~launchpad-pqm/launchpad/devel

= ExternalBugTracker: Trac =

This covers the implementation of the ExternalBugTracker class for Trac
bugwatches.


== Basics ==

The ExternalBugTracker descendant class which implements methods for updating
bug watches on Trac bug trackers is externalbugtracker.Trac, which implements
IExternalBugTracker.

    >>> from lp.bugs.externalbugtracker import Trac
    >>> from lp.bugs.tests.externalbugtracker import (
    ...     new_bugtracker)
    >>> from lp.services.webapp.testing import verifyObject
    >>> from lp.bugs.interfaces.bugtracker import BugTrackerType
    >>> from lp.bugs.interfaces.externalbugtracker import IExternalBugTracker
    >>> trac = Trac('http://example.com/')
    >>> verifyObject(IExternalBugTracker, trac)
    True


== LP plugin ==

Some Trac instances have a plugin installed to make it easier for us to
communicate with them. getExternalBugTrackerToUse() probes the bug
tracker for the special authentication mechanism the plugin uses, and
returns a TracLPPlugin if it's found.

If the LP plugin is installed, the URL will return 401, since it fails
to validate the token.

    >>> import urllib2
    >>> class TracHavingLPPlugin401(Trac):
    ...     def urlopen(self, url, data=None):
    ...         print url
    ...         raise urllib2.HTTPError(
    ...             url, 401, "Unauthorized", None, None)
    >>> trac = TracHavingLPPlugin401('http://example.com/')
    >>> chosen_trac = trac.getExternalBugTrackerToUse()
    http://example.com/launchpad-auth/check

    >>> from lp.bugs.externalbugtracker import (
    ...     TracLPPlugin)
    >>> isinstance(chosen_trac, TracLPPlugin)
    True
    >>> chosen_trac.baseurl
    'http://example.com'

Some Trac instances in the wild return HTTP 200 when the resource is
not found (HTTP 404 would be more appropriate). A distinguishing
difference between a 200 response from a broken Trac and a response
from a Trac with the plugin installed (when we've accidentally passed
a valid token, which is very unlikely), is that the broken Trac will
not include a "trac_auth" cookie.

    >>> from StringIO import StringIO
    >>> from httplib import HTTPMessage
    >>> from urllib import addinfourl

    >>> class TracReturning200ForPageNotFound(Trac):
    ...     def urlopen(self, url, data=None):
    ...         print url
    ...         return addinfourl(
    ...             StringIO(''), HTTPMessage(StringIO('')), url)

    >>> class TracHavingLPPlugin200(Trac):
    ...     def urlopen(self, url, data=None):
    ...         print url
    ...         return addinfourl(
    ...             StringIO(''), HTTPMessage(
    ...                 StringIO('Set-Cookie: trac_auth=1234')), url)

The plain, non-plugin, external bug tracker is selected for broken
Trac installations:

    >>> trac = TracReturning200ForPageNotFound('http://example.com/')
    >>> chosen_trac = trac.getExternalBugTrackerToUse()
    http://example.com/launchpad-auth/check

    >>> isinstance(chosen_trac, TracLPPlugin)
    False
    >>> chosen_trac.baseurl
    'http://example.com'

In the event that our deliberately bogus token is considered valid,
the external bug tracker that groks the plugin is selected:

    >>> trac = TracHavingLPPlugin200('http://example.com/')
    >>> chosen_trac = trac.getExternalBugTrackerToUse()
    http://example.com/launchpad-auth/check

    >>> isinstance(chosen_trac, TracLPPlugin)
    True
    >>> chosen_trac.baseurl
    'http://example.com'

If a 404 is returned, the normal Trac instance is returned.

    >>> class TracNotHavingLPPlugin(Trac):
    ...     def urlopen(self, url, data=None):
    ...         print url
    ...         raise urllib2.HTTPError(
    ...             url, 404, "Not found", None, None)
    >>> trac = TracNotHavingLPPlugin('http://example.com/')
    >>> chosen_trac = trac.getExternalBugTrackerToUse()
    http://example.com/launchpad-auth/check

    >>> chosen_trac is trac
    True

In the event that a connection error is returned, we return a normal Trac
instance. It will deal with the connection error later, if the situation
persists.

    >>> class TracWithConnectionError(Trac):
    ...     def urlopen(self, url, data=None):
    ...         print url
    ...         raise urllib2.URLError("Connection timed out")
    >>> trac = TracWithConnectionError('http://example.com/')
    >>> chosen_trac = trac.getExternalBugTrackerToUse()
    http://example.com/launchpad-auth/check

    >>> chosen_trac is trac
    True


== Status Conversion ==

The basic Trac ticket statuses map to Launchpad bug statuses.
Trac.convertRemoteStatus() handles the conversion.

    >>> trac = Trac('http://example.com/')
    >>> trac.convertRemoteStatus('open').title
    'New'
    >>> trac.convertRemoteStatus('new').title
    'New'
    >>> trac.convertRemoteStatus('reopened').title
    'New'
    >>> trac.convertRemoteStatus('accepted').title
    'Confirmed'
    >>> trac.convertRemoteStatus('assigned').title
    'Confirmed'
    >>> trac.convertRemoteStatus('fixed').title
    'Fix Released'
    >>> trac.convertRemoteStatus('closed').title
    'Fix Released'
    >>> trac.convertRemoteStatus('invalid').title
    'Invalid'
    >>> trac.convertRemoteStatus('wontfix').title
    "Won't Fix"
    >>> trac.convertRemoteStatus('duplicate').title
    'Confirmed'
    >>> trac.convertRemoteStatus('worksforme').title
    'Invalid'
    >>> trac.convertRemoteStatus('fixverified').title
    'Fix Released'

If the status isn't one that our Trac ExternalBugTracker can understand
an UnknownRemoteStatusError will be raised.

    >>> trac.convertRemoteStatus('eggs').title
    Traceback (most recent call last):
      ...
    UnknownRemoteStatusError: eggs


== Initialization ==

Calling initializeRemoteBugDB() on our Trac instance and passing it a set of
remote bug IDs will fetch those bug IDs from the server and file them in a
local variable for later use.

We use a test-oriented implementation for the purposes of these tests, which
overrides ExternalBugTracker.urlopen() so that we don't have to rely on a
working network connection.

    >>> from lp.bugs.tests.externalbugtracker import TestTrac
    >>> trac = TestTrac(u'http://test.trac/')
    >>> trac.initializeRemoteBugDB([1])
    >>> sorted(trac.bugs.keys())
    [1]

If we initialize with a different set of keys we overwrite the first set:

    >>> trac.initializeRemoteBugDB([6,7,8,9,10,11,12])
    >>> sorted(trac.bugs.keys())
    [6, 7, 8, 9, 10, 11, 12]


== Export Methods ==

There are two means by which we can export Trac bug statuses: on a bug-by-bug
basis and as a batch. When the number of bugs that need updating is less than
a given bug tracker's batch_query_threshold the bugs will be fetched
one-at-a-time:

    >>> trac.batch_query_threshold
    10

    >>> trac.trace_calls = True
    >>> trac.initializeRemoteBugDB([6, 7, 8, 9, 10])
    CALLED urlopen(u'http://test.trac/ticket/6?format=csv')
    CALLED urlopen(u'http://test.trac/ticket/7?format=csv')
    CALLED urlopen(u'http://test.trac/ticket/8?format=csv')
    CALLED urlopen(u'http://test.trac/ticket/9?format=csv')
    CALLED urlopen(u'http://test.trac/ticket/10?format=csv')

If there are more than batch_query_threshold bugs to update then they are
fetched as a batch:

    >>> trac.batch_query_threshold = 4
    >>> trac.initializeRemoteBugDB([6, 7, 8, 9, 10])
    CALLED urlopen(u'http://test.trac/query?id=6&id=7...&format=csv')

The batch updating method will also be used in cases where the Trac instance
doesn't support CSV exports of individual tickets:

    >>> trac.batch_query_threshold = 10
    >>> trac.supports_single_exports = False
    >>> trac.initializeRemoteBugDB([6, 7, 8, 9, 10])
    CALLED urlopen(u'http://test.trac/query?id=6&id=7...&format=csv')

If, when using the batch export method, the Trac instance comes across
invalid data, it will raise an UnparsableBugData exception. We will
force our trac instance to use invalid data for the purposes of this
test.

    >>> trac.csv_export_file = 'trac_example_broken_ticket_export.csv'
    >>> trac.initializeRemoteBugDB([6, 7, 8, 9, 10])
    Traceback (most recent call last):
      ...
    UnparsableBugData: External bugtracker http://test.trac does not
    define all the necessary fields for bug status imports (Defined
    field names: ['<html>']).

This is also true of the single bug export mode.

    >>> trac.supports_single_exports = True
    >>> trac.initializeRemoteBugDB([6])
    Traceback (most recent call last):
      ...
    UnparsableBugData: External bugtracker http://test.trac does not
    define all the necessary fields for bug status imports (Defined
    field names: ['<html>']).

Trying to get the remote status of the bug will raise a BugNotFound
error since the bug was never imported.

    >>> trac.getRemoteStatus(6)
    Traceback (most recent call last):
      ...
    BugNotFound: 6

Both the single and batch ticket import modes use the _fetchBugData()
method to retrieve the CSV data from the remote Trac instance. This
method accepts a URL from which to retrieve the data as a parameter.

    >>> trac.supports_single_exports = False
    >>> trac.csv_export_file = None

    >>> query_url = 'http://example.com/query?id=%s&format=csv'
    >>> query_string = '&id='.join(['1', '2', '3', '4', '5'])
    >>> query_url = query_url % query_string

    >>> remote_bugs = trac._fetchBugData(query_url)
    CALLED urlopen('http://example.com/query?id=1&id=2...&format=csv')

However, _fetchBugData() doesn't actually check the results it returns
except for checking that they are valid Trac CSV exports. in this case,
the IDs returned are nothing like the ones we asked for:

    >>> bug_ids = sorted(int(bug['id']) for bug in remote_bugs)
    >>> print bug_ids
    [1, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153]

If _fetchBugData() receives a response that isn't a valid Trac CSV
export, it will raise an UnparsableBugData error.

    >>> trac.csv_export_file = 'trac_example_broken_ticket_export.csv'
    >>> trac._fetchBugData(query_url)
    Traceback (most recent call last):
      ...
    UnparsableBugData: External bugtracker http://test.trac does not
    define all the necessary fields for bug status imports (Defined
    field names: ['<html>']).


== Updating Bug Watches ==

First, we create some bug watches to test with:

    >>> from lp.bugs.interfaces.bug import IBugSet
    >>> from lp.registry.interfaces.person import IPersonSet

    >>> sample_person = getUtility(IPersonSet).getByEmail(
    ...     'test@canonical.com')

    >>> example_bug_tracker = new_bugtracker(BugTrackerType.TRAC)

    >>> from lp.app.interfaces.launchpad import ILaunchpadCelebrities
    >>> example_bug = getUtility(IBugSet).get(10)
    >>> example_bugwatch = example_bug.addWatch(
    ...     example_bug_tracker, '1',
    ...     getUtility(ILaunchpadCelebrities).janitor)


Collect the Example.com watches:

    >>> for bug_watch in example_bug_tracker.watches:
    ...     print "%s: %s" % (bug_watch.remotebug, bug_watch.remotestatus)
    1: None

And have a Trac instance process them:

    >>> transaction.commit()

    >>> from lp.testing.layers import LaunchpadZopelessLayer
    >>> from lp.bugs.scripts.checkwatches import CheckwatchesMaster
    >>> txn = LaunchpadZopelessLayer.txn
    >>> bug_watch_updater = CheckwatchesMaster(txn)
    >>> trac = TestTrac(example_bug_tracker.baseurl)
    >>> bug_watch_updater.updateBugWatches(trac, example_bug_tracker.watches)
    INFO:...:Updating 1 watches for 1 bugs on http://bugs.some.where

    >>> for bug_watch in example_bug_tracker.watches:
    ...     print "%s: %s" % (bug_watch.remotebug, bug_watch.remotestatus)
    1: fixed

We'll add some more watches now.

    >>> from lp.bugs.interfaces.bugwatch import IBugWatchSet

    >>> bug_watch_set = getUtility(IBugWatchSet)
    >>> bug_watches = dict(
    ...     (int(bug_watch.remotebug), bug_watch.id)
    ...     for bug_watch in example_bug_tracker.watches)

    >>> for remote_bug, bug_watch_id in bug_watches.items():
    ...     bug_watch = getUtility(IBugWatchSet).get(bug_watch_id)
    ...     print "%s: %s" % (remote_bug, bug_watch.remotestatus)
    1: fixed

    >>> remote_bugs = [
    ...     (143, 'fixed'),
    ...     (144, 'assigned'),
    ...     (145, 'duplicate'),
    ...     (146, 'invalid'),
    ...     (147, 'worksforme'),
    ...     (148, 'wontfix'),
    ...     (149, 'reopened'),
    ...     (150, 'new'),
    ...     (151, 'new'),
    ...     (152, 'new'),
    ...     (153, 'new'),
    ... ]

    >>> for remote_bug_id, remote_status in remote_bugs:
    ...     bug_watch = bug_watch_set.createBugWatch(
    ...         bug=example_bug, owner=sample_person,
    ...         bugtracker=example_bug_tracker,
    ...         remotebug=str(remote_bug_id))
    ...     bug_watches[remote_bug_id] = bug_watch.id

    >>> trac.trace_calls = True
    >>> bug_watch_updater.updateBugWatches(trac, example_bug_tracker.watches)
    INFO:...:Updating 12 watches for 12 bugs on http://bugs.some.where
    CALLED urlopen(u'http://.../query?id=...

    >>> for remote_bug_id in sorted(bug_watches.keys()):
    ...     bug_watch = getUtility(IBugWatchSet).get(
    ...         bug_watches[remote_bug_id])
    ...     remote_status = bug_watch.remotestatus
    ...     print 'Remote bug %d: %s' % (remote_bug_id, remote_status)
    Remote bug 1: fixed
    Remote bug 143: fixed
    Remote bug 144: assigned
    Remote bug 145: duplicate
    Remote bug 146: invalid
    Remote bug 147: worksforme
    Remote bug 148: wontfix
    Remote bug 149: reopened
    Remote bug 150: new
    Remote bug 151: new
    Remote bug 152: new
    Remote bug 153: new

updateBugWatches() updates the lastchecked attribute on the watches, so
now no bug watches are in need of updating:

    >>> flush_database_updates()
    >>> example_bug_tracker.watches_needing_update.count()
    0

If the status isn't different, the lastchanged attribute doesn't get
updated. If we set a bug watch's lastchanged timestamp manually and call
update, lastchanged shouldn't be affected because the remote status of the bug
watch hasn't altered:

    >>> import pytz
    >>> from datetime import datetime, timedelta
    >>> from operator import attrgetter
    >>> sorted_bug_watches = sorted(
    ...     (bug_watch for bug_watch in example_bug_tracker.watches),
    ...     key=attrgetter('remotebug'))
    >>> bug_watch = sorted_bug_watches[-1]
    >>> now = datetime.now(pytz.timezone('UTC'))
    >>> bug_watch.lastchanged = now - timedelta(weeks=2)
    >>> old_last_changed = bug_watch.lastchanged
    >>> bug_watch.remotebug
    u'153'
    >>> bug_watch.remotestatus
    u'new'

    >>> trac.batch_query_threshold = 0
    >>> trac.trace_calls = False
    >>> bug_watch_updater.updateBugWatches(trac, [bug_watch])
    INFO:...:Updating 1 watches for 1 bugs on http://bugs.some.where

    >>> bug_watch.lastchanged == old_last_changed
    True
    >>> bug_watch.remotebug
    u'153'
    >>> bug_watch.remotestatus
    u'new'