~launchpad-pqm/launchpad/devel

ExternalBugTracker: TracLPPlugin
================================

This covers the implementation of the ExternalBugTracker class for Trac
instances having the LP XML-RPC plugin installed.

For testing purposes, a custom XML-RPC transport can be passed to it,
so that we can avoid network traffic in tests.

    >>> from lp.bugs.externalbugtracker import (
    ...     TracLPPlugin)
    >>> from lp.bugs.tests.externalbugtracker import (
    ...     TestTracXMLRPCTransport)
    >>> test_transport = TestTracXMLRPCTransport('http://example.com/')
    >>> trac = TracLPPlugin(
    ...     'http://example.com/', xmlrpc_transport=test_transport)
    >>> trac._xmlrpc_transport is test_transport
    True


Authentication
--------------

Before any XML-RPC methods can be used, we need to authenticate with the
Trac instance. To authenticate we create a special login token in
Launchpad. We then give that token to the Trac instance, which checks
whether the token is valid, and returns a cookie we can send with the
XML-RPC requests.

We give the token to Trac by issuing a HTTP request at
$base_url/launchpad-auth/$token. A request to such an URL will cause
Trac to validate $token and return a Set-Cookie header.

    >>> import random
    >>> from lp.services.verification.interfaces.logintoken import (
    ...     ILoginTokenSet)
    >>> from lp.services.webapp.url import urlappend
    >>> from lp.testing.dbuser import lp_dbuser

    >>> class FakeResponse:
    ...     def __init__(self):
    ...         self.headers = {}

    >>> class TestTracLPPlugin(TracLPPlugin):
    ...     def urlopen(self, url, data=None):
    ...         with lp_dbuser():
    ...             base_auth_url = urlappend(self.baseurl, 'launchpad-auth')
    ...             if not url.startswith(base_auth_url + '/'):
    ...                 raise AssertionError("Unexpected URL: %s" % url)
    ...             token_text = url.split('/')[-1]
    ...             token = getUtility(ILoginTokenSet)[token_text]
    ...             if token.tokentype.name != 'BUGTRACKER':
    ...                 raise AssertionError(
    ...                     'Invalid token type: %s' % token.tokentype.name)
    ...             if token.date_consumed is not None:
    ...                 raise AssertionError(
    ...                     "Token has already been consumed.")
    ...             token.consume()
    ...             print "Successfully validated the token."
    ...             cookie_string = (
    ...                 'trac_auth=random_token-' + str(random.random()))
    ...             self._xmlrpc_transport.setCookie(cookie_string)
    ...             response = FakeResponse()
    ...             response.headers['Set-Cookie'] = cookie_string
    ...
    ...         return response

To generate the token, the internal XML-RPC server is used. By using the
XML-RPC server rather than talking to the database directy means that we
don't have to bother about commiting the transaction to make the token
visible to Trac.

    >>> from cookielib import CookieJar
    >>> from lp.bugs.tests.externalbugtracker import (
    ...     TestInternalXMLRPCTransport)
    >>> cookie_jar = CookieJar()
    >>> test_transport = TestTracXMLRPCTransport(
    ...     'http://example.com/', cookie_jar)
    >>> trac = TestTracLPPlugin(
    ...     'http://example.com/', xmlrpc_transport=test_transport,
    ...     internal_xmlrpc_transport=TestInternalXMLRPCTransport(),
    ...     cookie_jar=cookie_jar)

The method that authenticates with Trac is _authenticate().

    >>> trac._authenticate()
    Using XML-RPC to generate token.
    Successfully validated the token.

After it has been called, the XML-RPC transport will have its
auth_cookie attribute set.

    >>> test_transport.cookie_processor.cookiejar
    <cookielib.CookieJar[Cookie(version=0, name='trac_auth'...

The XML-RPC transport's cookie_processor shares its cookiejar with the
TracLPPlugin instance. This is so that the TracLPPlugin can use the
cookiejar when authenticating with the remote Trac and then pass it to
the XML-RPC transport for further use, meaning that there's no need to
manually manipulate cookies.

    >>> test_transport.cookie_processor.cookiejar == trac._cookie_jar
    True

So if we look in the TracLPPlugin's CookieJar we'll see the same cookie:

    >>> trac._cookie_jar
    <cookielib.CookieJar[Cookie(version=0, name='trac_auth'...

And altering the cookie in the TracLPPlugin's CookieJar will mean, of
course, that it's altered in the XML-RPC transport's CookieJar, too.

    >>> from cookielib import Cookie
    >>> new_cookie = Cookie(
    ...     name="trac_auth",
    ...     value="Look ma, a new cookie!",
    ...     version=0, port=None, port_specified=False,
    ...     domain='http://example.com', domain_specified=True,
    ...     domain_initial_dot=None, path='', path_specified=False,
    ...     secure=False, expires=False, discard=None, comment=None,
    ...     comment_url=None, rest=None)

    >>> trac._cookie_jar.clear()
    >>> trac._cookie_jar.set_cookie(new_cookie)

    >>> trac._cookie_jar
    <cookielib.CookieJar[Cookie(version=0, name='trac_auth',
    value='Look ma, a new cookie!'...>

    >>> test_transport.cookie_processor.cookiejar
    <cookielib.CookieJar[Cookie(version=0, name='trac_auth',
    value='Look ma, a new cookie!'...>

If authentication fails, a BugTrackerAuthenticationError will be raised.

    >>> from urllib2 import HTTPError
    >>> class TestFailingTracLPPlugin(TracLPPlugin):
    ...     def urlopen(self, url, data=None):
    ...         raise HTTPError(url, 401, "Denied!", {}, None)

    >>> test_trac = TestFailingTracLPPlugin(
    ...     'http://example.com', xmlrpc_transport=test_transport,
    ...     internal_xmlrpc_transport=TestInternalXMLRPCTransport(),
    ...     cookie_jar=cookie_jar)
    >>> test_trac._authenticate()
    Traceback (most recent call last):
      ...
    BugTrackerAuthenticationError: http://example.com:
    HTTP Error 401: Denied!


Current time
------------

The current time is always returned in UTC, no matter if the Trac
instance returns another time zone.

    >>> test_transport = TestTracXMLRPCTransport('http://example.com/')
    >>> trac = TestTracLPPlugin(
    ...     'http://example.com/', xmlrpc_transport=test_transport,
    ...     internal_xmlrpc_transport=TestInternalXMLRPCTransport())

    >>> from datetime import datetime
    >>> # There doesn't seem to be a way to generate a UTC time stamp,
    >>> # without mocking around with the TZ environment variable.
    >>> datetime.utcfromtimestamp(1207706521)
    datetime.datetime(2008, 4, 9, 2, 2, 1)

    >>> HOUR = 60*60
    >>> test_transport.seconds_since_epoch = 1207706521 + HOUR
    >>> test_transport.local_timezone = 'CET'
    >>> test_transport.utc_offset = HOUR
    >>> trac.getCurrentDBTime()
    Using XML-RPC to generate token.
    Successfully validated the token.
    datetime.datetime(2008, 4, 9, 2, 2, 1, tzinfo=<UTC>)

An authorization request was automatically sent, since the method needed
authentication. Because the cookie is now set, other calls won't cause
an authorization request.

    >>> test_transport.auth_cookie
    Cookie(version=0, name='trac_auth'...)
    >>> trac.getCurrentDBTime()
    datetime.datetime(2008, 4, 9, 2, 2, 1, tzinfo=<UTC>)

If the cookie gets expired, an authorization request is automatically
sent again.

    >>> test_transport.expireCookie(test_transport.auth_cookie)
    >>> trac.getCurrentDBTime()
    Using XML-RPC to generate token.
    Successfully validated the token.
    datetime.datetime(2008, 4, 9, 2, 2, 1, tzinfo=<UTC>)


Getting modified bugs
---------------------

We only want to update the bug watches whose remote bugs have been
modified since the last time we checked.

In order to demonstrate this, we'll create some mock remote bugs for our
test XML-RPC transport to check.

    >>> from lp.bugs.tests.externalbugtracker import (
    ...     MockTracRemoteBug)

    >>> remote_bugs = {
    ...     '1': MockTracRemoteBug('1', datetime(2008, 4, 1, 0, 0, 0)),
    ...     '2': MockTracRemoteBug('2', datetime(2007, 1, 1, 1, 1, 1)),
    ...     '3': MockTracRemoteBug('3', datetime(2008, 1, 1, 1, 2, 3)),
    ...     }

    >>> test_transport.remote_bugs = remote_bugs

Calling the getModifiedRemoteBugs() method of our Trac instance and
passing it a list of bug IDs and a datetime object will return a list
of the IDs of the bugs which have been modified since that time.

    >>> bug_ids_to_check = ['1', '2', '3']
    >>> last_checked = datetime(2008, 1, 1, 0, 0, 0)
    >>> test_transport.expireCookie(test_transport.auth_cookie)
    >>> sorted(trac.getModifiedRemoteBugs(
    ...     bug_ids_to_check, last_checked))
    Using XML-RPC to generate token.
    Successfully validated the token.
    ['1', '3']

Different last_checked times will result in different numbers of bugs
being returned.

    >>> last_checked = datetime(2008, 2, 1, 0, 0, 0)
    >>> test_transport.expireCookie(test_transport.auth_cookie)
    >>> trac.getModifiedRemoteBugs(
    ...     bug_ids_to_check, last_checked)
    Using XML-RPC to generate token.
    Successfully validated the token.
    ['1']

If no bugs have been updated since last_checked, getModifiedRemoteBugs()
will return an empty list.

    >>> last_checked = datetime(2008, 5, 1, 0, 0, 0)
    >>> test_transport.expireCookie(test_transport.auth_cookie)
    >>> trac.getModifiedRemoteBugs(
    ...     bug_ids_to_check, last_checked)
    Using XML-RPC to generate token.
    Successfully validated the token.
    []

If we ask for bug ids that don't exist on the remote server, they will
also be returned. This is so that when we try to retrieve the status of
the missing bugs an error will be raised that we can then investigate.

    >>> bug_ids_to_check = ['1', '2', '3', '99', '100']
    >>> last_checked = datetime(2008, 1, 1, 0, 0, 0)
    >>> test_transport.expireCookie(test_transport.auth_cookie)
    >>> sorted(trac.getModifiedRemoteBugs(
    ...     bug_ids_to_check, last_checked))
    Using XML-RPC to generate token.
    Successfully validated the token.
    ['1', '100', '3', '99']


Getting the status of remote bugs
---------------------------------

Like all other ExternalBugTrackers, the TracLPPlugin ExternalBugTracker
allows us to fetch bugs statuses from the remote bug tracker.

To demonstrate this, we'll add some statuses to our mock remote bugs.

    >>> test_transport.remote_bugs['1'].status = 'open'
    >>> test_transport.remote_bugs['2'].status = 'fixed'
    >>> test_transport.remote_bugs['3'].status = 'reopened'

We need to call initializeRemoteBugDB() on our TracLPPlugin instance to
be able to retrieve remote statuses.

    >>> last_checked = datetime(2008, 1, 1, 0, 0, 0)
    >>> bugs_to_update = trac.getModifiedRemoteBugs(
    ...     bug_ids_to_check, last_checked)
    >>> test_transport.expireCookie(test_transport.auth_cookie)
    >>> trac.initializeRemoteBugDB(bugs_to_update)
    Using XML-RPC to generate token.
    Successfully validated the token.

Calling getRemoteStatus() on our example TracLPPlugin instance will
return the status for whichever bug we request.

    >>> trac.getRemoteStatus('1')
    'open'

    >>> trac.getRemoteStatus('3')
    'reopened'

If we try to get the status of bug 2 we'll get a BugNotFound error,
since that bug wasn't in the list of bugs that were modified since our
last_checked time.

    >>> trac.getRemoteStatus('2')
    Traceback (most recent call last):
      ...
    BugNotFound: 2


Importing Comments
------------------

The TracLPPlugin class allows Launchpad to import comments from remote
systems that have the Launchpad plugin installed.

TracLPPlugin implements the ISupportsCommentImport interface, providing
three methods: getCommentIds(), getPosterForComment() and
getMessageForComment().

    >>> from lp.bugs.interfaces.externalbugtracker import (
    ...     ISupportsCommentImport)
    >>> ISupportsCommentImport.providedBy(trac)
    True

We'll add some comments to our example bugs in order to demonstrate the
comment importing functionality.

    >>> import time
    >>> comment_datetime = datetime(2008, 4, 18, 17, 0, 0)
    >>> comment_timestamp = int(time.mktime(comment_datetime.timetuple()))

    >>> test_transport.remote_bugs['1'].comments = [
    ...     {'id': '1-1', 'type': 'comment',
    ...      'user': 'Test <test@canonical.com>',
    ...      'comment': 'Hello, world!',
    ...      'timestamp': comment_timestamp}]
    >>> test_transport.remote_bugs['2'].comments = [
    ...     {'id': '2-1', 'type': 'comment', 'user': 'test@canonical.com',
    ...      'comment': 'Hello again, world!',
    ...      'timestamp': comment_timestamp},
    ...     {'id': '2-2', 'type': 'comment', 'user': 'foo.bar',
    ...      'comment': 'More commentary.',
    ...      'timestamp': comment_timestamp}]

We also need an example Bug, BugTracker and BugWatch.

    >>> from lp.bugs.interfaces.bug import CreateBugParams
    >>> from lp.bugs.interfaces.bugtracker import BugTrackerType
    >>> from lp.registry.interfaces.person import IPersonSet
    >>> from lp.registry.interfaces.product import IProductSet
    >>> from lp.bugs.tests.externalbugtracker import (
    ...     new_bugtracker)

    >>> bug_tracker = new_bugtracker(BugTrackerType.TRAC)

    >>> with lp_dbuser():
    ...     sample_person = getUtility(IPersonSet).getByEmail(
    ...         'test@canonical.com')
    ...     firefox = getUtility(IProductSet).getByName('firefox')
    ...     bug = firefox.createBug(
    ...         CreateBugParams(sample_person, "Yet another test bug",
    ...             "Yet another test description.",
    ...             subscribe_owner=False))
    ...     bug_watch = bug.addWatch(bug_tracker, '1', sample_person)
    ...     bug_watch_two = bug.addWatch(bug_tracker, '2', sample_person)
    ...     bug_watch_three = bug.addWatch(bug_tracker, '3', sample_person)
    ...     bug_watch_broken = bug.addWatch(bug_tracker, '123', sample_person)

getCommentIds() returns all the comment IDs for a given remote bug.
bug_watch is against remote bug 1, which has one comment.

    >>> test_transport.expireCookie(test_transport.auth_cookie)
    >>> bugs_to_update = ['1', '2', '3']
    >>> trac.initializeRemoteBugDB(bugs_to_update)
    Using XML-RPC to generate token.
    Successfully validated the token.

    >>> trac.getCommentIds(bug_watch.remotebug)
    ['1-1']

bug_watch_two is against remote bug 2, which has two comments.

    >>> trac.getCommentIds(bug_watch_two.remotebug)
    ['2-1', '2-2']

bug_watch_three is against bug 3, which has no comments.

    >>> trac.getCommentIds(bug_watch_three.remotebug)
    []

Trying to call getCommentIds() on a bug that doesn't exist will raise a
BugNotFound error.

    >>> trac.getCommentIds(bug_watch_broken.remotebug)
    Traceback (most recent call last):
      ...
    BugNotFound: 123

The fetchComments() method is used to pre-load a given set of comments
for a given bug before they are parsed.

Before fetchComments() is called for a given remote bug, that remote
bug's 'comments' field will be a list of comment IDs.

    >>> trac.bugs[1]['comments']
    ['1-1']

After fetchComments() is called the bug's 'comments' field will contain
a dict in the form {<comment_id>: <comment_dict>}, which can then be
parsed.

    >>> remote_bug = bug_watch.remotebug
    >>> transaction.commit()

    >>> test_transport.expireCookie(test_transport.auth_cookie)
    >>> trac.fetchComments(remote_bug, ['1-1'])
    Using XML-RPC to generate token.
    Successfully validated the token.

    >>> for comment in trac.bugs[1]['comments'].values():
    ...     for key in sorted(comment.keys()):
    ...         print "%s: %s" % (key, comment[key])
    comment: Hello, world!
    id: 1-1
    timestamp: 1208518200
    type: comment
    user: Test <test@canonical.com>

getPosterForComment() returns a tuple of (displayname, emailaddress) for
the poster of a given comment.

    >>> trac.getPosterForComment(bug_watch.remotebug, '1-1')
    ('Test', 'test@canonical.com')

getPosterForComment() handles situations in which only an email address
is supplied for the 'user' field by returning None as the user's
displayname. When this is passed to IPersonSet.ensurePerson() a display
name will be generated for the user from their email address.

    >>> remote_bug = bug_watch_two.remotebug
    >>> transaction.commit()

    >>> trac.fetchComments(remote_bug, ['2-1', '2-2'])
    >>> trac.getPosterForComment(remote_bug, '2-1')
    (None, 'test@canonical.com')

getPosterForComment() will also return displayname, email tuples in
cases where the 'user' field is set to a plain username (e.g. 'foo').
However, in these cases it is the email address that will be set to
None.

    >>> trac.getPosterForComment(bug_watch_two.remotebug, '2-2')
    ('foo.bar', None)

Finally, getMessageForComment() will return a Message instance for a
given comment. For the sake of brevity we'll use test@canonical.com as
the comment's poster.

    >>> from zope.component import getUtility
    >>> poster = getUtility(IPersonSet).getByEmail('test@canonical.com')
    >>> message_one = trac.getMessageForComment(
    ...     bug_watch.remotebug, '1-1', poster)

The Message returned by getMessageForComment() contains the full text of
the original comment.

    >>> print message_one.text_contents
    Hello, world!

The owner of the comment is set to the Person passed to
getMessageForComment().

    >>> print message_one.owner.displayname
    Sample Person


Pushing comments
----------------

The TracLPPlugin ExternalBugTracker implements the
ISupportsCommentPushing interface, which allows Launchpad to use it to
push comments to the remote bug tracker.

    >>> from lp.bugs.interfaces.externalbugtracker import (
    ...     ISupportsCommentPushing)
    >>> ISupportsCommentPushing.providedBy(trac)
    True

ISupportsCommentPushing defines a method, addRemoteComment(), which is
responsible for pushing comments to the remote bug tracker. It accepts
two parameters: the ID of the remote bug to which to push the comment
and a Message instance containing the comment to be pushed. It returns
the ID assigned to the comment by the remote bug tracker.

To demonstrate this method, we'll create a comment to push.

    >>> from lp.services.messages.interfaces.message import IMessageSet
    >>> with lp_dbuser():
    ...     message = getUtility(IMessageSet).fromText(
    ...         "A subject", "An example comment to push.", poster)

Calling addRemoteComment() on our TracLPPlugin instance will push the
comment to the remote bug tracker. We'll add it to bug three on the
remote tracker, which as yet has no comments.

    >>> test_transport.remote_bugs['3'].comments
    []

addRemoteComment() requires authentication with the remote trac
instance. We'll expire our auth cookie to demonstrate this.

    >>> test_transport.expireCookie(test_transport.auth_cookie)

    >>> message_text_contents = message.text_contents
    >>> message_rfc822msgid = message.rfc822msgid
    >>> transaction.commit()

    >>> remote_comment_id = trac.addRemoteComment(
    ...     '3', message_text_contents, message_rfc822msgid)
    Using XML-RPC to generate token.
    Successfully validated the token.

    >>> print remote_comment_id
    3-1

If we look at our example remote server we can see that the comment has
been pushed to bug 3.

    >>> for comment in test_transport.remote_bugs['3'].comments:
    ...     for key in sorted(comment.keys()):
    ...         print "%s: %s" % (key, comment[key])
    comment: An example comment to push.
    id: 3-1
    time: ...
    type: comment
    user: launchpad


Linking remote bugs to Launchpad bugs
-------------------------------------

The TracLPPlugin class implements the ISupportsBackLinking interface,
which allows it to tell the remote bug tracker which Launchpad bug
links to a given one of its bugs.

    >>> from lp.bugs.interfaces.externalbugtracker import (
    ...     ISupportsBackLinking)
    >>> from zope.interface.verify import verifyObject
    >>> verifyObject(ISupportsBackLinking, trac)
    True

The getLaunchpadBugId() method will return the Launchpad bug ID for a
given remote bug. If no Launchpad bug has been linked to the remote bug,
getLaunchpadBugId() will return None.

getLaunchpadBugId() requires authentication.

    >>> test_transport.expireCookie(test_transport.auth_cookie)
    >>> launchpad_bug_id = trac.getLaunchpadBugId('3')
    Using XML-RPC to generate token.
    Successfully validated the token.

    >>> print launchpad_bug_id
    None

We call setLaunchpadBugId() to set the Launchpad bug ID for a remote
bug. setLaunchpadBugId() also requires authentication.

    >>> test_transport.expireCookie(test_transport.auth_cookie)
    >>> trac.setLaunchpadBugId('3', 15, 'http://bugs.launchpad.dev/bugs/xxx')
    Using XML-RPC to generate token.
    Successfully validated the token.

Calling getLaunchpadBugId() for remote bug 3 will now return 10, since
that's the Launchpad bug ID that we've just set.

    >>> print trac.getLaunchpadBugId('3')
    15

Passing a Launchpad bug ID of None to setLaunchpadBugId() will unset the
Launchpad bug ID for the remote bug.

    >>> trac.setLaunchpadBugId('3', None, None)
    >>> print trac.getLaunchpadBugId('3')
    None

If we try to call getLaunchpadBugId() or setLaunchpadBugId() for a
remote bug that doesn't exist, a BugNotFound error will be raised.

    >>> trac.getLaunchpadBugId('12345')
    Traceback (most recent call last):
      ...
    BugNotFound: 12345

    >>> trac.setLaunchpadBugId(
    ...     '12345', 1, 'http://bugs.launchpad.dev/bugs/xxx')
    Traceback (most recent call last):
      ...
    BugNotFound: 12345