~launchpad-pqm/launchpad/devel

Code Imports
============

CodeImport objects model the process surrounding the code import
service of Launchpad. A CodeImport object is created by a user
requesting an import, the import source is then reviewed by privileged
users. Then the code import servoce performs the initial import that
populates the import branch, and updates it regularly.

We can import code from CVS or Subversion.

To allow this test to modify CodeImports freely, we log in as a member
of the vcs-imports team.

    >>> login('david.allouche@canonical.com')


Code import set utility
-----------------------

CodeImports are created and found using the ICodeImportSet interface,
which is registered as a utility.

    >>> from lp.code.interfaces.branchtarget import IBranchTarget
    >>> from lp.code.interfaces.codeimport import ICodeImport, ICodeImportSet
    >>> from zope.component import getUtility
    >>> from zope.security.proxy import removeSecurityProxy
    >>> from lp.services.webapp.testing import verifyObject
    >>> code_import_set = getUtility(ICodeImportSet)
    >>> verifyObject(ICodeImportSet, removeSecurityProxy(code_import_set))
    True

CodeImports record who created them, so we're going to create a new
person with no special privileges.

    >>> nopriv = factory.makePerson(
    ...     displayname="Code Import Person", email="import@example.com",
    ...     name="import-person")


CodeImport events
-----------------

Most mutating operations affecting code imports should create
CodeImportEvent objects in the database to provide an audit trail.

    >>> from lp.code.interfaces.codeimportevent import ICodeImportEventSet
    >>> event_set = getUtility(ICodeImportEventSet)


Supported source systems
------------------------

The rcs_type field, which indicates whether the import is from CVS or
Subversion, takes values from the 'RevisionControlSystems' vocabulary.

    >>> from lp.code.enums import RevisionControlSystems
    >>> for item in RevisionControlSystems:
    ...     print item.title
    Concurrent Versions System
    Subversion via CSCVS
    Subversion via bzr-svn
    Git
    Mercurial
    Bazaar


Import from CVS
+++++++++++++++

Code imports from CVS specify the CVSROOT value, and the path to import
in the repository, known as the "module".

    >>> cvs = RevisionControlSystems.CVS
    >>> cvs_root = ':pserver:anonymous@cvs.example.com:/cvsroot'
    >>> cvs_module = 'hello'
    >>> target = IBranchTarget(factory.makeProduct(name='widget'))
    >>> cvs_import = code_import_set.new(
    ...     registrant=nopriv, target=target, branch_name='trunk-cvs',
    ...     rcs_type=cvs, cvs_root=cvs_root, cvs_module=cvs_module)
    >>> verifyObject(ICodeImport, removeSecurityProxy(cvs_import))
    True

When a new code import is created, an email is sent to the each of the
three members of the vcs-imports team.

    >>> import transaction
    >>> transaction.commit()
    >>> from lp.services.mail import stub
    >>> len(stub.test_emails)
    3
    >>> from lp.services.mail.helpers import get_contact_email_addresses
    >>> from lp.app.interfaces.launchpad import ILaunchpadCelebrities
    >>> vcs_imports = getUtility(ILaunchpadCelebrities).vcs_imports
    >>> len(get_contact_email_addresses(vcs_imports))
    3
    >>> import email
    >>> message = email.message_from_string(stub.test_emails[0][2])
    >>> print message['subject']
    New code import: widget/trunk-cvs
    >>> print message['X-Launchpad-Message-Rationale']
    Operator @vcs-imports
    >>> print message.get_payload(decode=True)
    A new CVS code import has been requested by Code Import Person:
        http://code.launchpad.dev/~import-person/widget/trunk-cvs
    from
        :pserver:anonymous@cvs.example.com:/cvsroot, hello
    <BLANKLINE>
    --
    You are getting this email because you are a member of the vcs-imports
    team.

Creating a CodeImport object creates a corresponding CodeImportEvent.

    >>> cvs_events = event_set.getEventsForCodeImport(cvs_import)
    >>> [event.event_type.name for event in cvs_events]
    ['CREATE']

The CodeImportSet is also able to retrieve the code imports with the
specified root and module.

    >>> existing_import = code_import_set.getByCVSDetails(
    ...     cvs_root=cvs_root, cvs_module=cvs_module)
    >>> cvs_import == existing_import
    True


Import from Subversion
++++++++++++++++++++++

Code imports from Subversion specify the URL used with "svn checkout" to
retrieve the tree to import.

    >>> svn = RevisionControlSystems.SVN
    >>> svn_url = 'svn://svn.example.com/trunk'
    >>> svn_import = code_import_set.new(
    ...     registrant=nopriv, target=target, branch_name='trunk-svn',
    ...     rcs_type=svn, url=svn_url)
    >>> verifyObject(ICodeImport, removeSecurityProxy(svn_import))
    True

Creating a CodeImport object creates a corresponding CodeImportEvent.

    >>> svn_events = event_set.getEventsForCodeImport(svn_import)
    >>> [event.event_type.name for event in svn_events]
    ['CREATE']

The CodeImportSet is also able to retrieve the code imports with the
specified subversion branch url.

    >>> existing_import = code_import_set.getByURL(svn_url)
    >>> svn_import == existing_import
    True


Import from Subversion via bzr-svn
++++++++++++++++++++++++++++++++++

Code imports from Subversion can also specify that they should be
imported with 'bzr-svn' rather than cscvs.  In most respects these
imports are similar to the Subversion via cscvs imports.

    >>> bzr_svn = RevisionControlSystems.BZR_SVN
    >>> bzr_svn_url = 'svn://svn.example.com/for-bzr-svn/trunk'
    >>> bzr_svn_import = code_import_set.new(
    ...     registrant=nopriv, target=target, branch_name='trunk-bzr-svn',
    ...     rcs_type=bzr_svn, url=bzr_svn_url)
    >>> verifyObject(ICodeImport, removeSecurityProxy(svn_import))
    True

The CodeImportSet.getBySVNDetails is also able to find bzr-svn
imports.

    >>> existing_bzr_svn_import = code_import_set.getByURL(bzr_svn_url)
    >>> bzr_svn_import == existing_bzr_svn_import
    True


Import from Git
+++++++++++++++

Code imports from Git specify the URL used with "git clone" to
retrieve the branch to import.

    >>> git = RevisionControlSystems.GIT
    >>> git_url = 'git://git.example.com/hello.git'
    >>> git_import = code_import_set.new(
    ...     registrant=nopriv, target=target, branch_name='trunk-git',
    ...     rcs_type=git, url=git_url)
    >>> verifyObject(ICodeImport, removeSecurityProxy(git_import))
    True

Creating a CodeImport object creates a corresponding CodeImportEvent.

    >>> git_events = event_set.getEventsForCodeImport(git_import)
    >>> [event.event_type.name for event in git_events]
    ['CREATE']

The CodeImportSet is also able to retrieve the code imports with the
specified git repo url.

    >>> existing_import = code_import_set.getByURL(git_url)
    >>> git_import == existing_import
    True

Import from Mercurial
+++++++++++++++++++++

Code imports from Mercurial specify the URL used with "hg clone" to
retrieve the branch to import.

    >>> hg = RevisionControlSystems.HG
    >>> hg_url = 'http://hg.example.com/metallic'
    >>> hg_import = code_import_set.new(
    ...     registrant=nopriv, target=target, branch_name='trunk-hg',
    ...     rcs_type=hg, url=hg_url)
    >>> verifyObject(ICodeImport, removeSecurityProxy(hg_import))
    True

Creating a CodeImport object creates a corresponding CodeImportEvent.

    >>> hg_events = event_set.getEventsForCodeImport(hg_import)
    >>> [event.event_type.name for event in hg_events]
    ['CREATE']

The CodeImportSet is also able to retrieve the code imports with the
specified hg repo url.

    >>> existing_import = code_import_set.getByURL(url=hg_url)
    >>> hg_import == existing_import
    True


Updating code import details
----------------------------

Members of the VCS Imports team (import operators), or Launchpad
administrators can update the details of the code import, including
the review status.  This is done using the code import method
'updateFromData'.  updateFromData returns a MODIFY CodeImportEvent if
any changes were made, or None if not.

    >>> code_import = factory.makeProductCodeImport(
    ...     svn_branch_url='http://svn.example.com/project')
    >>> print code_import.review_status.title
    Reviewed

When an import operator updates the code import emails are sent out to
the branch subscribers and members of VCS Imports that describe the
change.

The logged in user is normally subscribed to the new import as it is
created if done through the web UI, so we'll add nopriv here.

    >>> from lp.code.enums import (
    ...     BranchSubscriptionDiffSize,
    ...     BranchSubscriptionNotificationLevel,
    ...     CodeReviewNotificationLevel)
    >>> subscription = code_import.branch.subscribe(
    ...     nopriv,
    ...     BranchSubscriptionNotificationLevel.FULL,
    ...     BranchSubscriptionDiffSize.NODIFF,
    ...     CodeReviewNotificationLevel.FULL, nopriv)

    >>> from lp.testing.mail_helpers import (
    ...     pop_notifications, print_emails)
    >>> from lp.code.enums import CodeImportReviewStatus
    >>> ignore_old_emails = pop_notifications()
    >>> modify_event = code_import.updateFromData(
    ...     {'review_status': CodeImportReviewStatus.REVIEWED,
    ...      'url': 'http://svn.example.com/project/trunk'},
    ...     nopriv)
    >>> print_emails(group_similar=True)
    From: Code Import Person <import@example.com>
    To: david.allouche@canonical.com, ...
    Subject: Code import product.../name... status: Reviewed
    <BLANKLINE>
    ... is now being imported from:
        http://svn.example.com/project/trunk
    instead of:
        http://svn.example.com/project
    <BLANKLINE>
    -- =
    <BLANKLINE>
    http://code.launchpad.dev/~person.../product.../name...
    You are getting this email because you are a member of the vcs-imports
    team.
    <BLANKLINE>
    ----------------------------------------
    From: Code Import Person <import@example.com>
    To: import@example.com
    Subject: Code import product.../name... status: Reviewed
    <BLANKLINE>
    ... is now being imported from:
        http://svn.example.com/project/trunk
    instead of:
        http://svn.example.com/project
    <BLANKLINE>
    -- =
    <BLANKLINE>
    http://code.launchpad.dev/~person.../product.../name...
    You are receiving this email as you are subscribed to the branch.
    To unsubscribe from this branch go to .../+edit-subscription.
    <BLANKLINE>
    ----------------------------------------

updateFromData is smart enough to not send an email if no changes were
actually made.

    >>> code_import.updateFromData({}, nopriv)
    >>> print_emails(group_similar=True)

The person argument to updateFromData can be None, which is
appropriate for an automated change.  In that case, the email comes
from a 'noreply' address.

    >>> modify_event = code_import.updateFromData(
    ...     {'url': 'http://svn.example.org/project/trunk'},
    ...     None)
    >>> print_emails(group_similar=True)
    From: noreply@launchpad.net
    To: david.allouche@canonical.com, ...
    Subject: Code import product.../name... status: Reviewed
    ...
    From: noreply@launchpad.net
    To: import@example.com
    Subject: Code import product.../name... status: Reviewed
    ...


Update intervals
----------------

After an import is initially completed, it must be updated regularly. Each
code import can specify a custom update interval, or use a default value.

There is a separate default update interval for each version control system,
set in the Launchpad configuration system.

    >>> from lp.services.config import config
    >>> from datetime import timedelta
    >>> default_interval_cvs = timedelta(
    ...     seconds=config.codeimport.default_interval_cvs)
    >>> default_interval_subversion = timedelta(
    ...     seconds=config.codeimport.default_interval_subversion)
    >>> default_interval_git = timedelta(
    ...     seconds=config.codeimport.default_interval_git)
    >>> default_interval_hg = timedelta(
    ...     seconds=config.codeimport.default_interval_hg)

By default, code imports are created with an unspecified update interval.

    >>> print cvs_import.update_interval
    None
    >>> print svn_import.update_interval
    None

When the update interval interval is unspecified, the effective update
interval, which decides how often the import is actually updated, uses the
appropriate default value for the RCS type.

    >>> default_interval_cvs
    datetime.timedelta(0, 43200)
    >>> cvs_import.effective_update_interval
    datetime.timedelta(0, 43200)

    >>> default_interval_subversion
    datetime.timedelta(0, 21600)
    >>> svn_import.effective_update_interval
    datetime.timedelta(0, 21600)
    >>> bzr_svn_import.effective_update_interval
    datetime.timedelta(0, 21600)

    >>> default_interval_git
    datetime.timedelta(0, 21600)
    >>> git_import.effective_update_interval
    datetime.timedelta(0, 21600)

    >>> default_interval_hg
    datetime.timedelta(0, 21600)
    >>> hg_import.effective_update_interval
    datetime.timedelta(0, 21600)


If the update interval is set, then it overrides the default value.

As explained in the "Modify CodeImports" section, the interface does not allow
direct attribute modification. So we use removeSecurityProxy in this example.

    >>> removeSecurityProxy(cvs_import).update_interval = (
    ...     timedelta(seconds=7200))
    >>> cvs_import.effective_update_interval
    datetime.timedelta(0, 7200)

    >>> removeSecurityProxy(svn_import).update_interval = (
    ...     timedelta(seconds=3600))
    >>> svn_import.effective_update_interval
    datetime.timedelta(0, 3600)


Retrieving CodeImports
----------------------

You can retrive subsets of code imports with the `search` method of
ICodeImportSet.  Passing no arguments returns all code imports.

    >>> svn_import in code_import_set.search()
    True

You can filter the results by review status and by type.  For
instance, there is a single sample CodeImport with the "REVIEWED"
status:

    >>> reviewed_imports = list(code_import_set.search(
    ...     review_status=CodeImportReviewStatus.REVIEWED))
    >>> reviewed_imports
    [<...CodeImport...>]
    >>> reviewed_imports[0].review_status.name
    'REVIEWED'

And a single Git import.

    >>> git_imports = list(code_import_set.search(
    ...     rcs_type=RevisionControlSystems.GIT))
    >>> git_imports
    [<...CodeImport...>]
    >>> git_imports[0].rcs_type.name
    'GIT'

Passing both paramters is combined as "and".

    >>> reviewed_git_imports = list(code_import_set.search(
    ...     review_status=CodeImportReviewStatus.REVIEWED,
    ...     rcs_type=RevisionControlSystems.GIT))
    >>> reviewed_git_imports
    [<...CodeImport...>]
    >>> reviewed_git_imports[0].rcs_type.name
    'GIT'
    >>> reviewed_git_imports[0].review_status.name
    'REVIEWED'

You can also retrive an import by id and by branch, which will be used
to present the import's details on the page of the branch.

    >>> code_import_set.get(svn_import.id).url
    u'svn://svn.example.com/trunk'
    >>> code_import_set.getByBranch(cvs_import.branch).cvs_root
    u':pserver:anonymous@cvs.example.com:/cvsroot'

When you ask for an id that is not present ICodeImportSet.get() raises
lp.app.errors.NotFoundError, rather than some internal database exception.

    >>> code_import_set.get(-10)
    Traceback (most recent call last):
      ...
    NotFoundError: -10


Canonical URLs
--------------

We've registered the ICodeImportSet utility on the 'code' part of the
site:

    >>> from lp.services.webapp import canonical_url
    >>> print canonical_url(code_import_set)
    http://code.launchpad.dev/+code-imports

The code imports themselves have a canonical URL that is subordinate of
the branches, though they cannot currently be viewed that way in the webapp,
only over the API.

    >>> print canonical_url(svn_import.branch)
    http://code.launchpad.dev/~import-person/widget/trunk-svn
    >>> print canonical_url(svn_import)
    http://code.launchpad.dev/~import-person/widget/trunk-svn/+code-import


Modifying CodeImports
---------------------

Modifications to CodeImport objects must be done using setter methods
that create CodeImportEvent objects when appropriate. This is enforced
by preventing the setting of any attribute through the ICodeImport
interface.

Even though David can access CodeImportObjects, he cannot set attributes
on those objects.

    >>> login('david.allouche@canonical.com')
    >>> svn_import.url
    u'svn://svn.example.com/trunk'
    >>> svn_import.url = 'svn://svn.example.com/branch/1.0'
    Traceback (most recent call last):
      ...
    ForbiddenAttribute: ('url', <CodeImport ...>)

Modifications can be done using the CodeImport.updateFromData
method. If any change were made, this method creates and returns a
CodeImportEvent describing them. The CodeImportEvent records the user
that made the change, so we need to pass the user as an argument.

    >>> svn_import.url
    u'svn://svn.example.com/trunk'
    >>> data = {'url': 'svn://svn.example.com/branch/1.0'}
    >>> modify_event = svn_import.updateFromData(data, nopriv)
    >>> modify_event.event_type.name
    'MODIFY'
    >>> svn_import.url
    u'svn://svn.example.com/branch/1.0'
    >>> svn_events = event_set.getEventsForCodeImport(svn_import)
    >>> [event.event_type.name for event in svn_events]
    ['CREATE', 'MODIFY']

The launchpad.Edit privilege is required to use CodeImport.updateFromData.

    >>> login(ANONYMOUS)
    >>> svn_import.updateFromData({}, nopriv)
    Traceback (most recent call last):
    ...
    Unauthorized: (<CodeImport ...>, 'updateFromData', 'launchpad.Edit')

We saw above how changes to SVN details are displayed in emails above.
CVS details are displayed in a similar way.

    >>> from lp.code.mail.codeimport import (
    ...     make_email_body_for_code_import_update)
    >>> login('david.allouche@canonical.com')
    >>> data = {'cvs_root': ':pserver:anoncvs@cvs.example.com:/var/cvsroot'}
    >>> modify_event = cvs_import.updateFromData(data, nopriv)
    >>> print make_email_body_for_code_import_update(
    ...     cvs_import, modify_event, None)
    ~import-person/widget/trunk-cvs is now being imported from:
        hello from :pserver:anoncvs@cvs.example.com:/var/cvsroot
    instead of:
        hello from :pserver:anonymous@cvs.example.com:/cvsroot

For Git.

    >>> data = {'url': 'git://git.example.com/goodbye.git'}
    >>> modify_event = git_import.updateFromData(data, nopriv)
    >>> print make_email_body_for_code_import_update(
    ...     git_import, modify_event, None)
    ~import-person/widget/trunk-git is now being imported from:
        git://git.example.com/goodbye.git
    instead of:
        git://git.example.com/hello.git

Imports via bzr-svn are also similar.

    >>> data = {'url': 'http://svn.example.com/for-bzr-svn/trunk'}
    >>> modify_event = bzr_svn_import.updateFromData(data, nopriv)
    >>> print make_email_body_for_code_import_update(
    ...     bzr_svn_import, modify_event, None)
    ~import-person/widget/trunk-bzr-svn is now being imported from:
        http://svn.example.com/for-bzr-svn/trunk
    instead of:
        svn://svn.example.com/for-bzr-svn/trunk

And for Mercurial.

    >>> data = {'url': 'http://metal.example.com/byebye.hg'}
    >>> modify_event = hg_import.updateFromData(data, nopriv)
    >>> print make_email_body_for_code_import_update(
    ...     hg_import, modify_event, None)
    ~import-person/widget/trunk-hg is now being imported from:
        http://metal.example.com/byebye.hg
    instead of:
        http://hg.example.com/metallic

In addition, updateFromData can be used to set the branch whiteboard,
which is also described in the email that is sent.

    >>> data = {'whiteboard': 'stuff'}
    >>> modify_event = cvs_import.updateFromData(data, nopriv)
    >>> print make_email_body_for_code_import_update(
    ...     cvs_import, modify_event, 'stuff')
    The branch whiteboard was changed to:
    <BLANKLINE>
    stuff
    >>> print cvs_import.branch.whiteboard
    stuff

Setting the whiteboard to None is how it is deleted.

    >>> data = {'whiteboard': None}
    >>> modify_event = cvs_import.updateFromData(data, nopriv)
    >>> print make_email_body_for_code_import_update(
    ...     cvs_import, modify_event, '')
    The branch whiteboard was deleted.
    >>> print cvs_import.branch.whiteboard
    None