CVE IMPORT PROCESS
==================

Launchpad has a cronscript (in "cronscripts/update-cve.py") that will fetch
the latest CVE database, in XML format, from the web, and then ensure that
the CVE data in it is correctly reflected in the local Launchpad database.
It takes care to reflect the correct description, status (Entry, Candidate,
or Deprecated), as well as all the References for that CVE entry.

This documentation test will describe and test the behaviour of this
infrastructure. Note that it assumes that the XML format is not changing,
and it does not attempt to connect to the outside world to verify that is
the case.

    >>> import subprocess, os.path, sys
    >>> import transaction
    >>> from lp.services.config import config

OK. So now lets import the first XML database. First, lets se how many CVE
entries are in the database.

    >>> from lp.bugs.model.cve import Cve
    >>> print Cve.select().count()
    10

    >>> script = os.path.join(config.root, 'cronscripts', 'update-cve.py')
    >>> url_base = os.path.join(
    ...     config.root, 'lib', 'lp', 'bugs', 'tests', 'data'
    ...     )

Now run the cronscript.

    >>> url = 'file://%s' % os.path.join(url_base, 'cvedb_init.xml.gz')
    >>> process = subprocess.Popen(
    ...     [sys.executable, script, '-u', url], stdin=subprocess.PIPE,
    ...     stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
    ...     )
    >>> (output, empty) = process.communicate()
    >>> print output
    INFO    Creating lockfile: /var/lock/launchpad-updatecve.lock
    ...
    INFO    CVE-1999-0002 created
    INFO    Creating new SGI reference for 1999-0002
    INFO    Creating new BID reference for 1999-0002
    INFO    Creating new XF reference for 1999-0002
    INFO    CVE-1999-0003 created
    INFO    Creating new CERT reference for 1999-0003
    INFO    Creating new SGI reference for 1999-0003
    INFO    Creating new XF reference for 1999-0003
    INFO    Creating new XF reference for 1999-0003
    INFO    CVE-1999-0005 created
    INFO    Creating new CERT reference for 1999-0005
    INFO    Creating new BID reference for 1999-0005
    INFO    CVE-1999-0007 created
    INFO    Creating new CERT reference for 1999-0007
    INFO    Creating new XF reference for 1999-0007
    INFO    CVE-2005-2761 created
    INFO    Creating new MISC reference for 2005-2761
    INFO    Creating new DEBIAN reference for 2005-2761
    INFO    CVE-2005-2762 created
    INFO    CVE-2005-2763 created
    INFO    CVE-2005-2764 created
    ...
    <BLANKLINE>

And lets make sure we got the right number of CVE entries.

    >>> transaction.commit()
    >>> print Cve.select().count()
    18

We will make a note of the CVE modification time of 1999-0002. When we
update it later, we can use this modification time to check that its
modification time is being updated correctly.

    >>> c = Cve.bySequence('2005-2734')
    >>> mod_time = c.datemodified

And while we are here, make a note of the number of references for that CVE
entry.

    >>> print c.references.count()
    6

Now, let's run an import of the update db.

    >>> url = 'file://%s' % os.path.join(url_base, 'cvedb_update.xml.gz')
    >>> process = subprocess.Popen(
    ...     [sys.executable, script, '-u', url], stdin=subprocess.PIPE,
    ...     stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
    ...     )
    >>> (output, empty) = process.communicate()
    >>> print output
    INFO    Creating lockfile: /var/lock/launchpad-updatecve.lock
    ...
    INFO    Creating new CERT reference for 1999-0002
    INFO    Creating new CIAC reference for 1999-0002
    INFO    Creating new NAI reference for 1999-0003
    INFO    Creating new SGI reference for 1999-0003
    INFO    Creating new BID reference for 1999-0003
    INFO    Creating new SUN reference for 1999-0005
    INFO    Creating new XF reference for 1999-0005
    INFO    CVE-1999-0006 created
    INFO    Creating new CERT reference for 1999-0006
    INFO    Creating new SGI reference for 1999-0006
    INFO    Creating new AUSCERT reference for 1999-0006
    INFO    Creating new XF reference for 1999-0006
    INFO    Creating new BID reference for 1999-0006
    INFO    CVE-1999-0007 updated description
    INFO    Creating new SECUNIA reference for 2005-2761
    INFO    CVE-2005-2734 updated description
    INFO    Removing BID reference for 2005-2734
    INFO    Removing BUGTRAQ reference for 2005-2734
    INFO    Removing CONFIRM reference for 2005-2734
    INFO    Removing SECTRACK reference for 2005-2734
    INFO    Removing SECUNIA reference for 2005-2734
    INFO    Removing XF reference for 2005-2734
    INFO    CVE-2005-2764 updated description
    INFO    CVE-2005-2765 created
    INFO    Creating new MSKB reference for 2005-2765
    INFO    CVE-2005-2766 created
    INFO    Creating new BUGTRAQ reference for 2005-2766
    ...
    <BLANKLINE>

Let's make sure we got the new CVE's.

    >>> transaction.commit()
    >>> print Cve.select().count()
    21

And lets make sure the modification time of 2005-2734 was updated, as were
the number of comments.

    >>> c.sync()
    >>> print mod_time < c.datemodified
    True
    >>> print c.references.count()
    0


Script class
------------

The class the cronscript uses to do its work is CVEUpdater.

    >>> from lp.bugs.scripts.cveimport import CVEUpdater

The method that does all the processing of the CVE XML is
processCVEXML(). It accepts that XML as a string. If no CVEs are found
in the XML, a LaunchpadScriptFailure is raised. This helps us catch
issues that causes no CVEs to be processed, for example if the XML
schema is changed.

    >>> cve_updater = CVEUpdater('cve-updater', test_args=[])
    >>> cve_updater.processCVEXML('<some-xml />')
    Traceback (most recent call last):
    ...
    LaunchpadScriptFailure: No CVEs found in XML file.