~launchpad-pqm/launchpad/devel

CVE IMPORT PROCESS

==================

Launchpad has a cronscript (in "cronscripts/update-cve.py") that will fetch

the latest CVE database, in XML format, from the web, and then ensure that

the CVE data in it is correctly reflected in the local Launchpad database.

It takes care to reflect the correct description, status (Entry, Candidate,

or Deprecated), as well as all the References for that CVE entry.

This documentation test will describe and test the behaviour of this

infrastructure. Note that it assumes that the XML format is not changing,

and it does not attempt to connect to the outside world to verify that is

the case.

    >>> import subprocess, os.path, sys

    >>> import transaction

    >>> from lp.services.config import config

OK. So now lets import the first XML database. First, lets se how many CVE

entries are in the database.

    >>> from lp.bugs.model.cve import Cve

    >>> print Cve.select().count()

    10

    >>> script = os.path.join(config.root, 'cronscripts', 'update-cve.py')

    >>> url_base = os.path.join(

    ...     config.root, 'lib', 'lp', 'bugs', 'tests', 'data'

    ...     )

Now run the cronscript.

    >>> url = 'file://%s' % os.path.join(url_base, 'cvedb_init.xml.gz')

    >>> process = subprocess.Popen(

    ...     [sys.executable, script, '-u', url], stdin=subprocess.PIPE,

    ...     stdout=subprocess.PIPE, stderr=subprocess.STDOUT,

    ...     )

    >>> (output, empty) = process.communicate()

    >>> print output

    INFO    Creating lockfile: /var/lock/launchpad-updatecve.lock

    ...

    INFO    CVE-1999-0002 created

    INFO    Creating new SGI reference for 1999-0002

    INFO    Creating new BID reference for 1999-0002

    INFO    Creating new XF reference for 1999-0002

    INFO    CVE-1999-0003 created

    INFO    Creating new CERT reference for 1999-0003

    INFO    Creating new SGI reference for 1999-0003

    INFO    Creating new XF reference for 1999-0003

    INFO    Creating new XF reference for 1999-0003

    INFO    CVE-1999-0005 created

    INFO    Creating new CERT reference for 1999-0005

    INFO    Creating new BID reference for 1999-0005

    INFO    CVE-1999-0007 created

    INFO    Creating new CERT reference for 1999-0007

    INFO    Creating new XF reference for 1999-0007

    INFO    CVE-2005-2761 created

    INFO    Creating new MISC reference for 2005-2761

    INFO    Creating new DEBIAN reference for 2005-2761

    INFO    CVE-2005-2762 created

    INFO    CVE-2005-2763 created

    INFO    CVE-2005-2764 created

    ...

    <BLANKLINE>

And lets make sure we got the right number of CVE entries.

    >>> transaction.commit()

    >>> print Cve.select().count()

    18

We will make a note of the CVE modification time of 1999-0002. When we

update it later, we can use this modification time to check that its

modification time is being updated correctly.

    >>> c = Cve.bySequence('2005-2734')

    >>> mod_time = c.datemodified

And while we are here, make a note of the number of references for that CVE

entry.

    >>> print c.references.count()

    6

Now, let's run an import of the update db.

    >>> url = 'file://%s' % os.path.join(url_base, 'cvedb_update.xml.gz')

    >>> process = subprocess.Popen(

    ...     [sys.executable, script, '-u', url], stdin=subprocess.PIPE,

    ...     stdout=subprocess.PIPE, stderr=subprocess.STDOUT,

    ...     )

    >>> (output, empty) = process.communicate()

    >>> print output

    INFO    Creating lockfile: /var/lock/launchpad-updatecve.lock

    ...

    INFO    Creating new CERT reference for 1999-0002

    INFO    Creating new CIAC reference for 1999-0002

    INFO    Creating new NAI reference for 1999-0003

    INFO    Creating new SGI reference for 1999-0003

    INFO    Creating new BID reference for 1999-0003

    INFO    Creating new SUN reference for 1999-0005

    INFO    Creating new XF reference for 1999-0005

    INFO    CVE-1999-0006 created

    INFO    Creating new CERT reference for 1999-0006

    INFO    Creating new SGI reference for 1999-0006

    INFO    Creating new AUSCERT reference for 1999-0006

    INFO    Creating new XF reference for 1999-0006

    INFO    Creating new BID reference for 1999-0006

    INFO    CVE-1999-0007 updated description

    INFO    Creating new SECUNIA reference for 2005-2761

    INFO    CVE-2005-2734 updated description

    INFO    Removing BID reference for 2005-2734

    INFO    Removing BUGTRAQ reference for 2005-2734

    INFO    Removing CONFIRM reference for 2005-2734

    INFO    Removing SECTRACK reference for 2005-2734

    INFO    Removing SECUNIA reference for 2005-2734

    INFO    Removing XF reference for 2005-2734

    INFO    CVE-2005-2764 updated description

    INFO    CVE-2005-2765 created

    INFO    Creating new MSKB reference for 2005-2765

    INFO    CVE-2005-2766 created

    INFO    Creating new BUGTRAQ reference for 2005-2766

    ...

    <BLANKLINE>

Let's make sure we got the new CVE's.

    >>> transaction.commit()

    >>> print Cve.select().count()

    21

And lets make sure the modification time of 2005-2734 was updated, as were

the number of comments.

    >>> c.sync()

    >>> print mod_time < c.datemodified

    True

    >>> print c.references.count()

    0

Script class

------------

The class the cronscript uses to do its work is CVEUpdater.

    >>> from lp.bugs.scripts.cveimport import CVEUpdater

The method that does all the processing of the CVE XML is

processCVEXML(). It accepts that XML as a string. If no CVEs are found

in the XML, a LaunchpadScriptFailure is raised. This helps us catch

issues that causes no CVEs to be processed, for example if the XML

schema is changed.

    >>> cve_updater = CVEUpdater('cve-updater', test_args=[])

    >>> cve_updater.processCVEXML('<some-xml />')

    Traceback (most recent call last):

    ...

    LaunchpadScriptFailure: No CVEs found in XML file.