~launchpad-pqm/launchpad/devel

Gina Test
---------

This file is a simple test for gina. It uses a test archive (kept in
lp.soyuz.scripts.tests.archive_for_gina) and runs gina in
quiet mode over it.

Get the current counts of stuff in the database:

    >>> from lp.services.identity.model.emailaddress import EmailAddress
    >>> from lp.soyuz.interfaces.publishing import active_publishing_status
    >>> from lp.soyuz.model.publishing import (
    ...     BinaryPackagePublishingHistory,
    ...     SourcePackagePublishingHistory)
    >>> from lp.registry.model.person import Person, WikiName
    >>> from lp.registry.model.teammembership import TeamParticipation
    >>> from lp.registry.interfaces.pocket import (
    ...     PackagePublishingPocket)
    >>> from lp.soyuz.model.binarypackagebuild import BinaryPackageBuild
    >>> from lp.soyuz.model.binarypackagerelease import BinaryPackageRelease
    >>> from lp.soyuz.model.sourcepackagerelease import SourcePackageRelease
    >>> SSPPH = SourcePackagePublishingHistory
    >>> SBPPH = BinaryPackagePublishingHistory

    >>> orig_spr_count = SourcePackageRelease.select().count()
    >>> orig_sspph_count = SSPPH.select().count()
    >>> orig_person_count = Person.select().count()
    >>> orig_tp_count = TeamParticipation.select().count()
    >>> orig_email_count = EmailAddress.select().count()
    >>> orig_wiki_count = WikiName.select().count()
    >>> orig_bpr_count = BinaryPackageRelease.select().count()
    >>> orig_build_count = BinaryPackageBuild.select().count()
    >>> orig_sbpph_count = SBPPH.select().count()
    >>> orig_sspph_main_count = SSPPH.selectBy(
    ...     componentID=1, pocket=PackagePublishingPocket.RELEASE).count()

Create a distribution release and an arch release for breezy:

    >>> from lp.soyuz.model.distroarchseries import DistroArchSeries
    >>> from lp.soyuz.model.processor import ProcessorFamily
    >>> from lp.app.interfaces.launchpad import ILaunchpadCelebrities
    >>> celebs = getUtility(ILaunchpadCelebrities)
    >>> ubuntu = celebs.ubuntu
    >>> hoary = ubuntu.getSeries("hoary")

    # Only the distro owner and admins can create a new series.
    >>> login_person(ubuntu.owner.activemembers[0])
    >>> breezy = ubuntu.newSeries(
    ...     "breezy", "Breezy Badger", "My title",
    ...     "My summary", "My description", "5.10",
    ...     hoary, celebs.launchpad_developers)
    >>> login(ANONYMOUS)

    # Enable the Derived Series feature flag, to make sure gina likes it.
    >>> from lp.services.features.testing import FeatureFixture
    >>> from lp.soyuz.model.distroseriesdifferencejob import (
    ...     FEATURE_FLAG_ENABLE_MODULE,
    ...     )
    >>> fixture = FeatureFixture({FEATURE_FLAG_ENABLE_MODULE: u'on'})
    >>> fixture.setUp()
    >>> pf = ProcessorFamily.selectOneBy(name="x86")
    >>> breezy_i386 = DistroArchSeries(distroseries=breezy,
    ...                         processorfamily=pf,
    ...                         architecturetag="i386", official=True,
    ...                         owner=celebs.launchpad_developers)
    >>> import transaction
    >>> transaction.commit()

Now, lets run gina on hoary and breezy. This test imports a few
packages successfully (at least partially):

   * archive-copier, a source package which generates one udeb
     in debian-installer. Its maintainer has a name which contains a ","
   * archive-copier, again, with a different version number to see that
     both versions get correctly imported.
   * db1-compat, source package what generates 1 binary package. The same
     version was included in both hoary and breezy, but its source
     package's section and its binary package's priority were changed in
     breezy.
   * gcc-defaults, a source package that generates 8 binary packages with
     differing versions.
   * x11proto-damage, a package which is only present in breezy
   * libcap, a source package which generates 3 binary packages, and
     whose version number contains an epoch. It is not in the Breezy
     Sources list, but some binaries are in the Packages file. However, these
     binaries are unchaged in Breezy.
   * ubuntu-meta, a source package that generates 3 binary packages in
     Hoary and 5 in breezy. However, its breezy version is /not/ listed in the
     Sources list, so the binary packages will need to discover it.
   * ed, a source package what generates one binary package and
     misses a section entry in Sources. The same version exists in
     breezy, this time with a defined section. Its hoary binary package
     lacks a Priority.
   * python-sqllite, an arch-independent source package that generates
     one binary package. Its breezy packages are missing from the archive.
   * python-pam, an arch-independent source package that generates one
     binary package, whose changelog contains a busted urgency. Its hoary
     binary package contains lacks a Section. Its breezy packages are missing
     from the archive.
   * mkvmlinuz, a source package that generates one binary package,
     but which is missing a version field in its Sources file.
     Its breezy package has a version but is missing copyright and changelog.
   * 3dchess, a source package that generates a binary package.
   * 9wm, a source package that generates a binary package, and whose
     changelog file name starts with the binary package name.
   * rioutil, a source package that generates a binary package, and
     whose source contains a shlibs file. Its current binary package is
     actually an evil bin-only-NMU!

And two completely broken packages:

   * util-linux, a source package that is missing from the pool. It
     generates 4 binary packages, all missing. It's correctly listed in
     Sources and Packages, though.

   * clearlooks, a source package with no binaries listed, and which has
     a DSC file that refers to an inexistant tar.gz.

Let's set up the filesystem:

    >>> import subprocess, sys, os
    >>> try:
    ...     os.unlink("/var/lock/launchpad-gina.lock")
    ... except OSError:
    ...     pass
    >>> try:
    ...     os.remove('/tmp/gina_test_archive')
    ... except OSError:
    ...     pass
    >>> relative_path = ('lib/lp/soyuz/scripts/tests/gina_test_archive')
    >>> path = os.path.join(os.getcwd(), relative_path)
    >>> os.symlink(path, '/tmp/gina_test_archive')

And give her a spin:

    >>> gina_proc = [sys.executable, 'scripts/gina.py', '-q',
    ...              'hoary', 'breezy']
    >>> proc = subprocess.Popen(gina_proc, stderr=subprocess.PIPE)

Check STDERR for the errors we expected:

    >>> print proc.stderr.read()
    ERROR   Error processing package files for clearlooks
     -> http://...Error 2 unpacking source)
    WARNING Invalid format in db1-compat, assumed '1.0'
    WARNING Source package ed lacks section, assumed 'misc'
    ERROR   Unable to create SourcePackageData for mkvmlinuz
     -> http://...version: None)
    WARNING Invalid urgency in python-pam, None, assumed 'low'
    ERROR   Error processing package files for util-linux
     -> http://...dsc not in archive)
    ERROR   Error processing package files for bsdutils
     -> http://...deb not found)
    WARNING Binary package ed lacks valid priority, assumed 'extra'
    ERROR   Unable to create BinaryPackageData for mount
     -> http://...invalid version...)
    WARNING Binary package python-pam lacks a section, assumed 'misc'
    ERROR   Error processing package files for python2.4-pam
     -> http://...deb not found)
    ERROR   Error processing package files for python2.4-sqlite
     -> http://...deb not found)
    WARNING No source package rioutil (1.4.4-1.0.1) listed for rioutil (1.4.4-1.0.1), scrubbing archive...
    WARNING Nope, couldn't find it. Could it be a bin-only-NMU? Checking...
    ERROR   Error processing package files for util-linux
     -> http://...deb not found)
    ERROR   Unable to create BinaryPackageData for util-linux-locales
     -> http://...installed_size'])
    ERROR   Invalid Sources stanza in /tmp/tmp...
     -> http://...bogus\n')
    WARNING No changelog file found for mkvmlinuz in mkvmlinuz-14ubuntu1
    WARNING No copyright file found for mkvmlinuz in mkvmlinuz-14ubuntu1
    WARNING Invalid urgency in mkvmlinuz, None, assumed 'low'
    ERROR   Error processing package files for python-sqlite
     -> http://...dsc not in archive)
    ERROR   Error processing package files for util-linux
     -> http://...dsc not in archive)
    ERROR   Error processing package files for python-sqlite
     -> http://...deb not found)
    WARNING No source package ubuntu-meta (0.80) listed for ubuntu-base (0.80), scrubbing archive...
    <BLANKLINE>

The exit status must be 0, for success:

    >>> proc.wait()
    0
    >>> transaction.commit()
    >>> fixture.cleanUp()


Testing Source Package Results
..............................

We should have more source packages in the database:

    >>> existing = 9
    >>> hc = 13 - 2   # 2 packages fail
    >>> bc = 9 - 3 - 2 + 1  # 3 packages are the same as in hoary, 2 fail,
    ... #one that is imported forcefully (ubuntu-meta)
    >>> hc + bc
    16
    >>> count = SourcePackageRelease.select().count()
    >>> count - orig_spr_count
    17

Check that x11proto-damage has its Build-Depends-Indep value correctly set:

    >>> from lp.registry.model.sourcepackagename import SourcePackageName
    >>> n = SourcePackageName.selectOneBy(name="x11proto-damage")
    >>> x11p = SourcePackageRelease.selectOneBy(sourcepackagenameID=n.id,
    ...                                         version="6.8.99.7-2")

    >>> print x11p.builddependsindep
    debhelper (>= 4.0.0)

Check if the changelog message was stored correcly:

    >>> print x11p.changelog_entry #doctest: -NORMALIZE_WHITESPACE
    x11proto-damage (6.8.99.7-2) breezy; urgency=low
    <BLANKLINE>
    <BLANKLINE>
      * Add dependency on x11proto-fixes-dev.
    <BLANKLINE>
     -- Daniel Stone <daniel.stone@ubuntu.com>  Mon, 11 Jul 2005 19:11:11 +1000

    >>> from lp.registry.interfaces.sourcepackage import SourcePackageUrgency
    >>> x11p.urgency == SourcePackageUrgency.LOW
    True

Check that the changelog was uploaded to the librarian correctly:

    >>> print x11p.changelog.read()
    x11proto-damage (6.8.99.7-2) breezy; urgency=low
    <BLANKLINE>
      * Add dependency on x11proto-fixes-dev.
    <BLANKLINE>
     -- Daniel Stone <daniel.stone@ubuntu.com>  Mon, 11 Jul 2005 19:11:11 +1000
    <BLANKLINE>
    x11proto-damage (6.8.99.7-1) breezy; urgency=low
    <BLANKLINE>
      * First x11proto-damage release.
    <BLANKLINE>
     -- Daniel Stone <daniel.stone@ubuntu.com>  Mon, 16 May 2005 22:10:17 +1000

Same for the copyright:

    >>> print x11p.copyright
    $Id: COPYING,v 1.2 2003/11/05 05:39:58 keithp Exp $
    <BLANKLINE>
    Copyright ... 2003 Keith Packard
    ...
    PERFORMANCE OF THIS SOFTWARE.

Check that the dsc on the libcap package is correct, and that we
only imported one:

    >>> n = SourcePackageName.selectOneBy(name="libcap")
    >>> cap = SourcePackageRelease.selectOneBy(sourcepackagenameID=n.id)
    >>> print cap.dsc
    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1
    <BLANKLINE>
    Format: 1.0
    Source: libcap
    Version: 1:1.10-14
    Binary: libcap-dev, libcap-bin, libcap1
    Maintainer: Michael Vogt <mvo@debian.org>
    Architecture: any
    Standards-Version: 3.6.1
    Build-Depends: debhelper
    Files:
     291be97b78789f331499a0ab22d9d563 28495 libcap_1.10.orig.tar.gz
     b867a0c1db9e8ff568415bbcd1fa65dc 12928 libcap_1.10-14.diff.gz
    <BLANKLINE>
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.2.4 (GNU/Linux)
    <BLANKLINE>
    iD8DBQFAfGV8liSD4VZixzQRAlHoAJ4hD8yDp/VIJUcdQLLr9KH/XQSczQCfQH/D
    FVJMGmGr+2YLZfF+oRUKcug=
    =bw+A
    -----END PGP SIGNATURE-----
    >>> print cap.maintainer.displayname
    Michael Vogt
    >>> print cap.dsc_binaries
    libcap-dev, libcap-bin, libcap1

Test ubuntu-meta in breezy, which was forcefully imported.

    >>> n = SourcePackageName.selectOneBy(name="ubuntu-meta")
    >>> um = SourcePackageRelease.selectOneBy(sourcepackagenameID=n.id,
    ...         version="0.80")
    >>> print um.section.name, um.architecturehintlist, \
    ...         um.upload_distroseries.name
    base any breezy

And check that its files actually ended up in the librarian (these sha1sums
were calculated directly on the files):

    >>> from lp.soyuz.model.files import SourcePackageReleaseFile
    >>> files = SourcePackageReleaseFile.selectBy(
    ...     sourcepackagereleaseID=cap.id, orderBy="libraryfile")
    >>> for f in files:
    ...     print f.libraryfile.content.sha1
    107d5478e72385f714523bad5359efedb5dcc8b2
    0083da007d44c02fd861c1d21579f716490cab02
    e6661aec051ccb201061839d275f2282968d8b93

Check that the section on the python-pam package is correct, and that we
only imported one:

    >>> n = SourcePackageName.selectOneBy(name="python-pam")
    >>> pp = SourcePackageRelease.selectOneBy(sourcepackagenameID=n.id)
    >>> print pp.component.name
    main

In the hoary Sources, its section is listed as underworld/python. Ensure
this is cut up correctly:

    >>> print pp.section.name
    python

Make sure that we only imported one db1-compat source package.

    >>> n = SourcePackageName.selectOneBy(name="db1-compat")
    >>> db1 = SourcePackageRelease.selectOneBy(sourcepackagenameID=n.id)
    >>> print db1.section.name
    libs


Testing Source Package Publishing
.................................

We check that the source package publishing override facility works:

    >>> for pub in SSPPH.selectBy(
    ...     sourcepackagereleaseID=db1.id, orderBy='distroseries'):
    ...     print "%s %s %s" % (
    ...         pub.distroseries.name, pub.section.name,
    ...         pub.archive.purpose.name)
    hoary libs PRIMARY
    breezy oldlibs PRIMARY

We should have one entry for each package listed in Sources that was
successfully processed.

    - We had 2 errors (out of 10 Sources stanzas) in hoary: mkvmlinuz and
      util-linux.

    - We had 2 errors (out of 10 Sources stanzas) in breezy: python-sqllite
      and util-linux (again, poor thing).

    >>> print SSPPH.select().count() - orig_sspph_count
    21

    >>> new_count = SSPPH.selectBy(
    ...     componentID=1,
    ...     pocket=PackagePublishingPocket.RELEASE).count()
    >>> print new_count - orig_sspph_main_count
    21


Testing Binary Package Results
..............................

We have 26 binary packages in hoary. The 4 packages for util-linux fail, and 1
package fails for each of python-sqlite and python-pam. We should publish one
entry for each package listed in Releases.

We have 23 binary packages in breezy. db1-compat, ed, the 3 libcap packages
and python-pam is unchanged.  python-sqlite fails. The 5 ubuntu-meta packages
work.

    >>> BinaryPackageRelease.select().count() - orig_bpr_count
    40
    >>> BinaryPackageBuild.select().count() - orig_build_count
    13
    >>> SBPPH.select().count() - orig_sbpph_count
    46

Check that the shlibs parsing and bin-only-NMU version handling works as
expected:

    >>> from lp.soyuz.model.binarypackagename import BinaryPackageName
    >>> n = BinaryPackageName.selectOneBy(name="rioutil")
    >>> rio = BinaryPackageRelease.selectOneBy(binarypackagenameID=n.id)
    >>> print rio.shlibdeps
    librioutil 1 rioutil
    >>> print rio.version
    1.4.4-1.0.1
    >>> print rio.build.source_package_release.version
    1.4.4-1

Test all the data got to the ed BPR intact, and that the missing
priority was correctly munged to "extra":

    >>> n = BinaryPackageName.selectOneBy(name="ed")
    >>> ed = BinaryPackageRelease.selectOneBy(binarypackagenameID=n.id)
    >>> print ed.version
    0.2-20
    >>> print ed.build.processor.name
    386
    >>> print ed.build.status
    Successfully built
    >>> print ed.build.distro_arch_series.processorfamily.name
    x86
    >>> print ed.build.distro_arch_series.architecturetag
    i386
    >>> print ed.priority
    Extra
    >>> print ed.section.name
    editors
    >>> print ed.summary
    The classic unix line editor.

We now check if the Breezy publication record has the correct priority:

    >>> ed_pub = SBPPH.selectOneBy(binarypackagereleaseID=ed.id,
    ...                            distroarchseriesID=breezy_i386.id)
    >>> print ed_pub.priority
    Standard

Check binary package libgjc-dev in Breezy. Its version number must differ from
its source version number.

    >>> n = BinaryPackageName.selectOneBy(name="libgcj-dev")
    >>> lib = BinaryPackageRelease.selectOneBy(binarypackagenameID=n.id,
    ...         version="4:4.0.1-3")
    >>> print lib.version
    4:4.0.1-3
    >>> print lib.build.source_package_release.version
    1.28
    >>> print lib.build.source_package_release.maintainer.displayname
    Debian GCC maintainers

Check if the udeb was properly parsed and identified:

    >>> n = BinaryPackageName.selectOneBy(name="archive-copier")
    >>> ac = BinaryPackageRelease.selectOneBy(binarypackagenameID=n.id,
    ...         version="0.1.5")
    >>> print ac.version
    0.1.5
    >>> print ac.priority
    Standard
    >>> print ac.section.name
    debian-installer
    >>> print ac.build.source_package_release.version
    0.1.5
    >>> print ac.build.source_package_release.maintainer.name
    cjwatson
    >>> print ac.build.processor.name
    386

We check that the binary package publishing override facility works:

    >>> n = BinaryPackageName.selectOneBy(name="libdb1-compat")
    >>> db1 = BinaryPackageRelease.selectOneBy(binarypackagenameID=n.id,
    ...         version="2.1.3-7")
    >>> for pub in BinaryPackagePublishingHistory.selectBy(
    ...     binarypackagereleaseID=db1.id, orderBy='distroarchseries'):
    ...     print "%s %s %s" % (
    ...         pub.distroarchseries.distroseries.name, pub.priority,
    ...         pub.archive.purpose.name)
    hoary Required PRIMARY
    breezy Optional PRIMARY

XXX: test package with invalid source version
XXX: test package with maintainer with non-ascii name


Testing People Created
......................

Ensure only one Kamion was created (he's an uploader on multiple packages),
and that we imported exactly 9 people (13 packages with 3 being uploaded by
Kamion, 2 being uploaded by mdz and 2 by doko).

    >>> from sqlobject import LIKE
    >>> p = Person.selectOne(LIKE(Person.q.name, u"cjwatson%"))
    >>> print p.name
    cjwatson
    >>> print Person.select().count() - orig_person_count
    13
    >>> print TeamParticipation.select().count() - orig_tp_count
    13
    >>> print EmailAddress.select().count() - orig_email_count
    13


Re-run Gina
...........

The second run of gina uses a test archive that is a copy of the first
one, but with updated Packages and Sources files for breezy that do
three important changes, implemented as publishing entries (or
overrides):

    - Binary package ed changed priority from 30 to 10 (extra) in i386
    - Source package x11proto-damage changed section from "x11" to "net"
    - Source package archive-copier has been moved from component "main"
      to "universe".

Link to the "later" archive:

    >>> os.remove('/tmp/gina_test_archive')
    >>> relative_path = ('lib/lp/soyuz/scripts/'
    ...                  'tests/gina_test_archive_2nd_run')
    >>> path = os.path.join(os.getcwd(), relative_path)
    >>> os.symlink(path, '/tmp/gina_test_archive')

We do a re-run over the same components. We should get ERRORs indicating
packages that failed to import the last time. Overrides should also have
been updated for packages in breezy which have changed since the last
run.

    >>> gina_proc = [sys.executable, 'scripts/gina.py', '-q',
    ...              'hoary', 'breezy']
    >>> proc = subprocess.Popen(gina_proc, stderr=subprocess.PIPE)
    >>> print proc.stderr.read()
    ERROR   Error processing package files for clearlooks
     -> http://...Error 2 unpacking source)
    WARNING Source package ed lacks section, assumed 'misc'
    ERROR   Unable to create SourcePackageData for mkvmlinuz
     -> http://...version: None)
    ERROR   Error processing package files for util-linux
     -> http://...dsc not in archive)
    ERROR   Error processing package files for bsdutils
     -> http://...deb not found)
    WARNING Binary package ed lacks valid priority, assumed 'extra'
    ERROR   Unable to create BinaryPackageData for mount
     -> http://...invalid version...)
    WARNING Binary package python-pam lacks a section, assumed 'misc'
    ERROR   Error processing package files for python2.4-pam
     -> http://...deb not found)
    ERROR   Error processing package files for python2.4-sqlite
     -> http://...deb not found)
    ERROR   Error processing package files for util-linux
     -> http://...deb not found)
    ERROR   Unable to create BinaryPackageData for util-linux-locales
     -> http://...installed_size'])
    ERROR   Invalid Sources stanza in /tmp/tmp...
     -> http://...bogus\n')
    ERROR   Error processing package files for python-sqlite
     -> http://...dsc not in archive)
    ERROR   Error processing package files for util-linux
     -> http://...dsc not in archive)
    ERROR   Error processing package files for python-sqlite
     -> http://...deb not found)
    <BLANKLINE>
    >>> proc.wait()
    0
    >>> transaction.commit()

Nothing should happen to most of our data -- no counts should have
changed, etc.

    >>> SourcePackageRelease.select().count() - orig_spr_count
    17
    >>> print Person.select().count() - orig_person_count
    13
    >>> print TeamParticipation.select().count() - orig_tp_count
    13
    >>> print EmailAddress.select().count() - orig_email_count
    13
    >>> BinaryPackageRelease.select().count() - orig_bpr_count
    40
    >>> BinaryPackageBuild.select().count() - orig_build_count
    13

But the overrides do generate extra publishing entries:

    >>> SBPPH.select().count() - orig_sbpph_count
    47
    >>> print SSPPH.select().count() - orig_sspph_count
    23

Check that the overrides we did were correctly issued. We can't use
selectOneBy because, of course, there may be multiple rows published for that
package -- that's what overrides actually do.

    >>> from lp.services.database.sqlbase import sqlvalues
    >>> x11_pub = SSPPH.select("""
    ...     sourcepackagerelease = %s AND
    ...     distroseries = %s AND
    ...     status in %s
    ...     """ % sqlvalues(
    ...         x11p, breezy, active_publishing_status),
    ...     orderBy=["-datecreated"])[0]
    >>> print x11_pub.section.name
    net
    >>> ed_pub = SBPPH.select("""
    ...     binarypackagerelease = %s AND
    ...     distroarchseries = %s AND
    ...     status in %s
    ...     """ % sqlvalues(
    ...         ed, breezy_i386, active_publishing_status),
    ...     orderBy=["-datecreated"])[0]
    >>> print ed_pub.priority
    Extra
    >>> n = SourcePackageName.selectOneBy(name="archive-copier")
    >>> ac = SourcePackageRelease.selectOneBy(sourcepackagenameID=n.id,
    ...         version="0.3.6")
    >>> ac_pub = SSPPH.select("""
    ...     sourcepackagerelease = %s AND
    ...     distroseries = %s AND
    ...     status in %s
    ...     """ % sqlvalues(
    ...         ac, breezy, active_publishing_status),
    ...     orderBy=["-datecreated"])[0]
    >>> print ac_pub.component.name
    universe


Partner archive import
......................

Importing the partner archive requires overriding the component to
"partner", which also makes the archive on any publishing records the
partner archive.

First get a set of existing publishings for both source and binary:

    >>> comm_archive = ubuntu.getArchiveByComponent('partner')
    >>> hoary = ubuntu['hoary']
    >>> hoary_i386 = hoary['i386']
    >>> partner_source_set = set(
    ...     SSPPH.select("distroseries = %s" % sqlvalues(hoary)))

    >>> partner_binary_set = set(
    ...     SBPPH.select("distroarchseries = %s" % sqlvalues(hoary_i386)))

Now run gina to import packages and convert them to partner:

    >>> gina_proc = [sys.executable, 'scripts/gina.py', '-q', 'partner']
    >>> proc = subprocess.Popen(gina_proc, stderr=subprocess.PIPE)
    >>> proc.wait()
    0
    >>> transaction.commit()

There will now be a number of publishings in the partner archive:

    >>> partner_source_set_after = set(
    ...     SSPPH.select("distroseries = %s" % sqlvalues(hoary)))

    >>> partner_binary_set_after = set(
    ...     SBPPH.select("distroarchseries = %s" % sqlvalues(hoary_i386)))

    >>> source_difference = partner_source_set_after - partner_source_set
    >>> len(source_difference)
    12

    >>> binary_difference = partner_binary_set_after - partner_binary_set
    >>> len(binary_difference)
    24

All the publishings will also have the 'partner' component and the
partner archive:

    >>> print set(sspph.component.name for sspph in source_difference)
    set([u'partner'])

    >>> print set(sbpph.component.name for sbpph in binary_difference)
    set([u'partner'])

    >>> print set(sspph.archive.purpose.name for sspph in source_difference)
    set(['PARTNER'])

    >>> print set(sbpph.archive.purpose.name for sbpph in binary_difference)
    set(['PARTNER'])


Source-only imports
...................

Gina has a 'source-only' configuration option which allows it to
import only sources from the configured archive.

That's how we intend to start importing all debian source releases to
the launchpad system. This way we would have precise records of
"Ubuntu-Debian" packages relationships and expose this information,
not only in Soyuz (package managing) but also in Bugs and Blueprints,
for instance.

We will restore the initial 'gina_test_archive' because it contains a
entry for a suite called 'testing' which contains only the source
indexes from the 'hoary' suite.

    >>> os.remove('/tmp/gina_test_archive')
    >>> relative_path = ('lib/lp/soyuz/scripts/tests/gina_test_archive')
    >>> path = os.path.join(os.getcwd(), relative_path)
    >>> os.symlink(path, '/tmp/gina_test_archive')

We will also create the target distroseries for the imported
sources. We will import them into Debian/Lenny distroseries as
specified in the testing configuration.

    >>> from lp.registry.interfaces.distribution import (
    ...     IDistributionSet)
    >>> debian = getUtility(IDistributionSet).getByName('debian')

    # Only the distro owner and admins can create a new series.
    >>> login('mark@example.com')
    >>> lenny = debian.newSeries(
    ...     "lenny", "lenny", "Lenny",
    ...     "---", "!!!", "8.06",
    ...     hoary, celebs.launchpad_developers)
    >>> login(ANONYMOUS)

Note that we will create a Lenny/i386 port (DistroArchSeries) to check
if no binaries get imported by mistake. However this is not required
in production, i.e., just creating 'lenny' should suffice for the
source-only import to happen.

    >>> pf = ProcessorFamily.selectOneBy(name="x86")
    >>> lenny_i386 = DistroArchSeries(
    ...     distroseries=lenny, processorfamily=pf,
    ...     architecturetag="i386", official=True,
    ...     owner=celebs.launchpad_developers)

We will also store the number of binaries already published in debian
PRIMARY archive, so we can check later it was unaffected by the
import.

    >>> debian_binaries = SBPPH.selectBy(archive=debian.main_archive)
    >>> number_of_debian_binaries = debian_binaries.count()

Commit the changes and run the importer script.

    >>> transaction.commit()

    >>> gina_proc = [
    ...     sys.executable, 'scripts/gina.py', '-q', 'lenny']
    >>> proc = subprocess.Popen(gina_proc, stderr=subprocess.PIPE)
    >>> proc.wait()
    0

    >>> transaction.commit()

There is now a number of source publications in PUBLISHED status for the
targetted distroseries, 'lenny'.

    >>> lenny_sources = SSPPH.select("distroseries = %s" % sqlvalues(lenny))
    >>> lenny_sources.count()
    12

    >>> print set([pub.status.name for pub in lenny_sources])
    set(['PUBLISHED'])

As mentioned before, lenny/i386 is empty, no binaries were imported.
Also, the number of binaries published in the whole debian distribution
hasn't changed.

    >>> lenny_binaries = SBPPH.selectBy(distroarchseries=lenny_i386)
    >>> lenny_binaries.count()
    0

    >>> debian_binaries = SBPPH.selectBy(archive=debian.main_archive)
    >>> debian_binaries.count() == number_of_debian_binaries
    True


Processing multiple suites in the same batch
............................................

Both, 'lenny' and 'hoary' (as partner) will be processed in the same
batch.

    >>> gina_proc = [
    ...     sys.executable, 'scripts/gina.py', 'lenny', 'partner']
    >>> proc = subprocess.Popen(gina_proc, stderr=subprocess.PIPE)

    >>> print proc.stderr.read()
    INFO    Creating lockfile: /var/lock/launchpad-gina.lock
    ...
    INFO    === Processing debian/lenny/release ===
    ...
    INFO    === Processing ubuntu/hoary/release ===
    ...

    >>> proc.wait()
    0


Other tests
...........

For kicks, finally, run gina on a configured but incomplete archive:

    >>> gina_proc = [sys.executable, 'scripts/gina.py', '-q', 'bogus']
    >>> proc = subprocess.Popen(gina_proc, stderr=subprocess.PIPE)
    >>> print proc.stderr.read()
    ERROR   Failed to analyze archive for bogoland
     -> http://...)
    <BLANKLINE>
    >>> proc.wait()
    1


Wrap up
.......

Remove the tmp link to the gina_test_archive
    >>> os.remove('/tmp/gina_test_archive')