~launchpad-pqm/launchpad/devel : contents of lib/lp/soyuz/doc/archive-dependencies.txt at revision 13457.5.2

~launchpad-pqm/launchpad/devel : (revision 13457.5.2)
= Archive dependencies =

`ArchiveDependencies` class models archive dependencies mechanics and
is used to provided the contents of 'sources_list' file used to build
sources in the given IBuildQueue context.


== Testing scenario setup ==

We use `SoyuzTestPublisher` to generate a source publications and
build candidates for ubuntu/hoary.

    >>> from lp.registry.interfaces.distribution import (
    ...     IDistributionSet)
    >>> from lp.soyuz.tests.test_publishing import (
    ...     SoyuzTestPublisher)
    >>> from canonical.launchpad.ftests import login

    >>> login('foo.bar@canonical.com')

    >>> test_publisher = SoyuzTestPublisher()

    >>> ubuntu = getUtility(IDistributionSet).getByName('ubuntu')
    >>> hoary = ubuntu.getSeries('hoary')

    >>> test_publisher.addFakeChroots(hoary)
    >>> unused = test_publisher.setUpDefaultDistroSeries(hoary)


== Static dependency maps ==

`component_dependencies` contains a static map of the default ubuntu
component dependencies (known as the 'ogre-model').

    >>> from lp.soyuz.adapters.archivedependencies import (
    ...     component_dependencies)

    >>> def show_component_deps():
    ...     print "Component  |   Dependencies"
    ...     print "-----------+---------------"
    ...     for key, value in sorted(component_dependencies.items()):
    ...         print "%10s |" % (key,),
    ...         for dep in value:
    ...             print dep,
    ...         print

    >>> show_component_deps()
    Component  |   Dependencies
    -----------+---------------
          main | main
    multiverse | main restricted universe multiverse
       partner | partner
    restricted | main restricted
      universe | main universe

`pocket_dependencies` contains a static map of the default ubuntu
pocket dependencies.

    >>> from lp.soyuz.adapters.archivedependencies import (
    ...     pocket_dependencies)

    >>> def show_pocket_deps():
    ...     print "Pocket    |   Dependencies"
    ...     print "----------+---------------"
    ...     for (key, value) in sorted(pocket_dependencies.items()):
    ...         print "%7s |" % (key.name,),
    ...         for pocket in value:
    ...             print pocket.name,
    ...         print

    >>> show_pocket_deps()
    Pocket    |   Dependencies
    ----------+---------------
      RELEASE | RELEASE
     SECURITY | RELEASE SECURITY
      UPDATES | RELEASE SECURITY UPDATES
     PROPOSED | RELEASE SECURITY UPDATES PROPOSED
    BACKPORTS | RELEASE SECURITY UPDATES BACKPORTS


== 'Ogre' components ==

The ubuntu 'ogre-model' ensures that build dependencies are
consistently spread according the source target component, i.e. a
source published in 'main' component is only allowed to depend on
binaries also published in 'main', on the other hand a source
published in 'universe' is allowed to depend on binaries published in
'main' and 'universe' components.

A proper name for this "model" would be 'cross-component-dependency'.

    >>> from canonical.database.sqlbase import flush_database_caches
    >>> from lp.soyuz.interfaces.component import IComponentSet

    >>> ubuntu_components = [
    ...     'main', 'restricted', 'universe', 'multiverse', 'partner']

    >>> from lp.soyuz.adapters.archivedependencies import (
    ...     get_components_for_context)

    >>> ogre_pub = test_publisher.getPubSource(sourcename='ogre')
    >>> [ogre_build] = ogre_pub.createMissingBuilds()

    >>> def testOgreComponents():
    ...     print " Component | Ogre-Model"
    ...     print "-----------+---------------"
    ...     for component in ubuntu_components:
    ...         component = getUtility(IComponentSet)[component]
    ...         ogre_pub.component = component
    ...         flush_database_caches()
    ...         components_term = " ".join(
    ...             get_components_for_context(
    ...                 ogre_build.current_component, ogre_build.pocket))
    ...         print '%10s | %s' % (ogre_build.current_component.name,
    ...                              components_term)

    >>> testOgreComponents()
     Component | Ogre-Model
    -----------+---------------
          main | main
    restricted | main restricted
      universe | main universe
    multiverse | main restricted universe multiverse
       partner | partner

As fixed for bug #198936, builds for the BACKPORTS pocket are allowed
to use any component available, independently of the component they
are currently published. This special-case is important because it
avoids changes to accommodate the backported source in the already
released series.

    >>> from lp.registry.interfaces.pocket import (
    ...      PackagePublishingPocket)
    >>> from zope.security.proxy import removeSecurityProxy

    >>> naked_build = removeSecurityProxy(ogre_build)
    >>> naked_build.pocket = PackagePublishingPocket.BACKPORTS

Ultimately, it means that a build targeted to the BACKPORTS pocket
will behave as if it were published in the multiverse component,
despite the component it is actually published in.

    >>> testOgreComponents()
     Component | Ogre-Model
    -----------+---------------
          main | main restricted universe multiverse
    restricted | main restricted universe multiverse
      universe | main restricted universe multiverse
    multiverse | main restricted universe multiverse
       partner | main restricted universe multiverse


== Sources.list contents for building ==

We will use Celso's PPA for testing these mechanisms.

    >>> from lp.registry.interfaces.person import IPersonSet
    >>> cprov = getUtility(IPersonSet).getByName('cprov')
    >>> print cprov.archive.displayname
    PPA for Celso Providelo

Non-primary archives by default use primary Release, Security and Updates
pockets and all its available components.

    >>> from lp.soyuz.adapters.archivedependencies import (
    ...     default_component_dependency_name,  default_pocket_dependency)

    >>> print default_pocket_dependency
    Updates

    >>> print default_component_dependency_name
    multiverse

The default values get applied to their corresponding dependency maps
and then will expand to distinct values that will be used to produce
the building 'sources_list' contents.

    >>> for pocket in pocket_dependencies[default_pocket_dependency]:
    ...     print pocket
    Release
    Security
    Updates

    >>> for component_name in component_dependencies[
    ...     default_component_dependency_name]:
    ...     print component_name
    main
    restricted
    universe
    multiverse

We will create a testing source publication and probe its build
environment.

    >>> pub_source = test_publisher.getPubSource(
    ...     version='1.1', archive=cprov.archive)
    >>> [a_build] = pub_source.createMissingBuilds()

Now we can verify if get_sources_list_for_building() method returns the
expected content for building the just-created source.

    >>> from lp.soyuz.adapters.archivedependencies import (
    ...     get_sources_list_for_building)

    >>> def print_building_sources_list(candidate):
    ...     sources_list = get_sources_list_for_building(
    ...     candidate, candidate.distro_arch_series,
    ...     candidate.source_package_release.name)
    ...     for line in sources_list:
    ...         print line

Note that only the default ubuntu dependencies for a public PPA will be
considered when building the source candidate. That's because there is
no binary published in Celso's PPA hoary/i386, so there is
no need to request the builder to load its archive indexes.

    >>> from lp.soyuz.enums import (
    ...     PackagePublishingStatus)

    >>> cprov.archive.getAllPublishedBinaries(
    ...      distroarchseries=a_build.distro_arch_series,
    ...      status=PackagePublishingStatus.PUBLISHED).count()
    0

    >>> print_building_sources_list(a_build)
    deb http://archive.launchpad.dev/ubuntu hoary
        main restricted universe multiverse
    deb http://archive.launchpad.dev/ubuntu hoary-security
        main restricted universe multiverse
    deb http://archive.launchpad.dev/ubuntu hoary-updates
        main restricted universe multiverse

Once we publish a test binary in Celso's PPA hoary/i386,
this archive becomes relevant for building, and thus listed in the
returned 'sources_list' content.

    >>> pub_binaries = test_publisher.getPubBinaries(
    ...     binaryname='dep-bin', archive=cprov.archive,
    ...     status=PackagePublishingStatus.PUBLISHED)

    >>> print_building_sources_list(a_build)
    deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
    deb http://archive.launchpad.dev/ubuntu hoary
        main restricted universe multiverse
    deb http://archive.launchpad.dev/ubuntu hoary-security
        main restricted universe multiverse
    deb http://archive.launchpad.dev/ubuntu hoary-updates
        main restricted universe multiverse

Similarly, unpopulated PPA dependencies are *not* listed in the building
'sources_list'.

    >>> mark = getUtility(IPersonSet).getByName('mark')
    >>> archive_dependency = cprov.archive.addArchiveDependency(
    ...     mark.archive, PackagePublishingPocket.RELEASE,
    ...     getUtility(IComponentSet)['main'])
    >>> print_building_sources_list(a_build)
    deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
    deb http://archive.launchpad.dev/ubuntu hoary
        main restricted universe multiverse
    deb http://archive.launchpad.dev/ubuntu hoary-security
        main restricted universe multiverse
    deb http://archive.launchpad.dev/ubuntu hoary-updates
        main restricted universe multiverse

But *populated* PPA dependencies *are* listed in the building 'sources_list'.

    >>> pub_binaries = test_publisher.getPubBinaries(
    ...     binaryname='dep-bin', archive=mark.archive,
    ...     status=PackagePublishingStatus.PUBLISHED)
    >>> print_building_sources_list(a_build)
    deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
    deb http://ppa.launchpad.dev/mark/ppa/ubuntu hoary main
    deb http://archive.launchpad.dev/ubuntu hoary
        main restricted universe multiverse
    deb http://archive.launchpad.dev/ubuntu hoary-security
        main restricted universe multiverse
    deb http://archive.launchpad.dev/ubuntu hoary-updates
        main restricted universe multiverse

    >>> cprov.archive.removeArchiveDependency(mark.archive)

What when we supply invalid dependencies, an exception is raised.

    >>> cprov.archive.external_dependencies = (
    ...     "Malformed format string here --> %(series)")
    Traceback (most recent call last):
    InvalidExternalDependencies: (InvalidExternalDependencies(...),
    "Invalid external dependencies:\nMalformed format string here
    --> %(series): Must start with 'deb'\nMalformed format string here
    --> %(series): Invalid URL\n")


== Overriding default primary archive dependencies ==

Despite being private or public, default primary archive dependencies
can be overridden by simply creating a `ArchiveDependency`record
targeted to the primary archive.

The 'pocket' and 'component' dependency attributes can be adjusted to
produce the desired build behaviour.

By default, public PPAs depend on all of the pocket dependencies of
UPDATES, and all of the primary archive's 'multiverse' component
dependencies.

    >>> print_building_sources_list(a_build)
    deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
    deb http://archive.launchpad.dev/ubuntu hoary
        main restricted universe multiverse
    deb http://archive.launchpad.dev/ubuntu hoary-security
        main restricted universe multiverse
    deb http://archive.launchpad.dev/ubuntu hoary-updates
        main restricted universe multiverse

The default build behaviour will remain unchanged when we override the
default primary archive dependencies with exactly the same values.

    >>> default_dependency = cprov.archive.addArchiveDependency(
    ...     ubuntu.main_archive, PackagePublishingPocket.UPDATES,
    ...     getUtility(IComponentSet)['multiverse'])

    >>> print_building_sources_list(a_build)
    deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
    deb http://archive.launchpad.dev/ubuntu hoary
        main restricted universe multiverse
    deb http://archive.launchpad.dev/ubuntu hoary-security
        main restricted universe multiverse
    deb http://archive.launchpad.dev/ubuntu hoary-updates
        main restricted universe multiverse

    >>> cprov.archive.removeArchiveDependency(ubuntu.main_archive)

The dependency can be modified to behave as an embargoed archive that
builds security updates. This is done by setting the SECURITY pocket
dependencies (RELEASE and SECURITY) and following the component
dependencies of the component where the source was last published in
the primary archive.

    >>> security_dependency = cprov.archive.addArchiveDependency(
    ...     ubuntu.main_archive, PackagePublishingPocket.SECURITY)

    >>> from lp.soyuz.adapters.archivedependencies import (
    ...     get_primary_current_component)

    >>> print get_primary_current_component(a_build.archive,
    ...     a_build.distro_series, a_build.source_package_release.name)
    universe

    >>> print_building_sources_list(a_build)
    deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
    deb http://archive.launchpad.dev/ubuntu hoary
        main universe
    deb http://archive.launchpad.dev/ubuntu hoary-security
        main universe

    >>> cprov.archive.removeArchiveDependency(ubuntu.main_archive)

It's also possible to modify the PPA to act as a super-free and
pristine build environment based only on what was included in the
original ubuntu release.

    >>> release_dependency = cprov.archive.addArchiveDependency(
    ...     ubuntu.main_archive, PackagePublishingPocket.RELEASE,
    ...     getUtility(IComponentSet)['restricted'])

    >>> print_building_sources_list(a_build)
    deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
    deb http://archive.launchpad.dev/ubuntu hoary main restricted

    >>> cprov.archive.removeArchiveDependency(ubuntu.main_archive)

The PPA can also be configured to extend the ubuntu PROPOSED build
environment.

    >>> proposed_dependency = cprov.archive.addArchiveDependency(
    ...     ubuntu.main_archive, PackagePublishingPocket.PROPOSED,
    ...     getUtility(IComponentSet)['multiverse'])

    >>> print_building_sources_list(a_build)
    deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
    deb http://archive.launchpad.dev/ubuntu hoary
        main restricted universe multiverse
    deb http://archive.launchpad.dev/ubuntu hoary-security
        main restricted universe multiverse
    deb http://archive.launchpad.dev/ubuntu hoary-updates
        main restricted universe multiverse
    deb http://archive.launchpad.dev/ubuntu hoary-proposed
        main restricted universe multiverse

    >>> cprov.archive.removeArchiveDependency(ubuntu.main_archive)

Similarly an extension of the BACKPORTS environment can be set.

    >>> backports_dependency = cprov.archive.addArchiveDependency(
    ...     ubuntu.main_archive, PackagePublishingPocket.BACKPORTS,
    ...     getUtility(IComponentSet)['multiverse'])

    >>> print_building_sources_list(a_build)
    deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
    deb http://archive.launchpad.dev/ubuntu hoary
        main restricted universe multiverse
    deb http://archive.launchpad.dev/ubuntu hoary-security
        main restricted universe multiverse
    deb http://archive.launchpad.dev/ubuntu hoary-updates
        main restricted universe multiverse
    deb http://archive.launchpad.dev/ubuntu hoary-backports
        main restricted universe multiverse

    >>> cprov.archive.removeArchiveDependency(ubuntu.main_archive)


== Partner archive builds ==

Similarly to what happens with PPA builds, PARTNER builds may depend
on any Ubuntu component in the PRIMARY archive. This behavior allows
scenarios where partner packages may use other restricted/non-free
applications from 'multiverse', for instance 'sun-java', and also
other partner applications by default.

    # Populate the ubuntutest PARTNER archive with one built and one
    # pending build source.
    >>> primary, partner, debug = ubuntu.all_distro_archives
    >>> unused_source = test_publisher.getPubSource(
    ...     archive=partner, component='partner')
    >>> unused = test_publisher.getPubBinaries(
    ...     pub_source=unused_source,
    ...     status=PackagePublishingStatus.PUBLISHED)
    >>> pub_source = test_publisher.getPubSource(
    ...     version='1.2', archive=partner, component='partner')
    >>> [partner_build] = pub_source.createMissingBuilds()

    >>> print_building_sources_list(partner_build)
    deb http://archive.launchpad.dev/ubuntu-partner hoary partner
    deb http://archive.launchpad.dev/ubuntu hoary
        main restricted universe multiverse
    deb http://archive.launchpad.dev/ubuntu hoary-security
        main restricted universe multiverse
    deb http://archive.launchpad.dev/ubuntu hoary-updates
        main restricted universe multiverse


== External build dependencies ==

Via an administrator change, any PPA hosted in launchpad can be
assigned to one or more 'external' build dependencies additionally to
the internal ones.

There is a column on IArchive called 'external_dependencies' which can be set
for any hosted PPA.  It is a string listing the comma-separated external
dependencies in the debian sources_list format.

 deb http[s]://[user:pass@]<host>[/path] %(series)s[-pocket] [components]

The '%(series)s' part is optional and will be replaced on-the-fly with
the series name for the build record being dispatched.

We will create some dependencies for Celso's PPA.

    >>> cprov.archive.external_dependencies = (
    ... "deb http://user:pass@repository zoing everything\n"
    ... "deb http://user:pass@repository %(series)s public private\n"
    ... "deb http://user:pass@repository %(series)s-extra public")

Now builds in Celso's PPA will use the external dependencies.

    >>> print_building_sources_list(a_build)
    deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
    deb http://user:pass@repository zoing everything
    deb http://user:pass@repository hoary public private
    deb http://user:pass@repository hoary-extra public
    deb http://archive.launchpad.dev/ubuntu hoary
        main restricted universe multiverse
    deb http://archive.launchpad.dev/ubuntu hoary-security
        main restricted universe multiverse
    deb http://archive.launchpad.dev/ubuntu hoary-updates
        main restricted universe multiverse


== Overlays ==

An overlay distroseries is a derived distribution which works like a PPA.
This means that the parent's details gets added to the sources.list passed to
the builders.

    >>> depdistro = factory.makeDistribution('depdistro',
    ...     publish_base_url=u'http://archive.launchpad.dev/')
    >>> depseries = factory.makeDistroSeries(
    ...     name='depseries', distribution=depdistro)
    >>> deparchseries = factory.makeDistroArchSeries(
    ...     distroseries = depseries, architecturetag = 'i386')
    >>> universe_component = getUtility(IComponentSet)['universe']
    >>> dsp = factory.makeDistroSeriesParent(
    ...     derived_series=hoary, parent_series=depseries,
    ...     initialized=True, is_overlay=True,
    ...     pocket=PackagePublishingPocket.SECURITY,
    ...     component=universe_component)
    >>> pub_source = test_publisher.getPubSource(
    ...     version='1.1', archive=hoary.main_archive)
    >>> [hoary_build] = pub_source.createMissingBuilds()
    >>> print_building_sources_list(hoary_build)
    deb http://archive.launchpad.dev/ubuntu hoary main
    deb http://archive.launchpad.dev/depdistro depseries main universe
    deb http://archive.launchpad.dev/depdistro depseries-security
        main universe