~launchpad-pqm/launchpad/devel

= Distribution Mirrors =

A distribution mirror must always be associated with a single distribution, so
to create a new mirror you should use the Distribution.newMirror method.

    >>> from datetime import datetime, timedelta
    >>> from zope.component import getUtility
    >>> from lp.services.librarian.interfaces import ILibraryFileAliasSet
    >>> from lp.registry.interfaces.distribution import IDistributionSet
    >>> from lp.registry.interfaces.distributionmirror import (
    ...     IDistributionMirrorSet,
    ...     MirrorContent,
    ...     MirrorSpeed,
    ...     MirrorStatus,
    ...     )
    >>> from lp.registry.interfaces.distroseries import IDistroSeriesSet
    >>> from lp.registry.interfaces.person import IPersonSet
    >>> from lp.registry.interfaces.pocket import PackagePublishingPocket
    >>> from lp.registry.interfaces.series import SeriesStatus
    >>> from lp.services.worlddata.interfaces.country import ICountrySet
    >>> from lp.soyuz.interfaces.distroarchseries import IDistroArchSeriesSet
    >>> mirrorset = getUtility(IDistributionMirrorSet)
    >>> distroset = getUtility(IDistributionSet)
    >>> ubuntu = distroset.get(1)
    >>> owner = getUtility(IPersonSet).getByName('name16')
    >>> speed = MirrorSpeed.S2M
    >>> brazil = getUtility(ICountrySet)['BR']
    >>> content = MirrorContent.ARCHIVE
    >>> http_base_url = 'http://foo.bar.com/pub'
    >>> whiteboard = "This mirror is based deep in the Amazon rainforest."
    >>> new_mirror = ubuntu.newMirror(
    ...     owner, speed, brazil, content, http_base_url=http_base_url,
    ...     whiteboard=whiteboard)

When a new mirror is created we'll generate a unique name for it, based
on its host name and content.

    >>> new_mirror.name
    u'foo.bar.com-archive'

And all mirrors aren't considered one of the official ones until the
distribution's mirror admin has reviewed (and approved) it.

    >>> new_mirror.isOfficial()
    False
    >>> new_mirror.status
    <DBItem MirrorStatus.PENDING_REVIEW...

You cannot create a new mirror on a distribution that does not have the
full functionality of Launchpad enabled:

    >>> kubuntu = distroset.get(5)
    >>> kubuntu.name
    u'kubuntu'
    >>> poor_mirror = kubuntu.newMirror(
    ...     owner, speed, brazil, content, http_base_url=http_base_url)
    >>> print poor_mirror
    None

The contents of a distribution mirror are represented by its associated
MirrorDistroArchSeriess and MirrorDistroSeriesSources. This data is
updated at least once a day by the distributionmirror-prober cronscript,
to make sure the information we have is always up to date.

    >>> new_mirror.source_series.count()
    0
    >>> new_mirror.arch_series.count()
    0

To create a new MirrorDistroArchSeries (or MirrorDistroSeriesSource)
associated with a given mirror, we use the ensureMirrorDistroArchSeries
(or ensureMirrorDistroSeriesSource) method.

    >>> warty = getUtility(IDistroSeriesSet).get(1)
    >>> warty_i386 = getUtility(IDistroArchSeriesSet).get(1)
    >>> pocket = PackagePublishingPocket.RELEASE
    >>> warty_component = warty.components[0]
    >>> warty_i386_mirror = new_mirror.ensureMirrorDistroArchSeries(
    ...     warty_i386, pocket, warty_component)

    >>> warty_mirror = new_mirror.ensureMirrorDistroSeriesSource(
    ...     warty, pocket, warty_component)

    >>> new_mirror.source_series.count()
    1
    >>> new_mirror.arch_series.count()
    1

If we try to create another MirrorDistroArchSeries with the same arch
series and pocket, that method will simply return the existing one.

    >>> same_warty_i386_mirror = new_mirror.ensureMirrorDistroArchSeries(
    ...     warty_i386, pocket, warty_component)
    >>> same_warty_i386_mirror == warty_i386_mirror
    True

    >>> same_warty_mirror = new_mirror.ensureMirrorDistroSeriesSource(
    ...     warty, pocket, warty_component)
    >>> same_warty_mirror == warty_mirror
    True

It's also possible to delete a
MirrorDistroArchSeries/MirrorDistroSeriesSource if we find out their
contents are not in a mirror where they used to be.

    >>> new_mirror.deleteMirrorDistroSeriesSource(
    ...     warty, pocket, warty_component)
    >>> new_mirror.deleteMirrorDistroArchSeries(
    ...     warty_i386, pocket, warty_component)

    >>> new_mirror.source_series.count()
    0
    >>> new_mirror.arch_series.count()
    0

From every distribution, you can easily get a list with its official
ARCHIVE or RELEASE mirrors. This is available through the
archive_mirrors and cdimage_mirrors properties of IDistribution.

    >>> [(mirror.name, mirror.speed.title, mirror.status.name)
    ...  for mirror in ubuntu.archive_mirrors]
    [(u'canonical-archive', '100 Mbps', 'OFFICIAL'),
     (u'archive-404-mirror', '512 Kbps', 'OFFICIAL'),
     (u'archive-mirror', '128 Kbps', 'OFFICIAL'),
     (u'archive-mirror2', '128 Kbps', 'OFFICIAL')]

    >>> [(mirror.name, mirror.speed.title, mirror.status.name)
    ...  for mirror in ubuntu.cdimage_mirrors]
    [(u'canonical-releases', '100 Mbps', 'OFFICIAL'),
     (u'releases-mirror', '2 Mbps', 'OFFICIAL'),
     (u'releases-mirror2', '2 Mbps', 'OFFICIAL'),
     (u'unreachable-mirror', '512 Kbps', 'OFFICIAL')]

The list of pending review mirrors can easily be obtained, so that the
distribution owner can see all the ones that need to be reviewed.

    >>> [(mirror.name, mirror.speed.title, mirror.status.name)
    ...  for mirror in ubuntu.pending_review_mirrors]
    [(u'random-releases-mirror', '100 Mbps', 'PENDING_REVIEW')]

We also have a property which returns all unofficial mirrors.

    >>> [(mirror.name, mirror.speed.title, mirror.status.name)
    ...  for mirror in ubuntu.unofficial_mirrors]
    [(u'invalid-mirror', '2 Mbps', 'UNOFFICIAL')]

It's possible to retrieve a mirror by its name:

    >>> from canonical.launchpad.webapp.testing import verifyObject
    >>> from lp.registry.interfaces.distributionmirror import IDistributionMirror
    >>> example_mirror = mirrorset.getByName('archive-mirror')
    >>> verifyObject(IDistributionMirror, example_mirror)
    True

    >>> print mirrorset.getByName('non-existent-mirror')
    None

Or by any of its URLs (HTTP, FTP or Rsync)
First we'll have to add some of this URLs to sample data:

    >>> from lp.testing import login
    >>> login("mark@example.com")
    >>> example_mirror.ftp_base_url = 'ftp://localhost/example-ftp'
    >>> example_mirror.rsync_base_url = 'rsync://localhost/example-rsync'

The getBy*Url methods return the corresponding mirrors:

    >>> http_mirror = mirrorset.getByHttpUrl(
    ...     'http://localhost:11375/valid-mirror')

    >>> print mirrorset.getByHttpUrl('http://non-existent-url')
    None

    >>> ftp_mirror = mirrorset.getByFtpUrl('ftp://localhost/example-ftp')
    >>> ftp_mirror.has_ftp_or_rsync_base_url
    True

    >>> print mirrorset.getByFtpUrl('ftp://non-existent-url')
    None

    >>> rsync_mirror = mirrorset.getByRsyncUrl(
    ...     'rsync://localhost/example-rsync')
    >>> rsync_mirror.has_ftp_or_rsync_base_url
    True

    >>> print mirrorset.getByRsyncUrl('rsync://non-existent-url')
    None


== Finding the best mirror for a given user ==

In order to be able to guess what would be the best mirror for a given
user based on his IP address, we provide simple API to find official
mirrors of a given content type located on a given country (or any other
country in its continent if the country doesn't have any).

In Germany we have a single cdimage mirror, so the list will contain
that mirror followed by the main cdimage repository
(releases.ubuntu.com), which is always appended to the end of the list.

    >>> germany = getUtility(ICountrySet)['DE']
    >>> cdimage = MirrorContent.RELEASE
    >>> [(mirror.name, mirror.country.name)
    ...  for mirror in mirrorset.getBestMirrorsForCountry(germany, cdimage)]
    [(u'releases-mirror2', u'Germany'),
     (u'canonical-releases', u'United Kingdom')]

Since there are no archive mirrors in Germany, any user in that country
will be directed to other mirrors in the same continet.

    >>> archive = MirrorContent.ARCHIVE
    >>> mirrors = mirrorset.getBestMirrorsForCountry(germany, archive)
    >>> sorted((mirror.name, mirror.country.name)
    ...        for mirror in mirrors)
    [(u'archive-404-mirror', u'France'), (u'archive-mirror', u'France'),
    (u'canonical-archive', u'United Kingdom')]

In brazil we don't have official archive or cdimage mirrors, so all we
see is the main repository.

    >>> [(mirror.name, mirror.country.name)
    ...  for mirror in mirrorset.getBestMirrorsForCountry(brazil, archive)]
    [(u'canonical-archive', u'United Kingdom')]

    >>> [(mirror.name, mirror.country.name)
    ...  for mirror in mirrorset.getBestMirrorsForCountry(brazil, cdimage)]
    [(u'canonical-releases', u'United Kingdom')]

If we fail to identify the user's country and pass None to
getBestMirrorsForCountry() we'll get only the main repository as well.

    >>> [(mirror.name, mirror.country.name)
    ...  for mirror in mirrorset.getBestMirrorsForCountry(None, archive)]
    [(u'canonical-archive', u'United Kingdom')]


== Probing the mirrors ==

The distributionmirror-prober script is used to check what a mirror
contains and when it was last updated. This script should run at least
once a day, so we know the information we display is always up to date.

This script will probe only official ARCHIVE or RELEASE mirrors that
weren't probed in the last 23 (the value of PROBE_INTERVAL) hours.

    >>> [mirror.name
    ...  for mirror in mirrorset.getMirrorsToProbe(MirrorContent.ARCHIVE)]
    [u'canonical-archive', u'archive-404-mirror', u'archive-mirror',
     u'archive-mirror2']

    >>> [mirror.name
    ...  for mirror in mirrorset.getMirrorsToProbe(MirrorContent.RELEASE)]
    [u'canonical-releases', u'releases-mirror', u'releases-mirror2',
     u'unreachable-mirror']

    >>> valid_mirror = mirrorset[1]
    >>> valid_mirror.name
    u'archive-mirror'
    >>> from StringIO import StringIO
    >>> log_file = StringIO()
    >>> log_file.write("Fake probe, nothing useful here.")
    >>> log_file.seek(0)
    >>> library_alias = getUtility(ILibraryFileAliasSet).create(
    ...     name='foo', size=len(log_file.getvalue()),
    ...     file=log_file, contentType='text/plain')
    >>> proberecord = valid_mirror.newProbeRecord(library_alias)

    >>> [mirror.name
    ...  for mirror in mirrorset.getMirrorsToProbe(MirrorContent.ARCHIVE)]
    [u'canonical-archive', u'archive-404-mirror', u'archive-mirror2']

    >>> [mirror.name
    ...  for mirror in mirrorset.getMirrorsToProbe(MirrorContent.RELEASE)]
    [u'canonical-releases', u'releases-mirror', u'releases-mirror2',
     u'unreachable-mirror']

The getMirrorsToProbe() method also accepts an ignore_last_probe
argument, that, if True, will ignore previous probe records for all
mirrors.

    >>> mirrors = mirrorset.getMirrorsToProbe(
    ...     MirrorContent.ARCHIVE, ignore_last_probe=True)
    >>> [mirror.name for mirror in mirrors]
    [u'canonical-archive', u'archive-404-mirror', u'archive-mirror',
     u'archive-mirror2']

It's also possible to limit the number of mirrors we want to probe by
passing an extra limit= argument to mirrorset.getMirrorsToProbe().  If
any limit is given, the mirrors which have never been probed will be
given precedence, followed by the ones which have been probed longest
ago.  Note that although we have this precedence when a limit is given,
the actual results are not ordered by the date the mirrors were last
probed as we don't care about it anyway.

    >>> mirrors = mirrorset.getMirrorsToProbe(
    ...     MirrorContent.ARCHIVE, ignore_last_probe=True)
    >>> import pytz
    >>> utc = pytz.timezone('UTC')
    >>> now = datetime.now(utc)
    >>> for mirror in mirrors:
    ...     last_probe = mirror.last_probe_record
    ...     last_probe_date = 'NEVER'
    ...     if last_probe is not None:
    ...         if (now - last_probe.date_created).days < 1:
    ...             last_probe_date = 'TODAY'
    ...         else:
    ...             last_probe_date = last_probe.date_created.isoformat()
    ...     print mirror.name, last_probe_date
    canonical-archive NEVER
    archive-404-mirror NEVER
    archive-mirror TODAY
    archive-mirror2 2006-05-24...

    >>> mirrors = mirrorset.getMirrorsToProbe(
    ...     MirrorContent.ARCHIVE, ignore_last_probe=True, limit=3)
    >>> for mirror in mirrors:
    ...     last_probe = mirror.last_probe_record
    ...     last_probe_date = 'NEVER'
    ...     if last_probe is not None:
    ...         last_probe_date = last_probe.date_created.isoformat()
    ...     print mirror.name, last_probe_date
    canonical-archive NEVER
    archive-404-mirror NEVER
    archive-mirror2 2006-05-24...


If when we finish probing a mirror, that mirror doesn't have any
MirrorDistroSeriesSource or MirrorDistroArchSeries, that mirror is
marked as disabled and a notification is sent to its owner and to the
distribution's mirror admin. This is done using
IDistributionMirror.disable().  Disabling a mirror causes it not to show
up on the public mirror listings.

    >>> valid_mirror.enabled
    True
    >>> log = 'Got a 404 on http://foo.bar/baz\n'
    >>> valid_mirror.disable(notify_owner=True, log=log)

    # Commit, so the email is actually sent.
    >>> import transaction
    >>> transaction.commit()

    >>> import email
    >>> from lp.services.mail import stub
    >>> len(stub.test_emails)
    2
    >>> stub.test_emails.sort(lambda a, b: cmp(a[1], b[1])) # sort by to_addr
    >>> from_addr, to_addr, raw_message = stub.test_emails.pop()
    >>> msg = email.message_from_string(raw_message)
    >>> msg['To']
    'mark@example.com,karl@canonical.com'
    >>> from_addr, to_addr, raw_message = stub.test_emails.pop()
    >>> msg = email.message_from_string(raw_message)
    >>> msg['To']
    'mark@example.com'
    >>> valid_mirror.enabled
    False
    >>> print msg.get_payload()
    Dear mirror administrator,
    <BLANKLINE>
    This is an automated failure notice that has been generated by the mirror
    content checker.
    ...
    Following is a snippet of the log file above:
    -----------------------------------------------------------------------
    Got a 404 on http://foo.bar/baz
    <BLANKLINE>
    -----------------------------------------------------------------------
    ...

    # Enable the mirror again.
    >>> from lp.testing import login
    >>> valid_mirror.enabled = True

If we pass notify_owner=False to the disable() method, then we'll send a
single notification to the distribution's mirror admins.

    >>> valid_mirror.disable(notify_owner=False, log=log)
    >>> transaction.commit()
    >>> len(stub.test_emails)
    1
    >>> from_addr, to_addr, raw_message = stub.test_emails.pop()
    >>> msg = email.message_from_string(raw_message)
    >>> msg['To']
    'mark@example.com,karl@canonical.com'

    Now we delete the MirrorProbeRecord we've just created, to make
    sure this mirror is probed by our prober script.
    >>> from zope.security.proxy import removeSecurityProxy
    >>> naked_record = removeSecurityProxy(proberecord)
    >>> naked_record.destroySelf()
    >>> transaction.commit()


=== Checking what content a mirror should contain ===

After obtaining the list of mirrors that we need to probe, the script
will then check what content is mirrored in each mirror. This is done by
checking the existence of some control files on that mirror.

For Archive mirrors, these files are Packages.gz (one file per
[arch_series, component, pocket] tuple) and Sources.gz (one per [series,
component, pocket] tuple). The paths to these files are given by the
getExpectedPackagesPaths() and getExpectedSourcesPaths() methods of
IDistributionMirror.

    >>> paths = mirror.getExpectedPackagesPaths()
    >>> [path for (series, pocket, component, path) in paths]
    [u'dists/breezy-autotest/main/binary-i386/Packages.gz',
     u'dists/breezy-autotest/restricted/binary-i386/Packages.gz',
     u'dists/breezy-autotest/universe/binary-i386/Packages.gz',
     u'dists/breezy-autotest/multiverse/binary-i386/Packages.gz',
     u'dists/breezy-autotest-backports/main/binary-i386/Packages.gz',
     u'dists/breezy-autotest-backports/restricted/binary-i386/Packages.gz',
     u'dists/breezy-autotest-backports/universe/binary-i386/Packages.gz',
     u'dists/breezy-autotest-backports/multiverse/binary-i386/Packages.gz',
     u'dists/breezy-autotest-security/main/binary-i386/Packages.gz',
     u'dists/breezy-autotest-security/restricted/binary-i386/Packages.gz',
     u'dists/breezy-autotest-security/universe/binary-i386/Packages.gz',
     u'dists/breezy-autotest-security/multiverse/binary-i386/Packages.gz',
     u'dists/breezy-autotest-updates/main/binary-i386/Packages.gz',
     u'dists/breezy-autotest-updates/restricted/binary-i386/Packages.gz',
     u'dists/breezy-autotest-updates/universe/binary-i386/Packages.gz',
     u'dists/breezy-autotest-updates/multiverse/binary-i386/Packages.gz',
     u'dists/breezy-autotest-proposed/main/binary-i386/Packages.gz',
     u'dists/breezy-autotest-proposed/restricted/binary-i386/Packages.gz',
     u'dists/breezy-autotest-proposed/universe/binary-i386/Packages.gz',
     u'dists/breezy-autotest-proposed/multiverse/binary-i386/Packages.gz',
     u'dists/hoary/main/binary-i386/Packages.gz',
     u'dists/hoary/restricted/binary-i386/Packages.gz',
     u'dists/hoary-backports/main/binary-i386/Packages.gz',
     u'dists/hoary-backports/restricted/binary-i386/Packages.gz',
     u'dists/hoary-security/main/binary-i386/Packages.gz',
     u'dists/hoary-security/restricted/binary-i386/Packages.gz',
     u'dists/hoary-updates/main/binary-i386/Packages.gz',
     u'dists/hoary-updates/restricted/binary-i386/Packages.gz',
     u'dists/hoary-proposed/main/binary-i386/Packages.gz',
     u'dists/hoary-proposed/restricted/binary-i386/Packages.gz',
     u'dists/warty/main/binary-i386/Packages.gz',
     u'dists/warty/universe/binary-i386/Packages.gz',
     u'dists/warty-backports/main/binary-i386/Packages.gz',
     u'dists/warty-backports/universe/binary-i386/Packages.gz',
     u'dists/warty-security/main/binary-i386/Packages.gz',
     u'dists/warty-security/universe/binary-i386/Packages.gz',
     u'dists/warty-updates/main/binary-i386/Packages.gz',
     u'dists/warty-updates/universe/binary-i386/Packages.gz',
     u'dists/warty-proposed/main/binary-i386/Packages.gz',
     u'dists/warty-proposed/universe/binary-i386/Packages.gz']

    >>> warty.status = SeriesStatus.OBSOLETE

    >>> paths = mirror.getExpectedSourcesPaths()
    >>> [path for (series, pocket, component, path) in paths]
    [u'dists/breezy-autotest/main/source/Sources.gz',
     u'dists/breezy-autotest/restricted/source/Sources.gz',
     u'dists/breezy-autotest/universe/source/Sources.gz',
     u'dists/breezy-autotest/multiverse/source/Sources.gz',
     u'dists/breezy-autotest-backports/main/source/Sources.gz',
     u'dists/breezy-autotest-backports/restricted/source/Sources.gz',
     u'dists/breezy-autotest-backports/universe/source/Sources.gz',
     u'dists/breezy-autotest-backports/multiverse/source/Sources.gz',
     u'dists/breezy-autotest-security/main/source/Sources.gz',
     u'dists/breezy-autotest-security/restricted/source/Sources.gz',
     u'dists/breezy-autotest-security/universe/source/Sources.gz',
     u'dists/breezy-autotest-security/multiverse/source/Sources.gz',
     u'dists/breezy-autotest-updates/main/source/Sources.gz',
     u'dists/breezy-autotest-updates/restricted/source/Sources.gz',
     u'dists/breezy-autotest-updates/universe/source/Sources.gz',
     u'dists/breezy-autotest-updates/multiverse/source/Sources.gz',
     u'dists/breezy-autotest-proposed/main/source/Sources.gz',
     u'dists/breezy-autotest-proposed/restricted/source/Sources.gz',
     u'dists/breezy-autotest-proposed/universe/source/Sources.gz',
     u'dists/breezy-autotest-proposed/multiverse/source/Sources.gz',
     u'dists/hoary/main/source/Sources.gz',
     u'dists/hoary/restricted/source/Sources.gz',
     u'dists/hoary-backports/main/source/Sources.gz',
     u'dists/hoary-backports/restricted/source/Sources.gz',
     u'dists/hoary-security/main/source/Sources.gz',
     u'dists/hoary-security/restricted/source/Sources.gz',
     u'dists/hoary-updates/main/source/Sources.gz',
     u'dists/hoary-updates/restricted/source/Sources.gz',
     u'dists/hoary-proposed/main/source/Sources.gz',
     u'dists/hoary-proposed/restricted/source/Sources.gz',
     u'dists/warty-security/main/source/Sources.gz',
     u'dists/warty-updates/main/source/Sources.gz']

    >>> warty.status = SeriesStatus.SUPPORTED

For cdimage mirrors, these files are listed in a file stored in
http://releases.ubuntu.com/. This file is parsed by
get_expected_cdimage_paths(), which returns a list of (distroseries,
flavour name, path) elements.  (For testing we have a sample of that
file stored in our tree, which is pointed by the
config.distributionmirrorprober.releases_file_list_url option)

    >>> from lp.registry.scripts.distributionmirror_prober import (
    ...     get_expected_cdimage_paths)
    >>> for (series, flavour, paths) in get_expected_cdimage_paths():
    ...     for path in paths:
    ...         print series.name, flavour, path
    hoary kubuntu /kubuntu/hoary/kubuntu-5.04-install-amd64.iso
    hoary kubuntu /kubuntu/hoary/kubuntu-5.04-install-i386.iso
    hoary kubuntu /kubuntu/hoary/kubuntu-5.04-install-powerpc.iso
    hoary kubuntu /kubuntu/hoary/kubuntu-5.04-live-amd64.iso
    hoary kubuntu /kubuntu/hoary/kubuntu-5.04-live-i386.iso
    hoary kubuntu /kubuntu/hoary/kubuntu-5.04-live-powerpc.iso
    hoary ubuntu /hoary/ubuntu-5.04-install-amd64.iso
    hoary ubuntu /hoary/ubuntu-5.04-install-i386.iso
    hoary ubuntu /hoary/ubuntu-5.04-install-powerpc.iso
    hoary ubuntu /hoary/ubuntu-5.04-live-amd64.iso
    hoary ubuntu /hoary/ubuntu-5.04-live-i386.iso
    hoary ubuntu /hoary/ubuntu-5.04-live-powerpc.iso
    warty ubuntu /warty/warty-release-install-amd64.iso
    warty ubuntu /warty/warty-release-install-i386.iso
    warty ubuntu /warty/warty-release-install-powerpc.iso
    warty ubuntu /warty/warty-release-live-i386.iso


=== Checking how up-to-date the content is ===

After knowing what content a mirror is expected to contain, we need to
check when that mirror last synced its contents.

To do that we use the getURLsToCheckUpdateness() of either
MirrorDistroSeriesSource or MirrorDistroArchSeries. This method returns
a dictionary mapping MirrorFreshnesses to URLs on that mirror.

The prober will then check, between all reachable URLs, which one has
the freshness which corresponds to the most recent sync, and then set
that as the mirror's freshness.

On the warty release, component 'main' and pocket RELEASE , we had two
uploads between 2005-09-15 and 2005-09-17, so at that time we could've
checked if that mirror's last sync was in the last one or two days.

    >>> ubuntu = getUtility(IDistributionSet).getByName('ubuntu')
    >>> foo_mirror = ubuntu.getMirrorByName('foo.bar.com-archive')
    >>> warty = ubuntu.getSeries('warty')
    >>> warty_mirror = foo_mirror.ensureMirrorDistroSeriesSource(
    ...     warty, PackagePublishingPocket.RELEASE, warty.components[0])

    >>> when = datetime(2005, 9, 17, tzinfo=utc)
    >>> urls = warty_mirror.getURLsToCheckUpdateness(when=when)
    >>> [(freshness.name, url) for (freshness, url) in urls.items()]
    [('UP',
      u'http://foo.bar.com/pub/pool/main/a/alsa-utils/alsa-utils_1.0.9a-4.dsc'),
     ('TWODAYSBEHIND',
      u'http://foo.bar.com/pub/pool/main/a/alsa-utils/alsa-utils_1.0.8-1ubuntu1.dsc')]

But if we were to check that mirror today, we could only check if the
last upload was mirrored and then mark the mirror as up-to-date. This is
because there were no recent uploads there.

    >>> urls = warty_mirror.getURLsToCheckUpdateness()
    >>> [(freshness.name, url) for (freshness, url) in urls.items()]
    [('UP', u'http://foo.bar.com/pub/pool/main/c/cdrkit/foobar-1.0.dsc')]

If the mirror has no HTTP base url, we'll use the FTP one.

    >>> naked_mirror = removeSecurityProxy(warty_mirror)
    >>> http_url = naked_mirror.distribution_mirror.http_base_url
    >>> naked_mirror.distribution_mirror.http_base_url = None
    >>> naked_mirror.distribution_mirror.ftp_base_url = (
    ...     'ftp://foo.bar.com/pub')
    >>> urls = warty_mirror.getURLsToCheckUpdateness()
    >>> [(freshness.name, url) for (freshness, url) in urls.items()]
    [('UP', u'ftp://foo.bar.com/pub/pool/main/c/cdrkit/foobar-1.0.dsc')]
    >>> naked_mirror.distribution_mirror.http_base_url = http_url

The same goes for the warty i386 mirror, in which we had one upload on
2005-06-18 and two others on 2005-06-20. One slightly difference in this
case is that one of the uploads made on 2005-06-20 contains an .udeb
package instead of a .deb, and we don't check .udeb files on the mirror,
so we need to skip that upload.

    >>> from lp.soyuz.model.files import BinaryPackageFile

    >>> warty_i386_mirror = foo_mirror.ensureMirrorDistroArchSeries(
    ...     warty['i386'], PackagePublishingPocket.RELEASE,
    ...     warty.components[0])
    >>> warty_i386_mirror = removeSecurityProxy(warty_i386_mirror)
    >>> recent_freshness, threshold = warty_i386_mirror.freshness_times[0]
    >>> start = datetime(2005, 06, 20, tzinfo=utc)
    >>> end = datetime(2005, 06, 20, tzinfo=utc) + timedelta(hours=0.5)
    >>> time_interval = (start, end)
    >>> upload = warty_i386_mirror.getLatestPublishingEntry(
    ...     time_interval, deb_only=False)

    >>> bpf = BinaryPackageFile.selectOneBy(
    ...     binarypackagereleaseID=upload.binarypackagerelease.id)
    >>> upload.binarypackagerelease.version, bpf.filetype.title
    (u'3.14156', 'UDEB Format')

    >>> when = datetime(2005, 6, 22, tzinfo=utc)
    >>> urls = warty_i386_mirror.getURLsToCheckUpdateness(when=when)
    >>> [(freshness.name, url) for (freshness, url) in urls.items()]
    [('UP',
      u'http://foo.bar.com/pub/pool/main/p/pmount/pmount_1.9-1_all.deb'),
     ('ONEWEEKBEHIND',
      u'http://foo.bar.com/pub/pool/main/m/mozilla-firefox/mozilla-firefox_0.9_i386.deb')]

    >>> when = datetime(2005, 6, 20, 0, 1, tzinfo=utc)
    >>> urls = warty_i386_mirror.getURLsToCheckUpdateness(when=when)
    >>> [(freshness.name, url) for (freshness, url) in urls.items()]
    [('UP',
      u'http://foo.bar.com/pub/pool/main/p/pmount/pmount_1.9-1_all.deb'),
     ('TWODAYSBEHIND',
      u'http://foo.bar.com/pub/pool/main/m/mozilla-firefox/mozilla-firefox_0.9_i386.deb')]

If the mirror has no HTTP base url, we'll use the FTP one.

    >>> naked_mirror = removeSecurityProxy(warty_i386_mirror)
    >>> http_url = naked_mirror.distribution_mirror.http_base_url
    >>> naked_mirror.distribution_mirror.http_base_url = None
    >>> naked_mirror.distribution_mirror.ftp_base_url = (
    ...     'ftp://foo.bar.com/pub')
    >>> urls = warty_i386_mirror.getURLsToCheckUpdateness()
    >>> [(freshness.name, url) for (freshness, url) in urls.items()]
    [('UP', u'ftp://foo.bar.com/pub/pool/main/c/cdrkit/foobar_1.0_all.deb')]
    >>> naked_mirror.distribution_mirror.http_base_url = http_url


== Running the prober script ==

First we need to run the http server that's going to answer our requests.

    >>> from lp.registry.tests.test_distributionmirror_prober import (
    ...     HTTPServerTestSetup)
    >>> http_server = HTTPServerTestSetup()
    >>> http_server.setUp()

    >>> import subprocess
    >>> def run_prober(arguments):
    ...     cmd = ('cronscripts/distributionmirror-prober.py %s '
    ...            '--no-remote-hosts' % arguments)
    ...     prober = subprocess.Popen(
    ...         cmd, shell=True,stdin=subprocess.PIPE, stdout=subprocess.PIPE,
    ...         stderr=subprocess.PIPE)
    ...     stdout, stderr = prober.communicate()
    ...     return prober, stdout, stderr

Now we run the prober as another process, and check that the generated
output doesn't contain any errors and that the number of mirrors probed
is correct.

    >>> transaction.commit()
    >>> prober, stdout, stderr = run_prober('--content-type=archive')
    >>> print stdout
    <BLANKLINE>
    >>> print stderr
    INFO    Creating lockfile: /var/lock/launchpad-distributionmirror-prober.lock
    INFO    Probing Archive Mirrors
    INFO    Probed 4 mirrors.
    INFO    Disabling 1 mirror(s):
            http://launchpad.dev/ubuntu/+mirror/archive-404-mirror
    INFO    Re-enabling 1 mirror(s):
            http://launchpad.dev/ubuntu/+mirror/archive-mirror
    INFO    Done.
    <BLANKLINE>

    >>> prober, stdout, stderr = run_prober('--content-type=cdimage')
    >>> print stdout
    <BLANKLINE>
    >>> print stderr
    INFO    Creating lockfile: /var/lock/launchpad-distributionmirror-prober.lock
    INFO    Probing CD Image Mirrors
    INFO    Probed 4 mirrors.
    INFO    Disabling 1 mirror(s):
            http://launchpad.dev/ubuntu/+mirror/unreachable-mirror
    INFO    Done.
    <BLANKLINE>

If we run the prober again, it won't do anything, because it will realize
that the mirrors were probed recently, and, by default, don't need to be
probed again.

    >>> prober, stdout, stderr = run_prober('--content-type=cdimage')
    >>> print stdout
    <BLANKLINE>
    >>> print stderr
    INFO    Creating lockfile: /var/lock/launchpad-distributionmirror-prober.lock
    INFO    Probing CD Image Mirrors
    INFO    No mirrors to probe.
    INFO    Done.
    <BLANKLINE>

But we can override the default behaviour and tell the prober to check
all official mirrors independently if they were probed recently or not.

    >>> prober, stdout, stderr = run_prober('--content-type=cdimage --force')
    >>> print stdout
    <BLANKLINE>
    >>> print stderr
    INFO    Creating lockfile: /var/lock/launchpad-distributionmirror-prober.lock
    INFO    Probing CD Image Mirrors
    INFO    Probed 4 mirrors.
    INFO    Done.
    <BLANKLINE>

When a mirror is not reachabe or fail to mirror the content that it
should, it's marked as disabled (as you can see above) and thus not
shown on the public mirror listings. We'll keep probing these disabled
mirrors and once they're reachable and don't fail the content check
we'll enable them again.

    >>> cdimage_mirror = mirrorset.getByName('releases-mirror')
    >>> cdimage_mirror.enabled
    True
    >>> cdimage_mirror.enabled = False
    >>> transaction.commit()
    >>> prober, stdout, stderr = run_prober('--content-type=cdimage --force')
    >>> print stderr
    INFO    Creating lockfile: /var/lock/launchpad-distributionmirror-prober.lock
    INFO    Probing CD Image Mirrors
    INFO    Probed 4 mirrors.
    INFO    Re-enabling 1 mirror(s):
            http://launchpad.dev/ubuntu/+mirror/releases-mirror
    INFO    Done.
    <BLANKLINE>

    >>> http_server.tearDown()
    >>> transaction.abort()

    >>> cdimage_mirror.enabled
    True

    >>> [(mirror.name, mirror.speed.title, mirror.country.name)
    ...  for mirror in ubuntu.disabled_mirrors]
    [(u'archive-404-mirror', '512 Kbps', u'France'),
     (u'unreachable-mirror', '512 Kbps', u'France')]

== Mirror content ==

Now that we've probed some mirrors, we can check the content that was
found in them.

First, let's check the source and arch series of one archive mirror.

    >>> archive_mirror = mirrorset.getByName('archive-mirror')
    >>> archive_mirror.name
    u'archive-mirror'

    >>> mirror_arch_series = (
    ...     archive_mirror.getSummarizedMirroredArchSeries())

    # We only have a few publishing records, so most of the cdimage mirrors
    # will have Unknown as their freshness.
    >>> for mirror_arch_series in mirror_arch_series:
    ...     print (mirror_arch_series.distro_arch_series.displayname,
    ...            mirror_arch_series.freshness.title)
    (u'Ubuntu Warty i386', 'Up to date')

    >>> mirror_source_series = (
    ...     archive_mirror.getSummarizedMirroredSourceSeries())
    >>> for mirror_source_series in mirror_source_series:
    ...     print (mirror_source_series.distroseries.displayname,
    ...            mirror_source_series.freshness.title)
    (u'Warty', 'Up to date')
    (u'Hoary', 'Up to date')
    (u'Breezy Badger Autotest', 'Up to date')

Because all MirrorDistroArchSeriess and MirrorDistroSeriesSources of
this mirror have 'Up to date' as freshness, this mirror's overall
freshness will also be 'Up to date'.

    >>> archive_mirror.getOverallFreshness().title
    'Up to date'


Now we check the MirrorCDImageDistroSeriess of a cdimage mirror.

    >>> cdimage_mirror.name
    u'releases-mirror'

    >>> mirrored_series = []
    >>> for mirror_cdimage_series in cdimage_mirror.cdimage_series:
    ...     mirrored_series.append(
    ...         (mirror_cdimage_series.distroseries.displayname,
    ...          mirror_cdimage_series.flavour))
    >>> [series for series in sorted(mirrored_series)]
    [(u'Hoary', u'kubuntu'), (u'Hoary', u'ubuntu'), (u'Warty', u'ubuntu')]


In the case of cdimage mirrors, they're said to be up-to-date if they
mirror all ISO images contained in http://releases.ubuntu.com, which is
the case here.  Otherwise, their freshness is unknown.

    >>> cdimage_mirror.getOverallFreshness().title
    'Up to date'


== Mirror counts and bandwidth aggregates ==

We include a count of the number of mirrors per country, and the total
bandwidth (approximate) that they output. Here we test the aggregation
functions that achieve that.

First we import the classes required to test the view:

  >>> from zope.component import getMultiAdapter
  >>> from lp.registry.browser.distribution import (
  ...     DistributionMirrorsView)
  >>> from canonical.launchpad.webapp.servers import LaunchpadTestRequest

Create a view to test:

  >>> request = LaunchpadTestRequest()
  >>> view = getMultiAdapter((ubuntu, request), name='+archivemirrors')

Verify that the view is a DistributionMirrorsView:

  >>> isinstance(view, DistributionMirrorsView)
  True

We want to make sure that the view._sum_throughput method knows about all
the possible mirror speeds.

  >>> from lp.registry.interfaces.distributionmirror import MirrorSpeed
  >>> class MockMirror:
  ...     speed = None
  >>> mirrors = []
  >>> for speed in MirrorSpeed.items:
  ...     a = MockMirror()
  ...     a.speed = speed
  ...     mirrors.append(a)
  >>> print view._sum_throughput(mirrors)
  37 Gbps


Changing mirror owners
----------------------

The mirror owner can change the owner to another user.

    >>> owner = factory.makePerson(name='raccoon')
    >>> new_owner = factory.makePerson(name='bear')
    >>> mirror = ubuntu.newMirror(
    ...     owner, speed, brazil, content, http_base_url='http://a.ab/')


    >>> login_person(mirror.owner)
    >>> mirror.owner = new_owner
    >>> print mirror.owner.name
    bear


Deleting mirrors
----------------

Mirrors are sometimes created by spammers hoping that we'll publicize
whatever links they put there. Although we only publicize mirrors that
have been approved by an admin, it's annoying to the admins to have these
bogus mirrors on their list of unapproved mirrors, so we allow admins
(and only them) to delete these mirrors.

Only mirrors which have never been probed can be deleted this way.

    >>> login_person(cdimage_mirror.owner)
    >>> cdimage_mirror.destroySelf()
    Traceback (most recent call last):
    ...
    Unauthorized: ...

    >>> login('karl@canonical.com')
    >>> cdimage_mirror.last_probe_record is not None
    True
    >>> cdimage_mirror.destroySelf()
    Traceback (most recent call last):
    ...
    AssertionError: ...

    >>> invalid_mirror = mirrorset.getByName('invalid-mirror')
    >>> invalid_mirror.last_probe_record is None
    True
    >>> invalid_mirror.destroySelf()

    >>> mirrorset.getByName('invalid-mirror') is None
    True

Country DNS mirrors
-------------------

Country DNS mirrors are mirrors which have been assigned
$CC.archive.ubuntu.com or $CC.releases.ubuntu.com. These assignments are
tracked in Launchpad.

    >>> login('admin@canonical.com')
    >>> ubuntu_distro = getUtility(IDistributionSet).getByName('ubuntu')
    >>> de_archive_mirror = factory.makeMirror(ubuntu_distro,
    ...     "Technische Universitaet Dresden", country=82,
    ...     http_url="http://ubuntu.mirror.tudos.de/ubuntu/",
    ...     official_candidate=True)
    >>> davis_station_archive = factory.makeMirror(ubuntu_distro,
    ...     "Davis Station", country=9,
    ...     http_url="http://mirror.davis.antarctica.org/ubuntu",
    ...     official_candidate=True)
    >>> de_archive_mirror.status = MirrorStatus.OFFICIAL
    >>> de_archive_prober_log = factory.makeMirrorProbeRecord(de_archive_mirror)

Normal users can access country_dns_mirror, can see if a mirror is eligible
for the status, however, they may not change it:

    >>> login('test@canonical.com')
    >>> de_archive_mirror.canTransitionToCountryMirror()
    True
    >>> de_archive_mirror.transitionToCountryMirror(True)
    Traceback (most recent call last):
    ...
    Unauthorized: (<DistributionMirror at ...>, 'transitionToCountryMirror',
      'launchpad.Admin')

Mirror listing administrators may change the status however:

    >>> login('karl@canonical.com')
    >>> de_archive_mirror.transitionToCountryMirror(True)

Mirrors which are already set as country mirrors can't be 'set' as such
again:

    >>> de_archive_mirror.canTransitionToCountryMirror()
    False
    >>> de_archive_mirror.transitionToCountryMirror(True)

There cannot be multiple country mirrors of one type for one country:

    >>> login('karl@canonical.com')
    >>> proberecord = factory.makeMirrorProbeRecord(davis_station_archive)

    >>> print davis_station_archive.content.name
    ARCHIVE
    >>> print davis_station_archive.country_dns_mirror
    False
    >>> print davis_station_archive.country.name
    Antarctica

    >>> archive_mirror2 = getUtility(IDistributionMirrorSet).getByName(
    ...     'archive-mirror2')
    >>> print archive_mirror2.content.name
    ARCHIVE
    >>> print archive_mirror2.country_dns_mirror
    False
    >>> print archive_mirror2.country.name
    Antarctica

    >>> davis_station_archive.status = MirrorStatus.OFFICIAL

    >>> davis_station_archive.transitionToCountryMirror(True)
    >>> archive_mirror2.transitionToCountryMirror(True)
    Traceback (most recent call last):
    ...
    CountryMirrorAlreadySet: Antarctica already has a country Archive mirror
      set.

Mirrors which have not been probed may not be marked as country mirrors:

    >>> linux_au_mirror = factory.makeMirror(ubuntu_distro,
    ...     "Linux.org.au", country=14,
    ...     http_url="http://mirror.linux.org.au/ubuntu",
    ...     official_candidate=True)
    >>> linux_au_mirror.status = MirrorStatus.OFFICIAL
    >>> linux_au_mirror.transitionToCountryMirror(True)
    Traceback (most recent call last):
    ...
    MirrorNotProbed: This mirror may not be set as a country mirror as it has
      not been probed.

Mirrors which are not official or do not have an HTTP URL may not be set as
country mirrors:

    >>> login('admin@canonical.com')
    >>> osuosl_mirror = factory.makeMirror(ubuntu_distro,
    ...     "OSU Open Source Lab", country=226,
    ...     ftp_url="ftp://ubuntu.osuosl.org/pub/ubuntu/",
    ...     official_candidate=True)
    >>> osuosl_mirror.status = MirrorStatus.OFFICIAL
    >>> print osuosl_mirror.http_base_url
    None

    >>> osuosl_mirror.canTransitionToCountryMirror()
    False

    >>> osuosl_mirror.transitionToCountryMirror(None)
    Traceback (most recent call last):
    ...
    NoneError: None isn't acceptable as a value for
      DistributionMirror.country_dns_mirror

    >>> osuosl_mirror.transitionToCountryMirror(True)
    Traceback (most recent call last):
    ...
    MirrorHasNoHTTPURL: This mirror may not be set as a country mirror as it
      does not have an HTTP URL set.
    >>> logout()