~launchpad-pqm/launchpad/devel

= Debian Installer Custom Uploads =

Debian Installer uploads are treated as one of the Soyuz "custom"
types.

They come from buildd as result of the 'debian-installer' source
build. This binary upload usually contains:

 * The debian-installer binaries (DEB)
 * The debian-installer translation (TAR.GZ, custom ROSETTA_TRANSLATIONS)
 * The debian-installer special archive files (TAR.GZ, custom DEBIAN_INSTALLER)

We have a special processor to publish debian-installer special archive
file contents.

The debian-installer filename should be something like:

<BASE>_<VERSION>_<ARCH>.tar.gz

Where:

 * BASE: base name (usually 'debian-installer-images')

 * VERSION: encoded version (something like '20061102ubuntu14')
   * if the version string contains '.0.' we assume it is a
     'daily-installer', otherwise, it is a normal 'installer'

 * ARCH: targeted architecture tag ('i386', 'amd64', etc)

The contents are extracted in the archive, respecting its type
('installer' or 'daily-installer'), in the following path:

 * <ARCHIVE>/dists/<SUITE>/main/<TYPE>-<ARCH>/


Let's try one test upload and verify the results of its publication.
In order to keep the tests simple we will upload only the
debian-installer tarball, despite the fact that it's very unlikely to
happen in production:

  >>> from lp.testing.gpgkeys import import_public_test_keys
  >>> import_public_test_keys()
  >>> login('foo.bar@canonical.com')

  >>> from lp.registry.interfaces.distribution import IDistributionSet
  >>> ubuntutest = getUtility(IDistributionSet)['ubuntutest']

  >>> from lp.archiveuploader.nascentupload import NascentUpload
  >>> from lp.archiveuploader.tests import datadir, getPolicy

  >>> anything_policy = getPolicy(
  ...     name='anything', distro='ubuntutest', distroseries=None)
  >>> anything_policy.setDistroSeriesAndPocket('hoary-test')

Set the email address for annoucements emails:

  >>> anything_policy.distroseries.changeslist = 'announce@example.com'

  >>> from lp.services.log.logger import DevNullLogger
  >>> upload = NascentUpload.from_changesfile_path(
  ...     datadir(
  ...       'debian-installer/debian-installer_20070214ubuntu1_i386.changes'),
  ...     anything_policy, DevNullLogger())
  >>> upload.process()
  >>> success = upload.do_accept()
  >>> success
  True

Upload was successfully 'processed' and 'accepted'. Two email messages
were generated (acceptance and announcement). Inspect the queue (IDRQ)
information we find out that it contains one custom upload.

  >>> upload.queue_root.status.name
  'ACCEPTED'

Make librarian files available for publication:

  >>> import transaction
  >>> transaction.commit()

  >>> from lp.services.mail import stub
  >>> len(stub.test_emails)
  2
  >>> upload.queue_root.customfiles.count()
  1

Let's force an error simulating a conflict in archive:

  >>> import os
  >>> archive_path = "/var/tmp/archive/"
  >>> installer_path = os.path.join(
  ...     archive_path, "ubuntutest/dists/hoary-test/main/installer-i386")

  >>> os.makedirs(os.path.join(installer_path, '20070214ubuntu1'))

Try to publish the custom upload, but it will be skipped due the disk
conflict and the queue item will remain in ACCEPTED

  >>> from lp.services.log.logger import FakeLogger
  >>> pub_records = upload.queue_root.realiseUpload(FakeLogger())
  DEBUG Publishing custom debian-installer-images_20070214ubuntu1_i386.tar.gz to ubuntutest/hoary-test
  ERROR Queue item ignored: installer build i386 for architecture 20070214ubuntu1 already exists
  >>> upload.queue_root.status.name
  'ACCEPTED'

Let's remove the conflicted directory from disk and expect it to work
next time:

  >>> os.rmdir(os.path.join(installer_path, '20070214ubuntu1'))

The upload task requires that the umask be 022, otherwise things will end up
with the wrong permission.

  >>> old_mask = os.umask(002)
  >>> pub_records = upload.queue_root.realiseUpload(FakeLogger())
  DEBUG ...
  ERROR Queue item ignored: Bad umask; expected 022, got 002

Reset the umask, remove any residual directories, and try again.

  >>> os.rmdir(os.path.join(installer_path, '20070214ubuntu1'))
  >>> print '%03o' % os.umask(old_mask)
  002

Process (publish) the custom upload:

  >>> pub_records = upload.queue_root.realiseUpload(DevNullLogger())
  >>> upload.queue_root.status.name
  'DONE'

The current debian-installer directory (20070214ubuntu1) and a link to
the last version (we only have the one we uploaded) should be
presented:

  >>> sorted(os.listdir(installer_path))
  ['20070214ubuntu1', 'current']

As a consistency check, let's verify one of the files inside the
tarball:

  >>> the_file = ("current/images/netboot/ubuntu-installer/i386/"
  ...             "pxelinux.cfg.serial-9600/default")
  >>> the_file_path = os.path.join(installer_path, the_file)
  >>> print open(the_file_path).read()
  hey

Also check if the symbolic link included in the installer tarball
looks sane:

  >>> the_link = "current/images/netboot/foo"
  >>> the_link_path = os.path.join(installer_path, the_link)
  >>> os.readlink(the_link_path)
  'ubuntu-installer/i386/pxelinux.cfg.serial-9600/default'

  >>> the_dir_link = "current/images/netboot/link_to_dir"
  >>> the_dir_link_path = os.path.join(installer_path, the_dir_link)
  >>> os.readlink(the_dir_link_path)
  'ubuntu-installer/i386/pxelinux.cfg.serial-9600/'

Check permissions on the created directories to satisfy the
publication path. They are created with os.makedirs() and forced to
set permission '0755' instead of the default '0777' (see further
details in bug #107068):

  >>> oct(os.stat(installer_path).st_mode & 0777)
  '0755'

  >>> the_path = os.path.join(installer_path, os.pardir)
  >>> oct(os.stat(the_path).st_mode & 0777)
  '0755'

Check permissions of created files and directories:

  >>> the_file = ("current/images/netboot/ubuntu-installer/i386/"
  ...             "pxelinux.cfg.serial-9600/default")
  >>> the_file_path = os.path.join(installer_path, the_file)
  >>> oct(os.stat(the_file_path).st_mode & 0777)
  '0644'

  >>> the_dir = ("current/images/netboot/ubuntu-installer/i386/"
  ...            "pxelinux.cfg.serial-9600/")
  >>> the_dir_path = os.path.join(installer_path, the_dir)
  >>> oct(os.stat(the_dir_path).st_mode & 0777)
  '0755'


Remove local archive contents:

  >>> import shutil
  >>> shutil.rmtree(archive_path)