~launchpad-pqm/launchpad/devel

== DDTP-TARBALL upload ==

This test will describe how launchpad treats an upload of DDTP
tarball.

DDTP (Debian Description Translation Project) tarball upload is
consisted of a tarball (specified in the changesfile as a
DDTP-TARBALL) containing all the supported index files for the DDTP
contents driven by component.

The index-files needs to be published in the ubuntu archive, according
its target suite and component, under the 'i18n' (see
http://en.wikipedia.org/wiki/Internationalization_and_localization)
directory.

Soonish APT will be able to download and follow the indexes files,
then present the appropriated translated package description the users
in ubuntu.

For further info about DDTP see:
  https://wiki.ubuntu.com/TranslatedPackageDescriptionsSpec

  >>> from lp.registry.interfaces.distribution import IDistributionSet

  >>> ubuntutest = getUtility(IDistributionSet)['ubuntutest']
  >>> breezy_autotest = ubuntutest['breezy-autotest']

  >>> from lp.archiveuploader.nascentupload import NascentUpload
  >>> from lp.archiveuploader.tests import datadir, getPolicy

  >>> from canonical.launchpad.ftests import import_public_test_keys
  >>> import_public_test_keys()

Login as an admin (or ubuntutest.archive_admin if we have one), since
we need to access and modify PackageUpload records and other tables.

  >>> login('foo.bar@canonical.com')

Set the email address for announcements:

  >>> breezy_autotest.changeslist = 'test@example.com'

First, test the rejection of a missapplied changesfile name, which
doesn't follow the accepted format,  "<pkg>_<version>_<arch>.changes"

  >>> sync_policy = getPolicy(
  ...     name='sync', distro='ubuntutest', distroseries=None)

  >>> from lp.services.log.logger import DevNullLogger
  >>> upload = NascentUpload.from_changesfile_path(
  ...     datadir('ddtp-tarball/translations-main_20060728.changes'),
  ...     sync_policy, DevNullLogger())
  >>> upload.process()
  Traceback (most recent call last):
  ...
  EarlyReturnUploadError: An error occurred that prevented further processing.

Now do a proper upload into the system.

  >>> insecure_policy = getPolicy(
  ...     name='insecure', distro='ubuntutest', distroseries=None)

  >>> from lp.services.log.logger import FakeLogger
  >>> upload = NascentUpload.from_changesfile_path(
  ...     datadir('ddtp-tarball/translations-main_20060728_all.changes'),
  ...     insecure_policy, FakeLogger())
  DEBUG Verifying signature on translations-main_20060728_all.changes

  >>> upload.process()
  DEBUG Beginning processing.
  DEBUG Verifying the changes file.
  DEBUG Verifying files in upload.
  DEBUG Single Custom Upload detected.
  DEBUG Finished checking upload.

It was not rejected.

  >>> upload.is_rejected
  False

  >>> success = upload.do_accept()
  DEBUG Creating queue entry
  ...
  DEBUG Sent a mail:
  DEBUG   Subject: [ubuntutest/breezy-autotest] translations_main_20060728.tar.gz - (Accepted)
  DEBUG   Sender: Root <root@localhost>
  DEBUG   Recipients: Foo Bar <foo.bar@canonical.com>
  DEBUG   Bcc: Root <root@localhost>
  DEBUG   Body:
  DEBUG .
  DEBUG * Duhh
  DEBUG
  DEBUG
  DEBUG Date: Thu, 01 Aug 2006 14:20:25 +0300
  DEBUG Changed-By: Foo Bar <foo.bar@canonical.com>
  DEBUG Origin: Ubuntu/warty
  DEBUG
  DEBUG
  DEBUG ==
  DEBUG
  DEBUG Announcing to test@example.com
  DEBUG
  DEBUG Thank you for your contribution to Ubuntu Test.
  DEBUG
  DEBUG --
  DEBUG You are receiving this email because you are the uploader, maintainer or
  DEBUG signer of the above package.
  DEBUG Sent a mail:
  DEBUG   Subject: [ubuntutest/breezy-autotest] translations_main_20060728.tar.gz - (Accepted)
  DEBUG   Sender: Foo Bar <foo.bar@canonical.com>
  DEBUG   Recipients: test@example.com
  DEBUG   Bcc: Root <root@localhost>
  DEBUG   Body:
  DEBUG .
  DEBUG * Duhh
  DEBUG
  DEBUG
  DEBUG Date: Thu, 01 Aug 2006 14:20:25 +0300
  DEBUG Changed-By: Foo Bar <foo.bar@canonical.com>
  DEBUG Origin: Ubuntu/warty
  DEBUG

And all things worked.

  >>> success
  True

  >>> not upload.rejection_message
  True

We need to commit the transaction to be able to use the librarian files.

  >>> flush_database_updates()
  >>> transaction.commit()

Let's use the script to fetch the ddtp-tarball upload:

  >>> import subprocess
  >>> import os
  >>> import sys
  >>> from canonical.config import config

  >>> script = os.path.join(
  ...    config.root, "scripts", "ftpmaster-tools", "queue")

  >>> process = subprocess.Popen([sys.executable, script, "-Q", "accepted",
  ...                             "-s", "breezy-autotest", "fetch", "trans",
  ...                             "-d", "ubuntutest"],
  ...                            stdout=subprocess.PIPE)
  >>> stdout, stderr = process.communicate()
  >>> process.returncode
  0
  >>> print stdout
  Initializing connection to queue accepted
  Running: "fetch trans"
  Fetching ubuntutest/breezy-autotest (ACCEPTED) 1/1
  ---------------------------------------------------------------------------
  Constructing translations-main_20060728_all.changes
  Constructing translations_main_20060728.tar.gz
  ---------------------------------------------------------------------------
                                                                 1/1 total
  <BLANKLINE>

Check if the files were written:

  >>> os.path.exists('translations-main_20060728_all.changes')
  True
  >>> os.path.exists('translations_main_20060728.tar.gz')
  True

Cleanup the mess:

  >>> os.remove('translations-main_20060728_all.changes')
  >>> os.remove('translations_main_20060728.tar.gz')


Inspect the QUEUE looking for an ACCEPTED entry corresponding to the
previous upload:

  >>> from lp.soyuz.enums import PackageUploadStatus
  >>> queue_item = breezy_autotest.getQueueItems(
  ...      status=PackageUploadStatus.ACCEPTED)[0]
  >>> queue_item.customfiles[0].libraryfilealias.filename
  u'translations_main_20060728.tar.gz'

Do the publish, i.e process the CustomUpload decompressing and writing
the tarball contents in the archive, collect and check debug message:

The umask must be 022 in order to perform the upload.

  >>> old_mask = os.umask(022)
  >>> pub_records = queue_item.realiseUpload(FakeLogger())
  DEBUG Publishing custom translations_main_20060728.tar.gz to ubuntutest/breezy-autotest
  >>> print '%03o' % os.umask(old_mask)
  022

Check what was published in the target directory:

  >>> import os
  >>> archive_dir = '/var/tmp/archive/'

DDTP indexes are published in the "i18n" directory inside archive dists:

  >>> upgrade_dir = 'ubuntutest/dists/breezy-autotest/main/i18n'
  >>> target_dir = os.path.join(archive_dir, upgrade_dir)

We simply decompress the contents of the tarball, the files follow
the format:

  Translation-<lang-code>{'', '.gz', '.bz2'}

We have only used uncompressed files for tests:

  >>> dir_list = os.listdir(target_dir)
  >>> for filename in sorted(dir_list):
  ...    content = open(os.path.join(target_dir, filename), 'rb').read()
  ...    print filename, len(content)
  Translation-bn 0
  Translation-ca 0
  Translation-en 16
  Translation-pt_BR 15

** Note that a incomming directory inside the tarball "i18n" directory
was skipped by the DDTP processor

Upload a new DDTP tarball which is supposed to only add a new language
index , the "bn" one, and left the rest untouched, this feature is
know as "partial update" and will save some storage when fixing specific
language indexes or adding new ones:

  >>> insecure_policy = getPolicy(
  ...     name='insecure', distro='ubuntutest', distroseries=None)

  >>> upload = NascentUpload.from_changesfile_path(
  ...     datadir('ddtp-tarball/translations-main_20060817_all.changes'),
  ...     insecure_policy, DevNullLogger())
  >>> upload.process()
  >>> upload.is_rejected
  False
  >>> success = upload.do_accept()
  >>> success
  True


We need to commit the transaction to be able to use the librarian files.

  >>> flush_database_updates()
  >>> transaction.commit()


Our archive uses `dsync` to replace identical files by hard link in
order to save some space. This feature breaks the tarfile.extract method,
because it doesn't remove pre-existing files, it simply overwrite them.

Create a hard link to "bn" package file.

  >>> src = os.path.join(target_dir, 'Translation-ca')
  >>> dest = os.path.join(target_dir, 'Translation-bn')
  >>> os.remove(dest)
  >>> os.link(src, dest)

  >>> os.path.exists(dest)
  True
  >>> os.stat(dest).st_nlink
  2
  >>> os.stat(src).st_nlink
  2


Retrieve and publish new custom upload:

  >>> queue_item = breezy_autotest.getQueueItems(
  ...      status=PackageUploadStatus.ACCEPTED)[0]

  >>> pub_records = queue_item.realiseUpload()


Check if there is new content for "bn" index:

  >>> dir_list = os.listdir(target_dir)
  >>> for filename in sorted(dir_list):
  ...    content = open(os.path.join(target_dir, filename), 'rb').read()
  ...    print filename, len(content)
  Translation-bn 22
  Translation-ca 0
  Translation-en 16
  Translation-pt_BR 15


Check if the contents of "ca" were not affected by the "bn" contents
(this would happen if "bn" was not removed, left as hard link):

  >>> open(os.path.join(target_dir, 'Translation-bn')).read()
  'hardlinks are bad !!!\n'

  >>> open(os.path.join(target_dir, 'Translation-ca')).read()
  ''

Remove the directory to keep the test working.

  >>> import shutil
  >>> shutil.rmtree(target_dir)