~launchpad-pqm/launchpad/devel

DDTP-TARBALL upload
-------------------

This test will describe how launchpad treats an upload of DDTP
tarball.

DDTP (Debian Description Translation Project) tarball upload is
consisted of a tarball (specified in the changesfile as a
DDTP-TARBALL) containing all the supported index files for the DDTP
contents driven by component.

The index-files needs to be published in the ubuntu archive, according
its target suite and component, under the 'i18n' (see
http://en.wikipedia.org/wiki/Internationalization_and_localization)
directory.

Soonish APT will be able to download and follow the indexes files,
then present the appropriated translated package description the users
in ubuntu.

For further info about DDTP see:
  https://wiki.ubuntu.com/TranslatedPackageDescriptionsSpec

    >>> from lp.registry.interfaces.distribution import IDistributionSet

    >>> ubuntutest = getUtility(IDistributionSet)['ubuntutest']
    >>> breezy_autotest = ubuntutest['breezy-autotest']

    >>> from lp.archiveuploader.nascentupload import NascentUpload
    >>> from lp.archiveuploader.tests import datadir, getPolicy

    >>> from canonical.launchpad.ftests import import_public_test_keys
    >>> import_public_test_keys()

Login as an admin (or ubuntutest.archive_admin if we have one), since
we need to access and modify PackageUpload records and other tables.

    >>> login('foo.bar@canonical.com')

Set the email address for announcements:

    >>> breezy_autotest.changeslist = 'test@example.com'

First, test the rejection of a missapplied changesfile name, which
doesn't follow the accepted format,  "<pkg>_<version>_<arch>.changes"

    >>> sync_policy = getPolicy(
    ...     name='sync', distro='ubuntutest', distroseries=None)

    >>> from lp.services.log.logger import DevNullLogger
    >>> upload = NascentUpload.from_changesfile_path(
    ...     datadir('ddtp-tarball/translations-main_20060728.changes'),
    ...     sync_policy, DevNullLogger())
    >>> upload.process()
    Traceback (most recent call last):
    ...
    EarlyReturnUploadError: An error occurred that prevented further
    processing.

Now do a proper upload into the system.

    >>> insecure_policy = getPolicy(
    ...     name='insecure', distro='ubuntutest', distroseries=None)

    >>> from lp.services.log.logger import FakeLogger
    >>> upload = NascentUpload.from_changesfile_path(
    ...     datadir('ddtp-tarball/translations-main_20060728_all.changes'),
    ...     insecure_policy, FakeLogger())
    DEBUG Verifying signature on translations-main_20060728_all.changes

    >>> upload.process()
    DEBUG Beginning processing.
    DEBUG Verifying the changes file.
    DEBUG Verifying files in upload.
    DEBUG Single Custom Upload detected.
    DEBUG Finished checking upload.

It was not rejected.

    >>> upload.is_rejected
    False

    >>> success = upload.do_accept()
    DEBUG Creating queue entry
    ...
    DEBUG Sent a mail:
    DEBUG   Subject: [ubuntutest/breezy-autotest]
    translations_main_20060728.tar.gz - (Accepted)
    DEBUG   Sender: Root <root@localhost>
    DEBUG   Recipients: Foo Bar <foo.bar@canonical.com>
    DEBUG   Bcc: Root <root@localhost>
    DEBUG   Body:
    DEBUG .
    DEBUG * Duhh
    DEBUG
    DEBUG
    DEBUG Date: Thu, 01 Aug 2006 14:20:25 +0300
    DEBUG Changed-By: Foo Bar <foo.bar@canonical.com>
    DEBUG Origin: Ubuntu/warty
    DEBUG
    DEBUG
    DEBUG ==
    DEBUG
    DEBUG Announcing to test@example.com
    DEBUG
    DEBUG Thank you for your contribution to Ubuntu Test.
    DEBUG
    DEBUG --
    DEBUG You are receiving this email because you are the uploader,
    maintainer or
    DEBUG signer of the above package.
    DEBUG Sent a mail:
    DEBUG   Subject: [ubuntutest/breezy-autotest]
    translations_main_20060728.tar.gz - (Accepted)
    DEBUG   Sender: Foo Bar <foo.bar@canonical.com>
    DEBUG   Recipients: test@example.com
    DEBUG   Bcc: Root <root@localhost>
    DEBUG   Body:
    DEBUG .
    DEBUG * Duhh
    DEBUG
    DEBUG
    DEBUG Date: Thu, 01 Aug 2006 14:20:25 +0300
    DEBUG Changed-By: Foo Bar <foo.bar@canonical.com>
    DEBUG Origin: Ubuntu/warty
    DEBUG

And all things worked.

    >>> success
    True

    >>> not upload.rejection_message
    True

We need to commit the transaction to be able to use the librarian files.

    >>> flush_database_updates()
    >>> transaction.commit()

Let's use the script to fetch the ddtp-tarball upload:

    >>> import subprocess
    >>> import os
    >>> import sys
    >>> from canonical.config import config

    >>> script = os.path.join(
    ...    config.root, "scripts", "ftpmaster-tools", "queue")

    >>> process = subprocess.Popen([sys.executable, script, "-Q", "accepted",
    ...                             "-s", "breezy-autotest", "fetch", "trans",
    ...                             "-d", "ubuntutest"],
    ...                            stdout=subprocess.PIPE)
    >>> stdout, stderr = process.communicate()
    >>> process.returncode
    0
    >>> print stdout
    Initializing connection to queue accepted
    Running: "fetch trans"
    Fetching ubuntutest/breezy-autotest (ACCEPTED) 1/1
    ---------------------------------------------------------------------------
    Constructing translations-main_20060728_all.changes
    Constructing translations_main_20060728.tar.gz
    ---------------------------------------------------------------------------
								   1/1 total
    <BLANKLINE>

Check if the files were written:

    >>> os.path.exists('translations-main_20060728_all.changes')
    True
    >>> os.path.exists('translations_main_20060728.tar.gz')
    True

Cleanup the mess:

    >>> os.remove('translations-main_20060728_all.changes')
    >>> os.remove('translations_main_20060728.tar.gz')


Inspect the QUEUE looking for an ACCEPTED entry corresponding to the
previous upload:

    >>> from lp.soyuz.enums import PackageUploadStatus
    >>> queue_item = breezy_autotest.getQueueItems(
    ...      status=PackageUploadStatus.ACCEPTED)[0]
    >>> queue_item.customfiles[0].libraryfilealias.filename
    u'translations_main_20060728.tar.gz'

Do the publish, i.e process the CustomUpload decompressing and writing
the tarball contents in the archive, collect and check debug message:

The umask must be 022 in order to perform the upload.

    >>> old_mask = os.umask(022)
    >>> pub_records = queue_item.realiseUpload(FakeLogger())
    DEBUG Publishing custom translations_main_20060728.tar.gz to
    ubuntutest/breezy-autotest
    >>> print '%03o' % os.umask(old_mask)
    022

Check what was published in the target directory:

    >>> import os
    >>> archive_dir = '/var/tmp/archive/'

DDTP indexes are published in the "i18n" directory inside archive dists:

    >>> upgrade_dir = 'ubuntutest/dists/breezy-autotest/main/i18n'
    >>> target_dir = os.path.join(archive_dir, upgrade_dir)

We simply decompress the contents of the tarball, the files follow
the format:

  Translation-<lang-code>{'', '.gz', '.bz2'}

We have only used uncompressed files for tests:

    >>> dir_list = os.listdir(target_dir)
    >>> for filename in sorted(dir_list):
    ...    content = open(os.path.join(target_dir, filename), 'rb').read()
    ...    print filename, len(content)
    Translation-bn 0
    Translation-ca 0
    Translation-en 16
    Translation-pt_BR 15

** Note that a incomming directory inside the tarball "i18n" directory
was skipped by the DDTP processor

Upload a new DDTP tarball which is supposed to only add a new language
index , the "bn" one, and left the rest untouched, this feature is
know as "partial update" and will save some storage when fixing specific
language indexes or adding new ones:

    >>> insecure_policy = getPolicy(
    ...     name='insecure', distro='ubuntutest', distroseries=None)

    >>> upload = NascentUpload.from_changesfile_path(
    ...     datadir('ddtp-tarball/translations-main_20060817_all.changes'),
    ...     insecure_policy, DevNullLogger())
    >>> upload.process()
    >>> upload.is_rejected
    False
    >>> success = upload.do_accept()
    >>> success
    True


We need to commit the transaction to be able to use the librarian files.

    >>> flush_database_updates()
    >>> transaction.commit()


Our archive uses `dsync` to replace identical files by hard link in
order to save some space. This feature breaks the tarfile.extract method,
because it doesn't remove pre-existing files, it simply overwrite them.

Create a hard link to "bn" package file.

    >>> src = os.path.join(target_dir, 'Translation-ca')
    >>> dest = os.path.join(target_dir, 'Translation-bn')
    >>> os.remove(dest)
    >>> os.link(src, dest)

    >>> os.path.exists(dest)
    True
    >>> os.stat(dest).st_nlink
    2
    >>> os.stat(src).st_nlink
    2


Retrieve and publish new custom upload:

    >>> queue_item = breezy_autotest.getQueueItems(
    ...      status=PackageUploadStatus.ACCEPTED)[0]

    >>> pub_records = queue_item.realiseUpload()


Check if there is new content for "bn" index:

    >>> dir_list = os.listdir(target_dir)
    >>> for filename in sorted(dir_list):
    ...    content = open(os.path.join(target_dir, filename), 'rb').read()
    ...    print filename, len(content)
    Translation-bn 22
    Translation-ca 0
    Translation-en 16
    Translation-pt_BR 15


Check if the contents of "ca" were not affected by the "bn" contents
(this would happen if "bn" was not removed, left as hard link):

    >>> open(os.path.join(target_dir, 'Translation-bn')).read()
    'hardlinks are bad !!!\n'

    >>> open(os.path.join(target_dir, 'Translation-ca')).read()
    ''

Remove the directory to keep the test working.

    >>> import shutil
    >>> shutil.rmtree(target_dir)