~launchpad-pqm/launchpad/devel

= Build FAILEDTOUPLOAD workflow =

A FAILEDTOUPLOAD build record represents a build procedure that
resulted in a binary upload, already gathered in the production
machine disk, that could not be successfully uploaded to Launchpad.

It can be caused by several reasons, inconsistencies generated by the
builders or code failures in upload system.

In all of these cases we want the 'buildd-admins' to be aware of the
failure and provide them all the information required to debug the failure.

== FAILEDTOUPLOAD email notification ==

Once a build result is recognised as FAILEDTOUPLOAD by the
buildmaster/slave-scanner code, an build notification will be issued.
See more information in build-notification.txt file.

  >>> from lp.soyuz.interfaces.binarypackagebuild import (
  ...     IBinaryPackageBuildSet)
  >>> from lp.testing.mail_helpers import pop_notifications
  >>> buildset = getUtility(IBinaryPackageBuildSet)

Let's use a sampledata build record in FAILEDTOUPLOAD:

  >>> failedtoupload_candidate = buildset.getByID(22)

  >>> print failedtoupload_candidate.title
  i386 build of cdrkit 1.0 in ubuntu breezy-autotest RELEASE

  >>> print failedtoupload_candidate.status.name
  FAILEDTOUPLOAD

  >>> print failedtoupload_candidate.upload_log.filename
  upload_22_log.txt

FAILEDTOUPLOAD notification requires 'extra_info' argument to be filled:

  >>> failedtoupload_candidate.notify()
  Traceback (most recent call last):
  ...
  AssertionError: Extra information is required for FAILEDTOUPLOAD notifications.

Normally 'extra_info' will contain the output generated by the binary
upload procedure with instructions to reprocess it:

  >>> failedtoupload_candidate.notify(extra_info='ANY OUTPUT')

  >>> notifications = pop_notifications()
  >>> len(notifications)
  3

As for the other failure notifications we will send emails for the
'lp-buildd-admins' team members (celso.providelo & foo.bar) and for
source creator (mark) as specified in the test configuration:

  >>> from canonical.config import config
  >>> config.builddmaster.notify_owner
  True


  >>> for build_notification in notifications:
  ...      build_notification['To']
  'celso.providelo@canonical.com'
  'foo.bar@canonical.com'
  'mark@example.com'

Note that the generated notification contain the 'extra_info' content:

  >>> build_notification = notifications.pop(0)

  >>> build_notification['Subject']
  '[Build #22] i386 build of cdrkit 1.0 in ubuntu breezy-autotest RELEASE'

  >>> build_notification['X-Launchpad-Build-State']
  'FAILEDTOUPLOAD'

  >>> build_notification['X-Creator-Recipient']
  'mark@example.com'

  >>> notification_body = build_notification.get_payload()
  >>> print notification_body #doctest: -NORMALIZE_WHITESPACE
  <BLANKLINE>
   * Source Package: cdrkit
   * Version: 1.0
   * Architecture: i386
   * Archive: ubuntu primary archive
   * Component: main
   * State: Failed to upload
   * Duration: 1 minute
   * Build Log: http://launchpad.dev/ubuntu/+source/cdrkit/1.0/+build/22/+fil=
  es/netapplet-1.0.0.tar.gz
   * Builder: http://launchpad.dev/builders/bob
   * Source: http://launchpad.dev/ubuntu/+source/cdrkit/1.0
  <BLANKLINE>
  Upload log:
  ANY OUTPUT
  <BLANKLINE>
  If you want further information about this situation, feel free to
  contact a member of the Launchpad Buildd Administrators team.
  <BLANKLINE>
  --
  i386 build of cdrkit 1.0 in ubuntu breezy-autotest RELEASE
  http://launchpad.dev/ubuntu/+source/cdrkit/1.0/+build/22
  <BLANKLINE>


== Rescuing FAILEDTOBUILD record ==

Let's emulate the procedure of rescuing an FAILEDTOUPLOAD build.
A FAILEDTOUPLOAD build obviously has no binaries:

  >>> print failedtoupload_candidate.status.name
  FAILEDTOUPLOAD

  >>> failedtoupload_candidate.binarypackages.count()
  0

  >>> failedtoupload_candidate.verifySuccessfulUpload()
  False

But once the person responsible have received the information about
the binary upload failure via build notification, or accessing the
build page, he will be aware of:

 * The targeted Build ID
 * Path where the rejected/failed upload is, usually in some
   destination inside buildd upload queue (/srv/launchpad.net/builddmaster/*)

Having those informations, one of buildd-admin team members will be
able to reprocess the binary upload via an shell account following
these steps:

 * Move the rejected/failed upload directory to 'incoming' queue.
 * Issue a process-upload command as:
   `process-upload.py
        -C buildd
        -r <DISTRORELEASE>
        -b <BUILD_ID>
        -J <LEAF_NAME>
        -Mvv
        <UPLOAD_FUL_PATH>`
 * LEAF_NAME is the upload directory name, required to not affect
   other uploads in progress.
 * '-Mvv' will produce descriptive debug information and suppress
   upload email notifications, that are definetelly *not wanted*  for
   binary uploads.
 * It's possible to use '-n' (dry-run) if he is unsure about the parameters.

For simplicity we will process the binary upload using the
NascentUpload class instead of calling the script:

  >>> login('foo.bar@canonical.com')

  >>> from lp.archiveuploader.nascentupload import NascentUpload
  >>> from lp.archiveuploader.tests import datadir, getPolicy

The policy contains the right arguments to reprocess the build for the
right target location, they are passed via process-upload command-line:

  >>> buildd_policy = getPolicy(
  ...     name='buildd',
  ...     distro=failedtoupload_candidate.distribution.name,
  ...     distroseries=failedtoupload_candidate.distro_series.name)

  >>> from lp.services.log.logger import DevNullLogger
  >>> cdrkit_bin_upload = NascentUpload.from_changesfile_path(
  ...     datadir('suite/cdrkit_1.0/cdrkit_1.0_i386.changes'),
  ...     buildd_policy, DevNullLogger())
  >>> cdrkit_bin_upload.process()
  >>> cdrkit_bin_upload.is_rejected
  False
  >>> success = cdrkit_bin_upload.do_accept(build=failedtoupload_candidate)
  >>> print cdrkit_bin_upload.queue_root.status.name
  NEW

After successfully reprocessing binary upload the originally
FAILEDTOUPLOAD build record is already in FULLYBUILT state and the
previously stored upload_log is dereferenced (they are both updated
during the upload processing time):

  >>> print failedtoupload_candidate.status.name
  FULLYBUILT

  >>> print failedtoupload_candidate.upload_log
  None

And contains the associated binaries.

  >>> failedtoupload_candidate.binarypackages.count()
  1
  >>> failedtoupload_candidate.verifySuccessfulUpload()
  True