~launchpad-pqm/launchpad/devel

= BuildQueue =

BuildQueue class represents the run-time-records of builds being
processed.

See buildd-queuebuilder.txt for more information about the BuildQueue
initialisation.

BuildQueue should provide all the information needed for dispatching
and collecting build results.

As soon as a build job is processed succesfully (dispatched &
collected) the BuildQueue record representing it is removed.

    >>> from canonical.launchpad.webapp.testing import verifyObject
    >>> from lp.buildmaster.interfaces.buildqueue import (
    ...     IBuildQueue, IBuildQueueSet)

Get an instance of Builder from the current sampledata using the
IBuildQueueSet zope utility. The zope utility itself will be
documented later on this document, we are only using it to retrive an
IBuildQueue record with the security proxies in place:

    >>> bq = getUtility(IBuildQueueSet).get(1)

Instances returned by IBuildQueueSet provides the IBuildQueue interface:

    >>> verifyObject(IBuildQueue, bq)
    True

The IBuild record related to this job is provided by the 'build'
attribute:

    >>> from lp.soyuz.interfaces.binarypackagebuild import IBinaryPackageBuildSet
    >>> build = getUtility(IBinaryPackageBuildSet).getByQueueEntry(bq)
    >>> build.id
    8
    >>> build.status.name
    'BUILDING'

The static timestamps, representing when the record was initialised
(inserted) and when the job was dispatched are provided as datetime
instances:

    >>> bq.job.date_created
    datetime.datetime(2005, 6, 15, 9, 14, 12, 820778, tzinfo=<UTC>)

    >>> bq.date_started
    datetime.datetime(2005, 6, 15, 9, 20, 12, 820778, tzinfo=<UTC>)

Check Builder foreign key, which indicated which builder 'is processing'
the job in question:

The 'builder' attribute provides the IBuilder instance of the builder
currently processing this job:

    >>> bq.builder.name
    u'bob'

'logtail' is a text field which stores up to 1 Kbyte of the last
logged actions during the build process, it is updated according the
frequency set in buildd-slavescanner, normally 10 seconds, see
buildd-slavescanner.txt for further information:

    >>> bq.logtail
    u'Dummy sampledata entry, not processing'

'lastscore' is the heuristic index generated by queuebuilder to order
the dispatching process, see more information in
buildd-queuebuilder.txt:

    >>> bq.lastscore
    1

'manual' is a boolean, which is meant to suppress the auto-scoring
procedure, i.e., when it is set the job do not get re-scored, it is
used to manually reorder jobs via the web UI:

    >>> bq.manual
    False

BuildQueue provides the name for the logfile resulting from the build:

    >>> bq.getLogFileName()
    u'buildlog_ubuntu-hoary-i386.mozilla-firefox_0.9_BUILDING.txt'

BuildQueue provides a method to handle "manual scoring" procedure
properly, in order to retain the manually set score we need to
set 'manual' attribute atomically. For this we use manualScore:

    >>> pending_bq = getUtility(IBuildQueueSet).get(2)

    >>> pending_bq.manual, pending_bq.lastscore
    (False, 10)
    >>> pending_bq.manualScore(1000)

    >>> pending_bq.manual, pending_bq.lastscore
    (True, 1000)


The BuildQueue item is responsible for providing the required build behavior
for the item.

    >>> from zope.security.proxy import isinstance
    >>> from lp.soyuz.model.binarypackagebuildbehavior import (
    ...     BinaryPackageBuildBehavior)
    >>> isinstance(bq.required_build_behavior, BinaryPackageBuildBehavior)
    True


== Dispatching and Reseting jobs ==

The sampledata contains an active job, being built by the 'bob'
builder.

    >>> from lp.buildmaster.interfaces.builder import IBuilderSet

    >>> bob = getUtility(IBuilderSet)['bob']
    >>> job = bob.currentjob

    >>> print job.builder.name
    bob
    >>> job.date_started is not None
    True
    >>> print job.logtail
    Dummy sampledata entry, not processing
    >>> build = getUtility(IBinaryPackageBuildSet).getByQueueEntry(job)
    >>> print build.status.name
    BUILDING
    >>> print job.lastscore
    1

IBuildQueue.reset() will reset the job, enabling it to be dispatched
to another build. The score value of the job is preserved.

    >>> job.reset()

    >>> print bob.currentjob
    None

    >>> print job.builder
    None
    >>> print job.date_started
    None
    >>> print job.logtail
    None
    >>> print build.status.name
    NEEDSBUILD
    >>> print job.lastscore
    1

On the flip-side, IBuildQueue.markAsBuilding() will assign the job to
the specified builder.

    >>> job.markAsBuilding(bob)

    >>> bob.currentjob == job
    True

    >>> print job.builder.name
    bob
    >>> job.date_started is not None
    True
    >>> print build.status.name
    BUILDING


== BuildQueueSet utility ==

Now perform the tests for the BuildQueue zope utility, BuildQueueSet.
Check if the instance returned as utility corresponds to its
respective interface:

    >>> bqset = getUtility(IBuildQueueSet)
    >>> verifyObject(IBuildQueueSet, bqset)
    True

IBuildQueueSet utility is iterable:

    >>> for bq in bqset:
    ...     bq.id
    1
    2

Also provides a getter:

    >>> bqset[1].id
    1

a 'get' method:

    >>> bqset.get(2).id
    2

and a full counter:

    >>> bqset.count()
    2

IBuildQueueSet provides a method to retrieve the active job:

    >>> for bq in bqset.getActiveBuildJobs():
    ...     bq.id, bq.builder.name
    (1, u'bob')

Another method to fetch the BuildQueue being processed for a builder,
there should be only one since the builders are sigle-task
applications:

    >>> current_job = bqset.getByBuilder(bob)
    >>> print current_job.builder.name
    bob

BuildQueueSet provides a method used in the builddmaster context to
calculate retrieve every instance for a set of distroarchseriess
corresponding to a build record in NEEDSBUILD state.

Use all architectures available in ubuntu/hoary, i.e, i386 and hppa:

    >>> from lp.registry.interfaces.distribution import IDistributionSet
    >>> ubuntu = getUtility(IDistributionSet)['ubuntu']
    >>> hoary = ubuntu['hoary']
    >>> archseries = [arch for arch in hoary.architectures]

Create a new Builds and corresponding BuildQueues in ubuntu/hoary/hppa
for testing if we will also be able to deal with more than one architecture.

    >>> from lp.registry.interfaces.pocket import (
    ...     PackagePublishingPocket)
    >>> alsa_hoary = hoary.getSourcePackage('alsa-utils')
    >>> alsa_spr = alsa_hoary['1.0.9a-4'].sourcepackagerelease
    >>> print alsa_spr.title
    alsa-utils - 1.0.9a-4

    >>> alsa_build = alsa_spr.createBuild(
    ...     hoary['hppa'], PackagePublishingPocket.RELEASE,
    ...     hoary.main_archive)
    >>> alsa_build.queueBuild().lastscore = 500

    >>> pmount_hoary = hoary.getSourcePackage('pmount')
    >>> pmount_spr = pmount_hoary.currentrelease.sourcepackagerelease
    >>> print pmount_spr.title
    pmount - 0.1-2

    >>> pmount_build = pmount_spr.createBuild(
    ...     hoary['hppa'], PackagePublishingPocket.RELEASE,
    ...     hoary.main_archive)
    >>> pmount_build.queueBuild().lastscore = 1500

Materialize the changes in database:

    >>> from canonical.database.sqlbase import flush_database_updates
    >>> flush_database_updates()

Note that the results are ordered by descending BuildQueue.lastscore
and restricted

    >>> for bq in getUtility(IBinaryPackageBuildSet).calculateCandidates(archseries):
    ...     build = getUtility(IBinaryPackageBuildSet).getByQueueEntry(bq)
    ...     print "%s (%s, %d)" % (build.title, bq.lastscore, bq.id)
    hppa build of pmount 0.1-2 in ubuntu hoary RELEASE (1500, 4)
    i386 build of alsa-utils 1.0.9a-4ubuntu1 in ubuntu hoary RELEASE (1000, 2)
    hppa build of alsa-utils 1.0.9a-4 in ubuntu hoary RELEASE (500, 3)

Language packs will have a score of zero so they are processed
after other packages in the queue. First we need to create a fake
language pack build.

    >>> from zope.security.proxy import removeSecurityProxy
    >>> from lp.soyuz.interfaces.section import ISectionSet
    >>> section_set = getUtility(ISectionSet)
    >>> [transl_section] = [s for s in section_set
    ...     if s.name == u'translations']
    >>> na_hoary = hoary.getSourcePackage('netapplet')
    >>> na_spr = removeSecurityProxy(
    ...     na_hoary.currentrelease.sourcepackagerelease)
    >>> na_spr.section = transl_section

The build score for the fake language pack is zero.

    >>> na_build = na_spr.createBuild(
    ...     hoary['i386'], PackagePublishingPocket.RELEASE,
    ...     hoary.main_archive)
    >>> na_build.queueBuild().lastscore
    0

The fake language pack will be queued behind the other source packages
as intended.

    >>> for bq in getUtility(IBinaryPackageBuildSet).calculateCandidates(archseries):
    ...     build = getUtility(IBinaryPackageBuildSet).getByQueueEntry(bq)
    ...     print "%s (%s, %d)" % (build.title, bq.lastscore, bq.id)
    hppa build of pmount 0.1-2 in ubuntu hoary RELEASE (1500, 4)
    i386 build of alsa-utils 1.0.9a-4ubuntu1 in ubuntu hoary RELEASE (1000, 2)
    hppa build of alsa-utils 1.0.9a-4 in ubuntu hoary RELEASE (500, 3)
    i386 build of netapplet 1.0-1 in ubuntu hoary RELEASE (0, 5)


Restricting the domain to only hoary/hppa:

    >>> archseries = [hoary['hppa']]
    >>> for bq in getUtility(IBinaryPackageBuildSet).calculateCandidates(archseries):
    ...     build = getUtility(IBinaryPackageBuildSet).getByQueueEntry(bq)
    ...     print "%s (%s, %d)" % (build.title, bq.lastscore, bq.id)
    hppa build of pmount 0.1-2 in ubuntu hoary RELEASE (1500, 4)
    hppa build of alsa-utils 1.0.9a-4 in ubuntu hoary RELEASE (500, 3)

This method asserts a valid 'archseries' argument, None or an empty
list will result in a AssertionFailure.

    >>> getUtility(IBinaryPackageBuildSet).calculateCandidates(None).count()
    Traceback (most recent call last):
    ...
    AssertionError: Given 'archseries' cannot be None/empty.

    >>> getUtility(IBinaryPackageBuildSet).calculateCandidates([]).count()
    Traceback (most recent call last):
    ...
    AssertionError: Given 'archseries' cannot be None/empty.