~launchpad-pqm/launchpad/devel

= BuildQueue =

BuildQueue class represents the run-time-records of builds being

processed.

See buildd-queuebuilder.txt for more information about the BuildQueue

initialization.

BuildQueue should provide all the information needed for dispatching

and collecting build results.

As soon as a build job is processed succesfully (dispatched &

collected) the BuildQueue record representing it is removed.

    >>> from lp.services.webapp.testing import verifyObject

    >>> from lp.services.propertycache import get_property_cache

    >>> from lp.buildmaster.interfaces.buildqueue import (

    ...     IBuildQueue, IBuildQueueSet)

Get an instance of Builder from the current sampledata using the

IBuildQueueSet zope utility. The zope utility itself will be

documented later on this document, we are only using it to retrive an

IBuildQueue record with the security proxies in place:

    >>> bq = getUtility(IBuildQueueSet).get(1)

Instances returned by IBuildQueueSet provides the IBuildQueue interface:

    >>> verifyObject(IBuildQueue, bq)

    True

The IBuild record related to this job is provided by the 'build'

attribute:

    >>> from lp.soyuz.interfaces.binarypackagebuild import IBinaryPackageBuildSet

    >>> build = getUtility(IBinaryPackageBuildSet).getByQueueEntry(bq)

    >>> build.id

    8

    >>> build.status.name

    'BUILDING'

The static timestamps, representing when the record was initialized

(inserted) and when the job was dispatched are provided as datetime

instances:

    >>> bq.job.date_created

    datetime.datetime(2005, 6, 15, 9, 14, 12, 820778, tzinfo=<UTC>)

    >>> bq.date_started

    datetime.datetime(2005, 6, 15, 9, 20, 12, 820778, tzinfo=<UTC>)

Check Builder foreign key, which indicated which builder 'is processing'

the job in question:

The 'builder' attribute provides the IBuilder instance of the builder

currently processing this job:

    >>> bq.builder.name

    u'bob'

'logtail' is a text field which stores up to 1 Kbyte of the last

logged actions during the build process, it is updated according the

frequency set in buildd-slavescanner, normally 10 seconds, see

buildd-slavescanner.txt for further information:

    >>> bq.logtail

    u'Dummy sampledata entry, not processing'

'lastscore' is the heuristic index generated by queuebuilder to order

the dispatching process, see more information in

buildd-queuebuilder.txt:

    >>> bq.lastscore

    1

'manual' is a boolean, which is meant to suppress the auto-scoring

procedure, i.e., when it is set the job do not get re-scored, it is

used to manually reorder jobs via the web UI:

    >>> bq.manual

    False

BuildQueue provides the name for the logfile resulting from the build:

    >>> bq.getLogFileName()

    u'buildlog_ubuntu-hoary-i386.mozilla-firefox_0.9_BUILDING.txt'

BuildQueue provides a method to handle "manual scoring" procedure

properly, in order to retain the manually set score we need to

set 'manual' attribute atomically. For this we use manualScore:

    >>> pending_bq = getUtility(IBuildQueueSet).get(2)

    >>> pending_bq.manual, pending_bq.lastscore

    (False, 10)

    >>> pending_bq.manualScore(1000)

    >>> pending_bq.manual, pending_bq.lastscore

    (True, 1000)

The BuildQueue item is responsible for providing the required build behavior

for the item.

    >>> from zope.security.proxy import isinstance

    >>> from lp.soyuz.model.binarypackagebuildbehavior import (

    ...     BinaryPackageBuildBehavior)

    >>> isinstance(bq.required_build_behavior, BinaryPackageBuildBehavior)

    True

== Dispatching and Reseting jobs ==

The sampledata contains an active job, being built by the 'bob'

builder.

    >>> from lp.buildmaster.interfaces.builder import IBuilderSet

    >>> bob = getUtility(IBuilderSet)['bob']

    >>> job = bob.currentjob

    >>> print job.builder.name

    bob

    >>> job.date_started is not None

    True

    >>> print job.logtail

    Dummy sampledata entry, not processing

    >>> build = getUtility(IBinaryPackageBuildSet).getByQueueEntry(job)

    >>> print build.status.name

    BUILDING

    >>> print job.lastscore

    1

IBuildQueue.reset() will reset the job, enabling it to be dispatched

to another build. The score value of the job is preserved.

    >>> job.reset()

    >>> del get_property_cache(bob).currentjob

    >>> print bob.currentjob

    None

    >>> print job.builder

    None

    >>> print job.date_started

    None

    >>> print job.logtail

    None

    >>> print build.status.name

    NEEDSBUILD

    >>> print job.lastscore

    1

On the flip-side, IBuildQueue.markAsBuilding() will assign the job to

the specified builder.

    >>> job.markAsBuilding(bob)

    >>> del get_property_cache(bob).currentjob

    >>> bob.currentjob == job

    True

    >>> print job.builder.name

    bob

    >>> job.date_started is not None

    True

    >>> print build.status.name

    BUILDING

== BuildQueueSet utility ==

Now perform the tests for the BuildQueue zope utility, BuildQueueSet.

Check if the instance returned as utility corresponds to its

respective interface:

    >>> bqset = getUtility(IBuildQueueSet)

    >>> verifyObject(IBuildQueueSet, bqset)

    True

IBuildQueueSet utility is iterable:

    >>> for bq in bqset:

    ...     bq.id

    1

    2

Also provides a getter:

    >>> bqset[1].id

    1

a 'get' method:

    >>> bqset.get(2).id

    2

and a full counter:

    >>> bqset.count()

    2

IBuildQueueSet provides a method to retrieve the active job:

    >>> for bq in bqset.getActiveBuildJobs():

    ...     bq.id, bq.builder.name

    (1, u'bob')

Another method to fetch the BuildQueue being processed for a builder,

there should be only one since the builders are sigle-task

applications:

    >>> current_job = bqset.getByBuilder(bob)

    >>> print current_job.builder.name

    bob

BuildQueueSet provides a method used in the builddmaster context to

calculate retrieve every instance for a set of distroarchseriess

corresponding to a build record in NEEDSBUILD state.

Use all architectures available in ubuntu/hoary, i.e, i386 and hppa:

    >>> from lp.registry.interfaces.distribution import IDistributionSet

    >>> ubuntu = getUtility(IDistributionSet)['ubuntu']

    >>> hoary = ubuntu['hoary']

    >>> archseries = [arch for arch in hoary.architectures]

Create a new Builds and corresponding BuildQueues in ubuntu/hoary/hppa

for testing if we will also be able to deal with more than one architecture.

    >>> from lp.registry.interfaces.pocket import (

    ...     PackagePublishingPocket)

    >>> alsa_hoary = hoary.getSourcePackage('alsa-utils')

    >>> alsa_spr = alsa_hoary['1.0.9a-4'].sourcepackagerelease

    >>> print alsa_spr.title

    alsa-utils - 1.0.9a-4

    >>> alsa_build = alsa_spr.createBuild(

    ...     hoary['hppa'], PackagePublishingPocket.RELEASE,

    ...     hoary.main_archive)

    >>> alsa_build.queueBuild().lastscore = 500

    >>> pmount_hoary = hoary.getSourcePackage('pmount')

    >>> pmount_spr = pmount_hoary.currentrelease.sourcepackagerelease

    >>> print pmount_spr.title

    pmount - 0.1-2

    >>> pmount_build = pmount_spr.createBuild(

    ...     hoary['hppa'], PackagePublishingPocket.RELEASE,

    ...     hoary.main_archive)

    >>> pmount_build.queueBuild().lastscore = 1500

Materialize the changes in database:

    >>> from lp.services.database.sqlbase import flush_database_updates

    >>> flush_database_updates()

Note that the results are ordered by descending BuildQueue.lastscore

and restricted

    >>> for bq in getUtility(IBinaryPackageBuildSet).calculateCandidates(archseries):

    ...     build = getUtility(IBinaryPackageBuildSet).getByQueueEntry(bq)

    ...     print "%s (%s, %d)" % (build.title, bq.lastscore, bq.id)

    hppa build of pmount 0.1-2 in ubuntu hoary RELEASE (1500, 4)

    i386 build of alsa-utils 1.0.9a-4ubuntu1 in ubuntu hoary RELEASE (1000, 2)

    hppa build of alsa-utils 1.0.9a-4 in ubuntu hoary RELEASE (500, 3)

Language packs will have a score of zero so they are processed

after other packages in the queue. First we need to create a fake

language pack build.

    >>> from zope.security.proxy import removeSecurityProxy

    >>> from lp.soyuz.interfaces.section import ISectionSet

    >>> section_set = getUtility(ISectionSet)

    >>> [transl_section] = [s for s in section_set

    ...     if s.name == u'translations']

    >>> na_hoary = hoary.getSourcePackage('netapplet')

    >>> na_spr = removeSecurityProxy(

    ...     na_hoary.currentrelease.sourcepackagerelease)

    >>> na_spr.section = transl_section

The build score for the fake language pack is zero.

    >>> na_build = na_spr.createBuild(

    ...     hoary['i386'], PackagePublishingPocket.RELEASE,

    ...     hoary.main_archive)

    >>> na_build.queueBuild().lastscore

    0

The fake language pack will be queued behind the other source packages

as intended.

    >>> for bq in getUtility(IBinaryPackageBuildSet).calculateCandidates(archseries):

    ...     build = getUtility(IBinaryPackageBuildSet).getByQueueEntry(bq)

    ...     print "%s (%s, %d)" % (build.title, bq.lastscore, bq.id)

    hppa build of pmount 0.1-2 in ubuntu hoary RELEASE (1500, 4)

    i386 build of alsa-utils 1.0.9a-4ubuntu1 in ubuntu hoary RELEASE (1000, 2)

    hppa build of alsa-utils 1.0.9a-4 in ubuntu hoary RELEASE (500, 3)

    i386 build of netapplet 1.0-1 in ubuntu hoary RELEASE (0, 5)

Restricting the domain to only hoary/hppa:

    >>> archseries = [hoary['hppa']]

    >>> for bq in getUtility(IBinaryPackageBuildSet).calculateCandidates(archseries):

    ...     build = getUtility(IBinaryPackageBuildSet).getByQueueEntry(bq)

    ...     print "%s (%s, %d)" % (build.title, bq.lastscore, bq.id)

    hppa build of pmount 0.1-2 in ubuntu hoary RELEASE (1500, 4)

    hppa build of alsa-utils 1.0.9a-4 in ubuntu hoary RELEASE (500, 3)

This method asserts a valid 'archseries' argument, None or an empty

list will result in a AssertionFailure.

    >>> getUtility(IBinaryPackageBuildSet).calculateCandidates(None).count()

    Traceback (most recent call last):

    ...

    AssertionError: Given 'archseries' cannot be None/empty.

    >>> getUtility(IBinaryPackageBuildSet).calculateCandidates([]).count()

    Traceback (most recent call last):

    ...

    AssertionError: Given 'archseries' cannot be None/empty.