~launchpad-pqm/launchpad/devel

= Bugtask Expiration =

Old unattended Incomplete bugtasks clutter the search results of

Launchpad Bugs making the bug staff's job difficult. A script is run

daily to locate unattended Incomplete bugtasks that have not been

updated in 2 months, and sets their status to Expired. Only bugtasks

for projects that use Launchpad to track bugs and have

enable_bug_expiration set to True will be expired; this rule does not

apply to Bugs imported from upstream bug trackers. The preconditions

are:

1. The bugtask belongs to a project with enable_bug_expiration is True.

2. The bugtask has the status Incomplete.

3. The last update of the bug is older than 60 days.

4. The bug is not a duplicate.

5. The bug does not have any other valid bugtasks.

6. The bugtask is not assigned to anyone.

7. The bugtask does not have a milestone.

Bugtasks cannot transition to Expired automatically unless they meet

all the rules stated above.

== findExpirableBugTasks() Part 1 ==

BugTaskSet provides findExpirableBugTasks() to find bugtasks that

qualify for expiration. The bugtasks must must meet all the

preconditions stated in this tests introduction.

findExpirableBugTasks() requires a parameter for the minimum days old

(min_days_old) that the bugtask has been in the unattended Incomplete

status. It also requires specifying the user that is doing the search.

    >>> from lp.bugs.interfaces.bugtask import (

    ...     BugTaskStatus,

    ...     IBugTaskSet,

    ...     )

    >>> from storm.store import Store

    >>> bugtaskset = getUtility(IBugTaskSet)

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks()

    Traceback (most recent call last):

    ...

    TypeError: findExpirableBugTasks() takes at least 3 arguments (1 given)

Looking back 9,999 days, findExpirableBugTasks() reports that there are

no expirable bugtasks in the sampledata.

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(9999, None)

    >>> expirable_bugtasks.count()

    0

== Setup ==

Let's make some bugtasks that qualify for expiration. A Jokosher

bugtask and a conjoined pair of ubuntu_hoary and ubuntu bugtasks

will suffice.

IBug specifies two properties related to bug expiration. can_expire

tells you whether one or more of the bug's bug tasks may be expired, if

the bug doesn't get any more activity. permits_expiration on the other

hand, mainly tells you whether no bug tasks are in a state that they may

expire. If permits_expiration is True, it could very well be that no bug

tasks will be expired.  permits_expiration mainly exists to have

can_expire avoid an expensive db, in the case where we can easily tell

that no bug tasks can be expired.

    >>> from lp.registry.interfaces.distribution import IDistributionSet

    >>> from lp.registry.interfaces.person import IPersonSet

    >>> from lp.registry.interfaces.product import IProductSet

    >>> ubuntu = getUtility(IDistributionSet).getByName('ubuntu')

    >>> jokosher = getUtility(IProductSet).getByName('jokosher')

    >>> sample_person = getUtility(IPersonSet).getByEmail(

    ...     'test@canonical.com')

    # A expirable bugtask. It will be expired because its conjoined

    # master can be expired.

    >>> from lp.bugs.tests.bug import create_old_bug

    >>> ubuntu_bugtask = create_old_bug('expirable_distro', 351, ubuntu)

    >>> ubuntu_bugtask.bug.permits_expiration

    True

    >>> ubuntu_bugtask.bug.can_expire

    True

    # An expirable bugtask, a distroseries. The ubuntu bugtask is its

    # conjoined slave.

    >>> from lp.bugs.tests.bug import sync_bugtasks

    >>> hoary_bugtask = bugtaskset.createTask(

    ...     bug=ubuntu_bugtask.bug, owner=sample_person,

    ...     distroseries=ubuntu.currentseries)

    >>> ubuntu_bugtask.conjoined_master == hoary_bugtask

    True

    >>> ubuntu_bugtask.bug.permits_expiration

    True

    >>> sync_bugtasks([ubuntu_bugtask, hoary_bugtask])

    >>> ubuntu_bugtask.bug.can_expire

    True

    # A bugtask for a product that is expirable.

    >>> jokosher_bugtask = create_old_bug('jokosher', 61, jokosher)

    >>> jokosher_bugtask.bug.permits_expiration

    True

    >>> jokosher_bugtask.bug.can_expire

    True

A bugtask for a product with a bug watch. Note that this bugtask

has otherwise the same parameters as jokosher_bugtask. The

bugwatch prevents expiration, hence this bugtask will not appear

in the listings of expirable bugtasks below.

    >>> from lp.bugs.interfaces.bugtracker import IBugTrackerSet

    >>> mozilla_bugtracker = getUtility(IBugTrackerSet)['mozilla.org']

    >>> jokosher_bugtask_watched = create_old_bug('jokosher watched',

    ...     61, jokosher, external_bugtracker=mozilla_bugtracker)

    >>> jokosher_bugtask_watched.bug.can_expire

    False

Let's also make some bugs that almost qualify for expiration.

    # A bugtask whose status is not Incomplete is not expirable.

    # This one's status is New.

    >>> thunderbird = getUtility(IProductSet).getByName('thunderbird')

    >>> new_bugtask = bugtaskset.createTask(

    ...     bug=ubuntu_bugtask.bug, owner=sample_person,

    ...     product=thunderbird)

    >>> new_bugtask.status.title

    'New'

    >>> new_bugtask.bug.permits_expiration

    False

    >>> new_bugtask.bug.can_expire

    False

    # A bugtask that is not expirable because it is assigned.

    >>> assigned_bugtask = create_old_bug(

    ...     'assigned', 61, ubuntu, assignee=sample_person)

    >>> assigned_bugtask.bug.permits_expiration

    True

    >>> assigned_bugtask.bug.can_expire

    False

    # A bug with two Ubuntu tasks, one assigned Incomplete, and one

    # Invalid task, is not expirable.

    >>> ubuntu_alsa = ubuntu.getSourcePackage('alsa-utils')

    >>> another_assigned_bugtask = create_old_bug(

    ...     'assigned', 61, ubuntu, assignee=sample_person)

    >>> another_assigned_bugtask.transitionToTarget(ubuntu_alsa)

    >>> ubuntu_evolution = ubuntu.getSourcePackage('evolution')

    >>> invalid_bugtask = bugtaskset.createTask(

    ...     bug=another_assigned_bugtask.bug, owner=sample_person,

    ...     distribution=ubuntu,

    ...     sourcepackagename=ubuntu_evolution.sourcepackagename,

    ...     status=BugTaskStatus.INVALID)

    >>> another_assigned_bugtask.bug.permits_expiration

    True

    >>> another_assigned_bugtask.bug.can_expire

    False

    # A bugtask that is not expirable because its status is CONFIRMED.

    >>> confirmed_bugtask = create_old_bug(

    ...     'confirmed', 61, ubuntu, status=BugTaskStatus.CONFIRMED)

    >>> confirmed_bugtask.bug.permits_expiration

    False

    >>> confirmed_bugtask.bug.can_expire

    False

    # A bugtask that is not expirable because it is a duplicate.

    >>> duplicate_bugtask = create_old_bug(

    ...     'duplicate', 61, ubuntu, duplicateof=confirmed_bugtask.bug)

    >>> duplicate_bugtask.bug.permits_expiration

    True

    >>> duplicate_bugtask.bug.can_expire

    False

    # A bugtask that is not expirable because it does not use

    # Launchpad Bugs.

    >>> external_bugtask = create_old_bug('external', 61, thunderbird)

    >>> external_bugtask.bug.permits_expiration

    False

    >>> thunderbird.enable_bug_expiration

    False

    >>> external_bugtask.bug.can_expire

    False

    # A bugtask that is not expirable because it has a milestone.

    >>> milestone = ubuntu.currentseries.newMilestone("0.1")

    >>> Store.of(milestone).flush()

    >>> milestone_bugtask = create_old_bug(

    ...     'milestone', 61, ubuntu,

    ...     milestone=milestone)

    >>> milestone_bugtask.bug.permits_expiration

    True

    >>> milestone_bugtask.bug.can_expire

    False

    # Create a bugtask that is not old enough to expire

    >>> recent_bugtask = create_old_bug('recent', 31, ubuntu)

    >>> recent_bugtask.bug.permits_expiration

    True

    >>> recent_bugtask.bug.can_expire

    False

    # A bugtask that is not expirable; while the product uses Launchpad to

    # track bugs, enable_bug_expiration is set to False

    >>> firefox = getUtility(IProductSet).getByName('firefox')

    >>> no_expiration_bugtask = create_old_bug('no_expire', 61, firefox)

    >>> no_expiration_bugtask.bug.permits_expiration

    False

    >>> firefox.enable_bug_expiration

    False

    >>> no_expiration_bugtask.bug.can_expire

    False

The ubuntu, hoary, and jokosher bugs are the only ones that can be

expired. The other bugs do not meet one of the preconditions.

    >>> bugtasks = [ubuntu_bugtask, hoary_bugtask, jokosher_bugtask,

    ...     jokosher_bugtask_watched, new_bugtask, assigned_bugtask,

    ...     confirmed_bugtask, duplicate_bugtask, external_bugtask,

    ...     milestone_bugtask, recent_bugtask, no_expiration_bugtask]

    >>> from lp.bugs.tests.bug import summarize_bugtasks

    >>> summarize_bugtasks(bugtasks)

    ROLE             EXPIRE  AGE  STATUS      ASSIGNED  DUP    MILE   REPLIES

    ubuntu           False   351  Incomplete  False     False  False  False

    hoary            False   351  Incomplete  False     False  False  False

    jokosher         True     61  Incomplete  False     False  False  False

    jokosher watched False    61  Incomplete  False     False  False  False

    thunderbird      False   351  New         False     False  False  False

    assigned         False    61  Incomplete  True      False  False  False

    confirmed        False    61  Confirmed   False     False  False  False

    duplicate        False    61  Incomplete  False     True   False  False

    external         False    61  Incomplete  False     False  False  False

    milestone        False    61  Incomplete  False     False  True   False

    recent           False    31  Incomplete  False     False  False  False

    no_expire        False    61  Incomplete  False     False  False  False

== isExpirable() ==

In addition to can_expire bugs have an isExpirable method to which a custom

number of days, days_old, can be passed.  days_old is then used with

findExpirableBugTasks.  This allows projects to create their own janitor using

a different period for bug expiration.

    # Check to ensure that isExpirable() works without days_old, then set the

    # bug to Invalid so it doesn't affect the rest of the doctest

    >>> from lp.bugs.tests.bug import create_old_bug

    >>> very_old_bugtask = create_old_bug('expirable_distro', 351, ubuntu)

    >>> very_old_bugtask.bug.isExpirable()

    True

    >>> very_old_bugtask.transitionToStatus(

    ...     BugTaskStatus.INVALID, sample_person)

    # Pass isExpirable() a days_old parameter, then set the bug to Invalid so it

    # doesn't affect the rest of the doctest

    >>> from lp.bugs.tests.bug import create_old_bug

    >>> not_so_old_bugtask = create_old_bug('expirable_distro', 31, ubuntu)

    >>> not_so_old_bugtask.bug.isExpirable(days_old=14)

    True

    >>> not_so_old_bugtask.transitionToStatus(

    ...     BugTaskStatus.INVALID, sample_person)

== findExpirableBugTasks() Part 2 ==

The value of the min_days_old controls the bugtasks that are

returned. The oldest bug in this test is 351 days old, the youngest is

31 days old. There are no bugs older than 351 days.

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(351, None)

    >>> expirable_bugtasks.count()

    0

While there are bugtasks older than 350 days in the data, the hoary

bugtask does not display because its bug has other bugtasks that are

valid.

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(350, None)

    >>> expirable_bugtasks.count()

    0

    >>> hoary_bugtask.bug.can_expire

    False

    >>> summarize_bugtasks(hoary_bugtask.bug.bugtasks)

    ROLE         EXPIRE  AGE  STATUS      ASSIGNED  DUP    MILE   REPLIES

    ubuntu       False   351  Incomplete  False     False  False  False

    hoary        False   351  Incomplete  False     False  False  False

    thunderbird  False   351  New         False     False  False  False

If the valid bugtask becomes Invalid or Won't Fix, the hoary bugtask

will be expirable.

    >>> new_bugtask.transitionToStatus(BugTaskStatus.WONTFIX, sample_person)

    >>> sync_bugtasks(new_bugtask)

    >>> hoary_bugtask.bug.can_expire

    True

    >>> summarize_bugtasks(hoary_bugtask.bug.bugtasks)

    ROLE         EXPIRE  AGE  STATUS      ASSIGNED  DUP    MILE   REPLIES

    ubuntu       True    351  Incomplete  False     False  False  False

    hoary        True    351  Incomplete  False     False  False  False

    thunderbird  False   351  Won't Fix   False     False  False  False

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(350, None)

    >>> summarize_bugtasks(expirable_bugtasks)

    ROLE         EXPIRE  AGE  STATUS      ASSIGNED  DUP    MILE   REPLIES

    ubuntu       True    351  Incomplete  False     False  False  False

    hoary        True    351  Incomplete  False     False  False  False

The ubuntu bugtask is never returned; it is a conjoined slave to the

hoary bugtask. Slave bugtasks cannot be directly expired, so they are

not returned by findExpirableBugTasks().

    >>> ubuntu_bugtask.status.title

    'Incomplete'

    >>> ubuntu_bugtask.conjoined_master == hoary_bugtask

    True

Reducing the age to 60 days old, both hoary and jokosher bugtasks

are returned.

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(60, None)

    >>> summarize_bugtasks(expirable_bugtasks)

    ROLE         EXPIRE  AGE  STATUS      ASSIGNED  DUP    MILE   REPLIES

    ubuntu       True    351  Incomplete  False     False  False  False

    hoary        True    351  Incomplete  False     False  False  False

    jokosher     True     61  Incomplete  False     False  False  False

When a bug is passed as an argument to findExpirableBugTasks(), it

returns that bug's expirable BugTasks, or an empty list. Passing the bug

that has the hoary and ubuntu bugtasks with 0 min_days_old returns just

the hoary bugtask.

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(

    ...     0, None, bug=hoary_bugtask.bug)

    >>> summarize_bugtasks(expirable_bugtasks)

    ROLE         EXPIRE  AGE  STATUS      ASSIGNED  DUP    MILE   REPLIES

    ubuntu       True    351  Incomplete  False     False  False  False

    hoary        True    351  Incomplete  False     False  False  False

When a BugTarget is passed as an argument to findExpirableBugTasks(), it

returns all the target's expirable bugtasks, or an empty list. If the

target's pillar has not enabled bug expiration, None is always returned.

Passing ubuntu with 0 min_days_old shows that the distribution has two

bugtasks that can expire if they are not confirmed.

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(

    ...     0, None, target=ubuntu)

    >>> summarize_bugtasks(expirable_bugtasks)

    ROLE         EXPIRE  AGE  STATUS      ASSIGNED  DUP    MILE   REPLIES

    ubuntu       True    351  Incomplete  False     False  False  False

    hoary        True    351  Incomplete  False     False  False  False

    recent       False    31  Incomplete  False     False  False  False

findExpirableBugTasks also accepts a limit argument, which allows for limiting

the number of bugtasks returned.

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(

    ...     0, None, target=ubuntu, limit=2)

    >>> summarize_bugtasks(expirable_bugtasks)

    ROLE         EXPIRE  AGE  STATUS      ASSIGNED  DUP    MILE   REPLIES

    ubuntu       True    351  Incomplete  False     False  False  False

    hoary        True    351  Incomplete  False     False  False  False

Thunderbird has not enabled bug expiration. Even when the min_days_old

is set to 0, no bugtasks are replaced.

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(

    ...     0, None, target=thunderbird)

    >>> summarize_bugtasks(expirable_bugtasks)

    ROLE         EXPIRE  AGE  STATUS      ASSIGNED  DUP    MILE   REPLIES

== Privacy ==

The user parameter indicates which user is performing the search. Only

bugs that the user has permission to view are returned. A value of None

indicates the anonymous user.

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(

    ...     0, user=None, target=ubuntu)

    >>> visible_bugs = set(bugtask.bug for bugtask in expirable_bugtasks)

    >>> print sorted(bug.title for bug in visible_bugs)

    [u'expirable_distro', u'recent']

If one of the bugs is set to private, anonymous users can no longer see

it as being marked for expiration.

    >>> private_bug = ubuntu_bugtask.bug

    >>> private_bug.title

    u'expirable_distro'

    >>> private_bug.setPrivate(True, sample_person)

    True

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(

    ...     0, user=None, target=ubuntu)

    >>> visible_bugs = set(bugtask.bug for bugtask in expirable_bugtasks)

    >>> print sorted(bug.title for bug in visible_bugs)

    [u'recent']

No Privileges Person can't see the bug either...

    >>> no_priv = getUtility(IPersonSet).getByName('no-priv')

    >>> private_bug.unsubscribe(no_priv, no_priv)

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(

    ...     0, user=no_priv, target=ubuntu)

    >>> visible_bugs = set(bugtask.bug for bugtask in expirable_bugtasks)

    >>> print sorted(bug.title for bug in visible_bugs)

    [u'recent']

... unless he's subscribed to the bug.

    >>> private_bug.subscribe(no_priv, sample_person)

    <lp.bugs.model.bugsubscription.BugSubscription ...>

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(

    ...     0, user=no_priv, target=ubuntu)

    >>> visible_bugs = set(bugtask.bug for bugtask in expirable_bugtasks)

    >>> print sorted(bug.title for bug in visible_bugs)

    [u'expirable_distro', u'recent']

The Janitor needs to be able to access all bugs, even private ones, in

order to be able to expire them. If the Janitor is passed as the user,

even the private bugs are returned.

    >>> from lp.app.interfaces.launchpad import ILaunchpadCelebrities

    >>> janitor = getUtility(ILaunchpadCelebrities).janitor

    >>> private_bug.isSubscribed(janitor)

    False

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(

    ...     0, user=janitor, target=ubuntu)

    >>> visible_bugs = set(bugtask.bug for bugtask in expirable_bugtasks)

    >>> print sorted(bug.title for bug in visible_bugs)

    [u'expirable_distro', u'recent']

    >>> private_bug.setPrivate(False, sample_person)

    True

== The default expiration age ==

The expiration age is set using the

config.malone.days_before_expiration configuration variable. It

defaults to 60 days. The period is measured from the date_incomplete

field. We expire bugtasks that are Incomplete and unattended for 60

days or more.

    >>> from canonical.config import config

    >>> old_age_days = config.malone.days_before_expiration

    >>> old_age_days

    60

== Running the script ==

There are no Expired Bugtasks in sampledata, from the tests above.

    >>> from lp.bugs.model.bugtask import BugTask

    >>> BugTask.selectBy(status=BugTaskStatus.EXPIRED).count()

    0

    >>> # We want to check the hoary bugtask messages later.

    >>> starting_bug_messages_count = (hoary_bugtask.bug.messages.count())

The script 'expire-bugtasks.py' writes its report to stdout. It makes

its database changes as the user configured in

config.malone.expiration_dbuser.

    >>> config.malone.expiration_dbuser

    'bugnotification'

    # Commit the current transaction because the script will run in

    # another transaction, and thus it won't see the changes done on

    # this test unless we commit.

    >>> commit()

    >>> import subprocess

    >>> process = subprocess.Popen(

    ...     'cronscripts/expire-bugtasks.py', shell=True,

    ...     stdin=subprocess.PIPE, stdout=subprocess.PIPE,

    ...     stderr=subprocess.PIPE)

    >>> (out, err) = process.communicate()

    >>> print err

    INFO    Creating lockfile: /var/lock/launchpad-expire-bugtasks.lock

    INFO    Expiring unattended, INCOMPLETE bugtasks older than

            60 days for projects that use Launchpad Bugs.

    INFO    Found 3 bugtasks to expire.

    INFO    Expired 2 bugtasks.

    INFO    Finished expiration run.

    <BLANKLINE>

    >>> print out

    <BLANKLINE>

    >>> process.returncode

    0

    >>> bugtasks = [BugTask.get(bugtask.id) for bugtask in bugtasks]

== After the script has run ==

There are three Expired bugtasks. Jokosher, hoary and ubuntu were

expired by the expiration process. Although ubuntu was never returned

by findExpirableBugTasks(), it was expired because its master (hoary)

was expired. The remaining bugtasks are unchanged.

    >>> summarize_bugtasks(bugtasks)

    ROLE             EXPIRE  AGE  STATUS      ASSIGNED  DUP    MILE   REPLIES

    ubuntu           False     0  Expired     False     False  False  False

    hoary            False     0  Expired     False     False  False  False

    jokosher         False     0  Expired     False     False  False  False

    jokosher watched False    61  Incomplete  False     False  False  False

    thunderbird      False     0  Won't Fix   False     False  False  False

    assigned         False    61  Incomplete  True      False  False  False

    confirmed        False    61  Confirmed   False     False  False  False

    duplicate        False    61  Incomplete  False     True   False  False

    external         False    61  Incomplete  False     False  False  False

    milestone        False    61  Incomplete  False     False  True   False

    recent           False    31  Incomplete  False     False  False  False

    no_expire        False    61  Incomplete  False     False  False  False

The bugtasks statusexplanation was updated to explain the change in

status.

    >>> hoary_bugtask = BugTask.get(hoary_bugtask.id)

    >>> print hoary_bugtask.statusexplanation

    [Expired for Ubuntu Hoary because there has been no activity for 60 days.]

The message explaining the reason for the expiration was posted by the

Launchpad Janitor celebrity. Only one message was created for when the

master and slave bugtasks were expired.

    >>> starting_bug_messages_count

    2

    >>> hoary_bugtask.bug.messages.count()

    3

    >>> message = hoary_bugtask.bug.messages[-1]

    >>> print message.owner.name

    janitor

    >>> print message.text_contents

    [Expired for Ubuntu Hoary because there has been no activity for 60 days.]

The bug's activity log was updated too with the status change.

    >>> activity = hoary_bugtask.bug.activity[-1]

    >>> print "%s  %s  %s  %s" % (

    ...     activity.person.displayname, activity.whatchanged,

    ...     activity.oldvalue, activity.newvalue)

    Launchpad Janitor  Ubuntu Hoary: status  Incomplete  Expired

== enable_bug_expiration ==

The bugtask no_expiration_bugtask has not been expired because it does

not participate in bug expiration. When uses_bug_expiration is set to

True for a project, old bugs will be expired the next time the bugs are

expired.

    >>> no_expiration_bugtask.pillar.enable_bug_expiration = True

    >>> from canonical.launchpad.ftests import sync

    >>> sync(no_expiration_bugtask.pillar)

    >>> no_expiration_bugtask.bug.permits_expiration

    True

    >>> no_expiration_bugtask.bug.can_expire

    True

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(60, None)

    >>> summarize_bugtasks(expirable_bugtasks)

    ROLE         EXPIRE  AGE  STATUS      ASSIGNED  DUP    MILE   REPLIES

    no_expire    True    61   Incomplete  False     False  False  False