= Bugtask Expiration =

Old unattended Incomplete bugtasks clutter the search results of
Launchpad Bugs making the bug staff's job difficult. A script is run
daily to locate unattended Incomplete bugtasks that have not been
updated in 2 months, and sets their status to Expired. Only bugtasks
for projects that use Launchpad to track bugs and have
enable_bug_expiration set to True will be expired; this rule does not
apply to Bugs imported from upstream bug trackers. The preconditions
are:

1. The bugtask belongs to a project with enable_bug_expiration is True.
2. The bugtask has the status Incomplete.
3. The last update of the bug is older than 60 days.
4. The bug is not a duplicate.
5. The bug does not have any other valid bugtasks.
6. The bugtask is not assigned to anyone.
7. The bugtask does not have a milestone.

Bugtasks cannot transition to Expired automatically unless they meet
all the rules stated above.


== findExpirableBugTasks() Part 1 ==

BugTaskSet provides findExpirableBugTasks() to find bugtasks that
qualify for expiration. The bugtasks must must meet all the
preconditions stated in this tests introduction.

findExpirableBugTasks() requires a parameter for the minimum days old
(min_days_old) that the bugtask has been in the unattended Incomplete
status. It also requires specifying the user that is doing the search.

    >>> from lp.bugs.interfaces.bugtask import (
    ...     BugTaskStatus,
    ...     IBugTaskSet,
    ...     )
    >>> from storm.store import Store
    >>> bugtaskset = getUtility(IBugTaskSet)

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks()
    Traceback (most recent call last):
    ...
    TypeError: findExpirableBugTasks() takes at least 3 arguments (1 given)

Looking back 9,999 days, findExpirableBugTasks() reports that there are
no expirable bugtasks in the sampledata.

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(9999, None)
    >>> expirable_bugtasks.count()
    0


== Setup ==

Let's make some bugtasks that qualify for expiration. A Jokosher
bugtask and a conjoined pair of ubuntu_hoary and ubuntu bugtasks
will suffice.

IBug specifies two properties related to bug expiration. can_expire
tells you whether one or more of the bug's bug tasks may be expired, if
the bug doesn't get any more activity. permits_expiration on the other
hand, mainly tells you whether no bug tasks are in a state that they may
expire. If permits_expiration is True, it could very well be that no bug
tasks will be expired.  permits_expiration mainly exists to have
can_expire avoid an expensive db, in the case where we can easily tell
that no bug tasks can be expired.

    >>> from lp.registry.interfaces.distribution import IDistributionSet
    >>> from lp.registry.interfaces.person import IPersonSet
    >>> from lp.registry.interfaces.product import IProductSet
    >>> ubuntu = getUtility(IDistributionSet).getByName('ubuntu')
    >>> jokosher = getUtility(IProductSet).getByName('jokosher')
    >>> sample_person = getUtility(IPersonSet).getByEmail(
    ...     'test@canonical.com')

    # A expirable bugtask. It will be expired because its conjoined
    # master can be expired.
    >>> from lp.bugs.tests.bug import create_old_bug
    >>> ubuntu_bugtask = create_old_bug('expirable_distro', 351, ubuntu)
    >>> ubuntu_bugtask.bug.permits_expiration
    True
    >>> ubuntu_bugtask.bug.can_expire
    True

    # An expirable bugtask, a distroseries. The ubuntu bugtask is its
    # conjoined slave.
    >>> hoary_bugtask = bugtaskset.createTask(
    ...     bug=ubuntu_bugtask.bug, owner=sample_person,
    ...     distroseries=ubuntu.currentseries)
    >>> ubuntu_bugtask.conjoined_master == hoary_bugtask
    True
    >>> ubuntu_bugtask.bug.permits_expiration
    True
    >>> ubuntu_bugtask.bug.can_expire
    True

    # A bugtask for a product that is expirable.
    >>> jokosher_bugtask = create_old_bug('jokosher', 61, jokosher)
    >>> jokosher_bugtask.bug.permits_expiration
    True
    >>> jokosher_bugtask.bug.can_expire
    True

A bugtask for a product with a bug watch. Note that this bugtask
has otherwise the same parameters as jokosher_bugtask. The
bugwatch prevents expiration, hence this bugtask will not appear
in the listings of expirable bugtasks below.

    >>> from lp.bugs.interfaces.bugtracker import IBugTrackerSet
    >>> mozilla_bugtracker = getUtility(IBugTrackerSet)['mozilla.org']
    >>> jokosher_bugtask_watched = create_old_bug('jokosher watched',
    ...     61, jokosher, external_bugtracker=mozilla_bugtracker)
    >>> jokosher_bugtask_watched.bug.can_expire
    False

Let's also make some bugs that almost qualify for expiration.

    # A bugtask whose status is not Incomplete is not expirable.
    # This one's status is New.
    >>> thunderbird = getUtility(IProductSet).getByName('thunderbird')
    >>> new_bugtask = bugtaskset.createTask(
    ...     bug=ubuntu_bugtask.bug, owner=sample_person,
    ...     product=thunderbird)
    >>> new_bugtask.status.title
    'New'
    >>> new_bugtask.bug.permits_expiration
    False
    >>> new_bugtask.bug.can_expire
    False

    # A bugtask that is not expirable because it is assigned.
    >>> assigned_bugtask = create_old_bug(
    ...     'assigned', 61, ubuntu, assignee=sample_person)
    >>> assigned_bugtask.bug.permits_expiration
    True
    >>> assigned_bugtask.bug.can_expire
    False

    # A bug with two Ubuntu tasks, one assigned Incomplete, and one
    # Invalid task, is not expirable.
    >>> ubuntu_alsa = ubuntu.getSourcePackage('alsa-utils')
    >>> another_assigned_bugtask = create_old_bug(
    ...     'assigned', 61, ubuntu, assignee=sample_person)
    >>> another_assigned_bugtask.transitionToTarget(ubuntu_alsa)
    >>> ubuntu_evolution = ubuntu.getSourcePackage('evolution')
    >>> invalid_bugtask = bugtaskset.createTask(
    ...     bug=another_assigned_bugtask.bug, owner=sample_person,
    ...     distribution=ubuntu,
    ...     sourcepackagename=ubuntu_evolution.sourcepackagename,
    ...     status=BugTaskStatus.INVALID)
    >>> another_assigned_bugtask.bug.permits_expiration
    True
    >>> another_assigned_bugtask.bug.can_expire
    False

    # A bugtask that is not expirable because its status is CONFIRMED.
    >>> confirmed_bugtask = create_old_bug(
    ...     'confirmed', 61, ubuntu, status=BugTaskStatus.CONFIRMED)
    >>> confirmed_bugtask.bug.permits_expiration
    False
    >>> confirmed_bugtask.bug.can_expire
    False

    # A bugtask that is not expirable because it is a duplicate.
    >>> duplicate_bugtask = create_old_bug(
    ...     'duplicate', 61, ubuntu, duplicateof=confirmed_bugtask.bug)
    >>> duplicate_bugtask.bug.permits_expiration
    True
    >>> duplicate_bugtask.bug.can_expire
    False

    # A bugtask that is not expirable because it does not use
    # Launchpad Bugs.
    >>> external_bugtask = create_old_bug('external', 61, thunderbird)
    >>> external_bugtask.bug.permits_expiration
    False
    >>> thunderbird.enable_bug_expiration
    False
    >>> external_bugtask.bug.can_expire
    False

    # A bugtask that is not expirable because it has a milestone.
    >>> milestone = ubuntu.currentseries.newMilestone("0.1")
    >>> Store.of(milestone).flush()
    >>> milestone_bugtask = create_old_bug(
    ...     'milestone', 61, ubuntu,
    ...     milestone=milestone)
    >>> milestone_bugtask.bug.permits_expiration
    True
    >>> milestone_bugtask.bug.can_expire
    False

    # Create a bugtask that is not old enough to expire
    >>> recent_bugtask = create_old_bug('recent', 31, ubuntu)
    >>> recent_bugtask.bug.permits_expiration
    True
    >>> recent_bugtask.bug.can_expire
    False

    # A bugtask that is not expirable; while the product uses Launchpad to
    # track bugs, enable_bug_expiration is set to False
    >>> firefox = getUtility(IProductSet).getByName('firefox')
    >>> no_expiration_bugtask = create_old_bug('no_expire', 61, firefox)
    >>> no_expiration_bugtask.bug.permits_expiration
    False
    >>> firefox.enable_bug_expiration
    False
    >>> no_expiration_bugtask.bug.can_expire
    False

The ubuntu, hoary, and jokosher bugs are the only ones that can be
expired. The other bugs do not meet one of the preconditions.

    >>> bugtasks = [ubuntu_bugtask, hoary_bugtask, jokosher_bugtask,
    ...     jokosher_bugtask_watched, new_bugtask, assigned_bugtask,
    ...     confirmed_bugtask, duplicate_bugtask, external_bugtask,
    ...     milestone_bugtask, recent_bugtask, no_expiration_bugtask]

    >>> from lp.bugs.tests.bug import summarize_bugtasks
    >>> summarize_bugtasks(bugtasks)
    ROLE             EXPIRE  AGE  STATUS      ASSIGNED  DUP    MILE   REPLIES
    ubuntu           False   351  Incomplete  False     False  False  False
    hoary            False   351  Incomplete  False     False  False  False
    jokosher         True     61  Incomplete  False     False  False  False
    jokosher watched False    61  Incomplete  False     False  False  False
    thunderbird      False   351  New         False     False  False  False
    assigned         False    61  Incomplete  True      False  False  False
    confirmed        False    61  Confirmed   False     False  False  False
    duplicate        False    61  Incomplete  False     True   False  False
    external         False    61  Incomplete  False     False  False  False
    milestone        False    61  Incomplete  False     False  True   False
    recent           False    31  Incomplete  False     False  False  False
    no_expire        False    61  Incomplete  False     False  False  False

== isExpirable() ==

In addition to can_expire bugs have an isExpirable method to which a custom
number of days, days_old, can be passed.  days_old is then used with
findExpirableBugTasks.  This allows projects to create their own janitor using
a different period for bug expiration.

    # Check to ensure that isExpirable() works without days_old, then set the
    # bug to Invalid so it doesn't affect the rest of the doctest
    >>> from lp.bugs.tests.bug import create_old_bug
    >>> very_old_bugtask = create_old_bug('expirable_distro', 351, ubuntu)
    >>> very_old_bugtask.bug.isExpirable()
    True
    >>> very_old_bugtask.transitionToStatus(
    ...     BugTaskStatus.INVALID, sample_person)

    # Pass isExpirable() a days_old parameter, then set the bug to Invalid so it
    # doesn't affect the rest of the doctest
    >>> from lp.bugs.tests.bug import create_old_bug
    >>> not_so_old_bugtask = create_old_bug('expirable_distro', 31, ubuntu)
    >>> not_so_old_bugtask.bug.isExpirable(days_old=14)
    True
    >>> not_so_old_bugtask.transitionToStatus(
    ...     BugTaskStatus.INVALID, sample_person)


== findExpirableBugTasks() Part 2 ==

The value of the min_days_old controls the bugtasks that are
returned. The oldest bug in this test is 351 days old, the youngest is
31 days old. There are no bugs older than 351 days.

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(351, None)
    >>> expirable_bugtasks.count()
    0

While there are bugtasks older than 350 days in the data, the hoary
bugtask does not display because its bug has other bugtasks that are
valid.

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(350, None)
    >>> expirable_bugtasks.count()
    0

    >>> hoary_bugtask.bug.can_expire
    False
    >>> summarize_bugtasks(hoary_bugtask.bug.bugtasks)
    ROLE         EXPIRE  AGE  STATUS      ASSIGNED  DUP    MILE   REPLIES
    ubuntu       False   351  Incomplete  False     False  False  False
    hoary        False   351  Incomplete  False     False  False  False
    thunderbird  False   351  New         False     False  False  False

If the valid bugtask becomes Invalid or Won't Fix, the hoary bugtask
will be expirable.

    >>> new_bugtask.transitionToStatus(BugTaskStatus.WONTFIX, sample_person)
    >>> hoary_bugtask.bug.can_expire
    True
    >>> summarize_bugtasks(hoary_bugtask.bug.bugtasks)
    ROLE         EXPIRE  AGE  STATUS      ASSIGNED  DUP    MILE   REPLIES
    ubuntu       True    351  Incomplete  False     False  False  False
    hoary        True    351  Incomplete  False     False  False  False
    thunderbird  False   351  Won't Fix   False     False  False  False

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(350, None)
    >>> summarize_bugtasks(expirable_bugtasks)
    ROLE         EXPIRE  AGE  STATUS      ASSIGNED  DUP    MILE   REPLIES
    ubuntu       True    351  Incomplete  False     False  False  False
    hoary        True    351  Incomplete  False     False  False  False

The ubuntu bugtask is never returned; it is a conjoined slave to the
hoary bugtask. Slave bugtasks cannot be directly expired, so they are
not returned by findExpirableBugTasks().

    >>> ubuntu_bugtask.status.title
    'Incomplete'
    >>> ubuntu_bugtask.conjoined_master == hoary_bugtask
    True

Reducing the age to 60 days old, both hoary and jokosher bugtasks
are returned.

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(60, None)
    >>> summarize_bugtasks(expirable_bugtasks)
    ROLE         EXPIRE  AGE  STATUS      ASSIGNED  DUP    MILE   REPLIES
    ubuntu       True    351  Incomplete  False     False  False  False
    hoary        True    351  Incomplete  False     False  False  False
    jokosher     True     61  Incomplete  False     False  False  False


When a bug is passed as an argument to findExpirableBugTasks(), it
returns that bug's expirable BugTasks, or an empty list. Passing the bug
that has the hoary and ubuntu bugtasks with 0 min_days_old returns just
the hoary bugtask.

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(
    ...     0, None, bug=hoary_bugtask.bug)
    >>> summarize_bugtasks(expirable_bugtasks)
    ROLE         EXPIRE  AGE  STATUS      ASSIGNED  DUP    MILE   REPLIES
    ubuntu       True    351  Incomplete  False     False  False  False
    hoary        True    351  Incomplete  False     False  False  False

When a BugTarget is passed as an argument to findExpirableBugTasks(), it
returns all the target's expirable bugtasks, or an empty list. If the
target's pillar has not enabled bug expiration, None is always returned.
Passing ubuntu with 0 min_days_old shows that the distribution has two
bugtasks that can expire if they are not confirmed.

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(
    ...     0, None, target=ubuntu)
    >>> summarize_bugtasks(expirable_bugtasks)
    ROLE         EXPIRE  AGE  STATUS      ASSIGNED  DUP    MILE   REPLIES
    ubuntu       True    351  Incomplete  False     False  False  False
    hoary        True    351  Incomplete  False     False  False  False
    recent       False    31  Incomplete  False     False  False  False

findExpirableBugTasks also accepts a limit argument, which allows for limiting
the number of bugtasks returned.

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(
    ...     0, None, target=ubuntu, limit=2)
    >>> summarize_bugtasks(expirable_bugtasks)
    ROLE         EXPIRE  AGE  STATUS      ASSIGNED  DUP    MILE   REPLIES
    ubuntu       True    351  Incomplete  False     False  False  False
    hoary        True    351  Incomplete  False     False  False  False

Thunderbird has not enabled bug expiration. Even when the min_days_old
is set to 0, no bugtasks are replaced.

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(
    ...     0, None, target=thunderbird)
    >>> summarize_bugtasks(expirable_bugtasks)
    ROLE         EXPIRE  AGE  STATUS      ASSIGNED  DUP    MILE   REPLIES


== Privacy ==

The user parameter indicates which user is performing the search. Only
bugs that the user has permission to view are returned. A value of None
indicates the anonymous user.

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(
    ...     0, user=None, target=ubuntu)
    >>> visible_bugs = set(bugtask.bug for bugtask in expirable_bugtasks)
    >>> print sorted(bug.title for bug in visible_bugs)
    [u'expirable_distro', u'recent']

If one of the bugs is set to private, anonymous users can no longer see
it as being marked for expiration.

    >>> private_bug = ubuntu_bugtask.bug
    >>> private_bug.title
    u'expirable_distro'
    >>> private_bug.setPrivate(True, sample_person)
    True

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(
    ...     0, user=None, target=ubuntu)
    >>> visible_bugs = set(bugtask.bug for bugtask in expirable_bugtasks)
    >>> print sorted(bug.title for bug in visible_bugs)
    [u'recent']

No Privileges Person can't see the bug either...

    >>> no_priv = getUtility(IPersonSet).getByName('no-priv')
    >>> private_bug.unsubscribe(no_priv, no_priv)
    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(
    ...     0, user=no_priv, target=ubuntu)
    >>> visible_bugs = set(bugtask.bug for bugtask in expirable_bugtasks)
    >>> print sorted(bug.title for bug in visible_bugs)
    [u'recent']

... unless he's subscribed to the bug.

    >>> private_bug.subscribe(no_priv, sample_person)
    <lp.bugs.model.bugsubscription.BugSubscription ...>
    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(
    ...     0, user=no_priv, target=ubuntu)
    >>> visible_bugs = set(bugtask.bug for bugtask in expirable_bugtasks)
    >>> print sorted(bug.title for bug in visible_bugs)
    [u'expirable_distro', u'recent']

The Janitor needs to be able to access all bugs, even private ones, in
order to be able to expire them. If the Janitor is passed as the user,
even the private bugs are returned.

    >>> from lp.app.interfaces.launchpad import ILaunchpadCelebrities
    >>> janitor = getUtility(ILaunchpadCelebrities).janitor
    >>> private_bug.isSubscribed(janitor)
    False

    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(
    ...     0, user=janitor, target=ubuntu)
    >>> visible_bugs = set(bugtask.bug for bugtask in expirable_bugtasks)
    >>> print sorted(bug.title for bug in visible_bugs)
    [u'expirable_distro', u'recent']

    >>> private_bug.setPrivate(False, sample_person)
    True


== The default expiration age ==

The expiration age is set using the
config.malone.days_before_expiration configuration variable. It
defaults to 60 days. The period is measured from the date_incomplete
field. We expire bugtasks that are Incomplete and unattended for 60
days or more.

    >>> from canonical.config import config
    >>> old_age_days = config.malone.days_before_expiration
    >>> old_age_days
    60


== Running the script ==

There are no Expired Bugtasks in sampledata, from the tests above.

    >>> from lp.bugs.model.bugtask import BugTask
    >>> BugTask.selectBy(status=BugTaskStatus.EXPIRED).count()
    0

    >>> # We want to check the hoary bugtask messages later.
    >>> starting_bug_messages_count = (hoary_bugtask.bug.messages.count())

The script 'expire-bugtasks.py' writes its report to stdout. It makes
its database changes as the user configured in
config.malone.expiration_dbuser.

    >>> config.malone.expiration_dbuser
    'bugnotification'

    # Commit the current transaction because the script will run in
    # another transaction, and thus it won't see the changes done on
    # this test unless we commit.
    >>> commit()

    >>> import subprocess
    >>> process = subprocess.Popen(
    ...     'cronscripts/expire-bugtasks.py', shell=True,
    ...     stdin=subprocess.PIPE, stdout=subprocess.PIPE,
    ...     stderr=subprocess.PIPE)
    >>> (out, err) = process.communicate()
    >>> print err
    INFO    Creating lockfile: /var/lock/launchpad-expire-bugtasks.lock
    INFO    Expiring unattended, INCOMPLETE bugtasks older than
            60 days for projects that use Launchpad Bugs.
    INFO    Found 3 bugtasks to expire.
    INFO    Expired 2 bugtasks.
    INFO    Finished expiration run.
    <BLANKLINE>
    >>> print out
    <BLANKLINE>
    >>> process.returncode
    0

    >>> bugtasks = [BugTask.get(bugtask.id) for bugtask in bugtasks]


== After the script has run ==

There are three Expired bugtasks. Jokosher, hoary and ubuntu were
expired by the expiration process. Although ubuntu was never returned
by findExpirableBugTasks(), it was expired because its master (hoary)
was expired. The remaining bugtasks are unchanged.

    >>> summarize_bugtasks(bugtasks)
    ROLE             EXPIRE  AGE  STATUS      ASSIGNED  DUP    MILE   REPLIES
    ubuntu           False     0  Expired     False     False  False  False
    hoary            False     0  Expired     False     False  False  False
    jokosher         False     0  Expired     False     False  False  False
    jokosher watched False    61  Incomplete  False     False  False  False
    thunderbird      False     0  Won't Fix   False     False  False  False
    assigned         False    61  Incomplete  True      False  False  False
    confirmed        False    61  Confirmed   False     False  False  False
    duplicate        False    61  Incomplete  False     True   False  False
    external         False    61  Incomplete  False     False  False  False
    milestone        False    61  Incomplete  False     False  True   False
    recent           False    31  Incomplete  False     False  False  False
    no_expire        False    61  Incomplete  False     False  False  False

The bugtasks statusexplanation was updated to explain the change in
status.

    >>> hoary_bugtask = BugTask.get(hoary_bugtask.id)
    >>> print hoary_bugtask.statusexplanation
    [Expired for Ubuntu Hoary because there has been no activity for 60 days.]

The message explaining the reason for the expiration was posted by the
Launchpad Janitor celebrity. Only one message was created for when the
master and slave bugtasks were expired.

    >>> starting_bug_messages_count
    2
    >>> hoary_bugtask.bug.messages.count()
    3

    >>> message = hoary_bugtask.bug.messages[-1]
    >>> print message.owner.name
    janitor

    >>> print message.text_contents
    [Expired for Ubuntu Hoary because there has been no activity for 60 days.]

The bug's activity log was updated too with the status change.

    >>> activity = hoary_bugtask.bug.activity[-1]
    >>> print "%s  %s  %s  %s" % (
    ...     activity.person.displayname, activity.whatchanged,
    ...     activity.oldvalue, activity.newvalue)
    Launchpad Janitor  Ubuntu Hoary: status  Incomplete  Expired


== enable_bug_expiration ==

The bugtask no_expiration_bugtask has not been expired because it does
not participate in bug expiration. When uses_bug_expiration is set to
True for a project, old bugs will be expired the next time the bugs are
expired.

    >>> no_expiration_bugtask.pillar.enable_bug_expiration = True

    >>> no_expiration_bugtask.bug.permits_expiration
    True
    >>> no_expiration_bugtask.bug.can_expire
    True
    >>> expirable_bugtasks = bugtaskset.findExpirableBugTasks(60, None)
    >>> summarize_bugtasks(expirable_bugtasks)
    ROLE         EXPIRE  AGE  STATUS      ASSIGNED  DUP    MILE   REPLIES
    no_expire    True    61   Incomplete  False     False  False  False