~launchpad-pqm/launchpad/devel

Launchpad search page
=====================

Users can search for Launchpad objects and pages from the search form
located on all pages. The search is performed and displayed by the
LaunchpadSearchView.

    >>> from zope.component import getMultiAdapter, getUtility
    >>> from canonical.launchpad.webapp.interfaces import ILaunchpadRoot
    >>> from canonical.launchpad.webapp.servers import LaunchpadTestRequest

    >>> root = getUtility(ILaunchpadRoot)
    >>> request = LaunchpadTestRequest()
    >>> search_view = getMultiAdapter((root, request), name="+search")
    >>> search_view.initialize()
    >>> search_view
    <....SimpleViewClass from .../templates/launchpad-search.pt ...>


Page title and heading
----------------------

The page title and heading suggest to the user to search launchpad
when there is no search text.

    >>> print search_view.text
    None
    >>> search_view.page_title
    'Search Launchpad'
    >>> search_view.page_heading
    'Search Launchpad'

When text is not None, the title indicates what was searched.

    >>> def getSearchView(form):
    ...     search_param_list = []
    ...     for name in sorted(form):
    ...         value = form[name]
    ...         search_param_list.append('%s=%s' % (name, value))
    ...     query_string = '&'.join(search_param_list)
    ...     request = LaunchpadTestRequest(
    ...         SERVER_URL='https://launchpad.dev/+search',
    ...         QUERY_STRING=query_string, form=form, PATH_INFO='/+search')
    ...     search_view = getMultiAdapter((root, request), name="+search")
    ...     search_view.initialize()
    ...     return search_view

    >>> search_view = getSearchView(
    ...     form={'field.text': 'albatross'})

    >>> search_view.text
    u'albatross'
    >>> search_view.page_title
    u'Pages matching "albatross" in Launchpad'
    >>> search_view.page_heading
    u'Pages matching "albatross" in Launchpad'


No matches
----------

There were no matches for 'albatross'.

    >>> search_view.has_matches
    False

When search text is not submitted there are no matches. Search text is
required to perform a search. Note that field.actions.search is not a
required param to call the Search Action. The view always calls the
search action.

    >>> search_view = getSearchView(form={})

    >>> print search_view.text
    None
    >>> search_view.has_matches
    False


Bug and Question Searches
-------------------------

When a numeric token can be extracted from the submitted search text,
the view tries to match a bug and question. Bugs and questions are
matched by their id.

    >>> search_view = getSearchView(
    ...     form={'field.text': '5'})
    >>> search_view._getNumericToken(search_view.text)
    u'5'
    >>> search_view.has_matches
    True
    >>> search_view.bug.title
    u'Firefox install instructions should be complete'
    >>> search_view.question.title
    u'Installation failed'

Bugs and questions are matched independent of each other. The number
extracted may only match one kind of object. For example, there are
more bugs than questions.

    >>> search_view = getSearchView(
    ...     form={'field.text': '15'})
    >>> search_view._getNumericToken(search_view.text)
    u'15'
    >>> search_view.has_matches
    True
    >>> search_view.bug.title
    u'Nonsensical bugs are useless'
    >>> print search_view.question
    None

Private bugs are not matched if the user does not have permission to
see them. For example, Sample Person can see a private bug that he
created because he is the owner.

    >>> from canonical.launchpad.webapp.interfaces import ILaunchBag

    >>> login('test@canonical.com')
    >>> sample_person = getUtility(ILaunchBag).user
    >>> private_bug = factory.makeBug(owner=sample_person, private=True)

    >>> search_view = getSearchView(
    ...     form={'field.text': private_bug.id})
    >>> search_view.bug.private
    True

But anonymous and unprivileged users cannot see the private bug.

    >>> login(ANONYMOUS)
    >>> search_view = getSearchView(
    ...     form={'field.text': private_bug.id})
    >>> print search_view.bug
    None

The text and punctuation in the search text is ignored, and only the
first group of numbers is matched. For example a user searches for three
questions by number ('Question #15, #7, and 5.'). Only the first number
is used, and it matches a bug, not a question. The second and third
numbers do match questions, but they are not used.

    >>> search_view = getSearchView(
    ...     form={'field.text': 'Question #15, #7, and 5.'})
    >>> search_view._getNumericToken(search_view.text)
    u'15'
    >>> search_view.has_matches
    True
    >>> search_view.bug.title
    u'Nonsensical bugs are useless'
    >>> print search_view.question
    None

It is not an error to search for a non-existent bug or question.

    >>> search_view = getSearchView(
    ...     form={'field.text': '55555'})
    >>> search_view._getNumericToken(search_view.text)
    u'55555'
    >>> search_view.has_matches
    False
    >>> print search_view.bug
    None
    >>> print search_view.question
    None

There is no error if a number cannot be extracted from the search text.

    >>> search_view = getSearchView(
    ...     form={'field.text': 'fifteen'})
    >>> print search_view._getNumericToken(
    ...     search_view.text)
    None
    >>> search_view.has_matches
    False
    >>> print search_view.bug
    None
    >>> print search_view.question
    None

Bugs and questions are only returned for the first page of search,
when the start param is 0.

    >>> search_view = getSearchView(
    ...     form={'field.text': '5',
    ...           'start': '20'})
    >>> search_view.has_matches
    False
    >>> print search_view.bug
    None
    >>> print search_view.question
    None



Projects and Persons and Teams searches
---------------------------------------

When a Launchpad name can be made from the search text, the view tries
to match the name to a pillar or person. a pillar is a distribution,
product, or project group. A person is a person or a team.

    >>> search_view = getSearchView(
    ...     form={'field.text': 'launchpad'})
    >>> search_view._getNameToken(search_view.text)
    u'launchpad'
    >>> search_view.has_matches
    True
    >>> search_view.pillar.displayname
    u'Launchpad'
    >>> search_view.person_or_team.displayname
    u'Launchpad Developers'

A launchpad name is constructed from the search text. The letters are
converted to lowercase. groups of spaces and punctuation are replaced
with a hyphen.

    >>> search_view = getSearchView(
    ...     form={'field.text': 'Gnome Terminal'})
    >>> search_view._getNameToken(search_view.text)
    u'gnome-terminal'
    >>> search_view.has_matches
    True
    >>> search_view.pillar.displayname
    u'GNOME Terminal'
    >>> print search_view.person_or_team
    None

Since our pillars can have aliases, it's also possible to look up a pillar
by any of its aliases.

    >>> from lp.registry.interfaces.product import IProductSet
    >>> firefox = getUtility(IProductSet)['firefox']
    >>> login('foo.bar@canonical.com')
    >>> firefox.setAliases(['iceweasel'])
    >>> login(ANONYMOUS)
    >>> search_view = getSearchView(
    ...     form={'field.text': 'iceweasel'})
    >>> search_view._getNameToken(search_view.text)
    u'iceweasel'
    >>> search_view.has_matches
    True
    >>> search_view.pillar.displayname
    u'Mozilla Firefox'

This is a harder example that illustrates that text that is clearly not
the name of a pillar will none-the-less be tried. See the `Page searches`
section for how this kind of search can return matches.

    >>> search_view = getSearchView(
    ...     form={'field.text': "YAHOO! webservice's Python API."})
    >>> search_view._getNameToken(search_view.text)
    u'yahoo-webservices-python-api.'
    >>> search_view.has_matches
    False
    >>> print search_view.pillar
    None
    >>> print search_view.person_or_team
    None

Leading and trailing punctuation and whitespace are stripped.

    >>> search_view = getSearchView(
    ...     form={'field.text': "~name12"})
    >>> search_view._getNameToken(search_view.text)
    u'name12'
    >>> search_view.has_matches
    True
    >>> print search_view.pillar
    None
    >>> search_view.person_or_team.displayname
    u'Sample Person'

Pillars, persons and teams are only returned for the first page of
search, when the start param is 0.

    >>> search_view = getSearchView(
    ...     form={'field.text': 'launchpad',
    ...           'start': '20'})
    >>> search_view.has_matches
    True
    >>> print search_view.bug
    None
    >>> print search_view.question
    None
    >>> print search_view.pillar
    None

Deactivated pillars and non-valid persons and teams cannot be exact
matches. For example, the python-gnome2-dev product will not match a
pillar, nor will nsv match Nicolas Velin's unclaimed account.

    >>> from lp.registry.interfaces.person import IPersonSet

    >>> python_gnome2 = getUtility(IProductSet).getByName('python-gnome2-dev')
    >>> python_gnome2.active
    False

    >>> search_view = getSearchView(
    ...     form={'field.text': 'python-gnome2-dev',
    ...           'start': '0'})
    >>> search_view._getNameToken(search_view.text)
    u'python-gnome2-dev'
    >>> print search_view.pillar
    None

    >>> nsv = getUtility(IPersonSet).getByName('nsv')
    >>> nsv.displayname
    u'Nicolas Velin'
    >>> nsv.is_valid_person_or_team
    False

    >>> search_view = getSearchView(
    ...     form={'field.text': 'nsv',
    ...           'start': '0'})
    >>> search_view._getNameToken(search_view.text)
    u'nsv'
    >>> print search_view.person_or_team
    None


Shipit CD searches
------------------

The has_shipit property will be True when the search looks like the user
is searching for Shipit CDs. There is no correct object in Launchpad to
display. The page template decides how to handle when has_shipit is
True.

The match is based on an intersection to the words in the search text
and the shipit_keywords. The comparison is case-insensitive, has_shipit
is True when 2 or more words match.

    >>> sorted(search_view.shipit_keywords)
    ['cd', 'cds', 'disc', 'dvd', 'dvds', 'edubuntu', 'free', 'get', 'kubuntu',
     'mail', 'send', 'ship', 'shipit', 'ubuntu']
    >>> search_view = getSearchView(
    ...     form={'field.text': 'ubuntu CDs',
    ...           'start': '0'})
    >>> search_view.has_shipit
    True

    >>> search_view = getSearchView(
    ...     form={'field.text': 'shipit',
    ...           'start': '0'})
    >>> search_view.has_shipit
    False

    >>> search_view = getSearchView(
    ...     form={'field.text': 'get Kubuntu cds',
    ...           'start': '0'})
    >>> search_view.has_shipit
    True

There are shipit_anti_keywords too, words that indicate the search is
not for free CDs from Shipit. Search that have any of these word will
set has_shipit to False.

    >>> sorted(search_view.shipit_anti_keywords)
    ['burn', 'burning', 'enable', 'error', 'errors', 'image', 'iso',
     'read', 'rip', 'write']

    >>> search_view = getSearchView(
    ...     form={'field.text': 'ubuntu CD write',
    ...           'start': '0'})
    >>> search_view.has_shipit
    False

    >>> search_view = getSearchView(
    ...     form={'field.text': 'shipit error',
    ...           'start': '0'})
    >>> search_view.has_shipit
    False


The shipit FAQ URL is provides by the view for the template to use.

    >>> search_view.shipit_faq_url
    'http://www.ubuntu.com/getubuntu/shipit-faq'


Page searches
-------------

The view uses the GoogleSearchService to locate pages that match the
search terms.

    >>> search_view = getSearchView(
    ...     form={'field.text': " bug"})
    >>> search_view.text
    u'bug'
    >>> search_view.has_matches
    True
    >>> search_view.pages
    <...GoogleBatchNavigator ...>

The GoogleSearchService may not be available due to connectivity problems.
The view's has_page_service attribute reports when the search was performed
with Google page matches.

    >>> search_view.has_page_service
    True

The batch navigation heading is created by the view. The heading
property returns a 2-tuple of singular and plural heading. There
is a heading when there are only Google page matches...

    >>> search_view.has_exact_matches
    False
    >>> search_view.batch_heading
    (u'page matching "bug"', u'pages matching "bug"')

...and a heading for when there are exact matches and Google page
matches.

    >>> search_view = getSearchView(
    ...     form={'field.text': " launchpad"})
    >>> search_view.has_exact_matches
    True
    >>> search_view.batch_heading
    (u'other page matching "launchpad"', u'other pages matching "launchpad"')

The GoogleBatchNavigator behaves like most BatchNavigators, except that
its batch size is always 20. The size restriction conforms to Google's
maximum number of results that can be returned per request.

    >>> search_view.start
    0
    >>> search_view.pages.currentBatch().size
    20
    >>> pages = list(search_view.pages.currentBatch())
    >>> len(pages)
    20
    >>> for page in pages[0:5]:
    ...     page.title
    'Launchpad Bugs'
    'Bugs in Ubuntu Linux'
    'Bugs related to Sample Person'
    u'<b>Bug</b> #1 in Mozilla Firefox: ...Firefox does not support SVG...'
    'Bugs in Source Package "thunderbird" in Ubuntu Linux'

The batch navigator provides access to the other batches. There are two
batches of pages that match the search text 'bugs'. The navigator
provides a link to the next batch, which also happens to be the last
batch.

    >>> search_view.pages.nextBatchURL()
    '...start=20'
    >>> search_view.pages.lastBatchURL()
    '...start=20'

The second batch has only five matches in it, even though the batch size
is 20. That is because there were only 25 matching pages.

    >>> search_view = getSearchView(
    ...     form={'field.text': "bug",
    ...           'start': '20'})
    >>> search_view.start
    20
    >>> search_view.text
    u'bug'
    >>> search_view.has_matches
    True

    >>> search_view.pages.currentBatch().size
    20
    >>> pages = list(search_view.pages.currentBatch())
    >>> len(pages)
    5
    >>> for page in pages:
    ...     page.title
    u'<b>Bug</b> #2 in Ubuntu Hoary: \u201cBlackhole Trash folder\u201d'
    u'<b>Bug</b> #2 in mozilla-firefox (Debian): ...Blackhole Trash folder...'
    u'<b>Bug</b> #3 in mozilla-firefox (Debian): \u201cBug Title Test\u201d'
    '<b>Bug</b> trackers registered in Launchpad'
    u'<b>Bug</b> tracker \u201cDebian Bug tracker\u201d'

    >>> search_view.pages.nextBatchURL()
    ''
    >>> search_view.pages.lastBatchURL()
    ''

The PageMatch object has a title, url, and summary. The title and url
are used for making links to the pages. The summary contains markup
showing the matching terms in context of the page text.

    >>> page = pages[0]
    >>> page
    <...PageMatch ...>
    >>> page.title
    u'<b>Bug</b> #2 in Ubuntu Hoary: \u201cBlackhole Trash folder\u201d'
    >>> page.url
    'http://bugs.launchpad.dev/ubuntu/hoary/+bug/2'
    >>> page.summary
    u'<b>Bug</b> tracking <b>...</b> Search <b>bugs</b> reports ...'

See `google-searchservice.txt` for more information about the
GoogleSearchService and PageMatch objects.


No page matches
---------------

When an empty PageMatches object is returned by the GoogleSearchService to
the view, there are no matches to show.

    >>> search_view = getSearchView(form={'field.text': 'no-meaningful'})
    >>> search_view.has_matches
    False


Unintelligible searches
-----------------------

When a user searches for a malformed string, we don't OOPS, but show an
error. Also disable warnings, since we are tossing around malformed Unicode.

    >>> import warnings
    >>> with warnings.catch_warnings():
    ...     warnings.simplefilter('ignore')
    ...     search_view = getSearchView(
    ...         form={'field.text': '\xfe\xfckr\xfc'})
    >>> html = search_view()
    >>> 'Can not convert your search term' in html
    True


Bad Google response handling
----------------------------

Connectivity problems can cause missing or incomplete responses from
Google. The LaunchpadSearchView will display the other searches and
show a message explaining that the user can search again to find
matching pages.

    >>> search_view = getSearchView(form={'field.text': 'gnomebaker'})
    >>> search_view.has_matches
    True
    >>> search_view.pillar.displayname
    u'gnomebaker'
    >>> search_view.has_page_service
    False

The view provides the requested URL so that the template can make a
link to try the search again

    >>> print search_view.url
    https://launchpad.dev/+search?field.text=gnomebaker


SearchFormView and SearchFormPrimaryView
----------------------------------------

Two companion views are used to help render the global search form.
They define the required attributes to render the form in the
correct state.

The LaunchpadSearchFormView provides the minimum information to display
the form, but cannot handled the submitted data. It appends a suffix
('-secondary') to the id= and name= of the form and inputs, to prevent
them from conflicting with the other form. The search text is not the
default value of the text field; 'bug' was submitted above, but is not
present in the rendered form.

    >>> search_form_view = getMultiAdapter(
    ...     (search_view, request), name='+search-form')
    >>> search_form_view.initialize()
    >>> search_form_view.id_suffix
    '-secondary'
    >>> print search_form_view.render()
    <form action="http://launchpad.dev/+search" method="get"
      accept-charset="UTF-8" id="sitesearch-secondary"
      name="sitesearch-secondary">
      <div>
        <input class="textType" type="text" size="36"
          id="field.text-secondary" name="field.text" />
        <input class="button" type="submit" value="Search"
          id="field.text-secondary" name="field.actions.search-secondary" />
      </div>
    </form>

LaunchpadPrimarySearchFormView can handle submitted form by deferring to
its context (the LaunchpadSearchView) for the needed information. The
view does not append a suffix to the form and input ids. The search
field's value is 'bug', as was submitted above.

    >>> search_form_view = getMultiAdapter(
    ...     (search_view, request), name='+primary-search-form')
    >>> search_form_view.initialize()
    >>> search_form_view.id_suffix
    ''
    >>> print search_form_view.render()
    <form action="http://launchpad.dev/+search" method="get"
      accept-charset="UTF-8" id="sitesearch"
      name="sitesearch">
      <div>
        <input class="textType" type="text" size="36"
          id="field.text" value="gnomebaker" name="field.text" />
        <input class="button" type="submit" value="Search"
          id="field.text" name="field.actions.search" />
      </div>
    </form>

WindowedList and GoogleBatchNavigator
-------------------------------------

The LaunchpadSearchView uses two helper classes to work with
PageMatches.

The PageMatches object returned by the GoogleSearchService contains 20
or fewer PageMatches of what could be thousands of matches. Google
requires client's to make repeats request to step though the batches of
matches. The Windowed list is a list that contains only a subset of its
reported size. It is used to make batches in the GoogleBatchNavigator.

For example, the last batch of the 'bug' search contained 5 of the 25
matching pages. The WindowList claims to be 25 items in length, but
the first 20 items are None. Only the last 5 items are PageMatches.

    >>> from lp.app.browser.root import WindowedList
    >>> from lp.services.googlesearch import GoogleSearchService

    >>> google_search = GoogleSearchService()
    >>> page_matches = google_search.search(terms='bug', start=20)
    >>> results = WindowedList(
    ...     page_matches, page_matches.start, page_matches.total)
    >>> len(results)
    25
    >>> print results[0]
    None
    >>> results[24].title
    u'<b>Bug</b> tracker \u201cDebian Bug tracker\u201d'
    >>> results[18, 22]
    [None, None, <...PageMatch ...>, <...PageMatch ...>]

The GoogleBatchNavigator restricts the batch size to 20. the 'batch'
parameter that comes from the URL is ignored. For example, setting
the 'batch' parameter to 100 has no affect upon the Google search
or on the navigator object.

    >>> from lp.app.browser.root import GoogleBatchNavigator

    >>> GoogleBatchNavigator.batch_variable_name
    'batch'

    >>> search_view = getSearchView(
    ...     form={'field.text': "bug",
    ...           'start': '0',
    ...           'batch': '100',})

    >>> navigator = search_view.pages
    >>> navigator.currentBatch().size
    20
    >>> len(navigator.currentBatch())
    20
    >>> navigator.nextBatchURL()
    '...start=20'

Even if the PageMatch object to have an impossibly large size, the
navigator conforms to Google's maximum size of 20.

    >>> matches = range(0, 100)
    >>> page_matches._matches = matches
    >>> page_matches.start = 0
    >>> page_matches.total = 100
    >>> navigator = GoogleBatchNavigator(
    ...     page_matches, search_view.request, page_matches.start, size=100)
    >>> navigator.currentBatch().size
    20
    >>> len(navigator.currentBatch())
    20
    >>> navigator.nextBatchURL()
    '...start=20'

The PageMatches object can be smaller than 20, for instance, pages
without titles are skipped when parsing the Google Search XML. The size
of the batch is still 20, but when the items in the batch are iterated,
the true size can be seen. For example there could be only 3 matches in
the PageMatches object, so only 3 are yielded. The start of the next
batch is 20, which is the start of the next batch from Google.

    >>> matches = range(0, 3)
    >>> page_matches._matches = matches
    >>> navigator = GoogleBatchNavigator(
    ...     page_matches, search_view.request, page_matches.start, size=100)
    >>> batch = navigator.currentBatch()
    >>> batch.size
    20
    >>> len(batch)
    20
    >>> batch.endNumber()
    3
    >>> for item in batch:
    ...     print item
    0
    1
    2
    >>> navigator.nextBatchURL()
    '...start=20'