~launchpad-pqm/launchpad/devel

Librarian Access

================

The librarian is a file storage service for launchpad. Conceptually

similar to other file storage API's like S3, it is used to store binary

or large content - bug attachments, package builds, images and so on.

Content in the librarian can be exposed at different urls. To expose

some content use a LibraryFileAlias. Private content is supported as

well - for that tokens are added to permit access for a limited time by

a client - each time a client attempts to dereference a private

LibraryFileAlias a token is emitted.

Deployment notes

================

(These may seem a bit out of place - they are, but they need to be

written down somewhere, and the deployment choices inform the

implementation choices).

The basics are simple: The librarian talks to clients. However

restricted file access makes things a little more complex. As the

librarian itself doesn't do SSL processing, and we want restricted files

to be kept confidential the librarian will need a hint from the SSL

front end that SSL was in fact used. The semi standard header Front-End-

Https can be used for this if we filter it in incoming requests from

clients.

setUp

-----

    >>> from canonical.database.sqlbase import session_store

    >>> from lp.services.librarian.model import TimeLimitedToken

High Level

----------

    >>> from StringIO import StringIO

    >>> from lp.services.librarian.interfaces import (

    ...     ILibraryFileAliasSet)

    >>> data = 'This is some data'

We can create LibraryFileAliases using the ILibraryFileAliasSet utility.

This name is a mouthful, but is consistent with the rest of our naming.

    >>> lfas = getUtility(ILibraryFileAliasSet)

    >>> from lp.services.librarian.interfaces import NEVER_EXPIRES

    >>> alias = lfas.create(

    ...     'text.txt', len(data), StringIO(data), 'text/plain', NEVER_EXPIRES

    ...     )

    >>> alias.mimetype

    u'text/plain'

    >>> alias.filename

    u'text.txt'

We may wish to set an expiry timestamp on the file. The NEVER_EXPIRES

constant means the file will never be removed from the Librarian, and

because of this should probably never be used.

    >>> alias.expires == NEVER_EXPIRES

    True

    >>> alias = lfas.create(

    ...     'text.txt', len(data), StringIO(data), 'text/plain')

The default expiry of None means the file will expire a few days after

it is no longer referenced in the database.

    >>> alias.expires is None

    True

The creation timestamp of the LibraryFileAlias is available in the

date_created attribute.

    >>> alias.date_created

    datetime.datetime(...)

We can retrieve the LibraryFileAlias we just created using its ID or

sha1.

    >>> org_alias_id = alias.id

    >>> alias = lfas[org_alias_id]

    >>> alias.id == org_alias_id

    True

    >>> org_alias_id in [a.id for a in lfas.findBySHA1(alias.content.sha1)]

    True

We can get its URL too

    >>> from lp.services.config import config

    >>> import re

    >>> re.search(

    ...     r'^%s\d+/text.txt$' % config.librarian.download_url,

    ...     alias.http_url

    ...     ) is not None

    True

Librarian also serves the same file through https, we use this for

branding and similar inline-presented objects which would trigger

security warnings on https pages otherwise.

    >>> re.search(r'^https://.+/\d+/text.txt$', alias.https_url) is not None

    True

And we even have a convenient method which returns either the http URL

or the https one, depending on a config value.

    >>> config.vhosts.use_https

    False

    >>> re.search(

    ...     r'^%s\d+/text.txt$' % config.librarian.download_url,

    ...     alias.getURL()

    ...     ) is not None

    True

    >>> from textwrap import dedent

    >>> test_data = dedent("""

    ...     [librarian]

    ...     use_https: true

    ...     """)

    >>> config.push('test', test_data)

    >>> re.search(

    ...     r'^https://.+/\d+/text.txt$', alias.https_url

    ...     ) is not None

    True

However, we can force the use of HTTP by setting the 'HTTP_X_SCHEME'

header in the request to 'http', even when 'use_https' is True.

    >>> from zope.component import getMultiAdapter

    >>> from lp.services.webapp.servers import LaunchpadTestRequest

    >>> from urlparse import urlparse

    >>> request = LaunchpadTestRequest(

    ...     environ={'REQUEST_METHOD': 'GET', 'HTTP_X_SCHEME': 'http'})

    >>> view = getMultiAdapter((alias,request), name='+index')

    >>> view.initialize()

    >>> print urlparse(request.response.getHeader('Location'))[0]

    http

When the incoming scheme is 'https' then the redirect scheme is

unaffected.

    >>> request = LaunchpadTestRequest(

    ...     environ={'REQUEST_METHOD': 'GET', 'HTTP_X_SCHEME': 'https'})

    >>> view = getMultiAdapter((alias,request), name='+index')

    >>> view.initialize()

    >>> print urlparse(request.response.getHeader('Location'))[0]

    https

Reset 'use_https' to its original state.

    >>> test_config_data = config.pop('test')

However, we can't access its contents until we have committed

    >>> alias.open()

    Traceback (most recent call last):

        [...]

    LookupError: ...

Once we commit the transaction, LibraryFileAliases can be accessed like

files.

    >>> import transaction

    >>> transaction.commit()

    >>> alias.open()

    >>> alias.read()

    'This is some data'

    >>> alias.close()

We can also read it in chunks.

    >>> alias.open()

    >>> alias.read(2)

    'Th'

    >>> alias.read(6)

    'is is '

    >>> alias.read()

    'some data'

    >>> alias.close()

If you don't want to read the file in chunks you can neglect to call

open() and close().

    >>> alias.read()

    'This is some data'

Each alias also has an expiry date associated with it, the default of

None meaning the file will expire a few days after nothing references it

any more:

    >>> alias.expires is None

    True

Closing an alias repeatedly and/or without opening it beforehand is

tolerated and will not result in exceptions being raised.

    >>> alias.close()

    >>> alias.close()

Low Level

---------

We can also use the ILibrarianClient Utility directly to store and

access files in the Librarian.

    >>> from lp.services.librarian.interfaces.client import ILibrarianClient

    >>> client = getUtility(ILibrarianClient)

    >>> aid = client.addFile(

    ...     'text.txt', len(data), StringIO(data), 'text/plain', NEVER_EXPIRES

    ...     )

    >>> transaction.commit()

    >>> f = client.getFileByAlias(aid)

    >>> f.read()

    'This is some data'

    >>> url = client.getURLForAlias(aid)

    >>> re.search(

    ...     r'^%s\d+/text.txt$' % config.librarian.download_url, url

    ...     ) is not None

    True

When secure=True, the returned url has the id as part of the domain name

and the protocol is https:

    >>> expected = r'^https://i%d\..+:\d+/%d/text.txt$' % (aid, aid)

    >>> found = client.getURLForAlias(aid, secure=True)

    >>> re.search(expected, found) is not None

    True

Librarian reads are logged in the request timeline.

    >>> from lazr.restful.utils import get_current_browser_request

    >>> from lp.services.timeline.requesttimeline import get_request_timeline

    >>> request = get_current_browser_request()

    >>> timeline = get_request_timeline(request)

    >>> f = client.getFileByAlias(aid)

    >>> action = timeline.actions[-1]

    >>> action.category

    'librarian-connection'

    >>> action.detail.endswith('/text.txt')

    True

    >>> _unused = f.read()

    >>> action = timeline.actions[-1]

    >>> action.category

    'librarian-read'

    >>> action.detail.endswith('/text.txt')

    True

At this level we can also reverse the transactional semantics by using

the remoteAddFile instead of the addFile method. In this case, the

database rows are added by the Librarian, which means that the file is

downloadable immediately and will exist even if the client transaction

rolls back. However, the records in the database will not be visible to

the client until it begins a new transaction.

    >>> url = client.remoteAddFile(

    ...     'text.txt', len(data), StringIO(data), 'text/plain')

    >>> print url

    http://.../text.txt

    >>> from urllib2 import urlopen

    >>> urlopen(url).read()

    'This is some data'

If we abort the transaction, it is still in there

    >>> transaction.abort()

    >>> urlopen(url).read()

    'This is some data'

You can also set the expiry date on the file this way too:

    >>> from datetime import date, datetime

    >>> from pytz import utc

    >>> url = client.remoteAddFile(

    ...     'text.txt', len(data), StringIO(data), 'text/plain',

    ...     expires=datetime(2005,9,1,12,0,0, tzinfo=utc))

    >>> transaction.abort()

To check the expiry is set, we need to extract the alias id from the

URL. remoteAddFile deliberatly returns the URL instead of the alias id

because, except for test cases, the URL is the only thing useful

(because the client can't see the database records yet).

    >>> import re

    >>> match = re.search('/(\d+)/', url)

    >>> alias_id = int(match.group(1))

    >>> alias = lfas[alias_id]

    >>> print alias.expires.isoformat()

    2005-09-01T12:00:00+00:00

Restricted Librarian

--------------------

Some files should not be generally available publicly. If you know the

URL, any file can be retrieved directly from the librarian. For this

reason, there is a restricted librarian to which access is restricted

(at the system-level). This means that only Launchpad has direct access

to the host. You use the IRestrictedLibrarianClient to access this

librarian.

    >>> from zope.interface.verify import verifyObject

    >>> from lp.services.librarian.interfaces.client import IRestrictedLibrarianClient

    >>> restricted_client = getUtility(IRestrictedLibrarianClient)

    >>> verifyObject(IRestrictedLibrarianClient, restricted_client)

    True

File alias uploaded through the restricted librarian have the restricted

attribute set.

    >>> private_content = 'This is private data.'

    >>> private_file_id = restricted_client.addFile(

    ...     'private.txt', len(private_content), StringIO(private_content),

    ...     'text/plain')

    >>> file_alias = getUtility(ILibraryFileAliasSet)[private_file_id]

    >>> file_alias.restricted

    True

    >>> transaction.commit()

    >>> file_alias.open()

    >>> print file_alias.read()

    This is private data.

    >>> file_alias.close()

Restricted files are accessible with HTTP on a private domain.

    >>> print file_alias.http_url

    http://.../private.txt

    >>> file_alias.http_url.startswith(

    ...     config.librarian.restricted_download_url)

    True

They can also be accessed externally using a time-limited token appended

to their private_url. Possession of a token is sufficient to grant

access to a file, regardless of who is logged in. getURL can be asked to

provide such a token.

    >>> token_url = file_alias.getURL(include_token=True)

    >>> print token_url

    https://i...restricted.../private.txt?token=...

    >>> token_url.startswith('https://i%d.restricted.' % file_alias.id)

    True

    >>> private_path = TimeLimitedToken.url_to_token_path(

    ...        file_alias.private_url)

    >>> token_url.endswith(session_store().find(

    ...     TimeLimitedToken, path=private_path).any().token)

    True

LibraryFileAliasView doesn't work on restricted files. This is a

temporary measure until we're sure no restricted files leak into the

traversal hierarchy.

    >>> view = getMultiAdapter((file_alias, request), name='+index')

    >>> view.initialize()

    Traceback (most recent call last):

    ...

    AssertionError

If you try to retrieve this file through the standard ILibrarianClient,

you'll get a DownloadFailed error.

    >>> client.getFileByAlias(private_file_id)

    Traceback (most recent call last):

      ...

    DownloadFailed: Alias ... cannot be downloaded from this client.

    >>> client.getURLForAlias(private_file_id)

    Traceback (most recent call last):

      ...

    DownloadFailed: Alias ... cannot be downloaded from this client.

But using the restricted librarian will work:

    >>> restricted_client.getFileByAlias(private_file_id)

    <lp.services.librarian.client._File...>

    >>> file_url = restricted_client.getURLForAlias(private_file_id)

    >>> print file_url

    http://.../private.txt

Trying to access that file directly on the normal librarian will fail

(by switching the port)

    >>> sneaky_url = file_url.replace(

    ...     config.librarian.restricted_download_url,

    ...     config.librarian.download_url)

    >>> urlopen(sneaky_url).read()

    Traceback (most recent call last):

      ...

    HTTPError: HTTP Error 404: Not Found

But downloading it from the restricted host, will work.

    >>> print urlopen(file_url).read()

    This is private data.

Trying to retrieve a non-restricted file from the restricted librarian

also fails:

    >>> public_content = 'This is public data.'

    >>> public_file_id = getUtility(ILibrarianClient).addFile(

    ...     'public.txt', len(public_content), StringIO(public_content),

    ...     'text/plain')

    >>> file_alias = getUtility(ILibraryFileAliasSet)[public_file_id]

    >>> file_alias.restricted

    False

    >>> transaction.commit()

    >>> restricted_client.getURLForAlias(public_file_id)

    Traceback (most recent call last):

      ...

    DownloadFailed: ...

    >>> restricted_client.getFileByAlias(public_file_id)

    Traceback (most recent call last):

      ...

    DownloadFailed: ...

The remoteAddFile() on the restricted client, also creates a restricted

file:

    >>> url = restricted_client.remoteAddFile(

    ...     'another-private.txt', len(private_content),

    ...     StringIO(private_content), 'text/plain')

    >>> print url

    http://.../another-private.txt

    >>> url.startswith(config.librarian.restricted_download_url)

    True

The file can then immediately be retrieved:

    >>> print urlopen(url).read()

    This is private data.

Another way to create a restricted file is by using the restricted

parameter to ILibraryFileAliasSet:

    >>> restricted_file = getUtility(ILibraryFileAliasSet).create(

    ...     'yet-another-private.txt', len(private_content),

    ...     StringIO(private_content), 'text/plain', restricted=True)

    >>> restricted_file.restricted

    True

Even if one has the SHA1 of the file, searching the librarian for it

will only return the file if it was in the same restriction space.

So searching for the private content on the public librarian will fail:

    >>> transaction.commit()

    >>> search_query = "search?digest=%s" % restricted_file.content.sha1

    >>> print urlopen(config.librarian.download_url + search_query).read()

    0

But on the restricted server, this will work:

    >>> result = urlopen(

    ...     config.librarian.restricted_download_url + search_query).read()

    >>> result = result.splitlines()

    >>> print result[0]

    3

    >>> sorted(file_path.split('/')[1] for file_path in result[1:])

    ['another-private.txt', 'private.txt', 'yet-another-private.txt']

Odds and Sods

-------------

An UploadFailed will be raised if you try to create a file with no

content

    >>> client.addFile('test.txt', 0, StringIO('hello'), 'text/plain')

    Traceback (most recent call last):

        [...]

    UploadFailed: Invalid length: 0

If you really want a zero length file you can do it:

    >>> aid = client.addFile(

    ...     'test.txt', 0, StringIO(''), 'text/plain', allow_zero_length=True)

    >>> transaction.commit()

    >>> f = client.getFileByAlias(aid)

    >>> f.read()

    ''

An AssertionError will be raised if the number of bytes that could be

read from the file don't match the declared size.

    >>> client.addFile('test.txt', 42, StringIO(''), 'text/plain')

    Traceback (most recent call last):

        [...]

    AssertionError: size is 42, but 0 were read from the file

Filenames with spaces in them work.

    >>> aid = client.addFile(

    ...     'hot dog', len(data), StringIO(data), 'text/plain')

    >>> transaction.commit()

    >>> f = client.getFileByAlias(aid)

    >>> f.read()

    'This is some data'

    >>> url = client.getURLForAlias(aid)

    >>> re.search(r'/\d+/hot%20dog$', url) is not None

    True

Unicode file names work.  Note that the filename in the resulting URL is

encoded as UTF-8.

    >>> aid = client.addFile(

    ...     u'Yow\N{INTERROBANG}', len(data), StringIO(data), 'text/plain')

    >>> transaction.commit()

    >>> f = client.getFileByAlias(aid)

    >>> f.read()

    'This is some data'

    >>> url = client.getURLForAlias(aid)

    >>> re.search(r'/\d+/Yow%E2%80%BD$', url) is not None

    True

Files will get garbage collected on production systems as per

LibrarianGarbageCollection. If you request the URL of a deleted file,

you will be given None

    >>> alias = lfas[36]

    >>> alias.deleted

    True

    >>> alias.http_url is None

    True

    >>> alias.https_url is None

    True

    >>> alias.getURL() is None

    True

    >>> client.getURLForAlias(alias.id) is None

    True

Default View

------------

A librarian file has a default view that should redirect to the download

URL.

    >>> from zope.component import getMultiAdapter

    >>> from lp.services.webapp.servers import LaunchpadTestRequest

    >>> req = LaunchpadTestRequest()

    >>> alias = lfas.create(

    ...     'text2.txt', len(data), StringIO(data), 'text/plain',

    ...     NEVER_EXPIRES)

    >>> transaction.commit()

    >>> lfa_view = getMultiAdapter((alias, req), name='+index')

    >>> lfa_view.initialize()

    >>> req.response.getHeader("Location") == alias.getURL()

    True

File views setup

----------------

We need some files to test different ways of accessing them.

    >>> filename = 'public.txt'

    >>> content = 'PUBLIC'

    >>> public_file = getUtility(ILibraryFileAliasSet).create(

    ...     filename, len(content), StringIO(content), 'text/plain',

    ...     NEVER_EXPIRES, restricted=False)

    >>> filename = 'restricted.txt'

    >>> content = 'RESTRICTED'

    >>> restricted_file = getUtility(ILibraryFileAliasSet).create(

    ...     filename, len(content), StringIO(content), 'text/plain',

    ...     NEVER_EXPIRES, restricted=True)

    # Create a new LibraryFileAlias not referencing any LibraryFileContent

    # record. Such records are considered as being deleted.

    >>> from lp.services.librarian.model import LibraryFileAlias

    >>> from lp.services.webapp.interfaces import (

    ...     IStoreSelector, MAIN_STORE, MASTER_FLAVOR)

    >>> store = getUtility(IStoreSelector).get(MAIN_STORE, MASTER_FLAVOR)

    >>> deleted_file = LibraryFileAlias(

    ...     content=None, filename='deleted.txt', mimetype='text/plain')

    >>> ignore = store.add(deleted_file)

Commit the just-created files.

    >>> from canonical.database.sqlbase import commit

    >>> commit()

    >>> deleted_file = getUtility(ILibraryFileAliasSet)[deleted_file.id]

    >>> print deleted_file.deleted

    True

Clear out existing tokens.

    >>> _ = session_store().find(TimeLimitedToken).remove()

LibraryFileAliasMD5View

-----------------------

The MD5 summary for a file can be downloaded. The text file contains the

hash and file name.

    >>> view = create_view(public_file, '+md5')

    >>> print view.render()

    cd0c6092d6a6874f379fe4827ed1db8b public.txt

    >>> print view.request.response.getHeader('Content-type')

    text/plain

Download counts

---------------

The download counts for librarian files are stored in the

LibraryFileDownloadCount table, broken down by day and country, but

there's also a 'hits' attribute on ILibraryFileAlias, which holds the

total number of times that file has been downloaded.

The count starts at 0, and cannot be changed directly.

    >>> public_file.hits

    0

    >>> public_file.hits = 10

    Traceback (most recent call last):

    ...

    ForbiddenAttribute: ...

To change that, we have to use the updateDownloadCount() method, which

takes care of creating/updating the necessary LibraryFileDownloadCount

entries.

    >>> from lp.services.worlddata.interfaces.country import ICountrySet

    >>> country_set = getUtility(ICountrySet)

    >>> november_1st_2006 = date(2006, 11, 1)

    >>> brazil = country_set['BR']

    >>> public_file.updateDownloadCount(november_1st_2006, brazil, count=1)

    >>> public_file.hits

    1

This was the first hit for that file from Brazil on 2006 November first,

so a new LibraryFileDownloadCount was created.

    >>> from lp.services.librarian.model import (

    ...        LibraryFileDownloadCount)

    >>> from storm.locals import Store

    >>> store = Store.of(public_file)

    >>> brazil_entry = store.find(

    ...     LibraryFileDownloadCount, libraryfilealias=public_file,

    ...     country=brazil, day=november_1st_2006).one()

    >>> brazil_entry.count

    1

Below we simulate a hit from Japan on that same day, which will also

create a new LibraryFileDownloadCount.

    >>> japan = country_set['JP']

    >>> public_file.updateDownloadCount(november_1st_2006, japan, count=3)

    >>> public_file.hits

    4

    >>> japan_entry = store.find(

    ...     LibraryFileDownloadCount, libraryfilealias=public_file,

    ...     country=japan, day=november_1st_2006).one()

    >>> japan_entry.count

    3

If there's another hit from Brazil on the same day, the existing entry

will be updated.

    >>> public_file.updateDownloadCount(november_1st_2006, brazil, count=2)

    >>> public_file.hits

    6

    >>> brazil_entry.count

    3

If the hit happened on a different day, a separate entry would be

created.

    >>> november_2nd_2006 = date(2006, 11, 2)

    >>> public_file.updateDownloadCount(november_2nd_2006, brazil, count=10)

    >>> public_file.hits

    16

    >>> brazil_entry2 = store.find(

    ...     LibraryFileDownloadCount, libraryfilealias=public_file,

    ...     country=brazil, day=november_2nd_2006).one()

    >>> brazil_entry2.count

    10

    >>> last_downloaded_date = november_2nd_2006

Time to last download

---------------------

The .last_downloaded property gives us the time delta from today to the

day that file was last downloaded, or None if it's never been

downloaded.

    >>> today = datetime.now(utc).date()

    >>> public_file.last_downloaded == today - last_downloaded_date

    True

    >>> content = 'something'

    >>> brand_new_file = getUtility(ILibraryFileAliasSet).create(

    ...     'new.txt', len(content), StringIO(content), 'text/plain',

    ...     NEVER_EXPIRES, restricted=False)

    >>> print brand_new_file.last_downloaded

    None