= Hardware database = == Storage of raw submissions for the hardware database == The hardware database clients submits an XML file with collected hardware information and other data. This file is stored without any parsing in a librarian file. A reference to this file is stored in the table HWSubmission, together with the following data: >>> from lp.hardwaredb.interfaces.hwdb import IHWSubmissionSet >>> sample_submission = getUtility(IHWSubmissionSet).getBySubmissionKey( ... 'sample-submission') * reference to the librarian file. >>> print sample_submission.raw_submission <...LibraryFileAlias instance at ...> * date_created: Date and time of the submission (generated by the client). This value is not very reliable: * Some users do not care to set the correct time, date and time zone. * Some machines have a broken CMOS battery and the user does not set date and time after every boot. >>> print sample_submission.date_created 2008-09-26 17:19:18+00:00 * date_submitted: Date and time of the submission (generated by the server). >>> print sample_submission.date_submitted 2008-09-30 08:19:00.222131+00:00 * format: The format version of the submitted data, as given by the HWDB client. See HWSubmissionFormat for valid values. >>> print sample_submission.format.name VERSION_1 * status: The status of the submission. See HWSubmissionProcessingStatus for valid values. >>> print sample_submission.status.name PROCESSED * private: If True, the submitter allows public access to the data. If false, the data may be used only for statistical purposes. >>> print sample_submission.private False * contactable: If True, the submitter agrees to be contacted by upstream developers and package maintainers for tests etc. >>> print sample_submission.contactable False * submission_key: A unique submission ID. >>> print sample_submission.submission_key sample-submission * raw_emailaddress: The email address of the submitter. >>> print sample_submission.raw_emailaddress test@canonical.com * owner: A reference to the Person table. This value is null, if the email address does not belong (yet) to an entry in Person table. >>> print sample_submission.owner.displayname Sample Person * distroarchseries: A reference to the distroarchseries of the submission. This value is null, if the submitted values for distribution, distroseries and architecture do not match an existing entry in the Distroarchrelease table. >>> print sample_submission.distroarchseries.architecturetag i386 >>> print sample_submission.distroarchseries.distroseries.name hoary >>> print sample_submission.distroarchseries.distroseries.distribution.name ubuntu * system_fingerprint: A reference to an entry of the HWSystemFingerprint table. This table stores the system name as returned by HAL (system.vendor, system.product). >>> print sample_submission.system_fingerprint.fingerprint MSI MS-7369 * devices: The set set of HWSubmissionDevice records which belong to this submission. Access to all data except submissions is restricted to members of the Canonical team. >>> login('foo.bar@canonical.com') >>> for submission_device in sample_submission.devices[:2]: ... print submission_device.device.name MCP65 USB Controller MCP65 SATA Controller >>> login(ANONYMOUS) Limitations: * "No name" products like mainboards from companies like ASRock or Asus that are directly sold to end users have fingerprints like "American Megatrends Inc. Uknown 1.0". * A manufacturer may erroneously assign identical DMI values for product and vendor to different systems. * submissions for "counterfeit systems". * users submitting bogus values using a patched HWDB client. The system fingerprint is stored in a separate table, HWSystemFingerprint. IHWSystemFingerPrintSet.createFingerprint creates a new row in this table. >>> from lp.hardwaredb.interfaces.hwdb import IHWSystemFingerprintSet >>> from storm.store import Store >>> hw_fingerprint_set = getUtility(IHWSystemFingerprintSet) >>> fp = hw_fingerprint_set.createFingerprint(u'IBM T41') >>> fp.fingerprint u'IBM T41' Each fingerprint string in HWSystemFingerprint must be unique. >>> import transaction >>> transaction.commit() >>> fp = hw_fingerprint_set.createFingerprint(u'IBM T41') >>> Store.of(fp).flush() Traceback (most recent call last): ... IntegrityError: ... violates unique constraint "hwsystemfingerprint... >>> transaction.abort() IHWSystemFingerprint.getByName returns a fingerprint... >>> fp = hw_fingerprint_set.getByName(u'IBM T41') >>> fp.fingerprint u'IBM T41' ...or None, if no entry exists. >>> print hw_fingerprint_set.getByName(u'DEC PDP11') None IHWSubmissionSet.createSubmission creates a new entry in the HWSubmission table. >>> import pytz >>> from StringIO import StringIO >>> from datetime import datetime >>> from lp.hardwaredb.interfaces.hwdb import ( ... HWSubmissionFormat, ... IHWSubmissionSet, ... ) >>> date_created = datetime(2007, 04, 01, tzinfo=pytz.timezone('UTC')) >>> date_submitted = datetime(2007, 04, 02, tzinfo=pytz.timezone('UTC')) >>> submission_data = 'submission data' >>> hw_submission_set = getUtility(IHWSubmissionSet) >>> submission = hw_submission_set.createSubmission( ... date_created=date_created, ... format=HWSubmissionFormat.VERSION_1, ... private=False, ... contactable=False, ... submission_key=u'unique-id-1', ... emailaddress=u'test@canonical.com', ... distroarchseries=None, ... raw_submission=StringIO(submission_data), ... filename=u'hwinfo.txt', ... filesize=len(submission_data), ... system_fingerprint=u'Dell Inspiron 1234') submission_key must be unique. The attempt to create a HWSubmission entry with an already existing submission_key raises HWSubmissionError. >>> submission = hw_submission_set.createSubmission( ... date_created=date_created, ... format=HWSubmissionFormat.VERSION_1, ... private=False, ... contactable=False, ... submission_key=u'unique-id-1', ... emailaddress=u'test@canonical.com', ... distroarchseries=None, ... raw_submission=StringIO(submission_data), ... filename=u'hwinfo.txt', ... filesize=len(submission_data), ... system_fingerprint=u'Dell Inspiron 1234') Traceback (most recent call last): ... HWSubmissionKeyNotUnique: A submission with this ID already exists The field `owner` points to the Person record of the submitter, if the submitted emailaddress belongs to an existing Launchpad account. >>> submission.owner.displayname u'Sample Person' If no Person record is yet known for the submitter's email address, the field `owner` is None. >>> from lp.registry.interfaces.person import IPersonSet >>> beeblebrox = getUtility(IPersonSet).getByEmail( ... 'beeblebrox@example.com') >>> print beeblebrox None >>> submission = hw_submission_set.createSubmission( ... date_created=date_created, ... format=HWSubmissionFormat.VERSION_1, ... private=False, ... contactable=False, ... submission_key=u'unique-id-2', ... emailaddress=u'beeblebrox@example.com', ... distroarchseries=None, ... raw_submission=StringIO(submission_data), ... filename=u'hwinfo.txt', ... filesize=len(submission_data), ... system_fingerprint=u'Acer 6789') >>> print submission.owner None The submitted data can be retrieved via IHWSubmissionSet.getBySubmissionKey. >>> submission = hw_submission_set.getBySubmissionKey(u'unique-id-1') >>> submission.submission_key, submission.system_fingerprint.fingerprint (u'unique-id-1', u'Dell Inspiron 1234') If a submission is marked as private, it is only returned by IHWSubmissionSet.getBySubmissionKey if the parameter `user` matches the owner of the submission... >>> submission = hw_submission_set.createSubmission( ... date_created=date_created, ... format=HWSubmissionFormat.VERSION_1, ... private=True, ... contactable=False, ... submission_key=u'private-submission', ... emailaddress=u'test@canonical.com', ... distroarchseries=None, ... raw_submission=StringIO(submission_data), ... filename=u'hwinfo.txt', ... filesize=len(submission_data), ... system_fingerprint=u'Dell Inspiron 1234') >>> print hw_submission_set.getBySubmissionKey(u'private-submission') None >>> sample_person = getUtility(IPersonSet).getByEmail('test@canonical.com') >>> sample_person.displayname u'Sample Person' >>> submission = hw_submission_set.getBySubmissionKey( ... u'private-submission', sample_person) >>> submission ... or if `user` is an admin. >>> foobar = getUtility(IPersonSet).getByEmail('foo.bar@canonical.com') >>> submission = hw_submission_set.getBySubmissionKey( ... u'private-submission', foobar) >>> submission Only the owner and admins can view the submission details. >>> submission.submission_key, submission.system_fingerprint.fingerprint Traceback (most recent call last): ... Unauthorized: (...'submission_key', 'launchpad.View') >>> login('test@canonical.com') >>> submission.submission_key, submission.system_fingerprint.fingerprint (u'private-submission', u'Dell Inspiron 1234') >>> login('foo.bar@canonical.com') >>> submission.submission_key, submission.system_fingerprint.fingerprint (u'private-submission', u'Dell Inspiron 1234') Passing another person than the owner or an admin as the parameter `user` does not return private submissions. >>> no_priv = getUtility(IPersonSet).getByEmail('no-priv@canonical.com') >>> no_priv.displayname u'No Privileges Person' >>> print hw_submission_set.getBySubmissionKey(u'private-submission', ... no_priv) None >>> login(ANONYMOUS) If no submission exists for a given ID, None is returned. >>> print hw_submission_set.getBySubmissionKey(u'unknown-ID') None IHWSubmissionSet.getByFingerprintName returns all submissions for a given fingerprint name. Private submissions are excluded, if the parameter `user` is omitted or if user is neither the owner nor an admin. >>> submissions = hw_submission_set.getByFingerprintName( ... u'Dell Inspiron 1234') >>> [(s.submission_key, s.system_fingerprint.fingerprint, s.private) ... for s in submissions] [(u'unique-id-1', u'Dell Inspiron 1234', False)] >>> submissions = hw_submission_set.getByFingerprintName( ... u'Dell Inspiron 1234', no_priv) >>> [(s.submission_key, s.system_fingerprint.fingerprint, s.private) ... for s in submissions] [(u'unique-id-1', u'Dell Inspiron 1234', False)] >>> login('test@canonical.com') >>> submissions = hw_submission_set.getByFingerprintName( ... u'Dell Inspiron 1234', sample_person) >>> [(s.submission_key, s.system_fingerprint.fingerprint, s.private) ... for s in submissions] [(u'private-submission', u'Dell Inspiron 1234', True), (u'unique-id-1', u'Dell Inspiron 1234', False)] >>> submissions = hw_submission_set.getByFingerprintName( ... u'Dell Inspiron 1234', foobar) >>> [(s.submission_key, s.system_fingerprint.fingerprint, s.private) ... for s in submissions] [(u'private-submission', u'Dell Inspiron 1234', True), (u'unique-id-1', u'Dell Inspiron 1234', False)] If no submissions exist for the given system fingerprint, an empty result set is returned. >>> list(hw_submission_set.getByFingerprintName('DEC PDP11')) [] IHWSubmissionSet.getByOwner returns all submissions of a person. Again, only non-private submissions are returned if no parameter `user` is passed to the method, or if the passed parameter is neither the owner nor an admin. >>> submissions = hw_submission_set.getByOwner(sample_person) >>> [(s.submission_key, s.system_fingerprint.fingerprint) ... for s in submissions] [(u'unique-id-1', u'Dell Inspiron 1234'), (u'sample-submission', u'MSI MS-7369'), (u'test_submission_id_1', u'TONKA TUFFBOOK2600')] >>> submissions = hw_submission_set.getByOwner(sample_person, ... user=no_priv) >>> [(s.submission_key, s.system_fingerprint.fingerprint) ... for s in submissions] [(u'unique-id-1', u'Dell Inspiron 1234'), (u'sample-submission', u'MSI MS-7369'), (u'test_submission_id_1', u'TONKA TUFFBOOK2600')] >>> submissions = hw_submission_set.getByOwner(sample_person, ... user=sample_person) >>> [(s.submission_key, s.system_fingerprint.fingerprint) ... for s in submissions] [(u'private-submission', u'Dell Inspiron 1234'), (u'unique-id-1', u'Dell Inspiron 1234'), (u'sample-submission', u'MSI MS-7369'), (u'test_submission_id_1', u'TONKA TUFFBOOK2600')] >>> submissions = hw_submission_set.getByOwner(sample_person, ... user=foobar) >>> [(s.submission_key, s.system_fingerprint.fingerprint) ... for s in submissions] [(u'private-submission', u'Dell Inspiron 1234'), (u'unique-id-1', u'Dell Inspiron 1234'), (u'sample-submission', u'MSI MS-7369'), (u'test_submission_id_1', u'TONKA TUFFBOOK2600')] When an anonymous submitter later registers a Launchpad account, the field `owner` is updated. >>> from lp.services.identity.interfaces.emailaddress import IEmailAddressSet >>> from lp.registry.interfaces.person import PersonCreationRationale >>> user = getUtility(IPersonSet).ensurePerson( ... u'beeblebrox@example.com', u'Beeblebrox', ... PersonCreationRationale.OWNER_CREATED_LAUNCHPAD) >>> email = getUtility(IEmailAddressSet).getByEmail( ... u'beeblebrox@example.com') >>> user.validateAndEnsurePreferredEmail(email) >>> transaction.commit() >>> from lp.scripts.garbo import HWSubmissionEmailLinker >>> from lp.services.log.logger import DevNullLogger >>> HWSubmissionEmailLinker(log=DevNullLogger()).run() >>> submission = hw_submission_set.getBySubmissionKey(u'unique-id-2') >>> print submission.owner.displayname Beeblebrox If somebody has a Launchpad account, but submits HWDB test data using an email address which he has not added to his account, the submission will have the `owner` field set to None. >>> submission = hw_submission_set.createSubmission( ... date_created=date_created, ... format=HWSubmissionFormat.VERSION_1, ... private=False, ... contactable=False, ... submission_key=u'unique-id-4', ... emailaddress=u'beeblebrox@beeblebrox.net', ... distroarchseries=None, ... raw_submission=StringIO(submission_data), ... filename=u'hwinfo.txt', ... filesize=len(submission_data), ... system_fingerprint=u'Acer 6789') >>> Store.of(submission).flush() >>> print submission.owner None When he adds this email address to his list of addresses in Launchpad, the field `owner` is updated. >>> email = getUtility(IEmailAddressSet).new( ... u'beeblebrox@beeblebrox.net', user, account=user.account) >>> transaction.commit() >>> login_person(user) >>> user.validateAndEnsurePreferredEmail(email) >>> transaction.commit() >>> HWSubmissionEmailLinker(log=DevNullLogger()).run() >>> submission = hw_submission_set.getBySubmissionKey(u'unique-id-2') >>> print submission.owner.displayname Beeblebrox === Submission Records without an email address === A submission does not necessarily have an email address (anonymous submission). In this case the owner is always None. >>> submission_no_owner = hw_submission_set.createSubmission( ... date_created=date_created, ... format=HWSubmissionFormat.VERSION_1, ... private=False, ... contactable=False, ... submission_key=u'unique-id-5', ... emailaddress=None, ... distroarchseries=None, ... raw_submission=StringIO(submission_data), ... filename=u'hwinfo.txt', ... filesize=len(submission_data), ... system_fingerprint=u'Dell Inspiron 1234') >>> Store.of(submission_no_owner).flush() >>> print submission_no_owner.raw_emailaddress None >>> print submission_no_owner.owner None == Retrieving Submissions by Status == The set of submissions with a given status can be retrieved by IHWSubmissionSet.getByStatus(). Only the publicly visible submissions are returned if no user is specified. >>> from lp.hardwaredb.interfaces.hwdb import ( ... HWSubmissionProcessingStatus) >>> def print_submissions(submissions): ... for submission in submissions: ... print submission.submission_key, submission.private, ... if submission.owner is not None: ... print submission.owner.displayname, ... else: ... print '(no owner)', ... print submission.status.name >>> new_submissions = hw_submission_set.getByStatus( ... HWSubmissionProcessingStatus.SUBMITTED) >>> print_submissions(new_submissions) test_submission_id_1 False Sample Person SUBMITTED unique-id-1 False Sample Person SUBMITTED unique-id-2 False Beeblebrox SUBMITTED unique-id-4 False Beeblebrox SUBMITTED unique-id-5 False (no owner) SUBMITTED If a user is passed, private submissions of this user are returned too. >>> login('test@canonical.com') >>> from lp.hardwaredb.interfaces.hwdb import ( ... HWSubmissionProcessingStatus) >>> new_submissions = hw_submission_set.getByStatus( ... HWSubmissionProcessingStatus.SUBMITTED, user=sample_person) >>> print_submissions(new_submissions) test_submission_id_1 False Sample Person SUBMITTED unique-id-1 False Sample Person SUBMITTED unique-id-2 False Beeblebrox SUBMITTED private-submission True Sample Person SUBMITTED unique-id-4 False Beeblebrox SUBMITTED unique-id-5 False (no owner) SUBMITTED Submissions of other users are not returned. >>> from lp.hardwaredb.interfaces.hwdb import ( ... HWSubmissionProcessingStatus) >>> new_submissions = hw_submission_set.getByStatus( ... HWSubmissionProcessingStatus.SUBMITTED, user=no_priv) >>> print_submissions(new_submissions) test_submission_id_1 False Sample Person SUBMITTED unique-id-1 False Sample Person SUBMITTED unique-id-2 False Beeblebrox SUBMITTED unique-id-4 False Beeblebrox SUBMITTED unique-id-5 False (no owner) SUBMITTED For admins, IHWSubmissionSet.getByStatus() returns private and public submissions... >>> foobar = getUtility(IPersonSet).getByEmail('foo.bar@canonical.com') >>> login('foo.bar@canonical.com') >>> from lp.hardwaredb.interfaces.hwdb import ( ... HWSubmissionProcessingStatus) >>> new_submissions = hw_submission_set.getByStatus( ... HWSubmissionProcessingStatus.SUBMITTED, user=foobar) >>> print_submissions(new_submissions) test_submission_id_1 False Sample Person SUBMITTED unique-id-1 False Sample Person SUBMITTED unique-id-2 False Beeblebrox SUBMITTED private-submission True Sample Person SUBMITTED unique-id-4 False Beeblebrox SUBMITTED unique-id-5 False (no owner) SUBMITTED ...as well as for the janitor. >>> from lp.app.interfaces.launchpad import ILaunchpadCelebrities >>> janitor = getUtility(ILaunchpadCelebrities).janitor >>> new_submissions = hw_submission_set.getByStatus( ... HWSubmissionProcessingStatus.SUBMITTED, user=janitor) >>> print_submissions(new_submissions) test_submission_id_1 False Sample Person SUBMITTED unique-id-1 False Sample Person SUBMITTED unique-id-2 False Beeblebrox SUBMITTED private-submission True Sample Person SUBMITTED unique-id-4 False Beeblebrox SUBMITTED unique-id-5 False (no owner) SUBMITTED Searching for other statuses is possible too. >>> submission_no_owner.status = HWSubmissionProcessingStatus.PROCESSED >>> processed_submissions = hw_submission_set.getByStatus( ... HWSubmissionProcessingStatus.PROCESSED, user=foobar) >>> print_submissions(processed_submissions) sample-submission False Sample Person PROCESSED unique-id-5 False (no owner) PROCESSED == Links to Bugs == HWDB submissions can be linked to bugs. These links are created by IHWSubmissionBugSet.create(). >>> from lp.bugs.interfaces.bug import IBugSet >>> from lp.hardwaredb.interfaces.hwdb import IHWSubmissionBugSet >>> bug_one = getUtility(IBugSet).get(1) >>> submission_bug_set = getUtility(IHWSubmissionBugSet) >>> submission_bug = submission_bug_set.create( ... submission_no_owner, bug_one) >>> transaction.commit() >>> print submission_bug.submission.submission_key unique-id-5 >>> submission_bug.bug.id 1 Links between a bug and a submission are unique. If we call IHWSubmissionBugSet.create() twice for the same bug and HWDB submission, the method returns the existing record. >>> second_link = submission_bug_set.create(submission_no_owner, bug_one) >>> second_link.id == submission_bug.id True HWSubmissionBugSet.submissionsForBug() returns all submissions linked to a given bug. >>> for submission in submission_bug_set.submissionsForBug(bug_one): ... print submission.submission_key unique-id-5 Private submissions are only returned if the current user is an admin or the owner of the private submission. >>> submission = hw_submission_set.getBySubmissionKey( ... u'private-submission', sample_person) >>> submission_bug_set.create(submission, bug_one) >> for submission in submission_bug_set.submissionsForBug( ... bug_one, user=sample_person): ... print submission.submission_key private-submission unique-id-5 >>> for submission in submission_bug_set.submissionsForBug( ... bug_one, user=foobar): ... print submission.submission_key private-submission unique-id-5 Other registered persons can see only public submissions... >>> for submission in submission_bug_set.submissionsForBug( ... bug_one, user=no_priv): ... print submission.submission_key unique-id-5 ...as well as anonymous users. >>> for submission in submission_bug_set.submissionsForBug( ... bug_one): ... print submission.submission_key unique-id-5 Existing links can also be removed. >>> submission_bug_set.remove(submission_no_owner, bug_one) >>> submission_bug_set.submissionsForBug(bug_one).count() 0 >>> for submission in submission_bug_set.submissionsForBug( ... bug_one, user=foobar): ... print submission.submission_key private-submission == General Search Method == IHWSubmissionSet.search() allows the look up of submissions which contain a specfic device, a specific driver or which are made for a specific distribution or processor architecture. All parameters are optional; if none are given, all publicly visible submissions are returned. >>> for submission in hw_submission_set.search(): ... print submission.submission_key test_submission_id_1 sample-submission unique-id-1 unique-id-2 unique-id-4 unique-id-5 If the parameter `user` is specified, private submissions from this user are returned too. >>> for submission in hw_submission_set.search(user=sample_person): ... print submission.submission_key test_submission_id_1 sample-submission unique-id-1 unique-id-2 private-submission unique-id-4 unique-id-5 Admins have access to all submissions. >>> for submission in hw_submission_set.search(user=foobar): ... print submission.submission_key test_submission_id_1 sample-submission unique-id-1 unique-id-2 private-submission unique-id-4 unique-id-5 If a distribution is specified, only those submissions that were made for that distribution are returned. >>> from lp.registry.interfaces.distribution import ( ... IDistributionSet) >>> distribution_set = getUtility(IDistributionSet) >>> ubuntu = distribution_set.getByName('ubuntu') >>> for submission in hw_submission_set.search(distribution=ubuntu): ... print submission.submission_key sample-submission >>> debian = distribution_set.getByName('debian') >>> print hw_submission_set.search(distribution=debian).count() 0 If an architecture is specified, only those submissions that were made for that architecture are returned. >>> for submission in hw_submission_set.search(architecture='i386'): ... print submission.submission_key sample-submission >>> print hw_submission_set.search(architecture='amd64').count() 0 If a device is specified, only those submissions that include the device are returned. >>> from lp.hardwaredb.interfaces.hwdb import HWBus, IHWDeviceSet >>> device_set = getUtility(IHWDeviceSet) >>> msi_mainboard = device_set.getByDeviceID( ... HWBus.SYSTEM, 'MSI', 'MS-7369') >>> for submission in hw_submission_set.search(device=msi_mainboard): ... print submission.submission_key sample-submission If a driver is specified, only those submissions for that driver are returned. >>> from lp.hardwaredb.interfaces.hwdb import IHWDriverSet >>> driver_set = getUtility(IHWDriverSet) >>> ehci_driver = driver_set.getByPackageAndName( ... 'linux-image-2.6.24-19-generic', 'ehci_hcd') >>> for submission in hw_submission_set.search(driver=ehci_driver): ... print submission.submission_key sample-submission If a distroseries is specified, only submissions for that distroseries are returned. >>> from lp.testing.factory import LaunchpadObjectFactory >>> factory = LaunchpadObjectFactory() >>> warty = ubuntu['warty'] >>> warty_arch_series = factory.makeDistroArchSeries(distroseries=warty) >>> submission = factory.makeHWSubmission(distroarchseries=warty_arch_series) >>> for submission in hw_submission_set.search(distroseries=warty): ... print submission.distroarchseries.distroseries.name warty It is also possible to search for a distroseries and architecture. >>> from lp.soyuz.interfaces.processor import IProcessorFamilySet >>> amd64 = getUtility(IProcessorFamilySet).getByName('amd64') >>> warty_amd64 = factory.makeDistroArchSeries( ... distroseries=warty, architecturetag='amd64', ... processorfamily=amd64) >>> submission = factory.makeHWSubmission(distroarchseries=warty_amd64) >>> for submission in hw_submission_set.search(distroseries=warty, architecture='amd64'): ... print '%s %s ' % ( ... submission.distroarchseries.distroseries.name, ... submission.distroarchseries.architecturetag) warty amd64 And we can also search for submissions from a particular user. >>> from lp.registry.interfaces.person import IPersonSet >>> owner = getUtility(IPersonSet).getByName('name12') >>> set(submission.owner.name for submission ... in hw_submission_set.search(owner=owner)) set([u'name12']) Alternatively, we can get the same result by looking at IPerson.hardware_submissions. >>> set(submission.owner.name for submission ... in owner.hardware_submissions) set([u'name12'])