« back to all changes in this revision
Viewing changes to doc/buildout.txt
- browse files at revision 14649
- compare with another revision
- compare with revision 13845
- download diff
- download tarball
- view history from revision 14649
- Committer: Launchpad Patch Queue Manager
- Date: 2012-01-06 18:11:49 UTC
- mfrom: (14646.1.1 bug578854)
- Revision ID: launchpad@pqm.canonical.com-20120106181149-jr3427anqojecnjh
[r=sinzui][bug=578854] Update buildout docs for clarity
- files modified:
- doc/buildout.txt
added
removed
doc/buildout.txt
1
1
Launchpad Buildout
2
2
******************
3
3
4
Launchpad is moving to using the buildout_ (or "zc.buildout") build system. It
5
will gradually replace the sourcecode directory, and hopefully also the
6
Launchpad Makefile.
4
Launchpad uses the buildout_ (or "zc.buildout") build system.
5
Buildout's biggest strength is managing Python packages. That is also
6
our focus for it.
7
7
8
Buildout's biggest strength is managing Python packages. That will also be our
9
primary focus for it. Meanwhile, apt will continue to manage our Python
10
language installation, as well some or all of our non-Python dependencies, such
11
as PostgreSQL. While Bazaar will continue to be an essential part of our
12
toolchain, we will move away from using it to incorporate source code trees of
13
our dependencies.
8
We have at least two other ways of managing dependencies. Apt
9
manages our Python language installation, as well as many of our
10
non-Python system-level dependencies, such as PostgreSQL. The
11
sourcecode directory is our other way of managing dependencies. It is
12
supposed to only contain dependencies that are not standard Python
13
packages. bzr plugins and Javascript libraries are existing examples.
14
14
15
15
If you are not interested in our `Motivations`_ or in an `Introduction to
16
16
zc.buildout`_, all developers will at least want to read the very brief
36
36
* We want more careful specification of our dependencies across branches.
37
37
[INTERNAL] [EXTERNAL]
38
38
39
This is a real concern pertinent both for our "trunks" (devel, stable,
40
db-devel, db-stable) and for our development boxes. For instance, now, in our
41
trunks, when we want to update a dependency, we need to make sure that *all*
42
the current Launchpad trunks work with the dependency initially; then submit
43
a new Launchpad branch that uses the change dependency. A mistake can
44
even potentially break one or both of the db-* trunks, since PQM only tests
45
against one branch (usually devel), and sourcecode changes affect all
46
branches at once. For simplicity, speed, and safety, we want to be able to
47
submit a single branch that incorporates the source dependencies and the
48
associated changes at once.
39
This is a real concern pertinent both for our "trunks" (devel,
40
stable, db-devel, db-stable) and for our development boxes. For
41
instance, before incorporating buildout, in our trunks, when we want
42
to update a dependency, we needed to make sure that *all* the
43
current Launchpad trunks work with the dependency initially; then
44
submit a new Launchpad branch that uses the change dependency. A
45
mistake can even potentially break one or both of the db-* trunks,
46
since PQM only tests against one branch (usually devel), and
47
sourcecode changes affect all branches at once. For simplicity,
48
speed, and safety, we want to be able to submit a single branch that
49
incorporates the source dependencies and the associated changes at
50
once.
49
51
50
52
This is also true, if less critical and easier to work around, on developer
51
53
boxes. Without care, changes to sourcecode when working on dependencies will
58
60
A significant number of projects do not always have a pristine trunk, and
59
61
many also spend extra effort on polish, bug fixes, and compatibility before a
60
62
release. If we do not desperately need a new feature on trunk, using a
61
release is generally regarded as a safer, better practice. Our current usage
63
release is generally regarded as a safer, better practice. Our earlier usage
62
64
of bzr branches of the development trunks does not encourage this practice.
63
65
64
66
* We want to be encouraged to make the effort to interact with upstream
65
67
projects to have our patches integrated. [EXTERNAL]
66
68
67
Interacting and negotiating with upstream is undeniably more time-consuming
68
than our current practice of maintaining local bzr branches with our patches,
69
especially short-term. But our current practice is not good open-source
70
community behavior--an ironic characteristic for a project like Launchpad. It
71
also can cause problems down the road, for instance, if the patch becomes
69
Interacting and negotiating with upstream is undeniably more
70
time-consuming than our previous practice of maintaining local bzr
71
branches with our patches, especially short-term. But our previous
72
use of bzr branches is not good open-source community behavior--an
73
ironic characteristic for a project like Launchpad. It also can
74
cause problems down the road, for instance, if the patch becomes
72
75
stale and we want to migrate to new releases.
73
76
74
77
* We want to be protected from changes and differences in our operating system.
76
79
77
80
This is a concern both over time and across different Launchpad environments.
78
81
79
First, our operating system, Ubuntu, is driven by many needs and goals.
80
Launchpad is among them, but generally Launchpad serves Ubuntu, not the
81
reverse. For instance, recently Jaunty dropped Launchpad's Python version.
82
The Ubuntu developers had good reason--Python 2.4 has not been supported by
83
the Python developers for some time--but it caused a significant
84
inconvenience to the Launchpad team. Managing our dependencies, particularly
85
the Python library dependencies, can help alleviate these problems.
82
First, our operating system, Ubuntu, is driven by many needs and
83
goals. Launchpad is among them, but generally Launchpad serves
84
Ubuntu, not the reverse. For instance, Jaunty dropped Launchpad's
85
Python version. The Ubuntu developers had good reason--Python 2.4
86
has not been supported by the Python developers for some time--but
87
it caused a significant inconvenience to the Launchpad
88
team. Managing our dependencies, particularly the Python library
89
dependencies, can help alleviate these problems.
86
90
87
91
Second, Launchpad developers run a significantly different version of the
88
92
operating system than that run in production. Maintaining our Python library
420
424
421
425
./bin/buildout -v buildout:install-from-cache=false | tee out.txt | grep 'Picked'
422
426
423
The first part (``./bin/buildout -v buildout:install-from-cache=false``)
424
will run buildout, allowing it to download source packages from the
425
Internet to ``download-cache/dist``. The second part (``tee out.txt``) will
426
dump the full output to the ``out.txt`` file in case you need to debug a
427
problem. The last part (``grep 'Picked'``) will filter the output so that
428
only additional packages--dependencies of your dependency--will be listed.
429
Look at the output. You need to see if it means that you have dependencies,
430
some of which may be indirect dependencies. Here's an imaginary example
431
output::
427
The first part (``./bin/buildout -v
428
buildout:install-from-cache=false``) will run buildout, allowing
429
it to download source packages from the Internet to
430
``download-cache/dist``. The second part (``tee out.txt``) will
431
dump the full output to the ``out.txt`` file in case you need to
432
debug a problem. The last part (``grep 'Picked'``) will filter the
433
output so that only additional packages--dependencies of your
434
dependency--will be listed. You need to see if it means that you
435
have dependencies, some of which may be indirect
436
dependencies. We'll see how to do this with an example. Here's an
437
imaginary output::
432
438
433
439
Picked: ipython = 0.9.1
434
440
Picked: lazr.foom = 1.4
435
441
Picked: zope.bar = 3.6.1
436
442
Picked: z3c.shazam = 2.0.1
437
443
438
At this time, note that the output will include at least one, and possibly
439
more, spurious "Picked:" listings. ipython, in particular, shows up
440
repeatedly.
441
442
In our example, other than the spurious ``ipython`` listing, this means
443
that these packages have also been added to your ``download-cache/dist``
444
directory. You also need to add those versions to the ``versions.cfg``
445
file::
446
444
In our example, this means that these packages have been added to
445
your ``download-cache/dist`` directory. You now need to add those
446
versions to the ``versions.cfg`` file::
447
448
ipython = 0.9.1
447
449
lazr.foom = 1.4
448
450
zope.bar = 3.6.1
449
451
z3c.shazam = 2.0.1
450
452
453
Note that the output might include at least one, and possibly
454
more, spurious "Picked:" listings. ipython, in particular, has
455
shown up in the past incorrectly--that is, when you try to add the
456
file to the download-cache/dist directory, you'll discover that it
457
is already there; and you'll see that versions.cfg already
458
specifies the version. As long as the test passes (see step 5,
459
below), you can ignore this.
460
451
461
5. Test.
452
462
453
[TODO] Note that you can tell ec2test to include all uncommitted
454
distributions from the local download-cache in its tests. If you do this,
455
you cannot use the ec2test feature to submit on test success. Also, if you
456
have uncommitted distributions and you do *not* explicitly tell ec2test to
457
include or ignore the uncommitted distributions, it will refuse to start
458
an instance.
459
460
6. Check old versions in the download-cache. If you are sure that they are
461
not in use any more, *anywhere*, then remove them to save checkout space.
462
More explicitly, check with the LOSAs to see if they are in use in
463
production and send an email to launchpad-dev@lists.launchpad.net before
464
deleting anything if you are unsure.
465
466
7. Commit the changes (``cd download-cache``, ``bzr up``,
467
``bzr commit -m 'Add lazr.foom 1.1.2 and depdendencies to the download
468
cache'``) to the shared download cache when you are sure it is what you
469
want.
470
471
*Never* modify a package in the download-cache.
463
Note that you can tell ``ec2 test`` to include all uncommitted
464
distributions from the local download-cache in its tests with the
465
``--include-download-cache-changes`` flag (or ``-c``). If you do
466
this, you cannot use the ec2 test feature to submit on test
467
success. Also, if you have uncommitted distributions and you do
468
*not* explicitly tell ec2 test to include or ignore the
469
uncommitted distributions, it will refuse to start an instance.
470
471
6. Check old versions in the download-cache. If you are sure that
472
they are not in use any more, *anywhere*, then remove them to save
473
checkout space. More explicitly, check with the LOSAs to see if
474
they are in use in production and send an email to
475
launchpad-dev@lists.launchpad.net before deleting anything if you
476
are unsure. A rule of thumb is that it's worth starting this
477
investigation if the replacement has already been in use by the
478
Launchpad tree for more than a month. You can approximate this
479
information by using ``bzr log`` on the newer (replacement)
480
download-cache/dist file for the particular package.
481
482
7. Now you need to share your package changes with the rest of the
483
team. You must do this before submitting your Launchpad branch to
484
PQM or else your branch will not build properly anywhere else,
485
including buildbot. Commit the changes (``cd download-cache``,
486
bzr add the needed files, ``bzr up``, ``bzr commit -m 'Add
487
lazr.foom 1.1.2 and depdendencies to the download cache'``) to the
488
shared download cache when you are sure it is what you want.
489
490
*Never* modify a package in the download-cache. A change in code must mean a
491
change in version number, or else very bad inconsistencies and
492
confusion across build environments will happen.
472
493
473
494
474
495
Upgrade a Package
Loggerhead is a web-based interface for Breezy
Version: 2.0.1