~launchpad-pqm/launchpad/devel

8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
1
Launchpad Buildout
2
******************
3
4
Launchpad is moving to using the buildout_ (or "zc.buildout") build system.  It
5
will gradually replace the sourcecode directory, and hopefully also the
6
Launchpad Makefile.
7
8
Buildout's biggest strength is managing Python packages.  That will also be our
9
primary focus for it.  Meanwhile, apt will continue to manage our Python
10
language installation, as well some or all of our non-Python dependencies, such
11
as PostgreSQL.  While Bazaar will continue to be an essential part of our
12
toolchain, we will move away from using it to incorporate source code trees of
13
our dependencies.
14
15
If you are not interested in our `Motivations`_  or in an `Introduction to
16
zc.buildout`_, all developers will at least want to read the very brief
17
sections about the `Set Up`_ and the `Everyday Usage`_.
18
19
Developers who manage source dependencies probably should read the general
20
information about `Managing Dependencies and Scripts`_, but will also find
21
detailed instructions to `Add a Package`_, to `Upgrade a Package`_, to `Add a
22
Script`_, to `Add a File Modified By Buildout`_, and to `Work with Unreleased
23
or Forked Packages`_.
24
25
.. _buildout: http://www.buildout.org/
26
27
===========
28
Motivations
29
===========
30
31
These motivations are labeled as "[INTERNAL]" or "[EXTERNAL]" to indicate
32
whether it is pertinent for internal dependencies, which we on the Launchpad
33
team create and release ourselves; or external dependencies, which other
9012.1.3 by Maris Fogels
Added a bit to the buildout docs for using the filetemplate recipe.
34
parties, in and out of Canonical, create and release.
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
35
36
* We want more careful specification of our dependencies across branches.
37
  [INTERNAL] [EXTERNAL]
38
39
  This is a real concern pertinent both for our "trunks" (devel, stable,
40
  db-devel, db-stable) and for our development boxes. For instance, now, in our
41
  trunks, when we want to update a dependency, we need to make sure that *all*
42
  the current Launchpad trunks work with the dependency initially; then submit
43
  a new Launchpad branch that uses the change dependency.  A mistake can
44
  even potentially break one or both of the db-* trunks, since PQM only tests
45
  against one branch (usually devel), and sourcecode changes affect all
46
  branches at once.  For simplicity, speed, and safety, we want to be able to
47
  submit a single branch that incorporates the source dependencies and the
48
  associated changes at once.
49
50
  This is also true, if less critical and easier to work around, on developer
51
  boxes.  Without care, changes to sourcecode when working on dependencies will
52
  affect all a developer's branches, polluting test results with false
53
  negatives or false positives.
54
55
* We want to default to using released versions of our software dependencies.
56
  [EXTERNAL]
57
58
  A significant number of projects do not always have a pristine trunk, and
59
  many also spend extra effort on polish, bug fixes, and compatibility before a
60
  release.  If we do not desperately need a new feature on trunk, using a
61
  release is generally regarded as a safer, better practice. Our current usage
62
  of bzr branches of the development trunks does not encourage this practice.
63
64
* We want to be encouraged to make the effort to interact with upstream
65
  projects to have our patches integrated. [EXTERNAL]
9012.1.3 by Maris Fogels
Added a bit to the buildout docs for using the filetemplate recipe.
66
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
67
  Interacting and negotiating with upstream is undeniably more time-consuming
68
  than our current practice of maintaining local bzr branches with our patches,
69
  especially short-term.  But our current practice is not good open-source
70
  community behavior--an ironic characteristic for a project like Launchpad. It
71
  also can cause problems down the road, for instance, if the patch becomes
72
  stale and we want to migrate to new releases.
73
74
* We want to be protected from changes and differences in our operating system.
75
  [INTERNAL] [EXTERNAL]
76
77
  This is a concern both over time and across different Launchpad environments.
78
79
  First, our operating system, Ubuntu, is driven by many needs and goals.
80
  Launchpad is among them, but generally Launchpad serves Ubuntu, not the
81
  reverse.  For instance, recently Jaunty dropped Launchpad's Python version.
82
  The Ubuntu developers had good reason--Python 2.4 has not been supported by
83
  the Python developers for some time--but it caused a significant
84
  inconvenience to the Launchpad team. Managing our dependencies, particularly
85
  the Python library dependencies, can help alleviate these problems.
86
87
  Second, Launchpad developers run a significantly different version of the
88
  operating system than that run in production. Maintaining our Python library
89
  dependencies ourselves can also help alleviate these concerns.
90
91
* We want to be able to easily use standard packages of our primary
92
  programming language, Python. [EXTERNAL]
93
94
  Our Python library dependencies are distributed for many operating systems--
95
  Windows, Mac, and other flavors of Linux--in a unified location and format:
96
  PyPI, using distutils.  Using Python library dependencies in their standard
97
  distributions makes it easier for us to reuse code.
98
99
* We want to be encouraged to release Python packages of our open-source
100
  code. [INTERNAL]
101
102
  We are beginning to realize our aspirations of abstracting and releasing some
103
  of our code.  Much of that code is in Python.  Consuming standard Python
104
  packages encourages us to follow good practice in releasing our own Python
105
  packages.  Dogfooding can help keep us honest, and good official releases can
106
  help us support and encourage a community of users.
107
108
Meanwhile, we want to be able to keep certain aspects of the legacy story.
109
110
* We need deployment to not need network access.
111
112
* We need to be able to use custom releases of our Python dependencies, if
113
  absolutely necessary.
114
115
* We would like to be able to follow security releases in our operating system.
116
117
We will be able to support the first two of these aspects entirely.
118
119
As to the third, we will continue to follow operating system security releases
120
for most or all of the dependencies that are not Python packages--a very
121
similar situation to the one before Buildout integration.
122
123
===========================
124
Introduction to zc.buildout
125
===========================
126
127
Buildout is a relatively simple system that increases in complexity as it is
128
extended via "recipes".  It works on top of distutils_ and setuptools_.  It
129
uses declarative ini-style files to direct its work.  The `Buildout site`_
130
points to a variety of documents describing and documenting its use.
131
132
For Launchpad, Buildout is hidden behind a Makefile as of this writing.  If
133
the Makefile is removed, developers will typically run ``bootstrap.py`` in the
134
branch, and then run ``bin/buildout``.  After that, the ``bin`` directory will
135
have the commands to start, stop, test, and debug the software.  If you are
136
interested in example direct usage of Buildout, you may want to read `the
137
"Hacking" document in the Launchpad wiki`_ that describes the usage patterns in
138
``lazr.*`` packages.
139
140
.. _`the "Hacking" document in the Launchpad wiki`: https://dev.launchpad.net/Hacking
141
142
Launchpad's Buildout usage is roughly of medium complexity.  It is more
143
complex than that needed by a package with few dependencies and simple usage
144
(see lazr.uri, for instance), but less complex than that of other large
145
applications that use the buildout system.  More complexity can come by
146
building more non-Python tools and by having multiple configuration variations,
147
for instance.
148
149
The documentation below will focus on using Launchpad's buildout.  See the
150
links given above for a more thorough general review.
151
152
.. _distutils: http://docs.python.org/distutils/index.html
153
154
.. _setuptools: http://peak.telecommunity.com/DevCenter/setuptools
155
156
.. _`Buildout site`: http://www.buildout.org/
157
158
======
159
Set Up
160
======
161
162
If you use the ``rocketfuel-get`` script, run that, and you will be done.
163
164
If you don't, you have just a bit more initial set up.  I'll assume you
165
maintain a pattern similar to what the ``rocketfuel-*`` scripts use: you have a
166
local, pristine branch of trunk from which you make your other branches.  You
8329.1.7 by Gary Poster
convert layout to have eggs and download-cache on top; and add make target for eggs folder, for deployment.
167
manually update the trunk and rsync sourcecode when necessary.  When you make
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
168
a branch, you use ``utilities/link-external-sourcecode``.
169
170
Developers that take this approach should do the following, where ``trunk`` is
171
the trunk branch from which you make local branches.
172
173
::
174
175
    bzr co lp:lp-source-dependencies download-cache
176
8329.1.7 by Gary Poster
convert layout to have eggs and download-cache on top; and add make target for eggs folder, for deployment.
177
Then run ``make`` in the trunk.
178
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
179
See the `Everyday Usage: Manual`_ section for further instructions on how to
180
work without the ``rocketfuel-*`` scripts.
181
182
.. _`Everyday Usage: Manual`: Manual_
183
184
==============
185
Everyday Usage
186
==============
187
188
``rocketfuel`` Scripts
189
======================
190
191
If you typically use ``rocketfuel-get``, and you don't change source
192
dependencies, you should not have any further changes, except that ``bin/test``
193
has replaced ``test.py``.  ``rocketfuel-branch`` and
194
``link-external-dependencies`` will Do the Right Thing.
195
196
Manual
197
======
198
199
If you don't use the rocketfuel scripts, you will still use
200
``link-external-dependencies`` as before.  When a buildout complains that it
201
cannot find a version of a dependency, do the following, from within the
202
branch::
203
8329.1.7 by Gary Poster
convert layout to have eggs and download-cache on top; and add make target for eggs folder, for deployment.
204
    bzr up download-cache
205
206
After this, retry your make (or run ``bin/buildout`` from the branch).
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
207
208
That's it for everyday usage.
209
210
=================================
211
Managing Dependencies and Scripts
212
=================================
213
214
What if you need to change or add dependencies or scripts?  As you might
215
expect, you need to know a bit more about what's going on, although we can
216
still keep this at a fairly high level.
217
218
First let's talk a little about the anatomy of what we have set up.  To be
219
clear, much of this is based on our own decisions of what to do.  If you see
220
something problematic, bring it up with the Foundations team.  Maybe together
221
we can come up with another approach that meets our needs better.
222
223
If you saw the top-level Launchpad directory before we started using Buildout,
224
you might notice seven new items in the checkout.
225
226
``bootstrap.py``
227
    This is the standard bootstrapping file provided by the Buildout
228
    distribution.  As of this writing, the Makefile uses this file, and you do
229
    not have to modify it or call it directly.
9012.1.3 by Maris Fogels
Added a bit to the buildout docs for using the filetemplate recipe.
230
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
231
    Without a Makefile, the first step of developing a source tree that uses
232
    Buildout is to run this file in the top level directory with the Python
233
    executable that you wish to use for the source tree.  It creates a ``bin``
234
    directory, if it does not already exist, and puts a ``buildout`` executable
235
    there.  The next step is to run ``bin/buildout``, which, unless you supply
236
    a ``-c`` option to specify a configuration file, will look for a
237
    buildout.cfg file by default to discover what to do.
9012.1.3 by Maris Fogels
Added a bit to the buildout docs for using the filetemplate recipe.
238
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
239
    Again, as of this writing, the Makefile uses this file, and it does not
240
    need to be modified, so you need not concern yourself with it further at
241
    this time.
242
243
``ez_setup.py``
244
    This is another standard file from another project.  In this case, it is
245
    the file provided by the setuptools project to install setuptools.  It is
246
    used by the ``setup.py`` file, described below.  It does not need to be
247
    modified or called directly.
248
249
``setup.py``
250
    This is the file that uses ``distutils``, extended by ``setuptools``, to
251
    specify direct dependencies, scripts, and other elements of the local
252
    source tree.
9012.1.3 by Maris Fogels
Added a bit to the buildout docs for using the filetemplate recipe.
253
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
254
    Buildout uses it, but the reverse is not true: ``setup.py`` does not know
255
    about Buildout.  This means that packages that use Buildout for development
256
    do not have to require it when they are being installed in other software
257
    as a dependency.
9012.1.3 by Maris Fogels
Added a bit to the buildout docs for using the filetemplate recipe.
258
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
259
    Describing this file in full is well beyond the scope of this document.  We
260
    will give recipes for modifying it for certain tasks below. For more
261
    information beyond these recipes, see the setuptools and distutils
262
    documentation.
263
264
``buildout.cfg``
265
    This is the default configuration file that ``bin/buildout`` will look to
266
    for instructions.
267
268
    Describing it in full is well beyond the scope of this document.  However,
269
    we will give an overview here.
270
271
    Configuration files for Buildout are comprised of sections with key-value
272
    pairs.
273
274
    The key-value pairs are separated with new lines, when the subsequent line
275
    is not indented.  The key and value are each separated with an equals sign.
9012.1.3 by Maris Fogels
Added a bit to the buildout docs for using the filetemplate recipe.
276
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
277
    ::
9012.1.3 by Maris Fogels
Added a bit to the buildout docs for using the filetemplate recipe.
278
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
279
        foo = bar
280
        baz = bing
281
              bah
282
              boo
283
        sha = zam
284
285
    That example shows three keys, 'foo', 'baz', and 'sha'.  The 'baz' key has
286
    a string with two new lines (which might be interpreted one several ways,
287
    as defined for that key).
288
289
    The ``[buildout]`` section is the starting point for Buildout to determine
290
    what to do.  It looks for an ``extends`` key to find any additional files
291
    to merge in; we use this for ``versions.cfg``, discussed below.
9012.1.3 by Maris Fogels
Added a bit to the buildout docs for using the filetemplate recipe.
292
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
293
    In addition to general configuration and initialization such as this, it
294
    looks in the ``develop`` key to find source trees to develop as part of the
295
    buildout. In the standard Launchpad configuration, we develop only
296
    Launchpad itself (the current directory, or '.').  This means that the
297
    local ``setup.py`` will be run.  If you want to develop Launchpad while you
298
    develop another dependency, you can link another source tree in, and
299
    specify an additional ``develop`` directory in another line::
9012.1.3 by Maris Fogels
Added a bit to the buildout docs for using the filetemplate recipe.
300
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
301
        [buildout]
302
        develop = .
303
                  lazr_uri_branch
304
11132.2.4 by James Westby
Extend the buildout documentation to discuss using the develop option.
305
    See `Developing a Dependent Library In Parallel`_ for more on this.
306
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
307
    The other basic key in the ``[buildout]`` section that we'll highlight here
308
    is ``parts``.  This key identifies the other sections in buildout.cfg that
309
    will be processed.  A section that is not identified in the ``[buildout]``
310
    sections ``parts`` key will usually be ignored (unless chosen for another
311
    role by another key elsewhere).
9012.1.3 by Maris Fogels
Added a bit to the buildout docs for using the filetemplate recipe.
312
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
313
    Sections other than ``[buildout]`` that are specified as parts always must
314
    specify a ``recipe``: an identifier that determines what code should
315
    process that section.  You'll see a variety of recipes in Launchpad's
316
    buildout.cfg, including ``z3c.recipe.filetemplate``, ``zc.recipe.egg``, and
317
    others.
318
319
``versions.cfg``
320
    As mentioned above, ``buildout.cfg`` extends ``versions.cfg``  by
321
    specifying it in the ``extends`` key of the ``[buildout]`` section.
322
    Versions.cfg specifies the precise versions of the dependencies we use.
323
    This means that we can have several versions of a dependency available
324
    locally, but we only build the precise one we specify.  We give an
325
    example of its use below.  To read about the mechanism used, see the
326
    zc.buildout documentation of the ``versions`` option in the ``[buildout]``
327
    section.
328
8329.1.7 by Gary Poster
convert layout to have eggs and download-cache on top; and add make target for eggs folder, for deployment.
329
``eggs``
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
330
    The ``eggs`` directory holds the eggs built from the downloaded
8329.1.7 by Gary Poster
convert layout to have eggs and download-cache on top; and add make target for eggs folder, for deployment.
331
    distributions. Unless you set it up differently yourself, this directory is
332
    shared by all your branches. This directory is local to your system--we do
333
    not manage it in a branch. One reason for this is that eggs are often
334
    platform-specific.
335
336
``download-cache``
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
337
    The ``download-cache`` directory is a set of downloaded distributions--that
338
    is, exact copies of the items that would typically be obtained from the
339
    Python Package Index ("PyPI"), or another download source. We manage the
340
    download cache as a shared resource across all of our developers with a bzr
341
    branch in a Launchpad project called ``lp-source-dependencies``.
9012.1.3 by Maris Fogels
Added a bit to the buildout docs for using the filetemplate recipe.
342
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
343
    When we run buildout, Buildout reads a special key and value in the
344
    ``[buildout]`` section: ``install-from-cache = true``.  This means that, by
345
    default, Buildout will *not* use network access to find packages, but
9012.1.3 by Maris Fogels
Added a bit to the buildout docs for using the filetemplate recipe.
346
    *only* look in the download cache.  This has many advantages.
347
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
348
    - First, it helps us keep our deployment boxes from needing network access
349
      out to PyPI and other download sites.
350
351
    - Second, it makes the buildout much faster, because it does not have to
9012.1.3 by Maris Fogels
Added a bit to the buildout docs for using the filetemplate recipe.
352
      look out on the net for every dependency.
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
353
354
    - Third, it makes the buildout more repeatable, because we are more
355
      insulated from outages at download sites such as PyPI, and poor release
9012.1.3 by Maris Fogels
Added a bit to the buildout docs for using the filetemplate recipe.
356
      management.
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
357
358
    - Fourth, it makes our deployments more auditable, because we can tell
359
      exactly what we are deploying.
360
361
    - Fifth, it gives us a single obvious place to put custom package
362
      distributions, as we'll discuss below.
363
364
    The downside is that adding and upgrading packages takes a small additional
365
    step, as we'll see below.
366
367
``buildout-templates``
368
    The last additional item in the checkout is the ``buildout-templates``
369
    directory.  This is used to hold templates that are used by the
370
    section in buildout.cfg that uses the ``z3c.recipe.filetemplate`` recipe.
371
    This can be used for many things, but we are using it as an alternate way
372
    for producing scripts when the zc.recipe.egg approach is insufficient.
373
374
In addition to these seven listings, after you have run the Makefile (or
375
``bin/buildout``), you will see an additional listing:
376
377
``bin``
378
    The ``bin`` directory has already been discussed many times.  After running
379
    the bootstrap.py, it holds the ``buildout`` script which can be used to
380
    process Buildout configuration files.  In Launchpad's case, after running
381
    the buildout, it also holds many executables, including scripts to test
382
    Launchpad; to run it; to run Python or IPython with Launchpad's sourcetree
383
    and dependencies available; to run harness or iharness (with IPython) with
384
    the sourcetree, dependencies, and database connections; or to perform
385
    several other tasks.  For now, the Makefile provides aliases for many of
386
    these.
387
388
Now that you have an introduction to the pertinent files and directories, we'll
389
move on to trying to perform tasks in the buildout.  We'll discuss adding a
390
dependency, upgrading a dependency, adding a script, adding an arbitrary file,
391
and working with unreleased packages.
392
393
Add a Package
394
=============
395
396
Let's suppose that we want to add the "lazr.foo" package as a dependency.
397
398
1.  Add the new package to the ``setup.py`` file in the ``install_requires``
399
    list.
9012.1.3 by Maris Fogels
Added a bit to the buildout docs for using the filetemplate recipe.
400
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
401
    Generally, our policy is to only set minimum version numbers in this file,
402
    or none at all.  It doesn't really matter for an application like
403
    Launchpad, but it a good rule for library packages, so we follow it for
404
    consistency.  Therefore, we might simply add ``'lazr.foo'`` to
405
    install_requires, or ``'lazr.foo >= 1.1'`` if we know that we are depending
406
    on features introduced in version 1.1 of lazr.foo.
407
408
2.  [OPTIONAL] If you know it, add the desired version number to versions.cfg
409
    now.
9012.1.3 by Maris Fogels
Added a bit to the buildout docs for using the filetemplate recipe.
410
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
411
    For instance, if you know you want lazr.foo 1.1.2, add this line to the
412
    ``[versions]`` section of ``versions.cfg``::
9012.1.3 by Maris Fogels
Added a bit to the buildout docs for using the filetemplate recipe.
413
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
414
      lazr.foo = 1.1.2
415
416
3.  [OPTIONAL] Add the desired distribution of lazr.foo 1.1.2 to the
8329.1.7 by Gary Poster
convert layout to have eggs and download-cache on top; and add make target for eggs folder, for deployment.
417
    ``download-cache/dist`` directory.
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
418
419
4.  Run the following command (or your variation)::
420
421
        ./bin/buildout -v buildout:install-from-cache=false | tee out.txt | grep 'Picked'
422
423
    The first part (``./bin/buildout -v buildout:install-from-cache=false``)
424
    will run buildout, allowing it to download source packages from the
8329.1.7 by Gary Poster
convert layout to have eggs and download-cache on top; and add make target for eggs folder, for deployment.
425
    Internet to ``download-cache/dist``. The second part (``tee out.txt``) will
426
    dump the full output to the ``out.txt`` file in case you need to debug a
427
    problem. The last part (``grep 'Picked'``) will filter the output so that
428
    only additional packages--dependencies of your dependency--will be listed.
429
    Look at the output. You need to see if it means that you have dependencies,
430
    some of which may be indirect dependencies. Here's an imaginary example
431
    output::
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
432
8514.2.1 by Francis J. Lacoste
Refer to ipython as the spurious Picked example.
433
        Picked: ipython = 0.9.1
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
434
        Picked: lazr.foom = 1.4
435
        Picked: zope.bar = 3.6.1
436
        Picked: z3c.shazam = 2.0.1
437
438
    At this time, note that the output will include at least one, and possibly
8514.2.1 by Francis J. Lacoste
Refer to ipython as the spurious Picked example.
439
    more, spurious "Picked:" listings.  ipython, in particular, shows up
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
440
    repeatedly.
9012.1.3 by Maris Fogels
Added a bit to the buildout docs for using the filetemplate recipe.
441
8514.2.1 by Francis J. Lacoste
Refer to ipython as the spurious Picked example.
442
    In our example, other than the spurious ``ipython`` listing, this means
8329.1.7 by Gary Poster
convert layout to have eggs and download-cache on top; and add make target for eggs folder, for deployment.
443
    that these packages have also been added to your ``download-cache/dist``
7849.19.3 by Sidnei da Silva
- Fix typo
444
    directory. You also need to add those versions to the ``versions.cfg``
8329.1.7 by Gary Poster
convert layout to have eggs and download-cache on top; and add make target for eggs folder, for deployment.
445
    file::
9012.1.3 by Maris Fogels
Added a bit to the buildout docs for using the filetemplate recipe.
446
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
447
        lazr.foom = 1.4
448
        zope.bar = 3.6.1
449
        z3c.shazam = 2.0.1
450
451
5.  Test.
452
453
    [TODO] Note that you can tell ec2test to include all uncommitted
454
    distributions from the local download-cache in its tests.  If you do this,
455
    you cannot use the ec2test feature to submit on test success.  Also, if you
456
    have uncommitted distributions and you do *not* explicitly tell ec2test to
457
    include or ignore the uncommitted distributions, it will refuse to start
458
    an instance.
459
12751.1.1 by Julian Edwards
Amend buildout doc to remind user to delete old versions of sourcecode tarballs.
460
6. Check old versions in the download-cache.  If you are sure that they are
461
   not in use any more, *anywhere*, then remove them to save checkout space.
462
   More explicitly, check with the LOSAs to see if they are in use in
463
   production and send an email to launchpad-dev@lists.launchpad.net before
464
   deleting anything if you are unsure.
465
466
7.  Commit the changes (``cd download-cache``, ``bzr up``,
8329.1.7 by Gary Poster
convert layout to have eggs and download-cache on top; and add make target for eggs folder, for deployment.
467
    ``bzr commit -m 'Add lazr.foom 1.1.2 and depdendencies to the download
468
    cache'``) to the shared download cache when you are sure it is what you
469
    want.
470
471
*Never* modify a package in the download-cache.
472
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
473
474
Upgrade a Package
475
=================
476
477
Sometimes you need to upgrade a dependency.  This may require additional
478
dependency additions or upgrades.
479
480
If you already know what version you want, the simplest thing to try is to
481
modify versions.cfg to specify the new version and run steps 4, 5, and 6 of the
482
`Add a Package`_ instructions.
483
484
If you don't know what version you want, but just want to see what happens when
485
you upgrade to the most recent revision, you can clear out the versions of the
486
packages for upgrade and give it a try (that is, run steps 4, 5, and 6 of the
487
`Add a Package`_ instructions).  Note that, when not given an explicit version
488
number, our buildout is set to prefer final releases over alpha and beta
489
releases.  If you want to temporarily override this behavior, include
490
``buildout:prefer-final=false`` as another argument to ``bin/buildout``.
491
492
Add a Script
493
============
494
495
We often need scripts that are run in a certain environment defined by Python
496
dependencies, and sometimes even different Python executables.  Several of the
497
scripts we have are specified using the setuptools-based spelling that the
498
``zc.recipe.egg`` recipe supports.
499
500
For the common case, in ``setup.py``, add a string in the ``console_scripts``
501
list of the ``entry_points`` argument. Here's an example string::
502
14565.2.11 by Curtis Hovey
Moved runlaunchpad to lp.scripts.
503
    'run = lp.scripts.runlaunchpad:start_launchpad'
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
504
505
This will create a script named ``run`` in the ``bin`` directory that calls the
506
``start_launchpad`` function in the
14565.2.11 by Curtis Hovey
Moved runlaunchpad to lp.scripts.
507
``lp.scripts.runlaunchpad`` module.
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
508
509
See the `zc.recipe.egg documentation`_ for more information on how to add
510
scripts using this method.
511
512
.. _`zc.recipe.egg documentation`: http://pypi.python.org/pypi/zc.recipe.egg
513
514
Add a File Modified By Buildout
515
===============================
516
517
Sometimes we need more control for the way our scripts are generated, or we
518
need other files processed during a buildout.  Writing a custom zc.buildout
519
recipe is one way, but well out of the scope of this document.  Read the
520
zc.buildout documentation for direction.
521
522
A much easier, and more limited approach is to use `z3c.recipe.filetemplate`_
9012.1.3 by Maris Fogels
Added a bit to the buildout docs for using the filetemplate recipe.
523
to build the file.  The recipe uses the ``buildout-templates`` directory,
524
which is a mirror of the Launchpad source tree.  The recipe searches the tree
525
for files ending in '.in', performs variable substitution on them, and then be
526
copies them into the Launchpad source tree.
527
528
To add a file using the recipe, simply create mirrors of the source tree
529
directories that you need under ``buildout-templates/``, and create a .in file
530
template at the desired location.  Take a look at
531
``buildout-templates/bin/`` for examples of what is possible.
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
532
533
.. _`z3c.recipe.filetemplate`: http://pypi.python.org/pypi/z3c.recipe.filetemplate
534
535
Work with Unreleased or Forked Packages
536
=======================================
537
538
Sometimes you need to work with unreleased or forked packages.  FeedValidator_,
539
for instance, makes nightly zip releases but other than that only offers svn
540
access.  Similarly, we may require a patched or unreleased version of a package
541
for some purpose.  Hopefully, these situations will be rare, but they do occur.
542
543
While `other answers`_ are available for Buildout, our solution is to use the
544
download-cache.  Basically, make a custom source distribution with a unique
545
suffix in the name, and use it (and its version name) for the normal process of
546
adding or updating a package, as described above.  Because the custom package
547
is in the download-cache, it will be found and used.
548
549
Here's an example of making a custom distribution of FeedValidator.
550
551
FeedValidator is a Subversion project.  We check it out::
552
553
    svn co http://feedvalidator.googlecode.com/svn/trunk/feedvalidator/src feedvalidator
554
555
Next, we ``cd feedvalidator``, and, using a Python that has setuptools
556
installed, we run the following command::
557
558
    python setup.py egg_info -r -bDEV sdist
559
560
For this example, imagine that the current revision of the repository is 1049.
561
Because setuptools has built-in Subversion support, the command above will
562
create a tar.gz in the ``dist`` directory named
563
``feedvalidator-0.0.0DEV-r1049.tar.gz``. The -r option specifies that the
564
subversion revision should be part of the package name.  The -bDEV option
565
specifies that the 'DEV' suffix should be added to the version number.
566
8329.1.7 by Gary Poster
convert layout to have eggs and download-cache on top; and add make target for eggs folder, for deployment.
567
We could then put the tar.gz file in Launchpad's ``download-cache/dist``
568
directory, specify ``feedvalidator = 0.0.0DEV-r1049`` in the ``versions.cfg``
569
file, and proceed with the usual steps to update or add a new package.
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
570
571
If you use a bzr branch, you might use the ``-d`` option instead of the ``-r``
572
option when you create the distribution.  This will add the date instead of the
573
revision::
574
575
    python setup.py egg_info -d -bDEV sdist
576
577
For instance, this might produce a distribution for the ``lazr.restful``
578
project with a name like this: ``lazr.restful-0.9.1DEV-20090512.tar.gz``.
579
580
See the setuptools documentation for more information about `the egg_info
581
command`_.
582
13697.6.2 by Henning Eggers
Update buildout docucmentation.
583
It is also possible that setup.py does not support the egg_info command and
584
it is sufficient to just run the sdist command. It adds the current revision
585
of the bzr branch to the version number::
586
587
    python setup.py sdist
588
589
For instance, this might produce a distribution for the ``testtools``
590
project with a name like this: ``testtools-0.9.12-r228.tar.gz``.
591
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
592
.. _FeedValidator: http://feedvalidator.org/
593
594
.. _`other answers`: http://pypi.python.org/pypi/zc.buildoutsftp
595
596
.. _`the egg_info command`: http://peak.telecommunity.com/DevCenter/setuptools#tagging-and-daily-build-or-snapshot-releases
597
11132.2.4 by James Westby
Extend the buildout documentation to discuss using the develop option.
598
Developing a Dependent Library In Parallel
599
==========================================
600
601
Sometimes you need to iterate on change to a library used by Launchpad
602
that is managed by buildout as an egg. You could just edit what it is in
603
the ``eggs`` directory, but it is harder to produce a patch while doing
604
this. You could instead grab a branch of the libarary and produce an egg
605
everytime you make a change and make buildout use the new egg, but this is
606
slow.
607
608
buildout defaults to only using the current directory as code that will
609
be used without creating a distribution. We can arrange for it to use other
610
paths as well, so we can use a checkout of any code we like, with changes
611
being picked up instantly without us having to create a distribution.
612
613
To do this add the extra paths to the ``develop`` key in the ``[buildout]``
12392.1.1 by Jonathan Lange
Fix restructured text errors.
614
section of ``buildout.cfg``::
11132.2.4 by James Westby
Extend the buildout documentation to discuss using the develop option.
615
616
    [buildout]
617
    develop = .
618
              path/to/branch
619
620
and re-run ``make``.
621
622
Now any changes you make in that path will be picked up, and you are free
623
to make the changes you need and test them in the Launchpad environment.
624
625
Once you are finished you can produce a distribution as above for inclusion
626
in to Launchpad, as well as sending your patch upstream. At that point you
627
are free to revert the configuration to only develop Launchpad. You should
628
not submit your branch with this change in the configuration as it will
629
not work for others.
630
631
Be aware that you may have to change the version for the package in
632
``versions.cfg`` if there is a difference between the version in the
633
``eggs`` directory and the one in the ``setup.py`` that you pointed to
634
in the ``develop`` key.
635
636
One thing to be wary of is that setuptools treats "develop eggs" created
637
by this process with the same precedence as system packages. That means
638
that if the version in the ``setup.py`` at the path that you put in the
639
``develop`` key is the same as the version installed system wide, setuptools
640
may pick the wrong one. If that happens then increase the version in
641
setup.py and setuptools will take it.
642
8329.1.4 by Gary Poster
respond to reviews; in particular, add doc.
643
=====================
644
Possible Future Goals
645
=====================
646
647
- No longer use system site-packages.
648
- No longer use make.
649
- Get rid of the sourcecode directory.