~launchpad-pqm/launchpad/devel

= Launchpad configs =

This directory defines the configurations used to run Launchpad

applications in specific environments.

== Environments ==

This directory stores configs. The config used is selected on startup

using the LPCONFIG environment variable.

Each config directory contains a launchpad-lazr.conf file, and optionally

a number of .zcml files. These .zcml files are processed as ZCML

overrides allowing you to change behavior not yet configurable in

launchpad-lazr.conf.

If you want to create a temporary config, prefix the directory name with

'+' so that bzr ignores it and you won't accidentally commit it (this

pattern is listed in the top level .bzrignore in this tree).

If you need to make changes to the production, staging or dogfood configs

make sure you inform the people in charge of those systems. You may wish

to do this if you are adding a new required config option to

launchpad-lazr.conf.

The old ZConfig-based launchpad.conf files are still used to define

servers and log files. These will be replaced when lazr.config is

bootstrapped into the Zope startup process.

== CanonicalConfig ==

Launchpad uses a singleton CanonicalConfig object to select the config

to load and manage its state.

=== Instance directories and process files ===

The directories in configs/ represent environment's config `instance`.

Environment and instance are synonymous in this case. The instance is

often set by the LPCONFIG environment variable, but an app may override

this. The test.py calls:

    config.setInstance('testrunner')

to force the testrunner configuration to be loaded.

An instance directory may contain several lazr.config conf files.

CanonicalConfig loads the config file named for the process that is

running, eg. if the processes name is 'test', CanonicalConfig looks for

test-lazr.conf. launchpad-lazr.conf is loaded if not a lazr config files

named for the process.

All this information is available in the config object.

    >>> config.instance_name

    'testrunner'

    >>> config.process_name

    'test'

    >>> config.filename

    '.../configs/testrunner/launchpad-lazr.conf'

    >>> config.extends.filename

    '.../configs/development/launchpad-lazr.conf'

=== Accessing the CanonicalConfig in code ===

The CanonicalConfig singleton is exposed as config in its module.

    from canonical.config import config

The config can be accessed as a dictionary...

    >>> 'launchpad' in config

    True

    >>> config['launchpad']['default_batch_size']

    5

...though it is commonly accessed as an object.

    >>> config.librarian.download_host

    'localhost'

    >>> config.librarian.download_port

    58000

You can learn more about canonical.config in the doctest located at

    lib/canonical/launchpad/doc/canonical-config.txt

=== Testing with CanonicalConfig ===

Configurations are meant to be immutable--applications should never

alter the config. Nor should tests. Older code and tests assumed

that because the keys looked like attributes of the config, they

could be set. This was *wrong*. The code was actually adding an

attribute to the CanonicalConfig instance rather that updating to

underlying config object. While the code intended to reset the key's

value to the original value, it would have to delete the new attribute

to really restore the config singleton.

CanonicalConfig supports testing by exposing lazr.config's push() and

pop() methods to add and remove configurations to the stack of

ConfigData. The configuration can be modified and safely restored.

Tests can call push() with the configuration name and data to update

the config singleton.

    >>> test_data = ("""

    ...     [answertracker]

    ...     email_domain: answers.launchpad.dev""")

    >>> config.push('test_data', test_data)

    >>> config.answertracker.email_domain

    'answers.launchpad.dev'

And tests can remove the data with pop() when they are done to restore

the config.

    >>> config.pop('test_data')

    (<canonical.lazr.config.ConfigData ...>,)

    >>> config.answertracker.email_domain

    'answers.launchpad.net'

== lazr.conf schema and confs ==

All Launchpad configs inherit from the Launchpad schema defined

in ../lib/canonical/config/schema-lazr.conf (it is symlinked

as ./schema-lazr.conf for convenience).

lazr.config conf and schema files look like ini-based conf files, but

supports a number of features that make it easy for us to run multiple

versions of the application:

    1. The [meta] section's extends: key points to a file that defines

       the inherited sections, keys, and values.

    2. All config automatically inherit the default sections and

       key values from the schema. configs need only to define

       the keys that are unique to itself. Shared confs can be created

       to define a common set of key values for many configs.

    3. Except for optional sections ([<section>.optional]) which must

       be declared in a conf file for all the keys and values to be

       inherited.

    4. Schema's may define a template for a category of sections

       ([<category>.template]) to define a common set of keys and

       values.

    5. The schema and configs are validated. Configs cannot add

       sections or keys that are not defined in the schema. Sections

       and keys cannot be defined twice in a file.

    6. Launchpad uses implicit typing. The default value of a key is a

       str. If the vales looks like a int, it will be cast as an int.

       the tokens 'true', 'false', and 'none' (in any case) will map

       to Python types.

When adding sections and keys to the schema, include a comment that

documents their purpose.

The schema should define the default value for a key when the value is

safe for all environments. Values that enable a common feature, or set

the size of a list for example, should be in the schema. Values like

cause email to be sent, or development features should be disabled

in the schema--each environment that wants the enabled feature may do

so in its local conf file.

You can learn more about lazr.config in the doctest located at

    lib/canonical/lazr/doc/config.txt

=== schema template and optional sections ===

The schema can contain [<category>.template] sections that define a

common set of keys and default value for category of sections.

For example:

    [vhost.template]

    # Host name of this virtual host.

    # This is matched from the incoming Host header, and

    # also used to put together URLs if rooturl is not provided.

    # Example: launchpad.net

    # datatype: string

    hostname: none

    # Alternative host names to match, in addition to

    # the one given in hostname, comma separated.

    # Example: wwwww.launchpad.net, www.launchpad.net

    # datatype: string

    althostnames: none

    # Explicit root URL for this virtual host.

    # If this is not provided, the root URL is calculated

    # based on the host name.

    # Example: https://launchpad.net/

    # datatype: string

    rooturl: none

The [vhost.template] defines the keys and default values of a vhost.

"vhost" is a category, any section whose name is prefixed "vhost" will

inherit the keys and default values of the [vhost.template].

[vhost.answers] is an empty section in the schema...

    [vhost.answers]

...and lpnet-lazr.conf defines this:

    [vhost.answers]

    hostname: answers.launchpad.net

./lpnet1/launchpad-lazr.conf does not define anything for

[vhost.answers], yet when it is loaded, it has the all the keys and

default values of [vhost.template] from the schema, with the hostname

change defined in lpnet-lazr.conf:

    [vhost.answers]

    althostnames: None

    hostname: answers.launchpad.net

    rooturl: None

The schema may contain [<section>.optional] to define a section

that may declared in a conf, but is not automatically

inherited. This allows the application to define process configuration

data without exposing that information in every environment.

For example:

    [vhost.xmlrpc_private.optional]

is defined in the schema. It has the default keys and values defined

in [vhost.template]. The [vhost.xmlrpc_private] is not visible in

most confs because they do not declare that they use the section.

The production-xmlrpc-private/launchpad-lazr.conf file does though,

by including [vhost.xmlrpc_private] section:

    [vhost.xmlrpc_private]

    hostname: xmlrpc.lp.internal

    rooturl: https://launchpad.net/

Including just the section ([vhost.xmlrpc_private]) will suffice. In

this case, the two keys were redefined.

=== Implicit typing ===

lazr.config support implicit typing so that the application does not

need to coerce the config values:

    Integers: any value that is only made up of numbers, optionally

        prefixed with +/- is cast as an int: 0, 2001, -55, +404, 100.

    True, False, or None: any value that matches the boolean and None

        keywords is treated as the prescribed type. The match is

        case-insensitive: none, nOne, true, and False are all matched.

    Strings: any value that is not an int, bool, or None is treated as

        a str. Multi-line strings can be included by indenting the

        continuation lines to show that they are subordinate to the

        key:

            mykey: this line

                has a line break in it.

Implicit typing does not support lists, or compound types. Code must

split and unpack the value to make the desired object. For example,

the callsite must split the host:port compound object and coerce the

port to an int.

=== Config inheritance ===

The lazr configurations in this directory descend from the

Launchpad schema. This is a general outline of inheritance:

    ../lib/canonical/config/schema-lazr.conf

        |

        + development/launchpad-lazr.conf

        |    |

        |    + testrunner/launchpad-lazr.conf

        |         |

        |         + authserver-lazr.conf

        |         |

        |         + testrunner-appserver/launchpad-lazr.conf

        |             |

        |             + authserver-lazr.conf

        |

        + staging-lazr.conf

        |    |

        |    + bazaar-staging/launchpad-lazr.conf

        |    |

        |    + staging/launchpad-lazr.conf

        |    |    |

        |    |    + authserver-lazr.conf

        |    |

        |    + staging-mailman/launchpad-lazr.conf

        |

        + edge-lazr.conf

        |    |

        |    + edge<1-4>/launchpad-lazr.conf

        |

        + lpnet-lazr.conf

        |    |

        |    + lpnet<1-8>/launchpad-lazr.conf

        |    |

        |    + wildcherry/launchpad-lazr.conf

        |    |

        |    + librarian/launchpad-lazr.conf

        |    |

        |    + librarian-restricted/launchpad-lazr.conf

        |    |

        |    + production/launchpad-lazr.conf

        |    |

        |    + production-mailman/launchpad-lazr.conf

        |    |

        |    + production-xmlrpc-private/launchpad-lazr.conf

        |

        + demo-lazr.conf

        |    |

        |    + demo<1-4>/launchpad-lazr.conf

        |

        + beta-lazr.conf

        |    |

        |    + beta<1-3>/launchpad-lazr.conf

        |

        + dogfood/launchpad-lazr.conf

        |

        + ...

There are other configuration in this directory that are not

listed here

=== Viewing a configuration with lsconf.py ===

You can view the complete configuration for an process using the

lsconf.py utility to assemble the configuration from the lazr

conf file. eg:

    ./utilities/lsconf.py ./configs/production/launchpad-lazr.conf

The output looks like a lazr.conf file that lists all the sections

and keys in the configuration. The heading lists the order the conf

files were processed from child to ancestor.

Two useful options are -v and -s. The verbose option (-v) will annotate

each key with a comment explaining which conf file set the value.

The section name option (-s) will limit the output to the named

section. eg:

    ./utilities/lsconf.py -v -s answertracker \

        ./configs/production/launchpad-lazr.conf

    # This configuration derives from:

    #     ./configs/production/launchpad-lazr.conf

    #     ./configs/lpnet-lazr.conf

    #     ./lib/canonical/config/schema-lazr.conf

    [answertracker]

    # Defined in: lib/canonical/config/schema-lazr.conf

    days_before_expiration: 15

    # Defined in: lib/canonical/config/schema-lazr.conf

    dbuser: answertracker

    # Defined in: lib/canonical/config/schema-lazr.conf

    email_domain: answers.launchpad.net