1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
|
= Application Server Database Policy =
The database policy chooses the default Storm store to used. Its goal
is to distribute load away from the master databases to read only
stores where possible. It will benefit old code - new code should
explicitly select objects from the master or slave stores as needed.
To test this policy, lets point the MAIN SLAVE store to a Launchpad
database with a different name. This makes it easy to check if a
request is querying the master or slave database.
>>> from lp.services.config import config
>>> from textwrap import dedent
>>> config_overlay = dedent("""
... [database]
... rw_main_slave: dbname=launchpad_empty
... """)
>>> config.push('empty_slave', config_overlay)
>>> from zope.component import getUtility
>>> from lp.services.webapp.interfaces import (
... IStoreSelector, MAIN_STORE, MASTER_FLAVOR, SLAVE_FLAVOR)
>>> from lp.testing.layers import DatabaseLayer
>>> master = getUtility(IStoreSelector).get(MAIN_STORE, MASTER_FLAVOR)
>>> dbname = DatabaseLayer._db_fixture.dbname
>>> dbname == master.execute("SELECT current_database()").get_one()[0]
True
>>> slave = getUtility(IStoreSelector).get(MAIN_STORE, SLAVE_FLAVOR)
>>> slave.execute("SELECT current_database()").get_one()[0]
u'launchpad_empty'
We should confirm that the empty database is as empty as we hope it is.
>>> from zope.component import getUtility
>>> from lp.services.webapp.interfaces import (
... IStoreSelector, MAIN_STORE, MASTER_FLAVOR, SLAVE_FLAVOR)
>>> from lp.registry.model.person import Person
>>> slave_store = getUtility(IStoreSelector).get(
... MAIN_STORE, SLAVE_FLAVOR)
>>> master_store = getUtility(IStoreSelector).get(
... MAIN_STORE, MASTER_FLAVOR)
>>> slave_store.find(Person).count()
0
>>> master_store.find(Person).count() > 0
True
This helper parses the output of the +whichdb view (which unfortunately
needs to be created externally to this pagetest).
>>> def whichdb(browser):
... dbname = extract_text(find_tag_by_id(browser.contents, 'dbname'))
... if dbname == DatabaseLayer._db_fixture.dbname:
... return 'MASTER'
... elif dbname == 'launchpad_empty':
... return 'SLAVE'
... else:
... return 'UNKNOWN'
Read only requests such as GET and HEAD will use the MAIN SLAVE
Store by default.
>>> browser.open('http://launchpad.dev/+whichdb')
>>> print whichdb(browser)
SLAVE
POST requests might make updates, so they use the MAIN MASTER
Store by default.
>>> browser.getControl('Do Post').click()
>>> print whichdb(browser)
MASTER
This is an unauthenticated browser. These typically have no session, unless
special dispensation has been made. Without a session, subsequent requests
will then immediately return to using the SLAVE.
>>> browser.open('http://launchpad.dev/+whichdb')
>>> print whichdb(browser)
SLAVE
However, if the request has a session (that is, is authenticated; or is
unauthenticated, but under special dispensation to have a session), once a
POST request has been made, further GET and HEAD requests from the same client
continue to use the MAIN MASTER Store by default for 5 minutes. This ensures
that a user will see any changes they have made immediately, even though the
slave databases may lag some time behind the master database.
>>> browser.addHeader('Authorization', 'Basic mark@example.com:test')
>>> browser.getControl('Do Post').click() # POST request
>>> print whichdb(browser)
MASTER
>>> browser.open('http://launchpad.dev/+whichdb') # GET request
>>> print whichdb(browser)
MASTER
GET and HEAD requests from other clients are unaffected though
and use the MAIN SLAVE Store by default.
>>> anon_browser.open('http://launchpad.dev/+whichdb')
>>> print whichdb(anon_browser)
SLAVE
>>> admin_browser.open('http://launchpad.dev/+whichdb')
>>> print whichdb(admin_browser)
SLAVE
If no more POST requests are made for 5 minutes, GET and HEAD
requests will once again be using the MAIN SLAVE store as we
can assume that any changes made to the master database have
propagated to the slaves.
To test this, first we need to wind forward the database policy's clock.
>>> from lp.services.webapp import dbpolicy
>>> from datetime import timedelta
>>> _original_now = dbpolicy._now
>>> def _future_now():
... return _original_now() + timedelta(minutes=10)
>>> browser.open('http://launchpad.dev/+whichdb')
>>> print whichdb(browser)
MASTER
>>> dbpolicy._now = _future_now # Install the time machine.
>>> browser.open('http://launchpad.dev/+whichdb')
>>> print whichdb(browser)
SLAVE
>>> dbpolicy._now = _original_now # Reset the time machine.
When lag gets too bad, we stop using slave databases. This stops
replication oddities from becoming too bad, as well as lightening the load
on the slaves allowing them to catch up.
>>> anon_browser.open('http://launchpad.dev/+whichdb')
>>> print whichdb(anon_browser)
SLAVE
>>> dbpolicy._test_lag = timedelta(minutes=10)
>>> anon_browser.open('http://launchpad.dev/+whichdb')
>>> print whichdb(anon_browser)
MASTER
>>> dbpolicy._test_lag = None
A 404 error page is shown when code raises a LookupError. If a slave
database is being used, this might have been caused by replication lag
if the missing data was only recently created. To fix this surprising
error, requests are always retried using the master database before
returning a 404 error to the user.
>>> anon_browser.handleErrors = True
>>> anon_browser.raiseHttpErrors = False
# Confirm requests are going to the SLAVE
>>> anon_browser.open('http://launchpad.dev/+whichdb')
>>> print whichdb(anon_browser)
SLAVE
# The slave database contains no data, but we don't get
# a 404 page - the request is retried against the MASTER.
>>> anon_browser.open('http://launchpad.dev/~stub')
>>> anon_browser.headers['Status']
'200 Ok'
# 404s are still returned though if the data doesn't exist in the
# MASTER database either.
>>> anon_browser.open('http://launchpad.dev/~does-not-exist')
>>> anon_browser.headers['Status']
'404 Not Found'
# This session is still using the SLAVE though by default.
>>> anon_browser.open('http://launchpad.dev/+whichdb')
>>> print whichdb(anon_browser)
SLAVE
Reset our config to avoid affecting other tests.
>>> ignored = config.pop('empty_slave')
|