25
24
default global value is FALSE which will not include the query in the
26
25
messages. It can be controlled per session, as well. For example:
30
drizzle> set @@replicate_query = 1;
27
``drizzle> set @@replicate_query = 1;``
32
29
The stored query should be used as a guide only, and never executed
33
30
on a slave to perform replication as this will lead to incorrect results.
35
32
Message Definitions
38
35
The GPB messages are defined in .proto files in the drizzled/message
39
36
directory of the Drizzle source code. The primary definition file is
108
105
#. A TransactionContext message containing information about the
109
106
transaction as a whole, such as the ID of the executing server,
110
the start and end timestamp of the transaction, segmenting
111
metadata and a unique identifier for the transaction.
107
the start and end timestamp of the transaction, and a globally-
108
unique identifier for the transaction.
112
109
#. A vector of Statement messages representing the distinct SQL
113
110
statements which modified the state of the server. The Statement
114
111
message is, itself, a generic envelope message containing a
115
112
sub-message which describes the specific data modification which
116
occurred on the server (such as, for instance, an INSERT statement).
113
occurred on the server (such as, for instance, an INSERT statement.
118
115
The Statement Message
119
#####################
116
^^^^^^^^^^^^^^^^^^^^^
121
118
The generic "envelope" message containing information common to each
122
119
SQL statement executed against a server (such as a start and end timestamp
127
124
of the Statement should construct the inner Statement subclass representing
130
Statements are recorded separately as sometimes individual statements
127
Statements are recorded seperatley as sometimes individual statements
131
128
have to be rolled back.
134
131
.. _bulk-operations:
136
133
How Bulk Operations Work
137
########################
134
------------------------
139
136
Certain operations which change large volumes of data on a server
140
137
present a specific set of problems for a transaction coordinator or
154
151
replication system uses a mechanism which provides bulk change
157
A single transaction in the database can possibly be represented with
158
multiple protobuf Transaction messages if the message grows too large.
159
This can happen if you have a bulk transaction, or a single statement
160
affecting a very large number of rows, or just a large transaction with
161
many statements/changes.
163
For the first two examples, it is likely that the Statement sub-message
164
itself will get segmented, causing another Transaction message to be
165
created to hold the rest of the Statement's row changes. In these cases,
166
it is enough to look at the segment information stored in the Statement
167
message (see example below).
169
For the last example, the Statement sub-messages may or may not be
170
segmented, but we could still need to split the individual Statements up into
171
multiple Transaction messages to keep the Transaction message size from
172
growing too large. In this case, the segment information in the Statement
173
submessages is not helpful if the Statement isn't segmented. We need this
174
information in the Transaction message itself.
176
Segmenting a Single SQL Statement
177
*********************************
179
154
When a regular SQL statement modifies or inserts more rows than a
180
155
certain threshold, Drizzle's replication services component will begin
181
156
sending Transaction messages to replicas which contain a chunk
223
194
#. Construct an UpdateHeader message with information about the tables
224
195
and fields involved in the UPDATE statement. Push this UpdateHeader
225
196
message onto the Transaction message's statement vector.
226
#. Construct an UpdateData message. Set the *segment_id* member to 1.
227
Set the *end_segment* member to true.
197
#. Construct an UpdateData message. Set the segment_id member to 1.
198
Set the end_segment member to true.
228
199
#. For every record updated in a storage engine, the ReplicationServices
229
200
component builds a new UpdateRecord message and appends this message
230
201
to the aforementioned UpdateData message's record vector.
231
202
#. After a certain threshold of records is reached, the
232
203
ReplicationServices component sets the current UpdateData message's
233
*end_segment* member to false, and proceeds to send the Transaction
204
end_segment member to false, and proceeds to send the Transaction
234
205
message to replicators.
235
206
#. The ReplicationServices component then constructs a new Transaction
236
207
message and constructs a transaction context with the same
237
208
transaction ID and server information.
238
#. A new UpdateData message is created. The message's *segment_id* is
209
#. A new UpdateData message is created. The message's segment_id is
239
210
set to N+1 and as new records are updated, new UpdateRecord messages
240
211
are appended to the UpdateData message's record vector.
241
212
#. While records are being updated, we repeat steps 5 through 7, with
242
only the final UpdateData message having its *end_segment* member set
213
only the final UpdateData message having its end_segment member set
245
Segmenting a Transaction
246
************************
248
The Transaction protobuf message also contains *segment_id* member and a
249
*end_segment* member. These values are also set appropriately when a
250
Statement sub-message is segmented, as described above.
252
These values are also set when a Transaction must be segmented along
253
individual Statement boundaries (i.e., the Statement message itself
254
is **not** segmented). In either case, it is enough to check the
255
*end_segment* and *segment_id* values of the Transaction message
256
to determine if this is a multi-message transaction.
258
216
Handling ROLLBACKs
261
219
Both transactions and individual statements may be rolled back.
271
229
* For a transaction which is made up of multiple messages, and at least
272
230
one message has already been sent through the replication stream, then
273
231
the Transaction message will contain a Statement message with type =
274
ROLLBACK. This signifies to rollback the entire transaction.
276
A special Statement message type, ROLLBACK_STATEMENT, is used when
277
we have a segmented Statement message (see above) and we need to tell the
278
receiver to undo any changes made for this single statement, but not
279
for the entire transaction. If the receiver cannot handle rolling back
280
a single statement, then a message buffering strategy should be employed
281
to guarantee that a statement was indeed applied successfully before
282
executing on the receiver.
284
.. _replication_streams:
289
The Drizzle kernel handles delivering replication messages to plugins by
290
maintaining a list of replication streams. A stream is represented as a
291
registered *replicator* and *applier* pair.
293
When a replication message is generated within the kernel, the replication
294
services module of the kernel will send this message to each registered
295
*replicator*. The *replicator* will then do something useful with it and
296
send it to each *applier* with which it is associated.
301
A registered *replicator* is a plugin that implements the TransactionReplicator
302
API. Each replicator will be plugged into the kernel to receive the Google
303
Protobuf messages that are generated as the database is changed. Ideally,
304
each registered replicator will transform or modify the messages it receives
305
to implement a specific behavior. For example, filtering by schema name.
307
Each registered replicator should have a unique name. The default replicator,
308
cleverly named **default_replicator**, does no transformation at all on the
309
replication messages.
314
A registered *applier* is a plugin that implements the TransactionApplier
315
API. Appliers are responsible for applying the replication messages that it
316
will receive from a registered replicator. The word "apply" is used loosely
317
here. An applier may do anything with the replication messages that provides
318
useful behavior. For example, an applier may simply write the messages to a
319
file on disk, or it may send the messages over the network to some other
320
service to be processed.
322
At the point of registration with the Drizzle kernel, each applier specifies
323
the name of a registered replicator that it should be attached to in order to
324
make the replication stream pair.
added
removed
docs/replication.rst
