~drizzle-trunk/drizzle/development

2144.1.1 by patrick crews
Overhaul of code. We can run rabbitmq : ) We now better encapsulate a per-executor working environment = one step closer to --parallel >: ) using subprocess goodness for server control
1
#! /usr/bin/env python
2235.5.2 by Stewart Smith
dbqp source (and all its libs) should use emacs python mode, not emacs C mode
2
# -*- mode: python; indent-tabs-mode: nil; -*-
2088.9.1 by patrick crews
Updated tree so that test-run.pl and test-run.py may live together in peace for a time
3
# vim:expandtab:shiftwidth=2:tabstop=2:smarttab:
4
#
2194.2.1 by patrick crews
Integrated randgen with dbqp. We now have mode=randgen and a set of randgen test suites (very basic now). Output = same as dtr : ) We also have mode=cleanup to kill any servers we have started. Docs updates too. Gendata utility allows us to populate test servers
5
# Copyright (C) 2010,2011 Patrick Crews
2088.9.1 by patrick crews
Updated tree so that test-run.pl and test-run.py may live together in peace for a time
6
#
2121.3.2 by patrick crews
Updated license verbiage
7
# This program is free software; you can redistribute it and/or modify
8
# it under the terms of the GNU General Public License as published by
9
# the Free Software Foundation; either version 2 of the License, or
10
# (at your option) any later version.
11
#
12
# This program is distributed in the hope that it will be useful,
13
# but WITHOUT ANY WARRANTY; without even the implied warranty of
14
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
15
# GNU General Public License for more details.
16
#
17
# You should have received a copy of the GNU General Public License
18
# along with this program; if not, write to the Free Software
19
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
2121.3.1 by patrick crews
Added licensing text to dbqp files
20
2088.9.1 by patrick crews
Updated tree so that test-run.pl and test-run.py may live together in peace for a time
21
22
""" drizzled.py:  code to allow a serverManager
23
    to provision and start up a drizzled server object
24
    for test execution
25
26
"""
27
28
# imports
29
import os
2194.2.1 by patrick crews
Integrated randgen with dbqp. We now have mode=randgen and a set of randgen test suites (very basic now). Output = same as dtr : ) We also have mode=cleanup to kill any servers we have started. Docs updates too. Gendata utility allows us to populate test servers
30
from lib.server_mgmt.server import Server
2088.9.1 by patrick crews
Updated tree so that test-run.pl and test-run.py may live together in peace for a time
31
2194.2.1 by patrick crews
Integrated randgen with dbqp. We now have mode=randgen and a set of randgen test suites (very basic now). Output = same as dtr : ) We also have mode=cleanup to kill any servers we have started. Docs updates too. Gendata utility allows us to populate test servers
32
class drizzleServer(Server):
2088.9.1 by patrick crews
Updated tree so that test-run.pl and test-run.py may live together in peace for a time
33
    """ represents a drizzle server, its possessions
34
        (datadir, ports, etc), and methods for controlling
35
        and querying it
36
2144.1.1 by patrick crews
Overhaul of code. We can run rabbitmq : ) We now better encapsulate a per-executor working environment = one step closer to --parallel >: ) using subprocess goodness for server control
37
        TODO: create a base server class that contains
38
              standard methods from which we can inherit
39
              Currently there are definitely methods / attr
40
              which are general
41
2088.9.1 by patrick crews
Updated tree so that test-run.pl and test-run.py may live together in peace for a time
42
    """
43
2194.2.1 by patrick crews
Integrated randgen with dbqp. We now have mode=randgen and a set of randgen test suites (very basic now). Output = same as dtr : ) We also have mode=cleanup to kill any servers we have started. Docs updates too. Gendata utility allows us to populate test servers
44
    def __init__( self, name, server_manager, default_storage_engine
45
                , server_options, requester, workdir_root):
46
        super(drizzleServer, self).__init__( name, server_manager
47
                                           , default_storage_engine
48
                                           , server_options, requester
49
                                           , workdir_root)
2088.9.1 by patrick crews
Updated tree so that test-run.pl and test-run.py may live together in peace for a time
50
        self.code_tree = self.system_manager.code_tree
51
        self.preferred_base_port = 9306
2240.4.2 by patrick crews
Moved client file declarations earlier in init function for drizzle server
52
        
53
        # client files
54
        self.drizzledump = self.code_tree.drizzledump
55
        self.drizzle_client = self.code_tree.drizzle_client
56
        self.drizzleimport = self.code_tree.drizzleimport
57
        self.drizzleadmin = self.code_tree.drizzleadmin
58
        self.drizzleslap = self.code_tree.drizzleslap
59
        self.server_path = self.code_tree.drizzle_server
60
        self.drizzle_client_path = self.code_tree.drizzle_client
61
        self.schemawriter = self.code_tree.schemawriter
62
2088.9.1 by patrick crews
Updated tree so that test-run.pl and test-run.py may live together in peace for a time
63
        # Get our ports
64
        self.port_block = self.system_manager.port_manager.get_port_block( self.name
65
                                                                         , self.preferred_base_port
2283.4.9 by Stewart Smith
add JSON_SERVER_PORT to test-run.pl and dbqp. Make the basic json_server test actually test something useful, such as interaction between SQL from normal protocol and SQL from http
66
                                                                         , 6 )
2088.9.1 by patrick crews
Updated tree so that test-run.pl and test-run.py may live together in peace for a time
67
        self.master_port = self.port_block[0]
68
        self.drizzle_tcp_port = self.port_block[1]
69
        self.mc_port = self.port_block[2]
70
        self.pbms_port = self.port_block[3]
2144.1.1 by patrick crews
Overhaul of code. We can run rabbitmq : ) We now better encapsulate a per-executor working environment = one step closer to --parallel >: ) using subprocess goodness for server control
71
        self.rabbitmq_node_port = self.port_block[4]
2283.4.9 by Stewart Smith
add JSON_SERVER_PORT to test-run.pl and dbqp. Make the basic json_server test actually test something useful, such as interaction between SQL from normal protocol and SQL from http
72
        self.json_server_port = self.port_block[5]
2088.9.1 by patrick crews
Updated tree so that test-run.pl and test-run.py may live together in peace for a time
73
74
        # Generate our working directories
75
        self.dirset = { self.name : { 'var': {'std_data_ln':( os.path.join(self.code_tree.testdir,'std_data'))
76
                                             ,'log':None
77
                                             ,'run':None
78
                                             ,'tmp':None
79
                                             ,'master-data': {'local': { 'test':None
80
                                                                       , 'mysql':None
81
                                                                       }
82
                                                             }
83
                                             }  
84
                                    } 
85
                      }
86
        self.workdir = self.system_manager.create_dirset( workdir_root
87
                                                        , self.dirset)
88
        self.vardir = os.path.join(self.workdir,'var')
89
        self.tmpdir = os.path.join(self.vardir,'tmp')
90
        self.rundir = os.path.join(self.vardir,'run')
91
        self.logdir = os.path.join(self.vardir,'log')
92
        self.datadir = os.path.join(self.vardir,'master-data')
93
94
        self.error_log = os.path.join(self.logdir,('%s.err' %(self.name)))
95
        self.pid_file = os.path.join(self.rundir,('%s.pid' %(self.name)))
96
        self.socket_file = os.path.join(self.vardir, ('%s.sock' %(self.name)))
97
        self.timer_file = os.path.join(self.logdir,('timer'))
2194.2.1 by patrick crews
Integrated randgen with dbqp. We now have mode=randgen and a set of randgen test suites (very basic now). Output = same as dtr : ) We also have mode=cleanup to kill any servers we have started. Docs updates too. Gendata utility allows us to populate test servers
98
99
        # Do magic to create a config file for use with the slave
100
        # plugin
101
        self.slave_config_file = os.path.join(self.logdir,'slave.cnf')
102
        self.create_slave_config_file()
103
2088.9.1 by patrick crews
Updated tree so that test-run.pl and test-run.py may live together in peace for a time
104
        self.snapshot_path = os.path.join(self.tmpdir,('snapshot_%s' %(self.master_port)))
105
        # We want to use --secure-file-priv = $vardir by default
106
        # but there are times / tools when we need to shut this off
107
        if self.no_secure_file_priv:
108
            self.secure_file_string = ''
109
        else:
110
            self.secure_file_string = "--secure-file-priv='%s'" %(self.vardir)
111
        self.user_string = '--user=root'
112
113
        self.initialize_databases()
114
        self.take_db_snapshot()
115
116
        if self.debug:
117
            self.logging.debug_class(self)
118
119
    def report(self):
120
        """ We print out some general useful info """
121
        report_values = [ 'name'
122
                        , 'master_port'
123
                        , 'drizzle_tcp_port'
124
                        , 'mc_port'
125
                        , 'pbms_port'
2144.1.1 by patrick crews
Overhaul of code. We can run rabbitmq : ) We now better encapsulate a per-executor working environment = one step closer to --parallel >: ) using subprocess goodness for server control
126
                        , 'rabbitmq_node_port'
2088.9.1 by patrick crews
Updated tree so that test-run.pl and test-run.py may live together in peace for a time
127
                        , 'vardir'
128
                        , 'status'
129
                        ]
2194.2.1 by patrick crews
Integrated randgen with dbqp. We now have mode=randgen and a set of randgen test suites (very basic now). Output = same as dtr : ) We also have mode=cleanup to kill any servers we have started. Docs updates too. Gendata utility allows us to populate test servers
130
        self.logging.info("%s server:" %(self.owner))
2088.9.1 by patrick crews
Updated tree so that test-run.pl and test-run.py may live together in peace for a time
131
        for key in report_values:
132
          value = vars(self)[key] 
133
          self.logging.info("%s: %s" %(key.upper(), value))
134
2144.1.1 by patrick crews
Overhaul of code. We can run rabbitmq : ) We now better encapsulate a per-executor working environment = one step closer to --parallel >: ) using subprocess goodness for server control
135
    def get_start_cmd(self):
136
        """ Return the command string that will start up the server 
137
            as desired / intended
138
 
2088.9.1 by patrick crews
Updated tree so that test-run.pl and test-run.py may live together in peace for a time
139
        """
2151.8.4 by patrick crews
gdb! : ) We try to set things up to launch the native Mac Terminal if we are running on Darwin
140
141
        server_args = [ self.process_server_options()
142
                      , "--mysql-protocol.port=%d" %(self.master_port)
143
                      , "--mysql-protocol.connect-timeout=60"
144
                      , "--innodb.data-file-path=ibdata1:20M:autoextend"
145
                      , "--sort-buffer-size=256K"
146
                      , "--max-heap-table-size=1M"
147
                      , "--mysql-unix-socket-protocol.path=%s" %(self.socket_file)
148
                      , "--pid-file=%s" %(self.pid_file)
149
                      , "--drizzle-protocol.port=%d" %(self.drizzle_tcp_port)
2167.4.1 by patrick crews
Fixed default-storage-engine option, minor fixes + added repeat option, will run each testcase in a sequence times
150
                      , "--default-storage-engine=%s" %(self.default_storage_engine)
2151.8.4 by patrick crews
gdb! : ) We try to set things up to launch the native Mac Terminal if we are running on Darwin
151
                      , "--datadir=%s" %(self.datadir)
152
                      , "--tmpdir=%s" %(self.tmpdir)
153
                      , self.secure_file_string
154
                      , self.user_string
155
                      ]
156
157
        if self.gdb:
158
            server_args.append('--gdb')
159
            return self.system_manager.handle_gdb_reqs(self, server_args)
160
        else:
161
            return "%s %s %s & " % ( self.cmd_prefix
162
                                   , self.server_path
163
                                   , " ".join(server_args)
164
                                   )
165
2144.1.1 by patrick crews
Overhaul of code. We can run rabbitmq : ) We now better encapsulate a per-executor working environment = one step closer to --parallel >: ) using subprocess goodness for server control
166
167
    def get_stop_cmd(self):
168
        """ Return the command that will shut us down """
2088.9.1 by patrick crews
Updated tree so that test-run.pl and test-run.py may live together in peace for a time
169
        
2144.1.1 by patrick crews
Overhaul of code. We can run rabbitmq : ) We now better encapsulate a per-executor working environment = one step closer to --parallel >: ) using subprocess goodness for server control
170
        return "%s --user=root --port=%d --shutdown " %(self.drizzle_client_path, self.master_port)
2088.9.1 by patrick crews
Updated tree so that test-run.pl and test-run.py may live together in peace for a time
171
           
172
2144.1.1 by patrick crews
Overhaul of code. We can run rabbitmq : ) We now better encapsulate a per-executor working environment = one step closer to --parallel >: ) using subprocess goodness for server control
173
    def get_ping_cmd(self):
174
        """Return the command string that will 
175
           ping / check if the server is alive 
176
177
        """
178
179
        return "%s --ping --port=%d --user=root" % (self.drizzle_client_path, self.master_port)
2088.9.1 by patrick crews
Updated tree so that test-run.pl and test-run.py may live together in peace for a time
180
2194.2.1 by patrick crews
Integrated randgen with dbqp. We now have mode=randgen and a set of randgen test suites (very basic now). Output = same as dtr : ) We also have mode=cleanup to kill any servers we have started. Docs updates too. Gendata utility allows us to populate test servers
181
    def is_started(self):
182
        """ Determine if the server is up and running - 
183
            this may vary from server type to server type
2088.9.1 by patrick crews
Updated tree so that test-run.pl and test-run.py may live together in peace for a time
184
185
        """
2194.2.1 by patrick crews
Integrated randgen with dbqp. We now have mode=randgen and a set of randgen test suites (very basic now). Output = same as dtr : ) We also have mode=cleanup to kill any servers we have started. Docs updates too. Gendata utility allows us to populate test servers
186
187
        # We experiment with waiting for a pid file to be created vs. pinging
188
        # This is what test-run.pl does and it helps us pass logging_stats tests
189
        # while not self.ping_server(server, quiet=True) and timer != timeout:
190
191
        return self.system_manager.find_path( [self.pid_file]
192
                                            , required=0)
193
194
    def create_slave_config_file(self):
195
       """ Create a config file suitable for use
196
           with the slave-plugin.  This allows
197
           us to tie other servers in easily
198
199
       """
200
2260.1.1 by David Shrewsbury
Revert multi-master code
201
       config_data = [ "master-host=127.0.0.1"
2194.2.1 by patrick crews
Integrated randgen with dbqp. We now have mode=randgen and a set of randgen test suites (very basic now). Output = same as dtr : ) We also have mode=cleanup to kill any servers we have started. Docs updates too. Gendata utility allows us to populate test servers
202
                     , "master-port=%d" %self.master_port
203
                     , "master-user=root"
204
                     , "master-pass=''"
205
                     , "max-reconnects=100"
2239.7.4 by patrick crews
Tweaks to drizzled.py's slave.cnf file to play with multi-master rpl
206
                     #, "seconds-between-reconnects=20"
2194.2.1 by patrick crews
Integrated randgen with dbqp. We now have mode=randgen and a set of randgen test suites (very basic now). Output = same as dtr : ) We also have mode=cleanup to kill any servers we have started. Docs updates too. Gendata utility allows us to populate test servers
207
                     ]
208
       outfile = open(self.slave_config_file,'w')
209
       for line in config_data:
210
           outfile.write("%s\n" %(line))
211
       outfile.close()
212
213
214
215
2088.9.1 by patrick crews
Updated tree so that test-run.pl and test-run.py may live together in peace for a time
216
                  
2194.2.1 by patrick crews
Integrated randgen with dbqp. We now have mode=randgen and a set of randgen test suites (very basic now). Output = same as dtr : ) We also have mode=cleanup to kill any servers we have started. Docs updates too. Gendata utility allows us to populate test servers
217
218
219
220
221
 
2088.9.1 by patrick crews
Updated tree so that test-run.pl and test-run.py may live together in peace for a time
222
         
223
224
225
226