a peek into my MIND

January 26, 2010

Groovy script to restart stopped MySql slave

Filed under: Groovy & Grails — Tags: , , — Bharat Kondeti @ 7:08 pm

We are having an issue where the slave instance of MySql stops replicating with an error “Deadlock found when trying to get lock; try restarting transaction’ on query“. All our tables use InnoDB engine with default settings. So our current configuration uses Statement based replication. The table in question that is causing this problem is using auto-increment for primary key.

After bit of investigation realized that when there are many insert statements being replicated on the slave for the same table, each insert obtains an auto-increment lock and releases it only after the statement is executed. Supposedly this is a problem for statement based replication. So when multiple threads are inserting into the same table in parallel we can get this deadlock.

Of-course we need to do more investigation and fine tune our MySql infrastructure. So when ever the replication stops we have to restart the slave. Blog post Monitoring and Resetting MySQL Replication details how it can be done.

To do this in an automated way we came up with a groovy scripts. A cron job is scheduled for every 15 minutes to execute this script. Following are some higher level steps.

If replication is not working
–stop slave
–reset slave
–issue change master command
–start slave
–check again if replication is working fine, if so
—-send an email that every thing is good and reset was a success
—-send an email slave reset is a failure
–do nothing

Following is the groovy code..


January 20, 2010

Unit testing multi threaded code with EasyMock

Filed under: Java, Testing — Tags: , , , — Bharat Kondeti @ 4:02 pm

Using java.util.concurrent api, creating asynchronous tasks and executing them in parallel is quite easy.

Following is an example where I create 2 task’s Callable1 and Callable2 and execute them in parallel with Fixed ThreadPoolExecutor. In real world there will be some complex code logic for each of these tasks, and there might also be some logic where these two tasks are executed (callTasksInParallel).

import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;

public class MultiThreadExample {
	private static final int threadTimeOut = 10;
	private static final int maxThreadPool = 20;

	private ExecutorService pool = null;

	public MultiThreadExample() {
		pool = Executors.newFixedThreadPool(maxThreadPool);
        public void setPool(ExecutorService pool) {
		this.pool = pool;

	public void callTasksInParallel() {
		Future<String> callable1Future = pool.submit(new Callable1());
		Future<String> callable2Future = pool.submit(new Callable2());

		try {
			String response1 = callable1Future.get(threadTimeOut, TimeUnit.SECONDS);
			String response2 = callable2Future.get(threadTimeOut, TimeUnit.SECONDS);

		} catch (InterruptedException e) {
		} catch (ExecutionException e) {
		} catch (TimeoutException e) {
	public static void main(String[] args) {
		MultiThreadExample multiThreadExample = new MultiThreadExample();

class Callable1 implements Callable<String> {
	public String call() throws Exception {
		return "I am returned from Callable1";

class Callable2 implements Callable<String> {
	public String call() throws Exception {
		return "I am returned from Callable2";

Unit testing MultiThreadExample by itself will require mocking of ExecuterService and having Future mocks. Following is an example Junit test code that tests callTasksInParallel as a unit without a dependency for Callable1 and Callable2 classes.

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import org.easymock.EasyMock;
import org.junit.Before;
import org.junit.Test;

public class MultiThreadExampleTest {
	private ExecutorService executorServiceMock;
	private MultiThreadExample multiThreadExample;

	public void setUp() throws Exception {
		executorServiceMock = EasyMock.createMock(ExecutorService.class);
		multiThreadExample = new MultiThreadExample();
		EasyMock.makeThreadSafe(executorServiceMock, true);

	public void testCallTasksInParallel() throws Exception {

                //Future mock for Callable1 and set the expectation
		Future<String> callable1Mock = (Future<String>) EasyMock.createMock(Future.class);
		EasyMock.expect(callable1Mock.get(EasyMock.anyLong(), EasyMock.isA(TimeUnit.class))).andReturn("Mock response from callable1");

                //Future mock for Callable2 and set the expectation
		Future<String> callable2Mock = (Future<String>) EasyMock.createMock(Future.class);
		EasyMock.expect(callable2Mock.get(EasyMock.anyLong(), EasyMock.isA(TimeUnit.class))).andReturn("Mock response from callable2");

                //Submit the created futures to the ExecuterService mock and replay them


Above approach can be used to Unit test all the exception scenarios also.

January 18, 2010

Reading a properties file in linux shell script.

Filed under: General — Bharat Kondeti @ 10:12 pm

It’s pretty easy to read a key=value style properties file using some sed and grep magic in a shell script

Example properties file:




Following script takes in a profile name and read’s the profile information accordingly

workspace=`sed '/^\#/d' environment.properties | grep "$1_workspace"  | tail -n 1 | sed 's/^.*=//'`
profile=`sed '/^\#/d' environment.properties | grep "$1_profile"  | tail -n 1 | sed 's/^.*=//'`
serverLocation=`sed '/^\#/d' environment.properties | grep "$1_serverLocation"  | tail -n 1 | sed 's/^.*=//'`
serverType=`sed '/^\#/d' environment.properties | grep "$1_serverType"  | tail -n 1 | sed 's/^.*=//'`
remoteMachine=`sed '/^\#/d' environment.properties | grep "$1_remoteMachine"  | tail -n 1 | sed 's/^.*=//'`

sed ‘/^\#/d’ — This will skip any line’s that start with ‘#’. ‘/d’ is similar to ‘-v’ option in grep.
grep “$1_workspace” — will grep for the respective profile line
tail -n 1 — if for some reason there are multiple grep matches pick only first one.
sed ‘s/^.*=//’ — Substitution magic is happening over here. I am replacing the string part till ‘=’ with empty string.

So if I pass in DAS03-8090 as profile name to the script, a value ‘/opt/home/dasadmin/workspace/branches/Production-2010-01-14’ will be set to the variable ‘workspace’

January 4, 2010

WebService client in PHP

Filed under: General — Bharat Kondeti @ 8:38 pm

Its pretty simple to write a web-service client in PHP using SoapClient library that comes bundled with PHP.

By default when a SoapClient is created, the client will use the soap:address location from the WSDL. There might be different reasons why the address location inside the WSDL is different from the actual WSDL location. Its always safe to manually set the web-service location for the client.

one can do this by calling _setLocation() on the client.

$soapUrl = "some wsdl url";

$client = new SoapClient( $soapUrl, array('connection_timeout' => 200));




Blog at WordPress.com.