Creative process is always fascinating and mysterious to me, especially the ideas people come up to express themselves. Sometimes I wonder if there is a limit to one’s imagination. This video from ‘OK Go’ is just amazing. Coming up with a creative idea, defining the constrains for it and executing the idea to create a complex music video is amazing.
Of-course check out the TED video behind this music video
In my current project all our logs are mounted onto a Linux machine for easy access and they are numbered systematically like server 1 to 6 with side A and side B. We have a daily append-er for our application logs, so at any given time we will have file’s with same name’s on all the servers. For Ex.
/opt/logs/server1_A/log_2010-03-12.log
/opt/logs/server1_B/log_2010-03-12.log
……
……
/opt/logs/server6_A/log_2010-03-12.log
/opt/logs/server6_B/log_2010-03-12.log
Some interesting things can be done with this kind of setup. To start off, to list all the files of a particular day one can use Square Bracket Wild cards.
ls -all /opt/logs/das[1-6]_[AB]/log.2010-03-15.log ls -all /opt/logs/das[123456]_[AB]/log.2010-03-15.log
There may be cases when one has to break the outer for loop when a condition get’s satisfied in the inner for loop.
In the following example code, I am finding the country where the state belongs to..
Country rc = null;
boolean stop = false;
for (Country country : countries) {
if(stop) break;
for (State state : country.getStates()) {
if(state.getName().equalsIgnoreCase(stateName)) {
rc = country; stop = true; break;
}
}
}
In the following example I am naming my external for loop and passing that name to the break statement.
Country rc = null;
COUNTRY:
for (Country country : countries) {
for (State state : country.getStates()) {
if(state.getName().equalsIgnoreCase(stateName)) {
rc = country; break COUNTRY;
}
}
}
Integrating iBATIS to an application using spring framework is pretty straight forward. Testing a DAO code with live database will make it an integration test.
In general I write two kinds of tests for any DAO code
Following is an example DAO code. We are using iBATIS with spring. Code can be much more complex with multiple calls to database and can have some logic inside it.
import java.util.List;
import org.springframework.orm.ibatis.SqlMapClientTemplate;
import org.springframework.orm.ibatis.support.SqlMapClientDaoSupport;
public class SomeDAODBImpl extends SqlMapClientDaoSupport {
public List<Customer> lookupCustomers(String zip) {
List<Customer> customerList = null;
try {
SqlMapClientTemplate template = getSqlMapClientTemplate();
customerList = template.queryForList("Some-Ibatis-Statement-Name", zip);
} catch (Exception ex) {
//TODO: Do something with the caught exception
}
return customerList;
}
}
To unit test this code one have to create a mock datasource, session and client. Following is an example unit test code.
We are having an issue where the slave instance of MySql stops replicating with an error “Deadlock found when trying to get lock; try restarting transaction’ on query“. All our tables use InnoDB engine with default settings. So our current configuration uses Statement based replication. The table in question that is causing this problem is using auto-increment for primary key.
After bit of investigation realized that when there are many insert statements being replicated on the slave for the same table, each insert obtains an auto-increment lock and releases it only after the statement is executed. Supposedly this is a problem for statement based replication. So when multiple threads are inserting into the same table in parallel we can get this deadlock.
Of-course we need to do more investigation and fine tune our MySql infrastructure. So when ever the replication stops we have to restart the slave. Blog post Monitoring and Resetting MySQL Replication details how it can be done.
To do this in an automated way we came up with a groovy scripts. A cron job is scheduled for every 15 minutes to execute this script. Following are some higher level steps.
If replication is not working
–stop slave
–reset slave
–issue change master command
–start slave
–check again if replication is working fine, if so
—-send an email that every thing is good and reset was a success
–else
—-send an email slave reset is a failure
else
–do nothing
Following is the groovy code..
(more…)
Using java.util.concurrent api, creating asynchronous tasks and executing them in parallel is quite easy.
Following is an example where I create 2 task’s Callable1 and Callable2 and execute them in parallel with Fixed ThreadPoolExecutor. In real world there will be some complex code logic for each of these tasks, and there might also be some logic where these two tasks are executed (callTasksInParallel).
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
public class MultiThreadExample {
private static final int threadTimeOut = 10;
private static final int maxThreadPool = 20;
private ExecutorService pool = null;
public MultiThreadExample() {
pool = Executors.newFixedThreadPool(maxThreadPool);
}
public void setPool(ExecutorService pool) {
this.pool = pool;
}
public void callTasksInParallel() {
Future<String> callable1Future = pool.submit(new Callable1());
Future<String> callable2Future = pool.submit(new Callable2());
try {
String response1 = callable1Future.get(threadTimeOut, TimeUnit.SECONDS);
String response2 = callable2Future.get(threadTimeOut, TimeUnit.SECONDS);
System.out.println(response1);
System.out.println(response2);
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
} catch (TimeoutException e) {
e.printStackTrace();
}
}
public static void main(String[] args) {
MultiThreadExample multiThreadExample = new MultiThreadExample();
multiThreadExample.callTasksInParallel();
}
}
class Callable1 implements Callable<String> {
public String call() throws Exception {
return "I am returned from Callable1";
}
}
class Callable2 implements Callable<String> {
public String call() throws Exception {
return "I am returned from Callable2";
}
}
Unit testing MultiThreadExample by itself will require mocking of ExecuterService and having Future mocks. Following is an example Junit test code that tests callTasksInParallel as a unit without a dependency for Callable1 and Callable2 classes.
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import org.easymock.EasyMock;
import org.junit.Before;
import org.junit.Test;
public class MultiThreadExampleTest {
private ExecutorService executorServiceMock;
private MultiThreadExample multiThreadExample;
@Before
public void setUp() throws Exception {
executorServiceMock = EasyMock.createMock(ExecutorService.class);
multiThreadExample = new MultiThreadExample();
EasyMock.makeThreadSafe(executorServiceMock, true);
}
@SuppressWarnings("unchecked")
@Test
public void testCallTasksInParallel() throws Exception {
multiThreadExample.setPool(executorServiceMock);
//Future mock for Callable1 and set the expectation
Future<String> callable1Mock = (Future<String>) EasyMock.createMock(Future.class);
EasyMock.expect(callable1Mock.get(EasyMock.anyLong(), EasyMock.isA(TimeUnit.class))).andReturn("Mock response from callable1");
EasyMock.replay(callable1Mock);
//Future mock for Callable2 and set the expectation
Future<String> callable2Mock = (Future<String>) EasyMock.createMock(Future.class);
EasyMock.expect(callable2Mock.get(EasyMock.anyLong(), EasyMock.isA(TimeUnit.class))).andReturn("Mock response from callable2");
EasyMock.replay(callable2Mock);
//Submit the created futures to the ExecuterService mock and replay them
EasyMock.expect(executorServiceMock.submit(EasyMock.isA(Callable1.class))).andReturn(callable1Mock);
EasyMock.expect(executorServiceMock.submit(EasyMock.isA(Callable2.class))).andReturn(callable2Mock);
EasyMock.replay(executorServiceMock);
multiThreadExample.callTasksInParallel();
EasyMock.verify(callable1Mock);
EasyMock.verify(callable2Mock);
EasyMock.verify(executorServiceMock);
}
}
Above approach can be used to Unit test all the exception scenarios also.
It’s pretty easy to read a key=value style properties file using some sed and grep magic in a shell script
Example properties file:
DAS01-8090_workspace=/opt/home/dasadmin/workspace/trunk DAS01-8090_profile=development-resources DAS01-8090_serverLocation=/opt/apache/tomcat2 DAS01-8090_serverType=local #DAS03-8090_workspace=/opt/home/dasadmin/workspace/branches/Production-2009-12-09 DAS03-8090_workspace=/opt/home/dasadmin/workspace/branches/Production-2010-01-14 DAS03-8090_profile=nuance-resources DAS03-8090_serverLocation=/opt/apache/tomcat-iteration DAS03-8090_serverType=remote DAS03-8090_remoteMachine=dasadmin@pacdcdtadeva03 DAS03-8080_workspace=/opt/home/dasadmin/workspace/branches/Production-2010-01-14 DAS03-8080_profile=integration-resources DAS03-8080_serverLocation=/opt/apache/tomcat DAS03-8080_serverType=remote DAS03-8080_remoteMachine=dasadmin@pacdcdtadeva03
Following script takes in a profile name and read’s the profile information accordingly
workspace=`sed '/^\#/d' environment.properties | grep "$1_workspace" | tail -n 1 | sed 's/^.*=//'` profile=`sed '/^\#/d' environment.properties | grep "$1_profile" | tail -n 1 | sed 's/^.*=//'` serverLocation=`sed '/^\#/d' environment.properties | grep "$1_serverLocation" | tail -n 1 | sed 's/^.*=//'` serverType=`sed '/^\#/d' environment.properties | grep "$1_serverType" | tail -n 1 | sed 's/^.*=//'` remoteMachine=`sed '/^\#/d' environment.properties | grep "$1_remoteMachine" | tail -n 1 | sed 's/^.*=//'`
sed ‘/^\#/d’ — This will skip any line’s that start with ‘#’. ‘/d’ is similar to ‘-v’ option in grep.
grep “$1_workspace” — will grep for the respective profile line
tail -n 1 — if for some reason there are multiple grep matches pick only first one.
sed ‘s/^.*=//’ — Substitution magic is happening over here. I am replacing the string part till ‘=’ with empty string.
So if I pass in DAS03-8090 as profile name to the script, a value ‘/opt/home/dasadmin/workspace/branches/Production-2010-01-14′ will be set to the variable ‘workspace’
Its pretty simple to write a web-service client in PHP using SoapClient library that comes bundled with PHP.
By default when a SoapClient is created, the client will use the soap:address location from the WSDL. There might be different reasons why the address location inside the WSDL is different from the actual WSDL location. Its always safe to manually set the web-service location for the client.
one can do this by calling _setLocation() on the client.
<?php
ini_set("memory_limit","128M");
$soapUrl = "some wsdl url";
$client = new SoapClient( $soapUrl, array('connection_timeout' => 200));
$client->__setLocation($soapUrl);
$params->telephone=7065555558;
$return=$client->lookupCustomerByTN($params);
var_export($return);
?>
There are different ways of installing multiple operating systems on a single machine. One way is to use a virtualization software. One can have a host operating system running multiple guest operating system as virtual machines. For ex. Windows XP can be running Ubuntu, Windows 7 as virtual machines. This will effectively give access to multiple operating systems with out an actual reboot of entire machine.
There are number of commercial as well as free virtualization software’s available (Extensive list), but my favorite is Virtual Box from sun micro-systems. VirtualBox is a powerful virtualization tool available for free to download and easy to use.
To install Ubuntu as a guest operating system on Windows XP one has to first download and install VirtualBox on windows. Then one has to download Ubuntu ISO in to a hard disk or burn into a CD. One’s VirtualBox is installed, using virtual machine wizard one can install Ubuntu on the windows. Installation and configuration of new OS is quite self explanatory.
Logging is an important aspect in writing any software application. Logs properly utilized provide valuable information related to code flow. Mining logs is a way to gather data related to performance of the application and to debug the application.
As part of our current application we wrote tools in Perl and Groovy to mine the logs to collect statistics related to the application. In these tools, we primarily look for string patterns and if some one unintentionally changes the log statements inside the code it breaks our tools also. It’s always beneficial to surround SOME of the log statements with Junit test.
Extending AppenderSkeleton to create a custom append-er.
import java.util.ArrayList;
import java.util.List;
import org.apache.log4j.AppenderSkeleton;
import org.apache.log4j.spi.LoggingEvent;
public class TestingAppender extends AppenderSkeleton {
//We are collecting all the log messages in to a list
private static List messages = new ArrayList();
// This method is called when ever any logging happens
// We are simply taking the log message and adding to our list
@Override
protected void append(LoggingEvent event) {
messages.add(event.getRenderedMessage());
}
//This method is called when the appender is closed.
//Gives an opportunity to clean up resources
public void close() {
messages.clear();
}
public boolean requiresLayout() {
return false;
}
public static String[] getMessages() {
return (String[]) messages.toArray(new String[messages.size()]);
}
public static void clear() {
messages.clear();
}
}
Then we need to write couple of methods to setup the custom append-er and remove it. As part of setUp and tearDown methods of JUnit tests appropriate methods can be called.
private void setupLog4j(TestingAppender appender) {
//Get the root logger for the
Logger root = Logger.getRootLogger();
//If there are no appenders, means Log4j is not initialized properly
if (!root.getAllAppenders().hasMoreElements()) {
System.out.println("*******Log4j is not initilized**********");
} else {
//Set to appropriate log level and add custom appender
root.setLevel(Level.ERROR);
root.addAppender(appender);
}
}
private void resetLog4j(TestingAppender appender) {
TestingAppender.clear();
Logger root = Logger.getRootLogger();
root.setLevel(Level.FATAL);
root.removeAppender(appender);
}
As part of tests we can call TestingAppender.getMessages() to get the list of messages and assert for the appropriate log message. This way we can have unit tests for log statements also.
Theme: Shocking Blue Green. Blog at WordPress.com.
