a peek into my MIND

March 15, 2010

There is always something to learn in Linux ‘Bash’ shell

Filed under: General — Tags: , , , — Bharat Kondeti @ 11:08 pm

In my current project all our logs are mounted onto a Linux machine for easy access and they are numbered systematically like server 1 to 6 with side A and side B. We have a daily append-er for our application logs, so at any given time we will have file’s with same name’s on all the servers. For Ex.


Some interesting things can be done with this kind of setup. To start off, to list all the files of a particular day one can use Square Bracket Wild cards.

ls -all /opt/logs/das[1-6]_[AB]/log.2010-03-15.log

ls -all /opt/logs/das[123456]_[AB]/log.2010-03-15.log

Both commands do the same, one is a range while other is a explicit definition of wild-cards. Above syntax is pretty concise and very powerful. If I want to search for a particular string on all the log files for statistical analysis all I have to do is

grep ‘search-string’ /opt/logs/das[1-6]_[AB]/log.2010-03-15.log

#To get word count
grep ‘search-string’ /opt/logs/das[1-6]_[AB]/log.2010-03-15.log | wc -l 

Other useful Linux command is cut. Sometimes using this command is much simpler and easier to manipulate strings based on a delimiter. For Ex in the following log line I need to pick up account number. One way is to split the string with ‘,’ pick up 4th element, pipe it to cut again and split by ‘=’ and pick up 4th element resulting in a account number I am looking for.

10:42:43,089 03/10 EE6133E80098CC039FC6586CBB10D037 WARN dfsdfs - method execution[execution(afsdfsdf)], exec time = 7551ms, input = [dfsfsfsfd[fsdfs=sdfsf[accountNumber=1920445063401,sdfs=],sdfsf=b,sdfsdfs=dsfsf[/code]]]; output = java.lang.NullPointerException

Following is sample command to get all the account numbers that threw NullPointerException.

grep ‘search-string’ /opt/logs/das[1-6]_[AB]/log.2010-03-15.log | cut –f4 –d”,” | cut –f4 –d”=”

# -f field to return
# -d delimiter

In my case first five digits of an account number identifies customer market. So to return market id’s I would pipe the results of above command again to cut command.

grep ‘search-string’ /opt/logs/das[1-6]_[AB]/log.2010-03-15.log | cut –f4 –d”,” | cut –f4 –d”=” | cut –c1-5

# -c cut range

To get unique market Id’s where the NullPointerException is happening, I can pipe the results again to a sort command

grep ‘search-string’ /opt/logs/das[1-6]_[AB]/log.2010-03-15.log | cut –f4 –d”,” | cut –f4 –d”=” | cut –c1-5 | sort -u

# -u unique

I feel Linux shell is so powerful we can do pretty much any thing.


Leave a Comment »

No comments yet.

RSS feed for comments on this post.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: