a peek into my MIND

March 15, 2010

There is always something to learn in Linux ‘Bash’ shell

Filed under: General — Tags: , , , — Bharat Kondeti @ 11:08 pm

In my current project all our logs are mounted onto a Linux machine for easy access and they are numbered systematically like server 1 to 6 with side A and side B. We have a daily append-er for our application logs, so at any given time we will have file’s with same name’s on all the servers. For Ex.

/opt/logs/server1_A/log_2010-03-12.log
/opt/logs/server1_B/log_2010-03-12.log
……
……
/opt/logs/server6_A/log_2010-03-12.log
/opt/logs/server6_B/log_2010-03-12.log

Some interesting things can be done with this kind of setup. To start off, to list all the files of a particular day one can use Square Bracket Wild cards.

ls -all /opt/logs/das[1-6]_[AB]/log.2010-03-15.log

ls -all /opt/logs/das[123456]_[AB]/log.2010-03-15.log

Both commands do the same, one is a range while other is a explicit definition of wild-cards. Above syntax is pretty concise and very powerful. If I want to search for a particular string on all the log files for statistical analysis all I have to do is

grep ‘search-string’ /opt/logs/das[1-6]_[AB]/log.2010-03-15.log

#To get word count
grep ‘search-string’ /opt/logs/das[1-6]_[AB]/log.2010-03-15.log | wc -l 

Other useful Linux command is cut. Sometimes using this command is much simpler and easier to manipulate strings based on a delimiter. For Ex in the following log line I need to pick up account number. One way is to split the string with ‘,’ pick up 4th element, pipe it to cut again and split by ‘=’ and pick up 4th element resulting in a account number I am looking for.

10:42:43,089 03/10 EE6133E80098CC039FC6586CBB10D037 WARN dfsdfs - method execution[execution(afsdfsdf)], exec time = 7551ms, input = [dfsfsfsfd[fsdfs=sdfsf[accountNumber=1920445063401,sdfs=],sdfsf=b,sdfsdfs=dsfsf[/code]]]; output = java.lang.NullPointerException

Following is sample command to get all the account numbers that threw NullPointerException.

grep ‘search-string’ /opt/logs/das[1-6]_[AB]/log.2010-03-15.log | cut –f4 –d”,” | cut –f4 –d”=”

# -f field to return
# -d delimiter

In my case first five digits of an account number identifies customer market. So to return market id’s I would pipe the results of above command again to cut command.

grep ‘search-string’ /opt/logs/das[1-6]_[AB]/log.2010-03-15.log | cut –f4 –d”,” | cut –f4 –d”=” | cut –c1-5

# -c cut range

To get unique market Id’s where the NullPointerException is happening, I can pipe the results again to a sort command

grep ‘search-string’ /opt/logs/das[1-6]_[AB]/log.2010-03-15.log | cut –f4 –d”,” | cut –f4 –d”=” | cut –c1-5 | sort -u

# -u unique

I feel Linux shell is so powerful we can do pretty much any thing.

Advertisements

Leave a Comment »

No comments yet.

RSS feed for comments on this post.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: