A simple tip to analyse big logs by chaining grep -v on terminal

Sample Premise

Let's say you have logged on to your remote server via ssh and you have a log of around 200MB or more to analyse. To top that you have to do it on your server without the help of a graphical user interface. (based on a true story).

Sample Log

To make it easier you can download a sample log that we will be using here at the following link http://www.monitorware.com/en/logsamples/apache.php it should look like the following:

e.g. cat sample.log

Grep

grep is a power tool which performs pattern matching on output that is piped through it. If we cat the file it should like something above. As you will notice with many log files things get repeated, usually things you are not looking for. Specially in the case where you are trying to find what has gone wrong. 

Now if you knew for example that you are trying to look for "File does not exist" in the logs you could do: 

cat sample.log | grep "File does not exist"

this would output all the "File does not exist" lines in the log. But if you do that you could miss out on other errors.

Grep -v

In comes -v the inverse match parameter. Using the inverse match parameter we can remove repeating patterns that we don't want to see for example if we did(just an example).

cat sample.log | grep -v "Connection reset by peer" 

we get the following:

Notice how we have everything except the connection reset by peer.

But we can go further by chaining and piping grep -v further let's remove GET /twiki

cat sample.log | grep -v "Connection reset by peer" | grep -v "GET /twiki"

You can continue like that by removing redundant lines then at the end of you want to then pipe everything to a final log file for review.

cat sample.log | grep -v "Connection reset by peer" | grep -v "GET /twiki" > final.log

This method can allow you to easily decrease the size of logs and also help you with finding errors. Hope that this helps (it certainly has for me today)