Archive for the 'Linux' Category

Page 3 of 4

Probe average cpu utilisation (MRTG)

Published on October 21, 2004 in Linux. 1 Comment

There are two main tools to keep track of your CPU usage: top and vmstat.

  • top is an interactive tool: it shows you the CPU usage of each process, as well as overall statistics, updated every 5 seconds. It’s good for hands-on checking.

    #top 17:18:34 up 2 days, 8:14, 3 users, load average: 0.00, 0.00, 0.00
    47 processes: 46 sleeping, 1 running, 0 zombie, 0 stopped
    CPU states: 0.1% user 0.1% system 0.0% nice 0.0% iowait 99.6% idle
    Mem: 1030872k av, 1022256k used, 8616k free,
    0k shrd, 104844k buff
    777088k actv, 12k in_d, 22296k in_c
    Swap: 2048276k av, 8120k used, 2040156k free
    640080k cached
    PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
    30776 root 19 0 1140 1140 852 R 0.9 0.1 0:00 0 top
    1 root 15 0 504 464 436 S 0.0 0.0 0:03 0 init (...)
    But say you want to get just one number (percentage) back, so you can use it for logging.
  • vmstat wil give you the following output:

    #vmstat
    procs memory swap io system cpu
    r b w swpd free buff cache si so bi bo in cs us sy id
    0 0 0 7964 8804 104712 640224 0 0 2 16 129 27 0 0 100

    You can run vmstat 1 5 to get 5 consecutive measurements (1 second apart). The number we want is the average CPU usage, or (100% – idle). The following command will do the job:
    #vmstat 1 5 | gawk "/0/ {tot=tot+1; id=id+$16} END {print 100 - id/tot}"
    gives
    0.4

Estimate # of lines in a log file

Published on October 21, 2004 in Linux. Closed

Let’s say you need an (approximate) count of the number of lines in a huge file. The most obvious way of calculating this would be using wc, but this actually can be quite slow:
# time wc -l /var/log/squid/access.log
2812824 /var/log/squid/access.log
real 0m43.988s
(counting is done at 64.000 lines/sec)

Running wc without the -l (only count lines) would be ever slower because it would also count the words, instead of just the LF (linefeed) characters. But using wc -c is very fast! This is because the filesystem keeps track of each file’s filesize (= number of characters/bytes), so the file does not even have to be read to give this number. Can we estimate the # of lines from the # of bytes?

For the type of file we are talking about here (a Squid log file) there actually is a way. The file is more or less ‘square’, meaning that every line is about the same length (it contains date, status, URL, …).
If we take the beginning of the file (the first 10000 lines):
# head -10000 /var/log/squid/access.log | wc
10000 100000 1775257
we see that every line is about 177 chars long.

The end of the file (the last 10000 lines):
# tail -10000 /var/log/squid/access.log | wc
10000 100000 2047887
gives us a number of 204 chars/line.

Let’s take some more data and combine both:
# ( head -50000 /var/log/squid/access.log ; tail -50000 /var/log/squid/access.log ) | wc
100000 1000000 19488905
which gives us an average of 195 chars/line.

A file size of 533.229.920 bytes (533MB) would lead us to estimate the # of lines to 2.734.512, where the actual # of lines is 2.818.184 (3% difference). That is: we lose 3% accuracy but the calculation takes almost no CPU time, instead of 45 seconds. This might be a trade-off you are willing to accept!

Calculate hit rate from a log file

Published on October 21, 2004 in Linux. 1 Comment

You have a huge file that contains one line per request/transaction. Some of the lines are of one type (e.g. ‘HIT’), some of another (e.g. MISS). Let’s say you want to calculate the hitrate, but as fast as possible.
We take a Squid log file of about 140MB. How long does it take to count how many lines it has?
# time wc -l /var/log/squid/access.log
845212 /var/log/squid/access.log
real 0m6.523s (about 21.4 MB/s or 130.000 lines/s)

And now let’s just filter out the lines containing ‘HIT’ and count those:
#time sh -c "grep -i HIT /var/log/squid/access.log | wc -l"
Wow! This takes ages (I stopped it after 15 minutes) and the grep takes 100% CPU all the time. So let’s look for another solution.

Maybe gawk? First let’s see if it is much slower than wc -l for counting lines:
# time gawk "END {print NR}" /var/log/squid/access.log
845907
real 0m26.129s (5.3 MB/s or 32.000 lines/s – 4 times slower)
And now let it count the hits too:
]# time gawk "BEGIN {hit=0} /HIT/ {hit = hit+1} END {print hit/NR*100}" '/var/log/squid/access.log'
84.5023
real 0m32.836s (4MB/s or 25.000 lines/s – slow but acceptable)

Do we actually need a count on the whole file? What if we just took the last (i.e. most recent) 100.000 lines? The result would be a better indication of what the current hit rate is, and the speed of calculation would be more predictable.
# time sh -c "tail -100000 /var/log/squid/access.log | gawk 'BEGIN {hit=0} /HIT/ {hit = hit+1} END {print hit/NR*100}'"
92.305
real 0m3.332s (30.000 lines/s)

It is actually a bit slower the first time you run it, probably due to disk or filesystem caching. So if you want your hit rate calculation to take less than 2 seconds, you could take the last 50.000 lines. Done!

Squid: list top X referers

Published on October 19, 2004 in Linux. Closed

If your Squid server logs the referers of its request (i.e.
1. you’ve configured squid-cache with --enable-referer-log before compiling and
2. you’ve included a referer_log /var/log/squid/referer.log in your squid.conf file),
you can easily show top 50 of most popular referers with a simple Bourne shell:
#!/bin/bash

  1. this script is ‘top_referers.sh’
  2. (c) 2004 Peter Forret – Open Source
    REFERERS=/var/log/squid/referer.log
    OUTPUT=/var/www/html/stats/referer.txt
    MAXLINES=50(
    echo REPORT MADE AT `date`
    echo =============================
    $OUTPUT

Then add it to your crontab:
10 * * * * /(path)/top_referers.sh
and you have an hourly updated stat!
Add a little HTML formatting if you’re aesthetically demanding!