Generic structure of the .htaccess redirects

Redirect permanent /(old url) (new url)
Redirect ... (add all your one-2-one redirects here)
RedirectMatch permanent ^/old_stuff/.*html$ http://www.example.com/
RedirectMatch ... (add your catch-all redirects here)

RewriteEngine on
RewriteBase /blog/
RewriteRule ^([regex])$ http://blog.example.com/$1 [R,L]
RewriteRule ... (add all your variable redirects here)

EXAMPLE: old Blogger site (on your own server) to new Wordpress site
I’ve done a migration from a blog published by Blogger (via FTP) onto my own webspace, to a blog run by Wordpress. I’ve used the following Rewrite rules to handle the redirections.
* HOMEPAGE:
redirect /index.html and / to your new blog URL
Redirect permanent / http://blog.example.com/
Redirect permanent /index.html http://blog.example.com/

* FEED:
redirect e.g. /atom.xml to your Feedburner feed
Redirect permanent /atom.xml http://feeds.feedburner.com/(exampleblog)

* ARCHIVES:
redirect e.g. /archive/2005_03_posts.html to the new Wordpress archives
RedirectMatch permanent /archive/([0-9][0-9][0-9][0-9])_([0-9][0-9])_.*$ http://blog.example.com/$1/$2/

* POST PAGES:
This is tricky, because Blogger and Wordpress do not use exactly the same rules for constructing the text-like URL (the ‘post slug’). E.g a post called how-to-podcast-with-blogger-and.html on my old Blogger site became how-to-podcast-with-blogger-and-smartcast/ on the new Wordpress one. So what I did consisted of 2 type of rules:
a) redirecting individual pages
Redirect permanent 2004/10/how-to-podcast-with-blogger-and.html http://blog.example.com/2004/10/how-to-podcast-with-blogger-and-smartcast/
b) a generic rule for the others (this uses Rewrite instead of RedirectMatch!): each page is redirected to a search on the Wordpress blog within the correct month with the two first words of the title:
RewriteRule ^([0-9][0-9][0-9][0-9])/([0-9][0-9])/([a-z0-9]*)-([a-z0-9]*).*$ http://blog.example.com/$1/$2/?s=$3+$4 [R,L]
This method is far from perfect, but will bring visitors a lot closer to the right page. If you use pretty distinctive words for titles (e.g. “Myspace: bulletin and other spam“), chances are the right page show up first. If you start all your posts with “The ten best ways to …” then you will need a more sophisticated rule; e.g. using the 6th and 7th word:
RewriteRule ^([0-9][0-9][0-9][0-9])/([0-9][0-9])/[a-z0-9]*-[a-z0-9]*-[a-z0-9]*-[a-z0-9]*-[a-z0-9]*-([a-z0-9]*)-([a-z0-9]*).*$ http://blog.example.com/$1/$2/?s=$3+$4 [R,L]

Not losing the querystring
Redirect and RedirectMatch cannot redirect to a URL with a querystring (e.g. to newpage.php?param1=val1&param2=val2). For that you will need to use the RewriteRule. An example: redirect all links like test.asp?param=value on the old domain to the new domain while keeping all querystring parameters:
RewriteRule ^tools/test.asp\??(.*)$ http://web.example.com/tools/test.asp [L,QSA]
where the QSA = (query string append) keep existing querystring, and L = (last rule) stop looking further for rule matches.

No related posts.

if [ -f $DNSPRE ] ; then
cat $DNSPRE
| grep -w "PTR"
| gawk "BEGIN {OFS = "t" ;} {print $1,$2,$3,$4,";; FROM `basename $DNSPRE` "; }"
fi )
| sort -n
| uniq --check-chars=3


You would call it as follows:
revdns.sh 192.168.110 internal.example.com > new.db.192.168.110 and then replace the records of the original db.192.168.110 with the records of the new file. The script still requires manual intervention (you cannot pipe the result straight into a live Bind config file) but saves a lot of typing!

Example of the output:

201 IN PTR james.internal.example.be. ;; FROM db.internal.example.com
202 IN PTR wilbur.internal.example.be. ;; FROM db.internal.example.com
216 IN PTR appprd1.internal.example.com. ;; FROM db.192.168.110
217 IN PTR appprd2.internal.example.com. ;; FROM db.192.168.110
218 IN PTR appprd3.internal.example.com. ;; FROM db.192.168.110
219 IN PTR appprd4.internal.example.com. ;; FROM db.192.168.110
220 IN PTR appprd5.internal.example.com. ;; FROM db.192.168.110
221 IN PTR appprd6.internal.example.com. ;; FROM db.192.168.110


Related posts:

1. Calculate hit rate from a log file You have a huge file that contains one line per...
2. Probe disk performance (MRTG) The hdparam can be used to monitor the throughput speed...
3. Date formatting in GAWK: boot time I have one server with apparently an exceptional stability: #...

No related posts.

GOAL:

make an HTML scraper, i.e. a script that grabs another URL and outputs the results to the screen

TOOL:

let’s say … Perl (in my case: Perl 5.8 on RedHat)

INPUT:

a URL

OUTPUT:

the HTML code of that URL

The actual HTML retrieval is easy: you need get() from the LWP::Simple module:
use LWP::Simple;
my $page = get($url);

Some remarks:

#!/usr/bin/perl -w
use strict;
use CGI qw(:standard);
use LWP::Simple qw(!head);my $query = new CGI;
my $url = $query->param('url');
my $debug = 0;

print header();
if(length($url) > 0) {
print getpage($url);
} else {
showform();
}

sub getpage{
my $url = shift;
my $time1 = time();
debuginfo(“Scraping <a target=_blank href=’” . $url . “‘>link</a> …”);
my $page = get($url);
my $time2 = time();
debuginfo(“Time taken was <b>” . ($time2 – $time1) . “</b> seconds”);
debuginfo(“Total bytes scraped: <b>”. length($page)/1000 . “KB</b>” );
return $page;
}

sub debuginfo{
if ($debug > 0) {
my $text = shift;
print “<small>” , $text , “</small><br />n”;
}

}

sub showform{
print(“<html><head>”);
print(“<title>SCRAPER</title>”);
print(“<link rel=stylesheet type=text/css href=http://www.forret.com/blog/style.css>”);
print(“</head><body><center>n”);
print(“<form method=GET action=’scrape.pl’>”);
print(“URL: <input name=url type=text size=60 value=http://www.forret.com>”);
print(“<input type=submit></form>n”);
print(“</center></body></html>n”);
}

Next step: making sure image src= and hyperlink href keep on working (so convert relative links to absolute links!).

No related posts.

The cache manager (cachemgr.cgi) is a CGI utility for displaying statistics about the squid process as it runs. The cache manager is a convenient way to manage the cache and view statistics without logging into the server.
(from Squid FAQ)

The only thing is … it’s so ugly! It uses plain HTML and cannot be customized, the FAQ says. However, there is a way to do it:

I did something a bit different (I wanted to use the CSS of my own website), so I ‘ll show you the difference between the two versions.
In order to get to the following comparison, I did a strings cachemgr.cgi > cachemgr.txt to extract only the text parts, and I did a diff cachemgr.txt cachemgr2.txt to compare both files. You cannot do a file comparison of 2 binary files.

<em>173,174c173,174</em>
< <HTML><HEAD><TITLE>Cache Manager Interface</TITLE>
< <STYLE type="text/css"><!-- BODY{background-color:#ffffff;font-family:verdana,sans-serif} --></STYLE></HEAD>
---
> <HTML><HEAD><TITLE>Cache Manager (pforret)</TITLE>
> <link rel="stylesheet" type="text/css" href="http://www.forret.com/forret/forret.css" mce_href="http://www.forret.com/forret/forret.css" /> </HEAD>
<em>199c199</em>
< <STYLE type="text/css"><!-- BODY{background-color:#ffffff;font-family:verdana,sans-serif} TABLE{background-color:#333333;border:0pt;padding:0pt}TH,TD{background-color:#ffffff}--></STYLE>
---
> <link rel="stylesheet" type=text/css href="http://www.forret.com/forret/forret.css" mce_href="http://www.forret.com/forret/forret.css"><!-- TABLE{background-color:#333333;border:0pt;padding:0pt} TH,TD{background-color:#ffffff}--></STYLE>


Related posts:

1. Squid: list top X referers If your Squid server logs the referers of its request...

This would be an interesting performance metric to see plotted against time. So let’s convert it to a format ready for MRTG.

• The only numbers we need are the last ones: resulting speed. This can be parsed from the output as follows:
  #hdparm -tT /dev/hda | gawk -F = "/seconds/ { print $2}" 

  440.00 MB/sec   3.30 MB/sec

• if we could suppose that the results will always be in “MB/sec”, we could parse out the numbers with
  (...) | gawk "{print $1}"
  and then add a line to our MRTG config files to adjust the units:
  kMG[_]: M,G,T,P,X
  But let’s say that KB/sec or GB/sec speeds are possible.
• One gawk can do the conversion trick:
  #(...) | gawk "/GB/ {print $1*1000000000} /MB/ {print $1*1000000} /KB/ {print $1*1000}" 

  440000000 3300000

• To have a complete MRTG-ready output, we also add the boot time on line 3 and the name of the MRTG output on line 4
• Q: Do we need 2 gawks one after the other? Can’t one do it?
  A: You could do it in 1, I guess, but the parsing would be more complex. I use 2 because the FS (field separator) changes: the first gawk uses the ‘=’ character, the second uses the normal whitespace.

Related posts:

1. Probe average cpu utilisation (MRTG) There are two main tools to keep track of your...
2. Calculate hit rate from a log file You have a huge file that contains one line per...
3. Date formatting in GAWK: boot time I have one server with apparently an exceptional stability: #...

3:45pm  up 524 days,  1:22,  1 user,  load average: 0.44, 0.16, 0.13

Unfortunately I know this is not correct (I remember rebooting it some weeks ago). So what are other ways to get the date/time of the last boot?

Looking at the RedHat manuals, the following thing should work too:
# cat /proc/stat
cpu 33813143 210619911 30093342 59435750
cpu0 33813143 210619911 30093342 59435749
(...)
btime 1096157569
(...)

The btime gives us the last boot time in seconds since 1 Jan 1970. I can find and convert it with gawk:
# gawk "/btime/{ print (`date +%s` - $2) / (3600 * 24.0) ,"days -",strftime("%a %b %d %H:%M:%S %Z %Y",$2)}" /proc/stat
38.6473 days - Sun Sep 26 02:12:49 CEST 2004
Which gives us an uptime of 38,6 days – that looks more like it!

Another way of calculating the uptime:
# gawk "/cpu/ {print $1,($2 + $3 + $4 + $5)/(3600 * 24 * 100)}" /proc/stat
cpu 38.6515
cpu0 38.6515
Confirmation of the previous measurement!

# cat /proc/uptime
45282758.17 663091.26
The first number is the # of seconds since last boot. The other one (idle time) we don’t need. What is that in days?
# gawk "{print $1/(3600 * 24.0)}" /proc/uptime
524.106

This is where the wrong data is coming from! So I’ll ignore this data.

Remark: This server is one of my oldest ones and is still running Redhat 7.2 (Enigma). Looks like this bug was fixed in later versions of RedHat, since none of my other servers have it.

Related posts:

1. Probe disk performance (MRTG) The hdparam can be used to monitor the throughput speed...
2. Redhat versions: what am I running? If you manage multiple RedHat servers, or if you just...
3. Calculate hit rate from a log file You have a huge file that contains one line per...

• top is an interactive tool: it shows you the CPU usage of each process, as well as overall statistics, updated every 5 seconds. It’s good for hands-on checking.
  
#top 17:18:34 up 2 days, 8:14, 3 users, load average: 0.00, 0.00, 0.00
47 processes: 46 sleeping, 1 running, 0 zombie, 0 stopped
CPU states: 0.1% user 0.1% system 0.0% nice 0.0% iowait 99.6% idle
Mem: 1030872k av, 1022256k used, 8616k free,
0k shrd, 104844k buff
777088k actv, 12k in_d, 22296k in_c
Swap: 2048276k av, 8120k used, 2040156k free
640080k cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
30776 root 19 0 1140 1140 852 R 0.9 0.1 0:00 0 top
1 root 15 0 504 464 436 S 0.0 0.0 0:03 0 init (...)
  But say you want to get just one number (percentage) back, so you can use it for logging.
• vmstat wil give you the following output:
  
#vmstat
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 0 0 7964 8804 104712 640224 0 0 2 16 129 27 0 0 100


  You can run vmstat 1 5 to get 5 consecutive measurements (1 second apart). The number we want is the average CPU usage, or (100% – idle). The following command will do the job:
  #vmstat 1 5 | gawk "/0/ {tot=tot+1; id=id+$16} END {print 100 - id/tot}"
  gives
  0.4

Related posts:

1. Probe disk performance (MRTG) The hdparam can be used to monitor the throughput speed...
2. Date formatting in GAWK: boot time I have one server with apparently an exceptional stability: #...

Running wc without the -l (only count lines) would be ever slower because it would also count the words, instead of just the LF (linefeed) characters. But using wc -c is very fast! This is because the filesystem keeps track of each file’s filesize (= number of characters/bytes), so the file does not even have to be read to give this number. Can we estimate the # of lines from the # of bytes?

For the type of file we are talking about here (a Squid log file) there actually is a way. The file is more or less ’square’, meaning that every line is about the same length (it contains date, status, URL, …).
If we take the beginning of the file (the first 10000 lines):
# head -10000 /var/log/squid/access.log | wc
10000 100000 1775257
we see that every line is about 177 chars long.

The end of the file (the last 10000 lines):
# tail -10000 /var/log/squid/access.log | wc
10000 100000 2047887
gives us a number of 204 chars/line.

Let’s take some more data and combine both:
# ( head -50000 /var/log/squid/access.log ; tail -50000 /var/log/squid/access.log ) | wc
100000 1000000 19488905
which gives us an average of 195 chars/line.

A file size of 533.229.920 bytes (533MB) would lead us to estimate the # of lines to 2.734.512, where the actual # of lines is 2.818.184 (3% difference). That is: we lose 3% accuracy but the calculation takes almost no CPU time, instead of 45 seconds. This might be a trade-off you are willing to accept!

Related posts:

1. Calculate hit rate from a log file You have a huge file that contains one line per...
2. Squid: list top X referers If your Squid server logs the referers of its request...
3. Squid cachemgr.cgi UI hack Squid has a little system statistics viewer built-in: The cache...

And now let’s just filter out the lines containing ‘HIT’ and count those:
#time sh -c "grep -i HIT /var/log/squid/access.log | wc -l"
Wow! This takes ages (I stopped it after 15 minutes) and the grep takes 100% CPU all the time. So let’s look for another solution.

Maybe gawk? First let’s see if it is much slower than wc -l for counting lines:
# time gawk "END {print NR}" /var/log/squid/access.log
845907
real 0m26.129s (5.3 MB/s or 32.000 lines/s – 4 times slower)
And now let it count the hits too:
]# time gawk "BEGIN {hit=0} /HIT/ {hit = hit+1} END {print hit/NR*100}" '/var/log/squid/access.log'
84.5023
real 0m32.836s (4MB/s or 25.000 lines/s – slow but acceptable)

Do we actually need a count on the whole file? What if we just took the last (i.e. most recent) 100.000 lines? The result would be a better indication of what the current hit rate is, and the speed of calculation would be more predictable.
# time sh -c "tail -100000 /var/log/squid/access.log | gawk 'BEGIN {hit=0} /HIT/ {hit = hit+1} END {print hit/NR*100}'"
92.305
real 0m3.332s (30.000 lines/s)

It is actually a bit slower the first time you run it, probably due to disk or filesystem caching. So if you want your hit rate calculation to take less than 2 seconds, you could take the last 50.000 lines. Done!

Related posts:

1. Estimate # of lines in a log file Let’s say you need an (approximate) count of the number...
2. Convert Bind DNS zone into PTR records The following script I made in order to convert the...
3. Probe disk performance (MRTG) The hdparam can be used to monitor the throughput speed...

1. this script is ‘top_referers.sh’
2. (c) 2004 Peter Forret – Open Source
   REFERERS=/var/log/squid/referer.log
   OUTPUT=/var/www/html/stats/referer.txt
   MAXLINES=50(
echo REPORT MADE AT `date`
echo =============================
$OUTPUT

Then add it to your crontab:
10 * * * * /(path)/top_referers.sh
and you have an hourly updated stat!
Add a little HTML formatting if you’re aesthetically demanding!

Related posts:

1. Squid cachemgr.cgi UI hack Squid has a little system statistics viewer built-in: The cache...
2. Estimate # of lines in a log file Let’s say you need an (approximate) count of the number...
3. Calculate hit rate from a log file You have a huge file that contains one line per...

# more /proc/version
Linux version 2.4.20-24.9 (bhcompile@porky.devel.redhat.com)
(gcc version 3.2.2 20030222 (Red Hat Linux 3.2.2-5)) #1
Mon Dec 1 11:35:51 EST 2003
# more /proc/cpuinfo
vendor_id : GenuineIntel
model name : Intel(R) Pentium(R) 4 CPU 2.00GHz
cpu MHz : 1992.653
cache size : 512 KB
(…)
bogomips : 3971.48
# more /proc/meminfo
MemTotal: 1030872 kB
(…)
# cat /etc/redhat-release (only for RedHat distributions)
Red Hat Linux release 9 (Shrike)

So now you know: a 2GHz Pentium 4 with 1GB of memory, running RedHat 9 ‘Shrike’.
For more info on RedHat versions: Taroon, Shrike, Enigma, …

Related posts:

1. Date formatting in GAWK: boot time I have one server with apparently an exceptional stability: #...