So, I've been pinging hundreds of devices concurrently for years with a series of hand-me-down servers at work and home made C or Perl programs. And, for years, I've sat and tried to figure out a way around the high process count issues with my C or Perl attempts at doing mass, continuous ping polling.
Earlier this year, I had someone ask one of those network questions that they thought should be so simple to answer. It made me sit down and take another hard look at re-writing weialgo again in C.
To answer the question, I was going to need to ping thousands of devices, concurrently, every second if I could. I quickly determined that there wasn't a way to do it in Perl. I pretty much came to the conclusion that C was the only answer. I thought I was going to need to start by creating a memory mapped stream of the raw icmp packets that I would need to copy out to the network driver directly.
My first attempts at this didn't go so well. So, I googled, changed tactics, got some success, but still had a host of issues.
Frustration led to me reviewing the normal "ping" alternatives, hping, sing and the like. Nothing gave me the impression that it would be even close to doing the job, and most of those utilities didn't record times with any level of accuracy.
So, I'm sitting in the lab, chin in my hands looking at the server screen wondering how I'd ever do this. Don't ask me what made me think of it, I don't remember, but I got one of those "I wonder if nmap will do this?" random thoughts... When I thought of it, I didn't seriously think nmap would do the job, but I pulled up the --help and started reading.
Long story short, everything I had been attempting to do with weialgo, if I didn't want to do red/green alerting, was BLINDINGLY simple with nmap. Jean-Luc Picard /facepalm simple. Several days spent in denial, simple. OMGWTFwasIthinking simple. Sitting here typing this, I still want to shake my head at how stupidly easy it is with nmap.
Soooooo.... Here it is.
nice --adjustment=-10 nmap -n -sP -PN -PE -v --max-retries 0 --max-rtt-timeout 1000 --min-rtt-timeout 900 --initial-rtt-timeout 900 --max-rate 20000 -oX /mnt/ramdisk/$2.xml -e eth0:$3 -iL /weialgo5/weialgo5.lst >> /dev/null
$2 is the yearmonthdayhourminutesecond (eg: 20121207015611), and $3 is obviously specifying the network subinterface to get around polling overlap issues, but everything else is pretty straight forward.
Let me break it out.
(nice --adjustment=-10) The nice adjustment is obviously an attempt to get around any incurred system latency if at all possible. I'm trying to get accurate sub-millisecond measurements out of the Linux box I'm running this on, so nmap needs to have a higher priority than the basic system applications.
(nmap -n -sP -PN -PE -v --max-retries 0 --max-rtt-timeout 1000 --min-rtt-timeout 900 --initial-rtt-timeout 900) Ping. Yep, that's all it is really. Ping once, and wait up to 1 second (1000 milliseconds), don't ping multiple times, just once.
(--max-rate 20000) No more than 20,000 ping packets in a single second. So, 224 bits per packet * 20,000 packets in a second = maximum of 4,480,000 bits per second on the network. Now, this is a maximum, and depending on how the network is laid out, the likelihood that these pings would go over the same network links is pretty close to zero. --min-rate would be another good option, but it's not something that I need in the network I'm in.
(-oX /mnt/ramdisk/$2.xml) Save off the results to a xml file for parsing.
(-e eth0:$3) Round robin through a series of network subinterfaces to keep concurrent polling from giving false returns between the different instances of nmap.
(-iL /weialgo5/weialgo5.lst) List of devices to ping.
Once you have this, the rest is academic. Really. Freakin' nmap. With this, you could ping 10's of thousands, maybe hundreds of thousands of devices, every second if you wanted to. Someone will need to try to ping 100,000 unique devices and tell me if it works.
The reason I've talked this long about this is, you would not believe how many times I went to bed thinking "There's gotta be a better way to ping devices and record accurate round trip times...". Freakin' nmap. /facepalm
Anyways, it's all downhill from here. I'm just using a set of simple scripts down to fire off the nmaps, parse the data, and save it off for reporting later.
The box has 4 network subinterfaces, eth0:1, :2, :3, and :4. This allows 4 seconds between consecutive nmaps on the same subinterface.
I have a script calling a script which runs the nmap. Basic stuff.
#!/bin/bash
#/weialgo5/weialgo5.sh
while :
do
TARBZ=$(date --utc +%Y%m%d%H)
TARBZSEC=$(date --utc +%Y%m%d%H%M%S)
/bin/bash /weialgo5/weialgo5child.sh $TARBZ $TARBZSEC 1 &
sleep 1
TARBZ=$(date --utc +%Y%m%d%H)
TARBZSEC=$(date --utc +%Y%m%d%H%M%S)
/bin/bash /weialgo5/weialgo5child.sh $TARBZ $TARBZSEC 2 &
sleep 1
TARBZ=$(date --utc +%Y%m%d%H)
TARBZSEC=$(date --utc +%Y%m%d%H%M%S)
/bin/bash /weialgo5/weialgo5child.sh $TARBZ $TARBZSEC 3 &
sleep 1
TARBZ=$(date --utc +%Y%m%d%H)
TARBZSEC=$(date --utc +%Y%m%d%H%M%S)
/bin/bash /weialgo5/weialgo5child.sh $TARBZ $TARBZSEC 4 &
sleep 1
done
And, the child script is pretty easy to predict.
#!/bin/bash
#/weialgo5/weialg5child.sh
echo 'timestamp_utc,'$2 >> /mnt/ramdisk/$2.txt
nice --adjustment=-10 nmap -n -sP -PN -PE -v --max-retries 0 --max-rtt-timeout 1000 --min-rtt-timeout 900 --initial-rtt-timeout 900 --max-rate 20000 -oX /mnt/ramdisk/$2.xml -e eth0:$3 -iL /weialgo5/weialgo5.lst >> /dev/null
/usr/bin/perl /weialgo5/xmlweialgo5.pl /mnt/ramdisk/$2.xml >> /mnt/ramdisk/$2.txt
cat /mnt/ramdisk/$2.txt >> /mnt/ramdisk/weialgo5log$1.txt
rm -f /mnt/ramdisk/$2.xml
rm -f /mnt/ramdisk/$2.txt
Simple stuff. Just kick off an nmap, roughly once a second, and ping a thousand devices, and save off the results to an xml file.
Why the xml file (and the perl program to parse it)? nmap saves the rtt in a microsecond format in the xml file rather than a fraction of a second or millisecond format. The other output formats seem to use fraction of a second or milliseconds instead. Not a big deal, but, if you can get microsecond resolution, why not? Also, xml is fairly easy to parse through.
The next program is the perl script. I don't need anything other than the IP address and the rtt, so I strip out everything else, and save the result to a text file.
#!/usr/bin/perl
#!/weialgo5/xmlweialgo5.pl
use strict;
use warnings;
my $file = $ARGV[0];
my $line;
open my $fh, $file or die "Could not open $file: $!";
my $ipaddress = "error";
my $rttime = 2;
my $x001;
my $x002;
my $x003;
my @foo;
while( $line = <$fh>) {
$x001 = $line;
chomp $x001;
$x002 = substr ($x001,0,6);
if ( $x002 eq "<host>" ) { $ipaddress = "error"; $rttime = 2; }
$x002 = substr ($x001,0,12);
if ( $x002 eq "<times srtt=" ) { @foo = split ("\"", $x001); $rttime = $foo[1] / 1000000; @foo=(); }
$x002 = substr ($x001,0,14);
if ( $x002 eq "<address addr=" ) {
@foo = split ("\"",$x001);
$x003 = substr ($foo[3],0,3);
if ( $x003 eq "ipv" ) {
$ipaddress = $foo[1];
}
@foo=();
}
$x002 = substr ($x001,0,7);
if ( $x002 eq "</host>" ) { print "$ipaddress,$rttime\n"; }
}
close $fh;
This keeps the actual data that I'm storing off fairly small, and easily compressible.
Freakin' nmap..... /facepalm Thank you fyodor.
My next post will be on rolling up the data, reporting and generating the graphs.
No comments:
Post a Comment