20121219 Why Weialgo? Essay #2

Another email about my newly bad home Internet connection.  Having network problems is fun (for a short while, I always have my 4G phone if I -really- need something, so this isn't that big of a deal at the moment).



subject:  the ping graphs for this morning

Rob Luce <moo@moo.com>
5:05 AM (2 minutes ago)

to Todd 
Not able to sleep as usual, so here's some graphs.  Yes, they're huge, but, just roll with it for a minute.

It's the same information, weialgo at 2 second intervals, pingplotter at 100ms intervals, otherwise it's the same info.  What I'm trying to show is that the views are very similar, it's just a matter of presentation.

Weialgo is going to display information based on attempting to visually give a representation of the impact of variation.  Pingplotter is displaying simple absolute latency.

Ping plotter will show a 300ms rtt latency as 300ms.  Weialgo will show 300ms rtt latency, if that's the BEST latency for the site, as 0ms.  That's important because at a certain level, we can't do anything about the size of the earth.  If the network between Chicago and Singapore has zero congestion, and the best we'll ever get going from here to there and back is 300ms, then that's what it is.  A network should start with realistic expectations, and one of those is, photons and electrons have speed limits.   So, minimum latency is where weialgo starts and it shows it as 0ms.

50ms jitter will show up at 350ms on pingplotter, it'll show up as a green 50ms spike on weialgo.

100ms jitter will show up at 400ms on pingplotter, it'll show up as a yellow 400ms spike on weialgo.  Above 50ms jitter on weialgo, it switches from relative to absolute, to demonstrate the overall effect.  Sloppy packet handling (jitter) will have a bigger impact on a site that's far away than a site that's close by.  Weialgo visualizes that, it was intended to.

200ms jitter will show up at 500ms on pingplotter, it'll show up as a red 500ms spike on weialgo.  Anything above 450ms absolute rtt on weialgo goes red, as when you reach the half second mark, human beings start perceiving the latency in applications.

And a dropped packet is a red strike in both of them.

Looking at the graphs this morning, I wanted to remind/reinforce what I've already explained before.  It's useful, and I think you'll need it.

Weialgo is a modeled graph, pingplotter is an simple absolute graph.  Pingplotter is probably more useful to engineers, but weialgo is more useful to represent the network to the lay person.  Pingplotter has to be translated to people, weialgo attempts to explain why people may be complaining based on what their expectations should be.

For those of you that know Todd, you can now feel very sorry for him, as this is the type of stuff he's had to deal with for almost 20 years now.  24x7 non-stop engineering-ish usually-IT stuff.  IT is my job, my hobby, and my life.  I imagine it can be a little rough to be around me for people who don't take their contribution to the world seriously.  But, I suppose normal human beings like to do things that are easy and soft and safe.  Golf, or bowling, or maybe watching football, or stuff like that I suppose.   Todd is exceptionally gifted, but I guess he has a streak of normal in him.

I've learned to live with the fact that normal people aren't driven to try to do as much as they can.  Yes, I know that will sound funny if you know me.  But, think about it for a bit.  I don't do easy or soft or safe.  There was a time where I couldn't be stopped, so I think God stopped me for me, although I've never thanked him for it.

Anyways...

Here's why weialgo.  Weialgo tries to show the network to people, or why the network seems to change moment by moment throughout the day.  If the packet handling is sloppy, jitter is going to be all over the place.  But base latency isn't something we can control, the distance between point A and B will always be fixed, and there will be incurred latency due to that overall round trip.  So, an absolute graph of latency isn't really appropriate for showing the quality of the network between A and B.  Starting the graph at the minimum observed latency is how it should be shown.  I can't do anything about the fact that the end user is in South Africa, and the server is in Canada.  What I can do is make sure that the expectation for the network response time of that server doesn't change.  That's the key to all of this.

Now, there has to be a realization that if we are sloppy in handing the packets, overall round trip time (rtt) will bite us.  Think of jitter as a dog that bites, and the base rtt as the dog's teeth.  Some networks are going to have long base rtt, so you need to pay much better attention to how you handle the packets.  If a dog is toothless, ehhh...  who cares if he gums you a little bit.  But, if that dog has REALLY big, razor sharp teeth (Server in New York, the user is in India) then you REALLY need to pay attention or you'll end up without a leg.  Maybe a bit too visual of an analogy, but do you get it now?

In networking, it's popular to think that you only really need bandwidth graphs to be able to understand how to provision your network.  That's just plain false.  Knowing bandwidth utilization is important (95th percentile, 70%+ utilization over time with the expectation of that continuing into the future, upgrade the site), but it wont tell you what people are observing from the network at the site by itself.  You need bandwidth graphs and ping (rtt) graphs if you want to know how useful the network is to the people using it.

Normal ping graphs are absolute and need to be translated to non-network engineers.  Weialgo is a relative scale model and is designed to not need translation.  Anyone should be able to look at a weialgo graph and say "this looks good", or "this looks bad".  Because, it's the variation that matters at a point, not the absolute round trip time.

Well, rtt *does* matter, but I live in the real world.  Base rtt is a fact of life.   No one has invented a faster photon/electron, and moving everyone on the planet onto a small island so that base rtt never exceeds 10ms is a terrible idea.  So, distance is the rule, not the exception, and must be factored into everything IT, app design, networking management, and setting end user expectations.

But, because base rtt combined with jitter has a multiplicative effect on the performance of a interactive application, its impact needs to be represented in the graph.  Absolute rtt graphs do not do that, and non-network engineers don't understand this, nor do they get the idea when looking at a absolute rtt graph.

This point deserves some emphasis.  Take 3 identical server/client systems, one with 0ms base latency and 300ms jitter, one with 150ms base latency and 150ms jitter, and one with 300ms latency and 0ms jitter.  The 150/150 system will have the worst interactive performance.  I should detail this in a separate blog post some other time, but I'll stand on this statement at this time.  I have alot of experience with this particular issue of interactive applications over Internet connectivity, jitter by itself is bad, latency by iself is bad, but combine the two and people are out to shoot you because their Citrix sessions wont stay connected or because you wiped the raid.

Personally, I like packeteers in dq mode for doing this type of testing, but Linux has a similar latency simulation capability (since I always have a couple packeteers around, I have only used the Linux method once, you'll have to google it).  Prove this to yourself.  Take your favorite interactive app and start messing with different combinations of base latency + jitter = some set amount.  The reasoning for all of this has to do with how TCP works, but, you don't have to go that far into it.  Seeing it will be enough to get the idea.

The weialgo graphs attempt to take this fact and put it into a visual form that any lay person should be able to see and understand.  That's why weialgo and the format of the weialgo graphs.

Hopefully, my next article will be on the weialgo graphing scripts and reporting process.

Rob

20121217 Why Weialgo? Essay #1

I have two articles that I've already typed up but I haven't put them on the blog yet, mainly because they're so rambling that even I have a hard time understanding them.  One is on the graphing and reporting of weialgo data, the other on "Why Weialgo", which is more of me complaining about why this concept is so hard for some people to grasp /facepalm type of thing.

But, something happened over the weekend (thank you Lord, as the outage has been perfect to study and caused me to write the email in the first place) which caused me to write an email which did a better job of being an intro to "Why Weialgo" than my war and peace blog entry which goes over the same subject.

My Internet connection, for all intents and purposes, went down.  It's a wireless connection to a line of sight tower several miles away.  The "old" wireless connection was notoriously unreliable, but, it's failure mode was acceptable in that I was able to put a TCP tunnel between the house and the ISP using a Linux VM in their DC, and compensating for the loss that way.  When they installed the "new" wireless connection a year or so ago, network paradise was created.  It was so good that I had a hard time coming up with reasons why a copper/fiber connection would be better.

That was until last Friday.

Now, when you read this, yes, there is a weialgo graph.  But there's a more important undercurrent philosophy/process/knowledge going through this.  It's about TCP, and how it operates over a network.  And since TCP is how 95%+ of everything in data communications is done nowdays, a deep knowledge of TCP is key to making all, ALL, *-!ALL!-*, IT systems work.  If you don't understand how your systems communicate, you'll never understand why they don't work (or what they'll need to be made to work).

Admittedly, some of my explanations of TCP behavior are sometimes a bit simplistic, but the essence is true. Wireless is hard on TCP, that's just the fact of it.  Weialgo can provide a view of this relationship between wireless and TCP.


The email, lightly edited:


subject: failure mode of the *isp*.net internet link


Rob Luce <blah@blah.com>
6:09 AM (13 minutes ago)

to Todd, Ben 
I know you think I'm nuts, but seeing this graph makes me profoundly...   sad.

This is what the wireless link has been doing about 15 minutes out of every hour or so since last Friday.  This is worse than the old link.  It's also exceptionally educational at the same time.

The old wireless line would go down often, but would normally stay down for a matter of seconds (for the sake of discussion, say 5 to 10 seconds on average), then come back up for a matter of seconds (5 to 10 seconds on average) and "flap" like that while it was having issues.  When it came back up, it would stay up long enough though that the TCP tunnel between us and the ISP could clear it's entire queue of packets (which I set to 512 kByte), then requeue while the link was down.  BUT!!!  The old line would only hit TCP once before coming back up for a statistically significant amount of time (For an android, 5 seconds is nearly an eternity).  This would cause only one halving of the TCP window, as there was only "one" outage to compensate for from TCP's point of view, after which, TCP would "quickly" ramp back up to the full window so that the queue could clear out.

This new wireless failure mode is the death of TCP congestion control, and it details everything that's wrong with the IETF's insistence that dropped packets are to be treated as congestion, instead of what they are more likely to be nowdays - indications of a wireless network inline with the flow of communications.

Since both sides of the TCP tunnel, in this case, are Linux boxes, there are some parameters that are available that otherwise wouldn't be, I've tried to adjust as much as possible to compensate for the frequent drops in the network (BIC mainly, most of the other options aren't of much use in this case).  But, there's no getting around the "birdshot" pattern of packet loss that this link is having.

Every time TCP sees a dropped packet, it goes into "congestion recovery", but here, there is no congestion.  Normally, this means halving the cwnd, and eventually falling back to slow start.  Every one of those red strikes means the cwnd getting halved, and eventually falling back to slow start (mss * 3) (http://tools.ietf.org/html/rfc3782 http://tools.ietf.org/html/rfc2581).

"Down" and then "Up" (or "flap"), I can try to compensate for using a TCP tunnel with custom tweaking.   With "Shot full of holes like birdshot" (or "flutter"), there isn't really anything that can be done using TCP.  I'd have to write my own tunneling software using UDP as the carrier, with a set transmit window, and add something like a full SACK map to cover all in-flight packets.  Of course, there would be some benefits to this, but that level of coding isn't part of my normal skill set.

Every one of those red strikes on the weialgo graph that I've inspected has the exact same pattern.  It's the "TCP Killer Birdshot" or "flutter" pattern.   Which means, not much can be done about it using off the shelf technology.

Hopefully, this is something that *isp*.net finds this week of their own accord.  But, if you would, please call and let them know that the link has been performing strangely since last Friday.  

The ping plotter is using 28 byte packets at 100 ms.  The weialgo graphs are of the first hop router http://192.168.0.20/1.1.1.1.html after I determined that the issue wasn't being caused by the vm.  The vm graphs go back farther in time.  http://192.168.0.20/1.1.1.146.html

Moo
Rob

Sitting here typing/cutting-and-pasting this post, I'm having to watch the pingplotter between me and the ISP so that I know when to click "Save".  (By the way, pingplotter should be mandatory for every network person, and I wish they had an app for Linux.)

Wow, that is ugly.  Strangely enough, I tend to try to find ways to make things work.  If all you have is binder twine, grey tape, and determination, could you make a phone?  I have a tendency to say "you should really just go buy a couple phones and some phone wire", then go right to work making two cups on a string.

But, looking these graphs, I can't imagine how I'd make something work over that.  Change the tunnel to UDP, make a SACK map out of everything to make the tunnel reliable, and retransmit the individual TCP-in-UDP packets like a machine gun until the ISP gives up and fixes the link.  There would be alot of benefit to this in wireless environments, but, I'm a hack scripter, not a real coder. 

What do you think?

20121207 Weialgo/NMap, random notes

Just a couple notes on nmap and how it could be modified for the purpose that I'm using it for.

First, nmap could be changed to do polling with protocols other than ICMP.  There's no reason a TCP SYN/ACK poll couldn't be done, or some form of a UDP poll.  Having said that, it wouldn't normally be good form to use TCP or UDP for the purposes of determining pure network availability and rtt measurement.  Particularly if you're polling once a second.

Hitting a SSH port with a TCP SYN/ACK/RST poll, once a second, wouldn't create a DOS, I don't care what the newb geek in IT Security says.  It would work just fine, and nothing designed since 1993 would have an issue with it.  But, it would be considered poor form to use it in normal use (mostly because the newb geek in InfoSec would go all spazzy on you for doing it, even though he wouldn't have a foot to stand on in a real discussion about it).

So, you wouldn't have to do ICMP, you -*could*- do a "ping" through a firewall using a TCP poll, which nmap is expert at.  But, it would be poor form, normally.

The next thing is, I am not paid to do pings.  Yep, this is not my job, this is just one of my hobbies that I do at home (most of the time).  Yes, I consider this fun.  Actually, this type of stuff is a blast.

But, no, I do not get paid to do this.  But I let my hobbies benefit my work if at all possible.  :-)

The box that I'm using for this at my house is a Intel Atom D2700, which works absolutely fantastic for this.  I'm not pinging as much stuff as I am at work, but I dare say the Atom is faster than the throw-away box that I'm using at work (actually, that was meant to be a joke, but now that I think of it...).

So, 2 or 3 hundred bucks will get you a fantastic weialgo box.  The Atom boards are yet another thing that have an undeserved bad reputation.  Very respectable speed at 10 or 13 watts (I pay my home electric bill, so details like this are important to me).

I should do a rant about performance to watt comparisons at some point, but, that'll be a few blog posts from now.

20121206 nmap.... /facepalm

So, I've been pinging hundreds of devices concurrently for years with a series of hand-me-down servers at work and home made C or Perl programs.  And, for years, I've sat and tried to figure out a way around the high process count issues with my C or Perl attempts at doing mass, continuous ping polling.

Earlier this year, I had someone ask one of those network questions that they thought should be so simple to answer.  It made me sit down and take another hard look at re-writing weialgo again in C.

To answer the question, I was going to need to ping thousands of devices, concurrently, every second if I could.  I quickly determined that there wasn't a way to do it in Perl.  I pretty much came to the conclusion that C was the only answer.  I thought I was going to need to start by creating a memory mapped stream of the raw icmp packets that I would need to copy out to the network driver directly.

My first attempts at this didn't go so well.  So, I googled, changed tactics, got some success, but still had a host of issues.

Frustration led to me reviewing the normal "ping" alternatives, hping, sing and the like.  Nothing gave me the impression that it would be even close to doing the job, and most of those utilities didn't record times with any level of accuracy.

So, I'm sitting in the lab, chin in my hands looking at the server screen wondering how I'd ever do this.  Don't ask me what made me think of it, I don't remember, but I got one of those "I wonder if nmap will do this?" random thoughts...   When I thought of it, I didn't seriously think nmap would do the job, but I pulled up the --help and started reading.

Long story short, everything I had been attempting to do with weialgo, if I didn't want to do red/green alerting, was BLINDINGLY simple with nmap.  Jean-Luc Picard /facepalm simple.  Several days spent in denial, simple.  OMGWTFwasIthinking simple.  Sitting here typing this, I still want to shake my head  at how stupidly easy it is with nmap.

Soooooo....  Here it is.


nice --adjustment=-10 nmap -n -sP -PN -PE -v --max-retries 0 --max-rtt-timeout 1000 --min-rtt-timeout 900 --initial-rtt-timeout 900 --max-rate 20000 -oX /mnt/ramdisk/$2.xml -e eth0:$3 -iL /weialgo5/weialgo5.lst >> /dev/null


$2 is the yearmonthdayhourminutesecond (eg: 20121207015611), and $3 is obviously specifying the network subinterface to get around polling overlap issues, but everything else is pretty straight forward.

Let me break it out.

(nice --adjustment=-10)  The nice adjustment is obviously an attempt to get around any incurred system latency if at all possible.  I'm trying to get accurate sub-millisecond measurements out of the Linux box I'm running this on, so nmap needs to have a higher priority than the basic system applications.

(nmap -n -sP -PN -PE -v --max-retries 0 --max-rtt-timeout 1000 --min-rtt-timeout 900 --initial-rtt-timeout 900)    Ping.  Yep, that's all it is really.  Ping once, and wait up to 1 second (1000 milliseconds), don't ping multiple times, just once.

(--max-rate 20000)  No more than 20,000 ping packets in a single second.  So, 224 bits per packet * 20,000 packets in a second = maximum of 4,480,000 bits per second on the network.  Now, this is a maximum, and depending on how the network is laid out, the likelihood that these pings would go over the same network links is pretty close to zero.  --min-rate would be another good option, but it's not something that I need in the network I'm in.

(-oX /mnt/ramdisk/$2.xml)  Save off the results to a xml file for parsing.

(-e eth0:$3)  Round robin through a series of network subinterfaces to keep concurrent polling from giving false returns between the different instances of nmap.

(-iL /weialgo5/weialgo5.lst)  List of devices to ping.

Once you have this, the rest is academic.  Really.  Freakin' nmap.  With this, you could ping 10's of thousands, maybe hundreds of thousands of devices, every second if you wanted to.  Someone will need to try to ping 100,000 unique devices and tell me if it works.

The reason I've talked this long about this is, you would not believe how many times I went to bed thinking "There's gotta be a better way to ping devices and record accurate round trip times...".  Freakin' nmap.  /facepalm

Anyways, it's all downhill from here.  I'm just using a set of simple scripts down to fire off the nmaps, parse the data, and save it off for reporting later.

The box has 4 network subinterfaces, eth0:1, :2, :3, and :4.  This allows 4 seconds between consecutive nmaps on the same subinterface.

I have a script calling a script which runs the nmap.  Basic stuff.


#!/bin/bash
#/weialgo5/weialgo5.sh

while :

  do

TARBZ=$(date --utc +%Y%m%d%H)
TARBZSEC=$(date --utc +%Y%m%d%H%M%S)
/bin/bash /weialgo5/weialgo5child.sh $TARBZ $TARBZSEC 1 &
sleep 1

TARBZ=$(date --utc +%Y%m%d%H)
TARBZSEC=$(date --utc +%Y%m%d%H%M%S)
/bin/bash /weialgo5/weialgo5child.sh $TARBZ $TARBZSEC 2 &
sleep 1

TARBZ=$(date --utc +%Y%m%d%H)
TARBZSEC=$(date --utc +%Y%m%d%H%M%S)
/bin/bash /weialgo5/weialgo5child.sh $TARBZ $TARBZSEC 3 &
sleep 1

TARBZ=$(date --utc +%Y%m%d%H)
TARBZSEC=$(date --utc +%Y%m%d%H%M%S)
/bin/bash /weialgo5/weialgo5child.sh $TARBZ $TARBZSEC 4 &
sleep 1

  done

And, the child script is pretty easy to predict.


#!/bin/bash
#/weialgo5/weialg5child.sh

echo 'timestamp_utc,'$2 >> /mnt/ramdisk/$2.txt
nice --adjustment=-10 nmap -n -sP -PN -PE -v --max-retries 0 --max-rtt-timeout 1000 --min-rtt-timeout 900 --initial-rtt-timeout 900 --max-rate 20000 -oX /mnt/ramdisk/$2.xml -e eth0:$3 -iL /weialgo5/weialgo5.lst >> /dev/null
/usr/bin/perl /weialgo5/xmlweialgo5.pl /mnt/ramdisk/$2.xml >> /mnt/ramdisk/$2.txt
cat /mnt/ramdisk/$2.txt >> /mnt/ramdisk/weialgo5log$1.txt

rm -f /mnt/ramdisk/$2.xml
rm -f /mnt/ramdisk/$2.txt

Simple stuff.  Just kick off an nmap, roughly once a second, and ping a thousand devices, and save off the results to an xml file.

Why the xml file (and the perl program to parse it)?  nmap saves the rtt in a microsecond format in the xml file rather than a fraction of a second or millisecond format.  The other output formats seem to use fraction of a second or milliseconds instead.  Not a big deal, but, if you can get microsecond resolution, why not?  Also, xml is fairly easy to parse through.

The next program is the perl script.  I don't need anything other than the IP address and the rtt, so I strip out everything else, and save the result to a text file.


#!/usr/bin/perl
#!/weialgo5/xmlweialgo5.pl

use strict;
use warnings;

my $file = $ARGV[0];
my $line;
open my $fh, $file or die "Could not open $file: $!";
my $ipaddress = "error";
my $rttime = 2;
my $x001;
my $x002;
my $x003;
my @foo;

while( $line = <$fh>)  {

        $x001 = $line;
        chomp $x001;
        $x002 = substr ($x001,0,6);
        if ( $x002 eq "<host>" ) { $ipaddress = "error"; $rttime = 2; }
        $x002 = substr ($x001,0,12);
        if ( $x002 eq "<times srtt=" ) { @foo = split ("\"", $x001); $rttime = $foo[1] / 1000000; @foo=(); }
        $x002 = substr ($x001,0,14);
        if ( $x002 eq "<address addr=" ) {
                @foo = split ("\"",$x001);
                $x003 = substr ($foo[3],0,3);
                if ( $x003 eq "ipv" ) {
                        $ipaddress = $foo[1];
                }
                @foo=();
        }
        $x002 = substr ($x001,0,7);
        if ( $x002 eq "</host>" ) { print "$ipaddress,$rttime\n"; }

}



close $fh;

This keeps the actual data that I'm storing off fairly small, and easily compressible.

Freakin' nmap.....   /facepalm    Thank you fyodor.

My next post will be on rolling up the data, reporting and generating the graphs.

20121201 I never claimed my hacks were pretty.... Weialgo version 2 and 3.

I looked through my "stuff" and haven't found a version of weialgo older than 2008 here at home.  I probably have older versions squirreled away at work, but this seems to be the oldest version I have at home.


#!/usr/bin/perl

#Weialgo version 2

use Net::Ping;
use Time::HiRes qw (usleep gettimeofday);
use strict;
#use warnings;

my $host = $ARGV[0];
my $hostname = $ARGV[1];
if ( $host == "" ) { print "\nno IP to ping $ARGV[0] $ARGV[1] $ARGV[2]\n\n"; exit;}
open(LOG, '>>/mnt/ramdisk/v2logfile.csv');
select(LOG); $| = 1;
close(LOG);
select(STDOUT); $| = 1;
my ($seconds, $microseconds) = gettimeofday();
my $prevseconds = $seconds;
my $starttime = $seconds;
srand($microseconds);
my $offsetms = int(rand(1000000));
usleep(1000000-$microseconds+$offsetms);
my $down = 0;
my $totaldown = 0;
my $transitions = 0;
my $totaltime = 1;
my $i = 0;
my $j = 0;
my $ret = 0;
my $duration = 0;
my $ip = 0;
my $runtime = 0;
my $sentpackets = 0;
my $meetsla = 0;
my $minsla = 100000;
my $sec = 0;
my $min = 0;
my $hour = 0;
my $mday = 0;
my $mon = 0;
my $year = 0;
my $wday = 0;
my $yday = 0;
my $isdst = 0;

my $p = Net::Ping->new("icmp");
$p->hires();

while ( $i==0 ) {

  ($seconds, $microseconds) = gettimeofday();
  ($ret, $duration, $ip) = $p->ping($host, 0.6);
  $runtime = $seconds - $starttime;
  $sentpackets++;
  if ( $ret == 0 ) {
    if ( $down == 0 ) {
      open(LOG, '>>/mnt/ramdisk/v2logfile.csv');
      printf LOG ("$seconds,$host,$hostname,$runtime,$totaltime,$transitions,$totaldown,1,%.2f\n", 1000 * 10);
      close(LOG);
    }
    $duration = 10000;
  }
  if ( $ret == 1 ) {
    if ( $down > 1 ) {
      $j = $seconds - $down + 1;
      $totaldown = $totaldown + $j;
      $seconds--;
      open(LOG, '>>/mnt/ramdisk/v2logfile.csv');
      printf LOG ("$seconds,$host,$hostname,$runtime,$totaltime,$transitions,$totaldown,$j,%.2f\n", 1000 * 10);
      close(LOG);
      $seconds++;
    }
    open(LOG, '>>/mnt/ramdisk/v2logfile.csv');
    printf LOG ("$seconds,$host,$hostname,$runtime,$totaltime,$transitions,$totaldown,0,%.2f\n", 1000 * $duration);
    close(LOG);
    $duration = $duration * 1000;
  }
  if ( $duration < $minsla ) { $minsla = $duration; }
  if ( $duration < 10000 ) { if ( $duration < ( $minsla + $minsla + 50 ) ) { $meetsla++; } }
  ($seconds, $microseconds) = gettimeofday();
  if ( $microseconds < $offsetms ) {
    $j = $microseconds + 1000000;
    $microseconds = $j;
  }
  $j = 1000000+$offsetms-$microseconds;
  if ( $ret == 1 ) {
    $j = $j + 5000000;
    $totaltime = $totaltime + 6;
    if ( $down > 0 ) { $transitions++; }
    $down = 0;
  }
  if ( $ret == 0 ) {
    $j = $j + 1000000;
    $totaltime = $totaltime + 2;
    if ( $down == 0 ) { $transitions++; $down = $seconds}
    if ( -e "/mnt/ramdisk/pingslow.txt" ) {
      $j = $j + 4000000;
      $totaltime = $totaltime + 4;
    }
    #if ( $seconds - $prevseconds > 3 ) {
    #  $j = $j + (( $seconds - $prevseconds - 2 ) * 1000000 );
    #}
  }
  $prevseconds = $seconds;
#  print "$j,$down\n";
  usleep($j);
  if ( -e "/mnt/ramdisk/pingflag.txt" ) {
    $i = 1;
    $j = 0;
    if ( $ret == 0 ) {
      $j = $seconds - $down + 1;
      $totaldown = $totaldown + $j;
      open(LOG, '>>/mnt/ramdisk/v2logfile.csv');
      printf LOG ("$seconds,$host,$hostname,$runtime,$totaltime,$transitions,$totaldown,$j,%.2f\n", 10000);
      close(LOG);
    }

    ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)=localtime(time);
    open(LOG, '>>/storage/weialgo/rollups/v2sla.csv');
    printf LOG ("%4d-%02d-%02d %02d:%02d:%02d,$seconds,$host,$hostname,$minsla,$sentpackets,$meetsla,%.2f\n",$year+1900,$mon+1,$mday,$hour,$min,$sec, $meetsla / $sentpackets * 100);
    close(LOG);
    sleep(5);
  }

}

$p->close();


This seems to be the last version of version 2 weialgo that I put together.  Yes, it was written in Perl, mainly because that I didn't see much improvement in doing in a native binary in C.  Perl was much easier to update, and seemed to have similar performance.

I'm not going to bother including or describing the reporting modules.  Most of that was done in shell scripts or perl. You can use perl/sed/awk/grep/sort to go through the information that this script provides and get to some very useful information if so inclined.

Net::Ping and Time::HiRes are both CPAN Perl modules that do most of the magic.  Reading through the perl script, you can get an idea how those modules work, or you can go out to the Perl website and read up on the modules directly.

The thought process for this version was a carry-over of the original version, which was more for Red/Green alerting, with the side benefit of being able to report on data with a number of different statistical models.

At the time I was running upwards of 200 or so pings to individual systems/routers, which was about the limit of what this process was capable of.  It was a simple enough "/usr/bin/perl ./weialgo2.pl 10.1.1.1 myserver.com &" to get it started, and let it run continuously.

As I remember, I would let the program ping the device once every 6 seconds, and record the round trip time and the other data I though relevant at the time.  If the ping dropped, or didn't return in time, I changed the ping time to once a second, until the device responded again.

This last part was the "Weippert Algorithm".  It basically goes like this.  Decide how fast you want to know that a device is down (say, 60 seconds).  Divide that number in half, subtract 1, round down (29).  Using a fast, low bandwidth, low cpu status protocol (like ICMP ping, or TCP SYN/ACT port open check), check the device every int(X/2-1) seconds (29).  The moment the device misses a poll, check the device every second until it comes back up.  If the device misses/drops int(X/2) polls, declare the device down and send an alert.

(If you know Scott Weippert, you can tell him how brilliant he is.  :-) )

The Algorithm is simple, and very effective for Red/Green alerting.  Unfortunately, I couldn't convince anyone I worked with that it was better than SNMP for Red/Green (it's MUCH better, SNMP is a crappy system for basic availability alerting, from a network point of view, but I haven't won that fight yet).

As I worked through the different versions of Weialgo, I found that I was using Weialgo more for statistical reporting, and not for Red/Green.  From a statistical viewpoint, the int(X/2-1) aspect of weialgo complicates statistical reports quite a bit, as it means that all data has to be time indexed as part of the reporting process.  It's much easier to just ping every second or two, and report on the data using that assumption.

The main drawback to this version (and any other process-per-device based polling) is that it drives up the number of concurrent processes on the polling server.  On the P4 I was running this on, at around 200 instances of weialgo, the amount of incurred latency based on just process switching within Linux began to throw off the results.  CPU utilization would normally bounce off 100% continuously, and the box was useless for anything else.  So, I normally tried to keep the number of polled devices much less than 200, usually around 100, so that I could do some simple reporting on the same box against the data.

At one point in time, earlier versions of Weialgo would email out to a pager every time a device went down.  That lasted for a couple weeks (a router would lose it's E1 for 10 seconds at 2am in the morning, PAGE THE NETWORK TEAM!!!!  I wasn't very popular for a couple weeks.).  Adding that functionality back into the perl script would be easy enough to do.

Obviously, I'm over-reporting information in the log files, but the total amount of data is minor in my opinion even with the extra data.  gzip/bzip2 the log files after reporting on them, and you can keep decades of data in a few gigabytes.

Also, since I put all of the data into the same log file, it's possible to have concurrency problems, depending on the version of *nix you want to run this on.  I never had a problem on Linux, but Solaris tended to throw a mangled line in the log file every once in a while.  If you have problems with concurrent processes writing to the same logfile creating mangled entries, split the logfiles up by renaming the logfile with the device name.

$offsetms was put in because of the number of concurrent processes I was running.  I didn't want all of the pings to go out the same exact microsecond.  So, using $offsetms, I randomized the start time to different microseconds for each process.  This spread out the pings, and the processing.

To try this script out, it should be fairly simple.  You'd need Linux (or your favorite version of *nix), Perl, and CPAN install Net::Ping and Time::HiRes.  Copy and paste the script into a .pl on the box, and run the script with the ip address and hostname you want to ping.  Sit back and watch it ping the device until then end of time, or until the server is rebooted.

Oh, as you can probably see from the script, I'm a fan of RAM disks for transient data like this.  Since the log file isn't held open by the processes, you can mv the log file at any time and the processes will keep running.  So, rather than doing hundreds of individual writes to a single file every second on a hard drive, I do the "spam" individual writes to a file on a RAM disk, the mv/report the logfile data via a cron process every hour or so.  Saves wear and tear on the hard drive, and speeds up everything overall.  And, frankly, if I lose an hour or so of pings, no big deal.  RAM disks are another basic IT staple (like ping) that has been lost to an unearned bad reputation.

My next post will be on the "current" version of Weialgo, version 5.  Version 4 and 5 were both based on two issues with the previous versions.  #1  Process per device polling limited the scalability of weialgo to a few hundred devices per polling server.  #2  Red/Green alerting wasn't needed, I was only using the data to run reports and create graphs.

Version 4 was an attempt to create a single process that polled multiple devices (Version 4 never worked properly).  Version 5 was a complete re-write (eg: I threw out all of my previous work) when I discovered that there was a much easier way to ping thousands of devices.  It was a bit of a /facepalm moment.

Here's a copy of one of my Version 3 scripts.  Obviously, I had yanked out all of the int(X/2-1) logic, and I'm just logging straight pings to simplify statistical reporting.


#!/usr/bin/perl

# Weialgo version 3

use Net::Ping;
use Time::HiRes qw (usleep gettimeofday);
use Time::Local;
use strict;
#use warnings;

my $host = $ARGV[0];
my $hostname = $ARGV[1];
if ( $host == "" ) { print "\nno IP to ping $ARGV[0] $ARGV[1] $ARGV[2]\n\n"; exit;}
my $sec = 0;
my $min = 0;
my $hour = 0;
my $mday = 0;
my $mon = 0;
my $year = 0;
my $wday = 0;
my $yday = 0;
my $isdst = 0;
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = gmtime(time);
$year += 1900;
$mon += 1;
$mon = sprintf("%02d", $mon);
$mday = sprintf("%02d", $mday);
open(LOG, ">>/tmp/v3logfile_$hostname\_$year$mon$mday.csv");
select(LOG); $| = 1;
close(LOG);
select(STDOUT); $| = 1;
my ($seconds, $microseconds) = gettimeofday();
my $starttime = $seconds;
srand($microseconds);
#my $offsetms = int(rand(1000000));
#my $offsetms = 100;
my $offsetms = 1;
usleep(1000000-$microseconds+$offsetms);
my $totaldown = 0;
#my $totaltime = 1;
my $i = 0;
my $j = 0;
my $ret = 0;
my $duration = 0;
my $previousduration = 2;
my $ip = 0;
my $runtime = 0;
my $sentpackets = 0;

my $p = Net::Ping->new("icmp");
$p->hires();

while ( $i==0 ) {

  ($seconds, $microseconds) = gettimeofday();
  $j = 1000000+$offsetms-$microseconds;
  usleep($j);
  ($seconds, $microseconds) = gettimeofday();
  ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = gmtime(time);
  $year += 1900;
  $mon += 1;
  $mon = sprintf("%02d", $mon);
  $mday = sprintf("%02d", $mday);
  ($ret, $duration, $ip) = $p->ping($host, 0.5);
  $runtime = $seconds - $starttime;
  $sentpackets++;
  if ( $ret == 0 ) {
    $totaldown++;
    $previousduration = $previousduration + 0.2;
    if ( $previousduration > 2 ) {
      $previousduration = 2;
    }
    open(LOG, ">>/tmp/v3logfile_$hostname\_$year$mon$mday.csv");
    printf LOG ("$seconds,$host,$hostname,$runtime,$microseconds,$sentpackets,$totaldown,1,%.2f\n", 1000 * $previousduration);
    close(LOG);
  }
  if ( $ret == 1 && $duration > 0 ) {
    open(LOG, ">>/tmp/v3logfile_$hostname\_$year$mon$mday.csv");
    printf LOG ("$seconds,$host,$hostname,$runtime,$microseconds,$sentpackets,$totaldown,0,%.2f\n", 1000 * $duration);
    close(LOG);
    $previousduration = $duration;
  }
  if ( -e "/tmp/pingflag.txt" ) {
    $i = 1;
  }
  if ( $microseconds > 50000 ) {
    sleep (1);
  }
  if ( $microseconds > 100000 ) {
    sleep (1);
  }
  if ( $microseconds > 150000 ) {
    sleep (1);
  }
  if ( $microseconds > 200000 ) {
    sleep (1);
  }
  if ( $microseconds > 250000 ) {
    sleep (1);
  }
  if ( $microseconds > 300000 ) {
    sleep (1);
  }
  if ( $microseconds > 400000 ) {
    sleep (1);
  }
  if ( $microseconds > 500000 ) {
    sleep (1);
  }

}

$p->close();

20121128 Ping: I laughed, I cried! It is better than SNMP! I’m going to use it again and again!

If I could hypnotize you to get this idea through to you, I would.  As it is, I have to resort to a wall of text.

Ok, let's use our imagination for a moment.

Let's say you've been brought in to a new network.  There are no monitoring tools working.  All you have is people complaining about application and Internet performance, and everyone is pointing at the network as the cause.

Nothing is set up on the network to allow monitoring.  SNMP isn't configured properly.  Netflow isn't enabled.  You have limited server capability, even if you did get one of the monitoring protocols working on all the network equipment, you wouldn't have the server horsepower to properly use it.

What could you do that would give you data to hold back the tide of angry end users?  What tool should be your go to in tough situations like this?  What tool gives you the best information you could ask for in any network?

I'm here to talk about the much maligned, much ignored, but dependable workhorse, Ping.  Yes, that utility you go to first every time someone says "The network is down".  No, you've never said thank you to Ping.  It's just there, ready to do it's job at a moments notice.  It's the best friend that a network admin could ever have, and never asks for recognition from you, never takes a day off.

Why do we depend on Ping when we're in trouble, but never hear of it any other time?  What makes people think that ping is reliable and has a job to do when everything is coming apart, but any other time it's not good enough to use for any "serious" work?

Ping gets no respect.  That's just plain wrong.

Ping is the best tool to provide a network admin with the most important bits of information you'll ever want to know about a network.

Ping tells you two things.  #1 is the device responding.  #2 how long did the device take to respond.

Now, every network admin/tech/designer/engineer knows this.  Ping is everywhere for a reason.  But here's the reminder that everyone needs to remember.  THESE ARE THE TWO MOST IMPORTANT THINGS YOU NEED TO KNOW ABOUT THE NETWORK.

Sure, it's nice to know the types of information SNMP and Netflow can give you.  But both of those require network equipment to be configured properly, servers to be available with enough capability to handle the processing loads, and software to be purchased, loaded, configured and maintained.  To do a decent installation of your favorite HP SNMP monitoring suite for a mid-sized company could take months to do, thousands of man-hours to complete, and doesn't come for pocket change.  Proper Netflow collection and reporting isn't any better.

And neither of them will tell you the two most basic things about the network with the level of accuracy of Ping.  If you want to know, is it up and how quickly did it respond over the network?  Ping is the champ, bar none.

Now, I'm not trying to detract from SNMP and Netflow.  They, and systems like them, are essential tools to keep a developed network running properly.  But, most new Network Admins treat them as the first thing you should do to get a handle on a network that has gotten out of control.  I wont call them the last thing (after all, how hard is it to set up MRTG, really?), but, setting up a organization-wide Ping statistical data collection tool is simple, fast, and can be done from a spare laptop.

"Organization-wide Ping statistical data collection tool"?  Ok, I'm not sure if that is the best phrase for it, but it's what I call it, and it kind of describes where I'm going with this.

Let's say you collect a list of devices that are on the network that are considered important.  The IP address of every router, the important switches and other important network devices.  Throw in a list of the important server or system IP addresses.  You'll have to do this anyways, so, just type them all up in a list.

Now, let's say we ICMP Ping each one of them, once a second or so, continuously.  Yep, 24x7, 365.

It's ok, really.  Ping wont break your switches, or any other device for that matter.

See, here's the thing about Ping, the individual packets are as small as they get.   If you configure Ping properly (take a look at the -l 0 option in Windows ping, or -s in Linux), the total packet size is 28 bytes.  So, for every device you ping, you use 224 bits per second.   Even a 300 baud modem from 30 years ago can handle that just fine, and if you upgrade to a 1200 baud modem, you have bits to spare.  At today's network speeds, 224 bits each second for each device you want to ping doesn't come up to the level of background noise.  It's background noise to the background noise.

On top of the fact that Ping is extremely efficient with network bandwidth, responding to a ping requires next to zero CPU processing capability.  The most basic of network devices that are in use today have the ability to respond to ping with sub-millisecond variability.

Really, it's ok.  Pinging devices on your network once a second wont break anything.  This is the part that gets me when I start discussing this subject with other IT people.  For some reason, there seems to be an aversion to pinging something repeatedly on a continuous basis.  But, no one thinks snmp walking devices once every 5 minutes is a big deal.  SNMP and Netflow require orders of magnitude (yes, I'm using that term properly, and I think I'm safe saying it) more processing power than Ping ever could, period.  And, Ping gives you the two things you need most.

Now, let's collect all of that Ping data and turn it into something useful.  Something like graphs.  Non-network people love graphs.  Let's say we take all of that data and have our laptop turn it into a graph once a day for each device.  On top of that, let's create a composite metric for each device, letting us know where we need to start working first (rather than purely being slaves to end user complaints).

Here's a graph as an example.

The graph is based on ping data of a random site router.  No specific reason for pulling this graph in particular, it was the first graph I clicked on.  What's the graph mean?  Easy.  Yellow is bad.  Red is really bad.  If there's lots of red, it's really really bad.  I'll explain more later.

This graph was generated on a 6 to 8 year old box (2 ghz P4?  No one seems to remember how old the box actually is) that someone was throwing away.  That throw away box is pinging 800 devices every other second.  Collecting all of the data, and then generating a graph once a day for each device based on that data.  20+ hours of every day, CPU utilization on the box is less than 5% (remember ping takes next to nothing in CPU).  The only time that box "works" is when it's reporting on the data, and generating graphs, which it statically stores in a html directory to be looked at via a web server on the same box.

It took next to nothing to set up.  It takes next to nothing to run.  And, all of a sudden, poof, you have very useful data, that you wont see from any other tool.  I dare say, I don't know of any other tool that will do anything to this level.

Here's a monthly graph for a different site router.

How hard is it to look at that graph and see there is a problem?  The graph covers a whole month, but it's very easy to see the site circuit outage in the middle of the month.

I wrote the tool (hack) back in 2004, and have kept modifying it ever since.  It's much too useful to not use, it's cheap (free?), and takes up next to nothing in system resources to run.

It would be the first tool I deploy in any network I become involved with, and it makes an excellent supplement to every other monitoring tool out there.

And, it's just Ping.  Hundreds of Pings.  Thousands of Pings.  Once ever second, or every other second.  And then graph against a model.  All using open source software and a couple scripts.  Takes next to nothing to set up.

At one time, I had this system sending an email whenever a device failed to respond for 10 pings in a row.  Yes, Red/Green, with a hair trigger.  None of this, miss two SNMP polls and we send an alert.  If it was down for 10 seconds, email to a pager (boot to the head).

I named the system Weialgo, Weippert Algorithm.  Someone I worked with gave me the idea for it as a snide comment to how poor of a job another SNMP monitoring system did at alerting when a site/system was down.  In less than 5 minutes, he laid out how Red/Green should be done, including the underlying logic.  I'm sure he doesn't/wont remember the conversation, but I remember it as one of those moments where my eyes must have been wide open and I was thinking "That's Brilliant!".

Where I currently work, we call the graphs the "Grass on Fire" report, for obvious reasons.

Over the next couple weeks or so, I'll document the different versions, the setup, and the different modifications I've made with this system over the years.

20121127 Everything Packet: Where do you start? Build a Chair.

I'm sure, if you take a look at the last post, and this post, it probably looks like I forgot about the blog for a long period of time.  Honestly, I didn't.  It's just one of those things where you sit and look at it and ask yourself "Why?".  I don't expect alot of people to be interested in the finer details of packet handling on a communications network, so I'm not sure why I would invest alot of time typing up all of my thoughts on networking to put on the Internet so that very few people would read it.

But, then I take a look at the embarrassingly huge amount of time I've spent on other pursuits that had even less productive potential (WoW anyone?).  So, it's meaningless if this is a waste of time.  So, "Why" isn't a problem.

The other problem is a serious one.  Where to start.  Really.  I mean it.

I've been doing computers as long as I could remember.  It's not something I do for work.  It's who I am.  Even when I'm having fun, it usually has something to do with computers.  Nowdays, that's easy to say, 30 years ago, not so much.

I've built up more than my fair share of opinions on what is wrong with Computing (big C) over that time.  They cover the entire gamut.  Anything from the "Big Idea" things, all the way down to the "Nit Picky" details.

Where do you start?  When you take a look at everything wrong with the world, where you could literally start in on any section and just rip it to shreds, how do you decide what to tackle first?

I guess we'll start with how to describe the problems with a particular chair.

The problem with this analogy is that everyone knows what a chair is.  Or they think they do.  With technology (or any suitably advanced subject, like Nuclear Science or such), most people don't know what a "chair" is.  And those that should know what a "chair" is, probably don't.  Those people who actually know what a "chair" is, had to find out the hard way, and most are so filled with self doubt (because they've trialed-and-erred their way to this knowledge) that they don't volunteer the fact that they know what a "chair" is, just in case they haven't learned everything there is to know about "chairs".

So, when the subject about "chairs" comes up, you find that the information about "chairs" is primarily dominated by people who, by personality, are the ones that are most willing to stand up and say what a "chair" is, not by the ones that actually know "chairs".  This leads to suffering on a wide-scale basis.

So, what I'm saying is, "chair" design is primarily documented and promoted by people that are type A personalities, not by the people who have the best overall understanding on how to design a good all-around "chair".

So, what does it take to understand a "chair"?  You have to understand the Universe, Human Society, Human Government, Human History, Biochemistry, Biomechanics, Human Physiology, Mechanical Engineering, and Chemistry.... at the very least.

Do you really need to know about the Universe (which is a very big subject, to be sure) to understand how to build a chair?  Not really.  But, I will guarentee that you'll build a better chair, the more you understand the Universe.

And, the "Universe" here is really my term for a point of view.  I could have just said "you'll need to understand several points of view to properly build a chair", but phrases like that cause the brain to shut off.  No, one of the things you'll need to understand to properly build a chair is the Universe.  I didn't say "Build the BEST chair", I just said build a chair.

The Chair exists as part of a room, which exists in a house or building, which is owned by someone, which sits on the ground in a country, controlled by a government, which is part of a group of nations that exist on a planet, which orbits a star, which is one of billions in a galaxy, which is one of the billions of galaxies that make up the known universe.

Is it really that important to know about the Universal context of something when building a "chair"?  I don't know, what are you going to build the chair out of?  Let's say you decide to make the frame of the chair out of osmiridium.  Osmiridium is a fantastic material for building chairs out of as it's incredibly strong, very durable, and very corrosion resistant.

The problem with osmiridium is that it's very rare, and likely to stay that way.  But, you'd only know that if you understood how osmiridium was created in the first place, and that takes a much bigger view than you'll ever see by just looking at a "chair".  (On a side note:  I think, if osmiridium ever becomes commonplace, you'll find all chairs frames being made of it.)

The same situation exists for every context of "chair" building.  To build a proper chair, you have to know a great deal about "everything".  Heck, all I need to do is say OSHA, and that should be enough.

So... if building a proper chair, building a chair properly, not even a good chair just a proper one, is so hard, What makes everyone think that everything in IT is so damn easy?

Yes.  Why does everyone think that IT is somehow magic, and that people who have NO interest in it can, all of a sudden, sit down and make unbelievably broad statements on how IT systems are to work, and enforce standards that only make sense in the land of physics make believe?

-IT Systems are orders of magnitude more complex than a chair.
Documenting something as "simple" as network routing protocols has taken volumes of books and entire lifetimes of people to get right.  Ask John Moy to explain OSPF in detail in less than 5 minutes.  Now, try to explain a chair in detail, in less than 5 minutes.  The chair could probably be done in less that 30 seconds, and most people would probably understand it.  I don't know Mr. Moy, but I would guess that he could talk for 5 hours and 90% of people still wouldn't understand.

-IT Systems are conceptual in nature, chairs you can see.
Computer systems "work" in your head.  When you ask UPS how they track your package, there's no machine to take apart to point at the little stick that turns the gear that makes the package tracker point to your package in downtown Duluth.   It's all concepts, imagination put to productive use.  Public school beats imagination OUT of kids, it's not a productive skill to have according to most bureaucrats.  I hate to tell you this, but you CAN'T do IT without having a strong imagination.  Troubleshooting, design, support.  Imagination is king.  Imagination is lacking in the human race as a whole, and I believe it exists at an even smaller ratio for people with a business major.  I'm not saying people with business majors aren't smart.  Some are wickedly smart.  But, being technically minded, and having strong imaginations are not traits that are sought after in people with business degrees.

-IT Systems are unbelievably interwoven, chairs interact with the other furniture in your room.
A typical IT system will normally involve itself with every aspect of that business, and people and companies outside of that business.  All I have to do is say "Financial systems, Logistics, Human Resources, Voice and Video communications, Ecommerce, Manufacturing, Facilities..." and that's just a start.  Here's the fact, because of IT, every human being that works for a organization is directly tied to every other human being that works with that organization.  And the complexity of those interconnects are directly reflected in the IT Systems design.

-IT Systems require a diverse group of people to design, build, and maintain.  A chair takes one Amish carpenter.
I generally get the feeling that management feels that as time goes by, fewer people are required to maintain IT systems.  History shows that the exact opposite is true.  Remember the mainframe?  Mainframes died, not because the mainframe concept was backwards or out of date.  Mainframes died because they weren't supported, upgraded, and the knowledge of the people who built and supported those mainframes wasn't actively promoted through training of the next generations of IT people.  Everything you are doing today in IT is doomed if you don't continuously update the technology, and update the people.  And, let's face it, you can't pick just anyone to learn and love IT.  Most people would rather play fantasy football than go home and build a mainframe in their basement.

So, if designing a chair is complex, and IT Systems are orders of magnitude more complex than building a chair, why does everyone think it's so easy to design IT?

Here's my checklist for designing and building IT systems.

1.  You must love IT.  You must do IT for fun on your spare time.
2.  You must have at least 10 years of blood, sweat, and tears in IT before you can design the "smallest" IT production system which affects human lives.
3.  You must own it after you build it.  If it doesn't work right, it's your fault, don't avoid it.
4.  You must find and teach anyone you can who loves IT as well.  If you have a good design, you must share the idea with others.

Number 4 is why I'm doing this blog.  Let's see how far I get.