[rtg] rated, new version of rtgpoll

[Matt Provost]

On Mon, Mar 16, 2009 at 06:23:37AM -0500, Jeff Bacon wrote:
> Not to sound simplistic, but what about just putting the mysql backend
> on a compressing filesystem? If it's on another host with plenty of
> horsepower... 

Yes that will get you some constant factor of compression but it won't
be very good and it will eat a lot of CPU.

> 
> It's also possible to take older historical data and "refactor" it by
> going back and condensing N polling intervals into one, effectively
> increasing the stored poll length. You lose peaks that way of course but
> it does cut down on the number of rows. 

This is what RRD does. I think that's a bit simplistic. What I have in
mind is more of an mp3 for snmp data - a compression algorithm that
knows something about the data and can compress it while keeping the
stuff you're interested in and getting rid of things that are less
important. For snmp data that would be the peaks (and troughs).

There are a lot of good papers about compressing time series data out
there. I don't think I have to reinvent the wheel it's just a matter of
doing some research.

> 
> Detecting bursts of traffic... I have that problem now, and I deal with
> it by running on a 10-sec polling interval on the devices where it
> matters - I've refactored targetmaker and everything else
> (simplistically) so polling rate is per-host; I just don't need to poll
> my APC PDUs for power usage on a 10-second interval. 
> 

That's right, what I have in mind is something that can watch the rate
of change of a target and adjust the polling interval automatically. SO
things that are stable get checked infrequently while things that are
active get checked more often.

> What I would really like is to be able to do it on a per-OID basis, or
> per-class - poll these interfaces at 10, poll all PDUs at 120, poll temp
> sensors at 600. I can do it by further hacking the perl, but...

I also thought of doing it per-OID but that seems like a lot of work.
I'd rather come up with something that works automatically and doesn't
require configuration. But if that doesn't work then falling back to
per-OID is better than nothing.

> 
> But in terms of detecting bursts of traffic, I don't know how you would
> do it without a fixed really-fast polling interval - how do you know
> when the burst is going to happen? Admittedly I am dealing in bursts
> that last 1-10sec which may or may not be the norm, but an adaptive
> mechanism isn't going to work because any burst that I get is going to
> come up so fast and be gone that it only shows as a minor blip on a
> 60-second sale, so adapt all you want it doesn't matter. 
> 

I've been thinking of doing some random sampling. So in general things
would poll on their normal interval but randomly they would do a second
poll in the polling round so you'd get two polls within a second for
example. So you wouldn't catch every burst but you could catch some of
them. And if it's happening randomly then it wouldn't place a constant
burden on the target equipment by doing lots of fast polling. It would
be interesting to feed the data from these fast polls back into the
scheduler - so check things randomly for bursts and if you find one then
check that target more frequently...

There's plenty of room to play with this stuff and we'll see how much
time I have!

Thanks,
Matt