[rtg] Traffic is wrapping

[bill fumerola]

On Mon, Apr 07, 2008 at 11:19:12PM -0400, John Von Essen wrote:
> yrtg looks interesting... didn't know it existed.
> 
> Does anyone know if anyone has tried to tackle a pgsql implementation 
> with any success?

it shouldn't be that hard. i never much liked the libtool based db plugin
system that was written around that time. i did something similar though
and the sqlbuf functionality is [mostly] seperated from the specific DB
you use. IIRC, the libtool method meant that rtg could never be extended
such that you had some devices writing to a mysql server, some to sqlite,
some to oracle, some to rrd, etc. it's been a while.

as for sqlbuf: i didn't write pgsql support because i didn't use postgres,
i didn't know of any specific features it has that would help (nor did
i look), and not using postgres had no good way to test it. what i should
have done was wrote an sqlite version, just to keep myself honest.

> Out of curiosity, at what point (say, interface count) does scalability 
> start to become an issue? Right now I monitor 30 FE interfaces across 2 
> routers (7600 series) with 5 minute polling - and haven't noticed any 
> problems. Will I start to see some soon or is scalability problem much 
> higher in the interface/device count range?

that is nowhere near the amount of OIDs (which is really what matters
here, not interfaces) that you poll. part of the problem that spawned
the yrtg performance mods was situations where (at the time) one rtg
process was polling 55 different devices with hundreds of OIDs per-device.

the locking and threading model just didn't fit. if you increased threads
to e.g. 80, you had the possibility of having 80 threads all hit the
same device. if you limited the threads you had the possibility of one
device (dead, slow, or broken) stalling the poll past the next round
(taking longer to poll than the poll period). it would ruin data for all
the other devices.

the patch handles that (and other scalability problems, i think the web
page is pretty good about that) by locking based on host and introduces
a thread that works in the background flushing out large INSERT queries
while the polling continues.

while pgsql (and oracle and whatever) support was something i wrote
sqlbuf in mind for, i never had a reason (for my boss) to spend time
even trying it.

the rest of the desired mods (multiple PDUs per packet being one i
really thought would work nicely) didn't happen before i left Y!

-- bill