[rtg] Traffic is wrapping

[Leech, Jonathan]

What happens with RTG + the YRTG patch + a heterogeneous environment, is
the following:
The targets are randomly allocated to threads.
At the start of the polling cycle, the threads are well utilized, all
polling away.
The devices with the fewest targets tend to finish earliest.
Near the end of the polling cycle, the devices with the most targets and
/ or the slowest responses remain. The YRTG patch compounds this problem
by single threading at the device level. But the alternative is worse,
you don't want all the threads hammering away at the unfortunate last
device to get polled.
Let's say you've got T number of threads configured. Once there are D: D
< T devices left to poll, you've only got D threads working, until D
gets to 0 and your poll cycle is finished.

JRTG has the same limitation, as it is single-threaded at the device
level as well, but with less detrimental effect. First, the database
insertion is seperated from the polling via a buffer, with a fixed and
configurable number of inserter threads. This allows the polling to
proceed faster than would otherwise be possible. It also allows for
pausing, reconnecting, and restarting database inserts, or even changing
the database instance being written to.
Second, JRTG polls multiple targets at a time via a single SNMP get.
This drastically reduces the additive latency issue, and is a lot nicer
to any firewalls that happen to be in between the poller and the target.
Less traffic on the network = less dropped packets, fewer retries, and a
faster polling cycle.
So at the end of a polling cycle, JRTG is still polling the stragglers,
and not fully utilizing its poller threads, but it finishes them much
faster, and can effectively use that time to write records to the
database.

-Jonathan

-----Original Message-----
From: rtg-bounces at lists.grdata.com [mailto:rtg-bounces at lists.grdata.com]
On Behalf Of Bryan Wann
Sent: Monday, April 07, 2008 11:30 PM
To: rtg at lists.grdata.com
Subject: Re: [rtg] Traffic is wrapping

On Tue, 8 Apr 2008, Ryan DiRocco wrote:

> We're polling a few hundred devices and over 4000 total ports. The 
> only performance / scalability issues we've run into thus far is a 
> memory leak on the rehashing of the targets.
>
> We're running on a C2D 6550, 4gb ram.
>
> We actually 'roll up' the data out of the main RTG database and pull 
> it into separate databases so that eliminates the 'growth' problems of

> the rtg tables and the requirements for heavy repair / optimizes.
>
> We also then summarize the data into hourly figures, daily figures in 
> / out, etc. It helps keep down table size for highly 'active' data.

I've been polling a few dozen fully loaded 6509s and 6513s (336-528
interfaces each) on a dual x dual core Xeon system. I'm just barely able
to keep it around 240 seconds (300 sec intervals) by only measuring
in/outOctets, in/outErrors and discards.  It seems in my case the
limiting factor is the sheer speed getting data from the switches (lots
of sup1A and sup2s), as I get the same runtimes even with MySQL inserts
disabled. 
Past 40-50 threads I don't see any more improvement.

I'm actually writing a script to chop up targets.cfg into seperate
config files, one for octet counters, errors/discards, packets/s, so I
can see how I fare with three distinct rtgpoller processes.  If anything
this'll let me poll only errors on longer intervals.  Next up will be
rolling up my data as well, I really don't need five minute counters for
months and months.

Thus far I'm pretty impressed, RTG has been the only poller I've been
able to handle this sort of density with.  I've made mincemeat out of
Cacti and Torrus, constantly reading/writing to thousands of RRD files
was getting pretty expensive and didn't let me cover nearly as many
switches.



Met vriendelijke groet/kind regards,
bryan
_______________________________________________
RTG mailing list
RTG at lists.grdata.com
http://lists.grdata.com/mailman/listinfo/rtg