[rtg] Does anyone find this interesting?

[Matt Simerson]

I'm looking for feedback right now, questions, and concerns...


~/scripts/rtg % cat README

RTG is "a flexible, scalable, high-performance SNMP statistics  
monitoring system" available at http://rtg.sourceforge.net/  If you  
need to measure bandwidth based on router/switch port traffic, then  
RTG is an excellent tool that performs the task of polling the network  
devices via SNMP and storing the traffic data in a SQL RDBMS.

Layered Technologies uses RTG to monitor tens of thousands of network  
ports, feeding the bandwidth data from each of our networks into MySQL  
databases. Each of the scripts included in RTG::Reporter serve a  
particular need. We appreciate the generosity of RTGs authors and it  
is our hope that the RTG community finds this software useful as well.

billing_report.pl - Generate a bandwidth billing report. The script  
includes a slew of options for customizing the report to provide as  
much or as little data as is needed. The report exports a CSV file and  
emails it (as an attachment) to a list of email addresses specified in  
the config file. The report includes the following fields: network  
name, router, Interface ID, Interface Name, Port Speed, Description,  
and In/Out bytes. It can optionally include the Avg In/Out (bit/s),  
Peak In/Out (bit/s), and 95th In/Out (bits/s). For each network  
defined, a set of 4 summary records are included in the CSV data  
(uplink, downlink, isp_uplink, and total). An example CSV is included  
in examples/billing_report.csv.

uplink_report.pl - Generates an uplink report that shows how much  
traffic each ISP uplink port is utilizing. We run this report weekly  
and it gives our operations team and executives a high level overview  
of our network traffic. It also gives us handy reference points to  
compare and validate the usage our ISPs bill us for.

record_consolidator.pl - Consolidates RTG data into summary records.  
We monitor tens of thousands of network ports and have hundreds of  
gigabytes of RTG data. When disk space or performance becomes an  
issue, the RTG supplied tool is a pruning script that simply deletes  
old data. Rather than deleting that data, I wanted to condense it,  
preserving the essence of the data. I borrowed a concept from Tobias  
Otiker and his RRD databases--rather than discard old data, reduce its  
granularity.

In testing, I discovered that 15 minute averages still work perfectly  
with unaltered versions of the RTG CGI application. Any more than 15  
minute averages and graphs stop rendering. Dropping from 5 to 15  
minute averages reduces the disk space required by old data by 66%.  
The consolidator script is controlled by two settings in the config  
file:

  [Pruning]
  days     = 190
  interval = 60

Days controls what data you want to alter. A setting of 190 as shown  
will only condense data older than 190 days. Interval controls what  
interval the consolidated records will be. A setting of 15 will  
condense the data to 15 minute averages (66% reduction). A setting of  
60 as shown will reduce the data to 60 minute averages for a 91%  
reduction in disk space.

How you use the consolidator script will be dictated by your business  
needs. I suggest using the consolidator script with two sets of  
settings. Old data that is no longer needed for graph publication  
should be pruned to 60 minute averages. A setting of 190/60 will do  
this. Then have another tier of pruning where you consolidate data  
older than 2-3 months to 15 minute averages. A setting of 90/15 would  
do that.

rtg_target_maker.pl - not released yet...

A re-write of rtgtargetmkr.pl. Fixes a few notable problems with the  
existing script. Notably, it corrects errors in the RTG database. For  
example, say you see that a customer is utilizing 250% of his port  
speed in the billing report. Being gifted in math, you deduce that is  
not possible and figure out it's because the port was upgraded from  
100Mbit to Gigabit. This target maker script will automatically update  
port speeds and port descriptions so that the data in the database  
accurately reflects the configuration of the network device. It also  
sends a HUP signal to the rtgpoll process so it will re-read the  
routers file.



RTG::Reporter is written by Matt Simerson @ Layered Technologies, Inc.  
Layered Technologies is a hosting provider that you should consider  
for your web hosting needs: http://www.layeredtech.com/



Matt Simerson
Unix Automation Developer
Email: matt at layeredtech.com
Phone: 214.564.6085

Layered Technologies, Inc.
On-Demand Utility Computing & Hosting Solutions
Learn more>> http://www.layeredtech.com



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.grdata.com/pipermail/rtg/attachments/20080306/535f3c65/attachment.html