[rtg] rtg status?

[john at quonix.net]

Just my 2 cents, but...

After about a years worth of data, our RTG system started getting slow.

Adhoc queries I dont mind, because they are run by admins, and we can 
wait. 

The view.php interface for making daily/weekly/monthly graphs is what 
really started causing problems. End users (we're a colo) were getting 
significant wait times. 

So this is what we did:

Since the daily/weekly/monthly graph views are sort-of aggregate anyway, 
we created a second database that generates special "aggregate" data 
tables which are much smaller - and these tables recycle/remove data 
points not needed for their scope - so the tables remain small long-term. 
These tables are updated every hour from cron. Then we tweaked the 
view.php to look at these smaller tables. End result is the end-user 
graphs load much faster, but the original raw data tables are still there 
for admins to run queries against. 

Oddly enough, this is what I did years ago before I found RTG. I had a 
series of perl scripts scrap MRTG data into MySql. Had the same problem 
though after about a year and the db got enormous - the solution was these 
aggregate tables to aid in quicker graph generation. 

Obviously, database horsepower is one way to go. Right now I am only using 
a dual Pentium III 1.4Ghz with IDE. I'm sure a Quad-core with SAS would 
make things run just as fast. 

-John

----------------------------------------------------
>From : Matt Provost <mprovost at termcap.net>
To : John Center <john.center at villanova.edu>
Subject : Re: [rtg] rtg status?
Date : Wed, 7 Jan 2009 16:59:53 -0800
> On Tue, Jan 06, 2009 at 05:51:13PM -0500, John Center wrote:
> > Hi,
> > 
> > What is the status of this project?  We've been using an older version 
> > of rtg.  I saw a discussion about various patches floating around, but 
> > it doesn't look like there has been any further development.  Could 
> > someone please update the list?
> > 
> 
> I've been working on a fork of the rtg poller for a few months. I'll
> probably put it up online once I get to a good point if people are
> interested.
> 
> So far it's just been a big cleanup of the code, and adding a bunch of
> patches. The thread and signal code is a lot cleaner now. I also have
> it reconnecting to the database automatically. I got rid of all the
> memory leaks.
> 
> There's a lot that I'd like to do but we'll see how much time I have.
> The problem with the vast amounts of data that rtg collects is an
> interesting one. There's lots of good research out there about
> compressing time series data, but I'm not sure how you would do it and
> still keep it in SQL. Then again, you could write a backend for the
> poller that writes to any format...
> 
> Thanks,
> Matt
> _______________________________________________
> RTG mailing list
> RTG at lists.grdata.com
> http://lists.grdata.com/mailman/listinfo/rtg