[rtg] rated, new version of rtgpoll

[Matt Simerson]

On Mar 16, 2009, at 3:50 PM, Matt Provost wrote:

> On Mon, Mar 16, 2009 at 06:23:37AM -0500, Jeff Bacon wrote:
>> Not to sound simplistic, but what about just putting the mysql  
>> backend
>> on a compressing filesystem? If it's on another host with plenty of
>> horsepower...
>
> Yes that will get you some constant factor of compression but it won't
> be very good and it will eat a lot of CPU.

Actually, the CPU penalty is not as bad as you think. Servers these  
days have far more CPU available than disk I/O.

I have several backup servers with 24 1TB disks running ZFS with  
gzip-6 compression. I get about 2x compression on backup data, which  
I'd guess is more compressible than mysql tables.

>> It's also possible to take older historical data and "refactor" it by
>> going back and condensing N polling intervals into one, effectively
>> increasing the stored poll length. You lose peaks that way of  
>> course but
>> it does cut down on the number of rows.
>
> This is what RRD does. I think that's a bit simplistic. What I have in
> mind is more of an mp3 for snmp data - a compression algorithm that
> knows something about the data and can compress it while keeping the
> stuff you're interested in and getting rid of things that are less
> important. For snmp data that would be the peaks (and troughs).
>
> There are a lot of good papers about compressing time series data out
> there. I don't think I have to reinvent the wheel it's just a matter  
> of
> doing some research.

Data segregation is a much better solution. Create a new bandwidth  
table for each billing interval (typically months) so that every  
months bandwidth data in it's own table. If you are using MySQL, you  
can convert previous months data to MyISAM compressed for a far more  
significant reduction in disk space. The bandwidth queries for  
generating graphs will be much faster too, as the amount of data in  
each table would be much smaller.

Matt