[rtg] Traffic is wrapping

[Leech, Jonathan]

Bill,

I don't see any way to combine RTG and JRTG, other than what I've built
already (TargetMaker input and RTG schema output). Feel free to steal
any and all of my ideas from JRTG and implement them in RTG.
I am not a big fan of distributed / heterogenous environments, they tend
to be more fragile. JRTG is intented to scale to an enterprise / service
provider level on a single box, and beyond that, JRTG is intended to
scale horizontally; e.g. seperate boxes all polling distinct sets of
targets, but not distributing the various tasks involved in polling
across different machines.

Many of JRTG's improvements over RTG involve the poller itself; For
example the aggregation of multiple targets into a 
single SNMP get, the use of the uptime MIB to distinguish between
counter wrap and counter reset, etc. So I specifically wouldn't want the
RTG poller feeding JRTG to insert data.

TargetMaker has served me well, and my colleague made some enhancements
to effectively "multithread" it (via forking multiple processes), so
that it doesn't take multiple days to build a targets file. But, in all
likelyhood it will be the next target of our rewrite / conversion, and
its features included in the JRTG poller itself, with a database rather
than file backend. Perl is a poor choice for TargetMaker once the number
of targets gets bigger than a couple hundered, due to its poor
multithreading support. For the same reason RTG was written in C, to
scale better than Cricket via better multithreading, TargetMaker or its
equivalent needs to be implemented in a multithreading-capable language.

Once I've got TargetMaker or its equivalent's backend in a database, or
even an XML file, I'm either not parsing anything, or parsing XML, which
is dead simple in any language.

I appreciate the opportunity to bounce my ideas of others, and whether
my ideas make RTG better, or yours make JRTG better, it doesn't matter
to me one bit. The discussion alone is valuable to all involved.

-Jonathan

-----Original Message-----
From: bill fumerola [mailto:billf at mu.org] 
Sent: Wednesday, April 09, 2008 4:30 PM
To: Leech, Jonathan
Cc: rtg at lists.grdata.com
Subject: Re: [rtg] Traffic is wrapping

On Tue, Apr 08, 2008 at 03:16:10PM -0600, Leech, Jonathan wrote:
> What's on sourceforge is intended to be a starting point. I don't have

> any users I know of other than me, so I haven't prioritized updating 
> the code.

ok.

> However, JRTG hasn't changed much since the version that's up there. 
> One or two bug fixes. Also I've built an RRD module that uses RRD4J to

> write values to files instead of a database. I have also been kicking 
> around the idea of building a TargetParser that can read Cricket's 
> equivalent of a targets file.

normally, i like consolidating efforts on projects. unfortunately, the
only way i see to leverage work being done in your project and rtg would
be to use something like thrift[1] and seperate out the poller, config
parser, database insertion, etc. people could run different parts of the
system on different machines, JRTG could be used for the database layer,
the C poller could be used to feed it, etc.

this is just dreamworld stuff. i've wanted to seperate some of the C
code into seperate daemons for various reasons. thrift would be great,
but...

thrift doesn't have C bindings yet (and while many desire, it's a tough
nut to crack). that kills that idea unless we convert the existing rtg
to C++, which would be Yet Another Rewrite.

think of this, though... with thrift we could have a perl targetmaker, a
python config parser, a C or ruby poller, a java database aggregator, an
almost infinite amount of combinations.

it's neat. it also would require serious rototilling. not gonna happen.

> Thanks for the link to Joel, I hadn't read that one. He makes some 
> very good points. My particular school curriculum was mainly C++, a 
> little assembly. I already had a background in C and C++ prior to 
> college so I was immune to the weed out courses Joel mentions. I think

> the problem nowadays isn't in teaching Java, its the failure of the 
> schools to teach something hard enough to weed out the incapable.

i wasn't implying you were one of the people he mentions. it's just a
good summary on why language, while important, really isn't the mark of
a good program or programmer.

-- bill

1. http://developers.facebook.com/thrift/