[rtg] buffering mechanism

[Matt Provost]

On Tue, May 23, 2006 at 05:30:38PM -0600, Leech, Jonathan wrote:
> Matt,
> 
> I was initially thinking IPC so as to have a seperate process writing to the database... and that doesn't fit well with the RTG architecture.  
> 
> But I thought about it some more and I think it might be useful anyway as a Message Queue might survive all kinds of bad things like the RTG process being stopped, hanging, box losing power, etc.
> 
> As for buffering I'm thinking along the lines of the following:
> - every T time period
>  - invoke up to N database writer threads 
>   - each writing up to R rows from the queue to the database and commiting.
> 

Yeah splitting it up into two processes would be a big job, and I'm not
sure what it would gain us. IPC isn't great because you leak memory like
crazy when stuff goes wrong.

In any case, the code I attached does allow for N database writer
threads inside the poller process. It doesn't do more than one row at a
time, but that wouldn't be a big rewrite and probably should be added,
although it would have to know about tables which is why it's not done.

Anyway the queue code that I attached does about 90% of the job and just
needs to be integrated into the poller. I still have some of that code
sitting around if you're interested.

Thanks,
Matt