[rtg] handle database disconnects

[Chris Caputo]

On Wed, 7 Jan 2009, Matt Simerson wrote:
>   1. if the RTG poller loses the connection to the DB, it does not 
>      reconnect.  Ever.

For this I use the following simple patch in combination with having rtg 
be supervised by daemontools (http://cr.yp.to/daemontools.html) with "run" 
script:

  #!/bin/sh
  exec fghack /usr/local/rtg/bin/rtgpoll -v -t /usr/local/rtg/etc/targets.cfg

I run with my recent Postgres patches, but as long as the mysql and oracle 
db_insert() code returns an error on database insert failure, which they 
appear to do, this should work.

It isn't graceful, since you lose one or more periods of data, but at 
least it recovers from database issues once the database is nominal.  A 
better fix would be to have the reconnect code be in rtg, but this was 
easy.

Chris

---

diff -u -p -r1.41 rtgsnmp.c
--- src/rtgsnmp.c	19 Jan 2008 03:01:32 -0000	1.41
+++ src/rtgsnmp.c	7 Jan 2009 18:41:28 -0000
@@ -345,7 +345,9 @@ void *poller(void *thread_args)
 					PT_MUTEX_LOCK(&stats.mutex);
 					stats.db_inserts++;
 					PT_MUTEX_UNLOCK(&stats.mutex);
-				} 
+				} else {
+					fatal("Fatal database error.\n");
+				}
 			} /* insert_val > 0 or withzeros */	
 		} /* !dboff */