stats: allow the use of backends for statistics + stats tree about opensmtpd

[ericfaurot]

On Wed, Jun 20, 2012 at 2:02 PM, Gilles Chehade
[email protected]
wrote:

Statistics are handled by static structures stored on shared pages.

I would like to have the stats API use a backend system (akin to maps, scheduler, delivery, queue, ..) and provide a default backend that would store statistics in a tree.

The API needs to be redesigned slightly, but the idea is to have very simple primitives: increment / decrement / set / get ; and allow setting dynamic keys which is not possible today:

 /* example */
 {
   char key[STATS_KEY_SIZE];

   snprintf(key, sizeof key, "deliveries.%s", evp->dest.domain);
   stats_increment(key);
 }

The problem is that it will not work with shared pages easily anymore.
I can very well see what you intend to do, with SQL backend, but
what do you have in mind as a default out-of-the-box backend?

Eric.

[poolpOrg]

Well the goal is to get rid of shared pages and have the control process keep track of statistics.
To all processes, the stats_increment() [and related] functions will be imsg calls to the control process.

[ericfaurot]

On Wed, Jun 20, 2012 at 2:34 PM, Gilles Chehade
[email protected]
wrote:

Well the goal is to get rid of shared pages and have the control process keep track of statistics.
To all processes, the stats_increment() [and related] functions will be imsg calls to the control process.

That works well as long as your are just "pushing" stat counters, but as
soon as you want to query a stat counter, things get more complicated,
as you have to remain async. Unless you want to use a specific sync channel
for stats.

Eric.

[poolpOrg]

Nope, no sync channel.

If I'm correct, the only reason why we share statistics and where a process pops stats from another process is for runner to see if mda / mta will run into a fd exhaustion if we send them a batch.

In practice, every process should only care about its very own statistics and the new getdtablecount() system call can avoid us having to do this trick:

• every process maintains its OWN set of stats and pushes them to control via imsg
• every time mda / mta open a file, they use the getdtablecount() call to determine if they are below a limit
• if the limit is reached, they send an imsg to runner to notify it that they are too busy
• when they are below limit, they notify imsg that they are ok

with this scenario, statistics need only pushing

[ericfaurot]

On Wed, Jun 20, 2012 at 3:00 PM, Gilles Chehade
[email protected]
wrote:

Nope, no sync channel.

If I'm correct, the only reason why we share statistics and where a process pops stats from another process is for runner to see if mda / mta will run into a fd exhaustion if we send them a batch.

In practice, every process should only care about its very own statistics and the new getdtablecount() system call can avoid us having to do this trick:

• every process maintains its OWN set of stats and pushes them to control via imsg
• every time mda / mta open a file, they use the getdtablecount() call to determine if they are below a limit
• if the limit is reached, they send an imsg to runner to notify it that they are too busy
• when they are below limit, they notify imsg that they are ok

with this scenario, statistics need only pushing

Ok, this makes sense.

Eric.

[poolpOrg]

branch 'stats' @ poolp has initial stab at this ;-)

[ericfaurot]

It's in.