Provide function to close the current events log, create a new checkpoint and start a new events log about acid-state

[stepcut]

I think what you want is createCheckpoint + createArchive. When you create a checkpoint, acid-state will restore by loading the most recent checkpoint and replaying only the events that happened after that checkpoint.

createArchive will move all the old checkpoints and event files that are no longer required to restore the state into a subdirectory. You can then delete the files from that directory if you really don't want them anymore.

That said, issue #22 needs to be resolved. Pull request #67 does not quite seem right, but hopefully we will get it fixed soon.

Do you think createCheckpoint + createArchive is what you want?

Or perhaps you are suggesting that you are seeing a single events-XXXXX.log file that is getting bigger and bigger even after your call createCheckpoint? I can not imagine how that would happen -- but if it is, that is a bug, AFAIK.

[sdroege]

I'm running createCheckPoint and createArchive one after another here. That is creating a new checkpoint file and moving the old one into the Archive directory, however it still uses the same events log as before (same filename, same content) and that one is still growing.

It only starts a new events log when restarting the application (which in my case means closing the state and opening it again on the next run).

[stepcut]

Looking at the code, I am starting to think your observations are correct and my understand of how it works is wrong. I am thinking that either createLocalCheckpoint or createLocalArchive should call cutFileLog for the entries log. My inclination is that createLocalCheckpoint should do it.

[sdroege]

Yes, I agree with that. It should be the creation of a checkpoint, and not the creation of the archive.

The creation of the checkpoint will move everything that is inside the events log into the checkpoint, so after the creation of the checkpoint the events log will contain a lot of obsolete data.

createLocalArchive currently only moves unused files (old checkpoints and old events logs) away, it does not change any file contents.

[stepcut]

If you want this done soon, submit a pull request. Otherwise it will happen 'someday'. I'm thinking a call to cutFileLog for the entries log should go before the call to withCoreState here,

https://github.com/acid-state/acid-state/blob/master/src/Data/Acid/Local.hs#L180

[sdroege]

Yes I was looking at the code already, but I'm not 100% sure I understand what all those functions do yet. Need to read more code

Just to be sure, after calling cutFileLog on the entries log, a) the old events log file is closed and b) a new events log file was created. And all this in a threadsafe way (i.e. if other updates are happening in the meantime, they will only succeed after the new events log is opened and they are stored in there)?

[sdroege]

If the events log is cut before creating the checkpoint, then the new events log might contain events that are already in the checkpoint (new updates seem to be possible while the checkpoint is created), if it is cut afterwards then the old events log might contain events that are in the checkpoint.

I guess the first situation is better

[stepcut]

As far as I know that is correct. Better to have a few extra events in the
new event file than to still require some events from the old file.

• jeremy

On Tue, May 31, 2016 at 11:19 AM, Sebastian Dröge [email protected]
wrote:

If the events log is cut before creating the checkpoint, then the new
events log might contain events that are already in the checkpoint (new
updates seem to be possible while the checkpoint is created), if it is cut
afterwards then the old events log might contain events that are in the
checkpoint.

I guess the first situation is better

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#73 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAKpTak4fiqsCNw7v2JPyB3nJ8vf_DCyks5qHF-pgaJpZM4Iqfgk
.

[sdroege]

Great, thanks for confirming :) Works well for me

[matchwood]

I've run into exactly this issue as well - slightly horrified to find a 17GB events log sittting around on a server for a checkpoint of about 70MB. Like @sdroege I was calling createCheckpoint, createArchive via a cron script every night, so I was a little surprised!

Do you know if @sdroege's fix for this likely to be on hackage soon?

Cheers

[ddssff]

There seems to be an issue with the Travis builds under ghc-7.6 and ghc-7.8.

[gromakovsky]

@stepcut can provide us some information regarding when PR #74 (which fixes this issue) will be merged into master?

[gromakovsky]

Alright, thanks for merge.

[adamgundry]

Looks like this can be closed - fixed in 0.14.2.