how to delete buffer after certain time? about golongpoll

[fabioxgn]

I'm interested in this too. I'm using this to write a notification service and I'd like to have several categories and the notifications to expiry after a configurable amount of time, ex: 1 hour. So I don't keep a lot of buffers with notifications that will never reach the users.

[jcuga]

These are good, useful ideas. I can look into adding some sort of configurable expiration, and also an optional param to the subscription http handler that allows web clients to mark notifications for deletion. I should have time to tackle this starting a couple of days from now.

This way, notifications can auto-expire, or if users want to explicitly delete a notification earlier, they could do that too. Thoughts?

[fabioxgn]

@jcuga in my case just auto-expire is relevant as I'm broadcasting an event and I'm unaware how my clients have subscribed, so the event can't be deleted by the client as I don't know how many clients must receive the event.

[marwan-nwh]

@jcuga how would you implement this expiration?

[marwan-nwh]

@jcuga what about using a skiplist with unixtime as key and category(string) array as value (as some events could happen at the same time)
then range from beginning to certain time and delete those categories buffers every X of time, and delete the keys from the skiplist as well
also when publishing, we delete the category value from the array, or delete the whole key if contains one value

over engineering? good solution?

[marwan-nwh]

this https://github.com/diegobernardes/ttlcache could be useful too

[jcuga]

@mrwnmonm, one way to implement the expiration would be to remove any events in a given event buffer that are older than X amount of time whenever a new item is added to the event buffer. Events exist in chronological order, so starting at the one end of the buffer you could just keep deleting until you get to an event that is recent enough. The function GetEventsSince shows how the code is already inspecting the buffers for events that fall after a given input time.

This approach would not cover the case where a given category doesn't have any new events for quite some time, so we'd never perform an insert-new-and-remove-expired-events operation. Perhaps by associating a timestamp with each event subscription category that records the newest event in that channel, and then by periodically purging channels that haven't updated in a while.

In the next week or two I plan on sitting down and trying to provide options to allow expiration behavior. My initial thoughts are something along the lines of a cleanup upon new event insert, and then a periodic flush of old channels. But performing a flush on old channels can't be too expensive, or else performance might suffer. So I'll have to take a closer look at your suggested links and see what's the best option.

[jcuga]

Just to let you all know, I haven't forgotten about this issue. It's going to take some pretty major changes to support it, so I plan on making a branch and will start to tackle it sometime this week. Stay tuned.

[fabioxgn]

@jcuga thanks!

If you need any help testing or discussing something feel free to ping me.

[marwan-nwh]

great
On Sun, Jan 17, 2016 at 3:12 PM Fábio Gomes [email protected]
wrote:

@jcuga https://github.com/jcuga thanks!

If you need any help testing or discussing something feel free to ping me.

—
Reply to this email directly or view it on GitHub
#2 (comment).

[jcuga]

Okay, so I've given some thought to this. Since this is pretty major, I want to make sure I'm building the right thing from the start. So I tried to think of how I would be able to configure this new behavior.

What I've come up with is an Options structure that I've added here:
4dfbe6d

The new options would be:

• EventTimeToLiveSeconds
• DeleteEventAfterFirstRetrieval (bool)
• EmptyCategoryTimeToLiveSeconds

See the code comments for a full explanation of each option.

@mrwnmonm I believe these would support your needs. DeleteEventAfterFirstRetrieval would cause a user's event to be deleted as soon as they pull it down via the HTTP handler. If a user has not been on the site for a while, then EventTimeToLiveSeconds would automatically clean up old events that haven't been pulled down by the user yet. I'm not sure whether or not EmptyCategoryTimeToLiveSeconds is useful. The thought is that if you're deleting an event as soon as a user pulls it down, then the buffer could often times be empty (any time between when a user pulls down events and before any new events occur). I was planning on deleting buffers that are empty. But if a user is constantly emptying a buffer and getting new events, perhaps it is more efficient to wait a short period of time before deleting an empty buffer since it might fill back up again shortly.

By the way, If you are calling Publish() on on a category (in your case a user's id or whatever), and that user is not logged in, the event would be placed in the EventBuffer and would exist until it's Time To Live Seconds has been reached. You could only call Publish() in the first place if the given user is logged in. From what you've described, I can't tell whether or not the events you are publishing only occur for active users or not... But I think these new options should improve your situation.

@fabioxgn all you would need to do is set EventTimeToLiveSeconds and I think you would get exactly what you're asking for: that events disappear after a certain amount of time.

If these options make sense, I can go ahead and make a new factory function that takes an Options as an argument and creates a new LongpollManager with the proper behavior. The actual behavior may take a while to code the right way and unit test...

[fabioxgn]

@jcuga liked the Options struct and EventTimeToLiveSeconds is exactly what I need :)

About the EmptyCategoryTimeToLiveSeconds I would postpone this until it's a real issue.

[marwan-nwh]

@jcuga like you have said EventTimeToLiveSeconds and DeleteEventAfterFirstRetrieval solve my problem
about EmptyCategoryTimeToLiveSeconds, in my situation i will always have empty categories, because i will set EventTimeToLiveSeconds to be very short
but keeping the empty category or not, this depends on how the site are doing, if the users are very active i would leave it for a while, if they are not i would delete empty categories immediately

to get a better view on what i'm doing.
i don't know which users are logged in, actually the only way to know this is using this library (i don't do this now, but checking if the user is subscribed right now would do it), so i'm just notifying the user without checking if he is listening or not, and depend on events to be deleted after certain time (which i don't do it now, but it would be possible of course with EventTimeToLiveSeconds)

my notifications is exactly like the facebook notifications
i have three things to do this:

• notification table in the database that holds the actual notifications
• redis key to hold unread notifications count for each user
• this library to notify current active users immediately when something happens, to do live discussions

if the user is active, and some event happened, i do this:

• insert the notification in the database table
• increase the unread notifications count
• ping the UI (just with a simple response to tell the UI to increase the unread notifications count) and when the user click the notifications icon, i set the unread notifications count to 0, and retrieve the recent notifications from the database

if the user is not active, and some event happened, i will do exactly the same
and after the user logged in again, i check the unread notifications count from redis, and subscribe to current events

so notice i don't get the actual notifications or the unread notifications count from the poll, i'm using it for just notifying the current users, if the user has been notified i delete the event immediately, if not i have no interest to keep the event
so acually what would be more optimal for me is DeleteEventAfterPublish
and EmptyCategoryTimeToLiveSeconds depends on how the site is doing like i said before

i hope this describes my case well.
if something is not clear ask me, and if you want to go with a solution that you think is better but requires someone in my case to change his solution, i'm very much ok with it

[jcuga]

I think you all are right about holding off on EmptyCategoryTimeToLiveSeconds. I've started implementing DeleteEventAfterPublish and should have that option fully implemented the next time I sit down to code which should be tomorrow.

Once that option is implemented, you can test it via StartCustomLongpoll(opts Options).

EventTimeToLiveSeconds will be a bit harder and will likely take me a few extra days to finish.

[jcuga]

Update! I have made all of the feature changes in the branch 'buffer_cleanup'
See: https://github.com/jcuga/golongpoll/tree/buffer_cleanup

Once I add a bunch of unit tests, do some more stress testing, and update the README, I will merge this into master and tag a new release.

Until then, you all can try out the new changes by just doing a git checkout buffer_cleanup in the golongpoll repo.

Here's what I ended up doing. I deprecated the CreateManager() and CreateCustomManager functions in favor of the new StartLongpoll() function that takes an golongpoll.Options struct. (You can still use the deprecated functions but they'll log a warning.)

This Options struct has the options:

• LoggingEnabled bool
• MaxLongpollTimeoutSeconds int
• MaxEventBufferSize int
• EventTimeToLiveSeconds int
• DeleteEventAfterFirstRetrieval bool

The last two options are the ones that control how long events and buffers exist. EventTimeToLiveSeconds can be set to golongpoll.FOREVER to disable event expiration.

The way I implemented event/buffer expiration is that any time a new event is published on a category or any time a client starts a longpoll on a category, that specific category's event buffer is checked for expired events and they are removed. At those times we're already referencing the event buffer and the events are already stored in order so clearing expired events isn't hard or expensive to do.

This approach of cleaning up old events and deleting empty buffers whenever there is activity on a given subscription category solves most of the problem. But you could have an inactive category where there are no events being published and no clients requesting a longpoll. An example of this scenario would be if you're using per-user categories and that user is no longer on your web app. How do we purge that user's events after they've left? For that case, the expiration cleanup is more difficult.

To solve this problem, the code now keeps track of how old the event buffers are by keeping a priority queue (min heap) of references to the event buffers sorted by least-recent activity. Every so often the code will check that data structure and delete events/buffers that are too old. The min heap is generally O(log(n)) and since we're cleaning up old buffers the size of the heap should be roughly the number of recent/active categories, plus whatever hasn't expired yet. Also, we don't bother keeping the heap if we have EventTimeToLiveSeconds set to golongpoll.FOREVER.

It was a lot of work to get this event expiration behavior, but I think these changes should do a good job covering it. It was a good coding experience to think about the data structures. If either of you want to try things out feel free to do so.

[jcuga]

These features are now merged into master. I will update the README on the new Options in about a day or so.

@mrwnmonm , @fabioxgn Are there any objections to me closing this issue?

[fabioxgn]

@jcuga nice, thanks! No objections, I'll update my project and if I find any issues I'll report.

[marwan-nwh]

@jcuga thanks for you work, it will be also a good time for me to understand how you implemented this features