bslatkin / ringbuffer Goto Github PK

Once you know how far behind you are with #3, it'd be nice if a reader can elect to zip forward to sync with the writer and skip a bunch of data. This would be a method like force_sync_all(), except it would hold the reader lock and only allow one reader to be moved. It must only be called by the reader who owns the Pointer being moved. The writer should only ever call force_sync_all(), which blocks all readers because it holds the writer lock.

This would allow the reader to understand how far behind the writer they are, as a number of slots. You could also return this value as a percentage of total open slots.

It is greate work but I have one question:
The documentation mentioned that multiprocessing.Array is more is more atomic, and reading from the documentation, rawArray and Array seem to be the same, just Array is process safe. Could you explain a bit about why you are using rawArray?

PS: I am using Python3.5 & python3.6, and I am mainly interested in saving numpy ndarray (image) and timestamps in the ring buffer so I am creating a customized strcut and feed it into ringbuffer.

Similar idea to #1 except for readers. The idea is you'd quickly get a handle on a slot. Then you could have a ctypes.Structure map to it using from_buffer(). You'd read out everything you need quickly, and then call read_finished() on the slot when you were done.

Other readers would be able to read the same slot and do the same thing. The writer wouldn't be able to overwrite that slot until all readers had called read_finished() on it.

The idea is you called this first and it would return a buffer for the next available slot. The reader wouldn't be able to get to this buffer yet. You'd cast the buffer as a ctypes Structure and populate it with whatever information you have. Then you'd call write_finished() on the buffer. At that point readers would then be able to read the slot.

Doing it this way would avoid the extra copy that writers do and let them directly populate the slot. The key point of this design is it would have to avoid holding the writer lock during this time. It would just prevent other readers or writers from progressing until this slot was full.