bslatkin / ringbuffer Goto Github PK
View Code? Open in Web Editor NEWRing buffer that allows for high-throughput data transfer between multiproccessing Python processes.
License: Apache License 2.0
Ring buffer that allows for high-throughput data transfer between multiproccessing Python processes.
License: Apache License 2.0
Once you know how far behind you are with #3, it'd be nice if a reader can elect to zip forward to sync with the writer and skip a bunch of data. This would be a method like force_sync_all(), except it would hold the reader lock and only allow one reader to be moved. It must only be called by the reader who owns the Pointer being moved. The writer should only ever call force_sync_all(), which blocks all readers because it holds the writer lock.
This would allow the reader to understand how far behind the writer they are, as a number of slots. You could also return this value as a percentage of total open slots.
It is greate work but I have one question:
The documentation mentioned that multiprocessing.Array is more is more atomic, and reading from the documentation, rawArray and Array seem to be the same, just Array is process safe. Could you explain a bit about why you are using rawArray?
PS: I am using Python3.5 & python3.6, and I am mainly interested in saving numpy ndarray (image) and timestamps in the ring buffer so I am creating a customized strcut and feed it into ringbuffer.
Similar idea to #1 except for readers. The idea is you'd quickly get a handle on a slot. Then you could have a ctypes.Structure map to it using from_buffer(). You'd read out everything you need quickly, and then call read_finished() on the slot when you were done.
Other readers would be able to read the same slot and do the same thing. The writer wouldn't be able to overwrite that slot until all readers had called read_finished() on it.
The idea is you called this first and it would return a buffer for the next available slot. The reader wouldn't be able to get to this buffer yet. You'd cast the buffer as a ctypes Structure and populate it with whatever information you have. Then you'd call write_finished() on the buffer. At that point readers would then be able to read the slot.
Doing it this way would avoid the extra copy that writers do and let them directly populate the slot. The key point of this design is it would have to avoid holding the writer lock during this time. It would just prevent other readers or writers from progressing until this slot was full.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.