ninibe / netlog Goto Github PK

Conditions:

# create topic
curl -XPOST localhost:7200/bug3
{"ok":true,"message":"topic created"}
# add some data
curl -XPOST localhost:7200/bug3/payload -d "foo"
curl -XPOST localhost:7200/bug3/payload -d "foo"
curl -XPOST localhost:7200/bug3/payload -d "foo"
# create scanner from offset 1
curl -XPOST "localhost:7200/bug3/scanner?from=1"

Expected:

{"id":"...","next":1,"last":-1,"persistent":false}

Actual:

{"id":"...","next":0,"last":-1,"persistent":false}

Problem:

The scanner reports to read offset 0 next, despite internally being correctly 1.

It seems like this project tries to be very Kafka-like. The Jocko(https://github.com/travisjeffery/jocko) project does the same. Maybe joining forces could be beneficial instead of two separate projects?

As a follow up on #6 (comment) I'd like to propose a segment header for index files. The segment header should have the following properties:

• store metadata: for starters we want to put the segment creation timestamp
• non-intrusive: the existing code should basically continue to work like before
• easy to upgrade: its possible that we want to put more data into the segment header or maybe even change the index files format at some point. The header can support this by marking the schema of a segment

I am proposing the following format:

+----------------------------------------+
| version [1 byte] | createdTS [4 bytes] |
+----------------------------------------+
| first segment entry [ 16 bytes]        |
+----------------------------------------+
| second segment entry [ 16 bytes]       |
+----------------------------------------+
|  ...                                   |
+----------------------------------------+

Having a single byte for the version should be fine. That should at least be enough for the first 127 schema versions. If we ever exceed this we could think about using the highest bit of the version byte to indicate that another version byte is following. I doubt this will ever be the case but it is important for this reason that the version is the first field of the header.

Prepending a header before the actual index data is always non intrusive because it is only read once when a segment is opened and operation continues normally at the next byte offset after the header (segment.NiFO initially points to the next byte after the header).

I couldn't come up with more header fields for now so with this setup the header will be 5 bytes long. I kind of want to add three bytes up to the next power of two but there isn't really a good reason for this here so I left it as is. The version field enables us to add as much information as we want over time.

The gommap library has not been updated for over 5 years. In addition, it is also incompatible with Windows system(devs with Windows cannot compile/use the netlog). Hence it would be good to replace this library with a better one.

Hi, I just saw the Golang UK video https://www.youtube.com/watch?v=ysjcEN548yc
so I wanted to check out the project but it looks like it has been abandoned. Or is it rather considered completed?

The author has no new commits, and ninibe web shows nginx welcome screen. Seems like after that conference all work stopped.

if its 100% based how does notification work ? web sockets ?

Thankyou for open sourcing this project very useful.

I have a use case which requires that I only handle messages which have been persisted to disk. From reading the code I could see how changes to the index trigger a notification but no indication that the notified update is in sync.

Can you clarify how this works? (in case I've missed something). Is there a small window where a message could be lost if the process dies and hasn't flushed to disk but has notified a watcher about the new message?