Caching question in processing a KERI event log about tswg-did-method-webs-specification

[SmithSamuelM]

That is how any verifier that verifies a KEL can work. The code in keripy that does this type of verification is used in multiple places. More specifically its verifying a CESR Stream and a CESR stream can include streamed KELs. The CESR stream parser code then hands off as appropriate any KEY events in the stream to a Key Event Verifier (called a KEVer) In KERI parlance this stream parsing and Key Event Verification code is a library run by some component like a watcher, witness, controller, etc that is in essense "watching" a KEL. In the did:webs case the key event verifier is the did resolver.

The complication I want to avoid is in moving the verification to be downstream of the did doc rendering. I.e. interposing the did doc in the append only key event verification process. I don't see a reason to do that and it doesn't appear that that is what your are asking. I just wanted to clarify that that is not the case.

That means that in general we let the did resolver be a Key event verifier (technically CESR Stream parser that hands off to a key event verifier) and how it chooses to do that verification is implementation dependent. Certainly, becasue of the append only architecture (which by design enables incremental verification at scale) one can leverage that in doing a from scratch implementation of a KEL verifier in the form of a did resolver in order to render a did doc. But if you are using Python you can just import the keripy packages that do the parsing and key event verification for you and you don't have to write any new code other than to glue it on.

But conceptually what you described above is reasonable. Their are some important corner cases you would need to handle for recovery rotation events. And others for witnessed KELs. but the basic idea is the same.

[SmithSamuelM]

In keripy, we use LMDB, which is a highly performant persistent b-tree key-value store that enables asynchronous append-only key event verification at scale. But if performance is not so important, you could use a file as your append-only incremental cache. The limitation is that with a file all access is synchronous.

One of the reasons to use an asynchronous data store is that it makes it very convenient to, as we say, "walk the KEL" , but in a very performant manner. Each event in the KEL is accessible asynchronously via its SAID because the SAID is the index into the LMDB (b-tree). Essentially it's a de-duped content addressable database. There is also an index based on the sequence number of the event. So walking can be done by either Said or Sequence Number, sn , both forward and backward. We also store an index into the KEL based on the ordinal first seen number fn which provides the order of first seen for differeent versions of events at a given sn. This supports corner cases in duplicity detection.

[SmithSamuelM]

One of concerns with KERI light did methods like did:plc is that they haven't yet thought through the corner cases and are making some insecure naive assumptions.

[2byrds]

@swcurran and @SmithSamuelM this was an excellent issue to discuss and that I hope ties a bow on the discussions over the last month regarding .kel, .oobi, .keri, .cesr files/resources/streams. The current spec jargon that several PRs have migrated to, is the encompassing term 'KERI event stream'. KERI event stream is the CESR stream that @SmithSamuelM is referring to. And it is generic (agnostic) to how the KELs/TELs are being stored/updated/retrieved. When the resolver processes the GET command for the did:webs .keri resource the resolver could do all kinds of interesting things with files, databases, watcher code, etc. to get the KERI event stream for that AID and verify the DID Document. The spec will remain agnostic to those infrastructure/impl details, only specifying that a GET on the .keri resource is necessary to access the needed KERI event stream.

@swcurran you are describing a way that a resolver could handle generation and verification using files containing the KERI event stream needed. That implementation doesn't exist but the spec is agnostic to those implementation details, except to specify the security characteristics that have to be accounted for (if any are missing, we must add them).

@SmithSamuelM gave a general description with some important info, then described an already implemented way of verifying a did:webs DID document using the existing KERI python code that watchers, etc use. The details like using LMDB won't be discussed in the spec, but that implementation already exists and is very efficient so that is what the resolver reference implementation that @peacekeeper and I are building on Phils original work.

From the spec perspective we just say that the resolver needs the 'KERI event stream' found at the .keri web resource and then the spec will soon describe the KERI events (inception, rotation, interaction, etc) that a 'KERI event stream' will provide in order to process the verification of the DID document.

Happy to discuss further so that we are all in agreement about the division between what the spec says vs. what a reference implementation might support. To discuss implementation details we should discuss in the reference implementation repository here or if a different reference impl is started, it could be discussed there.

[swcurran]

The only thing I was after in this was if what I asked about was technically possible (seems like yes) and if it was already implemented such that a did:webs resolver could use it today (seem like that is a no). So possible, and worth mentioning in an implementers guide if the concerns @SmithSamuelM raised at the meeting a couple of weeks back becomes an issue.

Happy to close this as resolved.