Support upload of any binary content of the blob during the initial summary upload at routerlicious driver. about fluidframework

[DLehenbauer]

Here are the options we've considered so far:

• Adding an encoding field to ISummaryTree/ISummaryBlob.
• Changing the document creation endpoint to use whole document summaries.
• Using an empty initial document on creation (like ODSP).

Option 1: ISummaryBlob.encoding
The smallest change would be to add an optional encoding field to ISummaryBlob to differentiate between a utf-8 encoded string and a base64 encoded Uint8Array.

However, the root cause of this bug is arguably that it is incorrect to uses ISummaryTree as a wire format. When support for Uint8Array blobs was introduced, ISummaryTree began requiring an encoding step prior to serialization. It is dubious that encoding an ISummaryTree yields an ISummaryTree (see convertSummaryToCreateNewSummary).

The technically correct fix is to use distinct formats for decoded summaries and encoded summaries, which brings us to the next option:

Option 2: IWholeSummaryTree
FRS's historian endpoint already uses a distinct format for encoded summaries, which does correctly preserve encodings (see convertSummaryTreeToWholeSummaryTree). Ideally, we would use the same format for document creation as well. This however is likely to be nuanced and requires more design/investigation.

One reason is that the driver and Historian/GitRest services originally communicated via GitHub's public REST API. This made it possible to use GitHub as a storage service (either directly or via Historian) using the "shredded" summary protocol. To improve efficiency, FRS later introduced the "whole" summary protocol, which extends the GitHub public REST API with the new '/summaries/' route for bulk operations (see 'enableWholeSummaryUpload').

A potential complication of adopting IWholeSummaryTree everywhere is that Tinylicious and the default configuration of Routerlicious continue to use shredded summaries. Research is needed to understand if these would also need to be converted to use whole summary uploads.

Option 3: Empty document
This option would modify the creation flow to first initialize storage with an empty document, followed by an upload using the normal (whole document summary) path. It's non-ideal in that it introduces an extra round-trip through the service but is nice in that it unifies the document creation and summary upload paths. Possibly the impact is negligible, given that it's used by the ODSP driver.

[DLehenbauer]

Note: There is pending work to investigate refactoring the monorepos, which might make it easier to share the "whole summaries" implementation between Historian, GitRest, and Tinylicious without duplication. (The work is still being planned/prioritized.)

[dstanesc]

Option 3 appears natural as simplifies summary transmission. Conceptually createContainer covers the need of acquiring/generating the unique container identity (which in theory could be accomplished in different ways, hence optimizable in the future). In any case in the container lifecycle the referred extra round-trip is a one-time penalty not larger than the network latency. Supporting this proposal.

[NicholasCouri]

@jatgarg, can you confirm impact of #3? Does ODSP still make use of the extra round-trip? I might be missing something but I'm having a hard time finding it on the ODSP code.

[jatgarg]

Currently odsp only uses single round trip to create container on attach and upload the summary in same network call. Only in case of attachment blobs, it first creates an empty file and then upload the summary in a separate call. Double network call could impact perf in this case. Just handling the blob conversion properly which is specified as Option 1 could serve our purpose with lesser changes?

[DLehenbauer]

Confirm fixed. Tinylicious tracked by #17438.