theupdateframework / specification Goto Github PK

It could be valuable for potential adopters of TUF if there were some documentation beyond the specification, published papers and conversations captured on GitHub, that goes into detail about certain decisions, makes recommendations where the specification deliberately leaves things open and points to open implementations of the specification (i.e. Notary and PEP 458) as examples of the context for the various decisions that must be made when applying the TUF specification to a scenario.

The spec is a good document but provides several points where choices must be made without providing any explanation or guidance.

The papers which motivated various spec decisions and changes provide interesting reading but can be a little dense when trying to understand a nuance of the specification where the context for a decision may be difficult to elicit and, furthermore, the papers are a static document, unlike the specification itself.

In contrast to the specification, which should only say “do XYZ”, this additional document could say things like “do XYZ because foo, bar, baz” or “do X if your situation is quux (akin to projects ABC) or do Y if it is thud (akin to projects DEF)”.

cc @lukpueh

In section 5.1.9, it states:

1.9. If the timestamp and / or snapshot keys have been rotated, then delete the trusted timestamp and snapshot metadata files. This is done in order to recover from fast-forward attacks after the repository has been compromised and recovered. A fast-forward attack happens when attackers arbitrarily increase the version numbers of: (1) the timestamp metadata, (2) the snapshot metadata, and / or (3) the targets, or a delegated targets, metadata file in the snapshot metadata. Please see the Mercury paper for more details.

There some ambiguity here:

• We should define what it means to rotate a key. Do we only remove the metadata if we add a new key and remove an old key at the same time in a root metadata? Or should we also delete the local metadata if we remove a key?
• In the case where we have multiple keys and a threshold > 1 for a role, do we still delete the metadata if we only rotate one key? Theoretically, it should be safe to rotate less than the threshold number of keys in a given root metadata, since an attacker shouldn't be able to perform a fast-forward attack with less than the threshold number of compromised keys.
• Why do we delete both the local timestamp and snapshot metadata if just the timestamp key, rather than just deleting the timestamp metadata? Similarly for just a snapshot key rotation. Does this protect against a known attack, or is this more about cleaning up attacker controlled files that might contain other unknown attacks?
• Why do we not delete the targets metadata if those keys are rotated? (I think this might be addressed though by @lukpueh in #65 or 84103fc if that gets merged in).

Thanks again!

The TUF specification outlines the Root role and key revocations here.

We should add a separate section that more thoroughly covers key revocation of top-level roles, and how clients are expected to update metadata following these key revocations.

https://github.com/theupdateframework/specification/blob/master/tuf-spec.md
4.4. File formats: snapshot.json
The snapshot.json file is signed by the snapshot role. It lists the version
numbers of all metadata on the repository, excluding timestamp.json and
mirrors.json. For the root role, the hash(es), size, and version number are
listed.

What is the purpose of listing current root.json version in the snapshot.json?

Initially I thought that once you rotate any keys, the root.json will get
updated, its version will be bumped and the snapshot.json will get that version.
As a chain reaction the snapshot.json hash in the timestamp.json will be updated
as well. If an update client were to only periodically download
the timestamp.json file to answer the 'Is updated available?' question, then it
would make sense: one of the keys in root.json is updated -> root.json version is
bumped -> snapshot.json is updated with the new root.json version ->
timestamp.json is updated with the new snapshot.json hash -> client fetches the
timestamp.json -> client sees the snapshot.json hash is updated -> client
fetches the snapshot.json -> client sees the version of the root.json is updated
-> client fetches the root.json -> client updates trusted keys for this
repository.

However, the first step of the 5. Detailed Workflows section requires that the
root.json file is fetched and checked for changes prior to fetching of the
timestamp.json file. And it makes sense to do this because if the
least-privileged online timestamp key(s) is compromised and write permissions to
the repository storage are acquired, the attacker will be able to keep
refreshing timestamp.json keeping the snapshot.json hash stale and preventing
updates.

With that being said I can't figure out the purpose of mentioning root.json
version in the snapshot.json.

TAP 4 (the map file) describes how users may specify that a certain repository should be used for some targets, while other repositories are to be used for other targets.

We should reference the map file in the specification once TAP 4 is reviewed and accepted, which depends on the acceptance of TAP 7 (conformance testing). Pending pull request for the map file implementation is available here: theupdateframework/python-tuf#430

A detailed outline of the map file should be covered in the specification, similar to pull request theupdateframework/python-tuf#440 that has now been merged.

I'm trying to wrap my head around the whole delegated roles part of the spec. I see the possible use cases for delegating the authority of parts of a repository to different entities outside the main repository authority.

What I'm wondering about is the use of the backtrack flag: as I understand it, it states that "files that are not signed by this role might be provided by other (trusted) roles, go look in other roles as well." As clients should always check all target roles to resolve a particular target file using some sort of priority scheme this flag serves no use.

It might even be harmful: consider that all target files are mapped to only a single target role, with backtrack = false, this could lead to a situation that when another target role (purportedly or intentionally) lists the same target file the client can see different truths depending on the order in which it considers roles. Thus it has to specify a priority scheme and therefore consider all roles in order...

In 4.5 of the spec, the TARGETPATH is defined as:

 { TARGETPATH : {
       "length" : LENGTH,
       "hashes" : HASHES,
       ("custom" : { ... }) }
   , ...
 }

with custom being defined as:

If defined, the elements and values of "custom" will be made available to the client application. The information in "custom" is opaque to the framework and can include version numbers, dependencies, requirements, and any other data that the application wants to include to describe the file at TARGETPATH. The application may use this information to guide download decisions.

Just to be safe, is custom a json object (as the snippet suggests), or is it allowed to be any json value? As best as I can tell, python-tuf asserts it's an object. If this the case, I'll submit a patch to make the spec more explicit.

I see that the specification/README.rst now no longer calls the spec a draft, but the specification/tuf-spec.md itself (dated June 2018) still calls itself a draft.

• What is the status of the spec?
• Will there be a git tag when it is final?
• How does the version number relate to the tuf implementation versions and tuf/ROADMAP.md?

@jawi reports:

From my experience implementing TUF in a different language (Java), it always caused me several headaches when trying to get the signing right. Mostly this is due to the canonicalisation that needs to be performed that only few of the more popular libraries support. Alternatively, you could take the same approach as is done in the JWS: calculate the signature over the base64-encoded JSON. It would allow you to verify the signature before you unpack the payload and parse it further.

Would that something worth to consider?

@tarcieri comments:

For what it's worth, I've been moving the opposite direction lately, and have been experimenting with Ben Laurie's ObjectHash format:

https://github.com/benlaurie/objecthash

I think it still needs some work, and my use cases are a bit different (one signature that can be validated against different encodings e.g. protos and JSON), but I really like the approach. It requires supporting a visitor that can walk the structure, but eliminates problems with canonicalizing the format of the JSON prior to computing a signature.

There is some ambiguity in the spec regarding how to encode ed25519's public portion of the keyval:

The 'ed25519' format is:

  { "keytype" : "ed25519",
    "scheme" : "ed25519",
    "keyval" : {"public" : PUBLIC}
  }

where PUBLIC is a 32-byte string.

While the ed25519 public key is a 32-byte string, I could find no implementation that's serializing the public key to JSON as an array of integers. Instead, implementations are serializing it into a JSON string, but there is a fork in how this is implemented:

• python-tuf and go-tuf hex encode the public key
• rust-tuf encodes with base64url to encode the public key.
• notary, and hackage base64 standard encode the public key.

Can you provide any guidance on what serialization format we should be using?

edit: turns out rust-tuf uses base64url instead of base64standard for encoding.

For delegations, some examples seem to use a directory structure where foo/ means foo/bar/, foo/baz, and recursively everything under them. Other examples use what appears to be Unix style globbing like /foo/*.bar which would match foo/baz.bar and so on.

The spec should not actually specify either but say that a TUF implementation could use:

• A directory structure
• Regex
• Unix globbing
• Anything so long as it is well define and the server/client can agree on the method of representing delegated paths

Summary

In Section 5 (Detailed Workflows), item 4.5.1 of the spec, the meaning of "visited" should be clarified because an otherwise sensible interpretation can result in an unusual "attack" whereby early delegation confers early right to deny access to a specified role for the purpose of validating any targets included in the early delegation. (It is also prudent for it to be clearer for the purpose of avoiding cycles.)

4.5.1. If this role has been visited before, then skip this role (so that cycles in the delegation graph are avoided). Otherwise, if an application-specific maximum number of roles have been visited, then go to step 5 (so that attackers cannot cause the client to waste excessive bandwidth or time). Otherwise, if this role contains metadata about the desired target, then go to step 5.

Background on Cycle Prevention

When a client seeks target info (hashes, length, etc.) for a given target, it performs a pre-order depth-first traversal of a repository's delegations to find the delegation listing that target info that is trusted to list it (based on delegated paths/patterns), prioritized based on the delegation order.

To avoid clients falling into delegation cycles, a role X must be marked "visited" after that role's list of target info has been searched for the target info and before the recursion proceeds to X's delegations. (Otherwise, X->X, X->Y->X, etc. result in a cycle.) It is not made clear when a role is to be marked "visited", however, and this ambiguity can lead to a further issue beyond delegation cycles.

Particulars: unusual "attack"

The cycle prevention constraint in the preorder depth-first search algorithm says not to visit a role that has already been visited; however, if a role is deemed "visited" as soon as it has been obtained and before it has been validated (which is a place implementers might be tempted to put this for performance reasons), then an attack is possible:

Suppose some role "T" delegates some target "target" to some roles "R1" and "R2", in that order.

Suppose R1 and R2 both delegate to R3. R1 can prevent R2 from trusting R3 (for any target for which R1 has earlier delegation) by "delegating" to R3 with an expected list of keys that is impossible to satisfy (by e.g. requiring a fake or thrown away key.
This denial can cover all possible targets in a delegated space (instead of requiring targets to be specified individually). R1, being an earlier delegation, is allowed to specify targets and thereby prevent R2 from specifying those targets. In this "attack", however, R1 can prevent R2 from trusting arbitrary roles. This is undesirable, though the security impact is likely not noteworthy (since in principle, R1 has the right to deny later delegations from specifying targets by simply listing them).

Solution

This can be solved by defining what a visit entails in item 4.5.1 of Section 5 of the spec. To avoid cycles, a role X must be marked "visited" before the recursion proceeds to X's delegations. Further, if we want to avoid this "attack", role X must be marked "visited" only after the X role file is validated (and still before recursion proceeds to X's delegations).

In section 5.1.3, the spec states:

1.3. Check signatures. Version N+1 of the root metadata file MUST have been signed by: (1) a threshold of keys specified in the trusted root metadata file (version N), and (2) a threshold of keys specified in the new root metadata file being validated (version N+1). If version N+1 is not signed as required, discard it, abort the update cycle, and report the signature failure. On the next update cycle, begin at step 0 and version N of the root metadata file.

However, this seems inconsistent with section 6.1 when speaking about recovering from a compromised root key:

To replace a compromised root key or any other top-level role key, the root role signs a new root.json file that lists the updated trusted keys for the role. When replacing root keys, an application will sign the new root.json file with both the new and old root keys. Any time such a change is required, the root.json file is versioned and accessible by version number, e.g., 3.root.json. Clients update the set of trusted root keys by requesting the current root.json and all previous root.json versions, until one is found that has been signed by a threshold of keys that the client already trusts. This is to ensure that outdated clients remain able to update, without requiring all previous root keys to be kept to sign new root.json metadata.

As best as I can tell, 5.1 suggests that you update the root metadata from N, to N+1, N+2, to N+M etc. But 6.1 looks like it says you figure out the latest root version, and walk backwards from N+M, N+M-1, N+M-2, etc until you find a root metadata that's been signed by a quorum of keys trusted by the version N metadata.

In this case, it seems like we could have clients with different strategies could end up trusting different roots. For example:

• create a keys (K1) and sign root (R1) with them.
• roll the root metadata (R2) with new keys (K2) and sign the metadata with K1 and K2.
• we decide (R2) was compromised, so we generate (R3) and sign metadata with K1 and K3.

So a client thats updating from R1 with the forward updating updates to R2, and it will refuse to update to R3 because K2 didn't sign R3, but a client using the backwards strategy at R1 would skip R2 because R3 was signed by K1.

Looking at python-tuf implements the forward strategy. Is that the correct approach? If so, could section 6.1 be updated to reflect this?

The specification should provide recommendations about upload access to TUF repositories. An uploader should only be trusted to upload images that have been delegated to them, and in most cases they should not be allowed to replace images from other uploaders.

If developers are given unlimited upload access, they could create a denial of service by replacing valid images or metadata files.

Summary

Items 4.5.1-4.5.9 in the Detailed Workflows section (5.1) of the specification do not instruct the implementer in how to "process" the delegated role files, and there are tempting dangers here that must be averted.

Details

Consider this scenario (again... I know...):

• Targets delegates /a/\* to A
• Targets delegates /b/\* to B
• A delegates /a/\* to C, expecting threshold 1, key kA
• B delegates /b/\* to C, expecting threshold 1, key kB
• Targets, A, and B, are signed by all expected keys.
• C is signed by kA, but not by kB

Now, the client:

• updates top-level metadata, retrieving Root, Timestamp, Snapshot, and Targets.
• tries to get target info for target /a/foo.txt.
  • gets A.json
  • validates A.json based on the delegation info in Targets.json: valid
  • gets C.json
  • validates C.json based on the delegation info in A.json: valid, C.json was correctly signed by kA ((this step is not quite clear in the spec in 5.1:4.5.1+, but certainly implied))
• tries to get target info for target /b/bar.txt
  • gets B.json
  • validates B.json based on the delegation info in Targets.json: valid
  • regardless of whether or not C.json is cached, or whether we know that C.json has previously been validated (on a different delegation pathway, seeking a different target), MUST perform validation again, making sure that C.json was correctly signed by kB (and not just kA, as we validated before).

At that last step, it is likely that the client has cached C.json, kept it in a way that signals that it has been validated, in much the way that the top-level roles should be kept if and only if they are validated. The prior validation expecting kA cannot be allowed to pre-empt the validation expecting kB. This should probably be clear in the spec, as it is probably too easy to fail to catch. It requires that the implementer carefully consider the implications of promiscuous delegations.

Fix

A possible fix is to consider adding, between the 2nd and 3rd sentences in 5.1:4.5.1 in the spec, something like "Check the signature on the delegated role file to make sure that it satisfies the requirements (threshold, keys, etc.) of the delegation that brought us to it." A little more rearrangement might be better, but this would probably work.

While TUF protects against a malicious server tricking a client into rolling back metadata (especially if we can land #106), it doesn't have any protections against an attacker who has local control of the storage device that stores the trusted metadata. In that situation, the attacker could simply replace the trusted metadata with some older version that points at some compromised packages.

On Fuchsia, we support verified boot (see android's documentation for a nice description of it). In short, we establish a chain of trust from a hardware key, which has signed the bootloader, which has signed the kernel, which has signed our critical userspace services. We would like to use this functionality to establish a metadata version backstop to make sure that even if an attacker can modify the local storage, they could only rollback metadata as far back as the backstop version protected by verified boot. If we update this backstop with a high frequency, we can pretty much eliminate the risk of a local rollback attack.

We can think of two approaches for implementing this backstop. First, we can include the backstop metadata in the signed user space image. We would initialize our trust by first loading all the metadata from the immutable signed image, then load the metadata from the mutable storage, then finally load any metadata from the remote repository.

The other idea, but which isn't covered by the spec, we could just bake in the backstop version numbers into the signed user space image. This lets us skip having to actually bake in the metadata in our signed image, which could be useful on a space constrained device. When configured for a backstop, our update flow would be:

• In step 5.2.2 when updating the timestamp metadata, if we don't already have a trusted timestamp file, check if the new timestamp's version number is greater than or equal to the backstop timestamp version.
• In step 5.3.2 when updating the snapshot metadata, if we don't already have a trusted shapshot file, check if the new snapshot's version number is greater than or equal to the backstop snapshot version.
• In step 5.1.9, if the timestamp and / or snapshot keys are rotated, delete the backstop timestamp and metadata backstop versions.

Would this be safe to do? If so, I'd be happy to update the spec (or write a TAP) to reflect this change.

The detailed workflow section of the spec should clarify what clients should do after a failed metadata check. For instance, subsection 1.3 check signatures requires the client to verify that the metadata is properly signed, but what should happen if the metadata fails the check? The given metadata should definitely be discarded, but what about the client's current version of metadata?

In section 4.4.5, is it allowed for TUF clients to find the public keys in the root metadata, or is it required that the keys only be found in the targets (or delegated targets) metadata?

I am currently doing some exploration into how clients should handle interrupted, partially successful updates. For example, say we have a client that has a local cached copy of valid and unexpired metadata. We start an update process which includes a new timestamp, snapshot, and targets metadata. Unfortunately, we download the new timestamp and snapshot and persist them to disk, but the device loses power. Then when power is restored the network is down. We'd still like to make queries against the TUF targets file, but according to the workflow, we should get an error. We can only recover from this by restoring the network.

This is particularly relevant to Fuchsia, because of how we have created our packaging system. We want to treat the TUF targets as the list of executable packages, since it allows us to maintain a cryptographic chain of trust all the way down to the bootloader for what things can be executed. All our packages are stored in a content-addressed filesystem, and we use the custom field in a TUF target to provide the mapping from a human readable name to a merkle addressed package blob. When we try to open a package, we first look in TUF to find the merkle, then we check if we've already downloaded that blob. If so, we open up that package and serve it to the caller. See this slightly stale doc for more details. Due to this interrupted update problem, there's a chance a Fuchsia device could be made unusable until we are able to finish updating our metadata.

If not, we have had a few ideas on how to approach this:

• If an update fails, we could still query the local latest targets metadata, assuming it was signed with a key that's still trusted by the root metadata.
• During the update, we delay writing all the metadata to disk until all the files have been downloaded and verified. Then the files are written in one atomic transaction.
• For consistent snapshot metadata (which we only plan on supporting), fetch the timestamp metadata, but don't persist it to disk yet. Fetch and write the versioned-prefixed snapshot and targets metadata, and any other delegated metadata, to disk. Atomically write the timestamp metadata to disk, then clean up any old snapshot/targets/etc metadata files.

I'm not sure if these ideas would weaken the TUF security model though. Is there a better way for dealing with this, and could we incorporate this into the spec (or a POUF?), since I imagine other folks might need a solution for this.

I'm trying to get my head around the update workflow from a clients' perspective. Especially the initial phase, e.g., what to do if a client starts up for the 1st time and wants to check for updates, is missing in the documentation. I've entered them in a single issue, I can split them of to separate issues if needed...

For example, what should initially be available for the client-updater to initiate an update? Only the root keys (if so, which ones: all of them, or only the root key itself?), or the 'root.json' file? Or should the metadata of all roles be initially supplied? I think it is presumed that an initial version of all metadata is present, but it might be good to make this explicit in the specification.

Another shortcoming is the details on how client-updater workflow should do its update: the spec states that ``if at any point in the following process there is a problem (...) the Root file is downloaded and the process starts over''. Aside from the potential DoS that could lead from this for a client (it might be fed with bogus metadata over and over again, so how many times should it retry this?), it also implies that the local metadata is no longer to be trusted which, if true, could be used as an attack vector.
Suppose the following scenario: an adversary has control over the connection by which a client connects to the update server. He mangles the contents of Timestamp file which is detected by the client updater. This causes it to restart the update process with downloading the Root file again, which is now allowing the adversary to inject its own properly signed Root file. The client updater is now confronted with a problem: it gets valid metadata that is not signed by the root keys it knows, nor does it list the keys it currently trusts. This problem is mitigated by enforcing that new versions of Root files should always be cross-signed by the current keys and possibly the new keys. So, clients should accept a new Root file only if, and only if, it is signed by the keys it knows (and already trusts, thus not using the keys listed in the Root file for this!), it lists the current keys it trusts, and has a version that is strictly greater than its own Root file. These checks are subtly different than for other the other files. While this information is present in the spec, it should be made really explicit that this is a strict requirement for client updaters to implement.

What is also not really well described is the workflow for validating the metadata that is available locally: it should be checked for expiration and its signature should be verified. If this fails: first try to download the file again and recheck or always start completely from the beginning with a new Root file?

Also, it would be good to mention that clients are required to have a stable clock(source) in order to correctly verify timestamps. That said, would it not be better if the creation timestamp for metadata files is included in the file? This would allow client updater to no longer rely on information obtained from the webserver to determine whether files are "newer".

Sometimes, we need to check targets metadata from different sources, and make sure that the metadata from all sources matches. There are two broad cases where this is necessary: Multi-role delegations (TAP-3), and multi-repository consensus (TAP-4 use case 3).

In both cases, the TAP specifies that only the non-custom metadata must match, but does not provide any guidance about what to do with custom metadata in case it differs.

The reference implementation of the client exhibits inconsistent behaviour between the two cases: in the multi-repo (TAP-4) case, it creates directories for each repository and saves the complete targets metadata from each; custom metadata is not checked. In the multi-role (TAP-3) case, it finds a consensus group according to the specified behaviour, then returns and saves only the metadata (custom and non-custom) from the highest-priority role in the consensus group (based on the standard pre-order DFS).

The specs mentions

The framework must be secure to use in environments that lack support for SSL (TLS). This does not exclude the optional use of SSL when available, but the framework will be designed without it.

and

Each repository has one or more mirrors which are the actual providers of files to be downloaded. For example, each mirror may specify a different host where files can be downloaded from over HTTP.

I wonder whether TUF is compatible with the use of intermediate caching proxies when downloading (which can only cache http urls and not https urls).

Would it make sense to mention caching support in the spec ?

the spec does not have a notion of developer uploads. As a matter of fact, and I think this should be changed, right now it does not even describe how updates occur on the repository-side (as we do in PEP-458).

Originally posted by @lukpueh in #104 (comment)

In order to address concerns with canonical JSON (including #92) and to allow for more flexibility in the specification to make features like TAP 11 more useful, I propose we remove the canonical JSON requirement from the standard. This means removing the section:

All documents use a subset of the JSON object format, with floating-point numbers omitted. When calculating the digest of an object, we use the "canonical JSON" subdialect as described at http://wiki.laptop.org/go/Canonical_JSON

In addition, we will likely have to do a pass over the document for consistency.

Note that this change does not affect any existing implementations. Canonical JSON will still be a valid choice, but future implementations may chose to use a different canonical json dialect or an entirely different metadata format while still following the TUF specification.

We can probably leave the examples in the spec as is, but make it clear that implementers may chose to use a different metadata format and still have a valid TUF implementation.

The specification refers to files and filesystem operations done on a local filesystem. Some implementations may use TUF on distributed systems or a different non-traditional file backend. To support these implementations, the specification should leave flexibility in how files are represented. This includes changing requirements about paths, directory structure, and files.

For background see discussion in tuf#1009 and #102.

Timestamp and Root are likely to be the two most-downloaded files on a repository. Currently, Timestamp lists information for Snapshot in a different way than Snapshot lists information for Targets and delegated targets files: timestamp lists the hash of snapshot in addition to the version number of snapshot. It is not clear that this is actually useful.

The costs to this are a few:

• It adds size to the timestamp role file (one hash where timestamp would otherwise not contain hashes)
• It makes the timestamp role file a little harder to read and understand.
• It makes conceptually reconciling timestamp and snapshot definitions a bit harder for implementers / new folks.
• It requires a bulkier and less intuitive programmatic representation (in formats.py and tuf_metadata_definitions.asn1).

Snapshot definition
Timestamp definition

The current specification is too informal: it leaves many details unspecified (e.g. the exact algorithmic flow of a conformant updater, exactly what hashed delegations or consistent snapshots are, etc.).

Ideally, the TUF specification would be much more formal. (We should look into a suitable language.)

We should also unify our many different names for the same thing (e.g. 'consistent snapshots' == 'hashed snapshot trees', 'lazy bin walk' == 'hashed delegations', 'release' == 'snapshot'), and produce a standard glossary.

#87 added versioning/release policies, plus a script that helps to enforce them. One of the policies is:

Merges with 'master' must be followed by a git tag for the new version number.

This policy is not enforced by the script, as the script is auto-executed on PRs, but this policy would need to be enforced post-merge.

It would be nice to either:

• auto-tag on merge into master ( see #87 (comment)), or
• otherwise insistently remind the maintainer to tag the release

Note that the issue identified in Uptane Standard issue 36 applies to the TUF specification as well. There is a tiny smidge more implied here in the TUF spec, but it still needs to be clarified.

In section 4.6 of the spec, the description of the snapshot.json file states:

The snapshot.json file is signed by the snapshot role. It lists the version numbers of all metadata on the repository, excluding timestamp.json and mirrors.json. For the root role, the hash(es), size, and version number are listed.

However, the example does not include the hashes or size for the root.json file:

 {
 "signatures": [
  {
   "keyid": "fce9cf1cc86b0945d6a042f334026f31ed8e4ee1510218f198e8d3f191d15309",
   "sig": "f7f03b13e3f4a78a23561419fc0dd741a637e49ee671251be9f8f3fceedfc112e4
           4ee3aaff2278fad9164ab039118d4dc53f22f94900dae9a147aa4d35dcfc0f"
  }
 ],
 "signed": {
  "_type": "snapshot",
  "spec_version": "1",
  "expires": "2030-01-01T00:00:00Z",
  "meta": {
   "root.json": {
    "version": 1
   },
   "targets.json": {
    "version": 1
   },
   "project.json": {
    "version": 1
    },
   }
  "version": 1
  },
 }

Could they be added?

Also, I've seen mentioned in a few places that TAP 5 removes the need to actually include the root in the snapshot file, but I'm not sure if TAP 5 is integrated into the spec, or is just planned to be incorporated in some future version of TUF.

For almost all files listed in the snapshot, if no locally cached snapshot is available this just means that they should be considered changed and should be downloaded. However, the root information is somewhat special in this respect: clients in initial state with root info but no snapshot yet should consider this situation to mean that the root information has not changed (otherwise this would loop indefinitely, because the downloaded snapshot should not be written to the local cache when we find we need to download new root info; see #76).

In #76 I mention that we delete the cached snapshot and timestamp on a verification error to avoid attacks where the attackers sets the file version to MAX_INT. However, we now think that the cached snapshot should be deleted whenever the root info is updated:

If during the normal update process we notice that the root info was updated (because the hash of root.json in the new snapshot is different from the old snapshot) we download new root info and start over, without (yet) downloading a (potential) new index. This means it is important that we not overwrite our local cached snapshot, because if we did we would then on the next iteration conclude there were no updates and we would fail to notice that we should have updated the index. However, unless we do something, this means that we would conclude on the next iteration once again that the root info has changed (because the hash in the new shapshot still doesn't match the hash in the cached snapshot), and we would loop.

Section 2.1.2 of the TUF spec refers to "Delegated developer roles" suddenly. I'm guessing this was missed in a prior change and should just read "delegated roles". You can see this in the second paragraph below.

In addition, the targets role can delegate full or partial trust to other roles. Delegating trust means that the targets role indicates another role (that is, another set of keys and the threshold required for trust) is trusted to sign target file metadata. Partial trust delegation is when the delegated role is only trusted for some of the target files that the delegating role is trusted for.

Delegated developer roles can further delegate trust to other delegated roles. This provides for multiple levels of trust delegation where each role can delegate full or partial trust for the target files they are trusted for. The delegating role in these cases is still trusted. That is, a role does not become untrusted when it has delegated trust.

(I'm logging issues such as these mostly to help make the spec more precise. I hope it's useful.)

The update process described in Section 5.1 looks something like:

1. Download timestamp, verify, and compare to old timestamp.
2. If changed, download snapshot, verify, and compare to old snapshot.
3. If root metadata changed, download new, verify, and start over.
4. Download and verify other changed top-level metadata.

What does this not specify is when, say, the "old" timestamp that the client cached locally is replaced with the new timestamp from the server. This is somewhat subtle; it is not okay to replace the local copy of the timestamp with the server copy as soon as it has been verified and compared, because then we might do:

1. Download timestamp, verify, compare to local. Notice changed, replace local copy.
2. Download snapshot, verify, compare to local, notice root changed (perhaps just a new root key was added, nothing else), replace local copy.
3. Download root, verify, start process over.
4. Now we download the timestamp once again, verify it, and compare it to our local copy. Since we already replaced the local copy, we conclude it's not changed and we return "No changes" to the client.

This means we might fail to notice that there are other top-level metadata files that ought to have been replaced.

Incidentally, on a verification error in our implementation we actually delete the local timestamp and local snapshot after getting (and verifying) the new root info. We do this because, say, the timestamp key has been compromised and the attacker has set the file version to MAX_INT. Then if we don't do something to deal with this situation, clients would subsequently reject all further updates because the file version cannot change anymore. By removing the local files we basically reset the clients state to "we don't know what's on the server".

It's not a big deal, but it's too bad we lost a lot of git history in moving the specification to a new repository.

I noticed that section 5.2 of the spec does not specify if the client should persist the timestamp metadata to non-volatile storage:

2. Download the timestamp metadata file, up to Y number of bytes (because the size is unknown.) The value for Y is set by the authors of the application using TUF. For example, Y may be tens of kilobytes. The filename used to download the timestamp metadata file is of the fixed form FILENAME.EXT (e.g., timestamp.json).

In comparison, root, snapshot, and targets roles explicitly mention that the client should write the metadata to non-volatile storage. For example:

3. Download snapshot metadata file, up to the number of bytes specified in the timestamp metadata file. If consistent snapshots are not used (see Section 7), then the filename used to download the snapshot metadata file is of the fixed form FILENAME.EXT (e.g., snapshot.json). Otherwise, the filename is of the form VERSION_NUMBER.FILENAME.EXT (e.g., 42.snapshot.json), where VERSION_NUMBER is the version number of the snapshot metadata file listed in the timestamp metadata file. In either case, the client MUST write the file to non-volatile storage as FILENAME.EXT.

Was this intentional, or was this an oversight? As best as I can tell, all the implementations write the timestamp metadata to disk before downloading the rest of the metadata.

For multi-role delegations can min_roles_in_agreement include multiple instances of the same role? For example, if Targets delegates to Alice and Bob, who both delegate to Charlie, can Charlie's approval count for a min_roles_in_agreement of 2? If Bob also delegates to Daniela, how can he know to also check for her approval?

If Charlie's approval is not sufficient, the resolution of multi-role delegations in TAP 3 will need to keep track of which roles have been applied to min_roles_in_agreement. This will require the DFS to check all roles rather than returning after the first match, which could impact the efficiency of TUF.

This issue came out of a discussion about determining keyids.

In 4.6 of the spec, the timestamp role METAFILES is described as:

METAFILES is the same is described for the snapshot.json file. In the case of the timestamp.json file, this will commonly only include a description of the snapshot.json file.

Where the METAFILES described in snapshot.json is:

METAFILES is an object whose format is the following:

 { METAPATH : {
       "version" : VERSION }
   , ...
 }
METAPATH is the metadata file's path on the repository relative to the metadata base URL.

VERSION is listed for the root file and all other roles available on the repository.

However, the example in the spec includes the size and hashes field:

...
 "signed": {
  "_type": "timestamp",
  "spec_version": "1",
  "expires": "2030-01-01T00:00:00Z",
  "meta": {
   "snapshot.json": {
    "hashes": {
     "sha256": "c14aeb4ac9f4a8fc0d83d12482b9197452f6adf3eb710e3b1e2b79e8d14cb681"
    },
    "length": 1007,
    "version": 1
   }
...

Are the length and hashes field optional for the timestamp.json role, or are they required? I'm not sure if the arguments made in theupdateframework/python-tuf#451 and theupdateframework/python-tuf#465 to make these fields optional for the snapshot role also apply to the timestamp role.

Section 1.5.2 lists the attacks which the framework is designed to prevent against, but the phrasing used switches between "Attacker does X" and "Attacker should not be able to do X".

Attacker does X:

• "Arbitrary installation attacks. An attacker installs anything they want..."
• "Fast-forward attacks. An attacker arbitrarily increases the version numbers..."
• "Vulnerability to key compromises. An attacker who is able..." (This one is particularly confusing.)
• "Wrong software installation. An attacker provides a client..."

Attacker cannot:

• "Endless data attacks. Attackers should not be able to respond..."
• "Malicious mirrors preventing updates. Repository mirrors should be unable..."
• All the others, I think.

Please pick one and stick with it. I suggest the latter ("Attacker cannot")

It looks like https://updateframework.com has an invalid cert that doesn't match the domain. I'm not sure how much you care about that old domain, but it is linked from this discussion thread on securing RubyGems with TUF. That's how I found it. https://groups.google.com/forum/#!msg/rubygems-developers/lnnGTlfsuYo/TLDcJ2RPSDoJ

All documents use a subset of the JSON object format, with floating-point numbers omitted. When calculating the digest of an object, we use the "canonical JSON" subdialect as described at http://wiki.laptop.org/go/Canonical_JSON

Canonical JSON may be invalid JSON: canonical json says that control characters must not be escaped:

Because only two byte values are escaped, be aware that JSON-encoded data may contain embedded control characters and nulls.

Whereas JSON mandates that control characters are escaped.

I think this deserves a note in the specification, as normal json encoders and decoders cannot be used.

See, for example, the section 5.0 and 5.1 headings here. 5.1 is on the same outline level as 1.1 below it, and as a result, it's a bit hard to specify, say, item 4.5.1 in section 5 (or I guess it's in section 5.1, for that matter?).

We would like to make it possible to make changes to the TUF specification in a way that:

1. makes it easy to timestamp and archive versions of the TUF specification in semver style

2. enables there to be working drafts of new TUF versions. In particular, changes that are backwards incompatible (major), backwards compatible (minor), and typo fixes (patch) in parallel.

3. the manual effort to keep everything consistent is minimized.

To achieve these goals, I propose that there are three different "living" documents for the TUF spec (major-draft, minor-draft, and latest) that are edited in a very specific way. To provide a conceptual model that is easy to understand, the documents form a hierarchy, major (backwards-incompatible) -> minor (backwards compatible) -> patch (typo fixes). To minimize manual effort, changes in the document are only made to the drafts that need to be changed. To achieve this, all changes to lower documents propagate to higher documents. For example, if a typo is fixed in a patch, both the minor and major draft versions are updated to fix the typo. If a backwards incompatible change is made to the specification, only the major document changes.

Using such a hierarchy, it requires little effort to apply the correct semver number to a new release of TUF. Simply take the prior version and increment the version number that applies to the appropriate patch, while zeroing out all lower version numbers. For example, when at version 2.4.3 of the specification and accepting a minor draft, the new version number is 2.5.0. In all cases, the newly approved draft becomes the latest. If the major draft is accepted, then the previous major draft also becomes the new minor-draft and latest.

However, this does not handle the question of when to release a new version of the specification. A new version of the specification is released in two scenarios. First, whenever a patch change is accepted, the specification number is released. Second, when there is consensus in the community that a specific minor version or major version is ready, this is accepted and finalized.

Summary

The spec is not adequately clear on what a client should do when traversing a list of delegations if it cannot obtain and validate a role specified in that list. As a consequence, different clients may have different delegation order semantics -- intentionally or unintentionally.

Background

Near the bottom of section 4.5 in the spec is an explanation of delegation prioritization.

Details

Scenario:
Suppose some role "T" makes a non-terminating delegation of target files matching some_path/* to some roles "R1" and "R2", in that order.
(Note that the question is moot for a terminating delegation. "Terminating" is defined in the spec, but in short: if T makes a terminating delegation of namespace some_path/* to R1, then if the search for a target that matches some_path/* reaches the delegation to R1, it will not backtrack to proceed to R2 or further delegations in T's list of delegations. In other words, by making a terminating delegation of some_path/* to R1, T is excluding from its later delegations the right to specify a target that matches some_path/*)

Early delegation confers early right to specify a target. It is clear that if R1 specifies target info and a client finds a matching target file, that this should be considered the valid target file, and any target info R2 provides for this target should be disregarded. But what if a client can't find or validate an R1 role file? Should the client proceed to R2?

Solution

The behavior we think makes the most sense, to provide for a consistent meaning for delegations and prevent MITM attacks that exclude particular role files resulting in different targets being deemed valid, has the client halt its search for target file info for the given target if it cannot obtain or validate R1, and not proceed to R2 or backtrack, since R1 has the right to specify the target info for that target. This also is the behavior of the reference implementation.

The spec should indicate that this is the expected behavior, in spec section 5.1 (Detailed client workflows).

If this is not made rigid, it should at least be made clear that delegation order semantics will vary based on the behavior chosen. (People should not stumble into a behavior.)

In section 4.2, the spec states that RSA public keys are stored in PEM format, but may be vague. RSA PEM keys support two forms of encoding, RSAPublicKey form, as in:

-----BEGIN RSA PUBLIC KEY-----
MIIBCgKCAQEA4XLc9x7DX00eSQ4WvlqyojG1D/hr+X5Sn7dtd0Lq2MUarOqCRNDC
liKVni4ljwOADTn/+/JOgyS7Qf9RSi3KhKFMqEO83vKrbxJQHI4jK9kXlDtTPWmb
fdPzAMkc3tMNmp7M1DrYRUKR1++z5rb1AKCLIylhMp6j4oxOhKAc5ySKppMdr0EU
5Yih2kcqF/BmcIc6h/XNXS+iLOnOq9uT0+1VBhQ1iK542AM6XpXg3VADgq9PUzRf
ZfU4hVgbL6nTLH9j/lTyRn6Rl/rWgKSHo2xoAXrRTNJ5IEjfrSpsWrI4c9nSHs07
JosZDpmwJlC0GDET0ps18m1X8BTfj55UxQIDAQAB
-----END RSA PUBLIC KEY-----

And the newer SubjectPublicKeyInfo form, as in:

-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4XLc9x7DX00eSQ4Wvlqy
ojG1D/hr+X5Sn7dtd0Lq2MUarOqCRNDCliKVni4ljwOADTn/+/JOgyS7Qf9RSi3K
hKFMqEO83vKrbxJQHI4jK9kXlDtTPWmbfdPzAMkc3tMNmp7M1DrYRUKR1++z5rb1
AKCLIylhMp6j4oxOhKAc5ySKppMdr0EU5Yih2kcqF/BmcIc6h/XNXS+iLOnOq9uT
0+1VBhQ1iK542AM6XpXg3VADgq9PUzRfZfU4hVgbL6nTLH9j/lTyRn6Rl/rWgKSH
o2xoAXrRTNJ5IEjfrSpsWrI4c9nSHs07JosZDpmwJlC0GDET0ps18m1X8BTfj55U
xQIDAQAB
-----END PUBLIC KEY-----

Should we support both forms of key encoding, or just one? Presumably we need to at least support SubjectPublicKeyInfo for ECDSA keys.

The spec defines the spec_version" as:

SPEC_VERSION is the version number of the specification. Metadata is written according to version "spec_version" of the specification, and clients MUST verify that "spec_version" matches the expected version number. Adopters are free to determine what is considered a match (e.g., the version number exactly, or perhaps only the major version number (major.minor.fix).

Two requests:

• The document implies that spec_version is a string. Could this be explicitly mentioned?
• The document does not describe the format of the spec_version string. Can it contain arbitrary values, or is it only to be semver compatible?

Would it be acceptable to submit PRs to change the spec, or would these changes need to go through TAPs first?

The spec defines Timestamp and Snapshot metadata such that the meat of their data lies in a dictionary called meta. This should probably be called 'roleinfo' or something along those lines, if this metadata format is ever adjusted in an equally disruptive way. (It is probably not worth changing metadata definitions just for this, but the next time they do change, this is worth considering.)

Current:

"_type": "snapshot",
  "spec_version": "1",
  "expires": "2030-01-01T00:00:00Z",
  "meta": {
   "root.json": {
    "version": 1
   },
   "targets.json": {
    "version": 1
   },
   "project.json": {
    "version": 1
    },
   }
  "version": 1

I noticed that in 4.4.5 spec defines as TARGETPATH:

Each key of the TARGETS object is a TARGETPATH. A TARGETPATH is a path to a file that is relative to a mirror's base URL of targets. It should not have a leading path separator to avoid surprising behavior when constructing paths on disk.

Presumably this should be /, it says it's a file relative to the URL, and URLs use that as the path separator, but it'd be helpful if this was explicit.

This is relevant because in rust-tuf, we have a PathTranslator in order to use alternative path separators to support converting / to \ on windows, but I'm pretty sure this logic really should be moved to our FileSystemRepository.

spec says that all keys have the format

  { "keytype" : KEYTYPE,
    "scheme" : SCHEME,
    "keyval" : KEYVAL
  }

this matches the specific key format descriptions and later the examples.

The implementation at the moment looks like this however:

  { "keyid_hash_algorithms" : [ HASH_ALGORITHM, ... ] 
    "keytype" : KEYTYPE,
    "scheme" : SCHEME,
    "keyval" : KEYVAL
  }

where HASH_ALGORITHM is e.g. "sha256","sha512"

The Link to the submitted Mercury draft in Detailed Workflow 1.9 in the tuf_spec is broken.
it points to this link, but that is not available anymore.
https://ssl.engineering.nyu.edu/papers/kuppusamy_usenix_17.pdf

Is it maybe that link now:
https://ssl.engineering.nyu.edu/papers/kuppusamy-mercury-usenix-2017.pdf