project-oak / oak

Meaningful control of data in distributed systems.

Oak is a software platform for building distributed systems providing externally verifiable (or falsifiable) claims about system behaviors in a transparent way.

Oak provides building blocks that are used to build Enclave Applications, i.e. applications that are isolated from the host on which they run, and are remotely attestable.

In this doc we focus on distributed systems composed of heterogeneous "nodes"; a node may be an Enclave Application running on a server (e.g. private data center, public cloud provider, etc.), a regular application running on a client device (e.g. phone, laptop), a regular HTTP or gRPC service, a storage system, etc. When two nodes talk to each other, either or both of them may be Enclave Applications. For each interaction between two nodes, we refer to them as "initiator" and "responder". We avoid the terms "client" and "server" for those roles, because in some cases the "initiator" node may actually be running on a "server" - all combinations are possible.

An Enclave Application in an Oak distributed system can remotely attest itself to other nodes: this consists in providing evidence of its own software and hardware identity, bound to local cryptographic keys (stored in tamper-proof hardware, guaranteeing that keys cannot be exfiltrated or copied), ultimately signed by one or more software or hardware roots of trust available on the host of that node. The software identity of an Enclave Application consists of measurements (i.e. digests / hashes) of the executable binaries that contribute to it, as well as additional data (e.g. configuration).

Root of Trust

Oak focuses on VM-based Trusted Execution Environments (TEEs) (e.g. AMD SEV-SNP, Intel TDX) as hardware root of trust for Enclave Applications. This allows an Enclave Application to provide a remote attestation report signed by hardware-bound keys in the TEE, which are not accessible by the service provider. These keys are ultimately rooted in the TEE manufacturer (e.g. AMD, Intel); this reduces (ideally, removes) the need to trust the service provider of that node in order to validate its evidence, by distributing trust across multiple non colluding actors.

Encrypted Communication

At startup, each Enclave Application generates a local key pair, and binds the public key to its own evidence. When a node intends to communicate with an Enclave Application (as either initiator or responder), an end-to-end encrypted channel is established between the two nodes. As part of the channel establishment, each node also provides its own evidence (if any) to the other node, and may use the evidence provided by the other node to establish its trustworthiness, before deciding whether to exchange data with it.

This encrypted and attested channel provides:

• Confidentiality: each node knows that data sent over it may only be read by the other node
• Integrity: each node knows that data received over it must have been written by the other node
• Authenticity: each node knows the identity of the binary in any attested node processing the data it sends over the channel.

Split Architecture

In order to minimize the attack surface and the Trusted Computing Base (TCB) of an Enclave Application, Oak encourages a split architecture:

• Enclave Application: a binary built (reproducibly) from open source code, running inside the TEE. The Enclave Application has access to the decryption keys bound to the attestation evidence; it decrypts incoming requests from other nodes, processes them, encrypts responses and sends them back over the communication channel. This is the binary that is associated with externally verifiable claims. No other binary has access to incoming requests in plaintext.

• Host Application: a binary that runs on the host (untrusted) server environment that acts as the front-end for the Enclave Application; it exposes a gRPC endpoint which is externally reachable by other nodes. This binary only ever handles encrypted data; its main job is to create and maintain an instance of the Enclave Application in the TEE, and forward encrypted data to and from the Enclave Application. It may also contain proprietary logic that is needed to interact with the control plane or other internal systems necessary to run on the service provider infrastructure.

This architecture means the service provider would not be able to swap the Enclave Application binary with a different (potentially malicious) one without being detected. If the service provider were to introduce a backdoor in the Enclave Application, it would have to sign the binary and publish it in a verifiable log via the Transparent Release process, which would make the backdoor irreversibly visible to external observers.

Sealed Computing

A canonical use of Oak is to build privacy-preserving sealed computing applications.

In a sealed computing application, a node (usually a client device) sends data to an enclave application (usually a server), which processes data without the service provider hosting the enclave application being able to see the inputs, outputs, or side effects of the computation.

The following is a simplified architectural diagram.

Operating System

Oak supports two flavors of Operating Systems within the TEE, on which Enclave Applications may be deployed:

• Oak Restricted Kernel: a very minimal custom written microkernel that only supports a single CPU and a single application. It places very severe restrictions on what the application can do by only exposing a very limited set of syscalls to the application. Oak Restricted Kernel applications are suitable for cases where review of the entire trusted code base inside the TEE is critical, and the limited features and reduced performance is an acceptable trade-off. Ideally it should be possible for a single expert human reviewer to review the code of the Oak Restricted Kernel in its entirety in a couple of weeks.

• Oak Containers: The Enclave Application is provided as part of an OCI runtime bundle. This includes a full Linux kernel and a Linux distribution with a container runtime engine inside the TEE. Using the OCI runtime bundle as a compatibility layer provides a more familiar development experience and more flexibility. Oak Containers also supports multiple CPUs in the TEE and is therefore suitable for cases with high performance requirements. The drawback of this approach is that the size of the entire trusted code base running in the TEE is significantly larger, as it is not feasible for a single human reviewer to review the code for the entire Linux kernel and Linux distribution. The trust model for his approach relies on the security of the broader Linux ecosystem. The additional features and flexibility that the Linux kernel provides to the application also makes it more difficult to reason about the behavior of an application, which in turn makes the review process more complicated.

Remote Attestation

Remote Attestation is a process by which a TEE provides evidence to a remote party about its own state and the identity of the workload running inside the TEE (the Enclave Application).

Oak uses a multi-stage protocol in order to measure the identity of each layer in the boot process (beyond just the bootloader) and cryptographically bind it together with the TEE evidence. The Oak Stage 0 virtual firmware is pre-loaded into the VM memory, and it is covered by the TEE attestation report. To cover the later boot stages, Oak relies on the Device Identifier Composition Engine (DICE) approach, where each boot stage is responsible for measuring the next stage. Oak augments the attestation report with this additional evidence so that the identity of all the components of the workload is covered (e.g. kernel and application) by the evidence.

When a node connects to an Enclave Application, it first requests the attestation evidence and associated endorsements. The endorsements include a certificate chain from AMD to prove that the attestation report was signed by a legitimate AMD CPU and optionally signed statements from the developers of the various components running inside the enclave. The other node then verifies the evidence using these endorsements to make sure it matches its expectations. The client node should ensure that the enclave application is running in an up-to-date and correctly configured TEE (e.g. the CPU is running up to date versions of the microcode and SNP firmware and that debugging is disabled) and that the identity of the various components running inside the enclave matches expectations.

Once the node is satisfied that the enclave application meets all its expectations, it uses a public key that is bound to the evidence to set up an encrypted channel with the enclave application.

To be sure that the enclave application will handle data in accordance with expectations, external reviewers must review the code running in the TEE. Oak enables this by publishing provenance information of most Oak components. These act as a reverse index that allows a reviewer to look up the exact source code commit that was used to build each component given the measurement digest of the binary from the attestation evidence. All Oak components are reproducibly buildable (see Transparent Release below), which means that a reviewer can fetch the specific source code commit and rebuild the component to confirm that it produces the same exact measurement digest. This gives the reviewer confidence that the code version they are reviewing is the exact same code version that was loaded into the enclave during startup.

Transparent Release

Transparent Release is a set of formats and tools that allow developers to publish and consume claims about binary artifacts in a transparent and non-repudiable way.

As part of the remote attestation process, a node obtains the identity of the Enclave Application in the form of one or more binary digests (e.g. usually at least one per each layer of the boot chain); it then needs to establish whether these measurements are trustworthy.

As a first approximation, we could assume that the client knows exactly which digests to expect by hard-coding them, and can verify the actual values against the expected ones locally. This works, but it is very limiting with respect to the evolution of the server application: as soon as any of the Enclave Application binaries changes as part of a new release, the node will stop trusting the Enclave Application, and the two will not be able to interact any more.

We could solve this by hard-coding in the node a public key used to sign the binary digests, instead of the digests themselves. This way, when the Enclave Application binaries change, the developer can sign them with the corresponding private key in their possession, and distribute these signatures alongside the remote attestation report to the node, which can then verify the validity of these signatures. This works, but it has the problem of allowing the Enclave Application to present different signatures to different nodes, which would be very hard for users to detect.

The solution implemented is to sign not only the digests when a new release is created, but also to log this signature to an external append-only verifiable log. Oak relies on Rekor (by Sigstore) as the external verifiable log for this purpose. As part of the Transparent Release process the signature is logged to Rekor, and an inclusion proof (or inclusion promise) is obtained from it, signed by Rekor's root key. This is then stored alongside the binary artifact in question, and provided to nodes that may want to verify the identity of the Enclave Application. This makes it impossible for the developer to insert a backdoor in the Enclave Application without also logging it to the verifiable log.

Trust Model

We acknowledge that perfect security is impossible. In general, having a smaller Trusted Computing Base (TCB) is better, and Oak tries to minimize the TCB to the minimum components that need to be trusted.

Untrusted

Oak is designed to be verifiably trustworthy without needing to trust the following:

• most hardware (memory, disk, motherboard, network card, external devices)
• Host Operating System (kernel, drivers, libraries, applications)
• Hypervisor / VMM

The service provider may intend to access and exfiltrate users’ private data. They own the host machines in the data center, therefore have server root access and are capable of modifying the host machine OS, drivers, hypervisor, etc. They can also inspect network traffic that contains users’ data. In the Oak architecture, we treat the host hardware and the host OS operated by the service provider as untrusted. The data that need to be protected need to be encrypted when stored in the host OS’s memory system, with the decryption keys managed by the TEE, not accessible by the service provider hosting the hardware machines. The Oak architecture allows applications to make stronger claims by reducing the need to trust the service provider.

Trusted-but-verifiable

• Oak trusted platform
• Enclave application

Both the Oak platform libraries and components, and the Enclave Application that runs on that platform, run inside the TEE. They have access to unencrypted data. Oak establishes trust for these components by open sourcing them, and enables external inspection and verification via Transparent Release.

Trusted

• Hardware TEE manufacturer (e.g. AMD, Intel)

The hardware TEE is responsible for memory encryption and decryption, and generating the remote attestation report, etc.

No particular TEE manufacturer is absolutely trusted by Oak; rather, each client decides what TEE manufacturer(s) to trust when connecting to Enclave Applications running on other hosts. Oak makes it possible for Enclave Applications running on TEEs to present evidence rooted in the manufacturer of the TEE itself. Additional TEE models and manufacturers may be supported by Oak over time.

Side channel attacks

Side-channel attacks present significant challenges for confidential computing environments. We acknowledge that most existing TEEs have compromises and may be vulnerable to some side-channel attacks. This is an active research area, both hardware and software innovations are needed to defend fully against such attacks. Service providers hosting the TEE based servers need to implement strong host security. Strong security requires defense in depth.

Attacks that require physical access to the server hardware is another class of attacks that Oak and TEE hardware manufacturers cannot defend against alone. Physical attacks are expensive and therefore not scalable. To defend against them, service providers need to implement strong physical security in their data centers.

Getting involved

We welcome contributors! To join our community, we recommend joining the mailing list.

oak's Issues

Implement basic identity-based authorization

Currently each Oak Node allows clients to interact with it via gRPC based on just being able to connect to the port on which the Oak Node is listening, which means that anyone would be able to interact with it. Although the long-term story for policies is still being discussed, we will most definitely have to support some form of public key based authentication and authorization.

Asylo already supports cert identities as part of the gRPC channel credentials (see https://github.com/google/asylo/blob/master/asylo/identity/identity.proto#L40), so we should be able to obtain the credentials for any client connecting to the Oak Node and verify that they match to one of the expectations with which the Oak Node was initialized (by the Oak Scheduler). (cc @annasapek)

To start with we may just derive a single bit of information from this (authorized or not), but later on we should have roles, which will probably map to policies / capabilities (e.g. read, write, etc.)

handle panics in extern function calls

We currently rely on well-defined panic behaviour from extern functions. This is actually undefined behaviour.

oak/rust/oak/src/lib.rs

Line 283 in 6de88af

pub extern "C" fn oak_handle_grpc_call() {

oak/rust/oak/src/tests.rs

Lines 138 to 143 in 6de88af

	#[test]
	#[should_panic]
	fn test_handle_grpc_call_no_node() {
	reset_node();
	oak_handle_grpc_call(); // no node: panic!
	}

For some reason switching from stable to nightly Rust actually this manifests with a weird error:

error: process didn't exit successfully: `/usr/local/google/home/tzn/src/oak/rust/target/debug/deps/oak-092a7b70ef31d1ae` (signal: 4, SIGILL: illegal instruction)

Set up Continuous Integration

This may end up being more complicated than expected because of the additional dependencies needed by Asylo that need to be installed on the system or in a docker image in which tests are run.

Verify Asylo remote attestation in Oak `ApplicationClient`

Similar to #103, but at a lower level.
NodeClient must be configurable so that it can verify an attestation provided by Asylo.
See https://github.com/google/asylo/blob/master/asylo/identity/enclave_assertion_verifier.h

Improve mechanism for passing data from runtime to the Oak Module instance

Currently we use a combination of an owned vector and a cursor in the Oak Node to allow the Oak Module to consume data from the runtime.

oak/oak/server/oak_node.h

Lines 58 to 61 in c9c13c3

	std::unique_ptr<std::vector<char>> request_data_;
	// Cursor keeping track of how many bytes of request_data_ have been consumed by the Oak Module
	// during the current invocation.
	uint32_t request_data_cursor_;

For instance, when a gRPC call reaches the Oak Node, the oak_handle_grpc_call entry point of the module is invoked, with no arguments. The handler then needs to invoke the read_method_name and read host function calls in order to request the appropriate data (gRPC method name and payload, respectively) to be returned. This currently follows an approach similar to that of the read syscall (http://man7.org/linux/man-pages/man2/read.2.html) by which the module allocates a buffer of arbitrary size, passes its pointer and size to the runtime, and the runtime fills in the relevant data and returns the number of bytes written; the module then asks for more data, potentially having to reallocate the underlying buffer each time.

oak/oak/server/oak_node.cc

Lines 241 to 266 in c9c13c3

	::wabt::interp::HostFunc::Callback OakNode::OakRead(wabt::interp::Environment* env) {
	return [this, env](const wabt::interp::HostFunc* func, const wabt::interp::FuncSignature* sig,
	const wabt::interp::TypedValues& args, wabt::interp::TypedValues& results) {
	LOG(INFO) << "Called host function: " << func->module_name << "." << func->field_name;
	for (auto const& arg : args) {
	LOG(INFO) << "Arg: " << wabt::interp::TypedValueToString(arg);
	}
	// TODO: Synchronise this method.
	uint32_t p = args[0].get_i32();
	uint32_t len = args[1].get_i32();
	uint32_t start = request_data_cursor_;
	uint32_t end = start + len;
	if (end > request_data_->size()) {
	end = request_data_->size();
	}
	WriteMemory(env, p, request_data_->cbegin() + start, request_data_->cbegin() + end);
	results[0].set_i32(end - start);
	request_data_cursor_ = end;
	return wabt::interp::Result::Ok;
	};
	}

Note that we cannot just copy the memory from the runtime into the Wasm memory area, because the memory there is managed by the runtime (e.g. the Rust memory allocator), and therefore we cannot allocate it "from the outside" or it may conflict with it.

An initial improvement could be to remove the need for the vector and cursor (https://team.git.corp.google.com/project-oak-team/oak/+/refs/heads/master/oak/server/oak_node.h#61) and replace both with a single forward-only iterator, in order to avoid both copying the data in the Oak Module itself, and also storing explicit state to keep track of what the module has consumed so far; with an iterator, the state would be encapsulated in the iterator itself.

Test issue - please ignore

Testing slack integration

Use Bazel rules for Rust code

We are currently using Bazel for most of the C++ code, but Rust code is still built using cargo, and Rust protobuf are built manually by installing and running the protoc compiler, which makes achieving reproducible builds hard.

We should instead rely on https://github.com/bazelbuild/rules_rust , possibly by auto-generating BUILD files from the current Cargo.toml files using https://github.com/google/cargo-raze.

Migrate to Chrome V8 from wabt

V8 is Chrome's Javascript and WebAssembly engine. It is well tested in both server and client contexts, and provides multi-tenancy in form of isolates (see https://www.infoq.com/presentations/cloudflare-v8).

The plan is to replace wabt with V8, and have the Oak Manager create an isolate per Oak Node within the same Oak VM (as opposed to the current model in which each Oak Node runs a full independent Oak VM).

On server, a single Oak VM containing a single V8 instance will run within Asylo, relying on the underlying remote attestation mechanism to derive a key for the entire Oak VM; the Oak VM will then derive further keys (one per isolate) that it will use for attestation (both local and remote) between Oak Nodes.

On client, a hypervisor will run the Oak VM in a similar way, and provide it with the necessary key for attestation, which the Oak VM will then use in the same way as for the server case.

In both cases, the Oak VM as a whole is isolated from the host via the underlying enclave technology (Asylo or hypervisor), but isolation between Oak Nodes is only enforced by V8 itself.

Cache bazel artifacts in CI

I tried naively caching the whole bazel cache folder via the standard Travis mechanism, but it always fails to restore it, even if I increase the timeout. I think the cache is too large and there are too many small files for Travis to unpack it in a reasonable time.
Another option could be to use a Bazel remote cache, which has the advantage that we could also share the same cache when building locally.

Run examples as part of CI

Follow-up from #22.

Move OakScheduler class to separate file

oak/oak/server/oak_driver.cc

Lines 39 to 116 in c9c13c3

	class OakScheduler final : public ::oak::Scheduler::Service {
	public:
	OakScheduler() : Service(), node_id_(0) { InitializeEnclaveManager(); }
	::grpc::Status CreateNode(::grpc::ServerContext *context, const ::oak::CreateNodeRequest *request,
	::oak::CreateNodeResponse *response) override {
	std::string node_id = NewNodeId();
	CreateEnclave(node_id, request->module());
	oak::InitializeOutput out = GetEnclaveOutput(node_id);
	response->set_port(out.port());
	response->set_node_id(node_id);
	return ::grpc::Status::OK;
	}
	private:
	void InitializeEnclaveManager() {
	LOG(INFO) << "Initializing enclave manager";
	asylo::EnclaveManager::Configure(asylo::EnclaveManagerOptions());
	auto manager_result = asylo::EnclaveManager::Instance();
	if (!manager_result.ok()) {
	LOG(QFATAL) << "Could not initialize enclave manager: " << manager_result.status();
	}
	enclave_manager_ = manager_result.ValueOrDie();
	LOG(INFO) << "Enclave manager initialized";
	LOG(INFO) << "Loading enclave code from " << FLAGS_enclave_path;
	enclave_loader_ = absl::make_unique<asylo::SimLoader>(FLAGS_enclave_path, /*debug=*/true);
	}
	void CreateEnclave(const std::string &node_id, const std::string &module) {
	LOG(INFO) << "Creating enclave";
	asylo::EnclaveConfig config;
	oak::InitializeInput *initialize_input = config.MutableExtension(oak::initialize_input);
	initialize_input->set_node_id(node_id);
	initialize_input->set_module(module);
	asylo::Status status = enclave_manager_->LoadEnclave(node_id, *enclave_loader_, config);
	if (!status.ok()) {
	LOG(QFATAL) << "Could not load enclave " << FLAGS_enclave_path << ": " << status;
	}
	LOG(INFO) << "Enclave created";
	}
	oak::InitializeOutput GetEnclaveOutput(const std::string &node_id) {
	LOG(INFO) << "Initializing enclave";
	asylo::EnclaveClient *client = enclave_manager_->GetClient(node_id);
	asylo::EnclaveInput input;
	asylo::EnclaveOutput output;
	asylo::Status status = client->EnterAndRun(input, &output);
	if (!status.ok()) {
	LOG(QFATAL) << "EnterAndRun failed: " << status;
	}
	LOG(INFO) << "Enclave initialized";
	return output.GetExtension(oak::initialize_output);
	}
	std::string NewNodeId() {
	// TODO: Generate UUID.
	std::stringstream id_str;
	id_str << node_id_;
	node_id_ += 1;
	return id_str.str();
	}
	void DestroyEnclave(const std::string &node_id) {
	LOG(INFO) << "Destroying enclave";
	asylo::EnclaveClient *client = enclave_manager_->GetClient(node_id);
	asylo::EnclaveFinal final_input;
	asylo::Status status = enclave_manager_->DestroyEnclave(client, final_input);
	if (!status.ok()) {
	LOG(QFATAL) << "Destroy " << FLAGS_enclave_path << " failed: " << status;
	}
	LOG(INFO) << "Enclave destroyed";
	}
	asylo::EnclaveManager *enclave_manager_;
	std::unique_ptr<asylo::SimLoader> enclave_loader_;
	uint64_t node_id_;
	};

Run Rust tests in CI

Upgrade to asylo-3.4.0

As I see asylo-3.4.0 introduced some substantial changes in its build rules. I was able to build Oak with asylo-3.4.0 with some minor modifications, I'd advise doing so in master.

Improve gRPC async generic service loop

Currently we process gRPC generic events in a "synchronous" way by just queuing each even and processing it immediately after in the same loop, but this is not idiomatic, and I think it also has the potential of breaking (e.g. if an additional event is inserted in between in some other way, the loop may get out of sync). We should change this to pass a reference to each task as part of the gRPC event tag, and then looking up the task when processing each event from the queue.

cc @vjpai who has kindly advised on this in the past.

oak/oak/server/enclave_server.h

Lines 159 to 198 in c9c13c3

	// Consumes gRPC events from the completion queue in an infinite loop.
	void CompletionQueueLoop() {
	LOG(INFO) << "Starting gRPC completion queue loop";
	int i = 0;
	while (true) {
	::grpc::GenericServerContext context;
	::grpc::GenericServerAsyncReaderWriter stream(&context);
	// We request a new event corresponding to a generic call. This will create an entry in the
	// completion queue when a new call is available.
	generic_service_.RequestCall(&context, &stream, completion_queue_.get(),
	completion_queue_.get(), tag(i++));
	// Wait for a generic call to arrive.
	ProcessNextEvent();
	// Read request data.
	::grpc::ByteBuffer request;
	stream.Read(&request, tag(i++));
	// Process the event related to the read we just requested.
	ProcessNextEvent();
	::grpc::ByteBuffer response;
	// Invoke the actual gRPC handler on the Oak Node.
	::grpc::Status status = node_->HandleGrpcCall(&context, &request, &response);
	if (!status.ok()) {
	LOG(INFO) << "Failed: " << status.error_message();
	}
	::grpc::WriteOptions options;
	// Write response data.
	stream.WriteAndFinish(response, options, status, tag(i++));
	// Process the event related to the write we just requested.
	ProcessNextEvent();
	}
	}

Code generation for Oak Module gRPC interface

Currently each Oak Module needs to explicitly dispatch gRPC calls based on the incoming gRPC method name, then deserialise the incoming request according to the appropriate protobuf type, and eventually serialise the outgoing response. All of this is very mechanical and in fact is fully determined by the gRPC service definition in the source protobuf file. We should rely on code generation to automate and hide all of this complexity from module authors.

A possible approach would be to start from https://github.com/stepancheg/grpc-rust and modify it so that it generates the server trait / base struct. Note we do not need to generate clients, nor any of the transport layer logic.

See

oak/examples/running_average/src/lib.rs

Lines 43 to 64 in fbcd235

	// TODO: Generate this code via a macro or code generation (e.g. a protoc plugin).
	match method_name {
	"/oak.examples.running_average.RunningAverage/SubmitSample" => {
	let mut in_stream = protobuf::CodedInputStream::new(request);
	let mut req = proto::running_average::SubmitSampleRequest::new();
	req.merge_from(&mut in_stream)
	.expect("could not read request");
	self.sum += req.value;
	self.count += 1;
	}
	"/oak.examples.running_average.RunningAverage/GetAverage" => {
	let mut res = proto::running_average::GetAverageResponse::new();
	let mut out_stream = protobuf::CodedOutputStream::new(response);
	res.average = self.sum / self.count;
	res.write_to(&mut out_stream)
	.expect("could not write response");
	out_stream.flush().expect("could not flush");
	}
	_ => {
	panic!("unknown method name");
	}
	};

Add rust-clippy to CI

We should install this in the base Docker image and then run it in the same way as we currently run other format / lint checks.

See https://github.com/rust-lang/rust-clippy .

Check BUILD files format with buildifier

See https://github.com/bazelbuild/buildtools/releases

Implement Rust logging facade

We should have a way for an Oak Module to initialize a log connected to an output channel.

Note when I tried this in the past it resulted in some odd behaviour because the logger may perform multiple calls to write for a single log message (e.g. when interpolating arguments). Perhaps we should wrap the output channel with a LineWriter or something like that.

C++ example module not building on Travis

Looking inside a recent Travis build of master, the C++ example module is failing to build:

ERROR: /opt/my-project/examples/hello_world/module/cpp/BUILD:17:1: C++ compilation of rule '//examples/hello_world/module/cpp:hello_world.wasm' failed (Exit 127) clang.sh failed: error executing command toolchain/clang.sh -MD -MF bazel-out/wasm32-fastbuild/bin/examples/hello_world/module/cpp/_objs/hello_world.wasm/hello_world.d ... (remaining 20 argument(s) skipped)
Use --sandbox_debug to see verbose messages from the sandbox
[4 / 5] checking cached actions
/bin/clang -MD -MF bazel-out/wasm32-fastbuild/bin/examples/hello_world/module/cpp/_objs/hello_world.wasm/hello_world.d -frandom-seed=bazel-out/wasm32-fastbuild/bin/examples/hello_world/module/cpp/_objs/hello_world.wasm/hello_world.o -iquote . -iquote bazel-out/wasm32-fastbuild/bin -iquote external/bazel_tools -iquote bazel-out/wasm32-fastbuild/bin/external/bazel_tools -ffreestanding -isystem external/clang_llvm/lib/clang/8.0.0/include -isystem external/clang_llvm/include/c++/v1/ --target=wasm32-unknown-unknown -nostdlib -c examples/hello_world/module/cpp/hello_world.cc -o bazel-out/wasm32-fastbuild/bin/examples/hello_world/module/cpp/_objs/hello_world.wasm/hello_world.o
external/clang_llvm/bin/clang: error while loading shared libraries: libtinfo.so.5: cannot open shared object file: No such file or directory
[4 / 5] checking cached actions
ples/hello_world/module/cpp:hello_world.wasm failed to build
[4 / 5] checking cached actions
mand lines of failed build steps.
[4 / 5] checking cached actions
INFO: Elapsed time: 45.133s, Critical Path: 0.42s
[4 / 5] checking cached actions
INFO: 0 processes.
[4 / 5] checking cached actions
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

It would be good if:

• The C++ module built correctly on Travis.
• Failures to build the C++ module on Travis got reported as failed builds.
• The C++ module got run as part of the EXAMPLES_TEST=true Travis run.

New example: rustfmt

Create a new example that runs the rustfmt formatter in an Oak Node, and a sample client that sends a string containing a snippet of Rust code and gets back its formatted version.

Add rustfmt check to CI

Implement Private Set Intersection (PSI) example

Some references:

• https://www.microsoft.com/en-us/research/publication/fast-private-set-intersection-homomorphic-encryption/
• https://eprint.iacr.org/2017/670.pdf
• https://signal.org/blog/private-contact-discovery/

To start with we can probably model this as a gRPC-based Oak Module with the following methods:

• SubmitSet
• RetrieveCurrentIntersection

Initial assumptions:

• (unordered) sets of strings
• small size (max ~1MB / set)
• sets from different clients may have different size
• side channels (e.g. size, completion time, memory access patterns) are out of scope

The workflow would involve the clients connecting to the Oak Node one after the other, submitting their set, then assuming there is a way of directly signalling to each other, they would each call the method to retrieve the current intersection, which should return the same value for all of them.

e.g.

• client_0 sends the set {A, B, C, F}
• client_1 sends the set {C, D, F, G}
• client_0 retrieves current intersection, receiving {C, F}
• client_1 retrieves current intersection, receiving {C, F}

Oak Dev Server crashes on master when running examples with a segfault

Steps to repro:

1. bazel run //oak/server/dev:oak
2. Wait for server to start
3. ./scripts/run_example hello_world
4. Observe seg fault on server

Document build steps without Docker

Add documentation for building without Docker on selected platforms (Ubuntu?). I was able to build it on Arch, it should be possible to do so on arbitrary distros. Note that the project doesn't compile with Bazel version <0.2.3, it should be asserted too.

Target //oak/server:oak_manager can't be built on its own

If you build only oak_manager with bazel build --config=enc-sim //oak/server:oak_manager you'll meet the following error:

C++ compilation of rule '@linux_sgx//:urts_sim' failed (Exit 1) x86_64-elf-g++ failed: error executing command external/com_google_asylo_toolchain/toolchain/bin/x86_64-elf-g++ -isystemasylo/platform/posix/include -isystemasylo/platform/system/include ... (remaining 86 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
In file included from external/linux_sgx/common/inc/internal/se_wrapper.h:40:0,
                from external/linux_sgx/psw/urts/linux/urts_internal.cpp:40:
external/linux_sgx/common/inc/internal/se_thread.h:43:10: fatal error: sys/syscall.h: No such file or directory
#include <sys/syscall.h>
         ^~~~~~~~~~~~~~~
compilation terminated.
Target //oak/server:oak_manager failed to build

When it'll come to writing tests IMO it'll be problematic.

Upgrade Rust version to 1.36.0

oak/Dockerfile

Line 27 in 8006ba7

ARG RUST_VERSION=1.35.0

Create C++ SDK

Currently we only have a Rust SDK (https://github.com/project-oak/oak/blob/master/rust/oak/src/lib.rs) to facilitate interactions with the Oak WebAssembly runtime.
We should implement a similar abstraction layer for C++ so that we can port existing C++ software to Oak.

Add clang-format check to CI

API for creating static attestations

It would be useful to have a way for Oak Modules to produce a stand-alone attestation (e.g. a signature on top of user-provided data) rooted in the underlying enclave.

Some use cases that this could be useful for:

• attesting to the output of a long / expensive computation (e.g. compute 1B digits of pi)
• attesting to the artifact produced by a compiler given a specified source
• attesting to the verification / summary of a log performed by a trusted aggregation function (e.g. verifying a Trillian log)

This should probably be a native WebAssembly host function implemented by the Oak VM and exposed to the Oak Modules, and also correspondingly wrapped in more ergonomic API in Rust or future SDKs.

Verify Oak remote attestation in Oak Node Client

NodeClient must accept an attestation assertion in its constructor; then when establishing a connection to a OakNode instance it must verify whether the remote attestation provided by the OakNode is allowed by the assertion.
For now let us assume that the underlying enclave-level assertion is already verified (most likely via an Asylo assertion at the channel level, which will attest whether the gRPC channel terminates inside an SGX enclave or similar), even though in practice that is not implemented yet.
See #6 for more context.

Support Rust async programming model in SDK

This should allow writing idiomatic async / await code.

See

• https://doc.rust-lang.org/std/future/trait.Future.html
• https://docs.rs/tokio/0.1.22/tokio/reactor/struct.Reactor.html
• https://www.viget.com/articles/understanding-futures-in-rust-part-1/
• https://www.viget.com/articles/understanding-futures-is-rust-part-2/
• https://boats.gitlab.io/blog/post/wakers-i/
• https://boats.gitlab.io/blog/post/wakers-ii/

Build examples as part of CI

Requires installing Rust and Bazel as part the CI setup, among other things.

Install panic handler in Rust Oak Node

This will allow logging panics to the existing log channel, if one is present.

See https://doc.rust-lang.org/std/panic/struct.PanicInfo.html .

Project can't be loaded with CLion

This is probably an issue with either (a) Bazel plugin for CLion or (b) Asylo, but I'm still debugging. The problem occurs when CLion tries to load target oak_enclave.so

Traceback (most recent call last):
	File "/home/ahomolya/.cache/bazel/_bazel_ahomolya/344f6b368a5432cb81bfc09bc231b915/external/com_google_asylo_toolchain/toolchain/BUILD", line 74
		@intellij_aspect//:intellij_info_bundled.bzl%intellij_info_aspect(...)
	File "/home/ahomolya/.cache/bazel/_bazel_ahomolya/344f6b368a5432cb81bfc09bc231b915/external/intellij_aspect/intellij_info_bundled.bzl", line 53, in _aspect_impl
		intellij_info_aspect_impl(target, ctx, semantics)
	File "/home/ahomolya/.cache/bazel/_bazel_ahomolya/344f6b368a5432cb81bfc09bc231b915/external/intellij_aspect/intellij_info_impl.bzl", line 816, in intellij_info_aspect_impl
		collect_c_toolchain_info(target, ctx, semantics, ide_info, <2 more arguments>)
	File "/home/ahomolya/.cache/bazel/_bazel_ahomolya/344f6b368a5432cb81bfc09bc231b915/external/intellij_aspect/intellij_info_impl.bzl", line 418, in collect_c_toolchain_info
		cc_common.get_memory_inefficient_command_line(feature_configuration = feature_..., <2 more arguments>)
Invalid toolchain configuration: Cannot find variable named 'output_file'.

Run clang-tidy as part of CI

I found an other GitHub project who were able to get clang-tidy work with Bazel and run it in the CI pipeline: envoyproxy/envoy#921.

We could adopt this too.

Implement Oak remote attestation over gRPC

Each Oak Node should produce a remote attestation that confirms to the client the policy configuration and the Oak Module that are bound to that Oak Node. This should happen after the lower-level attestation (via Asylo) is verified, but before any actual data is transferred between client and server.

oak/oak/server/oak_node.cc

Lines 349 to 353 in c9c13c3

	::grpc::Status OakNode::GetAttestation(::grpc::ServerContext* context,
	const ::oak::GetAttestationRequest* request,
	::oak::GetAttestationResponse* response) {
	return ::grpc::Status(::grpc::StatusCode::UNIMPLEMENTED, "TODO");
	}

Implement Storage Service for Private Data Store

Unify error handling

We currently use a combination of the following approaches for error handling:

• return ::grpc::Status
• return ::asylo::StatusOr
• log errors (at inconsistent levels) and discard them

For gRPC methods it makes sense to keep returning ::grpc::Status, but for general methods that may fail we should probably converge on something like StatusOr; unfortunately this is not available in absl yet, but I think we could piggyback on the asylo implementation for now, and switch to the absl version once it becomes available.

@michael-kernel-sanders WDYT?

Support multi-node applications

Set docker UID

By default, Docker container has uid 0 (root). This is not only a security risk, but annoying because any files output by the container will be owned by root.

Oak Node lifetime management

We should support a way to manage lifetime of Oak Nodes. For now this may be an RPC on Oak Manager that allows the caller to shut down a running Application.

oak/oak/proto/manager.proto

Lines 135 to 143 in fb64cf6

	// Untrusted service in charge of creating Oak Applications on demand.
	service Manager {
	// Request the creation of a new Oak Application with the specified configuration.
	//
	// After the Oak Node is created, the client should connect to the returned
	// endpoint via gRPC and perform a direct attestation against the Node itself,
	// to verify that its configuration corresponds to what the client expects.
	rpc CreateApplication(CreateApplicationRequest) returns (CreateApplicationResponse);
	}

Add clang-format to the setup instructions

Implement host/module functions for Private Data Store

Allocate new port for each Oak Node instance

Currently the Oak Scheduler creates new Oak Node instances all listening on the same port (see

oak/oak/server/enclave_server.h

Line 107 in c9c13c3

builder.AddListeningPort("[::]:30000", credentials_, &port_);

), which means that when a client attempts to connect to it and there are multiple Oak Nodes listening there, the client will be connected to an arbitrary Oak Node among those running there.

We should instead let each Oak Node pick a free port on the host, and return that as part of the scheduling response.

Note the main reason why we have not done this yet is because we run the Oak Scheduler in Docker, and we need to list in advance all the ports to expose (see

oak/scripts/run_server_docker

Line 16 in c9c13c3

--publish=30000:30000 \

)

Embed Wasm engine using wasm-c-api

wasm-c-api

Provide[s] a "black box" API for embedding a Wasm engine in other C/C++ applications.

There should be prototypes of V8 implementing this interface, and hopefully also other Wasm engines; not sure if wabt, which we are currently using, also implements this.

This should also help with #74 (in fact I would say this is a prerequisite).

cc @hansman

Split the build scripts for rust and C++ modules

Right now we use params to configure things, which is not very nice.
We need to rethink the way we build server, client and modules.

Use `dyn` when dispatching on traits in generated gRPC code

warning: trait objects without an explicit `dyn` are deprecated
  --> rustfmt/module/rust/src/proto/hello_world_grpc.rs:38:28
   |
38 | pub fn dispatch(node: &mut HelloWorldNode, grpc_method_name: &str, grpc_pair: &mut oak::ChannelPair) {
   |                            ^^^^^^^^^^^^^^ help: use `dyn`: `dyn HelloWorldNode`
   |
   = note: `#[warn(bare_trait_objects)]` on by default

Bazelify V8: Link V8 engine in Bazel builds.

In preparation of #74 and #126 we want to link V8 so that it can be included in Bazel BUILD files as dependency.

stand-alone Oak VM

We should create new versions of the Oak VM and Oak Manager that do not depend on Asylo for isolation and remote attestation. The entry point should be a separate implementation of Oak Manager that implements the same gRPC interface as the current one:

oak/oak/proto/manager.proto

Lines 136 to 143 in 7481fe2

	service Manager {
	// Request the creation of a new Oak Application with the specified configuration.
	//
	// After the Oak Node is created, the client should connect to the returned
	// endpoint via gRPC and perform a direct attestation against the Node itself,
	// to verify that its configuration corresponds to what the client expects.
	rpc CreateApplication(CreateApplicationRequest) returns (CreateApplicationResponse);
	}

but instead of creating enclaves via Asylo, it just runs new Oak Nodes as separate processes (or potentially V8 isolates, in the future).

For now, each Oak Node will still run with a stand-alone gRPC server stack; in the future we should also refactor this so that we can run Oak Nodes with minimal support from the OS (ideally only communication channels), and the networking stack would live outside of the Oak Node itself. Obviously any data going in or out of the Node would still have to be encrypted by the Node itself.

Support client-streaming gRPC methods (in both gRPC server and client pseudo-Nodes)

Currently we only support unary RPCs, but gRPC allows any combination of client and server streaming.
See https://grpc.io/docs/guides/concepts/ for more context.