cloudevents / spec Goto Github PK
View Code? Open in Web Editor NEWCloudEvents Specification
Home Page: https://cloudevents.io
License: Apache License 2.0
CloudEvents Specification
Home Page: https://cloudevents.io
License: Apache License 2.0
It would be really helpful to have a list of contributors, to solve two problems:
Assigned to: @ultrasaurus
We need the content-type
attribute to be explicit (not part ofthe extensions), much like the schema is, without it there is no way to know how to de-serialize the data
Assigned to: @yaronha
The description for schema-url is "A link to the schema that the data attribute adheres to". What is meant by "data attribute"?
I'm trying to get what context this initiative is trying to cover.
I run the Proto.Actor project, cross platform actor framework (http://proto.actor)
For us, things like batching to increase throughput and reduce overall message size is important.
e.g. we buffer messages either until the buffer is full or deadline timeout and then compress the payload by replacing message names/namespaces with identifiers unique for the message batch.
We also heavily rely on things like location transparency, we have the concept of PID (process ID, copied from Erlang, ActorRef in Akka).
Are things like this taken into consideration when designing the spec?
At what level does the spec live, is it down to the raw bits and bytes serialization level?
Or is it more of a checklist that applies above this?
e.g. you can be cloudevent compatible while using either JSON or ProtoBuf or other serializers?
https://github.com/cloudevents/spec/blob/master/spec.md#event
The specification should either drop "occurrence" or nor restate a partial definition in "event".
Suggestion:
An "event" is a data record expressing an occurrence and its context or a set of occurrences and their shared context. Events are routed from the emitting [re]source to interested parties for the purpose of notifying the about the [re]source occurrence. The routing may be performed based on information contained in the event, but an event will generally not identify a specific routing destination.
Assigned to: @clemensv
The "source" property is, irrespective of being interpreted as a URI or URL, already an identifier for the source itself. The "source-id" therefore appears superfluous.
However, the present schema does not provide any further context qualification for the event relative to the source, so I am proposing to repurpose the source-id to provide that context qualification. In common messaging APIs and protocols (including the emails stack), the qualification property is called "subject".
Furthermore, with "source" already being a structured identifier (URI) and the event being sufficiently qualified by namespace/event-type and the subject, I question the practical value of the "source-type" property and propose for it to be removed.
Assigned to: @clemensv
Definition of Event introduces terms notification
and message
without clearly defining their meaning in the context of an event-driven system. There may be some implicit assumptions/understanding in there that be explicitly written.
Assigned to: @schatekar
Proposal to adjust the roadmap to have numbered milestones to help focus our work.
Based on a brainstorm with @ac360:
https://docs.google.com/document/d/1ZLCWPeNcNyjttvgo2QHY9CMtzDaA0_81vmh5C-r9QB4/edit
Assigned to: @ultrasaurus
This is a proposed metadata attribute (from Context Attribute Backlog):
causation-id
Type: String
Description: event-id of event that triggered this event to be created (if any). If not triggered in response to an event, then not present. Enables traceability through systems.
Examples:
Event A generated with unique event-id. Event handler receives event and generates a new event in response, Event B. Event B copies the event-id of Event A into its own causation-id so that the chain of events can be properly correlated.
Constraints:
Optional
Reliable specifications (protocols) require validation and testing to guarantee interoperability, particularly as a specification evolves (versions over time). We're in need of a tool (maybe two?) to validate event formats of operational services to verify their conformance to the normative statements in the CloudEvents specification. To facilitate validation, a checklist of assertions (as a document separate from the CloudEvents spec) may be needed.
Both the conformance tool and list of assertions (requisite tests) should be versioned alongside the CloudEvents specification.
This "CloudEvents Service Validator" would ideally be an open source framework for checking conformance of any generic service implementing the CloudEvents format as defined in the CloudEvents specification. The tool is designed to be vendor/project agnostic and driven purely based on the CloudEvents specification.
Integrity of validation results need to be guaranteed. Therefore, hosting of such a small service validator may be required.
Events occur on different layers of a system. This might lead to nesting of events. E.g. an API gateway may receive an HTTP POST from a web hook, which already contains event data from a SaaS application. In addition the API gateway may be configured to raise an event to a function, when a POST request to a certain URL is received.
How is this situation handled?
The API gateway wraps the SaaS application event into an API gateway event and sends it to the function.
The API gateway recognizes the event context in the HTTP header and forwards the application event to the function using the protocol of choice. No API gateway event would be triggered.
This is a proposed metadata attribute (from Context Attribute Backlog):
receipt-queue
Type: String
Description: An optional return endpoint for completion events
Decide and document if the event-type
attribute should include a version. A version is currently included in a separate attribute, event-type-version
.
This is a proposed metadata attribute (from Context Attribute Backlog):
content-type
Type: String
Description: The data encoding scheme (e.g. application/json). The data content type.
Constraints:
Required if the event contains a body.
Decide and document if the event-type
attribute should include a type format.
This is a proposed metadata attribute (from Context Attribute Backlog):
log-level
Type: String
Description: The level of logging that the publisher may want to log this specific event (e.g. debug)
Assigned to: @vbmade2000
On February 1st 2018, we tentatively changed "resource" to "source". We agreed that "resource" is too confusing, but also that "source" may not be the best name. After clarify the meaning of this term we can reaffirm or change the name.
Assigned to: @inlined
Objectives
Overview
This proposal is for a library that transforms common events (e.g. AWS S3 Object Created, webhooks) into the CloudEvents format.
The library could be used by publishers, middleware and subscribers. In the case of serverless, the library could be used at the beginning of serverless function handlers, so the user only needs to work with the CloudEvents format.
Example:
// Node.js Example
// Load the cloudevents Node.js library
var cloudevents = require('cloudevents')
module.exports = function(event, context, callback) {
// Perform transformation of AWS event format into CloudEvents event format
event = cloudevents.transform(event)
// Write your business logic around the familiar cloudevents format
return callback(null, 'success')
}
V.0 Specs
// AWS S3 Object Created > CloudEvents
// Keys are CloudEvents properties and values are provided by the original event format
{
"event-type": "aws.s3.(${eventName}='ObjectCreated:Put'?'object.created')",
"event-type-version": "${eventVersion}",
"data": "${s3}"
}
Future
Since today we don't have a defined set of "maintainers" to approve PRs we decided to follow a model that more akin to how a standard body works. Which is, sort of, ok for now, but not ideal for a variety of reasons - biggest being that not everyone can make the calls.
I'd like to use this issue to track ideas for how we migrate to a "maintainer" process and, in particular, how we identify the initial set of maintainers. It might be as simple as take nominations and vote - I dunno. But let's use this issue to gather ideas and then we'll work on creating a PR with a proposed solution.
This is a proposed metadata attribute (from Context Attribute Backlog):
correlation-id
Type: String
Description: Correlation ID of event that triggered this event to be created (if any). If not triggered in response to an event, a unique ID. Can be specified by the producer if event not produced in response to an event. Enables traceability through systems.
Examples:
Event A generated with unique correlation-id. Event handler receives event and generates a new event in response, Event B. Event B copies the correlation-id of Event A so that the chain of events can be properly correlated.
Constraints:
Optional
Assigned to: @cathyhongzhang
This is a question/suggestion. Should we have a "What OpenEvening is not section" in the spec?
Apologies if I have got the wrong end of the stick, but when we say events, a lot of things come to my mind - pub/sub, ESB, queueing systems etc. I don't think we are trying to address "event in every situation" problem. Or are we?
Assigned to: @schatekar
Assigned to: @cathyhongzhang
I think we need to add some description on:
Decide and document if the schema-url attribute is applicable to the whole event or to the data alone.
Assigned to: @duglin
What are the steps someone needs to take to “cut a release”?
E.g. tags vs branches, which docs are updated with version #’s...
I would suggest that when cut our first formal release, we just document the steps taken by whoever does it so that the next time people can just follow their lead and we're consistent across releases.
Add text: This is a list prospective event metadata attributes (“context”) for potential inclusion in the future.
Assigned to: @cathyhongzhang
This is a proposed metadata attribute (from Context Attribute Backlog):
path
Type: String
Description: The destination endpoint address (e.g. URL) or target topic
In some use cases, a function might be triggered to run by a combination of events. For example, a function could be triggered when both event-x and event-y happen or when either event-x or event-y happen. We may need to consider this in this spec?
Assigned to: @cathyhongzhang
Decide and document where the producer specifies the format/encoding of the data attribute in this schema.
Should we have nested properties for some things? eg. resource-type and resource-id under "resource". Or should we keep everything at the top-level?
Suppose the event comes from a AWS S3, what is the event source, event source type, event type, event Identifier? From current definition, it is not very clear.
Assigned to: @cathyhongzhang
namespace
identify event or event source (producer)?namespace
adds on top of source-id
?event-type
, event-type-version
) can be published by more than one event source, so with difference in at least one of (source-id
, source-type
)?namespace
identifies event source and there's value added of namespace
on top of source-id
would it make sense to include namespace
together with (source-id
, source-type
) as property of source object?This is a proposed metadata attribute (from Context Attribute Backlog):
method
Type: String
Description: The (http) method used in the call
The examples under the Reference section of the spec are useful, but it would be enlightening for a more outside observer like me if they had a comparison with what those events might look like when transformed into the CloudEvents spec.
This was mentioned in the call today. We're currently hosting a simple site on SquareSpace. Do we want to use something like Hugo and just run it using GitHub Pages? The current site is pretty simple, so it wouldn't take much to convert it.
Here are some potential theme options:
These basically represent three options that are quite different from each other, but all can allow the website to grow without changing much as more is added or linked to. The first option is the most basic and is very similar to what exists today. The second is a little more advanced and allows us to link to multiple items in GitHub from the main page. The third is the most elaborate with a picture and a simple blog capability.
I think my preference would be for the second option. I'm happy to do any work to transfer what exists today. I just want to know if there's any interest in this.
Rename attributes 'resource*' to 'source*', or alternatively 'producer*', for clarity.
Assigned to: @lfourie
We would like to write up some info so that folks can arrive at this repo and understand what kind of event system we're talking about.
Starting docs:
presentation -- slide 23 (below) shows the difference between general messaging, and "Eventing" which is what CloudEvents is focused on. Events are statements of fact.
https://github.com/cncf/wg-serverless/tree/master/proposals/cloudevents
OpenWhisk has a nice illustration of the kind of system we're talking about CloudEvents:
Assigned to: @ultrasaurus @yaronha
This is a proposed metadata attribute (from Context Attribute Backlog):
authentication
Type:
Description:
Just to capture a straw-man for HTTP mappings for discussion purposes - not about the mapping per-se (that might be too early at this point) but to illustrate concrete usage of the properties in the core spec.
With the properties that are in the spec as of today, I'm going to show two candidate mappings for HTTP. One carries an opaque binary payload as-is with the data field being mapped to the HTTP payload, the other carries an event consisting of JSON data. I believe that both scenarios have legitimate use cases.
Upside of the binary mapping is that binary payloads can be routed w/o size penalty. Downside is that you'll have to juggle HTTP headers, which is painful in many environments. Upside up the JSON format is that you have it all in one object, but non-JSON data effectively needs to be treated as binary and base64 encoded.
JSON mapping with JSON data has priority for Microsoft.
Binary: Cloud events properties are mapped to HTTP headers. Content-Type is mapped to the native header.
POST {external} HTTP/1.1
CloudEvents-Namespace: azure.microsoft.com/storage
CloudEvents-EventType: Microsoft.Storage.BlobCreated
CloudEvents-EventTypeVersion: 2018-01-20
CloudEvents-Version: 0.2
CloudEvents-Topic: subscriptions/{subscription-id}/resourceGroups/Storage/providers/Microsoft.Storage/storageAccounts/xstoretestaccount
CloudEvents-Subject: /blobServices/default/containers/oc2d2817345i200097container/blobs/oc2d2817345i20002296blob
CloudEvents-EventId: 831e1650-001e-001b-66ab-eeb76e069631
CloudEvents-EventTime: 2017-06-26T18:41:00.9584103Z
**Content-Type**: application/x-my-opaque-binary
Content-Length: 1234
{Data}
JSON: Here, all the cloud events properties are mapped inline at the outer scope of a JSON object. The "data" object contains event-specific data, here an Azure blob store event. I think this will be the more common form.
POST {external} HTTP/1.1
Content-Type: application/json
Content-Length: nnnn
{
"namespace" : "azure.com"
/oc2d2817345i20002296blob",
"cloud-events-version": "0.2"
"event-type": "Microsoft.Storage.BlobCreated",
"event-type-version": "1",
"event-time": "2017-06-26T18:41:00.9584103Z",
"event-id": "831e1650-001e-001b-66ab-eeb76e069631",
"topic": "subscriptions/{subscription-id}/resourceGroups/Storage/providers/Microsoft.Storage/storageAccounts/xstoretestaccount",
"subject": "/blobServices/default/containers/oc2d2817345i200097container/blobs
"schema-url": "https://raw.githubusercontent.com/Azure/azure-rest-api-specs/master/specification/eventgrid/data-plane/Microsoft.Storage/stable/2018-01-01/Storage.json",
"data": {
"api": "PutBlockList",
"clientRequestId": "6d79dbfb-0e37-4fc4-981f-442c9ca65760",
"requestId": "831e1650-001e-001b-66ab-eeb76e000000",
"eTag": "0x8D4BCC2E4835CD0",
"contentType": "application/octet-stream",
"contentLength": 524288,
"blobType": "BlockBlob",
"url": "https://oc2d2817345i60006.blob.core.windows.net/oc2d2817345i200097container/oc2d2817345i20002296blob",
"sequencer": "00000000000004420000000000028963",
"storageDiagnostics": {
"batchId": "b68529f3-68cd-4744-baa4-3c0498ec19f0"
}
}
}
}
I think it would be good to show some sample json serialization for each context attribute so there is no ambiguity around what it would look like. It would also be good to show what a complete "context" object would look like. These actually might not be "samples" - by that I mean it might be good to normatively say something like "When serialized in JSON, the format of the event MUST adhere to the following...". We'll see...
https://github.com/cloudevents/spec/blob/master/spec.md#context
Suggestion for clarification:
A set of metadata attributes included with the event that describe circumstances of the occurrence that are not specific to the occurrence. Tools and application code use context information to identify the relationship of events to aspects of the system or to other events. Context attributes may describe the event source, the originating system, the emitting process, and other circumstantial information.
Assigned to: @clemensv
Decide and document if the event-type-version
attribute is applicable to the whole event or to the data payload alone. Currently, event-type-version
covers the data, while open-events-version
covers the context.
Assigned to: @duglin
Roadmap says: Agree on design goals for CloudEvents V.1. for January
I don't think we've actually accomplished this.
@ac360 got together last week and brainstormed what might be an interim goal that would be useful in aligning a core subset of meta data:
Assigned to: @ultrasaurus
https://github.com/cloudevents/spec/blob/master/spec.md#occurrence
The definition of "occurrence" is unnecessarily narrowing the scope of what an occurrence may be, especially by calling out specific technologies/protocols.
Suggestion:
An "occurrence" is the capture of a statement of fact during the operation of a software system. This may occur because of a signal raised by the system or a signal being observed by the system, because of a state change, because of a timer elapsing, or any other noteworthy activity.
Assigned to: @clemensv
This repo's name has changed and its contents have been modified to use CloudEvents, but the description of the repository seems to still say "OpenEventing." I found this just a tiny bit confusing and would've put up a PR for it, but I think this needs to be changed by a project admin in the Github interface.
I'm attaching a screenshot for future reference:
Decide and document if the event-type
attribute can include a namespace as a prefix.
"This describes the software instance that emits the event at runtime (i.e. the producer). It contains sub-properties (listed below)"
The description using a "i.e." seems indicative of either the property name being too ambiguous to stand, or the description needing work.
"Software instance" is also often an incorrect term to describe a resource/source/producer, because a resource/source/producer is often a logical concept that may run multiple instances and even be made up of multiple separate implementations that all yield the same events from the same virtual context.
Assigned to: @clemensv
Before we ship a formal release I think we should remove the "References" section that shows what existing providers do for their events. This is very useful today as we work through the initial design, but once we get agreement on our stuff we no longer need to see what other people are doing since it doesn't align with what we're recommending.
Assigned to: @notque
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.