ssilverman / rdm-schema

The schema for the Parameter Metadata Language from Section 5 of E1.37-5.

README for the RDM Schema Project

Version: 1.0.0

This project contains the schema for the Parameter Metadata Language from Section 5 of E1.37-5.

The schema is subject to change.

Table of contents

1. Project intent
2. Design notes
   1. Relation to other RDM types and structure
   2. Exceptions
   3. Strings and string lengths
   4. Names and displayable strings
3. Usage notes
   1. Framing is at a different layer
   2. Arbitrary field sizes
   3. Constraints and errors
   4. Defaults and the "default" annotation
4. Best practices
5. Notes on the examples
   1. Manufacturer ID
6. Open questions
7. TODOs
8. Resources
   1. References mentioned in the schema

Project intent

This project intends to provide a machine-readable way to describe manufacturer-specific RDM messages. Controllers can then process, and perhaps provide a meaningful UI for, these messages.

Design notes

It was chosen to use the latest version of the schema specification, Draft 2019-09 to take advantage of the latest features. It is intended that the schema will be updated as later drafts and releases come out, as appropriate.

To develop the schema, all known RDM messages were implemented as schema instances so that every case in the specifications was accounted for. There were two exceptions, however, QUEUED_MESSAGE and CAPTURE_PRESET.

Relation to other RDM types and structure

The "DS_*" types in ANSI E1.20-202x, plus the types from related specifications, are not sufficient to describe a proper type system, so there is no direct mapping from ANSI E1.20-202x to this schema.

Whilst this schema can be used to represent everything in the specifications, the facilities in the specifications can't be used to represent everything RDM messages might wish to represent, manufacturer-specific or otherwise.

Exceptions

QUEUED_MESSAGE is the only message that allows responses having a different PID. First experiments used a response type of "different_pid". CAPTURE_PRESET is the only message with optional fields; the last few can be present as a group or not.

It was decided that these features would not be included because they're so rare and they're not desirable features for representing manufacturer messages, the original intent for this project.

Strings and string lengths

There are two facilities this schema provides for "string length". The first is "length in characters", defined by "minLength" and "maxLength". A "character" is defined the same way that JSON defines a character: a single code point, possibly composed using a UTF-16 surrogate pair.

The second facility is "byte length", expressed in "minBytes" and "maxBytes". Strings will use the UTF-8 encoding and the length in bytes gives bounds on the storage requirements.

From Validation Keywords for Strings:

The length of a string instance is defined as the number of its characters as defined by RFC 8259.

Specifically, see Section 7: Strings and Section 8: String and Character Issues.

In other words, "length" in this schema means the same things as "string length" per the JSON specification.

There are further considerations when using a string for display, say in a UI. However, this specification does not address those things. Those considerations may include, but are not limited to, normalization, canonicalization, glyph size, and character-to-display approaches. It is up to the manufacturer to decide what to use for string content.

See the discussion at What's the difference between a character, a code point, a glyph and a grapheme?.

Relevant terms: UCS-4, UTF-8, Unicode, Basic Multilingual Plane, Unicode Plane.

Names and displayable strings

For nameable things, "name" is intended to be a unique ID and "displayName" is intended to be a name for display. Note that both are optional.

In the case that a manufacturer wishes to provide localized names, this design takes a cue from how Java does localized strings. The name would be used as a lookup into some manufacturer-supplied table for the actual display string, and the "displayName" value could be used as a fallback or as the actual displayable name in the case where a manufacturer does not provide that out-of-band table.

Usage notes

Framing is at a different layer

The schema can describe complete messages, but does not describe message framing. For example, if a message is larger than can fit inside a single RDM packet, then the underlying implementation would use RESPONSE_TYPE_ACK_OVERFLOW appropriately.

Arbitrary field sizes

There are several field types that can be non-fixed sizes. For example, "a string having a length in the range 0-32" or "a list of arbitrary size". So that there is no ambiguity, a message should contain at most one non-fixed-size field, and that field should appear last, serially.

If a responder wishes for controllers to limit the number of bytes sent, then it should set appropriate values for the "maxLength" field for strings and the "maxLength" field for bytes.

Constraints and errors

Note that the schema does not capture every possible error. Some errors can only be caught after processing a schema instance. Please see the Best practices section for more details about avoiding these kinds of errors.

Defaults and the "default" annotation

In JSON Schema, the "default" keyword is merely an annotation that applies to the current schema location, if present; it does not describe the value to use when the property is absent. This is counterintuitive insofar as a "default" annotation does not provide a default value in the case of a missing property.

An implementation is expected to follow the usage notes in the description if a default value is needed. It can use the value of the "default" annotation, but this is not a JSON Schema feature.

In other words, a JSON parser/validator will not return values for absent properties; it is up to the application to supply values.

Best practices

It's certainly possible to create badly defined messages, even though they conform to the schema. These messages may just be ill-defined or may not be compatible with the responder serving these messages. This section describes some restrictions that, if followed, will prevent many of these kinds of problems.

1. If a responder wishes for controllers to limit the number of bytes sent for strings or bytes, then it should set appropriate values for the "maxBytes" field for strings and the "maxLength" field for bytes. It's conceivable that a responder doesn't need this, but many responders do because they're implemented on smaller systems that may need to preallocate memory.
2. Minimums should be less than maximums. For example, the "bytes" type has "minLength" and "maxLength" fields; "minLength" should be less than or equal to "maxLength".
3. A bit field size should be greater than or equal to the number of its defined bits.
4. A command should not refer to itself. For example, a "get_response" cannot have a value of "get_response". Please refer to the "Command Duplicate" subschema under the "command" schema.
5. References ("$ref") should refer to an object having a valid type. Also, there should not be any circular references.
6. String patterns should not contradict any minimum or maximum lengths, and vice versa.
7. The "format" value for bytes or strings, if a fixed-size type, should not contradict any minimum or maximum lengths.

Notes on the examples

Manufacturer ID

All the example messages should use a manufacturer ID of zero or 0xFFFF because those are ESTA's. However, those will not validate against the schema if the value is restricted to the range 0x0001-0x7FFFF. There was some discussion on this:

1. It is stated in several places that manufacturer IDs must be in the range 0x0001-0x7FFF, so we should restrict the range. See:
   1. ANSI E1.20
   2. ANSI E1.33
   3. Control Protocols Working Group - Manufacturer IDs
2. There was some concern that developers will copy & paste the examples and not choose their own manufacturer ID, so keeping the examples from validating will prevent this.

It is the opinion of the author of this document that it is not reasonable to restrict the schema just to prevent possible misuse. In addition, it's defined as a 16-bit value. Using the schema to restrict the value will cause future changes and uses to be invalid. It may be beyond the scope of the schema to accomplish any restriction.

If the manufacturer ID is required, if it is restricted to 0x0001-0x7FFF, and if the examples should validate, then there are two possible ways to express ESTA examples. Either:

1. Remove "manufacturer_id" from the examples and remove the field from the list of required fields, or
2. Use a valid manufacturer ID in the examples, such as 0x7FFF from the prototyping/experimental use region.

In other words, we can't simultaneously have all the following things:

1. Examples having manufacturer ID zero.
2. Manufacturer ID's limited to the range 0x0001-0x7FFF.
3. Examples that validate.
4. "manufacturer_id" being a required property.

For now, the value in all the examples is set to 0 (0x7FFF).

Open questions

Some open questions:

1. Versioning. Perhaps we change the schema's $id each time there's an update? We could include the version in the URI. Some possibilities:
   1. https://estalink.us/schemas/v1.0.1/rdm-schema.json
   2. https://estalink.us/schemas/rdm-schema-v1.0.1.json
2. How to have examples with the ESTA manufacturer ID (zero), while at the same time having them validate and requiring the manufacturer ID be a required property. The temporary solution is the change all the example manufacturer ID's to something in the prototyping/experimental use region (0x7FF0-0x7FFF).

TODOs

Work that still needs to be done:

1. Supply valid PIDs to some of the examples. Currently, they are using an invalid value of -1 because the PIDs are unassigned.

Resources

• TSP Published Documents
  • The latest ANSI E1.20 specification can be found here.
• TSP Public Review Documents
  • ANSI E1.20-202x can be found here when it's released for public review.
• JSON Schema
• RDM schema canonical URI (not necessarily a network locator): http://estalink.us/rdm-schema.json
• RDM schema main page (expected to be here, but not here yet): http://estalink.us/rdm-schema
• Online JSON schema validator: Hyperjump - JSON Schema Validator
• JSON
• JSON Schema Validation: Validation Keywords for Strings
• UTF-8
• The discussion at What's the difference between a character, a code point, a glyph and a grapheme?

References mentioned in the schema

• URI Syntax
• JSON Pointer
• JSON
  • Section 7: Strings
  • Section 8: String and Character Issues
• JSON Schema Validation: Defined Formats
• Format "hostname":
  • Requirements for Internet Hosts: Host Names and Numbers
    (You may need to scroll to Section 2.1.)
  • Application Techniques for Checking and Transformation of Names
  • Internationalized Domain Names for Applications (IDNA): Definitions and Document Framework
• Format "json" and string characters: JSON
• Format "url":
  • URI
    See also (from Section 1.1.3): URI, URL, URN Clarifications
  • URL
• ECMA-262 (Regular Expressions)

rdm-schema's Issues

Uniqueness

Schemas at the moment are inferred to be the same/valid for all devices by the absence of a device_model_id, software_version_id and root boolean value.

If that is the case, these should be made optional in the schema and stated in the standard as to their meaning when absent.

Provide Context

Schemas currently provide a good definition of the payload to and from a responder but little context about a parameter. In their current state a controller can only assume certain characteristics of a parameter when provided a schema or if forced not to assume, ultimately ignore the parameter altogether as it has no clue as to what to do with it.

I have found through developing a RDM Controller using the schemas as a generator for both the cached device parameter data (the responses from get requests) and the get requests themselves that there are a number of characteristics I have needed to attach to a parameter definition alongside the schema to allow the RDM controller to be both efficient with its generated get requests and automated in its process to gather parameter data.

---

Shareable

It is beneficial for a controller to know when a parameters get response payload can be shared across multiple devices. The scenario is that a controller may discover multiple devices with the same esta_manufacturer_id, device_model_id, software_version_id and whether it is the root device and if so filter the get requests it would make to each of those devices according to the current state of cached parameter values it has. For example, SOFTWARE_VERSION_LABEL is a shareable parameter, once you have the response from one device, any other devices with same esta_manufacturer_id, device_model_id, software_version_id and whether it is the root device can just use that cached value, the controller does not need to send a get request for that parameter to each of those devices.

Array

I call these type of parameters “array” parameters mainly because the responses are cached in a JSON array, but ultimately they are parameters that provide a collection of values, typicaly to be used in conjunction with another parameter. For example DMX_PERSONALITY_DESCRIPTION takes a uint8 personality index and returns the description for the personality at the given index. The idea being that a controller finds out the number of personalities from DEVICE_INFO and then generates the required get requests for each personality. To automate the process of generating the get requests after understanding a device supports this kind of parameter I have needed to first indicate that this is an “array” parameter but also provide another json schema to tell the controller what to do with it. Here is an example:

{
  "name": "DMX_PERSONALITY_DESCRIPTION",
  "pid": 225,
  "get_informed_by_parameter": {
    "name": "DEVICE_INFO",
    "pid": 96,
    "$ref": "$[7].value",
    "type": "uint8",
    "zero_based": false,
    "function": "enumerate"
  }
}

This schema indicates that for a controller to gather all possible values from DMX_PERSONALITY_DESCRIPTION, it needs to look at the responses payload from DEVICE_INFO, taking the value at index 7, and then enumerate from either 0 or 1 up to or including the value depending on zero_based, generating a get request with the index as a uint8. SENSOR_DEFINITION and SENSOR_VALUE use the value at index 10 from DEVICE_INFO, which is zero based… Without this kind of definition of what a controller should do with a parameter like this I’m uncertain how it should behave when it comes across one. The default would be to ignore it altogether, but the problem being that these types of parameters offer the visual text often presented to a user.

Action

If an application provides a representation of a device with all of its RDM parameters in an offline capacity, a boolean value indicating whether the parameter is an “action” parameter is necessary to provide a nice UX. An “action” parameters set request triggers an action on the device, CAPTURE_PRESET is a good example as it has no get response, so no values to be cached by the application, and the whole purpose of it is to trigger an action on the device when its online.

UI Representable

This one is quite self explanatory. There are a number of parameters that should never be shown to a “typical” user using an RDM controller. QUEUED_MESSAGES being a prime example. The idea here is that a manufacturer should be able to specify that they have a parameter that typically is helpful in gathering information for other functionality, but if the controller is showing a device with all of its supported parameters, this shouldn’t be shown visually.

UI Informs and Informed By

This is another front end thing, but there are certain parameters in the standard that inform the UI of another parameter. LANGUAGE_CAPABILITIES is a good example of this. When an application wants to show the available languages a device can be set to it would use this parameter to show them, but when actually doing the setting it uses LANGUAGE. It would be helpful if the schema could tell me this.

---

Sorry for the wall of text, there may be more but thats enough for now. The point I would like to get across though is that: - With the schema not specifying these points, a controller has to make an assumption about a parameter. Some of those assumptions are easy to make and whilst not as performant, functionally the controller would still work by assuming the worst, shareable for instance would default to false and the controller would send a good few messages more, but thats it. action would default to false and the application would show the parameter even when the device is offline, no big deal. The lack of array though can render a parameter useless.

As an example of perhaps how we could work this into the schema:

{
  "name": "DMX_PERSONALITY_DESCRIPTION",
  "pid": 225,
  "version": 1,
  "get_request_subdevice_range": [
    "root",
    "subdevices"
  ],
  "get_request": [
    {
      "name": "personality",
      "type": "uint8",
      "ranges": [
        {
          "minimum": 1,
          "maximum": 255
        }
      ]
    }
  ],
  "get_response": [
    {
      "name": "personality",
      "type": "uint8"
    },
    {
      "name": "dmx_slots_required",
      "type": "uint16",
      "ranges": [
        {
          "minimum": 0,
          "maximum": 512
        }
      ]
    },
    {
      "name": "description",
      "type": "string",
      "maxLength": 32,
      "restrictToASCII": true
    }
  ],
  "extensions": {
    "esta_backend": {
      "shareable": true,
      "array": {
        "name": "DMX_PERSONALITY_DESCRIPTION",
        "pid": 225,
        "get_informed_by_parameter": {
          "name": "DEVICE_INFO",
          "pid": 96,
          "$ref": "$[7].value",
          "type": "uint8",
          "zero_based": false,
          "function": "enumerate"
        }
      },
      "esta_frontend": {
        "action": false,
        "uiRepresentable": true,
        "uiInforms": {
          "name": "DMX_PERSONALITY",
          "pid": 224
        }
      }
    }
  },
  "extensions_used": [
    "esta_backend",
    "esta_frontend"
  ],
  "extensions_required": [
    "esta_backend",
    "esta_frontend"
  ]
}

Extensions

Referencing #14 and @peternewman comment I'm pulling out the general concept of extensions, allowing a schema to contain meta data alongside a parameters data definition.

A few topics up for discussion.

• The format of extensions within the .json file.
• Are there some extensions that are required and considered set to default values if missing?
• Manufacturer specific extensions.

response_time_setting_count in OUTPUT_RESPONSE_TIME_DESCRIPTION and MODULATION_FREQUENCY_DESCRIPTION

Hi,

In the json description of OUTPUT_RESPONSE_TIME_DESCRIPTION and MODULATION_FREQUENCY_DESCRIPTION there is a GET parameter: response_time_setting_count

Is this a (copy/paste) error, as ANSI E1.37-1 – 2012 (R2017) does not mention an additional GET parameter, aside from the requested setting's number?

Name Values

What is the logic behind the capitalisation of name values? It seems to be a mix between snake case and screaming snake case but I'm not sure why one value is one way and another is the other style.

restrictToLabeled = true for IDENTIFY_DEVICE

Shouldn't restrictToLabeled = true be added for the IDENTIFY_DEVICE PID?

As also interpreted by the ola tests: https://github.com/OpenLightingProject/ola/blob/master/tools/rdm/TestDefinitions.py#L4050 , any other value than 0 or 1 is not accepted.

Same goes for PAN_INVERT, TILT_INVERT, PAN_TILT_SWAP,

Although not listed explicitly mentioned in the spec, it seems for all these restrictToLabeled apply?

Incorrect Capture Preset Set Request

Set request contains "units" and "prefixPower". It shouldn't

DNS_IPV4_NAMESERVER vs DNS_IPV4_NAME_SERVER

In ANSI E1.37-2 – 2015 DNS_IPV4_NAME_SERVER is used as name for getting a name server. In the json "name server" is written as one word, DNS_IPV4_NAMESERVER. It would be logical to follow the naming in the spec?

DISPLAY_LEVEL's restrictToLabeled should be false?

https://github.com/ssilverman/rdm-schema/blob/master/examples/e1.20/DISPLAY_LEVEL.json

The spec states:
To turn the display off, Display Level shall be set to 0. To turn the display on full, Display Level shall be set to 0xFF. Any value in between represents a relative intensity setting. If the device does not support relative intensity settings, any non-zero value shall be interpreted as on.

So, it should not be (by default) restricted to the current: ?
{ name: "Off", value: 0 }, { name: "Full", value: 255 },

Manufacturer specific E1.20 schemas

There is a plausible scenario where a manufacturer may want to write a manufacturer specific version of an E1.20 parameter, for example:

{
  "scope": { 
    "esta_manufacturer_id": 12345,
    "device_model_id": 1337,
    "software_version_id": 12345678,
    "root": true
  }
  "name": "DMX_PERSONALITY",
  "pid": 224,
  "version": 1,
  "get_request_subdevice_range": [ "root", "subdevices" ],
  "get_request": [],
  "get_response": [
    {
      "name": "personality",
      "type": "uint8",
      "ranges": [
        { "minimum": 1, "maximum": 3 }
      ],
      "labels": [
        { "name": "1_ch", "value": 1 },
        { "name": "direct", "value": 2 },
        { "name": "direct_plus_strobe", "value": 3 }
      ]
    },
    { "name": "personality_count", "type": "uint8" }
  ],
  "set_request_subdevice_range": [ "root", "subdevices", "broadcast" ],
  "set_request": [
    { "$ref": "#/get_response/0" }
  ],
  "set_response": []
}

I've used the scope property referred to in #13, but do we think this should be a thing?

Extension: UI Informs/Informed By

Referencing #14 I'm pulling out the "Informs/Informed By" extension into its own issue for discussion.
There are currently two areas i can think of where it is useful to know if and how one parameter is informing another. In this issue we are looking at how the UI in an application is informed by another. For example, When an application wants to show the available languages a device can be set to it would use this parameter to show LANGUAGE_CAPABILITIES, but when actually setting it uses LANGUAGE.

An example:

"name": "LANGUAGE_CAPABILITIES",
"pid": 160,
...
"extensions": {
  "ui-informing": {
    "name" : "LANGUAGE",
    "pid" : 176
  }
}

"name" : "LANGUAGE",
"pid" : 176,
...
"extensions": {
  "ui-informed": {
    "name" : "LANGUAGE_CAPABILITIES",
    "pid" : 160
  }
}

Both extensions if null in the json would default to "This parameter is not informed by another, or does not inform another."

Versioning

Versioning. Perhaps we change the schema's $id each time there's an update? We could include the version in the URI. Some possibilities:
https://estalink.us/schemas/v1.0.1/rdm-schema.json
https://estalink.us/schemas/rdm-schema-v1.0.1.json

I'd favour the former example, you can then also have a symlink https://estalink.us/schemas/latest/rdm-schema.json which always points to the most recent version and the filename itself is always the same when downloaded, which is probably easier for software to manage (assuming the version is represented as a property within the JSON file too. I see it's currently in the id, it might be nice to have it duplicated as a clear semver field too, which doesn't require any fiddly parsing of the URL, although duplicating a value isn't great either.

Manufacturer ID zero

So currently, none of the examples will validate. Here's why (excerpt from the new "Notes on the examples" section in the README):

---

Manufacturer ID zero

All the example messages use a manufacturer ID of zero, even though that will
not validate against the schema. There was some discussion on this and these are
the reasons:

1. It is stated in several places that manufacturer IDs must be >= 1. See:
   1. ANSI E1.20
   2. Control Protocols Working Group - Manufacturer IDs
2. It will ensure that even if someone uses the example messages as a base for
   their own messages, say using cut & paste, they will still need to choose
   their own manufacturer ID.
3. Zero is ESTA's manufacturer ID.

---

Love your input if you would like to share here. It would be relayed to the group next time the group meets.

Originally posted by @ssilverman in #9 (comment)

Inconsistent get responses

The get response of ENDPOINT_TO_UNIVERSE uses a reference to the first field of the get request:

  "get_response": [
    { "$ref": "#/get_request/0" },
   ...

The get response of DMX_PERSONALITY_DESCRIPTION repeats the field:

  "get_response": [
    { "name": "personality", "type": "uint8" },
   ...

Whilst both are valid. What is the preferred approach?

Consistent Field Names

Fields duplicated across parameters should use the same "name" value.
For example DEVICE_INFO contains the personality index and dmx start address, but uses different names for those fields rather than what is used in DMX_PERSONALITY and DMX_START_ADDRESS.