Should we add an 'id' to the thing? about api

[benfrancis]

I've been trying to avoid this because I think the ID of a thing on the web should be its URI, which should be globally unique.

Whilst in an implementation of the gateway integration pattern an ID is proving useful in identifying Things internally, I'm hoping to keep it out of the Thing Description itself. What would be the point of an ID property when using the direct integration pattern? An id like the one we're using on the gateway is only useful in the context of the gateway itself, at the web level it makes more sense to identify Things by their URI.

I realise that we're currently exposing this ID in Thing descriptions in the API, but this is accidental on my part. I don't think we're actually using this ID anywhere in the front end, other than as part of the Thing URL. Let's see if we can keep it that way and remove it!

[benfrancis]

The latest Editor's Draft at the W3C includes a mandatory id member for a Thing Description. This ID is a "Unique identifier of the Thing (e.g., URI, custom URN, etc.)."

We could use the URL of the Thing Description as the thing's ID and I don't see any harm in including that in the Thing Description (though I'm not sure it should be mandatory). In fact that might be useful for the WebSocket API in differentiating between messages for multiple things on a single WebSocket connection.

[mrstegeman]

I think we should prioritize this. Things need to have a unique, persistent ID so that we can keep track of them across IP addresses, etc. Right now, the only thing we have to identify things by is their IP address/hostname and port. This is not good enough, especially if a web thing's IP changes.

Can we try to formalize this into our spec in the near future, so that we can begin relying on the presence of the id member?

[dhylands]

I agree. For zigbee and zwave the devices already have globally unique IDs. Using IP address is unreliable as the IP address can change over time. Adapters shoule probably combine the MAC address and some other unique identifier to come up with a unique ID. Add-ons should probably use UUIDs.

[benfrancis]

@mrstegeman wrote:

I think we should prioritize this. Things need to have a unique, persistent ID so that we can keep track of them across IP addresses, etc.

Can you explain the problem you're trying to solve in a bit more depth? Is it a problem of identifying the device, or locating it? (i.e. URI vs. URL).

The reason I ask is because the web associates a lot of trust with the origin of a web resource. If we allow the gateway to assume that messages coming from two separate origins which are self-advertising the same ID are in fact the same device, then we may break some security assumptions of the web. "Cool URIs don't change".

So far the gateway is responsible for assigning a unique ID to a device, which is a URI created from a unique string of some kind (chosen by the adapter) appended to a base URI. (e.g. https://smith.mozilla-iot.org/things/zb-0015bc001a006386).

The W3C spec does have an id member for a Thing Description, but that member is intended to be set to a URI. In our gateway implementation I would expect that to be the same as the external URL of the Thing Description, so I'm not sure that would actually help with your problem?

I'm guessing the use case you're talking about might be the ONVIF adapter storing the local IP address of a camera, and then that IP address changing when the DHCP lease runs out? That may be a separate issue.

How does this usually work with ONVIF? If the IP address of the camera changes, does the camera have some other way of identifying itself to a client in order to update the client with its new IP address? How does a device advertise itself on the network? Is there any kind of authentication mechanism in place to ensure the client is in fact still talking to the same device?

[mrstegeman]

The reason I ask is because the web associates a lot of trust with the origin of a web resource. If we allow the gateway to assume that messages coming from two separate origins which are self-advertising the same ID are in fact the same device, then we may break some security assumptions of the web. "Cool URIs don't change".

That works on the internet, but not really on a local network. On a local network, any device can broadcast any mDNS info it wants, versus the internet, where DNS names can be verified. We also can't rely on IP addresses not changing on a local network because of DHCP.

So far the gateway is responsible for assigning a unique ID to a device, which is a URI created from a unique string of some kind (chosen by the adapter) appended to a base URI. (e.g. https://smith.mozilla-iot.org/things/zb-0015bc001a006386).

The reason we're able to do this is that virtually all other smart home protocols (Zigbee, Z-Wave, HomeKit, and even all of the individual proprietary protocols) include some type of unique identifier that we can then turn into our own unique ID, so we're really only using what already exists on the device.

I'm guessing the use case you're talking about might be the ONVIF adapter storing the local IP address of a camera, and then that IP address changing when the DHCP lease runs out? That may be a separate issue.

No, even ONVIF devices have unique IDs. You also have to authenticate with ONVIF cameras.

Can you explain the problem you're trying to solve in a bit more depth? Is it a problem of identifying the device, or locating it? (i.e. URI vs. URL).

Let me give an example.

Let's say you go to the store and buy two brand new smart plugs which are native web things. They will be flashed with the exact same firmware which just exposes a basic thing description (with no ID). You put them on your network one at a time, adding them to your gateway and naming them as you go (Lamp 1, Lamp 2). Internally, the gateway (well, thing-url-adapter) has no way to tell these plugs apart, other than their IP address, because they're even trying to broadcast the same mDNS name. Now, let's say your router randomly assigns them both new IP addresses. How is thing-url-adapter now supposed to map those back to the original devices it found, since the IP address was the unique ID?

[benfrancis]

Let's say you go to the store and buy two brand new smart plugs which are native web things. They will be flashed with the exact same firmware which just exposes a basic thing description (with no ID). You put them on your network one at a time, adding them to your gateway and naming them as you go (Lamp 1, Lamp 2). Internally, the gateway (well, thing-url-adapter) has no way to tell these plugs apart, other than their IP address, because they're even trying to broadcast the same mDNS name. Now, let's say your router randomly assigns them both new IP addresses. How is thing-url-adapter now supposed to map those back to the original devices it found, since the IP address was the unique ID?

Thanks for explaining, I understand now. So this is really more about identifying the device on the local network, as opposed to how it is exposed by the gateway to the Web of Things on the Internet.

So how do you think it should work? Each native web thing should expose a URN as an id member of its Thing Description, separate from its local Thing Description URL and the URL assigned to it by the gateway?

This raises various questions in my head about the difference between web things on a local network and web things exposed to the Internet. There are maybe some assumptions built into the Thing Description spec which don't really work well on a local network, particularly if we implement the security member from the W3C spec. We still don't have a good answer to security for local web things, which may be tied up with this issue of identity.

[mrstegeman]

I don't see why there needs to be a difference between how a thing is exposed locally vs on the internet (or through the gateway). The gateway could in turn use the ID from the thing description as the ID it uses for the thing. There's really nothing special about the ID, it's just a random string/number/whatever. But yes, it should be part of the thing description.

How this works with most protocols is that the device is assigned a random ID at the time of manufacturing. In HomeKit, specifically, when the device is reset, a new random ID is generated, which I kind of like. That way, if someone is trying to spoof the ID for whatever reason, you can just generate a new one.

[benfrancis]

Picking this up again, because not having an id member is one thing which prevents our Thing Descriptions validating with the W3C Thing Description validator.

In principle I don't see anything particularly wrong with including an identifier like a MAC address as general informational metadata in a Thing Description, but I do worry a bit about using that as a canonical identifier for a device, separate from its URI. Let me explain...

A web thing is essentially a "digital twin" or "shadow" of a physical device and in reality it's that twin which is the web thing (a resource on the web), not the device itself.

With the same-origin security model of the web, if a user agent encounters the same resource at two different URIs (e.g. http://foo.com/cat.jpg and http://bar.com/cat.jpg), as far as it is concerned, they are two different resources, even if the content of those two resources is identical.

Similarly, if the same physical IoT device is represented at both https://foo.mozilla-iot.org/things/thing1 and http://gateway.local/things/thing1 then they are really two separate web things, even if they represent the same underlying physical device.

Therefore if a device which was previously represented at http://192.168.1.200 is now represented at http://192.168.1.300 (e.g. because its dynamic IP address changed) then from a web security model point of view, they should be treated as two separate web things.

It doesn't really seem safe to assume that http://192.168.1.200 is the same device as http://192.168.1.300, even if they both report the same unique ID in their thing description. http://192.168.1.300 could be trying to spoof http://192.168.1.200.

This is one reason why at the web layer I think the web thing's URL should be used as its canonical identifier, not an ID inside its thing description. (We've had similar debates at the W3C in the past about identifiers for web applications.)

I realise this causes some practical problems for devices with a dynamic local IP address (another area in addition to HTTPS where World Wide Web assumptions don't apply on local networks), but I don't think that relying on a self-reported ID as a canonical identifier is necessarily the best solution to that problem. It probably requires a protocol-specific mechanism for verifying identity (e.g. some kind of key exchange) to be handled by a gateway adapter, which is responsible for mapping unique IDs (used in the web thing URI) onto physical devices.

My conclusion is that we could include an id in the thing description as a convenience, in addition to being part of the Thing resource's URI (e.g. a Zigbee node ID or Wi-Fi MAC address prepended by a string unique to each adapter, so that each ID is unique to a given gateway), but that ID shouldn't be used as a canonical ID by our gateway implementation, at least not at the web layer where the web thing's full URI should be its canonical identifier.

What does everyone else think?

(Note that the W3C spec says that an ID must be a URI, so if it's something like a MAC address then that needs to somehow be represented in URI form. The obvious way to do that in our gateway implementation seems to be to simply use the URL of the Thing Description, which includes that ID. Which is why I kind of see it as redundant if the web client already knows the web thing URL. But I can see that it might be convenient to have it in the Thing Description.)

[dhylands]

I've already run into issues with DHCP when using my sonos speakers. Everytime my router restarts, the speakers come up with a different IP address, which meant that I had to remove all of my speakers and re-add them, and then also remove and re add all of the rules used to control them. To me this is totally unacceptable.

The speakers already have unique identifiers, and the speaker adapter knows what those unique identifiers are, so it seems totally reasonable to me that those unique identifiers be used and the IP address just considered as a way of getting to the device. This is what happens with named things anyways. It's just that there is an external service called DNS which does the mapping to come up with the IP address.

I think that using an IP address as a unique identifier is just plain wrong.

[dhylands]

As far as trying to spoof things go, just because it has the same IP address doesn't mean it's the same device. So let's say you have a device A at 192.168.1.200 device B at 192.168.1.201 and after the router reboots device A now comes up at 192.168.1.202 and device B comes up at 192.168.1.200. You're now talking to device B but thinking that you're talking to device A. How is that more secure than realizing that the device is incorrect based on some other unique ID which would be more correct.

I fear that you're throwing the baby out with the bath water by essentially making a normal system unusable just to cover the case where some device might try to spoof another.

[hobinjk]

I think I'm undercomplicating this but I believe that the best use for id is to mirror the device id (e.g. virtual-things-3) and potentially add some requirements/recommendations of best practices like uniqueness and prefixing with the adapter id

[benfrancis]

@dhylands wrote:

Everytime my router restarts, the speakers come up with a different IP address, which meant that I had to remove all of my speakers and re-add them, and then also remove and re add all of the rules used to control them. To me this is totally unacceptable.

I agree. I think we need to think separately about the adapter (network) layer and the web (application) layer.

At the adapter layer there needs to be some protocol-specific mechanism that does the best job it can of uniquely identifying a physical device (e.g. a MAC address) on a network, and ideally in future verifying that it's still talking to the same device. As I understand it Zigbee and Z-Wave (and probably HomeKit) already have mechanisms for this. Other IP-based protocols may have other mechanisms.

At the web layer the ID of a web thing is a URI (for which the URL of the thing description itself is an obvious choice).

The gateway, between the adapter and main process, is responsible for mapping the web thing's identity (URI) to the physical device's identity, and maintaining that relationship over time.

I think that using an IP address as a unique identifier is just plain wrong.

I agree with this too, and that's not what I'm suggesting. I'm saying that if an IP address forms part of a web thing's URI and that IP address changes, then it has to be considered to be a different web thing. That's how the origin-based security model of the web works. The same applies to websites. If a website that was previously hosted at 192.168.1.200 suddenly changes to 192.168.1.300 then a user agent will consider it to be a different website, and no permissions granted to the former will be valid for the latter.

This is usually not a problem on the World Wide Web using a fully qualified domain name, because that is expected to stay the same. It does cause problems on a local network if a dynamic IP address forms part of the URI.

As far as trying to spoof things go, just because it has the same IP address doesn't mean it's the same device.

That's exactly my point. There needs to be some other way of uniquely identifying the physical device on a network, at the adapter layer. The gateway needs to map a web thing URL onto that identity and keep it updated.

At the web layer there needs to be a URI that stays the same. And if that is the case, then including the ID in the Thing Description as well as the URI is arguably redundant (though it might be convenient).

@hobinjk wrote:

I think I'm undercomplicating this but I believe that the best use for id is to mirror the device id (e.g. virtual-things-3) and potentially add some requirements/recommendations of best practices like uniqueness and prefixing with the adapter id

I agree with this. But if the W3C specification says that the ID In the Thing Description must be a URI (which seems reasonable as that's how you generally identify things on the web), then in our gateway implementation we may as well just use the URL the gateway gives to the Thing resource for this purpose, which already includes that ID (e.g. https://foo.mozilla-iot.org/things/virtual-things-3).

It seems like the only real issue here is with local web things which use a dynamic IP address as part of their URL. Maybe the thing URL adapter needs to be using some other mechanism to uniquely identify that device on the network and map it onto the stable external URL the gateway gives that device.

Given we have other issues with local web things too, like with HTTPS and authentication/authorisation, maybe it's reasonable to say that a web thing isn't truly a web thing unless it has a URL on the World Wide Web. The same issues apply to a website which is only hosted at a local IP address on an Intranet, is that really a website?

Anyway, in conclusion, I think we can put an id member in the Thing Description, but if that's going to be a URI then in our gateway implementation that ID may as well be the URL of the Thing resource.

[dhylands]

Just to clarify, if I have a device at IP address 192.168.1.100 and it has an ID of foobar-123 then I don't have a problem with the thing URL being either https://foo.mozilla-iot.org/things/foobar-123 or https://localhost:4443/things/foobar-123.

The thing that I have a problem with is if the URL looks like https://foo.mozilla-iot.org/things/192.168.1.10.

[benfrancis]

The thing that I have a problem with is if the URL looks like https://foo.mozilla-iot.org/things/192.168.1.10

I agree that doesn't make sense. Are there currently circumstances under which we do that in our gateway implementation? I think maybe the thing-url adapter does something like this, but using the whole local URL as an ID?

The thing-url adapter is a bit of a strange example because we're adapting a web thing into a web thing so it's really a proxy rather than an adapter. I'm not really sure what a better solution is there.

Are there any other adapters that do this? And if so, is there an alternative identifier they could use instead of IP address?

[dhylands]

The thing-url-adapter should get some unique ID from the thing and use that. The thing-url-adapter should then probably keep some type of cache mapping thing-id to IP address, updating the IP address when it discovers that the thing has a new IP address.

At least then there is a chance to change the IP address (perhaps even manually) but this won't break any of the rules or other information that the gateway has about the thing.