eclipse-uprotocol / up-rust Goto Github PK

Currently UListener::on_error(UStatus) only provides error details in the UStatus struct but does not provide any information regarding the context in which the error occurred.

It seems reasonable to either pass in a full UMessage with the UStatusas payload or instead pass in both UStatus and UAttributes.

Currently, the PublishValidator verifies that a set of attributes for a Publish message contains either a source or a sink attribute.

However, based on eclipse-uprotocol/up-core-api#104 it should instead verify that a Publish message either has a source only (Publish message) or both source and sink (Notification).

A few things we want to do to our CI/CD:

• have build&test runs for amd64 and arm64
• have explicit build&test for the oldest-supported Rust toolset (v1.66 at the moment)

Related todos:

• clearly document backwards compatibility boundary for Rust toolset, and how we chose that
• remove Cargo.lock from .gitignore to facilitate reproducible builds for developers

The values to use in CloudEvent's type property are defined in options of UMessageType and should be used when serializing.

While implementing ULink library, I found some weird behaviors on validating UUID.
After some tracing, I'm not quite sure about the implementation here. It seems like it doesn't correspond to protocol-spec.
https://github.com/eclipse-uprotocol/uprotocol-rust/blob/main/src/proto/uprotocol/uuid.rs#L77

To be more specific, my questions are

1. Why do we need to check the version is Version::Custom here?
2. Is there any expected way to generate UUID that can pass is_uprotocol_uuid?

invoke_method can get the reply UMessage from RpcClientResult (pub type RpcClientResult = Result<UMessage, RpcMapperError>;)

async fn invoke_method(
        &self,
        method: UUri,
        request: UPayload,
        options: CallOptions,
    ) -> RpcClientResult;

I'm wondering how we get the reply when we use up-l1 API send to comprise the invoke_method.

After taking a look, I guess receive should be called after we use send to send out the request.
async fn receive(&self, topic: UUri) -> Result<UMessage, UStatus>
However, is it possible to send multiple requests to the same topic at the same time?
If yes, we'll have trouble to distinguish the mapping of request and reply.

Hi fellow up-rustaceans,
I want to do some fundamental cleanup and reorg of the SDK source file structure. Lining out the planned changes here, for feedback and also so we can maybe agree on a hands-off period for a week or so, where I then do all of this without making merging of parallel work more nightmarish than it needs to be.

The core things I want to do:

• clean out the src/proto folder (this is there for 'historical reasons', but the introduction of up-core-api was the final nail in the coffin for the original idea)
• make the cloudevents bits an optional crate feature

More detailed, I'm planning to do the following moves:

• the top-level uprotocol logic (mostly in /proto and the */datamodel folders atm) into the top-level /src folder
• the /uuid, /uri etc folders will remain, but contain the trait definitions for their associated types (validators, serializers etc)
• /rpc stays as it is
• /types only contains error definitions at this point, these well me merged and moved to top-level

Note: the external interface of the up-rust crate wouldn't change, so for users this should be transparent (with the possible exception of the cloudevents feature becoming optional, which I'd put into it's own PR though)

Any comments? Do any of you have some large changes in the pipeline that this should be time-synced with? Otherwise, I'd wait for the current batch of PRs to get merged, then pull of the above changes, then everyone could get back to work.

Hi all (@stevenhartley et al), I'm essentially done an initial version of the Rust SDK - some process ends being tied up on our end, etc, but getting close enough to ask this question:
How would you like me to make this available? One summary PR that includes the entire thing (matching the Java SDK in functionality)? Or rather a batch of individual PRs, maybe grouped by submodule (uuid, uri, cloudevent, transport etc)?

The following issue shall be to support publishing of release artifacts to crates.io registry so that developers can pull down stable versions.

I am working to clean up and pull together code belonging to UUri, specifically:

• make issues during serialization visible to the user - use TryFrom and Result<> instead of empty default objects and Option<>
• move the serialization/deserialization code directly into the TryFrom implementations, this gets rid of all the *serializer stuff while making the whole thing more rust-like
• general little cleanups and simplifications

This work will base on #18, so depends on that being merged.

The factory methods that build request(), response(), notificaiton(), and publish() should within the belly create the message ID as there is no real benefit to force the developer to call with_message_id().
Take a look at the up-java UAttributeBuilder.java, the factory methods when they instantiate the UAttributeBuilder object, the UUID is automatically created and populated. I don't think we need to add the flexibility to add different types of UUID, we're only supporting the one defined in the spec now.

The Eclipse Foundation's development process requires us to make sure that the license terms of all 3rd party dependencies are compatible with the project's license (ASL2).

The first step in doing so is to actually determine the licenses that the 3rd party dependencies are using.

I got this working locally by:

$ rustup target add aarch64-unknown-linux-gnu
$ sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu

then adding to .cargo/config.toml:

[target.aarch64-unknown-linux-gnu]
linker = "aarch64-linux-gnu-gcc"

then I could

$ cargo build --target=aarch64-unknown-linux-gnu

Could we add this to the CI/CD pipeline?

Any thoughts from folks? Tagging in @AnotherDaniel, @sophokles73

I love the PR #46, which makes the code cleaner.
However, I found that we still need to create the UAttributes by ourselves inside invoke_method while sending the RPC request.

async fn invoke_method(
    &self,
    method: UUri,
    request: UPayload,
    options: CallOptions,
) -> RpcClientResult;

Now I use UMessageBuilder to create the UMessage and extract the UAttributes.
https://github.com/eclipse-uprotocol/up-client-zenoh-rust/pull/5/files#diff-5cdcc3261dd77eadb37bcd60c5f85affcc9306454ae7bb8128e6d096ae51a631R61
Maybe we can still keep UAttributesBuilder here?

@sophokles73 Do you have any suggestions?

We should add the Cargo.lock file to the repository in order to make sure that we have reproducible builds at all times.
Updating any dependencies will still be possible, of course, but will turn into a controlled process instead of letting Cargo figure it out during each fresh build.

Now the short-form UUri is back in eclipse-uprotocol/up-spec#89, should we add the short-form UUri in up-rust?
It might be easier to transform the UUri into MQTT topic or Zenoh key if we have short-form UUri, but still need to sync with other languages to ensure interoperability.

With the changes introduced to UTransport::send, it seems more desirable to be able to easily build a UMessage to be sent instead of creating UAttributes.

uProtocol core library written in Rust

As part of the build pipeline rework ( #51 ), I going to make it so that we can repo-globally pin the version of the Rust toolchain that is used by the ci pipeline for up-rust.

While the actual decision on which specific version to use doesn't have to be made immediately, we can begin to discuss this...

Relevant factors that I'm aware of:

• As I write this, current rustc --version is 1.77.0
• Some of the dependencies of our dependencies require at least 1.74.0
• It should be extremely simple to set up/use latest/current Rust on any platform, via rustup, so no hard legacy-system-boundaries I'd hope?
• Ferrous Systems is planning to support 1.76.0 in the next Ferrous release

This leads me to think that 1.76.0 might be a sweet spot for us to support for now. Depending on what relevante we assign to ASIL certifyability going forward, me might decide to follow Ferrocene releases in the future...

WDYT?

The validate* functions of UriValidator seem to assume that a UUri always contains at least the logical names of authorities, entities, resources.

This means that the following code fails:

// up://10.0.5.101/24576/1/256
let uri = UUri {
    authority: Some(UAuthority {
        number: Some(Number::Ip(vec![10, 0, 5, 101])),
        ..Default::default()
    })
    .into(),
    entity: Some(UEntity {
        id: Some(24576),
        version_major: Some(1),
        ..Default::default()
    })
    .into(),
    resource: Some(UResource {
        id: Some(256),
        ..Default::default()
    })
    .into(),
    ..Default::default()
};
assert!(
    UriValidator::validate(&uri).is_ok(),
    "UUris that contain only identifiers (no logical names) should validate successfully"
);

This is a problem in situations where you want to use UMessageBuilder with a UUri that only contains identifiers, e.g. when interacting with services via SOME/IP. The bulder will not be able to create the message in this case because the build function invokes UriValidator::validate for the message's source/sink.

The same problem occurs with the other validator functions.
IMHO all functions should to be adapted to check on logical names but to also accept identifiers, if no logical name is set.

Hi there 👋

I noticed when serializing the UURI to micro form, we do some bit shifts and casting to get the desired 16 bits written for the UResource and UEntity ids.

I suppose it's possible based on what I see here and here that we could form ids which are larger than the allotted 16 bits when serializing.

Perhaps we should explicitly check to make sure we can fit the ids within the allotted bits, e.g.

const UENTITY_ID_LENGTH: usize = 16;
const UENTITY_ID_VALID_BITMASK: u32 = 0xffff << UENTITY_ID_LENGTH;
const URESOURCE_ID_LENGTH: usize = 16;
const URESOURCE_ID_VALID_BITMASK: u32 = 0xffff << URESOURCE_ID_LENGTH;
// ...
        // URESOURCE_ID
        if let Some(id) = uri.resource.as_ref().and_then(|resource| resource.id) {
            // this is new -- start
            if id & URESOURCE_ID_VALID_BITMASK != 0 {
                return Err(SerializationError::new(format!("UResource id larger than allotted 16 bits: 0x{:x}", id)));
            }
            // this is new -- end
            cursor.write_all(&[(id >> 8) as u8]).unwrap();
            cursor.write_all(&[id as u8]).unwrap();
        }

        // UENTITY_ID
        if let Some(id) = uri.entity.as_ref().and_then(|entity| entity.id) {
            // this is new -- start
            if id & UENTITY_ID_VALID_BITMASK != 0 {
                return Err(SerializationError::new(format!("UEntity id larger than allotted 16 bits: 0x{:x}", id)));
            }
            // this is new -- end
            cursor.write_all(&[(id >> 8) as u8]).unwrap();
            cursor.write_all(&[id as u8]).unwrap();
        }

And these are the accompanying tests which pass with the code changes above:

    #[test]
    fn test_uentity_id_greater_than_16_bits() {
        let uri = UUri {
            entity: Some(UEntity {
                id: Some(1 << UENTITY_ID_LENGTH),
                version_major: Some(254),
                ..Default::default()
            }),
            resource: Some(UResource {
                id: Some(19999),
                ..Default::default()
            }),
            ..Default::default()
        };

        let uprotocol_uri = MicroUriSerializer::serialize(&uri);
        assert!(uprotocol_uri.is_err());
        assert_eq!(
            uprotocol_uri.unwrap_err().to_string(),
            "UEntity id larger than allotted 16 bits: 0x10000"
        );
    }

    #[test]
    fn test_uresource_id_greater_than_16_bits() {
        let uri = UUri {
            entity: Some(UEntity {
                id: Some(29999),
                version_major: Some(254),
                ..Default::default()
            }),
            resource: Some(UResource {
                id: Some(1 << URESOURCE_ID_LENGTH),
                ..Default::default()
            }),
            ..Default::default()
        };

        let uprotocol_uri = MicroUriSerializer::serialize(&uri);
        assert!(uprotocol_uri.is_err());
        assert_eq!(
            uprotocol_uri.unwrap_err().to_string(),
            "UResource id larger than allotted 16 bits: 0x10000"
        );
    }

Perhaps a more thorough solution is to create some sort of internal-to-the-Rust-library version of the UURI that can ensure that this does not occur in the first place? But I'm not familiar enough yet to have found other instances like this.

Hey @stevenhartley, @sophokles73, @dakrbo:

With the issue closed regarding the implementation of the SDK, do you think now makes sense to publish a v0.1 to crates.io?

It's a pre 1.0 release, so we can still make breaking API changes like the one suggested to the listener type definition.

... to collect next steps in the evolution of our build pipeline and associated workflows... (and also so we can close #51 )

• review the scripts in ./tools, ensure we have locally-runnable versions especially of the check workflow
• #73 add x-compilation from x86 to arm, to go into our nighly workflow and mainly serve as a proof-point/canary which I'm not sure we'll ever see roll off the perch
• #25 incorporate the EF Dash tool for code scanning
• add cross-compile support for all valid Android targets
• add doctest steps to check and nightly workflows (as nexttest doesn't do doctests atm)

One of the unit tests of ucloudeventutils.rs has failed in a recent nightly build.

In beginning work on the uStreamer, I found the need to be able to put a ULinkZenoh struct inside an Arc to move it around in a thread safe manner and use a single instance of it.

However, I found that in order to do so I needed to add Send + Sync as required bounds on UTransport, RpcServer, and RpcClient.

Thankfully, the implementation of up-client-zenoh-rust was already written in a thread-safe manner, so I was able to fork up-rust, make that small change and keep working for now.

I would like to codify that on those traits tho, once the PR from CY merges which contains RpcServer.

WDYT @sophokles73, @AnotherDaniel, @evshary?

Perhaps we can also briefly discuss in tomorrow's meeting too.

Howdy 👋

@stevenhartley recently brought up on a PR going into up-client-zenoh-rust that the other language libs implement their UUID builder / factory as singletons to avoid an end-user accidentally foot-gunning themselves by using multiple UUIDBuilder within one uE.

He asked why the same wasn't done in up-rust. I had my thoughts as to why:

Making it a singleton in Rust forces upon the end user a certain way of interacting with the singleton and there are multiple (reasonable) ways of doing this in Rust. However, maybe it makes sense to give a "simple" path to an end-user not foot-gunning themselves with multiple UUIDBuilder, but allow the possibility for an advanced user to ensure it's a singleton themselves?

A couple of reasonable ways of doing this in Rust:

• lazy_static
• once_cell

Looking to hear feedback on this, hopefully from @sophokles73, @AnotherDaniel, and @evshary

As @stevenhartley has found by experimenting on uStreamer, it looks like it will make it easier to have a uSubscription trait.

Why?

Because up-streamer-rust (which is generic and should be able to work with any transports) will need to interact with not a concrete implementation of a uSubscription, but in fact some generic representation of one, because there will be different uSubscription implementations per transport.

Tagging in @AnotherDaniel & @sophokles73 for discussion

It is unclear how the UTransport::receive function handles situations where no message is (immediately) available on the given topic, i.e. when will the returned future be completed?

async fn receive(&self, topic: UUri) -> Result<UMessage, UStatus>;

FMPOV the function should accept an argument indicating for how long the client wants to wait, e.g. 0 = fail immediately if no message is available, and t > 0 = fail after t milliseconds without a being message available:

async fn receive(&self, topic: UUri, timeout: u16) -> Result<UMessage, UStatus>;

Now the definition of invoke_method is

async fn invoke_method(
        &self,
        method: UUri,
        request: UPayload,
        options: CallOptions,
    ) -> RpcClientResult;

https://github.com/eclipse-uprotocol/up-rust/blob/main/src/rpc/rpcclient.rs#L45C5-L50C26
While the structure of CallOptions is

pub struct CallOptions {
    timeout: u32,
    token: String,
}

https://github.com/eclipse-uprotocol/up-rust/blob/main/src/rpc/calloptions.rs#L18C1-L21C2

However, invoke_method is up-l2 API and it should be implemented by up-l1 API, which is async fn send(&self, message: UMessage) -> Result<(), UStatus>;
https://github.com/eclipse-uprotocol/up-rust/blob/main/src/transport/datamodel/utransport.rs#L42C5-L42C68

While UMessage comprises uPayload and uAttributes, does that mean we should generate our own uAttributes in invoke_method?

I wanted to discuss a potential overhead to our rust SDK's listener apis. Currently, the register_listener() function within the SDK requires a listener parameter of type Box<dyn Fn(Result<UMessage, UStatus>) + Send + Sync + 'static>, returning a listener string for later use with unregister_listener().
However, I've found it challenging to manage listeners effectively without associating them directly with a UUri and associated function. Other SDKs like Java and Python, which accept the listener function directly for both registration and unregistration, our current Rust SDK requires additional bookkeeping to track listeners.

Though in real life scenario same uE will rarely listen twice on same UUri but still this can be good correction. Either we expose functions to convert listener function to listener string to uE developer OR we modify rust SDK to match other SDKs.

Current listener type is defined as follows:

listener: Box<dyn Fn(UMessage) + Send + 'static>

From the current definition, I foresee two main issues:

1. The callback only accepts a UMessage. It is therefore impossible for the uTransport concrete implementation to notify the upper logic about errors.
2. The callback has the following trait bounds: Send + 'static. However, being Sync not required, it may be an issue in multi-threaded scenarios. E.g. on zenoh the subscriber callback is expected to be Sync because the async executor is multi-threaded and the callback could be called from different threads every time.

What I'd like then to discuss in this issue is the possibility to modify the listener type to the following:

listener: Box<dyn Fn(Result<UMessage, UStatus>) + Send + Sync + 'static>

I'm open to discussion.

I have just learned that cargo deny checks for security advisories for crates being used :-)

IMHO we should therefore add a cargo deny check command to the Nightly build in order to catch security advisories and abandoned crates, even if no new PRs have been created (and thus the standard Check workflow is not run).

I find the current iteration of the RpcMapper helper functions to be mostly useless; more precisely, they will only work in the special case where there's Any objects used to wrap UMessage payload content. While this might be a default case, it does forgo the opportunity to take into consideration the UPayload.type, and extract content accordingly.

I am working on an improved version, in conjunction with the up-subscription effort, this will result in an update/extension to RpcMapper sometimes in the coming weeks. This issue is to let you guys know that I'm touching things in this area, just in case someone else feels the same pain...

The following issue will be to track the work to replace the custom data model for the one from core-api and generated from proto