tokio-rs / valuable

Valuable

Valuable provides object-safe value inspection. Use cases include passing structured data to trait objects and object-safe serialization.

License

This project is licensed under the MIT license.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Valuable by you, shall be licensed as MIT, without any additional terms or conditions.

valuable's Issues

Consider including additional data in defintion types

For example, we could a type's module path.

Consider Visit::visit_key + Visit::visit_value

The Visit trait has a visit_entry method that accepts a key-value pair. In cases where you’re bridging different APIs you might not be able to guarantee both the key and value are present at the same time to pass to this method. For instance, serde::SerializeMap treats entries as an optimization and requires serializers support receiving keys and values independently. We ended up adding these separate methods to the standard library’s fmt builders for this reason.

We could consider making visit_entry forward to some new visit_key and visit_value methods.

Is it possible to allow to visit a value referencing data owned by the current function?

AFAIK, in the current API, Valuable cannot be implemented well if I need a temporarily owned value for serialization.

struct PathAsString(std::path::PathBuf);

impl Valuable for PathAsString {
    fn as_value(&self) -> Value<'_> {
        /*
        error[E0515]: cannot return value referencing temporary value
          --> lib.rs:74:9
           |
        74 |         Value::String(&self.0.display().to_string())
           |         ^^^^^^^^^^^^^^^----------------------------^
           |         |              |
           |         |              temporary value created here
           |         returns a value referencing data owned by the current function

        error: aborting due to previous error
        */
        Value::String(&self.0.display().to_string())
    }

    fn visit(&self, v: &mut dyn Visit) {
        v.visit_value(Value::String(&self.0.display().to_string()))
    }
}

Is there a way to handle such cases well?

(I was thinking a bit about removing the Valuable::as_value, but not sure if that would be a good idea.)

How to represent tuples?

Currently, tuples do not implement Valuable. How should they be represented? Probably Structable but w/o a name? Alternatively, we would need to add a Tuple variant to Value.

trait Tuplable? 😬

Consider changing Value::Error to &(dyn Error + 'static)

The current error variant in Value::Error is a plain &dyn Error, which unfortunately doesn't support downcasting. It might be worth changing this to &(dyn Error + 'static), which seems to be a reasonably accepted restriction on the type to support any scenario a consumer might have.

valuable-serde: consistency with serde

This is a list of cases where valuable-serde behaves differently from using serde directly.

• valuable/valuable-serde/tests/test.rs

  Lines 69 to 71 in 7a22d9d

  	// TODO:
  	// - valuable treats Option<T> as T or unit
  	// - serde treats Option<T> as Some(T) or None

  
  see also #23 (comment)

• valuable/valuable-serde/tests/test.rs

  Lines 85 to 87 in 7a22d9d

  	// TODO:
  	// - valuable treats unit struct as an empty tuple struct
  	// - serde treats unit struct as unit struct

  
  see also #23 (comment)

• valuable/valuable-serde/tests/test.rs

  Lines 103 to 105 in 7a22d9d

  	// TODO:
  	// - valuable treats unit variant as an empty tuple variant
  	// - serde treats unit variant as unit variant

Introduce a lifetime generic to `Visit` in order to allow borrowing out specific parts of the `valuable::Value` during visits

Motivating use-case

My motivating use-case is to write down in tracing 0.2 a visitor that is able to borrow Event’s message. Doing so today is impossible even though lifetimes should conceptually mostly work out. This problem extends to valuable::Value and its Visit trait as well, so the remaining of the issue will focus on valuable.

So lets say we begin by writing a function as such (any mistakes mine, I basically sketched this out in a github comment field):

fn extract_message<'val>(value: &'val valuable::Value<'val>) -> &'val str {
    struct Visitor<'val> {
        message: &'val str
    }
    impl<'val> valuable::Visit for Visitor<'val> {
        fn visit_entry(&mut self, key: valuable::Value<'_>, value: valuable::Value<'_>) {
            match (key, value) {
                (valuable::Value::String("message"), valuable::Value::String(v)) => {
                    self.message = v; // ERROR: `'_` does not outlive `'val`.
                }
                _ => {}
            }
        }
    }

    let mut visitor = Visitor<'val> { message: "" };
    value.visit(&mut visitor); 
    visitor.message    
}

Rust today provides no way to tell that, '_ will outlive 'val here, and so users are forced to allocate in the implementation to extract the string data here.

Proposed solution

The proposed solution here is to introduce a lifetime to Visit such that Visit becomes like this:

pub trait Visit<'a> {
    fn visit_value(&mut self, value: Value<'a>);
    fn visit_named_fields(&mut self, named_values: &NamedValues<'a>) { ... }
    fn visit_unnamed_fields(&mut self, values: &[Value<'a>]) { ... }
    fn visit_primitive_slice(&mut self, slice: Slice<'a>) { ... }
    fn visit_entry(&mut self, key: Value<'a>, value: Value<'a>) { ... }
}

This gives implementers an optional opportunity to tie the lifetime of values to the lifetime of the visitors contents. Any implementors wanting the current behaviour can write the implementation as such:

// Unfortunately a breaking change ­– you cannot leave out the `<'_>`.
impl Visit<'_> for SomeVisitor { ... }

While implementations such as one presented in the motivating use-case above are also made possible:

impl<'val> valuable::Visit<'val> for Visitor<'val> {
    fn visit_entry(&mut self, key: valuable::Value<'val>, value: valuable::Value<'val>) {
        match (key, value) {
            (valuable::Value::String("message"), valuable::Value::String(v)) => {
                self.message = v; // A-OK!
            }
            _ => {}
        }
    }
}

the trait `tracing::Value` is not implemented for `valuable::Value<'_>

The coerce actor crate uses the valuable crate. In my personal project, which uses both tracing and coerce, I'm unable to build the coerce ("full" features) dependency (i.e., cargo fails trying to check/build coerce v0.8.5 before it starts on my project). Oddly, I'm able to build and fully test the coerce library itself, which suggests something is missing in my project/dependencies definition, although I have strived to match coerce's dependencies. I have posted an issue in the coerce project, and I'm posting here in here there's something in using valuable I missed in the documentation. I appreciate any guidance you may be able to provide. My project can be found at https://github.com/dmrolfs/coerce-cqrs-rs.

The issue I see is:

cargo c
    ...
    Checking coerce v0.8.5
error[E0277]: the trait bound `valuable::Value<'_>: tracing::Value` is not satisfied
  --> /Users/rolfs/.cargo/registry/src/github.com-1ecc6299db9ec823/coerce-0.8.5/src/actor/lifecycle.rs:88:28
   |
88 |                   let span = tracing::info_span!(
   |  ____________________________^
89 | |                     "actor.recv",
90 | |                     ctx = log.as_value(),
91 | |                     message_type = msg.name(),
92 | |                 );
   | |_________________^ the trait `tracing::Value` is not implemented for `valuable::Value<'_>`
   |
   = help: the following other types implement trait `tracing::Value`:
             &'a T
             &'a mut T
             (dyn StdError + 'static)
             (dyn StdError + Sync + 'static)
             (dyn StdError + std::marker::Send + 'static)
             (dyn StdError + std::marker::Send + Sync + 'static)
             Arguments<'a>
             Box<T>
           and 34 others
   = note: required for the cast from `valuable::Value<'_>` to the object type `dyn tracing::Value`
   = note: this error originates in the macro `$crate::valueset` which comes from the expansion of the macro `tracing::info_span` (in Nightly builds, run with -Z macro-backtrace for more info)

...

Implement `Valuable` for Option<T> and `Result<T>`

I believe both types should just "flatten" the values instead of implementing Enumerable.

So, Some(u32) => Value::U32 or Value::Unit.

Create a span for plugin init

Plugin log messages should be correlated for initialization.
Suggest that a span is created around the call to plugin init that contains the following attributes:

• Plugin name
• A version supplied by the state machine (probably a UUID).
  Doing this is a central location will give all plugins better contextual information on startup.

Setup CI

Now that there are tests, we should setup the repo with Github Action for CI.

`NamedValues::get` footgun.

NamedValues::get has a subtle footgun: The given reference to a NamedField must be pointer-equal to a NamedField in NamedValues. But, because NamedField is Copy/Clone, it's trivial to construct programs that subtly violate this requirement; e.g., this program panics:

fn main() {
    use valuable::{NamedField, NamedValues, Value};

    let fields = [
        NamedField::new("foo"),
        NamedField::new("bar")
    ];
    let values = [
        Value::U32(123),
        Value::U32(456),
    ];

    let named_values = NamedValues::new(&fields, &values);

    let field = fields[0];

    assert_eq!(
        named_values.get(&field).and_then(Value::as_u32),
        Some(123)
    );
}

...but this program is fine:

fn main() {
    use valuable::{NamedField, NamedValues, Value};

    let fields = [
        NamedField::new("foo"),
        NamedField::new("bar")
    ];
    let values = [
        Value::U32(123),
        Value::U32(456),
    ];

    let named_values = NamedValues::new(&fields, &values);

    let field = &fields[0];

    assert_eq!(
        named_values.get(field).and_then(Value::as_u32),
        Some(123)
    );
}

Constructing objects from `Value`

I have a use case where I need to "deserialize" Values into instances of struct Foo.

This feels related to #17. Perhaps the impedance mismatch between valuable::Visit and serde::de::Visit is slight enough to be overcome with a generic implementation that allows deserializing a valuable::Value into anything that implements serde::de::Deserialize?

A while ago, I decided to start on an implementation that relies on a FromValue trait and a corresponding proc-macro, which can fallibly convert valuable::Value to any type that it's implemented for. I guess at the time, I must not have considered the serde solution. That implementation is available here.

Is this kind of functionality something that the team would be interested in incorporating into valuable or valuable-serde? If so, what should be the shape of that solution? Alternatively, if something seems fundamentally wrong about my use of tokio tracing, I would appreciate your feedback. Thanks very much!

Feature request: Expose doc comments for struct/enum/field definitions

Given that a big usecase is for generating UIs, the natural usecase for a doc comment is to act as a tooltip. It's surprisingly hard to find any derive macro that exposes doc comments, so I'm wondering if it's possible to do or add here. Maybe there's some limitation that makes it impossible I'm not aware of?

Visitor holding `Value`

As requested on the discord, posting a detailed issue about trying valuable for Tera.

Background

I am trying to see if valuable could replace serde_json::Value for Tera.
The goal in a template engine is to find the value the user wants to use in a given context.
Right now, the context in Tera is just serde_json::Value, which means we serialise everything even though we don’t really need it. See https://docs.rs/tera/1.10.0/tera/struct.Context.html for examples of Context.

In Tera, you need to evaluate expressions based on the context. For example:

• {{ get_metadata(path=page.path ~ “image.jpg”) }}: this concatenates into a single string which is then passed as an argument to a function
• {% if user.is_admin or data.public %}: this evaluates the fields truthiness
• {{ item.price * current_rate | format_currency }}: this multiplies the value found in the context and then applies a filter on the output
• {% for result in data.results[user.birth_year] %}

There are more examples on the Tera documentation: https://tera.netlify.app/docs/
If you are familiar with Django/Jinja2/Twig/Liquid this is essentially the same thing.

The way it currently work is the following:

1. Convert the whole context received into serde_json
2. Evaluates values in brackets, like user.birth_year in the example. If the value is an integer, we assume it’s the index of the array so we don’t change anything
3. Replace the values in brackets by their actual value and create a JSON pointer based on it
4. Use https://docs.serde.rs/serde_json/enum.Value.html#method.pointer on the context to retrieve the value (or not if it wasn’t found)
5. Do pattern matching on the Value kind: if we’re doing a math operation like *, we need to make sure both operands resolve to a Value::Number for example, string concatenation with ~ only work on Value::String and so on

Where valuable comes in

As mentioned, it currently serialises the whole context which can be very costly. For example, rendering a section in a Zola site will clone the content of all the sub-pages if pagination isn’t set up, which can quickly become slow.
The whole Tera cloning the context is actually one of the biggest current perf bottleneck in Zola as you need to clone the content every time you render the page, which can be pretty big considering it’s potentially thousands of HTML pages.

Examples

What I’d like to accomplish is be able to visit the context and extract the Value for each identifier. Let’s use https://github.com/tokio-rs/valuable/blob/master/valuable/examples/print.rs as an example data and say that the person variable is our context.

{{ name }}

The path is [“name”] so we only need to visit Structable and Mappable and look for that key

{{ addresses.1.city }}

The path is [“addresses”, 1, “city”]. We visit Structable/Mappable because it begins with a string and visit the field where it is equal to addresses. This is an array so we visit Listable. We know we want to only care about the element at index 1 so we could only visit that one ideally (see #52). Once there, there is only city left to match so we visit in Structable/Mappable again.

{% for address in addresses %}

Here we want to iterate on the Vec<Address> so I think we just need to visit the addresses and get a Vec<Structable> which we visit in its entirety one by one (since you can break from the loop).

The goal

In all cases above we want to get the Value corresponding to the path out of the visit and pattern match later. Since visit doesn’t return anything, I need to store it on the visitor struct. The Value found could be a primitive, a Mappable/Structable (for loops on key values) or not found at all.

However I get a lifetime error if I try to assign a Value in a small toy example: https://gist.github.com/Keats/150acda601910c5f8a6bd9aceba2ddfd so I’m not sure how I would accomplish that. Is there an example of a visitor holding a Value somewhere?

The article at https://tokio.rs/blog/2021-05-valuable specifically mention template engine so I’m wondering if the idea was limited to writing the value to a string buffer or whether it is too limited for Tera.

`dyn Display` primitive value

In some cases, it may be desirable for types implementing Valuable to record themselves as a pre-formatted human-readable text representation. Currently, this is only possible with the Value::String variant, which takes a &'a str.

For types that want to record themselves using a Display (or Debug) implementation, though, Value::String isn't really sufficient. Because it takes a borrowed string slice, those types would have to allocate ahead of time and store their own formatted representation, so that the borrowed string slice can be handed out in their Valuable impls. This is not ideal.

Therefore, we should probably add &dyn fmt::Display as a primitive Value variant.

derive Valuable for enum

The Enumerable trait and associated visit fns should be sketched out enough to implement derive for them.

Owned [NamedField] for Fields

The current signature of Fields::Named hold a borrowed slice of NamedField. As far as I can tell, it's designed for 'static definition of rust structs. This makes it difficult to define dynamic Structable for a map like we talked in #53 .

Perhaps we can make it a Cow for the dynamic scenario.

How should we implement Valuable for Uuid?

Hi! 👋

cc @QnnOkabayashi

Over in uuid-rs/uuid#583 we've been talking about adding unstable valuable support to Uuid and run into a bit of a design issue I wanted to get some input on.

It looks like we've got two possible directions we can go in:

1. Treat Uuid as its text-based format and use Value::Displayable (#86). This means end-users in the diagnostics use-case will see something resembling a UUID rather than a blob of bytes without needing to match on strings (assuming tracing will eventually run all values through valuable), but it means reconstructing a Uuid from that dyn Display will require more work.
2. Treat Uuid as its raw byte format and use Value::Listable. This is less work than 1, and a more accurate representation of the UUIDs internal structure, but if we want a text-based UUID we need to look at its stringly-typed name and parse it.

This seems like a general question that authors of primitive value types like UUIDs and IP addresses are going to need to answer, so was looking for some input on what you all think is the direction that best aligns with the goals of the valuable project.

question: is valuable expected to work with tracing macros?

I was expecting valuable to work out of the box with tracing's macros (info, debug etc). However, I get an issue saying Value is not implemented for my type (i.e. there is no blanket implementation covering all types that implement Valuable). This currently seems like the only way to make tracing happy:

for (service, old_state) in removed {
    info!(service = &service as &dyn Valuable,"Service unit removed");
    self.service_state.remove(&service);
}

Please let me know if this was an oversight, or if there's a fundamental limitation preventing the compiler from auto-coercing my type (RemoteServiceId) that implements Valuable to Value?

Implement valuable-serde crate

It should be possible to implement Serde's Serialize trait for valuable::Value. This should enable using Serde's existing serializer ecosystem with valuable.

derive: implement #[valuable] attributes

• #[valuable(rename = "...")]; #75
• #[valuable(transparent)]; #75
• #[valuable(skip)]; #75

Consider adding a concrete lifetime to the Visit trait

It looks like most of the work to support visiting borrowed data for some concrete lifetime (instead of just for any lifetime) is already done, the Visit trait just doesn't surface a lifetime that implementors can use to stash values across calls.

I at least find introducing lifetimes late in an API design can get a bit hairy so it might something to explore sooner rather than later 🙂

Consider other Value primitive variants

There are most likely other primitive types that should be passed around explicitly.

Ideas include:

• &[u8]
• Duration
• Instant
• SystemTime
• SocketAddr
• Bytes?
• &mut Display
• &mut Debug

Consider including the total number of fields as part of `Fields::Unnamed.

Consider passing a `container` argument to `visit_named_fields` and `visit_unnamed_fields`

This argument would inform the visitor of the current container being visited (enum, struct, tuple, ...).

publish `valuable-serde`

We've published valuable and valuable-derive, but the serde integration crate remains unpublished. We should probably get that ready to release as well?

Implement valuable-http, a bridge between the http crate and valuable

Until Valuable reaches 1.0, http will most likely not depend on valuable, so we should provide a bridge. This crate would provide wrappers for types in http that implement Valuable.

Then, users can do:

use valuable::Valuable;
use valuable_http::Request;

Request::from(my_request).as_value()

One open question: what argument should the Request type take?

• http::Request
• &http::Request
• impl AsRef<http::Request>

Release initial version

So valuable integration has started in tracing ( tokio-rs/tracing#1608 ) but a blocker for merging it is that valuable needs a release on crates.io. I'm aware there's a milestone for features for the 0.1 release so if that's not ready potentially an alpha release? Or depending on the milestone features punt them to a patch or the next version

Consider renaming NamedValues::entries() -> iter()

This should line up better w/ HashMap.

Add a way to iterate on non-primitives/visit specific slices for Listable

For example in templating engines, people will frequently print the value at the index n of an array. Right now keeping track of the indices of a Listable being visited is not triviall and as in the template engine case, we don't even want to visit all of them, just one.

Something like Listable::visit_slice(range: Range<usize>, &mut dyn Visit) to allow for more flexibility would be great to have. A .get(usize) would be nice too if possible

Feature request: Allow reading structable attributes

At the moment tracing-opentelemetry does a somewhat of a hack to know what fields are metrics. It requires the field name to be prefixed

To publish a new metric, add a key-value pair to your tracing::Event that contains following prefixes:

• monotonic_counter. (non-negative numbers): Used when the counter should only ever increase
• counter.: Used when the counter can go up or down
• value.: Used for discrete data points (i.e., summing them does not make semantic sense)

async fn current_example() { 
     tracing::info!(histogram.racecar.speed = 80_f64);
}

Instead if attributes were exposed it would be possible to do something like this:

/// How fast the racecar is going
#[derive(Valuable)]
#[histogram(key="racecar.speed", unit="mph")]
struct RacecarSpeed(f64);

#[tokio::test]
async fn proposed_example() {
    tracing::info!("Going at {}mph", speed=RacecarSpeed(777.0012_f64));
}

I think there's two approaches here: expose the attributes directly, or expose the typeid and require subscribers to map that typeid to the attribute

Strange happenings with manual implementation of `Valuable`

For the short term, I wanted to make myself a wrapper for anyhow::Error, to use the unstable support in tracing and log the values as structured.

What happened is that my manual implementation - although seemingly correct, fails to serialize. A different implementation, which stringifies early and uses#[derive(Valuable)] works just fine.

Because the code is a bit bigger than fits into an issue, I have created a reproduction in a separate repository, with both versions.

cc @hawkw