jonasbb / serde_with

This crate provides custom de/serialization helpers to use in combination with serde's `with`-annotation and with the improved `serde_as`-annotation.

Custom de/serialization functions for Rust's serde

---

This crate provides custom de/serialization helpers to use in combination with serde's with annotation and with the improved serde_as-annotation. Some common use cases are:

• De/Serializing a type using the Display and FromStr traits, e.g., for u8, url::Url, or mime::Mime. Check DisplayFromStr for details.
• Support for arrays larger than 32 elements or using const generics. With serde_as large arrays are supported, even if they are nested in other types. [bool; 64], Option<[u8; M]>, and Box<[[u8; 64]; N]> are all supported, as this examples shows.
• Skip serializing all empty Option types with #[skip_serializing_none].
• Apply a prefix to each field name of a struct, without changing the de/serialize implementations of the struct using with_prefix!.
• Deserialize a comma separated list like #hash,#tags,#are,#great into a Vec<String>. Check the documentation for serde_with::StringWithSeparator::<CommaSeparator, T>.

Getting Help

Check out the user guide to find out more tips and tricks about this crate.

For further help using this crate, you can open a new discussion or ask on users.rust-lang.org. For bugs, please open a new issue on GitHub.

Use serde_with in your Project

# Add the current version to your Cargo.toml
cargo add serde_with

The crate contains different features for integration with other common crates. Check the feature flags section for information about all available features.

Examples

Annotate your struct or enum to enable the custom de/serializer. The #[serde_as] attribute must be placed before the #[derive].

The as is analogous to the with attribute of serde. You mirror the type structure of the field you want to de/serialize. You can specify converters for the inner types of a field, e.g., Vec<DisplayFromStr>. The default de/serialization behavior can be restored by using _ as a placeholder, e.g., BTreeMap<_, DisplayFromStr>.

DisplayFromStr

#[serde_as]
#[derive(Deserialize, Serialize)]
struct Foo {
    // Serialize with Display, deserialize with FromStr
    #[serde_as(as = "DisplayFromStr")]
    bar: u8,
}

// This will serialize
Foo {bar: 12}

// into this JSON
{"bar": "12"}

Large and const-generic arrays

serde does not support arrays with more than 32 elements or using const-generics. The serde_as attribute allows circumventing this restriction, even for nested types and nested arrays.

On top of it, [u8; N] (aka, bytes) can use the specialized "Bytes" for efficiency much like the serde_bytes crate.

#[serde_as]
#[derive(Deserialize, Serialize)]
struct Arrays<const N: usize, const M: usize> {
    #[serde_as(as = "[_; N]")]
    constgeneric: [bool; N],

    #[serde_as(as = "Box<[[_; 64]; N]>")]
    nested: Box<[[u8; 64]; N]>,

    #[serde_as(as = "Option<[_; M]>")]
    optional: Option<[u8; M]>,

    #[serde_as(as = "Bytes")]
    bytes: [u8; M],
}

// This allows us to serialize a struct like this
let arrays: Arrays<100, 128> = Arrays {
    constgeneric: [true; 100],
    nested: Box::new([[111; 64]; 100]),
    optional: Some([222; 128]),
    bytes: [0x42; 128],
};
assert!(serde_json::to_string(&arrays).is_ok());

skip_serializing_none

This situation often occurs with JSON, but other formats also support optional fields. If many fields are optional, putting the annotations on the structs can become tedious. The #[skip_serializing_none] attribute must be placed before the #[derive].

#[skip_serializing_none]
#[derive(Deserialize, Serialize)]
struct Foo {
    a: Option<usize>,
    b: Option<usize>,
    c: Option<usize>,
    d: Option<usize>,
    e: Option<usize>,
    f: Option<usize>,
    g: Option<usize>,
}

// This will serialize
Foo {a: None, b: None, c: None, d: Some(4), e: None, f: None, g: Some(7)}

// into this JSON
{"d": 4, "g": 7}

Advanced serde_as usage

This example is mainly supposed to highlight the flexibility of the serde_as annotation compared to serde's with annotation. More details about serde_as can be found in the user guide.

use std::time::Duration;

#[serde_as]
#[derive(Deserialize, Serialize)]
enum Foo {
    Durations(
        // Serialize them into a list of number as seconds
        #[serde_as(as = "Vec<DurationSeconds>")]
        Vec<Duration>,
    ),
    Bytes {
        // We can treat a Vec like a map with duplicates.
        // JSON only allows string keys, so convert i32 to strings
        // The bytes will be hex encoded
        #[serde_as(as = "Map<DisplayFromStr, Hex>")]
        bytes: Vec<(i32, Vec<u8>)>,
    }
}

// This will serialize
Foo::Durations(
    vec![Duration::new(5, 0), Duration::new(3600, 0), Duration::new(0, 0)]
)
// into this JSON
{
    "Durations": [5, 3600, 0]
}

// and serializes
Foo::Bytes {
    bytes: vec![
        (1, vec![0, 1, 2]),
        (-100, vec![100, 200, 255]),
        (1, vec![0, 111, 222]),
    ],
}
// into this JSON
{
    "Bytes": {
        "bytes": {
            "1": "000102",
            "-100": "64c8ff",
            "1": "006fde"
        }
    }
}

License

Licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contribution

For detailed contribution instructions please read CONTRIBUTING.md.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual-licensed as above, without any additional terms or conditions.

serde_with's Issues

with_prefix! Making the module public

On a few occasions we have the same struct with the same prefix used in our applications. Currently the with_prefix creates a private module for serialize/deserialize and we have to do a hack in order to use the same prefix in other places and applications.

I'm opening this issue to see how this could be improved in future version.

// re-export the prefix module!
pub mod prefix_active {
    use serde_with::with_prefix;

    pub use prefix_active::*;

    with_prefix!(prefix_active "active_");
}

PS: Thank you all very much for creating and maintaining this crate!

Document the serde_as crate rename feature in the user guide

• Add documentation about serde_as to the user guide
• Add cate rename feature to DeserializeFromStr and SerializeDisplay
  • In their doc comment
  • In the user guide

v1.6.3 (v1.4.0 macros) Breaks Un-annotated Fields

I have a struct like this:

#[serde_with::serde_as]
#[derive(Serialize, Deserialize, Debug, PartialEq)]
pub struct Thing {
    pub id: u8,

    #[serde_as(as = "Vec<serde_with::hex::Hex>")]
    pub data: Vec<Vec<u8>>,
}

It worked well until today, presumably with the release of v1.6.3, when I started getting this error:

error: An empty `serde_as` attribute on a field has no effect. You are missing an `as`, `serialize_as`, or `deserialize_as` parameter.
  --> src/thing:14:5
   |
14 |     pub id: u8,
   |     ^^^

I did not previously have to annotate the id field. So I tried adding an annotation:

#[serde_with::serde_as]
#[derive(Serialize, Deserialize, Debug, PartialEq)]
pub struct Thing {
    #[serde_as(as = "u8")]
    pub id: u8,

    #[serde_as(as = "Vec<serde_with::hex::Hex>")]
    pub data: Vec<Vec<u8>>,
}

But now it says:

error[E0277]: the trait bound `u8: SerializeAs<_>` is not satisfied
   --> src/thing:11:1
    |
11  | / #[serde_with::serde_as]
12  | | #[derive(Serialize, Deserialize, Debug, PartialEq)]
    | |__________________^ the trait `SerializeAs<_>` is not implemented for `u8`
    | 

So how do I get the default serialization behavior back again?

Add serialize/deserialize adapters for chrono's Duration

Similar issue exists here: chronotope/chrono#117

Make `serde_as` support any value in serde's attributes

Currently, this panics in the proc-macro, because the rename is unknown to the darling parsing code.

#[serde_as]
#[derive(Debug, Deserialize)]
struct S {
    #[serde(rename = "def")]
    abc: String,
};

serde_with/serde_with_macros/src/lib.rs

Lines 376 to 385 in ec3fcd3

	#[derive(FromField, Debug)]
	#[darling(attributes(serde))]
	struct SerdeWithOptions {
	#[darling(default)]
	with: Option<String>,
	#[darling(default)]
	deserialize_with: Option<String>,
	#[darling(default)]
	serialize_with: Option<String>,
	}

serialize_always attribute mysteriously fails on array fields

The following runs fine:

#[serde_with::skip_serializing_none]
#[derive(serde::Serialize, serde::Deserialize)]
struct Huh {
    one: Option<String>,

   #[serialize_always]
    two: Option<String>,
}

fn main() {
    let thing = Huh { one: None, two: None };
    println!("{}", serde_json::to_string_pretty(&thing).unwrap());
}

Output is:

{
  "two": null
}

However, turning two into an array makes it fail to compile:

#[serde_with::skip_serializing_none]
#[derive(serde::Serialize, serde::Deserialize)]
struct Huh {
    one: Option<String>,

    #[serialize_always]
    two: [Option<String>; 1],
}

fn main() {
    let thing = Huh { one: None, two: [None] };
    println!("{}", serde_json::to_string_pretty(&thing).unwrap());
}

Compiler output:

error: cannot find attribute `serialize_always` in this scope
 --> src/main.rs:6:4
  |
6 |     #[serialize_always]
  |       ^^^^^^^^^^^^^^^^

error: aborting due to previous error

error: could not compile `sw-test`

This is extremely confusing if you never actually tried the non-array version of the code first, and just think you need to add a namespace qualification or enable a feature or something. If it's not supported on arrays that's odd but no big deal — it should definitely be documented though. (And of course if it is meant to be supported on arrays it's a bug.)

Using Rust 1.53, Cargo.toml:

[package]
name = "sw-test"
version = "0.1.0"
edition = "2018"

[dependencies]
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
serde_with = "1.9"

Ensure `serde_as` is compatible with references

Sometimes it is unecessary to take ownership of types, especially for serialization. Then it can be beneficial to simply serialize a reference. Right now this cannot be combined with #[serde_as].
Something like this should be possible:

#[serde_as]
#[derive(Serialize)]
struct S {
	#[serde_as(as = "Vec<(_, _)>")]
	map: &'a BTreeMap<u32, String>,
}

Documentation for inline proc-macros

Normally re-exported items can be documented directly, without the re-export section in rustdoc.
This is not possible for proc-macro functions due to a limitation in rust, tracked here: rust-lang/rust#58696.

What alternatives are there for documentation?

Reduce the number of derived traits

Right now, all types derive these traits. However, the conversion types are not used in normal code, such that they never need to be copied or default instantiated. By removing the derives less code would be emitted, maybe slightly reducing the compile time and artifact output size.

• Copy
• Clone
• Debug
• Default

Supports both flattened and nested input

Originally suggested at: serde-rs/serde#1485

Add compiletest to test the error messages of proc_macros in serde_with_macros

The compiletest crate can be used to test for error messages of proc_macros.

Currently it has a problem with running cargo check and cargo test after each other. Since we test also using clippy, this prevents compiletest from working correctly. The issue is tracked at Manishearth/compiletest-rs#101.

An initial test setup using compiletest is in the compiletest branch.

RUSTSEC-2020-0095: difference is unmaintained

difference is unmaintained

Details
Status	unmaintained
Package	difference
Version	2.0.0
URL	johannhof/difference.rs#45
Date	2020-12-20

The author of the difference crate is unresponsive.

Maintained alternatives:

• pretty_assertions

• dissimilar

• treediff

• diffus

See advisory page for additional details.

Use the Iterator → Array function from the Rust standard library

https://github.com/rust-lang/rust/blob/afaf33dcafe9c7068b63eb997df221aa08db7c29/library/core/src/array/mod.rs#L563-L628

This is not a public function, so the code needs to be copied.
This is very likely be better than the existing code.
The function uses too many unstable features to be useful right now.

These issues track adding such a function to the standard library.
rust-lang/rust#69985
rust-lang/rust#81615

core::array::try_from_fn: Currently blocked on try trait v2. See tracking issue rust-lang/rust#89379

Use clippy's MSRV support when available

Nightly clippy supports specifying the MSRV which disables lints incompatible with this Rust versions, e.g., because it requires library functions not yet available.
https://github.com/rust-lang/rust-clippy/blob/master/README.md#specifying-the-minimum-supported-rust-version
This is not yet available on stable clippy, so it cannot be used yet, since then clippy complains about unknown keys in the config file.
The alternative, custom inner attributes, is not yet stable in Rust and also cannot be used.

At some point, clippy will hopefully support the rust-version field from Cargo.toml (rust-lang/rust-clippy#7765) such that it is not necessary to have a separate configuration just for clippy.

Serialize a String into Vec<u8>

Found here: serde-rs/serde#1395

"wrong type for with_prefix" for flattening BTreeMap

use std::collections::BTreeMap;

use serde::Serialize;
use serde_with::with_prefix;

#[derive(Hash, PartialEq, Eq, Debug, Serialize, Ord, PartialOrd)]
enum Foo {
    One, Two, Three
}

#[derive(Hash, PartialEq, Eq, Debug, Serialize, Ord, PartialOrd)]
struct Data {
    stuff: String,

    #[serde(flatten, with = "foo")]
    foo: BTreeMap<Foo, i32>,
}
with_prefix!(foo "foo_");

fn main() {
    let mut my_foo = BTreeMap::new();
    my_foo.insert(Foo::One, 1);
    my_foo.insert(Foo::Two, 2);
    my_foo.insert(Foo::Three, 3);

    let row = Data {
        stuff: "Stuff".to_owned(),
        foo: my_foo
    };

    let my_str = serde_json::to_string(&row).unwrap();
    println!("{}", my_str);
}

I'm expecting to see

{"stuff":"Stuff","foo_One":1,"foo_Two":2,"foo_Three":3}

but I get Error("wrong type for with_prefix", line: 0, column: 0)',

If I just use the regular serde flatten I get

{"stuff":"Stuff","One":1,"Two":2,"Three":3}

Deserialize fallback on error

Is there a way to specify a fallback type to deserialize from? Or perhaps even better, a list of types (all implementing DeserializeAs) that could be tried in order when deserializing?

Specifically, my use case involves parsing some JSON values which might be either numbers, or numbers encoded as strings. Either {"value": 123 } or {"value": "123"} are possible.

Of course, writing a specific type for this case is trivial.

But I'm thinking about an interface like:

struct Container {
    #[serde_as(as = "DeserializeOptions<Self, DisplayFromStr>")]
    value: usize,
}

fn main() {
    let foo_json = r#"{ "value": 123 }"#;
    let foo: Container = serde_json::from_str(foo_json).unwrap();
    assert_eq!(foo.value, 123);

    let bar_json = r#"{ "value": "123" }"#;
    let bar: Container = serde_json::from_str(bar_json).unwrap();
    assert_eq!(bar.value, 123);
}

This also would simplify the implementation of various structs like BytesOrString

I'm definitely willing to work on this, if there's no ready available alternative.

Improved UX for custom code inside containers like `Option` and `Vec`

Currently, the serde(with = "...") annotations are not easily composable. This is a problem if you want to apply custom de/serialization code within container types, e.g., Option or Vec.
serde#723 tracks the issue on the serde side. The user @markazmierczak proposed a solution based around two new traits DeserializeAs<T> and SerializeAs<T>. The example code is in this gist as MIT/Apache-2 licensed code.

Todos:

• Remove serde private APIs such as __private_visit_untagged_option
• Ensure all types which implement SerializeAs also implement DeserializeAs, if possible.
• Try to achieve implementation parity with serde
• For each old module which has a SerializeAs replacement type add a tiny documentation peace pointing it to the new way.

Ideas:

• Instead of using SameAs<T> it is also possible to use a type like Same, without being generic over a type.
• Maybe it is possible with a proc-macro to convert an annotation like #[serde_with(as = "HashMap<_, DisplayString")] into the correct serde annotation #[serde(with = "As::<HashMap<Same, DisplayString>)]. That is, use _ as a placeholder when no transformation is wanted.
• A generic impl like

   impl<T: Serialize> SerializeAs<T> for T { .. }

  is not possible, as it might overlap with other implementations. But for every T, this could be implemented manually, e.g.,

   impl SerializeAs<i32> for i32 { .. }

  partially avoiding the need for the SameAs<T>/Same types.

Should `serde_as` apply `default` when the field is of type `Option<_>`

Using deserialize_with or serde_as on a Option field makes them mandatory while deserializing. In order to make the behavior match more closely with the default serde behavior, serde_as could apply the default attribute on the field. This would make the field optional again while deserializing.

The downside would be that is complicates the macro and would introduce additional behavior.
Right now it only translates the serde_as attribute into serde's deserialize_with and serialize_with.

Originally proposed in #183
Related to #184

Feature request: `serde_as` attribute: allow usage with re-exported `serde_with` crate at different path

It would be useful for my scenario to allow the serde_as attribute macro to function with the serde_with crate rooted at an alternate path - equivalently to Serde's #[serde(crate = "...")] attribute.

As the Serde docs mention, this would be used when a deserialise impl is being generated from within another macro. By re-exporting serde_with from my own crate, I allow consumers to not have to add the crate to their own manifest - and removing/modifying my usage of serde_with in future does not become a breaking change.

Happy to work on this if you'd like.

Generalizing serde_with::rust::double_option

Specifically, I am using this alternative module in my own project:

mod explicit_null {
    pub fn serialize<S, T>(values: &Option<T>, serializer: S) -> Result<S::Ok, S::Error>
    where
        S: serde::ser::Serializer,
        T: serde::ser::Serialize,
    {
        match values {
            None => serializer.serialize_unit(),
            Some(t) => serde::ser::Serialize::serialize(t, serializer),
        }
    }
    pub fn deserialize<'de, T, D>(deserializer: D) -> Result<Option<T>, D::Error>
    where
        T: serde::de::Deserialize<'de>,
        D: serde::de::Deserializer<'de>,
    {
        serde::de::Deserialize::deserialize(deserializer).map(Some)
    }
}

This allows the use of e.g. Option<serde_json::Value> to work where a nonpresent value is represented as None, and an explicit null is represented as Some(Value::Null).

There are no inference issues with using the more general API with Option<Option<T>> via serde(with), so the definition can probably be updated in-place (though would still technically be a breaking change in-place).

Provide a deserialize_ignore_any function

Hello !
Would you be interested in providing the following function :

fn deserialize_ignore_any<'de, D: Deserializer<'de>, T:Default>(deserializer: D) -> Result<T, D::Error> {
    serde::de::IgnoredAny::deserialize(deserializer).map(|_| T::default())
}

It is useful in combination with #[serde(deserialize_with = "...") when a field should accept any value without actually storing its contents.

In particular, it can be used to work around the fact that #[serde(other)] doesn't work with non-unit enum variants, with for instance :

#[derive(Deserialize)]
enum Item {
    Foo, Bar, Baz,
    #[serde(other, deserialize_with = "deserialize_ignore_any")]
    Other,
}

Create further Duration and Timestamp adapters which use milli, micro, and nanoseconds

Right now only DurationSeconds and TimestampSeconds exits. This should be extended to other common time units, meaning milliseconds, microseconds, and nanoseconds.

`string_empty_as_none` can not handle miss field

Without string_empty_as_none, if miss Option<T> field, I get None, with string_empty_as_none I only get the miss field error.

test tests/compile-fail/skip-none-always.rs ... mismatch

Hi. After mass rebuild in Fedora 33 serde_with_macros fails to build due to test error:

test tests/compile-fail/skip-none-always.rs ... mismatch

EXPECTED:
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
error: The attributes `serialize_always` and `serde(skip_serializing_if = "...")` cannot be used on the same field: `a`.
 --> $DIR/skip-none-always.rs:7:1
  |
7 | #[skip_serializing_none]
  | ^^^^^^^^^^^^^^^^^^^^^^^^

error: The attributes `serialize_always` and `serde(skip_serializing_if = "...")` cannot be used on the same field.
  --> $DIR/skip-none-always.rs:15:1
   |
15 | #[skip_serializing_none]
   | ^^^^^^^^^^^^^^^^^^^^^^^^

error: `serialize_always` may only be used on fields of type `Option`.
  --> $DIR/skip-none-always.rs:23:1
   |
23 | #[skip_serializing_none]
   | ^^^^^^^^^^^^^^^^^^^^^^^^
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈

ACTUAL OUTPUT:
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
error: The attributes `serialize_always` and `serde(skip_serializing_if = "...")` cannot be used on the same field: `a`.
 --> $DIR/skip-none-always.rs:7:1
  |
7 | #[skip_serializing_none]
  | ^^^^^^^^^^^^^^^^^^^^^^^^
  |
  = note: this error originates in an attribute macro (in Nightly builds, run with -Z macro-backtrace for more info)

error: The attributes `serialize_always` and `serde(skip_serializing_if = "...")` cannot be used on the same field.
  --> $DIR/skip-none-always.rs:15:1
   |
15 | #[skip_serializing_none]
   | ^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: this error originates in an attribute macro (in Nightly builds, run with -Z macro-backtrace for more info)

error: `serialize_always` may only be used on fields of type `Option`.
  --> $DIR/skip-none-always.rs:23:1
   |
23 | #[skip_serializing_none]
   | ^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: this error originates in an attribute macro (in Nightly builds, run with -Z macro-backtrace for more info)
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
note: If the actual output is the correct output you can bless it by rerunning
      your test with the environment variable TRYBUILD=overwrite

test tests/compile-fail/skip-none-not-struct.rs ... mismatch

EXPECTED:
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
error: The attribute can only be applied to struct or enum definitions.
 --> $DIR/skip-none-not-struct.rs:5:1
  |
5 | #[skip_serializing_none]
  | ^^^^^^^^^^^^^^^^^^^^^^^^
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈

ACTUAL OUTPUT:
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
error: The attribute can only be applied to struct or enum definitions.
 --> $DIR/skip-none-not-struct.rs:5:1
  |
5 | #[skip_serializing_none]
  | ^^^^^^^^^^^^^^^^^^^^^^^^
  |
  = note: this error originates in an attribute macro (in Nightly builds, run with -Z macro-backtrace for more info)
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
note: If the actual output is the correct output you can bless it by rerunning
test compile_test ... FAILED

failures:

---- compile_test stdout ----
thread 'compile_test' panicked at '2 of 2 tests failed', /usr/share/cargo/registry/trybuild-1.0.30/src/run.rs:60:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    compile_test

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

      your test with the environment variable TRYBUILD=overwrite

• Koji task
• Build log

#[serde_as(as = "Option<OneOrMany<_>>")]#[skip_serializing_none] fails to deserialize with None

I have this code which fails. I believe it should be supported.

#[serde_as]
#[skip_serializing_none]
#[derive(Debug, PartialEq, Serialize, Deserialize)]
struct Demo {
    #[serde_as(as = "Option<OneOrMany<_>>")]
    contents: Option<Vec<String>>,
}

It serializes fine with a single element in a vector or multiple elements in a vector

It fails with an error when the contents are set to None

failures:

---- tests::vec_none stdout ----
None
Serialized ---
{}

Error: Message("missing field contents", Some(Pos { marker: Marker { index: 4, line: 2, col: 0 }, path: "." }))

Please find a test case attached.

Workaround: Dont specify #[skip_serializing_none]

test.txt

All types of Bytes should be available in hex

I have a [u8, N] and want to deserialize it in the hex format. Base64 would also be cool but also way more complex.

Extend the `Timestamp*Seconds` types to support `chrono::NaiveDateTime`

Support for NaiveDateTime could treat the value equivalent to DateTime<Utc>. All operations would pretend it has a known timezone and use that for serialization purposes.

Question about const generics

I've seen some examples of using serde_as for explicitly-defined size ranges, but I'm struggling to figure out how to apply to const generics, if it's even possible.

use serde::{Serialize, Deserialize};
use serde_with::serde_as;

#[serde_as]
#[derive(Copy, Clone, PartialEq, Eq, Serialize, Deserialize)]
struct FixedMatrix<T, const ROW: usize, const COL: usize> {
    serde_as(as = "[[_; ROW]; COL]")
    data: [[T; ROW]; COL],
}

Was browsing through serde-rs/serde#1937 and saw mention of:

#[serde_with::serde_as]
#[derive(Serialize)]
struct Foo {
    #[serde_as(as = "[_; 2]")]
    bar: [String; 2],
    #[serde_as(as = "Vec<[_; 5]>")]
    foobar: Vec<[u8; 5]>,
}

Convert a map to a list of single elements

Sometimes it makes sense to deserialize a map as a list of single items, i.e., Vec<T>. The map-key becomes a special field in T.

{
	"id0": {
		"field_a": ...,
		...
	},
	"id1": {...},
	"id2": {...}, 
}

This could be converted to:

struct Data {
	id: IdType,
	field_a: ...,
	field_b: ...,
}

Prior art: https://crates.io/crates/serde-key-value-vec-map

Rework the serde_as array implementations

Remove the special casing for 0-length arrays such that the code might be compatible with const generics.

links to the user guide don't work

The documentation for the latest release has a few links to the user guide (in the first section, and also where it links to "feature flags"). These don't seem to work from any where I've tried to click them. They don't work from the docs.rs page for the crate, the docs.rs docs page for the same release, or the rendered GitHub README. I'm not sure where to find the guide.

Add `serde_as` support to more wrapper types

There are a bunch of single element wrapper types which are not currently usable with serde_as. It might be useful to implement the SerializeAs and DeserializeAs traits for them to.

• Arc

• ArcWeak

• Cell

• RefCell

• Mutex

• Rc

• RcWeak

• RwLock

• Result

• &mut

All of them are also supported by serde, at least opt-in.

Create a proc-macro attribute which applies a second attribute on all fields of a type.

Serialization code is often repetitive. One helper to improve on this already exists in this crate, the serde_with::skip_serializing_none attribute.

However, this could be generalized to work with any attribute on a user-specified type (list).
This could look like

#[apply(Vec, HashMap => #[serde(skip_serializing_if = "$TYPE::is_empty"])]
struct {}

This would apply the attribute on all fields of type Vec and HashMap. On the right hand side, the placeholder $TYPE would be replaced by the type of the field. This would make it easier to skip serializing all empty container types.

serde_with::skip_serializing_none could in the end be implemented on top of this more general proc-macro.

Serialize a HashMap as a list of tuples

serde-rs/serde#1476

• HashMap -> List of Tuples
• BTreeMap -> List of Tuples
• Vec<(T, T)> -> Map

#[serde_as(as = X)] fails silently without the quotes

When forgetting to add quotes to the type-string within the serde_as macro, the macro seems to do nothing, but won't throw an error, either. This results in the compiler complaining about the field in question not implementing De/Serialize. Example:

Works:

#[serde_as]
#[pyclass]
#[derive(Debug, Deserialize, Serialize, Clone)]
pub struct Container {
    #[pyo3(get, set)]
    #[serde_as(as = "Vec<LocalPy<Inner>>")]
    pub inner: Vec<Py<Inner>>,
}

Fails because Py<Inner> does not implement De/Serialize:

#[serde_as]
#[pyclass]
#[derive(Debug, Deserialize, Serialize, Clone)]
pub struct Container {
    #[pyo3(get, set)]
    #[serde_as(as = Vec<LocalPy<Inner>>)]
    pub inner: Vec<Py<Inner>>,
}

I guess you're using some kind of if-let on the structure of the macro and ignoring any incorrectly-constructed macros rather than throwing a compile-error if the macro structure does not match your expectations.

I hope you don't mind the frequent issues.

Macros for serializing a type as another type

I've been using these macros for a long time, to serialize a type as a different repr type.
They make it easy to serialize types from other crates that aren't Serialize/Deserialize.
I think it makes sense to include these macros in this crate, what do you think?

/// serializes a type as a different repr type that it can be converted Into and From
#[macro_export]
macro_rules! serde_as {
	($m:ident, $t:ty, $r:ty) => {
		pub mod $m {
			use super::*;
			use serde::{Deserialize, Deserializer, Serialize, Serializer};

			pub fn serialize<S: Serializer>(x: &$t, serializer: S) -> Result<S::Ok, S::Error> {
				let y: $r = (*x).into();
				y.serialize(serializer)
			}

			pub fn deserialize<'de, D: Deserializer<'de>>(deserializer: D) -> Result<$t, D::Error> {
				let y: $r = Deserialize::deserialize(deserializer)?;
				Ok(y.into())
			}
		}
	};
}

/// serializes a type as a different repr type using the given conversion functions
#[macro_export]
macro_rules! serde_conv {
	($m:ident, $t:ty, $ser:expr, $de:expr) => {
		pub mod $m {
			use super::*;
			use serde::{Deserialize, Deserializer, Serialize, Serializer};

			pub fn serialize<S: Serializer>(x: &$t, serializer: S) -> Result<S::Ok, S::Error> {
				let y = $ser(*x);
				y.serialize(serializer)
			}

			pub fn deserialize<'de, D: Deserializer<'de>>(deserializer: D) -> Result<$t, D::Error> {
				let y = Deserialize::deserialize(deserializer)?;
				Ok($de(y))
			}
		}
	};
}

Example usage:

serde_conv!(serde_timestamp, DOMTimeStamp, |x: DOMTimeStamp| x.0 as f64, |x: f64| DOMTimeStamp(x as _));

And then

#[serde(with = "serde_timestamp")] 
timestamp: DOMTimeStamp

Or:

serde_conv!(bool_str, bool, |x: &bool| x.to_string(), |x: String| x.parse::<bool>());

serde_conv!(flags_hashmap_bool_str, HashMap<Flag, bool>,
	|x: &HashMap<Flag, bool>| {
		x.iter().map(|(k, v)| (*k, v.to_string()))
			.collect::<HashMap<Flag, String>>()
	},
	|x: HashMap<Flag, String>| {
		x.into_iter().map(|(k, v)| Ok((k, v.parse::<bool>()?)))
			.collect::<Result<HashMap<Flag, bool>>>()
	}
);

serde_conv!(my_enum_i32, MyEnum, |x: &MyEnum| *x as i32, |x: i32| Ok(match x {
	0 => MyEnum::A,
	1 => MyEnum::B,
	2 => MyEnum::C,
	3 => MyEnum::D,
	_ => bail!("MyEnum: {}", x),
}));

Document the pitfalls of using `serde_as` on an `Option<_>` field

By default Option fields are optional, i.e., they do not need to appear in the serialized format.
By using deserialize_with or serde_as, they become mandatory.

Originally raised in #183

Conditional compilation ignores serde_as(as = X)

Using conditional compilation appears to break the use of serde_as. Using a struct such as StructC below, I would expect it to be serialized to JSON without issue, as happens without using non conditional compilation.
However, the attribute appears to have no effect, instead failing with the error Error("key must be a string", line: 0, column: 0).

Is this really a bug, or is there something I am not doing right ? I have also given a full example with tests that showcase the problem and what works.

#[cfg_attr(feature="serde_test" ,serde_as)]
#[derive(Default, Debug, Serialize, Deserialize)]
struct StructC {
    #[cfg_attr(feature="serde_test" ,serde_as(as = "Vec<(_, _)>"))]
    map :  HashMap<(i32,i32), i32>
}

lib.rs

use serde::{Deserialize, Serialize};
use serde_with::serde_as;
use std::collections::HashMap;

/// Works
#[serde_as]
#[derive(Default, Debug, Serialize, Deserialize)]
struct StructA {
    #[serde_as(as = "Vec<(_, _)>")]
    map: HashMap<(i32, i32), i32>,
}

/// Fails as expected
#[derive(Default, Debug, Serialize, Deserialize)]
struct StructB {
    map: HashMap<(i32, i32), i32>,
}

/// Should work
#[cfg_attr(feature = "serde_test", serde_as)]
#[derive(Default, Debug, Serialize, Deserialize)]
struct StructC {
    #[cfg_attr(feature = "serde_test", serde_as(as = "Vec<(_, _)>"))]
    map: HashMap<(i32, i32), i32>,
}

/// Should work or not ?
#[serde_as]
#[derive(Default, Debug, Serialize, Deserialize)]
struct StructD {
    #[cfg_attr(feature = "serde_test", serde_as(as = "Vec<(_, _)>"))]
    map: HashMap<(i32, i32), i32>,
}

/// Shows that other attributes work
#[derive(Default, Debug, Serialize, Deserialize)]
struct StructE {
    #[cfg_attr(feature = "serde_test", serde(rename = "renamed"))]
    map: HashMap<String, i32>,
}

#[cfg(test)]
mod tests {
    use crate::{StructA, StructB, StructC, StructD, StructE};

    #[test]
    fn test_struct_a() {
        assert!(cfg!(feature = "serde_test"));
        let mut struct_a = StructA::default();
        struct_a.map.insert((1, -10), -1);

        let a_str = serde_json::to_string(&struct_a);
        println!("{:?}", a_str);
        assert_eq!(a_str.unwrap(), r#"{"map":[[[1,-10],-1]]}"#.to_string());
    }

    #[test]
    fn test_struct_b() {
        assert!(cfg!(feature = "serde_test"));
        let mut struct_b = StructB::default();
        struct_b.map.insert((1, -10), -1);

        let b_str = serde_json::to_string(&struct_b);
        println!("{:?}", b_str);
        assert!(b_str.is_err());
    }

    #[test]
    fn test_struct_c() {
        assert!(cfg!(feature = "serde_test"));
        let mut struct_c = StructC::default();
        struct_c.map.insert((1, -10), -1);

        let c_str = serde_json::to_string(&struct_c);
        println!("{:?}", c_str);
        assert_eq!(c_str.unwrap(), r#"{"map":[[[1,-10],-1]]}"#.to_string());
    }
    #[test]
    fn test_struct_d() {
        assert!(cfg!(feature = "serde_test"));
        let mut struct_d = StructD::default();
        struct_d.map.insert((1, -10), -1);

        let d_str = serde_json::to_string(&struct_d);
        println!("{:?}", d_str);
        assert_eq!(d_str.unwrap(), r#"{"map":[[[1,-10],-1]]}"#.to_string());
    }
    #[test]
    fn test_struct_e() {
        assert!(cfg!(feature = "serde_test"));
        let mut struct_e = StructE::default();
        struct_e.map.insert("a_key".to_string(), -1);

        let e_str = serde_json::to_string(&struct_e);
        println!("{:?}", e_str);
        assert_eq!(e_str.unwrap(), r#"{"renamed":{"a_key":-1}}"#.to_string());
    }
}

cargo.toml

[package]
name = "mwe_issue"
version = "0.1.0"
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
serde = "1.0.130"
serde_with = "1.9.4"
serde_json = "1.0.67"


[features]
serde_test = []

Add support for base64 de/encoding

Support for base64 encoding would look similar to the existing hex module. One significant difference is the amount of different base64 character sets. Different character set can be supported using multiple format-like types. Custom character sets would require an approach similar to StringWithSeparator of a new trait or full const generics.

OneOrMany erroring with Err(Error("a list or single element", line: 1, column: 424))

As a user, when I pass the following JSON:

Expand JSON

{
  "result": {
    "queries": [
      [
        {
          "index": 0,
          "terms": {
            "type": "ref",
            "value": [
              {
                "type": "var",
                "value": "data"
              },
              {
                "type": "string",
                "value": "posts"
              },
              {
                "type": "var",
                "value": "$_term_1_21"
              }
            ]
          }
        },
        {
          "index": 1,
          "terms": [
            {
              "type": "ref",
              "value": [
                {
                  "type": "var",
                  "value": "eq"
                }
              ]
            },
            {
              "type": "ref",
              "value": [
                {
                  "type": "var",
                  "value": "$_term_1_21"
                },
                {
                  "type": "string",
                  "value": "department"
                }
              ]
            },
            {
              "type": "string",
              "value": "company"
            }
          ]
        },
        {
          "index": 2,
          "terms": {
            "type": "var",
            "value": "$_term_1_21"
          }
        }
      ]
    ]
  }
}

Using the following structs:

Click to expand ast.rs

use num_bigint::BigInt;
use serde::{Serialize, Deserialize};
use serde_with::{serde_as, OneOrMany};

#[derive(Debug, Serialize, Deserialize)]
pub struct CompileResponse {
    pub result: QuerySet
}

#[derive(Debug, Serialize, Deserialize)]
pub struct QuerySet {
    pub queries: Vec<Query>
}

pub type Query = Vec<Expr>;

#[serde_as]
#[derive(Debug, Serialize, Deserialize)]
pub struct Expr {
    pub index: isize,
    #[serde_as(as = "OneOrMany<_>")]
    pub terms: Vec<Term>
}

#[derive(Debug, Serialize, Deserialize)]
pub struct Array {
    pub elems: Vec<Term>,
    pub hash: isize
}

#[derive(Debug, Serialize, Deserialize)]
pub struct ArrayComprehension {
    pub term: Term,
    pub body: Vec<Expr>
}

#[derive(Debug, Serialize, Deserialize)]
pub struct SetComprehension {
    pub term: Term,
    pub body: Vec<Expr>
}

#[derive(Debug, Serialize, Deserialize)]
pub struct ObjectComprehension {
    pub key: Term,
    pub value: Term,
    pub body: Vec<Expr>
}


#[derive(Debug, Serialize, Deserialize)]
#[serde(tag = "type", rename_all="lowercase", content="value")]
pub enum ValueTypes {
    Bool(bool),
    Null,
    Number(BigInt),
    String(String),
    Var(String),
    Ref(Vec<Term>),
    Array(Array),
    Set(Vec<Term>),
    Object(Vec<(Term, Term)>),
    Call(Vec<Term>),
    ArrayComprehension(Box<ArrayComprehension>),
    SetComprehension(Box<SetComprehension>),
    ObjectComprehension(Box<ObjectComprehension>)
}

pub fn is_comprehension(x: ValueTypes) -> bool {
    match x {
        ValueTypes::ObjectComprehension(_) => return true,
        ValueTypes::SetComprehension(_) => return true,
        ValueTypes::ArrayComprehension(_) => return true,
        _ => return false
    }
}

pub fn is_scalar(x: ValueTypes) -> bool {
    match x {
        ValueTypes::Bool(_) => return true,
        ValueTypes::Null => return true,
        ValueTypes::String(_) => return true,
        ValueTypes::Number(_) => return true,
        _ => return false
    }
}

#[serde_as]
#[derive(Debug, Serialize, Deserialize)]
pub struct Term {
    #[serde(alias = "type")]
    pub term_type: String,
    #[serde_as(as = "OneOrMany<_>")]
    pub value: Vec<ValueTypes>
}

I receive an ambiguous error that about "a list or single element". This appears, from testing various pieces of my structures, to be related to parsing with respect to me using OneOrMany with an Enum type, though I'm not entirely sure. If I could get some insight into what this error means (or if this is a bug), that would be very helpful :)

EDIT: Looking more closely, this seems to be an error related to when one of my top level vectors (i.e. "terms") maybe a mixture of objects which contain OneOrMany where some are One and others are Many (i.e. values is an array of objects on some objects, but a single object on others).

Deserialize `Vec` while ignoring errors

#[serde_with::serde_as]
#[derive(Debug, serde::Deserialize)]
struct FooBar(#[serde_as(as = "VecSkipError<_>")] Vec<u32>);

serde_json::from_value::<FooBar>(serde_json::json!([
    0,
    "String",
    1,
    [],
    2,
    {},
    3,
]))?

// Results in
// FooBar([0, 1, 2, 3])

VecSkipError implementation

struct VecSkipError<T>(PhantomData<T>);

impl<'de, T, U> DeserializeAs<'de, Vec<T>> for VecSkipError<U>
where
    U: DeserializeAs<'de, T>,
{
    fn deserialize_as<D>(deserializer: D) -> Result<Vec<T>, D::Error>
    where
        D: Deserializer<'de>,
    {
        #[derive(serde::Deserialize)]
        #[serde(
            untagged,
            bound(deserialize = "DeserializeAsWrap<T, TAs>: Deserialize<'de>")
        )]
        enum GoodOrError<'a, T, TAs>
        where
            TAs: DeserializeAs<'a, T>,
        {
            Good(DeserializeAsWrap<T, TAs>),
            // This consumes one "item" when `T` errors while deserializing.
            // This is necessary to make this work, when instead of having a direct value
            // like integer or string, the deserializer sees a list or map.
            Error(serde::de::IgnoredAny),
            #[serde(skip)]
            _JustAMarkerForTheLifetime(PhantomData<&'a u32>),
        }
        
        struct SeqVisitor<T, U> {
            marker: PhantomData<T>,
            marker2: PhantomData<U>,
        }

        impl<'de, T, U> Visitor<'de> for SeqVisitor<T, U>
        where
            U: DeserializeAs<'de, T>,
        {
            type Value = Vec<T>;

            fn expecting(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
                formatter.write_str("a sequence")
            }

            fn visit_seq<A>(self, mut seq: A) -> Result<Self::Value, A::Error>
            where
                A: SeqAccess<'de>,
            {
                let mut values = Vec::with_capacity(seq.size_hint().unwrap_or_default());

                while let Some(value) = seq
                    .next_element()?
                {
                    if let GoodOrError::<T, U>::Good(value) = value {
                        values.push(value.into_inner());
                    }
                }

                Ok(values.into())
            }
        }

        let visitor = SeqVisitor::<T, U> {
            marker: PhantomData,
            marker2: PhantomData,
        };
        deserializer.deserialize_seq(visitor)
    }
}

• https://stackoverflow.com/questions/67702612/how-to-ignore-unknown-enum-variant-while-deserializing

Serialize struct with additional root element

Sometimes structs/enums need an additional root element. This can be done with a single element enum like so:

#[derive(Serialize)]
enum Foobar {
    #[serde(rename = "root_field_name")]
    RootName{
        field_a: bool,
        field_b: u32,
    }
}

// as JSON
{
  "root_field_name": {
    "field_a": false,
    "field_b": 123
  }
}

A better solution would be a macro which adds the root element without requiring changing the struct.

Prior art

• serde-rs/serde#1345
• https://crates.io/crates/serde_struct_wrapper

Better support for `StringWithSeparator` with non-string types

Currently StringWithSeparator only supports using Display and FromStr which is not as convenient when using non-string types (such as enums), because unlike Serde's normal Serialize and Deserialize traits, they cannot be derived automatically. This means developers need to write their own implementations, which can complicate things and also don't support some "fancy" Serde features like attributes for renaming or skipping, for example, enum variants (yes, this can still be done in the actual traits implementation, but doesn't look as good as declarative approch with attributes).

Is it possible to implement custom derive macro(s) that can automatically implement all traits needed by StringWithSeparator for simple non-string types (such as enums) with support for configuring them through attributes (preferably with standard Serde attributes)? I know not all types normally supported by Serde can be supported (for example, nested types), but for things like enums and types that hold strings, numbers, etc., I think it would be quite useful.

Migrate all tests to the `*_expect` versions of the util functions

Check documentation of `tuple_list_as_map`

This part seems to be wrong. The rust type should be a Vec.

#[serde_as]
#[derive(Deserialize, Serialize)]
struct S {
    #[serde_as(as = "BTreeMap<_, _>")] // HashMap will also work
    s: BTreeMap<(String, u32), u32>,
}

https://docs.rs/serde_with/1.6.2/serde_with/rust/tuple_list_as_map/index.html

Create two new derives: `DeserializeFromStr` and `SerializeDisplay`

They would derive Deserialize and Serialize but using the Display/FromStr traits for implementation.
This would be a nicer way to configure the de/serialization where the struct is defined, without needing a wrapper type and without specifying this per field.

This proposal was also made here serde-rs/serde#553 (comment) and here serde-rs/serde#908

Improve error messages when `serde_as` fails to parse attributes

error: custom attribute panicked
   --> src/main.rs:311:5
    |
311 |     #[serde_as]
    |     ^^^^^^^^^^^
    |
    = help: message: called `Result::unwrap()` on an `Err` value: Error { kind: UnknownField(ErrorUnknownField { name: "as", did_you_mean: None }), locations: [], span: Some(#0 bytes(10102..10104)) }

Instead of unwrap the error should be converted into a proper error message:

serde_with/serde_with_macros/src/lib.rs

Line 393 in ec3fcd3

let serde_as_options = SerdeAsOptions::from_field(field).unwrap();

Clearly explain what the `#[serde_as]` macro does

On Discord somebody mentioned that it is unclear that _ and Same mean the same.
The documentation should explain somewhere (probably on the macro itself) what the macro does. Right now this includes the following steps:

• Convert the serde_as field attribute into a serde attribute with the matching with/deserialize_with/serialize_with arguments.

• Wrap the type inside a As::<...>. This converts the SerializeAs trait into a Serialize compatible signature.

• Replace any _ with serde_with::Same. The Same type implement SerializeAs whenever the type is Serialize. Basically the opposite of As to glue the two traits together.

• Possible in the future also applying serde's default on Option fields (ref. #185).

Some conversion steps are mentioned here:
#231 (comment)

Make serde_as easier to use for identity transformations

As seen in #201 it might be confusing why this code doesn't work:

// Can't get this working at all
#[serde_as(as = "BTreeMap<String, ADef>")]
pub map: BTreeMap<String, A>,

// Confusingly enough defining identity transform like this doesn't compile either
#[serde_as(as = "BTreeMap<String, String>")]
pub map2: BTreeMap<String, String>,

The reason is that a blanket implementation for DeserializeAs/SerializeAs is not possible as it would overlap with other implementations. It is possible to special case the impl SerializeAs<String> for String case. This has to be done for every type and would bless the standard library types compared to types from crates.

Another possibility might be to move this logic into the serde_as macro. If it detects two identical types it replaces the type in the as part with _/Same. In the map case the BTreeMap<..., ...> compares not equal, but the String subtype does and would be replaced. In the map2 case already the BTreeMap<..., ...> type is equal and it would convert it to as = "_".

This could technically change how the serialization is performed as it then uses Serialize instead of SerializeAs. So this behavior might require an opt-out which disables the type equality check.
Another option might be to limit this to leaf types, i.e., ones without any generic parameters, since I assume it might be more likely there.

A problem with this approach is how to generalize this to different transformations, for example the BTreeMap<K, V> to Vec<(K, V)> transformation. The first one has two generic arguments while the Vec only has one (the tuple). So the fear here is that it simplifies it for the trivial cases but as soon as somebody want to use other transformations the help would fail and that might cause even more confusion.