sof3 / include-flate

A variant of include_bytes!/include_str! with compile-time deflation and runtime lazy inflation

include-flate

!

A variant of include_bytes!/include_str! with compile-time deflation and runtime lazy inflation.

Why?

include_bytes!/include_str! are great for embedding resources into an executable/library without involving the complex logistics of maintaining an assets manager. However, they are copied as-is into the artifact, leading to unnecessarily large binary size. This library automatically compresses the resources and lazily decompresses them at runtime, allowing smaller binary sizes.

Nevertheless, this inevitably leads to wasting RAM to store both the compressed and decompressed data, which might be undesirable if the data are too large. An actual installer is still required if the binary involves too many resources that do not need to be kept in RAM all time.

Warning

This library compresses included data independently. It is usually more effective to compress the whole output binary together (e.g. distributing .exe.gz ) than to compress independently. In addition, compression algorithms usually produce smaller artifacts by processing the raw input together than by processing already-compressed output. #[cfg_attr] might come handy for conditionally using compression or direct data inclusion.

include-flate's Issues

Publish a new crate version

Could you publish a new version to crates.io so I can get dependencies updated?

#24 Follow-up

LazyLock is being stabilized in the Rust standard library with 1.80, releasing July 25th. The once_cell dependency can be removed in favor of it. rust-lang/rust#121377

[BUG] Modifying the file included with `flate!` doesn't automatically trigger rebuild

To reproduce:

1. Paste this code into main.rs:

fn main() {
    include_flate::flate!(static FOO_STR: str from "foo.bar");
    println!("{}", FOO_STR.as_str());
}

2. Create a foo.bar under crate root, and write some content
3. Run cargo run, and observe that the content of foo.bar is correctly printed
4. Modify the contents of foo.bar and save
5. Run cargo run again, and observe that a rebuild is not triggered, and the old contents of foo.bar are printed
6. Replace flate! with include_str!, repeat steps 2-5, and observe that a rebuild is triggered

---

I am reporting this as a bug primarily because include-flate is advertised as a drop-in replacement for include_str, so this difference in behaviour could be confusing.

Also I agree with the standard library that automatically triggering a rebuild is the sensible default, so include-flate should try to replicate this behaviour. That being said, I'm not too familiar with the implementation details of include_str!, so perhaps it's driven by compiler magic and the automatic rebuild behaviour is not replicable in third-party crates, I'm not sure.

Support for compile-time environment variable lookup

First off, thank you for the great crate!

I'm now generating some plaintext in build.rs to $OUT_DIR/foo.txt, and including it in the library code like include_str!(concat!(env!("OUT_DIR"), "/foo.txt")). It'd be great if flate! also supports either

• flate!(concat!(env!("OUT_DIR"), "/foo.txt")), or
• flate!("$OUT_DIR/foo.txt")

so that one can use include-flate for generated file stored in OUT_DIR (or wherever else relative to an environment variable).

Emitting &[u8] literal is very slow

Issue

rustc takes forever to parse the generated &[u8] literal. I have checked that it is indeed compiled into a raw byte array (no delimiters, no boxing, etc.) in the output executable (thus satisfying the size requirements), but rustc doesn't seem to know to parse the contents in a faster way.

Alternatives

An alternative would be to write the deflated stream into a temp file and use include_bytes!(), but I don't know where the file should be created.

• If I write it to the OS temp directory, it would result in an OS-level resource leak before rebooting (because nobody is deleting the temp files).
• If I always write to the same file in the target directory, concurrent compilations might overwrite each other.
• If I write to somewhere next to the included file, it corrupts the user filesystem. Rust does not recommend build scripts to affect files beyond the OUT_DIR. However, for macros, we don't even have OUT_DIR or anything within the target/ directory.

A solution (still bad but affects less negatively) is to create a hash based on the full file name in the operating system, and save the file content hash + compressed contents using a file within target uniquely identified by the hash. A unique file lock can be acquired while writing. If the file content hash is already identical to the compressed contents, this means (although unlikely) that another compilation has already compressed the file, and there is no need to recompile, preventing reducing the chance of race condition between rustc's handling include_bytes!() and another process's macro compilation.
But where within the target directory? I am not sure.

Allow absolute path again

This means we can include files from OUT_DIR which is useful for dynamic file generation (I need to merge several folders into one place). Using relative path is simply not enough here.

Possibly related: Tracking Issue for proc_macro::{tracked_env, tracked_path} · Issue #99515 · rust-lang/rust (github.com)

Related: Fix compression with interpolated env · Issue #215 · pyrossh/rust-embed (github.com)

Include a usage example in the README

I really like it when READMEs show how to properly use a project to quickly get a feel for it. I think it would make the project more approachable in this case as well.

Support WASM

The include-flate crate is great and would be super useful for WASM apps and libraries. It would be a way to include data files that WASM couldn't otherwise upload.

It's possible that you can make it compatible just by changing some ::std's to ::alloc's.

When I expand the flate! macro on my I get code like this (noticed the ::std's)

struct NAME_TO_PROB_STR {
    __private_field: (),
}
#[doc(hidden)]
static NAME_TO_PROB_STR: NAME_TO_PROB_STR = NAME_TO_PROB_STR {
    __private_field: (),
};
impl ::lazy_static::__Deref for NAME_TO_PROB_STR {
    type Target = ::std::string::String;
    fn deref(&self) -> &::std::string::String {
        #[inline(always)]
        fn __static_ref_initialize() -> ::std::string::String {
            ::include_flate::decode_string(
                b"\x05\xc0;\n\x00 \x08\x00\xd0Y\xc1\xa3\x14.\xd1l\xd0A\x14\x1a\x84~DK\xb7\xefM\x1d\r\xf6Y\xa6\xe6\xdd\xef#\x14)\x90\"s\r\x9c\t?",
            )
        }
        #[inline(always)]
        fn __stability() -> &'static ::std::string::String {
            static LAZY: ::lazy_static::lazy::Lazy<::std::string::String> = ::lazy_static::lazy::Lazy::INIT;
            LAZY.get(__static_ref_initialize)
        }
        __stability()
    }
}
impl ::lazy_static::LazyStatic for NAME_TO_PROB_STR {
    fn initialize(lazy: &Self) {
        let _ = &**lazy;
    }
}

This works if I manually change the ::std's to ::alloc's. You use lazy_static which says that it is 'no_std' compatible, so it should be able to work with WASM.

Thanks for considering adding this capability.

• Carl
  p.s. I recently converted one of my non-macro crates to WASM and no_std compatibility and wrote a blog post about the process that you might find interesting. Thanks again for creating this crate. It did just want I needed.

no-std support

So we can implement lexer and parser automaton compression according to here