axodotdev / axoasset

📮 load, write, and copy remote and local assets

axoasset

This library offers read, write, and copy functions, for local and remote assets given a string that contains a relative or absolute local path or a remote address using http or https.

Example

use axoasset;

let assets = vec!("https://my.co/logo.png", "./profile.jpg", "README.md");
let dest = "public";

for asset in assets {
    axoasset::copy(asset, dest)?;
}

License

Licensed under either of

• Apache License, Version 2.0, (LICENSE-APACHE or apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or opensource.org/licenses/MIT)

at your option.

Contributions

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

If you are interested in contributing, please read our CONTRIBUTING notes and our Code of Conduct.

Copyright 2022 Axo Developer Co.

axoasset's Issues

review diagnostic text, make sure it is agnostic

there's mentions of oranda in there, etc, see also #38 (comment)

update licensing

filenaming

per #2 (comment) via @Gankra

currently if fetching a remote asset where the origin path is a url whose final segment is not identifiable as a filename, we do the following transform:

https://mywebsite.com/some/path/image.png
/some/path/image.png
_some_path_image.png
some_path_image.png

for several reasons this is suboptimal and we should review this and how we name files in general.

add guardrails to specific apis/types to ensure paths are right type

now that the specific apis are available- we should add guardrails and type safety to ensure that folks have passed the path type they tell us they passed. the agnostic api is stringly typed to give input flexibility, but the specific APIs should use Path and URL

diagnostics should not refer to oranda or other oranda specific logic

unescape file names from URLs

See axodotdev/oranda#419 for details

Probably it's not as simple as blindly unescaping things, as we don't want to unescape a slash naively..?

non-image MIME/extension

Consider adding a method for creating and/or opening a file

We have this snippet:

https://github.com/axodotdev/axoasset/blob/main/src/compression.rs#L33

We could replace this with a method from LocalAsset that creates a file and then returns it.

Add tests for SourceFile

From #43.

Create destination directory for LocalAsset copy / write if it does not exist

When using Asset::copy("legit-src", "non-existant-destination") or asset.write("non-existant-destination"), the call will raise an error if the destination path does not exist for a LocalAsset.

Potential solution would be to check if the destination Path is a valid directory and create it (plus any parent directories) if not using something like fs::create_dir_all before the complete destination path (including filename) is built.
e.g.

let dest_path = Path::new(dest_dir);
if !dest_path.is_dir() {
    std::fs::create_dir_all(dest_path)?;
}

If this is intended behaviour then feel free to ignore this!

support receiving a batch of asset paths

currently the API receives a single origin_path for an asset and fetches it. we should also introduce an API that can receive multiple origin_paths at once, be it a path to a dir, a vector, etc. it's not clear to me if we should pick one way (an iter?) or support multiple ways, of doing this, but in general this logic would not add a ton of complexity/maintenance burden (which is also why it seems ok to leave it up to the user to iterate through assets...)

release 0.2.0

make RemoteAsset::copy work with .zips

axodotdev/cargo-dist#1036 (comment)

tests

add asset new

need a way to create an asset after loading so that it can be written

Support recursive copy of files

Right now axoasset only supports copying one asset at a time, it would be nice if it supported a recursive option where we could just pass a origin_folder and a dist_folder and it would copy the files and create the folders

example code used in fs_extra: https://docs.rs/fs_extra/latest/src/fs_extra/lib.rs.html#190-223

related axodotdev/oranda#82

wrap miette labels in Option

Right now we do .unwrap_or((0..0).into()) on some Option<SourceSpan>s (see the toml and json deserializers), but I'm pretty sure the Errors should just have an Option for those spans and miette will understand what to do then. Like 90% sure that just works but too tired to check right now (done for the day).

add image cases to tests

for the sake of getting the test PR (#2) over the finish line, i eliminated the image case from several of the tests. because images are a key asset in the pipelines that axoasset is used in, we should add image tests cases to these tests.

Clean up error handling in zip creation

We currently condense a whole lot of std::io::Errors into one handler, without being able to specify which file exactly threw the error. This is not good, and should be improved.

cc #38

type aliases for public api

per #2 (comment) via @Gankra

the api is rather stringly typed, for reasons i think are fair, but it would be useful for us internally to use type aliases to make clear what we're doing

improve identification of filename in last URL segment

currently we "identify" if the last segment of a URL for a remote asset is a filename by checking if it contains a . which is not a robust way to do so. this should be improved.

support YAML

now that oranda reads funding.yaml it'd be good to support it so we can have nice errors

add optional "root dir name" arg

This was introduced in cargo-dist to allow npm tarballs to be made with "package" as the root dir name: axodotdev/cargo-dist#246

Unfortunately it's impl'd inconsistently in cargo-dist because we rely on zips to have no root dir (emulating --strip-prefix=1). As such the axoasset impl should take an Option and emit a flat zip/tarball if it's None.

This is a blocker on merging axodotdev/cargo-dist#229

Support nested directory reads for zip file creation

From #38, when the zip file ReadDir instance comes along a directory, it currently panics due to a todo!. We should fix this sooner than later.

https://crates.io/crates/walkdir is pretty much the standard way of recursively walking a directory.

zstd is not actually zlib

axoasset/src/compression.rs

Lines 118 to 120 in 2285366

	CompressionImpl::Zstd => {
	// Wrap our file in compression
	let zip_output = ZlibEncoder::new(final_zip_file, Compression::default());

Using this (via cargo-dist's unix-archive = ".tar.zstd") produces archives that neither tar nor zstd can handle:

$ tar xf zarc-cli-x86_64-unknown-linux-gnu.tar.zstd
tar: This does not look like a tar archive
tar: Skipping to next header
tar: Exiting with failure status due to previous errors

$ zstd --test zarc-cli-x86_64-unknown-linux-gnu.tar.zstd
zstd: zarc-cli-x86_64-unknown-linux-gnu.tar.zstd: unsupported format

Because they are instead raw zlib encoded:

$ file zarc-cli-x86_64-unknown-linux-gnu.tar.zstd
zarc-cli-x86_64-unknown-linux-gnu.tar.zstd: zlib compressed data

$ zlib-flate -uncompress < zarc-cli-x86_64-unknown-linux-gnu.tar.zstd > zarc-cli.tar

$ file zarc-cli.tar
zarc-cli.tar: POSIX tar archive (GNU)

$ tar xvf zarc-cli.tar
zarc-cli-x86_64-unknown-linux-gnu/
zarc-cli-x86_64-unknown-linux-gnu/LICENSE-APACHE
zarc-cli-x86_64-unknown-linux-gnu/README.md
zarc-cli-x86_64-unknown-linux-gnu/LICENSE-MIT
zarc-cli-x86_64-unknown-linux-gnu/zarc
zarc-cli-x86_64-unknown-linux-gnu/CHANGELOG.md

cargo-dist version 0.8.0

documentation