algorithmiaio / algorithmia-rust Goto Github PK

The current rust client has a useful .put_file() for the DataDir:

client.dir(some_remote_dir).put_file(some_local_file);

Which is super useful! however, when you have a particular remote filename that's different from the local file name, this the method isn't implemented:

client.file(some_remote_filename).put_file(some_local_file);

I propose that DataFile should have this method exposed to make rust life less painful!

According to Rust release milestone predictions, rustc-serialize is scheduled to be deprecated in next month's beta release of Rust.

Need to implement the multipart portion of API spec

when trying to download a file on a codex installation, I discovered that it was throwing the following exception:

couldn't download file from url: data://zeryx/binaries/ffmpeg-static.tar.gz 
401 Unauthorized

when talking with @pmcq it was discovered that on the api logs it was reported as:

 {"start_date":"2016-11-03T21:22:11.578Z","start_time":1478208131578,"end_time":1478208131581,"method":"GET","uri":"//v1/data/zeryx/bin/ffmpeg-static.tar.gz","status_code":401,"request_time":3,"ip":"156.34.17.89","user_agent":"algorithmia-rust/1.3.0 (Rust unknown","auth_user":"zeryx"}

2016-11-03T21:10:03.987 [application-akka.actor.default-dispatcher-48982] INFO  application : data_v0 GET error, no auth - path: zeryx/binaries/ffmpeg-static.tar.gz error: Authorizat
ionRequired(missing auth) request_headers: ArrayBuffer((User-Agent,algorithmia-rust/1.3.0 (Rust unknown), (Authorization,Simple <snip>), (Host,api.codexonazure.
com))

it looks like it's trying to call the data_v0 endpoint which is invalid on codex, breaking the rust client for all codex installations.

Hi,

It seems algorithmia-rust uses a version of the base64 library which is vulnerable to security issues. Please refer to https://github.com/RustSec/advisory-db/blob/master/Advisories.toml#L53 .

patch commit : marshallpierce/rust-base64@24ead98
patched version : [">= 0.5.2"]
used version : base64 = ^"0.3.0"

This is done using crates.io data. Feel free to close this issue if it's not relevant.

Thanks

When downloading a file using the client.file(..).get();... io::copy(..) operation, it occasionally fails to complete downloading the file in what seems like a broken pipe failure mode.

The output from .get() is a DataFile, which looks like a stream object, and the .get() is supposed to return a Result<DataFile>, however, no exception is ever thrown when said suspected broken pipe occurs, instead it hangs for indefinitely while keeping the httpClient connection open until manually killed.

https://gist.github.com/zeryx/3cdbe5c8b0d3da30041a1b0348f4cb83

note: The error is intermittent, and won't occur on every run; it seems to have a causal relationship with cpu load, which might hint towards the underlying cause.

There might be a more refined minimal way of defining a reproducible output, but I haven't reached that yet.
As you can see, the bizarre workaround is to split the client.file(..).get() operation into a two line client.file() and file.get().

The only hypothetical cause is a bug in the underlying http-client library reqwest, however that seems farfetched.

line 362 of dir.rs contains this line:
let mut file = File::open(path_ref).unwrap();
if path_ref doesn't refer to an existing file path, the unwrap fails. I propose this is replaced with:

let mut file try!(File::open(path_ref);

where try!() is a macro that is able to parse an io::Result()