chrismacnaughton / cargo-cacher

A caching server for crates + cargo

cargo-cacher

cargo-cacher is a caching server in the same spirit as apt-cacher-ng. The goal is to allow recursive caching of the canonical crates.io index.

Usage

To configure your system to use a copy of cargo-cacher, you need to setup a .cargo/config file in your project, or in a containing folder. The contents of that folder should look like:

[source]

[source.mirror]
registry = "http://localhost:8080/index"

[source.crates-io]
replace-with = "mirror"

Once this is in place, your builds will go through the local proxy, and the crates will be pulled down to the local filesystem when they are first requested. The path can be a remote host as long as the path is to /index. To run cargo-cacher, there are several arguments you probably want to use:

USAGE:
    cargo-cacher [FLAGS] [OPTIONS]

FLAGS:
    -a, --all        Prefetch entire Cargo index
    -d               Sets the level of debugging information
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -g <git>             Upstream git index (Default: https://github.com/rust-lang/crates.io-index.git)
    -i <index>           Path to store the indexes (git and fiels) at (Default: $HOME/.crates)
    -p <port>            Port to listen on (Default: 8080)
    -f <prefetch>        Path with a list of crate_name=version OR lock file to pre-fetch
    -r <refresh>         Refresh rate for the git index (Default: 600)
    -t <threads>         How many threads to use to fetch crates in the background
    -u <upstream>        Upstream Crate source (Default: https://crates.io/api/v1/crates/)

Full mirror

cargo-cacher supports providing a full mirror with the -a flag. Passing this flag will create a background thread pool on startup that will fetch all crates currently in the upstream index.

Prefetch

Prefetch is an option that I feel deserves further attention. Prefetch is a path to a file containing one line per crate/version, example:

log=0.3.6
libc=0.1.12

The above input will fetch log version 0.3.6 and libc version 0.1.12 before being requested by a user. This happens on a separate thread so the server can continue to start up without waiting on the pre-fetching to complete.

TODO

• Add expiration on background thread
• Add statistics

If you would like to support me:

• Gratipay
• Bountysource

cargo-cacher's Issues

`-p` option description looks suspicious

In README:

-p <port> Output file to put compiled crushmap into (Default: 8080)

Looks like it is a port to listen, not a file to output.

Add default thread number to command line help

pull request to follow...

Invalid Content-Type: application/

Thanks for the project, but having the following issues when running the cargo-cacher program.
My application can't seem to use the local cargo-cacher program.

$ cargo build

    Updating `http://localhost:8080/index` index
warning: spurious network error (2 tries remaining): invalid Content-Type: application/; class=Net (12)
warning: spurious network error (1 tries remaining): invalid Content-Type: application/; class=Net (12)
error: failed to query replaced source registry `https://github.com/rust-lang/crates.io-index`

Caused by:
  failed to fetch `http://localhost:8080/index`

Caused by:
  invalid Content-Type: application/; class=Net (12)

$ ./target/debug/cargo-cacher -f config -i crates -d -d -d -d

2020-02-28 11:07:41,905 INFO  [cargo_cacher] Configuration: Config { all: false, prefetch_path: Some("config"), index_path: "crates", crate_path: "crates/crates", git_index_path: "crates/index", upstream: "https://static.crates.io/crates/", index: "https://github.com/rust-lang/crates.io-index.git", extern_url: "http://localhost:8080", port: 8080, refresh_interval: 600s, threads: 16, log_level: Trace }
2020-02-28 11:07:41,905 DEBUG [cargo_cacher::index_sync] Syncing git repo at crates/index with https://github.com/rust-lang/crates.io-index.git, setting API url to http://localhost:8080
2020-02-28 11:07:41,905 DEBUG [cargo_cacher::index_sync] Repo path is "crates/index/.git"
2020-02-28 11:07:41,909 WARN  [cargo_cacher::index_sync] Command was not a success
2020-02-28 11:07:41,909 WARN  [cargo_cacher::index_sync] Failed to update index
2020-02-28 11:07:41,909 DEBUG [cargo_cacher::crates] Prefetching file at config!
2020-02-28 11:07:41,909 TRACE [cargo_cacher::crates] CoreFoundation-sys : "0.1.4" is already fetched
2020-02-28 11:07:41,909 TRACE [cargo_cacher::crates] IOKit-sys : "0.1.5" is already fetched
2020-02-28 11:07:41,909 TRACE [cargo_cacher::crates] aho-corasick : "0.7.7" is already fetched
2020-02-28 11:07:41,909 TRACE [cargo_cacher::crates] anyhow : "1.0.26" is already fetched
2020-02-28 11:07:41,909 TRACE [cargo_cacher::crates] arrayvec : "0.3.25" is already fetched
2020-02-28 11:07:41,909 TRACE [cargo_cacher::crates] bitflags : "1.2.1" is already fetched
2020-02-28 11:07:41,909 TRACE [cargo_cacher::crates] cfg-if : "0.1.10" is already fetched
2020-02-28 11:07:41,909 TRACE [cargo_cacher::crates] hexdump : "0.1.0" is already fetched
2020-02-28 11:07:41,909 TRACE [cargo_cacher::crates] itertools : "0.4.19" is already fetched
2020-02-28 11:07:41,909 TRACE [cargo_cacher::crates] lazy_static : "1.4.0" is already fetched
2020-02-28 11:07:41,909 TRACE [cargo_cacher::crates] libc : "0.2.66" is already fetched
2020-02-28 11:07:41,909 TRACE [cargo_cacher::crates] libudev : "0.2.0" is already fetched
2020-02-28 11:07:41,909 TRACE [cargo_cacher::crates] libudev-sys : "0.1.4" is already fetched
2020-02-28 11:07:41,909 TRACE [cargo_cacher::crates] mach : "0.1.2" is already fetched
2020-02-28 11:07:41,909 TRACE [cargo_cacher::crates] memchr : "2.3.0" is already fetched
2020-02-28 11:07:41,910 TRACE [cargo_cacher::crates] nix : "0.14.1" is already fetched
2020-02-28 11:07:41,910 TRACE [cargo_cacher::crates] nodrop : "0.1.14" is already fetched
Listening on :::8080
2020-02-28 11:07:41,910 TRACE [cargo_cacher::crates] odds : "0.2.26" is already fetched
2020-02-28 11:07:41,910 TRACE [cargo_cacher::crates] pkg-config : "0.3.17" is already fetched
2020-02-28 11:07:41,910 TRACE [cargo_cacher::crates] proc-macro2 : "1.0.8" is already fetched
2020-02-28 11:07:41,910 TRACE [cargo_cacher::crates] quote : "1.0.2" is already fetched
2020-02-28 11:07:41,910 DEBUG [hyper::server] threads = 96
2020-02-28 11:07:41,910 TRACE [cargo_cacher::crates] regex : "1.3.4" is already fetched
2020-02-28 11:07:41,910 TRACE [cargo_cacher::crates] regex-syntax : "0.6.14" is already fetched
2020-02-28 11:07:41,910 TRACE [cargo_cacher::crates] serde : "1.0.104" is already fetched
2020-02-28 11:07:41,910 TRACE [cargo_cacher::crates] serde_derive : "1.0.104" is already fetched
2020-02-28 11:07:41,910 TRACE [cargo_cacher::crates] serialport : "3.3.0" is already fetched
2020-02-28 11:07:41,910 TRACE [cargo_cacher::crates] serialport : "3.3.0" is already fetched
2020-02-28 11:07:41,910 TRACE [cargo_cacher::crates] syn : "1.0.14" is already fetched
2020-02-28 11:07:41,910 TRACE [cargo_cacher::crates] thread_local : "1.0.1" is already fetched
2020-02-28 11:07:41,910 TRACE [cargo_cacher::crates] toml : "0.5.6" is already fetched
2020-02-28 11:07:41,910 TRACE [cargo_cacher::crates] unicode-xid : "0.2.0" is already fetched
2020-02-28 11:07:41,910 TRACE [cargo_cacher::crates] void : "1.0.2" is already fetched
2020-02-28 11:07:41,910 TRACE [cargo_cacher::crates] winapi : "0.3.8" is already fetched
2020-02-28 11:07:41,910 TRACE [cargo_cacher::crates] winapi-i686-pc-windows-gnu : "0.4.0" is already fetched
2020-02-28 11:07:41,910 TRACE [cargo_cacher::crates] winapi-x86_64-pc-windows-gnu : "0.4.0" is already fetched
2020-02-28 11:02:41,031 DEBUG [hyper::server] Incoming stream
2020-02-28 11:02:41,031 TRACE [hyper::buffer] get_buf []
2020-02-28 11:02:41,031 TRACE [hyper::buffer] read_into_buf buf[0..4096]
2020-02-28 11:02:41,031 TRACE [hyper::buffer] get_buf [u8; 4096][0..130]
2020-02-28 11:02:41,031 TRACE [hyper::http::h1] try_parse([71, 69, 84, 32, 47, 105, 110, 100, 101, 120, 47, 105, 110, 102, 111, 47, 114, 101, 102, 115, 63
, 115, 101, 114, 118, 105, 99, 101, 61, 103, 105, 116, 45, 117, 112, 108, 111, 97, 100, 45, 112, 97, 99, 107, 32, 72, 84, 84, 80, 47, 49, 46, 49, 13, 10,
85, 115, 101, 114, 45, 65, 103, 101, 110, 116, 58, 32, 103, 105, 116, 47, 50, 46, 48, 32, 40, 108, 105, 98, 103, 105, 116, 50, 32, 48, 46, 50, 56, 46, 48,
 41, 13, 10, 72, 111, 115, 116, 58, 32, 108, 111, 99, 97, 108, 104, 111, 115, 116, 58, 56, 48, 56, 48, 13, 10, 65, 99, 99, 101, 112, 116, 58, 32, 42, 47,
42, 13, 10, 13, 10])
2020-02-28 11:02:41,031 TRACE [hyper::http::h1] Request.try_parse([Header; 100], [u8; 130])
2020-02-28 11:02:41,031 TRACE [hyper::http::h1] Request.try_parse Complete(130)
2020-02-28 11:02:41,032 TRACE [hyper::header] raw header: "User-Agent"=[103, 105, 116, 47, 50, 46, 48, 32, 40, 108, 105, 98, 103, 105, 116, 50, 32, 48, 46
, 50, 56, 46, 48, 41]
2020-02-28 11:02:41,032 TRACE [hyper::header] raw header: "Host"=[108, 111, 99, 97, 108, 104, 111, 115, 116, 58, 56, 48, 56, 48]
2020-02-28 11:02:41,032 TRACE [hyper::header] raw header: "Accept"=[42, 47, 42]
2020-02-28 11:02:41,032 DEBUG [hyper::server::request] Request Line: Get AbsolutePath("/index/info/refs?service=git-upload-pack") Http11
2020-02-28 11:02:41,032 DEBUG [hyper::server::request] Headers { User-Agent: git/2.0 (libgit2 0.28.0)
, Host: localhost:8080
, Accept: */*
, }
2020-02-28 11:02:41,032 TRACE [hyper::http] should_keep_alive( Http11, None )
2020-02-28 11:02:41,032 DEBUG [cargo_cacher::git] Raw GIT request: Request {
    url: Url { generic_url: "http://localhost:8080/index/info/refs?service=git-upload-pack" }
    method: Get
    remote_addr: V6([::1]:45300)
    local_addr: V6([::]:8080)
}
Not a git repository: 'crates/index'2020-02-28 11:02:41,035 TRACE [hyper::header] Headers.set( "Content-Type", ContentType(Mime(Application, Ext(""), []))
 )
2020-02-28 11:02:41,035 INFO  [logger] GET http://localhost:8080/index/info/refs?service=git-upload-pack 200 OK (2.911345 ms)
2020-02-28 11:02:41,035 TRACE [hyper::header] Headers.set( "Access-Control-Allow-Origin", Any )
2020-02-28 11:02:41,035 TRACE [hyper::header] Headers.set( "Content-Type", ContentType(Mime(Application, Ext(""), [])) )
2020-02-28 11:02:41,035 DEBUG [hyper::server::response] writing head: Http11 Ok
2020-02-28 11:02:41,035 TRACE [hyper::header] Headers.set( "Date", Date(HttpDate(Tm { tm_sec: 41, tm_min: 2, tm_hour: 16, tm_mday: 28, tm_mon: 1, tm_year:
 120, tm_wday: 5, tm_yday: 58, tm_isdst: 0, tm_utcoff: 0, tm_nsec: 35419769 })) )
2020-02-28 11:02:41,035 TRACE [hyper::header] Headers.set( "Transfer-Encoding", TransferEncoding([Chunked]) )
2020-02-28 11:02:41,035 DEBUG [hyper::server::response] headers [
Headers { Content-Type: application/
, Access-Control-Allow-Origin: *
, Date: Fri, 28 Feb 2020 16:02:41 GMT
, Transfer-Encoding: chunked
, }]
2020-02-28 11:02:41,035 TRACE [hyper::server::response] ending
2020-02-28 11:02:41,035 TRACE [hyper::http::h1] chunked write, size = 0
2020-02-28 11:02:41,035 TRACE [hyper::http] should_keep_alive( Http11, None )
2020-02-28 11:02:41,035 DEBUG [hyper::server] keep_alive = true for [::1]:45300
2020-02-28 11:02:41,035 TRACE [hyper::buffer] get_buf []
2020-02-28 11:02:41,035 TRACE [hyper::buffer] read_into_buf buf[0..4096]
2020-02-28 11:02:41,035 TRACE [hyper::buffer] get_buf []
2020-02-28 11:02:41,035 DEBUG [hyper::server] ioerror in keepalive loop = Custom { kind: UnexpectedEof, error: "end of stream before headers finished" }
2020-02-28 11:02:41,035 DEBUG [hyper::server] keep_alive loop ending for [::1]:45300
2020-02-28 11:02:41,035 DEBUG [hyper::server] Incoming stream
2020-02-28 11:02:41,035 TRACE [hyper::buffer] get_buf []
2020-02-28 11:02:41,035 TRACE [hyper::buffer] read_into_buf buf[0..4096]
2020-02-28 11:02:41,035 TRACE [hyper::buffer] get_buf [u8; 4096][0..130]
2020-02-28 11:02:41,036 TRACE [hyper::http::h1] try_parse([71, 69, 84, 32, 47, 105, 110, 100, 101, 120, 47, 105, 110, 102, 111, 47, 114, 101, 102, 115, 63
, 115, 101, 114, 118, 105, 99, 101, 61, 103, 105, 116, 45, 117, 112, 108, 111, 97, 100, 45, 112, 97, 99, 107, 32, 72, 84, 84, 80, 47, 49, 46, 49, 13, 10,
85, 115, 101, 114, 45, 65, 103, 101, 110, 116, 58, 32, 103, 105, 116, 47, 50, 46, 48, 32, 40, 108, 105, 98, 103, 105, 116, 50, 32, 48, 46, 50, 56, 46, 48,
 41, 13, 10, 72, 111, 115, 116, 58, 32, 108, 111, 99, 97, 108, 104, 111, 115, 116, 58, 56, 48, 56, 48, 13, 10, 65, 99, 99, 101, 112, 116, 58, 32, 42, 47,
42, 13, 10, 13, 10])
2020-02-28 11:02:41,036 TRACE [hyper::http::h1] Request.try_parse([Header; 100], [u8; 130])
2020-02-28 11:02:41,036 TRACE [hyper::http::h1] Request.try_parse Complete(130)
2020-02-28 11:02:41,036 TRACE [hyper::header] raw header: "User-Agent"=[103, 105, 116, 47, 50, 46, 48, 32, 40, 108, 105, 98, 103, 105, 116, 50, 32, 48, 46
, 50, 56, 46, 48, 41]
2020-02-28 11:02:41,036 TRACE [hyper::header] raw header: "Host"=[108, 111, 99, 97, 108, 104, 111, 115, 116, 58, 56, 48, 56, 48]
2020-02-28 11:02:41,036 TRACE [hyper::header] raw header: "Accept"=[42, 47, 42]
2020-02-28 11:02:41,036 DEBUG [hyper::server::request] Request Line: Get AbsolutePath("/index/info/refs?service=git-upload-pack") Http11
2020-02-28 11:02:41,036 DEBUG [hyper::server::request] Headers { User-Agent: git/2.0 (libgit2 0.28.0)
, Host: localhost:8080
, Accept: */*
, }
2020-02-28 11:02:41,036 TRACE [hyper::http] should_keep_alive( Http11, None )
2020-02-28 11:02:41,036 DEBUG [cargo_cacher::git] Raw GIT request: Request {
    url: Url { generic_url: "http://localhost:8080/index/info/refs?service=git-upload-pack" }
    method: Get
    remote_addr: V6([::1]:45302)
    local_addr: V6([::]:8080)
}
Not a git repository: 'crates/index'2020-02-28 11:02:41,039 TRACE [hyper::header] Headers.set( "Content-Type", ContentType(Mime(Application, Ext(""), []))
 )
2020-02-28 11:02:41,039 INFO  [logger] GET http://localhost:8080/index/info/refs?service=git-upload-pack 200 OK (3.089164 ms)
2020-02-28 11:02:41,039 TRACE [hyper::header] Headers.set( "Access-Control-Allow-Origin", Any )
2020-02-28 11:02:41,039 TRACE [hyper::header] Headers.set( "Content-Type", ContentType(Mime(Application, Ext(""), [])) )
2020-02-28 11:02:41,039 DEBUG [hyper::server::response] writing head: Http11 Ok
2020-02-28 11:02:41,039 TRACE [hyper::header] Headers.set( "Date", Date(HttpDate(Tm { tm_sec: 41, tm_min: 2, tm_hour: 16, tm_mday: 28, tm_mon: 1, tm_year:
 120, tm_wday: 5, tm_yday: 58, tm_isdst: 0, tm_utcoff: 0, tm_nsec: 39820490 })) )
2020-02-28 11:02:41,039 TRACE [hyper::header] Headers.set( "Transfer-Encoding", TransferEncoding([Chunked]) )
2020-02-28 11:02:41,039 DEBUG [hyper::server::response] headers [
Headers { Content-Type: application/
, Access-Control-Allow-Origin: *
, Date: Fri, 28 Feb 2020 16:02:41 GMT
, Transfer-Encoding: chunked
, }]
2020-02-28 11:02:41,039 TRACE [hyper::server::response] ending
2020-02-28 11:02:41,039 TRACE [hyper::http::h1] chunked write, size = 0
2020-02-28 11:02:41,040 TRACE [hyper::http] should_keep_alive( Http11, None )
2020-02-28 11:02:41,040 DEBUG [hyper::server] keep_alive = true for [::1]:45302
2020-02-28 11:02:41,040 TRACE [hyper::buffer] get_buf []
2020-02-28 11:02:41,040 TRACE [hyper::buffer] read_into_buf buf[0..4096]
2020-02-28 11:02:41,040 TRACE [hyper::buffer] get_buf []
2020-02-28 11:02:41,040 DEBUG [hyper::server] ioerror in keepalive loop = Custom { kind: UnexpectedEof, error: "end of stream before headers finished" }
2020-02-28 11:02:41,040 DEBUG [hyper::server] keep_alive loop ending for [::1]:45302
2020-02-28 11:02:41,040 DEBUG [hyper::server] Incoming stream
2020-02-28 11:02:41,040 TRACE [hyper::buffer] get_buf []
2020-02-28 11:02:41,040 TRACE [hyper::buffer] read_into_buf buf[0..4096]
2020-02-28 11:02:41,040 TRACE [hyper::buffer] get_buf [u8; 4096][0..130]
2020-02-28 11:02:41,040 TRACE [hyper::http::h1] try_parse([71, 69, 84, 32, 47, 105, 110, 100, 101, 120, 47, 105, 110, 102, 111, 47, 114, 101, 102, 115, 63
, 115, 101, 114, 118, 105, 99, 101, 61, 103, 105, 116, 45, 117, 112, 108, 111, 97, 100, 45, 112, 97, 99, 107, 32, 72, 84, 84, 80, 47, 49, 46, 49, 13, 10,
85, 115, 101, 114, 45, 65, 103, 101, 110, 116, 58, 32, 103, 105, 116, 47, 50, 46, 48, 32, 40, 108, 105, 98, 103, 105, 116, 50, 32, 48, 46, 50, 56, 46, 48,
 41, 13, 10, 72, 111, 115, 116, 58, 32, 108, 111, 99, 97, 108, 104, 111, 115, 116, 58, 56, 48, 56, 48, 13, 10, 65, 99, 99, 101, 112, 116, 58, 32, 42, 47,
42, 13, 10, 13, 10])
2020-02-28 11:02:41,040 TRACE [hyper::http::h1] Request.try_parse([Header; 100], [u8; 130])
2020-02-28 11:02:41,040 TRACE [hyper::http::h1] Request.try_parse Complete(130)
2020-02-28 11:02:41,040 TRACE [hyper::header] raw header: "User-Agent"=[103, 105, 116, 47, 50, 46, 48, 32, 40, 108, 105, 98, 103, 105, 116, 50, 32, 48, 46
, 50, 56, 46, 48, 41]
2020-02-28 11:02:41,040 TRACE [hyper::header] raw header: "Host"=[108, 111, 99, 97, 108, 104, 111, 115, 116, 58, 56, 48, 56, 48]
2020-02-28 11:02:41,040 TRACE [hyper::header] raw header: "Accept"=[42, 47, 42]
2020-02-28 11:02:41,040 DEBUG [hyper::server::request] Request Line: Get AbsolutePath("/index/info/refs?service=git-upload-pack") Http11
2020-02-28 11:02:41,040 DEBUG [hyper::server::request] Headers { User-Agent: git/2.0 (libgit2 0.28.0)
, Host: localhost:8080
, Accept: */*
, }
2020-02-28 11:02:41,040 TRACE [hyper::http] should_keep_alive( Http11, None )
2020-02-28 11:02:41,040 DEBUG [cargo_cacher::git] Raw GIT request: Request {
    url: Url { generic_url: "http://localhost:8080/index/info/refs?service=git-upload-pack" }
    method: Get
    remote_addr: V6([::1]:45304)
    local_addr: V6([::]:8080)
}
Not a git repository: 'crates/index'2020-02-28 11:02:41,043 TRACE [hyper::header] Headers.set( "Content-Type", ContentType(Mime(Application, Ext(""), []))
 )
 )
2020-02-28 11:02:41,043 INFO  [logger] GET http://localhost:8080/index/info/refs?service=git-upload-pack 200 OK (3.1143 ms)
2020-02-28 11:02:41,043 TRACE [hyper::header] Headers.set( "Access-Control-Allow-Origin", Any )
2020-02-28 11:02:41,043 TRACE [hyper::header] Headers.set( "Content-Type", ContentType(Mime(Application, Ext(""), [])) )
2020-02-28 11:02:41,044 DEBUG [hyper::server::response] writing head: Http11 Ok
2020-02-28 11:02:41,044 TRACE [hyper::header] Headers.set( "Date", Date(HttpDate(Tm { tm_sec: 41, tm_min: 2, tm_hour: 16, tm_mday: 28, tm_mon: 1, tm_year: 120, tm_wday: 5, tm_yday: 58, tm_isdst: 0, tm_utcoff: 0, tm_nsec: 44058283 })) )
2020-02-28 11:02:41,044 TRACE [hyper::header] Headers.set( "Transfer-Encoding", TransferEncoding([Chunked]) )
2020-02-28 11:02:41,044 DEBUG [hyper::server::response] headers [
Headers { Content-Type: application/
, Access-Control-Allow-Origin: *
, Date: Fri, 28 Feb 2020 16:02:41 GMT
, Transfer-Encoding: chunked
, }]
2020-02-28 11:02:41,044 TRACE [hyper::server::response] ending
2020-02-28 11:02:41,044 TRACE [hyper::http::h1] chunked write, size = 0
2020-02-28 11:02:41,044 TRACE [hyper::http] should_keep_alive( Http11, None )
2020-02-28 11:02:41,044 DEBUG [hyper::server] keep_alive = true for [::1]:45304
2020-02-28 11:02:41,044 TRACE [hyper::buffer] get_buf []
2020-02-28 11:02:41,044 TRACE [hyper::buffer] read_into_buf buf[0..4096]
2020-02-28 11:02:41,044 TRACE [hyper::buffer] get_buf []
2020-02-28 11:02:41,044 DEBUG [hyper::server] ioerror in keepalive loop = Custom { kind: UnexpectedEof, error: "end of stream before headers finished" }
2020-02-28 11:02:41,044 DEBUG [hyper::server] keep_alive loop ending for [::1]:45304
2020-02-28 11:05:47,831 DEBUG [cargo_cacher::index_sync] Syncing git repo at crates/index with https://github.com/rust-lang/crates.io-index.git, setting API url to http://localhost:8080
2020-02-28 11:05:47,831 DEBUG [cargo_cacher::index_sync] Repo path is "crates/index/.git"
2020-02-28 11:05:47,836 WARN  [cargo_cacher::index_sync] Command was not a success
2020-02-28 11:05:47,836 WARN  [cargo_cacher::index_sync] Failed to update index

when run with "-d" option, triggers "Could not determine the UTC offset on this system" error

Steps to reproduce:
install and run as in docs, but with "-d" flag, and then try to update index from a project actually using the cached mirror.
On my machine it resulted in "Could not determine the UTC offset on this system"

thread '<unnamed>' panicked at 'Could not determine the UTC offset on this system. Possible causes are that the time crate does not implement "local_offset_at" on your system, or that you are running in a multi-threaded environment and the time crate is returning "None" from "local_offset_at" to avoid unsafe behaviour. See the time crate's documentation for more information. (https://time-rs.github.io/internal-api/time/index.html#feature-flags): IndeterminateOffset', /home/andrzej/.cargo/registry/src/github.com-1ecc6299db9ec823/simple_logger-1.16.0/src/lib.rs:409:85
stack backtrace:
   0: rust_begin_unwind
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/panicking.rs:142:14
   2: core::result::unwrap_failed
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/result.rs:1805:5
   3: <simple_logger::SimpleLogger as log::Log>::log
   4: log::__log
   5: logger::Logger::log
   6: <logger::Logger as iron::middleware::AfterMiddleware>::after
   7: iron::middleware::Chain::continue_from_before
   8: <iron::middleware::Chain as iron::middleware::Handler>::handle
   9: <iron::iron::RawHandler<H> as hyper::server::Handler>::handle
  10: hyper::server::Worker<H>::handle_connection
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

I guess an update to logger or simplifying output format might help

If the initial clone fails, cargo-cacher will never recover on its own.

You end up with this,

$ tree ~/.crates/index
~/.crates/index
└── config.json

so the folder exists, but .git doesn't, so the clone will keep failing.

I feel a slight nudge towards rewriting index_sync.rs using the git2 crate, but git2 doesn't exactly provide an interface that is neat when used in rust, so hmm…

After hour run of cacher zombie processes appears

I'm running cacher and after hour of run, zombie processes of git start to appear

  216 ?        Ssl    0:00 /opt/cargo-cacher/cargo-cacher -e http://cargo-kacer.gcx.com -u https://crates.io/api/v1/crates/
  426 ?        Z      0:00  \_ [git] <defunct>
  430 ?        Z      0:00  \_ [git] <defunct>

Is cli port option redundant?

As a new user it was confusing that you can set the port in the external URL and a separate port option.
Would it be less confusing to remove the port option in favour of extracting the port from the external URL via rust's URL crate?

https://docs.rs/url/2.1.1/url/

DNS doesn't work in registry URL - and I'm really sad because of that :(

Hello Chris
Hope you're doing well
In case you ask me - I don't feel good like you

...That's because your great cargo-cacher doesn't work when I point to http://localhost:8080/index
However, It works when referring to http://127.0.0.1:8080/index

I get the following error

error:  Downloading crates ...
warning: spurious network error (2 tries remaining): [7] Couldn't connect to server (Failed to connect to localhost port 8080 after 1 ms: Couldn't connect to server)

But I do however want to put this application on a remote server and have a DNS to cargo-cacher.local

I also tried to build other people's forks of this repo and use it, but the result is the same

Here's also full .cargo/config.toml: (even though It's the same as in your example)

[source]

[source.mirror]
registry = "http://localhost:8080/index"

[source.crates-io]
replace-with = "mirror"

I'll be commenting on this issue If I discover anything

Config.json fails to commit/rebase

Everytime I start cargo-cacher I end up with config.json being modified but not committed to the index repo, this then in turn results in crates not being written to the crates directory. I resolved the issue by manually committing config.json to the index repo.

Maybe on startup we should always check to make sure config.json is committed and if not commit it.

WARN [cargo_cacher::index_sync] Failed to update index

Hello, could you help me pls?

what is it means? WARN [cargo_cacher::index_sync] Failed to update index
how to fix this WARN?

how I run cargo-cacher: cargo-cacher -i /opt/data

Access Denied

I get AccessDenied for every crate. I compared with cargo-download: Cargo-cache and cargo-download use the same URLs. cargo-download is working fine. I also read that the people running crates.io will never block mirroring tools, the are ok with mirroring people mirroring crates.io.

$ find -type f | wc -l; ag -l AccessDenied | wc -l
362
362

I use revision 8f07ed7. Is anybody else experiencing this?