cybozu-go / aptutil Goto Github PK
View Code? Open in Web Editor NEWGo utilities for Debian APT repositories
License: MIT License
Go utilities for Debian APT repositories
License: MIT License
Can you explain how to build this more clearly? I've never used go before and the conventions are weird. I tried this
mkdir -p ~/gocode/bin
mkdir ~/gocode/src
cd ~/gocode/src
go get -u github.com/cybozu-go/aptutil
go install -v github.com/cybozu-go/aptutil
and nothing happens. I tried
go build -v github.com/cybozu-go/aptutil/cacher
and got a face full of errors about missing dependencies.
It would be nice to be able to set the go-apt-cacher listen address in the .toml file instead of with e.g. -s :9999
on the command line. I can't see this documented anywhere. Is it currently possible, or is new code required?
(I don't think we need to support Windows. This issue is just for the record.)
> go get -u github.com/cybozu-go/aptutil/...
# github.com/cybozu-go/aptutil/mirror
Desktop\Go\src\github.com\cybozu-go\aptutil\mirror\flock.go:17: undefined: syscall.Flock
Desktop\Go\src\github.com\cybozu-go\aptutil\mirror\flock.go:17: undefined: syscall.LOCK_EX
Desktop\Go\src\github.com\cybozu-go\aptutil\mirror\flock.go:17: undefined: syscall.LOCK_NB
Desktop\Go\src\github.com\cybozu-go\aptutil\mirror\flock.go:23: undefined: syscall.Flock
Desktop\Go\src\github.com\cybozu-go\aptutil\mirror\flock.go:23: undefined: syscall.LOCK_UN
For Concourse, I'm currently implementing a new apt-package
resource type that can easily monitor the availability of new versions of a certain package. Neither the existing aptly-cli
nor the debian-sources
resource type fulfils my needs.
This repository contains 90% of the functionality to support me in implementing this, weren't it for the fact that a number of functions in the mirror
package are not exported. E.g. I would like to use functions downloadRelease
and downloadIndices
, possibly some more.
I'm not in favor of duplicating code or maintaining my own fork. Are you open to a PR that contains the changes I need to get my Concourse resource implemented?
Hi,
I just found an interesting failure in go-apt-cacher (I'm running the latest master/HEAD version).
Using the default config file (which maps ubuntu
to http://archive.ubuntu.com/ubuntu
), the server panics when querying http://go-apt-cacher:3142/ubuntu/
$ go-apt-cacher --loglevel debug
2017-12-19T13:19:32.594886Z 1eec099b553c go-apt-cacher debug: "request path" path="ubuntu"
panic: path must has a prefix: ubuntu
goroutine 33 [running]:
github.com/cybozu-go/aptutil/cacher.(*Cacher).download(0xc42017a240, 0x89dcc0, 0xc420420120, 0xc42043c005, 0x6, 0xc42018e200, 0x0)
/go/src/github.com/cybozu-go/aptutil/cacher/cacher.go:338 +0x142a
github.com/cybozu-go/aptutil/cacher.(*Cacher).Download.func1(0x89dc40, 0xc420400100, 0x89dc40, 0xc420400100)
/go/src/github.com/cybozu-go/aptutil/cacher/cacher.go:256 +0x62
github.com/cybozu-go/cmd.(*Environment).Go.func1(0xc4200982a0, 0xc42041a060)
/go/src/github.com/cybozu-go/cmd/env.go:133 +0x86
created by github.com/cybozu-go/cmd.(*Environment).Go
/go/src/github.com/cybozu-go/cmd/env.go:130 +0xaf
caused by a simple:
curl -i 172.17.0.2:3142/ubuntu/
The panic is caused by the following statement in cacher/cacher.go:L338
t := strings.SplitN(path.Clean(p), "/", 2)
if len(t) != 2 {
panic("path must has a prefix: " + p)
}
I don't think its adequate behavior to kill the server just because of this. go-apt-cacher should either serve the index.html of http://archive.ubuntu.com/ubuntu/ or just reply with 404.
Some repositories are badly configured to return wrong contents for uncompressed indices.
For instance, pkg.jenkins.io returned wrong Packages
as pointed by @takumin:
#25 (comment)
Zabbix also returned wrong Sources
for which go-apt-mirror
implements a workaround already:
Lines 243 to 257 in 66c646f
Although these indices are bad, apt
or apt-get
will work nicely as they look compressed indices
such as Packages.bz2
or Sources.gz
first.
To help users of go-apt-mirror
, we should provide a general workaround for this problem.
Readme says: "Pre-built binaries are available on releases.". But they are missing for 1.4.1 and 1.4.2.
In addition: This is software for caching/mirroring Debian packages. Shouldn't it be shipped as Debian packages itself (aka "Eat your own dogfood")?
I was using apt-cacher-ng but like you, I found it was unstable and I had intermittent issues. I'm trying to use go-apt-cacher as a satellite cache server. Our main repository is hosted with aptly.
I commented the existing mapping found in your sample config file and added mine:
aptly = "https://aptly.example.com/ubuntu"
The cacher is working fine. The first time a client ask for a package, it downloads it from the main repository and cache it locally. When another client ask for the same file again, the cacher get it from the local cache and doesn't get it from the central repository. All fine!
Problem is when I stop the cacher and try to start again. I get this error:
2020-04-07T20:28:26.549038Z deploiement-ext go-apt-cacher error: "ExtractFileInfo(aptly/dists/focal/main/binary-amd64/Packages.gz): parser.Read: bufio.Scanner: token too long"
The only thing I found to "fix" that is to remove the "meta" directory content. But by doing that, this seem to invalidate my local cache.
Currently, go-apt-cacher
uses http.DefaultTransport
which looks for proxy configurations from environment variables.
It would be more handy to be able to specify proxies in TOML.
I have been looking into writing a systemd unit file for this. One thing that came up was reloading the configuration file after it has been modified. USAGE.md
says "go-apt-cacher reads a configuration file at start up."
My question is: after a change to the configuration file, does go-apt-cacher notice the file has been updated and automatically reread it?
Or does one need to restart the service, ie kill the existing process and start it again?
Line 280 in ec162e2
go-apt-mirror keeps downloaded data in memory.
This may result in excessive memory usage while downloading debs.
go-apt-cacher has a similar issue.
I have successfully mirrored a Ubuntu repository, I'm not clear from the documentation if it is possible to have the go-apt-cacher mounted in such a way so that the cacher has the local mirrored repository available to it.
How is a client to be setup to use both? I guess you would change the source.list to point to the mirrored repository url that is being served with nginx? Then you would have the client http proxy setup to point to the go-apt-cacher url. How do you avoid the go-apt-cacher from caching the mirrored data?
Thanks for any clarification you can provide.
The default is 15 seconds. My concern is, if this utility comes into widespread use, won't it put a large load on the upstream servers? The whole point of having an apt-cache is to reduce the load on the upstream servers.
Can you say what the rationale for 15s was? What would be the downsides of setting this to, say 3601s?
caching repos like Kali Linux's (http://kali.download/kali) works just for the first run: after a few apt-get install
s pointing to go-apt-cacher, stopping and restarting the service results in an endless loop of this message:
SVC:/var/log/supervisord# tail -f go-apt-cacher*
==> go-apt-cacher1.log <==
2020-05-07T11:27:35.101356Z SVC go-apt-cacher warning: "well: got signal" signal="terminated"
2020-05-07T11:27:36.961230Z SVC go-apt-cacher error: "ExtractFileInfo(kali/dists/kali-rolling/main/binary-arm64/Packages.gz): parser.Read: bufio.Scanner: token too long"
2020-05-07T11:27:50.858243Z SVC go-apt-cacher error: "ExtractFileInfo(kali/dists/kali-rolling/main/binary-arm64/Packages.gz): parser.Read: bufio.Scanner: token too long"
2020-05-07T11:27:54.403333Z SVC go-apt-cacher error: "ExtractFileInfo(kali/dists/kali-rolling/main/binary-arm64/Packages.gz): parser.Read: bufio.Scanner: token too long"
2020-05-07T11:27:57.930828Z SVC go-apt-cacher error: "ExtractFileInfo(kali/dists/kali-rolling/main/binary-arm64/Packages.gz): parser.Read: bufio.Scanner: token too long"
2020-05-07T11:28:01.439917Z SVC go-apt-cacher error: "ExtractFileInfo(kali/dists/kali-rolling/main/binary-arm64/Packages.gz): parser.Read: bufio.Scanner: token too long"
2020-05-07T11:28:04.932175Z SVC go-apt-cacher error: "ExtractFileInfo(kali/dists/kali-rolling/main/binary-arm64/Packages.gz): parser.Read: bufio.Scanner: token too long"
2020-05-07T11:28:08.437772Z SVC go-apt-cacher error: "ExtractFileInfo(kali/dists/kali-rolling/main/binary-arm64/Packages.gz): parser.Read: bufio.Scanner: token too long"
2020-05-07T11:28:11.930364Z SVC go-apt-cacher error: "ExtractFileInfo(kali/dists/kali-rolling/main/binary-arm64/Packages.gz): parser.Read: bufio.Scanner: token too long"
==> go-apt-cacher2.log <==
go-apt-mirror at its first stage downloads Release
and InRelease
indices.
Ref: https://wiki.debian.org/DebianRepository/Format#A.22Release.22_files
These should contain the same checksums for all other indices, but in reality
they may contain different checksums for some indices temporarily.
If Release
and InRelease
have different checksums for some indices,
current go-apt-mirror may overlook them and create incomplete mirror.
This bug may be fixed by detecting mismatches or by implementing
by-hash
indices acquisition.
Only directories that have been linked to the repo-id are cleaned up.
Randomly
Own test systems
go-apt-mirror: v1.4.2
Not really clear why a sync would fail to an apt repo, but the causes could be unreliable IP connectivity or incorrectly configured upstream apt repos.
Start synchronizing an apt repository to which there is bad IP connectivity.
Should the synchronization fail, a message is displayed (it is) and the incorrectly downloaded apt repository is deleted from the disk.
The incorrectly and incompletely downloaded apt directories are laying around and slowly taking up disk space, no mechanism takes care of cleaning them.
Delete incomplete downloaded apt directories.
Implement a mechanism into the code that would keep a number of [configurable] temporary downloaded directories, and start wiping the older ones.
This tools is quite friendly when synchronizing mirrors, I've hit my head for a few days in regards to synchronizing mirrors, Icinga being amongst them.
From the debugging I've done, the common denominator is how aptly does not provide an uncompressed Contents- file, such as Contents-amd64. You can download the file without the file extension as the Release says is available, the Header "Content-Location" is returned with the filename you are really getting (filename.gz), which the net/http library might not handle that well.
Example:
As a consequence when trying to get https://packages.icinga.com/debian/dists/icinga-buster/main/Contents-amd64, you get served https://packages.icinga.com/debian/dists/icinga-buster/main/Contents-amd64.gz while the function checks against the file without compression. If the function checks in case it fails against a .gz entry for the same filename, it would work.
Toml to replicate the error:
dir = "/var/spool/go-apt-mirror"
max_conns = 10
[log]
level = "debug"
format = "plain"
[mirror.icinga]
url = "https://packages.icinga.com/debian"
suites = ["icinga-buster"]
sections = ["main"]
mirror_source = false
architectures = ["amd64"]
Some APT repositories have a Release file that contains wrong checksums of Release itself.
As go-apt-cacher fails to handle such repositories, such checksums should be ignored.
With 50appstream in apt.conf.d, the following errors occur (example):
Err:10 http://apt-cacher.foo.org:3141/ubuntu bionic/main DEP-11 48x48 Icons
File has unexpected size (162304 != 118319). Mirror sync in progress? [IP: 10.1.2.3 3141]
Hashes of expected file:
- Filesize:118319 [weak]
- SHA256:72a445c7d354db04d1ffca6f52f7ab66953a3fa889bb66bf6d3210d3ab100f14
- SHA1:176f580a423a2fcf3e257f0d9a00713e05545a35 [weak]
- MD5Sum:bdfd4c6a690b2f9d0790443956b327fd [weak]
Release file created at: Thu, 26 Apr 2018 23:37:48 +0000
Going directly to the mirror set in the config toml, everything works fine.
Full outputs:
apt-get-update.notok.txt
go-apt-cacher.si-ubuntu.notok.log
apt-get-update.ok.txt
Configs:
go-apt-cacher.toml.txt
50appstream.txt
Tested on Ubuntu 18.04 desktop live iso. (same on 16.04/18.04 desktop if installed with 50appstream)
Downloding large package cause go-apt-cacher with "out of memory" error. By execute apt-get download texlive-latex-extra-doc texlive-pstricks-doc
Package: texlive-latex-extra-doc
Download-Size: 384 MBPackage: texlive-pstricks-doc
Download-Size: 248 MB
On 1GB mem VM. Is this intended?
Log attached.
fatal error: runtime: out of memory
runtime stack:
runtime.throw(0x6f8ba7, 0x16)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/panic.go:566 +0x95
runtime.sysMap(0xc4427b0000, 0x20000000, 0x0, 0x8621b8)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/mem_linux.go:219 +0x1d0
runtime.(*mheap).sysAlloc(0x848be0, 0x20000000, 0x0)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/malloc.go:407 +0x37a
runtime.(*mheap).grow(0x848be0, 0x10000, 0x0)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/mheap.go:726 +0x62
runtime.(*mheap).allocSpanLocked(0x848be0, 0x10000, 0x42f87f)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/mheap.go:630 +0x4f2
runtime.(*mheap).alloc_m(0x848be0, 0x10000, 0x100000000, 0xc42014bec8)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/mheap.go:515 +0xe0
runtime.(*mheap).alloc.func1()
/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/mheap.go:579 +0x4b
runtime.systemstack(0xc42014bed0)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/asm_amd64.s:314 +0xab
runtime.(*mheap).alloc(0x848be0, 0x10000, 0x10100000000, 0xc421598990)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/mheap.go:580 +0x73
runtime.largeAlloc(0x1ffffe00, 0xc421598901, 0xc420147990)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/malloc.go:774 +0x93
runtime.mallocgc.func1()
/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/malloc.go:669 +0x3e
runtime.systemstack(0xc420016000)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/asm_amd64.s:298 +0x79
runtime.mstart()
/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/proc.go:1079
goroutine 18 [running]:
runtime.systemstack_switch()
/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/asm_amd64.s:252 fp=0xc420147898 sp=0xc420147890
runtime.mallocgc(0x1ffffe00, 0x68d040, 0xc420147901, 0x44fe90)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/malloc.go:670 +0x903 fp=0xc420147938 sp=0xc420147898
runtime.makeslice(0x68d040, 0x1ffffe00, 0x1ffffe00, 0x0, 0x0, 0xc421e88c10)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/slice.go:57 +0x7b fp=0xc420147990 sp=0xc420147938
bytes.makeSlice(0x1ffffe00, 0x0, 0x0, 0x0)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/bytes/buffer.go:198 +0x77 fp=0xc4201479d0 sp=0xc420147990
bytes.(*Buffer).ReadFrom(0xc420147a70, 0x824040, 0xc421e88c00, 0xc420078a00, 0x0, 0x200)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/bytes/buffer.go:170 +0x2a9 fp=0xc420147a40 sp=0xc4201479d0
io/ioutil.readAll(0x824040, 0xc421e88c00, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/io/ioutil/ioutil.go:33 +0x147 fp=0xc420147af0 sp=0xc420147a40
io/ioutil.ReadAll(0x824040, 0xc421e88c00, 0xc421e88c00, 0x824040, 0xc421e88c00, 0xc42203c0d0, 0xc421fd8ba0)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/io/ioutil/ioutil.go:42 +0x3e fp=0xc420147b40 sp=0xc420147af0
github.com/cybozu-go/aptutil/cacher.(*Cacher).download(0xc42006c360, 0x8286c0, 0xc421a878c0, 0xc420270245, 0x50, 0xc42276e200, 0xc420a81320)
/home/travis/gopath/src/github.com/cybozu-go/aptutil/cacher/cacher.go:310 +0x295 fp=0xc420147f00 sp=0xc420147b40
github.com/cybozu-go/aptutil/cacher.(*Cacher).Download.func1(0x828640, 0xc421e88480, 0x828640, 0xc421e88480)
/home/travis/gopath/src/github.com/cybozu-go/aptutil/cacher/cacher.go:256 +0x62 fp=0xc420147f48 sp=0xc420147f00
github.com/cybozu-go/cmd.(*Environment).Go.func1(0xc420040310, 0xc421598420)
/home/travis/gopath/src/github.com/cybozu-go/cmd/env.go:133 +0x86 fp=0xc420147f90 sp=0xc420147f48
runtime.goexit()
/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420147f98 sp=0xc420147f90
created by github.com/cybozu-go/cmd.(*Environment).Go
/home/travis/gopath/src/github.com/cybozu-go/cmd/env.go:138 +0xaf
goroutine 1 [chan receive]:
github.com/cybozu-go/cmd.(*Environment).Wait(0xc420040310, 0x0, 0x0)
/home/travis/gopath/src/github.com/cybozu-go/cmd/env.go:95 +0x5a
github.com/cybozu-go/cmd.Wait(0xc4204ad180, 0x0)
/home/travis/gopath/src/github.com/cybozu-go/cmd/default.go:47 +0x2d
main.main()
/home/travis/gopath/src/github.com/cybozu-go/aptutil/cmd/go-apt-cacher/main.go:52 +0x30f
goroutine 5 [syscall]:
os/signal.signal_recv(0x0)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/sigqueue.go:116 +0x157
os/signal.loop()
/home/travis/.gimme/versions/go1.7.linux.amd64/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
/home/travis/.gimme/versions/go1.7.linux.amd64/src/os/signal/signal_unix.go:28 +0x41
goroutine 6 [select, locked to thread]:
runtime.gopark(0x71ada8, 0x0, 0x6f376f, 0x6, 0x18, 0x2)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/proc.go:259 +0x13a
runtime.selectgoImpl(0xc420020730, 0x0, 0x18)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/select.go:423 +0x11d9
runtime.selectgo(0xc420020730)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/select.go:238 +0x1c
runtime.ensureSigM.func1()
/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/signal1_unix.go:304 +0x2f3
runtime.goexit()
/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/asm_amd64.s:2086 +0x1
goroutine 7 [chan receive]:
github.com/cybozu-go/cmd.handleSignal.func1(0xc4200123c0, 0xc420040310)
/home/travis/gopath/src/github.com/cybozu-go/cmd/signal.go:27 +0x5a
created by github.com/cybozu-go/cmd.handleSignal
/home/travis/gopath/src/github.com/cybozu-go/cmd/signal.go:32 +0x9d
goroutine 9 [select]:
github.com/cybozu-go/aptutil/cacher.(*Cacher).maintRelease(0xc42006c360, 0x828640, 0xc420947980, 0xc42003e81e, 0x1b, 0xc42001af01)
/home/travis/gopath/src/github.com/cybozu-go/aptutil/cacher/cacher.go:214 +0x280
github.com/cybozu-go/aptutil/cacher.(*Cacher).maintMeta.func1(0x828640, 0xc420947980, 0x828640, 0xc420947980)
/home/travis/gopath/src/github.com/cybozu-go/aptutil/cacher/cacher.go:192 +0x55
github.com/cybozu-go/cmd.(*Environment).Go.func1(0xc420040310, 0xc4208d3600)
/home/travis/gopath/src/github.com/cybozu-go/cmd/env.go:133 +0x86
created by github.com/cybozu-go/cmd.(*Environment).Go
/home/travis/gopath/src/github.com/cybozu-go/cmd/env.go:138 +0xaf
goroutine 10 [select]:
github.com/cybozu-go/aptutil/cacher.(*Cacher).maintRelease(0xc42006c360, 0x828640, 0xc420947b80, 0xc420042c4e, 0x25, 0xc42001b700)
/home/travis/gopath/src/github.com/cybozu-go/aptutil/cacher/cacher.go:214 +0x280
github.com/cybozu-go/aptutil/cacher.(*Cacher).maintMeta.func2(0x828640, 0xc420947b80, 0x828640, 0xc420947b80)
/home/travis/gopath/src/github.com/cybozu-go/aptutil/cacher/cacher.go:197 +0x55
github.com/cybozu-go/cmd.(*Environment).Go.func1(0xc420040310, 0xc4208d36e0)
/home/travis/gopath/src/github.com/cybozu-go/cmd/env.go:133 +0x86
created by github.com/cybozu-go/cmd.(*Environment).Go
/home/travis/gopath/src/github.com/cybozu-go/cmd/env.go:138 +0xaf
goroutine 11 [select]:
github.com/cybozu-go/aptutil/cacher.(*Cacher).maintRelease(0xc42006c360, 0x828640, 0xc420947d00, 0xc420012dfe, 0x2e, 0xc42001bf00)
/home/travis/gopath/src/github.com/cybozu-go/aptutil/cacher/cacher.go:214 +0x280
github.com/cybozu-go/aptutil/cacher.(*Cacher).maintMeta.func2(0x828640, 0xc420947d00, 0x828640, 0xc420947d00)
/home/travis/gopath/src/github.com/cybozu-go/aptutil/cacher/cacher.go:197 +0x55
github.com/cybozu-go/cmd.(*Environment).Go.func1(0xc420040310, 0xc4208d3880)
/home/travis/gopath/src/github.com/cybozu-go/cmd/env.go:133 +0x86
created by github.com/cybozu-go/cmd.(*Environment).Go
/home/travis/gopath/src/github.com/cybozu-go/cmd/env.go:138 +0xaf
goroutine 12 [select]:
github.com/cybozu-go/aptutil/cacher.(*Cacher).maintRelease(0xc42006c360, 0x828640, 0xc420947800, 0xc4200434be, 0x1d, 0xc42001c700)
/home/travis/gopath/src/github.com/cybozu-go/aptutil/cacher/cacher.go:214 +0x280
github.com/cybozu-go/aptutil/cacher.(*Cacher).maintMeta.func2(0x828640, 0xc420947800, 0x828640, 0xc420947800)
/home/travis/gopath/src/github.com/cybozu-go/aptutil/cacher/cacher.go:197 +0x55
github.com/cybozu-go/cmd.(*Environment).Go.func1(0xc420040310, 0xc4208d38a0)
/home/travis/gopath/src/github.com/cybozu-go/cmd/env.go:133 +0x86
created by github.com/cybozu-go/cmd.(*Environment).Go
/home/travis/gopath/src/github.com/cybozu-go/cmd/env.go:138 +0xaf
goroutine 14 [chan receive]:
github.com/cybozu-go/cmd.(*HTTPServer).wait(0xc4204ad180, 0x828640, 0xc420947f00, 0xc42001bf68, 0xc42001bf78)
/home/travis/gopath/src/github.com/cybozu-go/cmd/http.go:212 +0x6c
github.com/cybozu-go/cmd.(*HTTPServer).(github.com/cybozu-go/cmd.wait)-fm(0x828640, 0xc420947f00, 0x828640, 0xc420947f00)
/home/travis/gopath/src/github.com/cybozu-go/cmd/http.go:208 +0x3e
github.com/cybozu-go/cmd.(*Environment).Go.func1(0xc420040310, 0xc420ab2830)
/home/travis/gopath/src/github.com/cybozu-go/cmd/env.go:133 +0x86
created by github.com/cybozu-go/cmd.(*Environment).Go
/home/travis/gopath/src/github.com/cybozu-go/cmd/env.go:138 +0xaf
goroutine 15 [chan receive]:
github.com/cybozu-go/cmd.(*HTTPServer).Serve.func1(0xc4204ad180, 0x8281c0, 0xc420022000)
/home/travis/gopath/src/github.com/cybozu-go/cmd/http.go:278 +0x5a
created by github.com/cybozu-go/cmd.(*HTTPServer).Serve
/home/travis/gopath/src/github.com/cybozu-go/cmd/http.go:280 +0xba
goroutine 16 [IO wait]:
net.runtime_pollWait(0x7f1d4a4a4e98, 0x72, 0x0)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc4204ad250, 0x72, 0xc422049d88, 0xc42000a1c0)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc4204ad250, 0x825b00, 0xc42000a1c0)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).accept(0xc4204ad1f0, 0x0, 0x824640, 0xc42204a000)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/fd_unix.go:419 +0x238
net.(*TCPListener).accept(0xc420022000, 0xc4204ad1b0, 0x1, 0xc422049e50)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/tcpsock_posix.go:132 +0x2e
net.(*TCPListener).AcceptTCP(0xc420022000, 0xc422049eb0, 0xc422049eb8, 0xc422049ea8)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/tcpsock.go:209 +0x49
github.com/cybozu-go/netutil.TCPKeepAliveListener.Accept(0xc420022000, 0x71a7c0, 0xc42276e000, 0x828700, 0xc421e9abd0)
/home/travis/gopath/src/github.com/cybozu-go/netutil/tcpkeepalive.go:19 +0x2f
net/http.(*Server).Serve(0xc42054d780, 0x8281c0, 0xc420022000, 0x0, 0x0)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/server.go:2273 +0x1ce
github.com/cybozu-go/cmd.(*HTTPServer).Serve.func2(0xc4204ad180, 0x8281c0, 0xc420022000)
/home/travis/gopath/src/github.com/cybozu-go/cmd/http.go:283 +0x42
created by github.com/cybozu-go/cmd.(*HTTPServer).Serve
/home/travis/gopath/src/github.com/cybozu-go/cmd/http.go:284 +0xf0
goroutine 17 [chan receive]:
github.com/cybozu-go/aptutil/cacher.(*Cacher).Get(0xc42006c360, 0xc420270245, 0x50, 0x50, 0x1, 0xc4201301e0, 0x0)
/home/travis/gopath/src/github.com/cybozu-go/aptutil/cacher/cacher.go:434 +0x2aa
github.com/cybozu-go/aptutil/cacher.cacheHandler.ServeHTTP(0xc42006c360, 0x827d00, 0xc42204a0e0, 0xc4201301e0)
/home/travis/gopath/src/github.com/cybozu-go/aptutil/cacher/handler.go:35 +0x185
github.com/cybozu-go/cmd.(*HTTPServer).ServeHTTP(0xc4204ad180, 0x827f80, 0xc421a6e1a0, 0xc4201300f0)
/home/travis/gopath/src/github.com/cybozu-go/cmd/http.go:132 +0x232
net/http.serverHandler.ServeHTTP(0xc42054d780, 0x827f80, 0xc421a6e1a0, 0xc4201300f0)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/server.go:2202 +0x7d
net/http.(*conn).serve(0xc42276e000, 0x828640, 0xc421e881c0)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/server.go:1579 +0x4b7
created by net/http.(*Server).Serve
/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/server.go:2293 +0x44d
goroutine 24 [select]:
net/http.(*persistConn).readLoop(0xc42000ca00)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/transport.go:1536 +0x9c9
created by net/http.(*Transport).dialConn
/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/transport.go:1057 +0x4e9
goroutine 25 [select]:
net/http.(*persistConn).writeLoop(0xc42000ca00)
/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/transport.go:1641 +0x3bd
created by net/http.(*Transport).dialConn
/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/transport.go:1058 +0x50e
2017-10-04T11:36:23.589582Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher warning: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=404 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="93ef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=19 response_time=0.058151219000000004 type="access" url="/debian/dists/jessie/InRelease"
2017-10-04T11:36:23.594537Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher info: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=200 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="90ef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=63066 response_time=0.0030334760000000002 type="access" url="/debian-security/dists/jessie/updates/InRelease"
2017-10-04T11:36:23.632614Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher info: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=200 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="91ef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=2373 response_time=0.00025303500000000003 type="access" url="/debian/dists/jessie/Release.gpg"
2017-10-04T11:36:23.635683Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher info: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=200 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="96ef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=148400 response_time=0.001911944 type="access" url="/debian/dists/jessie/Release"
2017-10-04T11:36:24.311989Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher info: "downloaded and cached" path="debian-security/dists/jessie/updates/main/binary-amd64/Packages.bz2"
2017-10-04T11:36:24.316877Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher info: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=200 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="97ef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=433331 response_time=0.635777987 type="access" url="/debian-security/dists/jessie/updates/main/binary-amd64/Packages.bz2"
2017-10-04T11:36:24.354651Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher info: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=200 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="94ef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=2506 response_time=0.00017602900000000002 type="access" url="/debian-security/dists/jessie/updates/contrib/binary-amd64/Packages.bz2"
2017-10-04T11:36:24.356700Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher info: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=200 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="95ef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=1211 response_time=0.0014163930000000002 type="access" url="/debian-security/dists/jessie/updates/contrib/i18n/Translation-en.bz2"
2017-10-04T11:36:24.363795Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher info: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=200 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="9aef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=231458 response_time=0.0065162020000000005 type="access" url="/debian-security/dists/jessie/updates/main/i18n/Translation-en.bz2"
2017-10-04T11:36:24.372568Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher warning: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=404 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="9bef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=19 response_time=0.000028965 type="access" url="/debian/dists/jessie/main/binary-amd64/Packages.xz"
2017-10-04T11:36:24.373048Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher warning: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=404 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="98ef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=19 response_time=0.000019858000000000002 type="access" url="/debian/dists/jessie/contrib/binary-amd64/Packages.xz"
2017-10-04T11:36:24.375672Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher info: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=200 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="99ef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=38307 response_time=0.002113989 type="access" url="/debian/dists/jessie/contrib/i18n/Translation-en.bz2"
2017-10-04T11:36:24.541312Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher info: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=200 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="9eef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=4582022 response_time=0.164297343 type="access" url="/debian/dists/jessie/main/i18n/Translation-en.bz2"
2017-10-04T11:36:24.873764Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher info: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=200 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="9fef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=9063195 response_time=0.225623379 type="access" url="/debian/dists/jessie/main/binary-amd64/Packages.gz"
2017-10-04T11:36:24.996407Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher info: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=200 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="9cef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=59182 response_time=0.00016119000000000002 type="access" url="/debian/dists/jessie/contrib/binary-amd64/Packages.gz"
It seems that go-apt-cacher can't handle missing path case.
Are there any workaround configuration on this issue?
There is a workaround configuration in go-apt-cacher.toml or something.
It works with apt-line:
deb http://security.debian.org/debian-security/ jessie/updates main
deb-src http://security.debian.org/debian-security/ jessie/updates main
and go-apt-cacher.toml.
[mapping]
debian = "http://ftp.jp.debian.org/debian"
debian-security = "http://security.debian.org/debian-security"
But It doesn't work with apt-line and above toml:
deb http://security.debian.org/ jessie/updates main
deb-src http://security.debian.org/ jessie/updates main
% sudo apt update
...(skip)...
Fetched 23.1 MB in 2s (10.4 MB/s)
W: Failed to fetch http://security.debian.org/dists/jessie/updates/main/source/Sources 404 Not Found
W: Failed to fetch http://security.debian.org/dists/jessie/updates/main/binary-amd64/Packages 404 Not Found
E: Some index files failed to download. They have been ignored, or old ones used instead.
How to configure go-apt-cacher.toml mapping correctly for http://security.debian.org/? (without modifiyng /etc/apt/sources.list?)
Hi,
One of the advantages of apt-cacher-ng was that sources.list files didn't need updating.
Just the Acquire::http::Proxy statement, or http(s)_proxy enviroment line were required.
When you start adding other repos for software projects, either manually or using PPAs, this looks like it will be messy. How does the proxy know which repo you wish to use?
Over at https://github.com/jacksgt/docker-aptutil I have created a Dockerfile for aptutil.
It includes both go-apt-cacher and go-apt-mirror.
Usage is rather simple:
Mount the configuration file for go-apt-cacher to /etc/go-apt-cacher.toml
(read-only for added security).
Optionally, mount /var/spool/go-apt-cacher
for persistence (required for --read-only
).
$ docker container run \
--read-only \
-v /etc/go-apt-cacher.toml:/etc/go-apt-cacher.toml:ro \
-v /var/spool/go-apt-cacher:/var/spool/go-apt-cacher \
jacksgt/aptutil /go-apt-cacher
Mount the configuration file for go-apt-cacher to /etc/apt/mirror.toml
(read-only for added security).
Mount /var/spool/go-apt-mirror
to store the mirrored files permanently (required for --read-only
).
$ docker container run \
--read-only \
-v /etc/go-apt-mirror.toml:/etc/apt/mirror.toml:ro \
-v /var/spool/go-apt-mirror:/var/spool/go-apt-mirror \
jacksgt/aptutil /go-apt-mirror
Are you interested in merging this into your repository?
If so, it would be great if you created a DockerHub account and set up an automatic build. That way, the image will be automatically built each time someone pushes into the source repo.
By 'linking' against the golang
docker image repository you can also ensure the image is rebuilt when there is a new version of the golang image ("Repository Links").
See https://hub.docker.com/r/jacksgt/docker-aptutil/
Getting the following errors:
2020-03-21T16:46:05.021453Z apt-cacher go-apt-cacher warning: "downloaded data is not valid" url="http://si.archive.ubuntu.com/ubuntu/dists/trusty-updates/main/i18n/Translation-en.bz2"
2020-03-21T16:46:05.032053Z apt-cacher go-apt-cacher warning: "downloaded data is not valid" url="http://si.archive.ubuntu.com/ubuntu/dists/xenial-updates/main/i18n/Translation-en.xz"
> docker run -ti --rm resin/armhf-alpine-golang bash
bash-4.4# go get -v -u github.com/cybozu-go/aptutil/...
github.com/cybozu-go/aptutil (download)
github.com/pkg/errors (download)
github.com/cybozu-go/cmd (download)
github.com/cybozu-go/log (download)
github.com/cybozu-go/netutil (download)
Fetching https://golang.org/x/net/netutil?go-get=1
Parsing meta tags from https://golang.org/x/net/netutil?go-get=1 (status code 200)
get "golang.org/x/net/netutil": found meta tag get.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/netutil?go-get=1
get "golang.org/x/net/netutil": verifying non-authoritative meta tag
Fetching https://golang.org/x/net?go-get=1
Parsing meta tags from https://golang.org/x/net?go-get=1 (status code 200)
golang.org/x/net (download)
github.com/spf13/pflag (download)
github.com/spf13/viper (download)
github.com/fsnotify/fsnotify (download)
Fetching https://golang.org/x/sys/unix?go-get=1
Parsing meta tags from https://golang.org/x/sys/unix?go-get=1 (status code 200)
get "golang.org/x/sys/unix": found meta tag get.metaImport{Prefix:"golang.org/x/sys", VCS:"git", RepoRoot:"https://go.googlesource.com/sys"} at https://golang.org/x/sys/unix?go-get=1
get "golang.org/x/sys/unix": verifying non-authoritative meta tag
Fetching https://golang.org/x/sys?go-get=1
Parsing meta tags from https://golang.org/x/sys?go-get=1 (status code 200)
golang.org/x/sys (download)
github.com/hashicorp/hcl (download)
github.com/magiconair/properties (download)
github.com/mitchellh/mapstructure (download)
github.com/pelletier/go-toml (download)
github.com/spf13/afero (download)
Fetching https://golang.org/x/text/transform?go-get=1
Parsing meta tags from https://golang.org/x/text/transform?go-get=1 (status code 200)
get "golang.org/x/text/transform": found meta tag get.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/transform?go-get=1
get "golang.org/x/text/transform": verifying non-authoritative meta tag
Fetching https://golang.org/x/text?go-get=1
Parsing meta tags from https://golang.org/x/text?go-get=1 (status code 200)
golang.org/x/text (download)
Fetching https://golang.org/x/text/unicode/norm?go-get=1
Parsing meta tags from https://golang.org/x/text/unicode/norm?go-get=1 (status code 200)
get "golang.org/x/text/unicode/norm": found meta tag get.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/unicode/norm?go-get=1
get "golang.org/x/text/unicode/norm": verifying non-authoritative meta tag
github.com/spf13/cast (download)
github.com/spf13/jwalterweatherman (download)
Fetching https://gopkg.in/yaml.v2?go-get=1
Parsing meta tags from https://gopkg.in/yaml.v2?go-get=1 (status code 200)
get "gopkg.in/yaml.v2": found meta tag get.metaImport{Prefix:"gopkg.in/yaml.v2", VCS:"git", RepoRoot:"https://gopkg.in/yaml.v2"} at https://gopkg.in/yaml.v2?go-get=1
gopkg.in/yaml.v2 (download)
github.com/BurntSushi/toml (download)
github.com/cybozu-go/aptutil
github.com/hashicorp/hcl/hcl/strconv
github.com/spf13/afero/mem
golang.org/x/text/transform
golang.org/x/sys/unix
github.com/pkg/errors
github.com/cybozu-go/aptutil/apt
github.com/cybozu-go/log
golang.org/x/net/netutil
github.com/cybozu-go/netutil
github.com/spf13/pflag
github.com/hashicorp/hcl/hcl/token
github.com/hashicorp/hcl/hcl/ast
github.com/hashicorp/hcl/hcl/scanner
github.com/hashicorp/hcl/json/token
github.com/hashicorp/hcl/hcl/parser
github.com/hashicorp/hcl/json/scanner
github.com/hashicorp/hcl/json/parser
github.com/hashicorp/hcl/hcl/printer
github.com/fsnotify/fsnotify
github.com/hashicorp/hcl
github.com/mitchellh/mapstructure
github.com/pelletier/go-toml
github.com/magiconair/properties
golang.org/x/text/unicode/norm
github.com/spf13/jwalterweatherman
github.com/spf13/cast
gopkg.in/yaml.v2
github.com/BurntSushi/toml
github.com/spf13/afero
github.com/spf13/viper
github.com/cybozu-go/cmd
github.com/cybozu-go/aptutil/cacher
github.com/cybozu-go/aptutil/mirror
# github.com/cybozu-go/aptutil/mirror
src/github.com/cybozu-go/aptutil/mirror/config.go:9:2: imported and not used: "github.com/cybozu-go/cmd" as well
src/github.com/cybozu-go/aptutil/mirror/config.go:168:11: undefined: cmd
src/github.com/cybozu-go/aptutil/mirror/control.go:10:2: imported and not used: "github.com/cybozu-go/cmd" as well
src/github.com/cybozu-go/aptutil/mirror/mirror.go:16:2: imported and not used: "github.com/cybozu-go/cmd" as well
# github.com/cybozu-go/aptutil/cacher
src/github.com/cybozu-go/aptutil/cacher/cacher.go:20:2: imported and not used: "github.com/cybozu-go/cmd" as well
src/github.com/cybozu-go/aptutil/cacher/config.go:3:8: imported and not used: "github.com/cybozu-go/cmd" as well
src/github.com/cybozu-go/aptutil/cacher/config.go:61:6: undefined: cmd
src/github.com/cybozu-go/aptutil/cacher/server.go:6:2: imported and not used: "github.com/cybozu-go/cmd" as well
Hey,
I've got an error while mirroring the debian buster repo.
2020-04-16T19:54:42.020106Z goaptmirror go-apt-mirror error: "update failed" error="debian: no Files in dists/buster-updates/main/source/Sources.gz"
2020-04-16T19:54:42.020187Z goaptmirror go-apt-mirror error: "debian: no Files in dists/buster-updates/main/source/Sources.gz"
My mirror.toml:
# Directory to store mirrored files and other control files.
# The directory must be writable by go-apt-mirror.
dir = "/mnt/mirror"
# Maximum concurrent connections for an upstream server.
# Setting this 0 disables limit on the number of connections.
# Default: 10
max_conns = 10
# log specifies logging configurations.
# Details at https://godoc.org/github.com/cybozu-go/well#LogConfig
[log]
level = "info"
format = "plain"
#filename = "go-apt-mirror.log"
# [mirror.xxx] defines a mirror configuration for a debian repository.
# "xxx" must match this regexp: ^[a-z0-9_-]+$
#
# url: The repository base URL.
# suites: List of suites to mirror. see sources.list(5).
# sections: List of sections to mirror. see sources.list(5).
# mirror_source: true to mirror source archives. Default is false.
# architectures: List of architectures to mirror. "all" is always mirrored.
[mirror.debian]
url = "http://debian.uni-duisburg-essen.de/debian"
suites = ["stretch", "stretch-updates", "buster", "buster-updates"]
sections = ["main"]
mirror_source = true
architectures = ["amd64"]
#[mirror.ubuntu]
#url = "http://archive.ubuntu.com/ubuntu"
#suites = ["bionic", "bionic-updates"]
#sections = ["main"]
#mirror_source = false
#architectures = ["amd64"]
[mirror.security]
url = "http://security.debian.org/debian-security"
suites = ["buster/updates"]
sections = ["main"]
mirror_source = true
architectures = ["amd64"]
So with the security repo its run fine. In the buster, buster-updates repo go-apt-mirror stops with the error above. But only if i set the "source = true". When source = false, go-apt mirror download the 78Gb repo.
But the terminal output show me a lot of "missing file" like:
2020-04-16T19:54:41.910018Z goaptmirror go-apt-mirror warning: "missing file" path="dists/buster-updates/main/Contents-udeb-mips" repo="debian"
2020-04-16T19:54:41.911268Z goaptmirror go-apt-mirror warning: "missing file" path="dists/buster-updates/contrib/binary-mips/Packages" repo="debian"
2020-04-16T19:54:41.914372Z goaptmirror go-apt-mirror warning: "missing file" path="dists/buster-updates/contrib/Contents-udeb-i386" repo="debian"
2020-04-16T19:54:41.924035Z goaptmirror go-apt-mirror warning: "missing file" path="dists/buster-updates/non-free/debian-installer/binary-mips/Packages" repo="debian"
this output i get always when i start go-apt-mirror. What do this mean? This warning give me the feeling that anything isnt right. i am not sure.
I use the 1.4.1 release. This is my first issue. so i hope i make it right :-)
best regards
Chris
go-apt-mirror can detect by-hash support if all suites provide Acquire-By-Hash: yes
in Release files.
But if one suite does not support by-hash while the others do, go-apt-mirror misinterprets that
all suites do not support by-hash. This is not ideal.
Problem report template:
Apparently, there are many broken debian repositories out there, where the "Release file" contains entries, which are missing from the repository.
Some examples like this are some public Zabbix repositories etc.
Can happen at any time
Company test systems
go-apt-mirror: v1.4.2
Configure sync to some repositories, which where there are some 'broken' files
Execute go-apt-mirror -f mybrokenrepos.toml
All available files are synced.
The sync software, it it encounters an error (file missing from remote repo), will attempt to sync repo again.
Not any
When calling downloadFiles at:
https://github.com/cybozu-go/aptutil/blob/master/mirror/mirror.go#L524
we are setting allowMissing to false. This causes the caller function https://github.com/cybozu-go/aptutil/blob/master/mirror/mirror.go#L173 to return error causing the control function get get an error return https://github.com/cybozu-go/aptutil/blob/master/mirror/control.go#L40 stopping the update.
Suggestions: Let's make the setting allowMissing configurable in the .toml file on a per repository basis (but defaulting to false, to preserve the original behavior), and to allow handling the situation gracefully.
The following settings will fail.
go-apt-mirror v1.3.2 linux/amd64
# Directory to store mirrored files and other control files.
# The directory must be writable by go-apt-mirror.
dir = "/home/takumin/test"
# Maximum concurrent connections for an upstream server.
# Setting this 0 disables limit on the number of connections.
# Default: 10
max_conns = 10
# log specifies logging configurations.
# Details at https://godoc.org/github.com/cybozu-go/cmd#LogConfig
[log]
level = "info"
format = "plain"
# [mirror.xxx] defines a mirror configuration for a debian repository.
# "xxx" must match this regexp: ^[a-z0-9_-]+$
#
# url: The repository base URL.
# suites: List of suites to mirror. see sources.list(5).
# sections: List of sections to mirror. see sources.list(5).
# mirror_source: true to mirror source archives. Default is false.
# architectures: List of architectures to mirror. "all" is always mirrored.
[mirror.jenkins]
url = "http://pkg.jenkins.io/debian-stable"
suites = ["binary/"]
[mirror.nvidia-cuda]
url = "http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64"
suites = ["/"]
[mirror.nvidia-digits]
url = "http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64"
suites = ["/"]
2017-10-05T13:48:41.478084Z srv go-apt-mirror info: "update starts"
2017-10-05T13:48:41.478292Z srv go-apt-mirror info: "download Release/InRelease" repo="jenkins" suite="binary/"
2017-10-05T13:48:41.478396Z srv go-apt-mirror info: "download Release/InRelease" repo="nvidia-digits" suite="/"
2017-10-05T13:48:41.478509Z srv go-apt-mirror info: "download Release/InRelease" repo="nvidia-cuda" suite="/"
2017-10-05T13:48:41.945865Z srv go-apt-mirror info: "detected by-hash support" repo="nvidia-digits" suite="/"
2017-10-05T13:48:41.946167Z srv go-apt-mirror error: "update failed" error="nvidia-digits: found no Release/InRelease"
2017-10-05T13:48:41.946550Z srv go-apt-mirror error: "nvidia-digits: found no Release/InRelease"
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.