tuna / tunasync

Mirror job management tool.

tunasync

Get Started

• 中文文档

Download

Pre-built binary for Linux x86_64 and ARM64 is available at Github releases.

Design

# Architecture

- Manager: Central instance for status and job management
- Worker: Runs mirror jobs

+------------+ +---+                  +---+
| Client API | |   |    Job Status    |   |    +----------+     +----------+ 
+------------+ |   +----------------->|   |--->|  mirror  +---->|  mirror  | 
+------------+ |   |                  | w |    |  config  |     | provider | 
| Worker API | | H |                  | o |    +----------+     +----+-----+ 
+------------+ | T |   Job Control    | r |                          |       
+------------+ | T +----------------->| k |       +------------+     |       
| Job/Status | | P |   Start/Stop/... | e |       | mirror job |<----+       
| Management | | S |                  | r |       +------^-----+             
+------------+ |   |   Update Status  |   |    +---------+---------+         
+------------+ |   <------------------+   |    |     Scheduler     |
|   BoltDB   | |   |                  |   |    +-------------------+
+------------+ +---+                  +---+


# Job Run Process


PreSyncing                           Syncing                               Success
+-----------+     +----------+    +-----------+    +-------------+     +--------------+
|  pre-job  +--+->| pre-exec +--->|  job run  +--->|  post-exec  +-+-->| post-success |
+-----------+  ^  +----------+    +-----------+    +-------------+ |   +--------------+
               |                                                   |
               |                +-----------------+                | Failed
               +----------------+    post-fail    |<---------------+
                                +-----------------+

Building

Go version: 1.13

> make all

Binaries in the build-linux-amd64/.

tunasync's Issues

Use Redis or MQ to pass status and signals

So that tunasync can work upon multiple servers, and code can be simplified.

two-stage-sync provider cannot log to file properly

二级帮助提示

建议在tunasync -h里面体现出可以参阅 tunasync manager -h

你们开源镜像站挂了一个星期了，请问还修复么

你们开源镜像站挂了一个星期了，请问还修复么

permission error for zfs

The owner of new created zfs datasets is root, while syncing commands are run as tunasync.

Register at Pypi

Register before the name is taken! We also need a proper setup.py.

Milestone: 1.0 version

feature list:

rsync error no to error msg

translate the error no returned from rsync to error msg.

CPU使用率过高

服务器是2C 8G内存的配置。目前一共配置了6个mirrors同步job。concurrent设置为3。但是一直会报CPU使用率过高。请问是job的参数设置有问题吗

没办法同步，希望得到帮助

manager 收到worker 的信息，但worker 似乎没有执行mirrors里边的同步脚本，按照中文教程执行了几遍，也换过几台服务器，都不行，希望能够得到帮助。

How to configure the sync time with different jobs?

Hello, the server's disk I/O is alway very high when tunasync syncs mirrors, so we want to setup different sync times for these mirrors sync jobs, coud you tell me how to configure it.

Process code reviews from @Arthur2e5

commits:

• b7d3467
• 1c2d8a9
• 6205d4a

Web frontend for manager

Tuna/tunasync

Synchronize apps

How to solve bandersnatch via proxy?

HI，

I know this is not one "bug".But I noticed that tuna also uses the bandersnatch to sync Pypi .
I use the same way in my company, but the different is that ,my company has proxy. Each time I execute bandersntach will be failed by the connection time out.

Do you have any ideas?

Hot reloading config file for worker

docker executor

mirror jobs can be spawned using docker, which has these pros:

• mirror directory is isolated
• execution environment is stateless

Removed spam

tunasynctl cannot parse " 08:42:13 +0800"" as "T"

执行 ./tunasynctl list --all 报错：

/Users/Ray/.config/tunasync/ctl.conf
[17-06-06 17:11:34][INFO] Use manager address: http://localhost:14242
Failed to correctly get information of all jobs from manager server: parsing time ""2017-06-06 08:42:13 +0800"" as ""2006-01-02T15:04:05Z07:00"": cannot parse " 08:42:13 +0800"" as "T"

请问如何解决？

dockerhub

post it to dockerhub making it easier to run

improve test coverage

Test coverage need to be at lease 80%

"cannot delete non-empty directory" with rsync syncing

When folders were removed on server-side, rsync refused to delete them.

See https://serverfault.com/questions/573392/rsync-cannot-delete-non-empty-directory-errors-even-with-force-option for it.

Anaconda mirror script

Currently using lftp mirroring does not seems to be quite reliable since there's no checksum verifying.

Anaconda use repodata.json to store metadata.

curl -s https://repo.continuum.io/pkgs/free/linux-32/repodata.json.bz2 | bzip2 -d -

Aggregate syncing logs to manager

so that the logs can be viewed from management interface

go cli action signature depreacated

https://github.com/codegangsta/cli/blob/master/CHANGELOG.md#deprecated-cli-app-action-signature

Android SDK

参考:

https://dl.google.com/android/repository/repository-10.xml
http://mirrors.neusoft.edu.cn/android/repository/

Documentation

Background? Motivation? Trade-Off?

cannot delete mirror name from mirror list

When a mirror is deleted, it should be deleted from the public mirrorlist.

Usage statistics

Maybe we need a network/request statistics on web page.

make scripts a submodule to tunasync-scripts

Currently, scripts directory is some legacy content of tunasync-scripts. It's confusing to users.

How about making it a submodule pointed to tunasync-scripts?

use cgroup to manage child progresses

To make sure is job can be really killed

repo sync aosp code have warning

# ubuntu 14.04
$ repo init -u https://aosp.tuna.tsinghua.edu.cn/platform/manifest -b android-6.0.1_r67
$ repo sync

remote: warning: unable to access '/root/.config/git/attributes': Permission denied        
From https://aosp.tuna.tsinghua.edu.cn/platform/external/sonivox
   1d73da2..902ce06  master     -> aosp/master
remote: warning: unable to access '/root/.config/git/attributes': Permission denied        
From https://aosp.tuna.tsinghua.edu.cn/platform/packages/providers/MediaProvider
   dcf3957..9eb7709  master     -> aosp/master
remote: warning: unable to access '/root/.config/git/attributes': Permission denied        
From https://aosp.tuna.tsinghua.edu.cn/platform/external/conscrypt
   adb03d6..0da48d5  master     -> aosp/master
remote: warning: unable to access '/root/.config/git/attributes': Permission denied        
From https://aosp.tuna.tsinghua.edu.cn/platform/external/tinyalsa

同步过程中报错

Centos7的服务器，同步过程中，报端口连接被拒绝。查看了下防火墙及SELINUX的状态都被禁用了。查看服务状态后，报错如下
systemctl status tunasync-worker.service -l

3月 06 22:09:24 p-awscn-paas tunasync[62593]: [ERROR] Failed to update mirror(centos) status: Post https://mirrors.xxxxx.com:12345/workers/mirror_worker/jobs/centos: dial tcp ip地址:12345: getsockopt: connection refused

包的确是被正常的同步下来了，但是无法使用tunasync list mirrors_worker状态查看每个包的同步状态。

Cannot make

How to reproduce:

~> git clone [email protected]:tuna/tunasync.git
~> cd tunasync
~> make

failed with following errors:

~/Source/tunasync on master   / ~> make
go get ./cmd/tunasync
package ./cmd/tunasync
        imports github.com/tuna/tunasync/internal: use of internal package not allowed
make: *** [Makefile:6: get] Error 1
Exception: make exited with 2
Traceback:
  [interactive], line 1:
    make

Software versions:

~/Source/tunasync on master   / ~> go version
go version go1.7.3 linux/amd64
~/Source/tunasync on master   / ~> make -v
GNU Make 4.2.1
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
~/Source/tunasync on master   / ~> uname -a
Linux ancient 4.8.4-1-ARCH #1 SMP PREEMPT Sat Oct 22 18:26:57 CEST 2016 x86_64 GNU/Linux

ftpsync provider

ftpsync is the only offical syncing tool for debian and kali, we should either reimplement ftpsync or better integrate with it.

It seems to be easier to wrap ftpsync as a mirror provider.

报错: Error initializing DB adapter: open /var/lib/tunasync/tunasync.db: no such file or directory

(sid)scateu@localhost:~$ ./tunasync --help
NAME:
tunasync - A new cli application

USAGE:
tunasync [global options] command [command options] [arguments...]

VERSION:
0.1

COMMANDS:
manager, m start the tunasync manager
worker, w start the tunasync worker

GLOBAL OPTIONS:
--help, -h show help
--version, -v print the version

(sid)scateu@localhost:~$ ./tunasync manager
[16-10-31 00:16:51][ERROR] Error initializing DB adapter: open /var/lib/tunasync/tunasync.db: no such file or directory
[16-10-31 00:16:51][ERROR] Error intializing TUNA sync worker.

rsync exclude cannot work with docker executor

rsync exclude files are not considered in docker executor

`status_file` not work

I tried to set status_file in manager.conf, but it does not work.

I read through the sources but there's no codes for it. Does the status_file deprecated? Is there any way to get mirrors' status from a webpage?

Multiple reposync instances are confused about repomd.xml

We take use of reposync to sync yum repo.

When launching multiple instance of reposync which share similar configuration(mainly name of repo), one may falsely consider repomd.xml downloaded by another instance as fresh cache and finally results in failure.

Forcing reposync to use temporary directory(with -t) does not fix this problem.

Set retry in configuration

Make it configurable for retries in configuration.

For e.g.

in worker.conf

[global]
retry = 3

[[mirrors]]
retry = 0

Code needs comments

Add meaningful comments to the code so that other people can understand.

建议提供二进制Release

Meaning of exit status

Hi, I got some error code(Exit status 10) while syncing mirrors. I tried to look through source code to find their meaning, but I didn't find out. Can you please give more information about these error codes? Thanks.

Token and authentication

Security needs to be enhanced

Reduce CPU Usage

CPU usage is quite high, pprof indicates that runtime.siftdownTimer consumed a lot of CPU time.
It seems that we had abused golang timer objects.

Multi-file mirror config

Make it possible to write worker config as

[global]
blah =  ""

[server]
blah = ""

[include]
include_mirrors = "/etc/tunasync/mirorrs/*.conf"

[[mirrors]]
blah = ""

In the included config files, like /etc/tunasync/mirorrs/*.conf, only [[mirrors]] section is allowed.

RsyncProvider has no "--links" in the default options.

I see that the official mirror scripts provided by Archlinux and Ubuntu both has the option "--links" in it. I'm wondering why there's no "--links" in the default options.

class RsyncProvider(MirrorProvider):

    _default_options = ['-aHvh', '--no-o', '--no-g', '--stats',
                        '--exclude', '.~tmp~/',
                        '--delete', '--delete-after', '--delay-updates',
                        '--safe-links', '--timeout=120', '--contimeout=120']
...

从日志中获得 rsync 仓库的大小

refer tuna/issues#160

stop jobs when receiving SIGTERM and SIGINT

同步问题

mirror远程源里的rpm包没有了，为什么我本地的rpm包也会被删除，有没有办法保留本地的rpm包