mihaigalos / aim

🎯 A command line download/upload tool with resume.

aim

A command line download/upload tool with resume.

Table of Contents

❓ Motivation 💿︎ Installation 💡 Features Feature matrix Download / Upload Optional check of sha256 Interactive mode Resume Output during downloading Sharing a folder Indicators Output

🔑 Authentication Basicauth in url Netrc SSH keys .aws folder 🆕 Updating

🐳 Docker Hosting on machine A Downloading on machine B 🛠️ Similar work

❓ Motivation

Simplicity: download or upload files depending on parameter order with default settings.

💿︎ Installation

Download a release for Linux or MacOS from releases. See the Docker section on how to run it platform-independently.

If you want to build from source, use:

cargo install aim

💡 Features

Feature matrix

Protocol	Download	Upload	Resume	Interactive mode
http(s)	✅	✅	✅	✅
ftp	✅	✅	✅	❌
sftp	✅	✅	✅	❌
ssh	✅	✅	❌	❌
s3	✅	✅	❌	❌

Download / Upload

• default action implied from parameter order.
  • aim https://domain.com/ -> Display contents.
  • aim https://domain.com/source.file . -> Download.
  • aim source.file https://domain.com/destination.file -> Upload.
• support for http(s), (s)ftp, ssh, s3 (no resume at the moment).

Optional check of sha256

To validate that a download matches a desired checksum, just list it at the end when invoking aim.

aim https://github.com/XAMPPRocky/tokei/releases/download/v12.0.4/tokei-x86_64-unknown-linux-gnu.tar.gz . 0e0f0d7139c8c7e3ff20cb243e94bc5993517d88e8be8d59129730607d5c631b

Resume

Please consult the Feature matrix to find out if transfers via your desired protocol are resumable.

Resumable transfers pick up from a specific byte offset and continue. Extensive testing ensures that transfers are byte-exact (hash comparison between expected and actual transfer artefacts).

Node: If you're hosting a http(s) server yourself, upload needs PUT ranges (or a patched version of nginx).

Interactive mode

This feature can be activated by passing the -i or --interactive flag to the invocation.

It allows you to specify an initial URL and then navigate through links found in it using fuzzy search.

Controls:

• Start typing, partial matches are listed.
• Tab expands the path and goes into it, lists contents.
• / goes into path without expanding, lists contents.
• .. goes one level up.
• Enter finalizes the interaction and takes the result, performs the required operation on it.

This feature can be used in conjunction with Output during downloading and/or Sharing a folder.

Output during downloading

Several output formats can be specified:

• aim source . - downloads to the same basename as the source.
• aim source + - downloads to the same basename as the source and attempts to decompress. Target extensions are read and the system decompressor is called. Further info here.
• aim source destination - download to a new or existing file called destination.

---

Sharing a folder

aim can serve a folder over http on one device so that you can download it on another. By default, the serving port is 8080 or the next free port.

You can optionally set the AIM_HOSTING_PORT environment variable in your shell or .env file for a specific port.

Machine A

aim . # to serve current folder

Machine B

aim http://ip_of_Machine_A:8080 # list contents
aim http://ip_of_Machine_A:8080/file . # download

Moreover, since hosting is done over http, the client can even be a browser:

The server prints logs to the standard output. To colorize them, you can use pipeview:

aim . | pipeview --aim

Indicators

By default, a progressbar is displayed when up/downloading. The indicators can be configured via the internally used indicatif package.

You can change the display template and progress chars by either setting correct environment variables or creating a .env file in the folder you are calling from:

AIM_PROGRESSBAR_DOWNLOADED_MESSAGE="🎯 Downloaded {input} to {output}"
AIM_PROGRESSBAR_MESSAGE_FORMAT="🎯 Transferring {url}"
AIM_PROGRESSBAR_PROGRESS_CHARS="=>-"
AIM_PROGRESSBAR_TEMPLATE="{msg}\n{spinner:.cyan}  {elapsed_precise} ▕{bar:.white}▏ {bytes}/{total_bytes}  {bytes_per_sec}  ETA {eta}."
AIM_PROGRESSBAR_UPLOADED_MESSAGE="🎯 Uploaded {input} to {output}"
AIM_HOSTING_PORT=8080

By default, no progressbar is displayed if content length <1MB (easy display contents of remote).

---

Output

Because default output is stdout, aim is pipe-able to other commands:

aim https://github.com/XAMPPRocky/tokei/releases/download/v12.0.4/tokei-x86_64-unknown-linux-gnu.tar.gz | tar xvz
aim https://www.rust-lang.org/ | htmlq --attribute href a

---

🔑 Authentication

Basicauth in url

Just use the syntax protocol://user:pass@server:port. This can be used for all http(s), ftp, ssh and s3.

Example for downloading:

aim ftp://user:[email protected]:21/myfile .

Netrc

Create a file named .netrc with read permissions in ~ or the current folder you're running aim from to automate login to that endpoint:

machine mydomain.com login myuser password mypass port server_port

SSH keys

Keys that match the following patterns are automatically tried:

• id_ed25519
• id_rsa
• keys/id_ed25519
• keys/id_rsa
• ~/.ssh/id_rsa
• ~/.ssh/keys/id_ed25519

.aws folder

Credentials for AWS interaction (i.e.: S3) are automatically read from ~/.aws/credentials.

Alternatively, the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables are read.

---

🆕 Updating

aim can self update in-place using:

aim --update

🐳 Docker

For convenience, alpine-based docker images for aarch64 and x64 are available, so arguments can be passed directly to them.

docker run --rm -it -v $(pwd):/src --net=host --user $UID:$UID mihaigalos/aim https://raw.githubusercontent.com/mihaigalos/aim/main/LICENCE.md

Hosting on machine A

cd $(mktemp -d)
echo hello > myfile
docker run --rm -it -v $(pwd):/src --user $UID:$UID -p 8080:8080 mihaigalos/aim /src

Downloading on machine B

Adapt IP to match that of machine A.

docker run --rm -it -v $(pwd):/src --user $UID:$UID mihaigalos/aim http://192.168.0.24:8080/myfile /src/myfile

---

🛠️ Similar work

duma, grapple, rget.

aim's Issues

RUSTSEC-2022-0048: xml-rs is Unmaintained

xml-rs is Unmaintained

Details
Status	unmaintained
Package	xml-rs
Version	0.8.4
URL	https://github.com/netvl/xml-rs/issues
Date	2022-01-26

xml-rs is a XML parser has open issues around parsing including integer
overflows / panics that may or may not be an issue with untrusted data.

Together with these open issues with Unmaintained status xml-rs
may or may not be suited to parse untrusted data.

Alternatives

• quick-xml

See advisory page for additional details.

performance comparison

In order to ease the evaluation process, is it suggested to compare aim with other products, such as aria2 and wget.

Particularly, I am interested if aim handles multi source downloads, metalinks and torrents.

Also, I like to congratulate you to what looks like an awesome piece of software ☺️

RUSTSEC-2023-0052: webpki: CPU denial of service in certificate path building

webpki: CPU denial of service in certificate path building

Details
Package	webpki
Version	0.22.0
Date	2023-08-22

When this crate is given a pathological certificate chain to validate, it will
spend CPU time exponential with the number of candidate certificates at each
step of path building.

Both TLS clients and TLS servers that accept client certificate are affected.

This was previously reported in
<briansmith/webpki#69> and re-reported recently
by Luke Malinowski.

rustls-webpki is a fork of this crate which contains a fix for this issue
and is actively maintained.

See advisory page for additional details.

Migrate to autoclap for consuming Cargo.toml data

S3 support

RUSTSEC-2021-0141: dotenv is Unmaintained

dotenv is Unmaintained

Details
Status	unmaintained
Package	dotenv
Version	0.15.0
URL	dotenv-rs/dotenv#74
Date	2021-12-24

dotenv by description is meant to be used in development or testing only.

Using this in production may or may not be advisable.

Alternatives

The below may or may not be feasible alternative(s):

• dotenvy

See advisory page for additional details.

RUSTSEC-2021-0146: Crate `twoway` deprecated by the author

Crate twoway deprecated by the author

Details
Status	unmaintained
Package	twoway
Version	0.1.8
URL	https://github.com/bluss/twoway
Date	2021-05-20

The commit e99b3c7 releasing version 0.2.2 explicitely deprecates twoway in favour of memchr crate.

See advisory page for additional details.

Executable name overlap

Where does the name come from? Maybe not a big issue, but ABINIT package has aim executable, too. See the doc. (abinit package is available in the universe/science repository of Ubuntu.)

Interactive mode

Use / for diving into folders, Enter to download file or whole folder.
Possible solutions via fzf/skim augmented navigation.

Fix HTTP PUT when self-hosting

Currently not possible to perform PUT operations when self-hosting the http server (i.e. via aim .).

Download/Upload with wildcards

• Use i.e.: *.mp3, *.* for file matching.
• Use i.e.: **/*.mp3 for recursive file matching.

SSH-Copy Support

See this.

extern crate ssh2;

use std::net::TcpStream;
use ssh2::Session;
use std::path::Path;
use std::fs::File;
use indicatif::ProgressBar;

fn main() -> Result<(), Box<std::error::Error>> {
    let ssh_host_port = "prokerala.com:22";
    let ssh_user = "prokeral";
    let remote_temp_file = "/tmp/dbimport_Rk6Iwwm5.sql.bz2";

    let tcp = TcpStream::connect(&ssh_host_port).unwrap();
    let mut sess = Session::new().unwrap();
    sess.handshake(&tcp).expect("SSH handshake failed");
    let _ = sess.userauth_agent(ssh_user);

    let path = Path::new(&remote_temp_file);
    let (mut remote_file, stat) = sess.scp_recv(path).unwrap();

    let stream:ssh2::Stream = remote_file.stream(1);

    // Update: solved by using io::copy as suggested by @apemanzilla below
    // let mut target = File::create("/tmp/done.txt").unwrap();
    // let pb = ProgressBar::new(stat.size());
    // std::io::copy(&mut pb.wrap_read(remote_file), &mut target)?;
    Ok(())
}

show reply when uploading

aim doesn't currently show the reply when an upload happens, this makes services like transfer.sh or 0x0.st unusable

usage
aim file.txt http://0x0.st
aim file.txt http://transfer.sh/file.txt

example curl

curl -F [email protected] http://0x0.st
link to file here

curl --upload-file file.txt https://transfer.sh/file.txt
link to file here 

Fix CD

Because of a missing schema definition (sftp), aim is currently using a hard fork of rust-url.
This means cargo publish will fail in the CD pipeline, because we cannot publish a crate that relies on git deps.

This PR tried to contribute changes upstream but got closed.

url-parse, once reaching maturity, will be able to unblock the blocked pipeline.

Use Sha2 crate instead of Sha256

This enables hashing the file without copying it into memory.
From here:

use sha2::{Sha256, Digest};
use std::{io, fs};

let mut hasher = Sha256::new();
let mut file = fs::File::open("file.tar.gz")?;

let bytes_written = io::copy(&mut file, &mut hasher)?;
let hash_bytes = hasher.finalize();

RUSTSEC-2021-0139: ansi_term is Unmaintained

ansi_term is Unmaintained

Details
Status	unmaintained
Package	ansi_term
Version	0.12.1
URL	ogham/rust-ansi-term#72
Date	2021-08-18

The maintainer has adviced this crate is deprecated and will not
receive any maintenance.

The crate does not seem to have much dependencies and may or may not be ok to use as-is.

Last release seems to have been three years ago.

Possible Alternative(s)

The below list has not been vetted in any way and may or may not contain alternatives;

• anstyle
• nu-ansi-term
• yansi

See advisory page for additional details.

HTTPS self-hosting for easy sharing

See also:

• https://github.com/genonullfree/teleport
• https://gitlab.com/skubalj/serve-directory/-/tree/main
• https://github.com/kettle11/devserver

RUSTSEC-2020-0056: stdweb is unmaintained

stdweb is unmaintained

Details
Status	unmaintained
Package	stdweb
Version	0.4.20
URL	koute/stdweb#403
Date	2020-05-04

The author of the stdweb crate is unresponsive.

Maintained alternatives:

• wasm-bindgen
• js-sys
• web-sys

See advisory page for additional details.

Multi-arch docker images

See this.

---
name: 'build images'

on:
  push:
    branches:
      - master

jobs:
  docker:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Prepare
        id: prep
        run: |
          DOCKER_IMAGE=${{ secrets.DOCKER_USERNAME }}/${GITHUB_REPOSITORY#*/}
          VERSION=latest
          SHORTREF=${GITHUB_SHA::8}

          # If this is git tag, use the tag name as a docker tag
          if [[ $GITHUB_REF == refs/tags/* ]]; then
            VERSION=${GITHUB_REF#refs/tags/v}
          fi
          TAGS="${DOCKER_IMAGE}:${VERSION},${DOCKER_IMAGE}:${SHORTREF}"

          # If the VERSION looks like a version number, assume that
          # this is the most recent version of the image and also
          # tag it 'latest'.
          if [[ $VERSION =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]; then
            TAGS="$TAGS,${DOCKER_IMAGE}:latest"
          fi

          # Set output parameters.
          echo ::set-output name=tags::${TAGS}
          echo ::set-output name=docker_image::${DOCKER_IMAGE}

      - name: Set up QEMU
        uses: docker/setup-qemu-action@master
        with:
          platforms: all

      - name: Set up Docker Buildx
        id: buildx
        uses: docker/setup-buildx-action@master

      - name: Login to DockerHub
        if: github.event_name != 'pull_request'
        uses: docker/login-action@v1
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_PASSWORD }}

      - name: Build
        uses: docker/build-push-action@v2
        with:
          builder: ${{ steps.buildx.outputs.name }}
          context: .
          file: ./Dockerfile
          platforms: linux/amd64,linux/arm64,linux/ppc64le
          push: true
          tags: ${{ steps.prep.outputs.tags }}

SFTP resume support

Async SSH support

#52 will introduce async sftp support.
The complexity of offering async ssh support (including resumable streams) is then low.

Fix non-resumable FTP PUT

Currently, transferred bytes is hard-coded to zero.
A restart from a specific byte is necessary. The already transferred number of bytes should be fetched from the remote here.

Verify sha256 checksum via commandline argument

Pipe to other commands instead of redirecting stdout

Seems there's a pipe problem with i.e. large html files.

cargo run https://nixos.org/nixos/about.html | htmlq  --text .main

RUSTSEC-2023-0028: buf_redux is Unmaintained

buf_redux is Unmaintained

Details
Status	unmaintained
Package	buf_redux
Version	0.8.4
URL	https://github.com/abonander/buf_redux/issues
Date	2023-01-24

Last release was over three years ago.

The maintainer(s) have been unreachable to respond to any issues that may or may not include security issues.

The repository is now archived and there is no security policy in place to contact the maintainer(s) otherwise.

The safety-undocumented unsafe in the crate may or may not be safe to use.

The crate also has a current future incompatibility warning buf_redux/23.

Possible Alternatives

The below may or may not provide alternative(s)

• Rust alloc / std vec::Vec, collections::VecDeque
• buffer-redux - fork
• bytes
• crates.io search for 'buffer'

See advisory page for additional details.

aim_1.1.1_amd64.deb has odd description

Description: Syntax highlighter for git.
Delta provides language syntax-highlighting, within-line insertion/deletion
detection, and restructured diff output for git on the command line.

Support invalid TLS/SSL

This currently fails, especially in the case of a Man-in-the-Middle corporate certificates.

mosh support

mosh.io
https://github.com/mobile-shell/mosh

This is ssh, just better.

Like aim for ssh.

I suggest supporting it 😇

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update Rust crate clap to 4.1.8
• Update Rust crate crossbeam-utils to 0.8.15
• Update Rust crate warpy to 0.3.7
• Update Rust crate tokio to 1.26.0
• Click on this checkbox to rebase all open PRs at once

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• Update Rust crate ssh2 to 0.9.4
• Update Rust crate rust-s3 to 0.32.3

Detected dependencies

cargo

Cargo.toml

• async-ssh2-lite 0.4
• async-std 1.12.0
• async-stream 0.3.4
• async_ftp 6.0.0
• autoclap 0.3.15
• clap 4.1.6
• crossbeam-utils 0.8.14
• custom_error 1.9.2
• dotenvy 0.15.6
• futures 0.3
• futures-util 0.3.26
• http 0.2.9
• indicatif 0.17.3
• lazy_static 1.4.0
• log 0.4.17
• melt 0.1.6
• native-tls 0.2
• netrc 0.4.1
• openssl 0.10.45
• question 0.2.2
• regex 1.7.1
• reqwest 0.11.14
• rust-s3 0.31.0
• self_update 0.36.0
• sha2 0.10.6
• skim-navi 0.1.7
• ssh2 0.9.3
• strfmt 0.2.4
• tokio 1.25.0
• tokio-util 0.7.7
• untildify 0.1.1
• url-parse 1.0.5
• uuid 1.3
• warpy 0.3.5
• serial_test 1.0.0

dockerfile

Dockerfile

• alpine 3.17

github-actions

.github/workflows/audit.yaml

.github/workflows/cd.yaml

• actions/checkout v3

.github/workflows/ci.yaml

---

• Check this box to trigger a request for Renovate to run again on this repository