curl/wget (need help) about pure-bash-bible

[dylanaraps]

I got it working. It's a little slow as it requires two while loops. I'm going to work on making this even faster but for now it's an example. Usage is script url > file.

Example script:

#!/usr/bin/env bash
#
# Download a file in pure bash.

download() {
    IFS=/ read -r _ _ host query <<< "$1"

    # Send the HTTP request.
    exec 3<"/dev/tcp/${host}/80"; {
        printf '%s\r\n%s\r\n\r\n' \
               "GET /${query} HTTP/1.0" \
               "Host: $host"
    } >&3

    # Strip the HTTP headers.
    while IFS= read -r line; do
        [[ "$line" == $'\r' ]] && break
    done <&3

    # Output the file.
    nul='\0'
    while IFS= read -d '' -r line || { nul=""; [[ -n "$line" ]]; }; do
        printf "%s%b" "$line" "$nul"
    done <&3

    exec 3>&-
}

download "$1"

[131]

The fist loop is reasonably slow as it will just drop a sane amount of headers, i can't understand how the 2nd loop (a simple cat !!) can be so complicated (hence, slow i guess)

[dylanaraps]

Bash is slow at file IO and it doesn't handle binary data very well. I'm sure it can be optimized but I have some doubts as to whether or not this will ever be faster than wget/curl.

[131]

According to the "bash bible" - yours :p a simple cat alternative might be

file_data="$(<"file")"

Yet i cannot make this work with my design, but i do not understand why

[dylanaraps]

cat handles binary data correctly iirc, bash doesn't. What causes a larger problem is that bash handles binary data and null bytes differently depending on which version you're using (In 4.4+ null bytes are skipped and never reach the variable).

[darnir]

All the other examples here make sense and can often be faster than invoking another program. However, in the case of networking, I think it makes sense to depend on the binaries, both for useability and performance.

In the case of wget / curl replacements, all of these only work when you have a HTTP endpoint. This code is not going to work for HTTPS.