curl / wcurl

a simple wrapper around curl to easily download files

Curl is a command-line tool for transferring data specified with URL syntax. Find out how to use curl by reading the curl.1 man page or the MANUAL document. Find out how to install Curl by reading the INSTALL document.

libcurl is the library curl is using to do its job. It is readily available to be used by your software. Read the libcurl.3 man page to learn how.

You can find answers to the most frequent questions we get in the FAQ document.

Study the COPYING file for distribution terms.

Contact

If you have problems, questions, ideas or suggestions, please contact us by posting to a suitable mailing list.

All contributors to the project are listed in the THANKS document.

Commercial support

For commercial support, maybe private and dedicated help with your problems or applications using (lib)curl visit the support page.

Website

Visit the curl website for the latest news and downloads.

Git

To download the latest source from the Git server, do this:

git clone https://github.com/curl/curl.git

(you will get a directory named curl created, filled with the source code)

Security problems

Report suspected security problems via our HackerOne page and not in public.

Notice

Curl contains pieces of source code that is Copyright (c) 1998, 1999 Kungliga Tekniska Högskolan. This notice is included here to comply with the distribution terms.

Backers

Thank you to all our backers! 🙏 Become a backer.

Sponsors

Support this project by becoming a sponsor.

wcurl's Issues

Add `--compressed` to default

I think it should be the default

[idea] Content-Disposition Header for file name

As I did not see it in the script,
Would it be possible for wcurl to possibly read the Content-Disposition Header to autogenerate the filename ?

Some endpoints especially API endpoints may end on "/download" but respond with a Content-Disposition Header like

attachment; filename="File-From-Download.txt"; filename*=UTF8''File-From-Download.txt

What's the default protocol if none is specified?

In the README, I see examples like:

Download a single file:
wcurl example.com/filename.txt

I'm wondering whether this uses https://example.com/filename.txt if no scheme is given. Perhaps the example could indicate that?

Use a pronounceable name? How about wurl?

Thanks for this project, I can sympathize with the rationale because I never managed to learn the syntax to make curl do what wget does either. The name is really hard to pronounce though, and command line completion is stuck on the well-know Unix tool wc. Did you consider renaming this to something else? How about wurl? The two-letter prefix wu doesn't complete to anything on my machine, so presumably no too common Unix tool clashes.

I'm sure you already thought about this, but I didn't see a issue about it for having a public discussion on it, so here is :)

Support -V and --version flags

With wcurl 2024-07-02:

% wcurl --version
wcurl: unrecognized option `--version'

% wcurl -V    
wcurl: invalid option -- V

With wget 1.24.5:

% wget --version
GNU Wget 1.24.5 built on darwin21.6.0.

-cares +digest +gpgme +https +ipv6 +iri +large-file +metalink +nls 
+ntlm +opie +psl +ssl/gnutls 

Wgetrc: 
    /opt/local/etc/wgetrc (system)
Locale: 
    /opt/local/share/locale 
Compile: 
    /usr/bin/clang -DHAVE_CONFIG_H 
    -DSYSTEM_WGETRC="/opt/local/etc/wgetrc" 
    -DLOCALEDIR="/opt/local/share/locale" -I. -I../lib -I../lib 
    -I/opt/local/include 
    -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk 
    -DNDEBUG -pipe -Os -std=c99 
    -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk 
    -arch x86_64 
Link: 
    /usr/bin/clang -DNDEBUG -pipe -Os -std=c99 
    -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk 
    -arch x86_64 -L/opt/local/lib -Wl,-headerpad_max_install_names 
    -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk 
    -arch x86_64 -L/opt/local/lib -lproxy -L/opt/local/lib -lmetalink 
    -L/opt/local/lib -lpcre2-8 -L/opt/local/lib -lidn2 -L/opt/local/lib 
    -lnettle /opt/local/lib/libgnutls.dylib -L/opt/local/lib -lz 
    -L/opt/local/lib -lpsl -L/opt/local/lib -lgpgme ../lib/libgnu.a 
    /opt/local/lib/libiconv.dylib /opt/local/lib/libintl.dylib 
    -Wl,-framework -Wl,CoreFoundation -Wl,-framework -Wl,CoreServices 
    /opt/local/lib/libunistring.dylib 

Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://www.gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Originally written by Hrvoje Niksic <[email protected]>.
Please send bug reports and questions to <[email protected]>.

and wget -V outputs the same.

Keeping README.md and wcurl.1 in sync

When I initially submitted https://salsa.debian.org/debian/wcurl/-/merge_requests/3 to fix a problem in wcurl.1 I had not realized that the contents of wcurl.1 were incorporated into README.md. I later corrected it by adding the same change to README.md.

The same problem happened in e4a7f67 and has not yet been corrected. The changes made to wcurl.1 in that commit have yet to be made to README.md.

Having to make changes to both files independently leads to the files getting out of sync. What is the intended method by which they should remain in sync? Is there a script that formats wcurl.1 into Markdown and replaces what's in README.md? If so could that script be added to the repo, and could documentation covering this and any other developer-specific topics be added to the repo?

Two different license files

There are now two different curl license files: the original one at LICENSE and the newly added one at LICENSES/curl.txt. It should be possible to have a single license file that satisfies both REUSE requirements and allows GitHub to identify and display the license, in which one of these files is a symlink to the other. I forget which direction the symlink needs to be made to make both parties happy but I have seen it done in another repository.

URL without filename fails

With wcurl 2024-07-02:

% wcurl https://github.com/Debian/     
curl: Remote file name has no length
curl: (23) Failed writing received data to disk/application

However with wget 1.24.5:

% wget https://github.com/Debian/
--2024-07-04 09:53:47--  https://github.com/Debian/
Resolving github.com (github.com)... 140.82.114.3
Connecting to github.com (github.com)|140.82.114.3|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html’

index.html                                                             [  <=>                                                                                                                                                          ] 245.03K  1.17MB/s    in 0.2s    

2024-07-04 09:53:48 (1.17 MB/s) - ‘index.html’ saved [250913]

wcurl tries to continue downloading an existing file

When wget 1.24.5 encounters an existing file it appends a dot and a number:

% touch index.html
% ls -l index.html*  
-rw-r--r--  1 rschmidt  wheel  0 Jul  4 12:52 index.html
% wget github.com/index.html
URL transformed to HTTPS due to an HSTS policy
--2024-07-04 12:52:46--  https://github.com/index.html
Resolving github.com (github.com)... 140.82.114.3
Connecting to github.com (github.com)|140.82.114.3|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html.1’

index.html.1                                                           [ <=>                                                                                                                                                           ] 126.43K   771KB/s    in 0.2s    

2024-07-04 12:52:47 (771 KB/s) - ‘index.html.1’ saved [129465]

% ls -l index.html*         
-rw-r--r--  1 rschmidt  wheel       0 Jul  4 12:52 index.html
-rw-r--r--  1 rschmidt  wheel  129465 Jul  4 12:52 index.html.1
% rm index.html*            

wcurl 2024-07-02 just overwrites the existing file which may be unexpected:

% touch index.html
% ls -l index.html*
-rw-r--r--  1 rschmidt  wheel  0 Jul  4 12:51 index.html
% wcurl github.com/index.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  126k    0  126k    0     0   222k      0 --:--:-- --:--:-- --:--:--  334k
% ls -l index.html*          
-rw-r--r--  1 rschmidt  wheel  129466 Jul  4 12:51 index.html
% rm index.html*

Fix this by adding --no-clobber to the curl flags that get added by default:

% touch index.html          
% ls -l index.html*
-rw-r--r--  1 rschmidt  wheel  0 Jul  4 12:54 index.html
% wcurl --opts=--no-clobber github.com/index.html 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  126k    0  126k    0     0   214k      0 --:--:-- --:--:-- --:--:--  798k
% ls -l index.html*                              
-rw-r--r--  1 rschmidt  wheel       0 Jul  4 12:54 index.html
-rw-r--r--  1 rschmidt  wheel  129478 Jul  4 12:54 index.html.1

Filenames are not percent-decoded

wget 1.24.5 percent-decodes filenames:

% wget https://packages.macports.org/itstool/itstool-2.0.7_2%2Bpython312.any_any.noarch.tbz2
--2024-07-04 13:35:28--  https://packages.macports.org/itstool/itstool-2.0.7_2%2Bpython312.any_any.noarch.tbz2
Resolving packages.macports.org (packages.macports.org)... 146.75.106.132
Connecting to packages.macports.org (packages.macports.org)|146.75.106.132|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 19316 (19K) [application/x-bzip2]
Saving to: ‘itstool-2.0.7_2+python312.any_any.noarch.tbz2’

itstool-2.0.7_2+python312.any_any.noarch.tbz2                       100%[==================================================================================================================================================================>]  18.86K  --.-KB/s    in 0.001s

2024-07-04 13:35:28 (33.7 MB/s) - ‘itstool-2.0.7_2+python312.any_any.noarch.tbz2’ saved [19316/19316]

% ls -l itstool*
-rw-r--r--  1 rschmidt  wheel  19316 Jan 19 09:52 itstool-2.0.7_2+python312.any_any.noarch.tbz2
% rm itstool*

wcurl 2024-07-02 doesn't:

% wcurl https://packages.macports.org/itstool/itstool-2.0.7_2%2Bpython312.any_any.noarch.tbz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 19316  100 19316    0     0   287k      0 --:--:-- --:--:-- --:--:--  285k
% ls -l itstool*
-rw-r--r--  1 rschmidt  wheel  19316 Jan 19 09:52 itstool-2.0.7_2%2Bpython312.any_any.noarch.tbz2
% rm itstool*

Requires util-linux version of getopt

As pointed out by tssva on Hacker News, wcurl 2024-07-02 requires the util-linux version of getopt. If using the version of getopt that comes with macOS, it does this:

% wcurl --help             
curl: option --longoptions: is unknown
curl: try 'curl --help' or 'curl --manual' for more information

Create tags for released versions

While I know wcurl is included in the debian curl package, it was announced to the curl-distros list and other distributions were invited to package it as well. To do that, other package systems would prefer to download an official version from a tag rather than from some random commit.

Please create tags for all of the released versions:

• 2024-05-14 = 110ec46
• 2024-06-26 = 2deb288
• 2024-07-02 = e4a7f67