danmindru / cncnet-map-downloader.js

Downloads & unzips all cncnet maps for a given command and conquer game (i.e. Yuri's Revenge)

cncnet-map-downloader.js ⬇️💻

Downloads all maps from cncnet.org into a directory of your choice.

Usage

Grab the latest binary release on the release page 💽.

Double-click your favorite executable or run the executable through the command line. You will be guided through the options (see demo).

• By default, it only downloads Yuri (yr) maps, but can be configured to download other types. See config below.
• If the process is somehow killed, it will resume downloading maps that haven't been downloaded yet (skips previously downlaoded maps).
• It sorts maps in directories by name (alpha numeric).
• It checks for duplicate maps (size, id, sha).

Demo

Windows preview	Mac preview

Build it yourself

Clone the repo, then run npm install.
You can start a download with npm run download. Downloads can be configured in package.json under scripts.

Configuration

You can configure the downloader through enviornment variables.

• MAP_AGE - the age of the map in months (default: not specified - will get all). Example: 12 will get maps that are 1 year old and newer.
• GAME_TYPE - the type of maps to download (default: yr). Available: ts, ra, d2, td, dta.
• DESTINATION_DIR - the location to download files (default: cncnet-maps).

Configuration examples

• Get all Yuri's Revenge / Red Alert 2 maps in a custom directory

DESTINATION_DIR=my-yuri-maps ./cncnet-map-downloader-js-x-x-x-macos

• Get all Red alert maps

GAME_TYPE=ra ./cncnet-map-downloader-js-x-x-x-macos

• Get all Tiberian Sun maps, not older than 1 year

GAME_TYPE=ts MAP_AGE=12 ./cncnet-map-downloader-js-x-x-x-macos

Dev config

• RUN_UNPACKAGED - run the script in unpacked mode[1] (default: false; see npm run download in package.json for an example).
• DEBUG - print additional debug info.

[1] essentially, run the non-binary script. Requires node v12.9+ & npm install

About the method

Uses the cncnet search endpoint to retrieve all existing maps, then requests each map zip individually with a delay to be nice on the server.

Previously, scarping was considered, but not necessary at this point in time.

Debugging

Run

DEBUG=true MAX_NUMBER_OF_MAPS=5 REQUEST_DELAY=5000 MAP_AGE=1 RUN_UNPACKAGED=true GAME_TYPE=yr DESTINATION_DIR=./cncnet-yuri-maps node --inspect-brk ./scripts/main.js

And open chrome://inspect to debug the node process.

Other art

Check out the cncnet search, where you can search and download maps.

cncnet-map-downloader.js's Issues

Improve DX with husky, prettier

Better way to deal with dupes, versions

Currently dupes can pile up because some maps get tweaked an named i.e v2.
Others have dates in the name.
Should somehow manage this better.

Make age configurable

Update outdated maps

On script re-runs, maps can be checked by date and replaced instead of skipped.

This requires the date to be in the file name of the map.

Allow defining a 'skip list' / filter out tests/betas

Should be able to skip maps that are known to have issue (partial name, sha)

Nicer UX of progress

Use a loader (spinner) and possibly progress bar for promise chain method.

For reference, available game types

• td
• ra
• d2
• ts
• dta
• yr (default)

Remove empty dirs

It can happen during duplicate removal

Windows screenshot

Update deps & docs

Do not exit on windows

Windows terminal window is closed when done. Would prefer to have to press a key instead

Skip already downloaded maps

This makes it possible to pick up downloads if the script fails.

Mac screenshot

Unsort existing files

Essentially flatten existing dirs

Implement scraper

Demos

linux.zip
windows.zip

Use github CI to build binaries

Use more actions

• publish automatically with action
• add release & artifacts automatically with action (is it possible)?
• publish downloads to gh-pages through action?

Idea: cncnet-map-indexer

is there a good way to index / sort maps by types, creators, etc?

Write exit results to file

md? html?

Mark maps that crash & skip

Use prompts for better UX

i.e. to pass in env vars, chose game

Handle unarchive rejections

Better progress feedback

possible flow:

• looking for existing maps
  • no maps found
• found x maps, skipping download (could be a prompt y/n)
• write finish log to file:
  • date, size, no of maps downloaded, no of maps failed, no of maps skipped?

Set options through cli input

in addition to env variables, there should be a user friendlt way to set options during startup

Sort in folders by name

Speed up initial existing file checks

Startup is a bit slow for a few thousands existing files.

Skip duplicates

Need some sort of smart duplicate detection.
Many maps have the same name with v1, v2, etc

Upload to s3

resulting maps could be uploaded to s3 periodically