unreal-archive / unreal-archive

Unreal series games mod community content indexer, static site generator, and related tooling.

Unreal Archive

Scans, categorises and produces metadata for Unreal, Unreal Tournament, Unreal Tournament 2003/4, and Unreal Tournament 3 content, and builds a static browsable website of the content, currently published at https://unrealarchive.org/.

Requirements

• Java JDK 21 for building

Tested with both OpenJDK 21 on Linux and Azul's Zulu Java 21 on Windows.

Building

The project is build with Gradle. The provided gradlew wrapper may be invoked as follows to produce an executable file:

On Linux

./gradlew jlink

To run, execute:

./build/unreal-archive/bin/unreal-archive

On Windows

gradlew.bat jlink

To run, execute:

build\unreal-archive\bin\unreal-archive.bat

Components and Modules

The project is arranged into several modules, and published as Java Modules so components may be reused by other software projects.

These are:

• common: the usual assortment of shared boilerplate code and utility classes and functions.
• content: the definition of the content types, Maps, Map Packs, Skins, Game Types, etc. These are the elements which are serialiased to and from YAML format stored in the archive data repository, and projects wishing to work with the Unreal Archive data types should re-use these if possible. In addition, the code for loading, saving, and generally working with the data sets are here too.
• storage: the remote storage/mirror implementations, for S3 and Azure and associated interfaces.
• www: the code and templates responsible for generating the static website output.
• root project: the content indexing and mirroring implementation and management. This will likely move into its own module(s) later.

Usage and Functionality

TODO: complete this section

Run with no arguments to see input arguments and additional help for each command.

Browsing and Information:

• ls: List indexed content filtered by game, type or author.
• filter: List indexed content filtered by attribute=value pairs.
• show: Show data for the content items specified.
• summary: Show stats and counters for the content index.

General Content Management

• scan: Dry-run scan the contents of files or paths, comparing to known content where possible.
• index: Index the contents of files or paths, writing the results to the content path.
• edit: Edit the metadata for the provided.
• set: Convenience, set an attribute for the provided. Eg: set <hash> author Bob.
• sync: Sync managed files' local files to remote storage.

Gametype Management

• All commands prefixed by gametype:
• init <game> <gametype name>: Create a skeleton gametype file structure.
• locate <game> <gametype name>: Show the local file path to the provided gametype.
• index <game> <game type name> <release name>: Indexes the content of the release specified.
• add <game> <game type name> <release name> <file>: Convenience, which adds a gametype if it does not yet exist, adds a release, and indexes the release. A sync command afterwards is still required to sync download files to mirrors.
• addmirror <game> <game type name> <release name> <url>: Adds a secondary mirror to the gametype specified.

Mirrors:

• local-mirror: Create a local mirror of all file content.
• mirror: Create a remote mirror of all file content and add mirror links.

Utilities and Tools:

• unpack: Unpack the contents of a umod file to a directory.
• install: Unpack the contents of a file, URL or hash, and place within an Unreal game's standard directory layout (Maps, System, Textures etc).

Website Build:

• www: Generate the HTML website for browsing content.

Content Submission and Indexing Pipeline

• For any given file, a Submission instance is created.
  • If the file's directory contains an _override.yml file, that file is loaded as a SubmissionOverride instance and associated with the Submission.
• The Submission is used to instantiate an Incoming instance, which contains additional information like the file's SHA1 hash.
• If the content is new (hash is not yet known), the file is unpacked onto disk (via system-installed archiving tools 7z and unrar).
• The Incoming instance is passed to ContentType.classify() for content type classification, where the type of content is determined (see Content Identification).
• If the content type can be identified, the content type will be used as a factory to instantiate a new Content element of the appropriate type with common content information and metadata.
• The content type will again be used as a factory to instantiate an IndexHandler instance, which will take the new Content instance and populate it with more type-specific metadata extracted from the actual content being indexed (eg. extract author and screenshot information from map packages, or determine whether or not a skin has team colours defined).
• Once indexed, the result is returned via a Consumer<>, and this is then "checked into" the ContentManager, which will write the content metadata to disk in YAML format, as well as sync the package and any additional files to the nominated data store.

Most of this process uses Consumer<>s for providing feedback, with the intention of eventually being able to make this process more parallelised for better performance. Currently the performance is "good enough" that such parallel behaviour is not quite worth the implementation time.

Mirroring

If you have storage capacity available and would like to contribute some of it as a public mirror for archive content, the following steps should be taken.

1. Fork and clone the unreal-archive-data repository.
   • This dataset will be updated during the mirroring process.
2. Download or build the unreal-archive project binary
3. Execute unreal-archive mirror with the following command-line options:
   • --content-path=/path/to/unreal-archive-data
   • --store=[dav|s3|az] and appropriate configuration and credentials.
   • --concurrency=3 with an appropriate concurrency value for your bandwidth and processing power (3 is default)
   • --since=yyyy-mm-dd [optional] - only mirror content added after the date specified
4. Wait while the mirror process completes. If you want to abort, just Ctrl+C the process and whatever content has been mirrored so far can be used as-is or resumed later.
5. Ensure that the URLs added to the data files are publicly accessible by doing a spot-check on some of them.
6. Once the mirror is complete (or partially complete if you're only doing a partial mirror) and some URLs have been eyeballed, git commit your changes to the unreal-archive-data repository and push to your fork, then use GitHub to open a Pull Request to the main repository master branch.

Storage Configuration

A storage backend must be configured in order to store content during indexing.

When interrogating, mirroring or downloading content, no store needs to be specified, the default no-op store (--store=nop) will be used.

Environment variables may be used for configuration, eg.: replace --s3-key-images=key with S3_KEY_IMAGES=key.

HTTP/DAV

This is mostly for testing purposes, but in some circumstances may be useful.

• --store=dav
  • --store-[images|attachments|content]=dav
• --dav-url=http://hostname/path/
  • --dav-url-[images|attachments|content]=...

S3 Bucket Storage

Supports S3-compatible storage implementations.

• --store=s3
  • --store-[images|attachments|content]=bs3
• --s3-key=key-id
  • --s3-key-[images|attachments|content]=key-id
• --s3-secret=secret
  • --s3-secret-[images|attachments|content]=secret
• --s3-bucket=bucket-id
  • --s3-bucket-[images|attachments|content]=bucket-id
• --s3-endpoint=https://s3.amazonaws.com/
  • (provide the root of your storage API)
  • --s3-endpoint-[images|attachments|content]=https://s3.amazonaws.com/
• --s3-url=https://__BUCKET__.s3.eu-west-2.amazonaws.com/__NAME__
  • (provide the public URL of your storage bucket in the appropriate region. __BUCKET__ and __NAME__ will be replaced by the bucket and uploaded filenames respectively)

Note: Amazon S3 bucket policy to allow public downloads:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicRead",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::testing-unreal-archive-mirror/*"
        }
    ]
}

Azure Blob Storage

• --store=az
  • --store-[images|attachments|content]=az
• --az-acc=storage-account-name
  • --az-acc-[images|attachments|content]=storage-account-name
• --az-container
  • --az-container-[images|attachments|content]=container-name
• --az-sas
  • --az-sas-[images|attachments|content]="shared-access-signature"
• --az-endpoint (Optional, default="blob.core.windows.net")
  • --az-endpoint-[images|attachments|content]=endpoint-suffix

Note: The shared access signature (SAS) should be created with full permissions to the specified container. Use double quotes around the SAS when specifying it via the command line.

unreal-archive's Issues

Friendly string for release dates

Release dates are rendered as "2005-03" on content pages, whereas it may be more friendly to display them as "March 2005"

Map theme/keyword classification

For many maps (at least those using the standard texture sets), we can attempt to classify them based on exports, and maybe even props present in the map. Possibly even try to inspect things like slime or lava zones, or indoor and outdoor.

Process:

• create list of all standard (and maybe popular non-standard) texture packages
• categorise texture packages with some keywords, for example, "ancient", "castle", "nali", "outdoor", "mines", etc.
• inspect exported brushes or polys of a map, and count up texture package usage
• based on frequent texture use, create a list of matching keywords
• potentially also inspect exported decorations and zones to see if they hold any value and possible weighting on categorisation

Unreal Tournament 3 Domination Gamemodes

Classic Domination and just Domination are the same gamemode and maps from both sections go into the same pool in UT3 when installed (I can confirm this myself, as I have (or had) all of those installed). I suggest merging sections. The name should probably be just Domination.

Install utility command

Add an install command for downloading and installing content into the correct directories of an Unreal, Unreal Tournament, or Unreal Tournament 2004 installation.

Sample usage:

$ ./unreal-archive install 16d818a5359df7a6100643dddbb08d6ecfdf76a2 018d52453826aa1bc0e2bbc36f64106eaeea1181 ~/games/ut/

This would download archives for the content identified by the variable number of content hash arguments provided, and unpack their content into the correct sub-directories under the last argument, which should be a path.

An "is-a-hash" check could be performed on each argument, and any which are not hashes may be assumed to be files, allowing clean installation/unpacking of local files:

$ ./unreal-archive install ./DM-CoolBeans.zip ~/games/ut/

A wish: add sorting

Your site is awesome, you have a lot of maps.
i'd like to sort maps in a group (e. g. UT >CTF) by downloads or players.
Also, unrealplayground doesn't have maps for downloading. When i select it for download site, i always get this: "Invalid Page URL. If this is an error and the page should exist, please contact the system administrator and tell them how you got this message. "

Missing footer on latest additions

Looks like the footer content is not included on the latest content additions page.

Map pack maps duplicated with zips in zip

When a map pack contains a bunch of zip files, the map entries are duplicated.

See unreal-archive/unreal-archive-data#1833 and associated map pack for reference.

Miscellaneous Improvement Ideas

Good time of the day. Unreal Archive is an incredibly useful website in the current web landscape, were most Unreal Tournament modding websites have fallen. Hopefully, it will have a lot more longevity, than its predecessors.

I would also like to leave a couple more suggestions that might be beneficial to the website’s goal of archiving mods (I hope it’s fine, that I post them here):

1. Announcer Voicepack support for UT2004. Currently, whenever I try to upload any of them, I get: “No recognisable content found in file announcer.rar”.
2. Monster check for UT2004 mutators, if possible. More information on what’s inside mutators, like the following will be very useful (https://unrealarchive.org/mutators/unreal-tournament-2004/H/hl2monsters_84fd5bee.html)
3. Dark/Night Theme. Could be toggled on and off by clicking the Unreal logo at the top of the page.
4. Like I’ve said previously in UT3 discussion: VCTF and WAR map sections have more than 100 files each, should be enough for maps to be sorted alphabetically.
5. It doesn’t happen particularly often, but I tend to get “No recognisable content found in file mod.rar” error in perfectly ordinary files for UT2004. Here are a few examples, that I was not able to upload: https://gamebanana.com/mods/233123 and http://www.mediafire.com/download/yb6xduqo73dgzht/DM+Ariza+3.zip
6. Low priority suggestion, but maybe map info could also list music track(s) used in the level?

Finally, I want to say, that I would very much welcome more Game Types & Mods for UT2004 (and, perhaps, even UT3 at some point). Personal picks would be Jailbreak, CTF4/TDM4 and Chaos UT2: Evolution.

Best regards!

Website fails to detect some UT3 models if there are multiple UTCustomChar.ini files

Title. The following file can be used as an example, it has two custom factions (COG and Locust), with characters from the former missing in the model list (they can be seen on screenshots): https://unrealarchive.org/models/unreal-tournament-3/S/sniper_9d1bbbd4.html

Attempt local SHA1 hash of content before uploading

Perhaps we can SHA1 hash and lookup content before uploading.

This would allow validation of at least some duplicates before wasting time and resources uploading to the server.

submit/index.html should be able to load and hash files selected by the user, then perform some sort of request to the backend to look up content by hash.

This would likely also involve generating a new file structure based on hashes, perhaps with meta redirects to the appropriate content pages.

References:

• https://stackoverflow.com/questions/18586839/read-file-stream-using-javascript-in-web-browse
• https://gist.github.com/eungjun-yi/11282867

Implement mirror and sync

We can create a local mirror based on content in the repository, but cannot yet create a remote mirror easily.

Implement the ability to create and synchronise a full content mirror, adding the mirror URLs to the content downloads.

Recently added content

To promote discovery of new things, create a "recently added" page listing the last N (100?) content things added.

Symbol is eating entire words... with unfortunate results.

This was detected yesternight by @SeriousBuggie:

However the actual code of such maps is this:

name: "CTF-XV-ToX-STORAGEWARS"
author: "Toxic made · edited by EvilGrins"

name: "DM-XV-monsters-Brikby"
author: "Moron made · edited by EvilGrins"

I've sent a PR to unreal-archive-data (unreal-archive/unreal-archive-data#1377) thinking it was just those two maps. However, it turns out that EvilGrins has been using that symbol for quite some time. For example, in CTF-XV_FortressX, the map appears credited as "Deviate EvilGrins & ANUBITEK". HOWEVER, when taking a look at the code:

name: "CTF-XV-Fortress[X]"
author: "Deviate made · edited by EvilGrins & ANUBITEK"

That "·" symbol is screwing with a lot of things.

Dependency Resolution

A convenient feature, as well as something useful for #15, would be the ability to perform deep dependency resolution, if possible. May depend on expanding Import support in https://github.com/shrimpza/unreal-package-lib as well.

Given a Content package:

• find which other packages (USX, UTX, etc) it imports, find which resources are used from those packages
• if the dependent package is not within the archive of the content, we can search other content for a matching package
• even if we find matching packages, scan the Exports of those packages for the resources used by the content package being analysed
• the end result is a "best match" dependency package containing the resources required

Considerations:

• may need to retrieve packages during resolution
• found packages may also require further dependency analysis
• constrain package resolution to the game/engine of the package being analysed
  -require a full list of stock packages for each game, which can be excluded from resolution attempts

Reference: https://unrealadmin.org/forums/showthread.php?t=26321

Unreal Tournament 3 Gamespy Patch

A topic, that I brought up here: (#59), that I think should be highlighted on it's own. Gamespy Patch is crucial for Unreal Tournament 3 functionality for both retail and digital versions of the game. Uploading it to Unreal Archive would be the most welcome addition.

Image filenames overlapping

Image files are stored without any hash in their filenames, leading to same-name content sharing or overwriting eachother's files.

Image files should include the first 8 characters of the content hash on the end of the filename, as per content files themselves.

Existing same-name content will have to be re-indexed to correct their images.

[UT99][UT2004] Domination and Double Domination

https://github.com/unreal-archive/unreal-archive/blob/master/src/main/java/net/shrimpworks/unreal/archive/content/maps/GameTypes.java

Officially, both UT99 and UT2004 use the DOM-prefix for Domination and Double Domination respectively. (UT99's Double Domination mod does use DDOM, though).

However, in UT2004, Double Domination appears as simply "Domination", which can cause confusion considering there's an actual Classic Domination mod which uses the cDOM- prefix.

Filter deleted items from latest content listing

deleted: true content still shows up on the "Latest Content Additions" page, but it probably shouldn't.

Error when trying to upload anything regardless of the file itself

Title. Here's an example of the error in question:
Failed to index content in file ShowDamage.zip: java.lang.RuntimeException: java.io.IOException: Failed to read YAML file /tmp/ua-submit-data-17931778450830106458/content/Unreal Tournament 3/Maps/Capture The Flag/C/2/6/35533d/ctf-citius_[2635533d].yml

Remove Variations from Recent Additions

If indexing resulting in attaching a bunch of content as variations of existing content, don't list them in the Latest Additions view.

Mirroring: javax.net.ssl.SSLException: Connection timed out

And another one ;-) This time the connection timed out:

Failed to download content /[...]/Mirror/Unreal Tournament/Mutators/U: javax.net.ssl.SSLException: Connection timed out (Read failed)
[...]
Failed to download content /[...]/Mirror/Unreal/Maps/Unknown/S: javax.net.ssl.SSLException: Connection timed out (Read failed)

Concurrency is set to 3 on a 400MBit line, not sure where this is coming from. Since the CPU is rather slow, I reduced the concurrency to 2 and hope it goes away.

However, it is not clear if the download will be retried, or if some data will be missing from the mirror and some user interaction is required.
I think the error message could be improved; e.g.

• "The download will be retried later"
• or "Please re-run the mirror task to try again"

The later case could be signaled via a special exit code (e.g. 2), so a cron job knows it has to try again.

[Brainstorming] How to deal with multiple authors...

It's not uncommon for many people to work in the same map. Take for example CTF-XV-Battlegrounds. It has four authors:
Metalfist (original)
Maeze (original)
EvilGrins (XV conversion)
Nelsona (XV conversion pathing)

This is the code of said map:

game: "Unreal Tournament"
name: "CTF-XV-BattleGrounds"
author: "Metalfist & Maeze (original), EvilGrins (XV), Nelsona (pathing)"

How about the following instead:

game: "Unreal Tournament"
name: "CTF-XV-BattleGrounds"
author: "Various"
author1: "Metalfist"
author1role: "Original"
author2: "Maeze"
author2role: "Original"
author3: "EvilGrins"
author3role: "XVehicles conversion"
author4: "Nelsona"
author4role: "XVehicles botpathing"

Unable to upload Unofficial Greed Maps by Odedge (size cap?)

Title. I sense an archive size cap - map pack exceeds 500 mb. I really do want to upload it, since Odedge's site is no more.

Add fo each section output sorted by date

Will be good if in each section exists output sorted by date addition.
This useful for track newly added files.

For example enter here:
https://unrealarchive.org/maps/unreal-tournament/assault/index.html
And able choose view by date, where 100 maps per page (for example).
For make link persistent - newest stuff must be last. If this not matter, newest can be first.
Also if content able be deleted it can shift on pages.

https://unrealarchive.org/latest/index.html not enough. Too small, All in one.

Release date detection

Release date detection is currently broken, as it looks at the date of any Unreal package within an archive, rather than the actual content element being indexed.

Collect addition info about maps for later make sort on it

Will be good if for each map collect and store some stat info. I mean count of

1. Brushes
2. Movers
3. Light
4. AmbientSound
5. Decoration
6. Texture

After that maps can be sorted by this criteria.
In fact I think is most important metric is count light actors.
Exist very big correlation between count Light actors and quality of maps, independent from map size and presence brushes.

Here simple example of Assault maps, where you can see correlation between Light count and general quality of map (top by Light count):

For counting I use ucc listobjects PKG_NAME TYPE which count not only exact TYPE but any subclasses too.
If this hard to implement, I think for Light enough even simple count for few well know Light classes.

Lots of files that can't be added through the uploader

As requested several months ago, I've uploaded all the patches for U1, UT and UT2004 in a single location.

Unreal: https://mega.nz/folder/zA4EBARK#-LQ9asoGdx8AyjS-YP8bhw
UT: https://mega.nz/folder/PIhSWCiJ#N2mBUyktnmiW0uQzCT6XVA
UT2004: https://mega.nz/folder/iQAzQaKS#qgygdVIyC7QyKgjlUfFlHw

I've also uploaded the mods I've been saving throughout the years, classified by mutators, gametypes and TCs:

UT99: https://mega.nz/folder/rIIRgSAJ#sHiRTWJ8bC9xje5q-Y6Y4g
UT2004: https://mega.nz/folder/7IATWKzS#45DJl-fJ8yqgDwoSGSCbyA

Can these be added to the Archive?

Trim metadata

There are several metadata elements unlikely to actually be used, like initial index date (just use a single updated date, when something was added isn't really useful), as well as several things in downloads.

Removing a bunch of things will reduce storage requirements and speed up loading.

Cursor not set in search field on search page

Reproduce:

1. Open index page.
2. Click search on top right.
3. Start type text.

Expected result: Typed text goes to search field.
Actual result: Typed text goes to nothing, because you need put cursor to search field firstly.

Lightbox/Screenshot Viewer

Many screenshots are available in a size bigger than that which is shown on the content detail pages.

Open up the images in full size in a lightbox-like viewer.

Gametypes and Total Conversions support

Gametypes become a bit more difficult to mange due to their higher reliance on versioning than maps, mutators, etc; and they also incorporate many elements of all the other content types, and often involve more intricate packaging, sometimes in proprietary format Windows executable installers.

I believe most gametypes can be easily found by looking for TournamentGameInfo and friends definitions in .int files (for UT, at least). However, this largely tells us nothing about the gametype.

I think this needs to be highly hand-curated, with perhaps some initial metadata identification and collection via indexing used to help the process along. This content may be managed and defined as somewhat as a hybrid between the current content and "managed content" - with some additional specific metadata and the ability to associate a document (more than a one-liner description/summary).

Some elements of a Gametype content type (probably also applies to TCs and other mods):

• included gametypes, with associated map types if applicable (indexed)
• included maps, if applicable (indexed)
• included mutators, if applicable (indexed)
• link to Maps and Map Packs pages for the gametype, if applicable
• associated document with original description, unarchived homepage content, something along those lines
• screenshots
• credits, if a credits listing is available
  • larger mods usually have more than one author
• homepage link, otherwise link to best matching archive.org content

Logo link not lead to website root

Reproduce:

1. Open https://unrealarchive.org/maps/unreal-tournament/xvehicles/C/index.html
2. Click on website logo.

Expected result: You move on website root: https://unrealarchive.org/
Actual result: You move to same page as before: https://unrealarchive.org/maps/unreal-tournament/xvehicles/C/index.html

Dependencies are not installed?

I tried creating a mirror on my Arch Linux box, but cloning the two repos and then running

./gradlew execJar

in the unreal-archive git results in

lib:unreal-archive$ ./gradlew execJar
Downloading https://services.gradle.org/distributions/gradle-5.4.1-all.zip
..............................................................................................................................

Welcome to Gradle 5.4.1!

Here are the highlights of this release:
 - Run builds with JDK12
 - New API for Incremental Tasks
 - Updates to native projects, including Swift 5 support

For more details see https://docs.gradle.org/5.4.1/release-notes.html

Starting a Gradle Daemon (subsequent builds will be faster)

> Task :compileJava
/mnt/raid/archive/External/UnrealArchive/unreal-archive/src/main/java/net/shrimpworks/unreal/archive/content/IndexUtils.java:77: error: cannot find symbol
                                        Named pkg = ((Import)shotResolved).packageName.get();
                                                                          ^
  symbol:   variable packageName
  location: class Import
/mnt/raid/archive/External/UnrealArchive/unreal-archive/src/main/java/net/shrimpworks/unreal/archive/content/IndexUtils.java:79: error: cannot find symbol
                                                String parentPkg = pkg instanceof Import ? ((Import)pkg).packageName.get().name().name : "None";
                                                                                                        ^
  symbol:   variable packageName
  location: class Import
2 errors

> Task :compileJava FAILED

FAILURE: Build failed with an exception.
[...]

I found your unreal-package-lib, which seems to contain the mssing class - but even though it is listed as a dependency, it is not automatically fetched/built. I tried a little bit, but since I am not familiar with cradle, I could not get it to fetch the dependency.

If you could extend the docs a little bit, that'd be great :)

As a workaround I fetched the latest travis build from https://code.shrimpworks.za.net/artefacts/unreal-archive/latest/; this seems to work well and tbh - at least for underpowered boxes like my NAS - I find this much more comfortable.

Other than that: Great work! I was totally frustrated that a lot of the old content was difficult to find, and preserving this is definitely worthwhile. Thank you! I can't host an online mirror, but after tonight it'll be on an additonal RAID.

Unreal Tournament 3 UT2D Gamemode

A dedicated section for UT2D maps in Unreal Tournament 3 would be a welcome addition. There are currently only three uploaded to Unreal Archive, and while I don't have others on me, there are definitely more out there. Current amount is pretty small, but website has sections with even less maps for other custom gamemodes (Strike Force and CarBall in UT2004), so that shouldn't be an issue. Available UT2D maps can be found here: https://unrealarchive.org/maps/unreal-tournament-3/unknown/index.html

When generating site locally on Windows, invalid characters can occur in file names

When generating the site locally on Windows and images are being downloaded, there is at least one image that fails due to an invalid character in the filename (*):

Failed to download image *SDA_SuperSniper_by_bull*_shot_1.png: java.nio.file.InvalidPathException: Illegal char <*> at index 9: b0dbb7e5_*SDA_SuperSniper_by_bull*_shot_1.png

In Linux, it appears * can exist fine in the filename but not in Windows.

Repack utility command

Implement a utility command to repack existing archives, respecting the expected installation structure.

For example:

$ ./unreal-archive repack 16d818a5359df7a6100643dddbb08d6ecfdf76a2 ./repacks/

This would download the content specified by the provided hash, unpack it, then create a new zip archive in the path specified by final argument, with the zip file named after the content name.

The structure within the .zip file would respect the System, Maps, Textures and other directories expected by the Unreal games.

Content Packs

I think it would be nice to allow the definition of something like a "Content Pack", being a collection of other content.

For example, a collection of mutators and maps which work well together, could be listed as a content pack. The install command (#14) could support installing these as a one-shot, or it could be offered as a repacked download for convenience.

It would offer a nice way of discovering content.

Issues with selecting file types for an upload

Well, just one. It makes no difference: I still get the same error with content not being recognized, no matter the file type I select. I've tried Unreal map pack, UT2004 models, maps and mutators and UT3 model. All have the same result as with auto-detection.

Support for Rune content

It's been requested that Rune content be added.

Since it's a UE1 game, it should be possible to index and store the stuff.

New game and gametype definitions would be required. If there are skins and content types like that, additional investigations into metadata format definitions may be required.

Windows support

Ensure that at a minimum, the mirroring and download functionalities run without issues on Windows, and document the process.

Files Search

Add feature/capability to search for individual files.

Often it's difficult to track down a required dependency, so adding a file search may help.

Screenshots not being found

See map: https://unrealarchive.org/maps/unreal-tournament/deathmatch/A/3/dm-ainum-b5_d0e8be76.html

This map has a Screenshot export, which isn't referenced in its LevelInfo, and which wasn't picked up by the secondary screenshot search (should be searching for any exports named Screenshot).

See why this wasn't found; and potentially re-index all maps with missing screenshots in case they weren't picked up 😭.

[Brainstorming] UT3 support

Regardless of your opinion about the game, it's an undeniable fact that Unreal Tournament III is still part of the Unreal family, and as such has quite the amount of content that should be preserved as much as the content for the other games covered here. There are mods, maps, mutators, that are synonymous with quality (the five CBPs, the many HOLPs, Hourences's "The Ball"... just to name a few). Not to mention, the PS3 version is one of the very few, selected games that can actually support custom content! With the obvious conclusion being that there's quite a lot of content for UT3 on PS3.

At the same time, it's also an undeniable fact that the Unreal Engine 3 has changed quite a lot ever since the previous generation, and according to Shrimp it made its content more difficult to support, which is the main reason it's not supported here.

So, how about we start thinking on how the UA can support UT3?

For starters, ut-files has an entire section about UT3, so the initial content to support is there.

Submission page layout

The submission page doesn't feel great or overly logical to use.

• Switch to a 2-column view:
  • when selecting files, wall of text in one column, selected files in another.
  • when click submit, switch wall of text to the log view, maintaining the list of files that were submitted.
  • perhaps use a grid layout for this and manipulate visible elements by changing the grid columns via javascript.

Front page layout

The front page is too text heavy... It describes what's going on but feels like it makes the content feel secondary.

Rearrange it somehow, or put the About text on its own page perhaps.

Common content URL creation

At the moment URLs for content are created on an ad-hoc bases per content type, within the WWW generator and output for that content type. These URLs also include things like pagination.

This makes it significantly more difficult to link to specific contents from other pages, such as #6.

A wish for your site

Can you sort maps by number of downloads?

Forced filetypes in upload menu should reset in-between uploads to the default auto-detect

Title. This change is needed for convenience sake, so things like this: unreal-archive/unreal-archive-data#2100 won't happen. Yes, it's mostly the problem with me, the uploader, but still.

S3 Data Store

Add datastore implementation and associated options to support writing content to an S3 bucket.

Mirroring: FileAlreadyExistsException

I am currently mirroring, and am seeing some errors regarding duplicates:

No images store specified, this will be necessary for indexing new content. Falling back to no-op store.
No attachments store specified, this will be necessary for indexing new content. Falling back to no-op store.
No content store specified, this will be necessary for indexing new content. Falling back to no-op store.
Loaded content index with 46236 items (195.85GB) in 45.14s
Writing files to /mnt/raid/archive/External/UnrealArchive/Mirror with concurrency of 3
[ 42     / 46236  ] Processed dm-howdys-xmas-2004.zip
Failed to download content /mnt/raid/archive/External/UnrealArchive/Mirror/Unreal Tournament/Maps/DeathMatch/S: java.nio.file.FileAlreadyExistsException: /mnt/raid/archive/External/UnrealArchive/Mirror/Unreal Tournament/Maps/DeathMatch/S/DM-ShaMaNCastle.zip
[ 336    / 46236  ] Processed ctf-echochamber_final.zip
Failed to download content /mnt/raid/archive/External/UnrealArchive/Mirror/Unreal Tournament 2004/Maps/DeathMatch/B: java.nio.file.FileAlreadyExistsException: /mnt/raid/archive/External/UnrealArchive/Mirror/Unreal Tournament 2004/Maps/DeathMatch/B/dm-backyarddd.zip
[ 700    / 46236  ] Processed bt-puff.zip
Failed to download content /mnt/raid/archive/External/UnrealArchive/Mirror/Unreal Tournament/Maps/DeathMatch/C: java.nio.file.FileAlreadyExistsException: /mnt/raid/archive/External/UnrealArchive/Mirror/Unreal Tournament/Maps/DeathMatch/C/dm-cemetarygates.zip
[ 828    / 46236  ] Processed ctf-lavapits.zip

System is a Arch Linux box, mirroring to a btrfs; filesystem is case-sensitive.

It is not clear if this is a problem (e.g. the mirror might be missing data), or if it can be ignored.