About the new playlist file format (tab-delimited CSV) about ludo

[kivutar]

Your remarks are perfectly valid. Ludo wasn't honoring this column anyway, so for now I've removed it.

When the time comes to add History or Favorites I may change the playlist format. But I'm not yet sure how these things should look. For example:

• in History, I may want to display a date of last played
• Favorites may simply be pinned entries at the top of each system playlist

There is no requirement to exactly replicate the UI patterns of RetroArch. I may even change what Ludo looks like before v1.0.0.

But there are two thing that won't change: priority goes to stability fixes over everything else, and the code will have to remain small, I'm ready to sacrifice acclaimed features and platforms to achieve these two goals.

[fmaida]

BTW congratulations on having launched the very first public release of ludOS. I did not know where to write it because there is not yet a channel dedicated to Ludo on the libretro forum.

[RobLoach]

@fmaida Join us on Discord! https://discord.gg/YXYSEQD

[fmaida]

I know that this has been suggested before, but what about JSON?
It could allow you to keep a very flexible format, by adding a removing fields per entry while developing Ludo, without fearing to break the code that read the playlist.

But if I'm not mistaken, you once wrote that JSON is too slow to read for a very long playlist?

Anyway, here's an imaginary playlist file written in JSON, just to make an example:

{
    "title": "Nintendo - Nintendo Entertainment System",
    "defaults": {
        "core_id": "fceumm_libretro",
        "aspect_ratio": "16:10"
    },
    "entries": [
        {
            "id": 3415201714,
            "path": "/path/to/games/Balloon Fight (U) [!].nes.zip",
            "title": "Balloon Fight (USA)",
            "region": "USA",
            "core_id": "nestopia_libretro",
            "last_played": "2019-02-13 08:42:39",
            "pinned": "True"
        },
        {
            "id": 3305800452,
            "path": "/path/to/games/Contra (U) [!].nes.zip",
            "title": "Contra (USA)",
            "pinned": "True"
        },
        {
            "id": 3703128897,
            "path": "/path/to/games/Duck Tales (U) [!].nes.zip",
            "title": "DuckTales (USA)",
            "last_played": "2019-02-11 22:35:16"
        }
    ]
}

[kivutar]

It's not JSON per se that is too slow. It's just that my scanner code is appending to a playlist file.
Using json there, with the closing }, wouldn't support the append mode.

I would have to change the logic of the scanner writer to keep all the playlists in memory, and dump them to the file system only at the end.

It's a tradeoff between using RAM of having a lot of IO.

But since I need to keep all playlists in memory anyway for the deduplication code, I might as well migrate to JSON.

[i30817]

Deduplication... I'd like to warn that the RA scanner is fairly simple and doesn't consider games which are softpatched different if their underlying rom is the same (well, the situation got worse again after it started to use serials as 'ids' which aren't even unique and don't vary for hardpatches). This leads to duplicated (and wrong) entries on the playlist, but that is still 'better' than the hack not even existing.

I'm thinking of changing my 'hack infrastructure' to use a tool that hardpatches roms and simply keeps a revert patch (i already do something very similar with rhdndat, which could be extended to handle that management); but then i'm fairly sure the scanner in the main RA GUIs would just pretend the roms don't exist if the database doesn't have their entries if using crc32 for the scanner id (which is a problem because both the nes and snes databases are not accepting my more complete PRs), or that the hacks are the same as the original if using serials for the scanner id.

Uh, resuming any attempt to make the playlist better will probably run full tilt into the wall of limitations of the scanner (primarly) and libretro-database (secondarily), and some of those limitations are 'by design' in spite of being clearly sub-optimal (in a attempt to chase scanner performance).

[kivutar]

It's not really surprising that RA doesn't scan ROMs that are patched at launch time. How would you even implement this? Detect the .ups or .ips file with the name name, apply the patch, get the new CRC and match it in the database?

[i30817]

my best (and most complicated) idea is to make the scanner - and maybe even the emulators - operate on top of a FUSE filesystem that pretends the game and patch are a single hardpatched file.

This would also finally make the softpatching core independent.

This has 'consequences' if the 'ROM' is supposed to be writable though (either another copy-on-write fuse layer or the writable medium being volatile and modifications discarded after a run).

My other idea is to make a tool (for me, because i doubt it could ever be implemented on RA) that hardpatches but keeps a 'revert patch' (i already do this manually for isos because RA refuses to softpatch large files).

But it would currently backfire anyway (because RA libretro-database is not accepting my much more complete hack list for NES and SNES consoles because they have this idea fixee that the size of the database will slow down the scanner and they want to use serials now - which can't actually recognize hacks).

This also has the problem of needing COW for certain consoles, but i already have that on linux too; i implemented a script for it to use with my dosbox ppa because i wanted to play the games on a compressed read only filesystem, it also works for (for ex.) pc-98 games on RA if i mount the shared filesystem and scan in RA those playlists from that mounted filesystem. Then i only have to remember to load the script before playing games from those playlists.

[i30817]

Resuming RA needs to do two things to bring sanity to hack playlists:

add a option to always use checksums again (or a strategy that uses a hybrid scan for 'serials' which are also on a dedicated hack database to distinguish those hacks that could be caught from the OG game) instead of serials (serials are not useless because 'fuzzy' ids are sometimes valuable to only have a single configuration etc, but only if they're paired with a higher ranked unique id that can serve as database primary key when necessary).

Accept big libretro-database PRs for the SNES and NES again. (and in the future for the cd consoles that MAME is turning into chds, because chds have a truly unique 'internal data sha1' which could serve as id independent of the dump type being originally segmented into multiple files or not).

Optionally (and i doubt it will ever happen, though savvy users can do it themselves), implement a softpatcher and COW filesystem layer and identify which games playlists need them and use that when passing a game to a core (this game needs a patch and is for pc, apply patches, apply cow, but cow already has a copy, redirect the file without doing those things, that sort of thing).

[fmaida]

Good idea, thanks.