wonderfultoolchain / wf-issues Goto Github PK

Writing tooling in Lua turned out to be great for development agility (at least from my point of view - better than expected), until it wasn't.

Lua's strings are immutable, so right now f.e. every wswan target linker relocation involves a malloc() + memcpy(), and there are already codebases which apply them in the tens of thousands.

A mutable "typed array" library would allow circumventing this issue without massively increasing the memory footprint of the linker.

Of course, ideally, the linker would be rewritten in Not Lua...

https://github.com/bbbbbr/romusage from the Game Boy scene would be a good inspiration.

This tool could either work on ELF files (using the existing ELF parser logic, or cleaning it up further) or emit the data as part of the ROM builder itself.

With the new non-contiguous ROM linker (and, eventually, the WonderWitch target's use of SRAM for data storage), it is viable to add limited support for dynamic memory allocation and have it not be entirely useless. sbrk() would be a good first step.

The non-contiguous linker could be adjusted to allocate the heap and stack in separate "gaps" on user request to maximize the benefit of this, for example placing the stack in 0x0040 - 0x0FFF and the heap in 0xC000 - 0xFDFF.

One can disable optimizations to work around this, but it'd be great to have this fixed.

tkchia/gcc-ia16#120

There are probably multiple bugs contained here, but they all together create a wonderful mess.

sprites.zip

See src/main.c. In line 25 there are three different ways for defining the array SwanOffset.

// this gives the intended effect (doesn't work with bitfields):
int SwanOffset[NUM_SWANS] = {0, 8, 16, 48, 10};

// all the swans end up on one place (doesn't work with bitfields):
const int SwanOffset[NUM_SWANS] = {0, 8, 16, 48, 10};

What happens when bitfields are used with the two preceeding ones:

// there is a full Swan visible, but also a white tile (only the first iteration works?):
// (which likely didn't get the right palette assigned)
// works with both bitfields and manual assignment
const int __wf_rom SwanOffset[NUM_SWANS] = {0, 8, 16, 48, 10};

In line 79 it is accessed, but independently from that the behaviour changes based on whether the attr field of the sprite struct is set using bitfields or manually.

	// Setup the Swan sprites
	for (int swan = 0; swan < NUM_SWANS; swan++)
	{
		for (int tile = 0; tile < 4; tile++) {
			sprites[swan*4+tile].x = (tile & 1) * 8;
			sprites[swan*4+tile].y = (tile >> 1) * 8 + SwanOffset[swan];
			// When bitfields are used nothing works
			// (there is only one white square, so not even the palette is properly assigned)
			//sprites[swan*4+tile].palette = SPRITE_PALETTE;
			//sprites[swan*4+tile].tile = tile;
			// When the attribute field is composed manually some things work
			sprites[swan*4+tile].attr = tile | (SPRITE_PALETTE << 9);
		}
	}

Requested by a few developers. How do we get there when nobody on the dev team owns a Mac?

For various tools that work with WonderSwan software, it'd be good to define a common format of homebrew metadata (name, author, etc) that can be contained within the ROM.

It'd be good for it to be simple enough to be parseable on-device (but that's not a particularly low bar, the WS does have some power).

I'd say the primary "prior art" here is the SDSC header, but arguably a better solution would be to use a RIFF chunk pointer or Vorbis comments, with a small footer placed just before the official WonderSwan footer. Notes from Maxim/SMS Power on that regard:

I mean, it’s OK and a bit sympathetic to assembler syntax but I think it’d be better to do something more “standard” if possible. Check out the tagging formats used in media, like the Vorbis comment
Producing a valid SDSC header in a basic assembler from decades ago is a lot easier than producing a valid Vorbis comment but maybe it’s easier for others to decode if it’s a standard
SDSC limits you to nn.nn version numbers, three text fields and a date. The number parts can all have ill-formed values, like version FF.A7 and date AB/CD/DEF0
I recently took the liberty to at least specify the text is UTF-8 now…
So yeah, SDSC header is a crappy old-fashioned inextensible format. If we were to extend it we might start putting machine-readable text in the comment I suppose.
https://www.xiph.org/vorbis/doc/v-comment.html for reference. It's a bit unclear but basically, followed by many , where each UTF-8 text is in the form KEY=value. You might use the vendor string part to act as a locator or you might make a well-known offset for a pointer to the tag. But this will naturally take more bytes than a more efficient format.

I'd arrange it as follows:

As for the RIFF chunks (or Vorbis comments), they could have an official list of types (name, auth, desc, etc), but also allow people to define custom ones according to their needs.

Do you think this is overengineered? What would you propose?

ws-bootfriend/installer/src/main.c built using Makefile.witch -> menu_main

In a large software project, including homebrew projects, it is currently difficult to make a list of all licenses which a given program ought to comply with. This is less of a problem for dynamic linking, as the user often provides these libraries himself - however, as homebrew is statically linked, the requirement comes up for every library used in a project.

However, as the Wonderful toolchain is a controlled ecosystem, we can consider and support this scenario.

There's some things to consider:

• some licenses (CC0, zlib) do not put any requirements on a binary distributor; others (MIT, MPLv2) put requirements limited to the code being distributed; others (GPL) may have a viral effect on the entire binary.
• some libraries are header-only, but nonetheless meet the minimum criteria for copyrightability.
• some libraries (crt0, libgcc, libc, libstdc++) are default and thus not explicitly requested.
• some libraries have one global header (libtonc), others may have a few dozen headers depending on which files are linked in (newlib).

There's many sources of information which could be used here:

• ELF/map files - actually linked objects, functions and data,
• GNU make dependency files - headers pulled in by object files,
• Makefile - list of requested libraries.

Feedback wanted!

I think [...] overlays don't work properly
And they seem the only [reasonable] way to have more than 32KB of iwram code

My idea here is to offer a configuration system like so:

[overlays]
iwram = [
  ["shared", "overlay_a"],
  ["shared", "overlay_b", "overlay_c"],
  ["shared", "overlay_c", "overlay_d"]
]

One makes a list of all the combinations of the various IWRAM sections in memory, that they want to be accessible simultaneously, and the linkscript is dynamically generated based on that. This high-level abstraction allows breaking fine details about the linker script's operation without forcing everyone to carry their own modified link script; as well as expressing overlay relationships in a way possibly slightly more intuitive to a programmer.

The Windows installer could use checkbox options to automatically run commands such as wf-pacman -S target-gba, wf-pacman -S target-wswan, etc.

Seems like it could be fun.

• The GBA Video cartridge exposes sixteen 512-byte banks, for a total of 8KB. The header should, as such, fit a small crt0 which copies the actual executable code to EWRAM, think multiboot layout. The linkscript should be adjusted to match, and a new target added.
• New helper functions of the style gba_cart_matrix_* to copy out further data at will.
• Integration with gba_overlay_load, as ROM->IWRAM/EWRAM copies require special treatment on this mapper.

• The Windows target could have an installer tool or script written - it is much more consistent than Linux installations.
• Maybe the Linux target could have a partial script too.

Ideally, we'd have #embed, but we're not in that world right now.

.s files may be faster - it would be fairly easy to adapt the converter to those - as would .o files - which would be a bit harder.

The Windows installer could use code to add a Windows Terminal profile that calls the Wonderful Toolchain in its own shell.

The guide for adding a Windows Terminal profile to MSYS2 could be adapted and automated.

Yay, 2048 Mbit games!

• Figure out how the EG0xx mapper actually works. (Documentation would be useful here, but I can probably figure it out myself if I ever get my hands on a cart.)
• Add some ROM banking facility, dependent on the above.
• Contribute EG0xx mapper support to the first willing emulator.

• Bring up an initial crt0/linkscript - based on LunarLambda's design.
• Port -MJ file.cc.json from Wonderful's gcc-ia16 fork to GCC 13, for compile_commands.json handling.
• Write a tool to adjust and configure the .gba file header.
• Write an initial libgba (do not confuse with the other libgba - following convention here), covering low-level hardware access. For this initial bringup, code can be reused from the friendly-licensed gba-hpp, as well as agbabi.
• Write a linker wrapper (like we used to have for the wswan target) which can handle generating a linkscript dynamically as well as calling the ROM fixer.
  • This will allow implementing IWRAM overlays in a later stage without breaking Makefiles.

While unlikely if you know how to write code in general, it is possible that a user may not have make installed. This does not come with Wonderful Toolchain, and is neither present in the installation instructions nor anywhere else on the docs. (My use case for W-T is BlocksDS, though the wswan target likely has the same issue given the templates also use make).

• Note that devkitPro's APT package for devkitpro-pacman actually lists make as a dependency, making it a non-issue at least on Debian-based systems when people install that. Though by design and choice, W-T doesn't, nor wants to, install this way.

This is especially noticeable for a user that may not have touched Linux ever in their life and set up WSL2 purely out of interest for this toolchain; for instance, I had to reinstall my WSL2 distro today due to hardware issues, and found that Microsoft's Ubuntu image does not have either make nor build-essential installed.

Maybe some note about relevant packages to install would be a good idea (i.e. "Please go to Google and search how to install these particular applications for your Linux distribution.") It's the kind of thing that a LMGTFY would solve, but IMO such a note would change the situation from "figuring it out while debugging why I'm missing this" to "figuring it out since the docs told me I need this".

A long-standing request for the toolchain has been Windows and macOS support. RSDuck has created a proof of concept for the former, showing that the required changes to build scripts are minimal.

However, the Bash-based build system I wrote as a stop-gap solution to get wf-pacman out the door has limitations:

• ./rebuild-repository.sh always rebuilds the entire repository; due to the need to extract every single package to memory across all architectures, this takes a substantial amount of time already, and will take even more.
• There is no way to do things like "check which packages need a rebuild", or "automatically build all outdated/missing packages", or "automatically remove the old package on successful build". It is possible to scan packages for such information (makepkg --printsrcinfo), but one would need to act on it. Fixing this would open up the possibility for continuous integration, as well as for moving compilation to a dedicated ARM builder - currently QEMU user mode is used, which isn't the fastest.

Ideally, a Python-based build system would be built that supersedes the hacky Bash scripts. From there, Windows support can be designed in from day one.

Right now, constant structures have to be explicitly annotated with __far to be placed in ROM memory. This is less than ideal:

• Existing code most likely doesn't use the __far annotation, requiring adaptation;
• Literal strings cannot be placed in ROM memory at all.

There are many options to improve this:

• Patch gcc-ia16 to place literal strings in .fartext and as __far-annotated variables;
• Patch gcc-ia16 to place read-only data in .farrodata and as __far-annotated unless __near is present;
• Patch gcc-ia16 to support placing variables in the code segment; this would allow using CS-relative pointers for a slight performance benefit compared to __far annotations for literal strings.
• Add a large memory model to gcc-ia16, as the times when we want a near pointer are notably less numerious than times when we want a far pointer.

This is a rather difficult task, unfortunately.

Doesn't affect clangd, which we recommend, but is rather annoying.

Right now, emulating the old wfconfig.toml defaults, 64 bytes at the beginning of IRAM are reserved for interrupt handlers. However, many of these interrupt indices are never or rarely used by homebrew:

• Interrupt 6 and 7 are unused on the V30MZ,
• Interrupt 2 (NMI) is unused on the console's SoC,
• Interrupts 1, 3, 4 and 5 are only used if the user deliberately emits the relevant opcodes or flags,
• Hardware interrupts 1 (keypad) and 2 (cartridge IRQ) is rarely used in a homebrew context.

In total, that's 9 interrupts that are rarely used, potentially freeing up to 36 additional bytes of the console' 16 KB of IRAM - especially given the support for non-contiguous memory allocation.

As such, it may be a good idea to allow defining IRQ vectors using a section syntax such as .iram_irq0.

There are two approaches available here:

• wait for segelf support to be added to NASM: https://bugzilla.nasm.us/show_bug.cgi?id=3392533
• write an OMF -> segelf object converter - this would also allow supporting other OMF-compatible assemblers and compilers, such as MASM/TASM.

Elm-ChaN's FatFS can be used for this; I already have experience with doing so via my work on BlocksDS.

There's a few particularly popular/viable flashcarts to support:

• GBA Movie Player CF / SuperCard CF (talks to IDE bus; easy to emulate, good emulation target due to old homebrew supporting them well)
• SuperCard SD (talks to SD card via SPI)
• EverDrive GBA (talks to SD card via SPI with autoread?)
• EZ-Flash Omega (talks to SD card via high-level interface)

The migration part should be easy enough now that wswan/small is ported, however currently it's the crt0 objects which decide how the memory is laid out: screens, sprite tables, etc. The question of how to approach this remains open.