strikerx3 / openxbox Goto Github PK

Do what the title says. ;)

The Object Manager defines and manages objects that represent devices (including disks, volumes and files), synchronization objects such as events, semaphores, mutexes and timers, and system objects such as threads, processes, APCs and DPCs. Objects are accessed by their HANDLE on user space, and either handles or pointers by the kernel. There are two reference-counting mechanisms, one for handles and one for pointers.

The Object Manager has very close relations with many other subsystems, such as the I/O system (functions prefixed with Io). Keep in mind that we want to provide secure access to the host file system for both the hard disk and the DVD-ROM drive.

As mentioned on this comment, using pointers into the memory array with virtual memory addresses is dangerous and will corrupt memory or cause crashes if the data being accessed crosses the boundary of two or more virtual pages that map to non-contiguous physical addresses.

The goal here is to create a mechanism that allows host code to manipulate guest data using virtual addresses as transparently and lightweight as possible, ideally reproducing the behavior of the CPU, including causing exceptions such as General Protection Faults if the code attempts to read from or write to inaccessible memory pages. The existing StructRef template class can be used as a starting point for research.

Pointers are still safe to use when the address is known to be physical. Most (if not all) Mm* functions and kernel initialization code can continue using ToPointer. Stack variables might be safe as well; this needs investigation.

The command line front-end should be able to parse command line arguments.

https://github.com/skeeto/optparse seems like a good portable library to help with this.

In the remove-glib branch I removed GLib as a dependency on Windows platforms due to the excessive complexity of the library and the fact that we were barely using its features. The only real reason for it to be included was the (incomplete) NV2A renderer which I haven't touched because the kernel barely works and doesn't even reach the main function of Microsoft XDK titles anyway, so there is no point in having a renderer yet.

After mulling over the problem, I figured that the emulator would be in a much better shape if it had a modular plugin system where the core provides the basic emulation features, including the custom kernel implementation, and interfaces for plugins to implement Xbox hardware, such as the CPU, NV2A and others. That way, the core can be small, clean and have very few dependencies, while the plugins can provide implementations using whatever compiler and libraries they want, as long as they obey the supplied interfaces.

These two basic devices (8254 and 8259) need to be emulated since the kernel depends on the timer (which produces a constant stream of IRQ 0s once every millisecond) for thread scheduling, DPCs and other housekeeping duties.

The implementation should also be aware of the possible presence of the devices in virtualization platforms (such as KVM) and take advantage of them.

Reference sources for the implementation:

• 8253/8254 common code and emulation from QEMU
• 8259 common code and emulation from QEMU
• Additional changes to 8259 common code and emulation in XQEMU

Some devices are raising/lowering the wrong IRQs and erroneously sending interrupts to the CPU, which is causing the kernel to bug check due to unexpected interrupts.

Also check if the i8259 implementation is handling IRQs correctly.

Due to the excessive complexity in implementing and maintaining the high-level kernel implementation, I'm making a shift in focus to full LLE, as seen in the full-lle branch. The current kernel work will be put on hold until a better solution is found.

I know this goes against the original goal of the project, but the added complexity of the full kernel implementation is not worth the trouble of facilitating usage of the emulator, especially considering that Xbox hardware emulation is still incipient. Besides, there are many other successful emulators out there that don't work without a BIOS file (such as PCSX2), and it's relatively easy to extract the necessary files from an Xbox machine.

Dynamic recompilation is likely to provide the best performance for any Original Xbox emulator on x86 systems. But before that can be achieved, the kernel implementation needs to be redesigned for even greater performance.

One option I envisioned is to maintain two different versions of the kernel: the current one that indirectly manipulates Xbox memory through dumb pointers, and another version that can access these data structures directly. The second version would be used with the dynamic recompiler, as all game code and kernel structures will reside in host memory. The biggest issue with this is the fact that two versions of the same code need to be maintained, which is less than ideal.

Another possibility is to rewrite the kernel code in a separate project that is compiled into a binary ROM file which is then loaded into the top of the guest memory space and executed directly by the CPU emulator. The kernel would have to abuse certain invalid x86 instructions to communicate with the host in order to provide access to host resources, such as the file system, similar to what is done by features like VirtualBox Guest Additions or VMware Tools on the respective hypervisors. Advantages of this approach include:

• Only one codebase for the kernel that can be used by other emulator projects
• Can be executed by CPU emulators and the dynamic recompiler without modification, and they will all provide their own benefits as well
• Effectively becomes equivalent to writing a kernel for a real Xbox machine (and if done correctly, could even be installed on a real Xbox!)
• The project can be separated from OpenXBOX

KVM is Linux's virtualization platform, supporting both Intel and AMD hardware-assisted virtualization solutions.

We want to implement this as a Linux-only CPU module.

The current CPU plugin is based on Unicorn. I found a blocking issue that's preventing progress on the memory manager implementation: the emulated CPU seems to ignore paging settings and tries to access physical addresses instead of translating them from virtual addresses using the paging data structures in memory. The emulator has two disabled unit tests that confirm the bug.

The goal here is to research other CPU emulation or virtualization projects and create a new CPU plugin that can be used with OpenXBOX. Good candidates are mentioned in the README.md file. Another alternative is to create a dynamic recompiler that can take advantage of the design of the emulator to bypass trampolines and perform direct calls to kernel functions.

The goal is to implement the entire I/O subsystem, but not necessarily every I/O device available, simply because there is no hardware emulation other than the CPU and system clock at this point. However, we'd like to lay the foundation for the guest-to-host file system driver (#15) as it will push emulation further.

Several Nt* functions will become fully implementable as a consequence of this.

Unicorn (the only CPU emulator used on OpenXBOX) doesn't handle interrupts on its own. We'll need to emulate them while leaving the possibility of taking advantage of interrupt emulation features of more capable CPU emulators.

OSDev's page on Interrupts and the Interrupt Descriptor Table are good starting points for research. The Intel® 64 and IA-32 Architectures Software Developer Manuals are a great resource for understanding interrupts in depth.

This will serve as the basis for emulating certain pieces of Xbox hardware such as the System Clock that plays a role in thread switching implemented on #1.

The current implementation is mostly a placeholder that gets the job done. The memory manager in the kernel does much more work.

The goal is to implement all Mm* functions (including any internal Mi* functions and data structures), and go further and implement all Nt* functions related to virtual memory and Ex* functions related to pools.

Note that in the real kernel, the addresses produced by the internal memory allocation functions are in the virtual memory range reserved for the kernel -- 0x80000000 and above. These addresses map 1:1 to the physical memory area at 0x00000000, but we still need to map addresses to pointers and back again safely.

The big things:

• Initialization (#1)
• Thread scheduling (#1)
• Hardware interrupts (#4)
• Software interrupts (#7)
• Memory management, part 1 (#8)
  • Physical memory (#8)
  • Virtual memory (#8)
  • Pools (#8)
• Object management (#12)
• Device I/O (#13)
  • Host file system access (#15)
• Memory management, part 2
  • File system cache
• Exception handling
  • Remaining interrupt handlers
• Hardware abstraction layer

One of the biggest features missing from the emulator is support for port- and memory-mapped I/O. Without it, no hardware devices can be implemented.

Hardware devices will be able to map whatever I/O addresses they require. The CPU module must be able to intercept any attempts to perform I/O operations using either technique and redirect them to the specific piece of hardware that they're mapped to.

Implement the guest drivers that provide access to the host file system for the hard drive and DVD-ROM media. Ideally we want to provide a layer of abstraction that allows us to use different sources, such as a folder or XISO file for DVD media, for instance.

This can be done as a set of replacement drivers for the FATX, GDFX, UDFX and RAWX file systems implemented in the Xbox kernel.

While implementing the custom Xbox kernel outside of the emulated environment, I found that some of the kernel functions need to suspend execution due to various reasons (yielding, waiting for a signal, trying to enter a locked critical section).

Since we cannot suspend execution of the host thread, we need to figure out a way to allow the emulator to "suspend" execution of the host thread (but not really) and continue with the emulation, so that at a later point in time, when the suspension condition is no longer met, the original thread can resume execution from the point where it stopped.

Allow certain aspects of the emulator to be configured programmatically such as:

• Type of Xbox system to emulate: Retail, Debug, DevKit, Chihiro
  • Changes how much memory is available to the system and the behavior of many kernel functions
  • Enables additional hardware emulation, like the media board on Chihiro or Super I/O on the DevKit
• AV pack to report
• System clock tick rate
• Paths to MCPX and BIOS ROMs
• Path to EEPROM.bin (if the user doesn't want to use the default path)
• CPU emulator plugin to use
  • HAXM is the only option available at the moment. KVM (#19), Hypervisor.Framework (#20) and Windows Hypervisor Platform (#17) are planned
• GDB server: enable/disable, listen port, whether to wait for a connection or start emulation immediately, etc.
• Log level
• Other debugging features such as XBE dump, single CPU stepping, CPU register dump and possibly more (only for Debug builds)

The solution should be easy to integrate with the command line, a GUI, Libretro and others.

This is the last piece remaining to complete thread scheduling. The system clock (and many other processes) request software interrupts through HalRequestSoftwareInterrupt, which may invoke the handler itself or defer to the next time the IRQ level is lowered or a hardware interrupt handler finishes.