oxsomi / core3

Oxsomi Core3 is a combination of standalone C libraries useful for building applications, such as types, platform, graphics abstraction and file formats

OxC3 (Oxsomi core 3)

OxC3 (0xC3 or Oxsomi core 3) is the successor to O(x)somi core v2 and v1. Specifically it combines the ostlc (standard template library), owc (window core) and ogc (graphics core) in the future. Focused more on being minimal abstraction compared to the predecessors by using C17 instead of C++20. Written so it can be wrapped with other languages (bindings) or even a VM in the future. Could also provide a C++20 layer for easier usage, such as operator overloads.

• OxC3_types
  • The basic types that you might need and useful utilities.
  • Archive for managing zip-file like entries.
  • 16-bit float casts and arbitrary floating point formats.
  • AllocationBuffer for managing block allocations.
  • Buffer manipulation such as compares, copies, bit manipulation,
    • Encryption (aes256gcm), hashing (sha256, crc32c), cryptographically secure PRNG (CSPRNG).
    • Buffer layouts for manipulating buffers using struct metadata and a path.
  • List, CharString, CharStringList and CDFList.
  • Error type including stacktrace option.
  • Time utility.
  • Vectors (mathmetical) such as F32x2, F32x4, I32x2, I32x4.
• OxC3_formats: deps(OxC3_types)
  • A library for reading/writing files. Currently only for BMP and oiCA/oiDL (proprietary zip-style formats).
• OxC3_platforms: deps(OxC3_types, OxC3_formats)
  • For everything that's platform dependent (excluding some exceptions for OxC3_types).
  • Helpers for default allocator to simplify OxC3_types functions that require allocators.
  • File manipulation (in working or app dir only) such as read, write, move, rename, delete, create, info, foreach, checking.
  • Virtual file system; for accessing files included into the exe, apk, etc.
  • Input devices: multiple mice and keyboards (all accessible individually).
  • Lock and Thread for multi threading purposes.
  • Log for colored and proper cross platform logging.
  • Window for physical (OS-backed) and virtual (in memory) windows.
• Coming soon ™️ :OxC3_graphics: deps(OxC3_platforms)
  • Abstraction layer possible to port to newer graphics APIs such as D3D12, Vulkan, Metal and WebGPU. Vulkan would be the first important thing supported.
• OxC3: deps(OxC3_platforms)
  • Useful command line tool that exposes useful functions from OxC3.
  • File manipulation:
    • Conversions between oiCA/oiDL and raw files (zip-like).
    • Encryption/decryption.
    • File inspection for oiCA/oiDL files.
• Hash tool for files and strings (supporting sha256, crc32c).
• Random key, char, data and number generator.
• Profile tool for testing speed of float casts, csprng, crc32c, sha256 and aes256 (encryption and decryption).

One of the useful things about C is that files are incredibly easy to compile and parse compared to C++; resulting in reduced build times and allowing easy parsing for reflection or even auto generated documentation for things like types, function signatures and errors a function might return.

Requirements

• CMake >=3.13.
• (Optional): Git or any tool that can work with GitHub.

Running requirements

• Windows.
• A CPU.
  • Even though SSE4.2+ is recommended, this can be explicitly turned off. SSE can only be turned off if relax float is turned off; this is because normal floats (without SSE) aren't always IEEE754 compliant. SIMD option requires AES, PCLMULQDQ and RDRAND extensions too.
  • Recommended CPUs are AMD Zen, Intel Rocket lake (Gen 11) and up. This is because SHA256 is natively supported on them. These CPUs are faster and more secure. Minimum requirements for SSE build is Intel Broadwell+ (Gen 6+) and AMD Zen+ (1xxx+). The SSE-less build doesn't have any security guarantees for encryption, as these are software based instead of hardware based. Making them less secure, since no time and effort was put into preventing cache timing attacks. SSE-less build only exists for emulation purposes or for debugging, it's also notoriously slow since it doesn't use any intrinsics (SHA, AES, CRC, SIMD, etc.). The SSE-less build is also meant for easily porting to a new system without having to support the entire SIMD there first, before finally supporting SIMD after the base has been ported.

Installing OxC3

git clone --recurse-submodules -j8 https://github.com/Oxsomi/core3

Or just download the repo from GitHub.

Building OxC3

Windows

build Release On Off

This assumes you have VS2022 installed, if not, you can specify a different CMake generator by editing build.bat.

Other platforms

Unfortunately it has only been implemented on Windows as of yet, but will be supporting other platforms like Android in the future.

Virtual file system

OxC3 supports a virtual file system that allows baking/preprocessing of external file formats to our own, as well as embedding these files into the exe/apk directly. The executable embeds sections, which can be loaded individually and support dependencies. For example;

myLibrary
	shaders
	fonts
	textures
myOtherLibrary
	shaders
	fonts
	textures

The example above shows the sections that are supported for our example executable. To access these resources from our application we have to load either the root or the specific sections:

_gotoIfError(clean, File_loadVirtual("//myLibrary/fonts", NULL));	//Load section.
_gotoIfError(clean, File_loadVirtual("//myLibrary", NULL));		//Load myLibrary.
_gotoIfError(clean, File_loadVirtual("//.", NULL));			//Load everything.

These files are decompressed and unencrypted (if they were) and kept in memory, so they can be quickly accessed. They can then be unloaded if they're deemed unimportant.

The example layout above is only useful if the dependencies have very little resources. The moment you have lots of resources (or little RAM) then you probably want to split them up based on how they're accessed. For example; if assets are only used in a certain level then consider splitting up the file structure per level to ensure little RAM is wasted on it. Another example could be splitting up assets based on level environment, so if only all forest environment levels use a certain tree, then it's wasteful to load that for every level. The levels would then reference dependencies to sections and these sections would then be loaded.

When a virtual file system is loaded, it can be accessed the same way as a normal file is loaded. The exception being that write access isn't allowed anymore. The only limitation is that files need both a library and a section folder //myLibrary/fonts/* rather than //myLibrary/* and the section has to be loaded. To ensure it is loaded, a File_loadVirtual can be called or File_isVirtualLoaded can be called.

The same limitations as with a normal file system apply here; the file names have to be windows compatible and section/library names are even stricter (Nytodecimal only; 0-9A-Za-z$_). These are case insensitive and will likely be transformed to a different casing depending on platform.

The only reserved library names besides the windows ones (NUL, COM, etc.) are: access, function. So //access/... is reserved for future access to directories and files outside of the working directory and app directory (access has to be allowed through selecting for example from the Windows file explorer). //function/... is reserved for future functionality to allow custom functionality to emulate for example old file formats; the fully resolved path would be passed to a user function to allow custom behavior.

Usage in CMake

to add the files to your project, you can use the following:

add_virtual_files(TARGET myProject NAME mySection ROOT ${CMAKE_CURRENT_SOURCE_DIR}/res/mySectionFolder)
configure_icon(myProject "${CMAKE_CURRENT_SOURCE_DIR}/res/logo.ico")
configure_virtual_files(myProject)

Virtual files are then linked into the project.

To add a dependency, use the following:

add_virtual_dependencies(TARGET myProject DEPENDENCIES myDep)

This should be done before the configure_virtual_files and ensures the files for the dependency are present in this project. A dependency itself can't include an icon or use configure_virtual_files; as this is reserved for executables only.

Note: Dependencies can't be overlapping. So if B and C both include A then including B and C in D won't work.

Contributions

To contribute to this repository, you agree to the contribution guidelines. Before merging a PR as an external party, you have to sign a contributor license agreement.

License

This repository is available under two licenses:

• LGPL3 open source license.
• Commercial license.

Any company not wanting to adhere to the LGPL3 license can contact us as [email protected].

core3's Issues

Virtual files

The embedded files in exe/apk from core2/core1.

Define & implement code standard

Add string

For now we'll use ASCII. Should have nice helper functions.

Implement working dir option cmake

_WORKING_DIR should be used to set base directory to pick between working dir and app dir

Extendible error types

If error codes are added for clients, we'd have to have extendible error codes.

Prevent fingerprinting

We should ensure fingerprinting is minimized (as long as the deps don't link to anything besides our framework). This means things like monitor sizes should only be exposed to a text renderer and shouldn't be available to the app (?)

String_format

This would greatly simplify this code:

	struct String prefix = String_createRefUnsafeConst("Finished in ");
	struct String suffix = String_createRefUnsafeConst(" ms");
	struct String temp = String_createEmpty(), temp0 = String_createEmpty();

	if(!String_createCopyx(prefix, &temp).genericError) {

		if(!String_createDecx(Timer_elapsed(start) / ms, false, &temp0).genericError) {
		
			if(
				!String_appendStringx(&temp, temp0).genericError && 
				!String_appendStringx(&temp, suffix).genericError
			)
				Log_debug(temp, ELogOptions_Default);

			String_freex(&temp0);
		}

		String_freex(&temp);
	}

into

Error err = String_format("Finished in %u ms", Timer_elapsed(start) / ms);
if(!err.genericError)
	Log_debug(temp, ELogOptions_Default);

List is inconsistent with String

Create x functions for "types" functions that allocate memory

Passing Platform_instance.alloc to every function manually is a bit annoying. Perfectly fine for times when someone wants to do that. But we should also provide an x version. Example; bitx.h provides Bit_createBytesx and is part of the platforms dependency.

More functions for better errors

(struct Error){ ... } is not very nice if we ever want to add functionality to it. So replace it by Error_createGeneric for example

Float should be deterministic

https://randomascii.wordpress.com/2013/07/16/floating-point-determinism/

Do the following:
~~ - [ ] Make sure rsqrt/rsqrt isn't used.~~
~~ - [x] Remove F64.~~
~~ - [ ] Use precise fp.~~
~~ - [ ] SSE is not deterministic (?)~~
~~ - [ ] fp-contact~~
~~ - [ ] float-store~~
~~ - [ ] excess precision~~
~~ - [ ] flt eval method~~
~~ - [ ] no more printf ~~
~~ - [ ] don't sin or other trig or pow or log...~~
~~ - [ ] no reordering of instructions~~
~~ - [ ] check cpu flags on thread (main thread and new ones)~~
~~ - [ ] disable fast-math~~
~~ - [ ] force 32-bit intermediate instead of 80~~
~~ - [ ] -Qprec~~
~~ - [ ] Use doubles as intermediate: #pragma fenv_access(on), _PC_53 using _controlfp_s~~
~~ - [ ] mfp math sse~~

Avoid signed int arithmetic and modulo

These are not properly defined by C++ standard.

Keep track of the monitors a window is on

This is very important for text rendering later.

Const Buffer

Refactor: Remove struct X / enum X syntax everywhere

Simple regex. The reason: Compiler doesn't understand if you're misspelling something or not. It's also a bit too long and non standard.

HashMap, HashSet and BiHashMap

These should use the normal bucket approach.

• HashMap just uses hash of key to select a bucket and == into that bucket to get value.
• HashSet is a valueless HashMap.
• BiHashMap allows indexing two ways so has key and value buckets.

Reset cmd color on program exit

Otherwise it'll make everything after green, red, or whatever color.

BMP read/write support

• Read
• Write

File API that can map closer to DirectStorage

https://github.com/microsoft/DirectStorage/blob/main/Docs/DeveloperGuidance.md
https://github.com/microsoft/DirectStorage/blob/main/Samples/HelloDirectStorage/README.md

This will allow speedups when loading a lot of data at once. For old stuff we should still support the existing File functions.

Rename bit to Buffer

Bit_createBytes is kind of weird.

Documentation

We probably want to do this in the .c files with some tool that parses our documentation from there. I don't want to polute .h files because they're like a blueprint of the class and it's nicer to keep an overview without code and lots of comments in there.

• code_style
• types
• formats/oiCA
• formats/oiDL
• formats/oiXX
• formats/oiSH
• formats
• platforms
• graphics api
• graphics spec
• shader compiler
• OxC3 tool

Support F16?

Halfs are really nice on the GPU.
We might want to have a type that:

• Defaults to F64 if available otherwise F32
• Defaults to F16 if available otherwise F32.

F16 can be supported on CPU too; but needs manual implementation which is SLOW.

Keyboard and mouse support

This should be handled like this:

• Keyboard<i> and Mouse<i>
  Because with RawInput we can support a few keyboards or mice.

InputManager

Here we'd create actions and assign them a name and suggested bindings.
This is required if we want to abstract OpenXR at some point.

Idea; oiSc -> oiBc

Expose the types and some functions to a scripting language. Oxsomi script and bytecode. Bytecode could be generated per unique binary (JIT) or executed by the OVM.

This will allow us to scan through the bytecode to see what high level operations are done. This would allow us to force disable file calls for example. When still called, we could return an Unauthorized error.

Execute each window in a separate thread

This includes virtual windows and the physical window loop. Means the wait for windows function needs to be aware of this.

• Virtual
• Physical

Rename u<x> i<x> f<x> to U<x>, I<x>, F<x>?

This'd be more consistent with the rest of the codebase.

Replace Window's management with a Texture class.

This texture class should be able to handle all linear texture formats. Compressed formats should be readonly.
Should also allow Z-order curves or tiling for optimizations in cases such as CPU raytracing.

Virtual //access folder

The //access folder should provide files that we have received access to from the explorer.
For example; we say export file x, then an explorer should be spawned to give us writeonly access to this file.
Import file would have readonly.

This way, after calling File_writeVirtual or File_readVirtual; the permission will be revoked (read/write once).

An example could be an app folder at //access/app. Which would be %Appdata% on windows.

PNG read/write support

• Read
• Write

RefPtr; for Java-like reference counting

void* and u64 and some functions.

Better file support

• Checking if a file or folder is present
• Checking date / size of a file without reading
• Looping through a folder
• Querying file/folder count
• Removing files
• Adding files
• Writing empty file
• Adding folders
• Adding file parent automatically (recursive folder)
• Moving files
• Renaming files
• Checking file write/read permissions
• File max timeout for reading/writing
  - [ ] File streaming?
  - [ ] Locking files?

Safety check file names for consistency

Linux and windows have to be consistent with each other to make development easier.

• Disallow old MS DOS names to not run into that later.
• Disallow non standard MS path syntax such as C:myLocalFolder/...
• Disallow reading from servers by ip on Windows.
• Figure out what we want to do with network drives that don't hardcode ip?
• Disallow [A-Z]:/ syntax on Linux for consistency at start of string.
• Disallow other non allowed symbols in a string such as *:?"<>|
• Look into other problems Linux, Android, OS X, iOS or webasm might have with our current file system.
• Only allow [0-9A-Za-z\ _.-] for consistency

Protected file writing

Only files in the working directory are allowed to be modified.

ARM support

Such as:

• Replacing Timer_clocks from __rdtsc to something that ARM uses.
• Supporting ARM neon.

Rename enums to EX

More allocation control?

https://github.com/Oxsomi/core2/blob/master/include/system/allocator.hpp

Controller support

Controller support requires the following:
https://github.com/gabomdq/SDL_GameControllerDB/blob/master/gamecontrollerdb.txt

Without this database, essential controllers might be missing or it might be hard to configure settings.

Add macro for making texture format more readable

Something like MAKE_TEXTURE_FORMAT(..., ....) and MAKE_TEXTURE_FORMAT_COMPRESSED(...)

String UTF-8

This would mean that we need an extra check when looping over the string; are you the special character that indicates more bytes are needed? If no, read character, else; continue building character. Things like tolower/toupper would be way more confusing. But these shouldn't affect string size so would not need an allocator.

Support doubles

Even though niche, doubles can be good for large scale games. Especially the transformation part of it.
We would only support scalars and vectors as double. And double to float / float to double. Matrices or anything heavier shouldn't be used.
GPUs don't natively support doubles, so to support two types: compile shader twice if double is used; one for GPUs that support it, another if they don't. If they don't, we need to emulate doubles :(

• Default math operations
• Reading/writing operations
• Convert things like time to double
• Quat32 vs Quat64
• Conversions
• Sort
• Conversions between F32 and F64

File watcher

Force little endian

We don't support and will never support big endian. Some examples of big endian systems:

• Raspberry pi 0w.
• PS3 and below.
• Xbox 360 and below.

This is because of how we serialize data. Big endian would be a pain to implement in a safe way and would require a penalty on read/writes on big endian to swap around bytes. We will always store little endian, because everyone uses it:

• PC (Windows, OS X, Linux)
• Mobile (Android, iOS)
• Console (PS4+, Switch, Xbox One+)

Casting is also basically free on little endian systems because it doesn't have to swap any data, just add or remove the bytes after. Ofc for us we can still do an error check if required.

Because of this, we should disable building on a big endian platform somehow in cmake or in types.h or something.

Support platform dependent error

Error_platform should allow fetching current error for HRESULT on windows.

Make errors optional

There should be something like CHECK_ERROR(), RETURNS_ERROR and RET_ERROR(). This should ignore errors if you're not on secure mode.
Because if this isn't done everywhere from the beginning, it will become slow and a pain to fix later.

Allow heavier error checking on vectors and math

If they generate a nan or inf it should catch it.

String regex

Text rendering

Might have to be moved to Oxsomi/graphics_core2 when it's there https://medium.com/@calebfaith/implementing-msdf-font-in-opengl-ea09a9ab7e00

Create list

Allow free on null

This simplifies it for already freed objects