knightos / kernel

Kernel for z80 calculators

KnightOS Kernel

The KnightOS kernel is a kernel for Texas Instruments calculators. It offers many unix-like features for z80 systems, including:

• A tree-based filesystem
• Multitasking (up to 32 concurrent processes)
• Dynamic memory management

This kernel is the basis of KnightOS, which is a good resource for others hoping to implement a userspace.

This project is only useful to systems programmers. Please look into KnightOS instead if you are not a systems programmer.

Compiling

If you have a pre-compiled kernel image, skip this section.

The toolchain for the kernel was built mostly from scratch, since old z80 tools are, well, old. The new toolchain supports a lot of the kernel's needs on newer platforms, and works well on Linux, Mac, and Windows. You'll need to install:

• Mono (or Microsoft.NET on Windows)
• GNU Make
• sass (aur/sass)
• mktiupgrade (aur/mktiupgrade)
• genkfs (aur/genkfs)
• mkrom (aur/mkrom)
• patchrom (aur/patchrom)

genkfs is only strictly neccessary if you hope to build a userspace on top of the kernel. On Windows, install Cygwin and perform the build from there. Windows users should install sass into their %PATH%.

The kernel needs to be rebuilt for any system you'd like to target (different calculator models). For each supported calculator model, use the given make target:

Model	make Target
TI-73	TI73
TI-73 Explorer	TI73
TI-83+	TI83p
TI-83+ SE	TI83pSE
TI-84+	TI84p
TI-84+ SE	TI84pSE
TI-84+ CSE	TI84pCSE
TI-84 Pocket.fr	TI84p
TI-84 Plus Pocket SE	TI84pSE

Simply run make [target] to build the kernel. The default target, when omitted, is TI84pSE. This will produce a kernel image in the bin/ directory. This will also generate a kernel.inc file, which you can use to link your userspace with the kernel. Run make clean before trying to switch platforms.

Usage

The kernel does not do anything on its own. Instead, it forms the basis for more complex systems. Upon booting, the kernel loads up the filesystem and runs /bin/init. You need to provide this init program yourself. Here is a simple example init program, which can be assembled with the assembler of your choice:

#include "kernel.inc"
    ; Program header
    .db "KEXC"
    .db KEXC_STACK_SIZE
    .dw 20
    .db KEXC_ENTRY_POINT
    .dw start
    .db KEXC_HEADER_END
start:
    pcall(getLcdLock)
    pcall(allocScreenBuffer)
    kld(hl, message)
    ld de, 0
    pcall(drawStr)
    pcall(fastCopy)
    jr $
message:
    .db "Hello, userspace!", 0

When you compile the kernel, you'll get a ROM file with an empty filesystem. To build the filesystem, you will need to make an example on your own system to build it from. Then, you can use genkfs to generate and write a filesystem to the ROM. If you wish to build an OS upgrade that you can send to your calculator, use mktiupgrade.

When you build the kernel, in addition to the ROM file, you will receive a kernel upgrade file (this will be the .73u, .8xu, or .8cu file in the output directory). This can be used on any KnightOS system to upgrade the kernel without touching the userspace filesystem.

Kernel API

The kernel offers an API to userspace to interact with things like threads, memory, hardware, the filesystem, and more. The API is documented through special comments in the source code, which are extracted to generate the online API reference.

Versioning

The kernel uses semantic versioning. Version numbers are indicated by the latest git tag, and take the form of major.minor.patch. "Patch" is updated when bugs are fixed and for very minor changes. "Minor" is updated for new features and major non-breaking changes. "Major" is updated with breaking changes. When you compile your kernel, the kernel version (as an ASCII string) will be written to address 0x64 on page 0x00.

If you are working with a kernel that is not built from a major release, you will have a sligthly different kernel version. Appended to the version will be "-nn-hhhhhhhh". "nn" is the number of commits that have been made since your kernel release. "hhhhhhhh" is the git shorthash of the commit you're currently on. A "+" will be appended to this if your working directory is dirty.

Getting Help

You're free to ask questions about the kernel (or the userspace) in #knightos on irc.freenode.net. We're not always listening, so stick around - it may be a while before your question is answered.

Help, Bugs, Feedback

If you need help with KnightOS, want to keep up with progress, chat with developers, or ask any other questions about KnightOS, you can hang out in the IRC channel: #knightos on irc.freenode.net.

To report bugs, please create a GitHub issue or contact us on IRC.

If you'd like to contribute to the project, please see the contribution guidelines.

Licensing

The kernel uses the permissive MIT license. It permits use and modification in most scenarios, including commercial.

kernel's Issues

FAT cannot extend beyond first page

The kernel doesn't support reading a FAT across several pages.

Figure out how to detect stack overflows

I think it might be useful to write a tool that guesses the amount of stack a function needs by following it through its execution path. We might also try a custom emulator that's aware of the space each program owns and can alert us when something writes outside of its own memory. There also might be a strictly on-calc solution, but I'm not sure what it might be yet.

Platform: TI-84+ Color Silver Edition

Progress:

• Color LCD boot up
• LCD legacy mode
• Basic filesystem support
• 4MB filesystem support
• Flash driver compatability
• Power management
• Switching between legacy and color modes
• Color drawing functions (#53)
• Color versions of KnightOS userland apps

Thread ID assignment changes

Change thread IDs to be assigned in a similar fashion to file stream IDs, where the thread ID is an index.

Benefits:

• Thread allocation is faster and cannot cause ID conflicts
• Thread killing is faster
• Frees up a byte in the thread table that I intend to use for thread priority

The new structure of the thread table will be as follows:

Offset	Length	Description
0000	1	Priority
0001	2	Executable address
0003	2	Stack pointer
0006	1	Flags
0007	2	Reserved for future use

A priority of 0xFF would never be executed, and indicates an unused thread entry.

Publish documentation for kernel routines

Currently they can be accessed by looking at an old commit (like here) but they need to be either online or in the repo. If the latter, they need to stay up to date when people commit (so perhaps auto-generated stuff should stay online).

Add magic number and version to kernel executable format

Clear unclaimed semaphores when a thread is killed

TI Keyboard support

Modify sort/integerSort to use BC for length

Instead of DE for "last element".

sha1AddRange does not work as expected

In the writable_fs branch, I was going to use this to confirm that a file was read correctly. However, sha1AddRange does not return normally. Probably some stack mishandlng.

Implement getFreeMem

Should figure out just how much memory is in use.

Color graphics

• Text
• Images (#52)
• Lines
• Rectangles

RLE (de)compression functions

Implement a set of RLE compression and decompression routines.

• Compression
• Decompression
• Predict size of compressed data

deleteFile doesn't disable interrupts

Looks like it gets all ready to, but then doesn't.

Write a general purpose sort function

Where the caller can offer you a little function to do the comparison for you. This could be used to alphabetize strings and such. See if you can make the size of each element in the offered memory variable, so we can have you sort 16-bit pointers.

Allow for font customization

Maybe pull from /etc/fonts? Not sure how this could best be implemented.

Make basic link interface routines

Nothing fancy, just enough so that applications can send/receive bytes/bits. It would be nice if the PHY layer were compatible with existing protocols, like the TIOS's or CalcNet 2.2.

Add all remaining "setInitial" functions for setting the initial value of registers in new threads

No need to bother with individual 8-bit registers, A is special. Don't worry about shadow registers, either.

Provide better kernel error reporting

We moved libtext into the kernel, so we can give meaningful error messages.

Limit FAT to 1 sector of Flash

Benefits:

• Simplifies process to determine when a garbage collection is needed
• Simplifies garbage collection by keeping FAT and DAT off of each other's pages

Downsides:

• Reduces maximum size of FAT to 64K

Flash routines should execute somewhere other than kernelGarbage

We need kernelGarbage for garbage collection at some point.

Discussion - Compressed filesystem

Would it be worth our time to compress the filesystem at a low level, which could make files take up less of our very limited space? What kind of compression techniques would be best, and what would be required to implement them?

getBatteryLevel does not work correctly

On the 83+, it shows the battery on the Castle far beyond the bounds of the icon.

On all other platforms, it always shows as 3/4 full.

Implement realloc

Add applib function to display user-friendly error message

Error should be passed in via A. The function should immediately and silently return if Z is set or A = 0.

Time and date functions

All platforms should be able to do math related to time. Only platforms with an RTC should be able to get the current time.

Useful code: https://github.com/torvalds/linux/blob/master/kernel/time/timeconv.c

@thirtythreeforty is this something you'd be interested in?

drawStrFromStream can not possibly work

It says that it takes B as a stream ID although streamReadByte takes a stream ID in D, which is used by drawStrFromStream as an X coordinate.

Implement md5 routines

On the TI-83+ and TI-73, a full MD5 implementation will need to be written. On all other supported calculators, the hardware is able to perform the bulk of the work.

• md5init - Allocate working memory at DE
• md5update - Adds BC bytes from (HL) to the hash-in-progress at DE
• md5finish - Finishes the hash, returns pointer to completed hash in (HL)

Remove margin argument from drawStr

Don't forget to change all the code that calls drawStr.

Design and implement a color image format

• Design
• Decode in kernel
• Create tool to encode them on PC

Changes to kernel reserved memory

The following changes should be made to kernel reserved memory:

• 32 active threads (0x100 byte table)
• 16 active libraries (0x40 byte table)
• 16 pending signals (0x40 byte table)
• 16 active file streams (0x80 byte table)

The total size of kernel state memory would become 0x200 bytes. This should be followed by 0x100 bytes of miscellaneous state (i.e. hardware locks). Finish this off with 0x200 bytes of kernel garbage memory. This would leave 0x8500-0xFFFF for userspace memory.

When changing this, use clever equates and such in defines.inc that make this easier to change in the future.

Make filesystem writable

We'll also be changing existing read-only functions to use a buffer in memory so that read functions can use the same code for each kind of stream.

Orion TI-84+ support

Talking KnightOS!

Write unit tests for sorting functions

Discussion - Utilizing extra RAM pages

We have anywhere from 1 to 6 extra RAM pages on our supported calculators. Any ideas on how we much make use of them? I want to ideally be able to put allocated memory in them, but some algorithmic and performance considerations need to be addressed.

Deduplicate code in malloc routines in memory.asm

Things such as

a:
    push af
    push hl
        ; code
        call b
.fail:
    pop hl
    pop af
    ret

b:
    push af
    push hl
        ; code
.fail:
    pop hl
    pop af
    ret

can be changed to

a:
    push af
    push hl
        ; code
        jp b_nopush
.fail:
    pop hl
    pop af
    ret

b:
    push af
    push hl
b_nopush:
        ; code
.fail:
    pop hl
    pop af
    ret

Not critical but it saves a good deal of stack usage.

Move monochrome font to page 0x01

We're running a bit low on space on page 0x00 and we'll need to use 0x01 for color fonts anyway.

Add version information to kernel header

Implement calloc

Allocates a memory block of size BC * A and fills it with 0s.

Static linking and dependency resolution

Wanted to start a discussion here about static linking and dependency management. I've had some ideas, but they aren't fully fleshed out in the code (also note that it might be good to implement this, for the most part, in userland).

The idea is to make sure that software works when there are different versions of libraries required and such. So, here's how we do it. All libraries are required to use semantic versioning, and we store them in /lib/libraryname.v1.2.3. Instead of loadLibrary("/lib/libraryname"), userland programs would loadLibrary("/lib/libraryname", majorVersion), and the kernel would grab the best version.

We include as many versions as we need. If two userland programs call for different versions of the same library, we have both of them and each userland program loads the correct one.

Since we'd have semantic versioning here, the major version of each library would only increment for breaking changes. This means that for most cases, we wouldn't need multiple versions of libraries.

Static linking and the library jump table mechanics that are already in place fit into this paradigm without modification.

Rewriting unit tests in javascript

Since OpenTI is progressing nicely, I'm thinking that it might be a good idea to rewrite unit tests in JS instead of assembly. Then, we can have more flexible tests and we don't have to worry about having room in the kernel for the test runner and the tests themselves.

Implement thread priority

The thread priority byte in the thread table consists of two four-bit items: the priority and the counter.

A thread will be executed on a context switch if the upper four bytes of the priority are zero. Each context switch (regardless of whether or not a thread is executed), the counter nibble will be incremented, and then ANDed with the lower nibble.

Since 0xFF is an unused thread entry, this leaves us with four priorities:

• 0b0000: Priority 1, executes every context cycle
• 0b0001: Priority 2, executes every other context cycle
• 0b0011: Priority 3, executes every fourth context cycle
• 0b0111: Priority 4, executes every eighth context cycle

It is acceptable to use priority 1 for foreground threads (anything receiving user input). Any background threads (including daemons) are encouraged to use lower priorities to lighten the load on the foreground threads.

As an example, KnightOS will include a USB daemon, which will listen for USB activity and interact with devices as they are plugged in. This should be set to priority 4, since it needs to update infrequently. The Castle, on the other hand, is a priority 1 thread and interacts with the user in real time.

Rename sort -> integerSort

Deduplicate code in setInitial functions in thread.asm

Currently there is about 2 lines' difference between each of the functions. They can't quite be made into one function, but most of the code can be reused.

Support carraige return in text drawing

Kernel debugging

I want to build an on-hardware debugging system.

Code style in display.asm

Needs to be modified to meet the standard. Might be a good idea to make a tool that normalizes code automatically.

The following needs to happen:

• Fix indentation
• Fix instruction spacing and case
• Remove needless labels
• Convert labels to local

Add random number generator

Also include a custom seed function, unless the seed is constantly changed in the interrupt.

Modify getStreamInfo to return length in EBC, not DBC

Since the D register is kind of important for FS stuff.

Thread IDs could potentially be re-assigned while still in use

This can be corrected by confirming that the desired thread ID isn't already in use, instead of just picking the next one.