duhrer / fluid-osk

Fluid Onscreen Keyboard

About Me

I'm a developer focused on accessibility and interactive experiences.

I work a lot with things like MIDI controllers, game pads, and live coding environments.

If you want to see some examples of my work, check out my GitHub pages or YouTube channel

fluid-osk's Issues

Initial implementation

For various features in the [https://github.com/fluid-lab/gamepad-navigator][Gamepad Navigator], we need an in-browser on-screen keyboard. This ticket covers an initial implementation that can be reused for projects with similar use cases. The initial implementation should:

1. Provide a base grade for a single key that can accept mouse, touch, and keyboard input.
2. Provide a base grade for a row of keys, with sensible keyboard navigation between keys.
3. Provide a grouping grade for arrangements of rows of keys, i..e a keyboard layout.
4. Provide initial styles and markup.

The key grade should initially at least be able to support "momentary" key presses, i.e. the equivalent of a key down and key up in rapid succession. "Latched" key presses are key for use cases like the caps lock (press to toggle on and off) and shift key (press to capitalise next key). This may be added later, or may be covered here depending on how it goes. For the future, this grade will need to also provide a means to displaying effects such as "held", "inactive", this may be covered in another ticket depending.

The gamepad navigator primary needs to enter a URL. Although we will want other layouts, for the initial implementation and proof of concept, a keyboard layout with the letters, numbers, and special characters required for a URL that lacks query data would be adequate.

Only ASCII character equivalents will likely be supported for this pass, but the initial work should leave room for overlaying alternates for a single on-screen key, such as accents or variations.

Add ability to enter diacriticals and other international characters.

Currently, the keyboard only supports entering a primary or secondary ("shift") key. It does not support diacriticals or other "international" characters.

One way to accomplish this would be to use a pattern like that used in Windows or Mac OS X.

In this pattern, a combination of keys indicates that the next character will be modified. Sometimes, the initial "dead" (unprinted) key has a rough correspondence to the diacritical. In windows, colons become umlauts as in ü. In OS X, backquotes become grave accents as in è, and tildes are used to make ñ and other characters.

This "leading modifier" approach requires us to track the fact that some things are leading modifiers, and to define what should be output for which modifier. It also requires that we somehow indicate that the modifier is active, and provide a means to cancel entering a modified character.

Another pattern is to have some kind of modified or long press on the key to be modified, which is fairly common in mobile keyboards. Holding e or another character to be modified presents a menu with acute, grave, and other variations on that character. In our case, we would probably create a new "special character" latched key.

This seems more compatible with the existing code base. First, we already have the concept of a latched key. We also already have the concept of defining a payload (what to display, what to send). We could use the same concept to add a defined list of variants. When the "special character" key is latched, clicking a key presents a menu centred on the key's position, and selecting an option sends the "variant" key.

This does present some UI concerns that need unpacking, namely, what is focused, how do we move between variants, and how do we cancel the selection. Presenting the non-variant option as the first item and focusing on that seems appropriate.

Movement between options should follow the same convention as the larger keyboard, i.e. either tabs or "arrow only" (see #3).

Clicking outside of the variant menu should obviously close it, but I'm not sure what to do for keyboard navigation. One idea is to have navigating out of the menu in either direction close it, but this seems like it punishes users who explore too far.

In Weavly, there is a similar menu, which presents a focus trap where the original item (the key in our case) is included as the toggle for the menu. That might work, but is a little concerning in that we'd have to manage levels of focus traps when working within our own modal.

Another option is to avoid a focus trap and allow navigating in and out of the variant menu to neighbouring keys. In this scenario, we would close the variant menu when:

1. An item is selected from the menu.
2. The user hits escape.
3. A mouse click outside the variant menu occurs.
4. A "close" or other control inside the menu is clicked.
5. The "special character" key is clicked again (to turn off the latch).
6. The user opens the variant menu for another key.

We could choose to have navigating out of bounds open the variant menu for an adjoining key. In that scenario, making a mistake and navigating too far is easier to reverse.

Add (optional) support for "arrow only" navigation.

In downstream work on the gamepad navigator, the built-in tab navigation of the onscreen keyboard is proving a bit problematic. When the keyboard is part of a modal, you have to either tab through every key, or use the arrows to navigate to the last key (the space bar) and then tab to get to the modal controls.

Instead, I'd like to add (configurable) support for "arrow only" navigation where:

1. Only one item in the keyboard has tabindex=0 at a time.
2. The rest have tabindex=-1
3. The "live" tabindex follows the arrow focus, so that tabbing in and out of the keyboard returns to the last used key.

Given that we have a focus management element already, it seems easy enough to add logic to focus on the first key by default, and then to manage the tabindex values as we change the focus.

As this will be tied to a user preference in the gamepad navigator, the behaviour needs to be configurable using a model variable. The keyboard also needs to respond immediately when the preference changes:

1. When the behaviour is turned on, the current key needs to get tabindex=0 and the rest need to get tabindex=-1.
2. When the behaviour is turned off, all keys need to allow tab focus again.