tzlaine / text

What a c++ standard Unicode library might look like.

text

Motivation

C and C++ are the only major production languages that do not have robust Unicode support. This library attempts to provide that support for C++.

This library provides a Unicode implementation suitable for use by anyone — including those who know next to nothing about Unicode.

Features

• Iterators, views, and algorithms that convert among UTF-8, UTF-16, and UTF-32.

• An implementation of the Unicode algorithms for dealing with encodings, normalization, text segmentation, etc.

• A family of types for use in Unicode-aware text processing. This includes text, text_view, rope, and rope_view.

• Code familiarity -- all of the above types and algorithms work like the STL containers and algorithms.

This library targets submission to Boost and eventual standardization.

It's more interesting than it sounds.

Online docs: https://tzlaine.github.io/text

text's Issues

Myers singletons for all data tables that aren't already.

Except maybe ccc(). Can't live without that one, and we want it to be bare-metal fast.

Add generated tests for normalized_nf*() .

Replace {save,load}_table with iterator-based API

Add sentence break algorithms (forward and back).

Default collation (no locale)

Sort key compression

It's pretty easy now that the FractionalUCA.txt data are driving collation. It just needs to be done.

Case-sensitive find.

Apply API changes like those done to string in ccefda3 to the other unencoded types

Tailoring for Bidi

Not sure is this is necessary....

Apply space savings hack from TR15

http://www.unicode.org/reports/tr15/tr15-45.html#NFD_Table_Optimization

to_title()

Add NFKD normalization

Add Unicode bidirectional algorithm.

Performance tuning.

Consider adding reverse versions of break lazy ranges (reverse_paragraphs(), reverse_words(), etc.)

Add NFC normalization

Provide dual-build support (header only and compiled)

Add normalization perf tests.

Add vertical property lookup.

Export symbols for DLL/SO builds

So far, text has only been built as a static lib.

'make install' installs gtest, gmock, and benchmark, but not text

As part of playing with text, I performed a build/install with commands like the following and found that this installed the gtest, gmock, and benchmark dependencies, but not the text library headers, docs, or other artifacts.

cd build
cmake -DCMAKE_BUILD_TYPE=DEBUG ..
cmake --build . --config DEBUG --target install -- DESTDIR=/my/sandbox

Check the entire string in text{,_view} literal operators.

Case-insensitive find.

Add collation table lookup perf test.

Add line break algorithms (forward and back).

Another character data consolidation pass

This should be done after all the Unicode algorithms and their data are finished.

Tailored collation tables need canonical closure

This probably only needs to be generated for the tailored table entries. Some of the ICU collation tests (in tailoring_data.cpp. currently commented out) still fail because of this.

Try space-optimizing the character hash tables

It may be possible to combine the properties like quick check, ccc, etc. into a single struct, all placed into a single hash map. If there is not a large runtime penalty for this, this could be a nice space savings.

is_upper() applied to a single grapheme

to_upper()

Make the encoding breakage checks asserts instead of exceptions

Add stream safe conversion function.

Tailoring for break algorithms

Break lazy ranges need to be range-friendly themselves

They currently accept begin/end pairs, but should also accept a CPRange.

is_lower() applied to a single grapheme

Add word break algorithms (forward and back).

Consider adding a string builder, based on Peter Bindels' design

Said design being:

https://github.com/dascandy/s2/blob/master/include/s2/detail/rope.h#L26

Try to optimize normalization functions to concentrate around QC=maybe characters

http://www.unicode.org/reports/tr15/tr15-45.html#Detecting_Normalization_Forms

Add normalized_nf*() functions, featuring quick checks.

Add end-to-end collation perf test.

String algorithms for the string layer.

This should bring over many (most?) of the implementations from Boost.StringAlgo, but possibly with LEWG-friendlier names.

Add perf test for break property lookups

This is necessary to establish a baseline from which faster implementations can be produced.

Add NFKC normalization

Add built-in tailoring for german "ss" and that funny looking special character

http://www.unicode.org/reports/tr10/#Special_Cases

Add NFD normalization

Collation correctness and perf tests using much longer strings

Add tailoring support to break property functions.

Collation (with tailoring)

All range and iterator-pair interfaces should be range friendly

to_lower()