Unilex is a new subproject of Unicode CLDR. The purpose is to provide a central repository of lexical data: information about words in different languages. For each language, the goal is to collect properties such as word frequency, pronunciation, hyphenation, and eventually part of speech and other linguistic information. Such data can be useful for building spell checkers, text-to-speech synthesis systems, and a variety of other services that require language processing.
As a central repository, it can serve as a known location where software developers look to find lexical data, and a place where native language speakers can contribute data that can then be used to improve support for their language. To flesh out the support in different languages, the we are soliciting contributions of lexical data for different languages. That data is contributed as machine-readable list of words and their properties, as tab-separated plaintext files. For more about the current formats, see Unilex Properties
Because the project has just started, there is a limited amount of data. Please create an issue if you can contribute lexical data for a language.
Eventually, we plan releasing lexical data for the world’s languages as part of the Unicode Common Locale Data Repository. The final format has has not been decided yet: one option could be to format everything into XML files according to the Lexical Markup Framework; or we might define some new, possibly JSON-based format; or maybe we’ll just publish the data in its present form of tab-separated plaintext files.
The Unicode Consortium releases this data under the same license as all its other data files. Currently the format is tab-separated plaintext files, but additional formats may be offered in the future.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent