simplepie / text-encoder

Don't use this yet. Modern character encoding detection and conversion using Intl and ICU.

---

TextEncoder

An implementation of the WHATWG Encoding Specification in PHP.

The UTF-8 encoding is the most appropriate encoding for interchange of Unicode, the universal coded character set.

We agree with this statement. The goal of this project is to detect the current encoding of a string or document. If it isn't UTF-8, make it easy to convert to UTF-8.

Badges

Health

Quality

Social

Compliance

Workflow

Before we can convert content to UTF-8, we first need to understand the current character encoding so that we know what we need to convert from. In order to determine this, we rely on a few specific signals:

1. The Content-Type header in an HTTP response.
2. The XML prologue.
3. Well-known byte sequences in certain encodings.

Since character encodings can be fairly esoteric, and a understanding of encodings is limited, it is common that an HTTP response or XML prologue will lie about their encoding. As such, while they may be helpful, content sniffing provides the most confident results.

Once we have high confidence in the input encoding, converting to UTF-8 becomes trivial.

Specifications

• UTS #22: Unicode Character Mapping Markup Language, §1.4: Charset Alias Matching
• Extensible Markup Language (XML) 1.0 (Fifth Edition), Appendix F.1: “Detection Without External Encoding Information”
• RFC 3023: XML Media Types
• HTML Living Standard: Determining the Character Encoding
• Encoding Living Standard
• BCP-47: Tags for Identifying Languages
• FAQ: UTF-8, UTF-16, UTF-32 & BOM

Coding Standards

PSR-1/2/5/12/19 are a solid foundation, but are not an entire coding style by themselves. We automate a large part of our style requirements using PHP CS Fixer and PHP CodeSniffer. (The things that we cannot yet automate are documented in the SimplePie NG Coding Standards.)

These can be applied/fixed automatically by running the (lightweight) linter:

make lint

Additionally, in our quest to write excellent code, we use a variety of tools to help us catch issues with what we've written, including:

Type	Description
Linting Tools	PHP CS Fixer, PHP CodeSniffer
QA Tools	PDepend, PHPLOC, PHP Copy/Paste Detector, PHP Code Analyzer
Static Analysis	Phan, PHPStan, Psalm, PHP Dependency Analysis

These reports can be generated by running the (heavyweight) analyzer:

make analyze

Please Support or Sponsor Development

The SimplePie project is a labor of love. Development of the next-generation of SimplePie was started in June 2017 as because it's a project I love, and I believe our community would benefit from this tool.

If you use SimplePie — especially to make money — it would be swell if you could kick down a few bucks. As the project grows, and we start leveraging more services and architecture, it would be great if it didn't all need to come out of my pocket.

You can also sponsor the development of a particular feature. If there's a feature that you want to see implemented, and I believe it's the right fit for the SimplePie project, you can sponsor the development of the feature to get it prioritized.

Your contributions are greatly and sincerely appreciated. See the Sponsor button along the top of the page for more information.

text-encoder's Issues

[ENH] Sources for implementation

This should be a PHP implementation of the following:

• https://encoding.spec.whatwg.org/
• https://developer.mozilla.org/en-US/docs/Web/API/TextEncoder
• https://github.com/anonyco/FastestSmallestTextEncoderDecoder/blob/master/EncoderDecoderTogether.src.js
• https://github.com/web-platform-tests/wpt/tree/master/encoding-detection
• https://github.com/web-platform-tests/wpt/tree/master/encoding
• https://github.com/web-platform-tests/wpt/tree/master/mimesniff/mime-types