wayfind / is-utf8 Goto Github PK

用sublimetext3保存为utf8文件，检测失败

var fs = require('fs');
var isUtf8 = require('is-utf8');

var utf8 = fs.readFileSync('test.html');

console.log('test.html is utf8: '+isUtf8(utf8));

Can you mention in the readme the minimum amount of bytes needed to reliably detect utf8?

Hey,
Thanks for the good effort in building this library. We would like to use your node in our project.

Can you let us know what BSD license have been assigned to this library?

Thanks
Kandy

NULL terminated strings (NULL is part of the buffer) should qualify as UTF-8 but don't.
In fact only a few characters below 0x20 qualify as valid.

The check for ASCII should simply be
if(bytes[i]<=0x7F)

And not

        if(     (// ASCII
                    bytes[i] == 0x09 ||
                    bytes[i] == 0x0A ||
                    bytes[i] == 0x0D ||
                    (0x20 <= bytes[i] && bytes[i] <= 0x7E)
                )

UTF-8 is supposed to be backward compatible with ASCII.

UTF-8 Wiki

The salient features of this scheme are as follows:

Backward compatibility: One-byte codes are used only for the ASCII values 0 through 127. In this case the UTF-8 code has the same value as the ASCII code. The high-order bit of these codes is always 0. This means that ASCII text is valid UTF-8....

The string is sent by a device and is received by a third party software (node-red) which uses is-UTF8. I can't change either.

We like to take code from github to make debian packages.