Accept-Language comparing standard currently differs between `getLanguagePriority` and `compareSpecs` about negotiator

[dougwilson]

Hi everyone, sorry I have been away with other things.

For Accept-Language header, this module specifically only implements the algo linked to from the HTTP RFC (https://tools.ietf.org/html/rfc7231#section-5.3.5). To quote from that section of RFC 7231:

For matching, Section 3 of [RFC4647] defines several matching
schemes. Implementations can offer the most appropriate matching
scheme for their requirements. The "Basic Filtering" scheme
([RFC4647], Section 3.3.1) is identical to the matching scheme that
was previously defined for HTTP in Section 14.4 of [RFC2616].

The "Basic Filtering" algo is what this module currently implements for Accept-Language header, as it is the algo web servers have always traditionally implemented for selecting a language from a set list for HTTP applications. The full test of this algo is in RFC 4647 here: https://tools.ietf.org/html/rfc4647#section-3.3.1

How the basic filter algo works is also illustrated in a previous comment I made here #46 (comment) which applies to the original post as well.

I'm not really familiar with the ago in RFC 4647 that would produce the expected results in the original post. @tangye1234 would you be able to link directly to the section in RFC 4647 that defines the algo you're expected and than use your example to walk though it? I'm 100% open to adding additional defined algos to the list such that you can choose which one to use when matching language tags 👍 the PR above was changing the existing standards-defined algo which is why it was marked as wontfix, though adding additional algos from the RFC as options to alter behavior is perfectly acceptable.

I hope this makes sense.

[julianlam]

How coincidental, I just ran into this issue as well.

var Negotiator = require('negotiator');
var req = { headers: { 'accept-language': 'en-CA,en;q=0.9,en-GB;q=0.8,en-US;q=0.7,fr;q=0.6,pt;q=0.5,th;q=0.4' } }
var negotiator = new Negotiator(req);
negotiator.languages(['en-GB', 'en-US', 'en-x-pirate']);
[ 'en-x-pirate', 'en-GB', 'en-US' ]   // output

In my scenario the en should match en-GB first at priority of 0.8. en substring match with en-x-pirate should be considered only after all exact matches have been exhausted.

[julianlam]

@tangye1234 #46 provides a bit more context, and @dougwilson has shown that this module adheres to RFC 4647, so this unfortunately is a wontfix

[tangye1234]

@julianlam, I think @dougwilson is trying to explain the case Apache mod_negotiation and jshttp/negotiator behave the same, but that is not what rfc4647 describes, the RFC only describes that in this case language cannot be selected out, but only be filtered with a fallback list, while users would decide for their own.

Maybe he is right because the negotiator cannot make decisions for the user, unless user send another accept-language list: en-GB, en;q=0.9 that follows the the suggestion from RFC:

the user SHOULD specify the most specific (largest number of subtags) range
first and then supply shorter prefixes later in the list

[julianlam]

Assuming we want to define additional algos to the call to negotiator, how would this be achieved, as a second argument for .languages?