The lang subtags is cleaned about antisamy

[spassarop]

@LiuXing-R the answer is in the policy. The current definition of lang attribute on every predefined policy is:

<attribute name="lang" description="The 'lang' attribute tells the browser what language the element's attribute values and content are written in">
  <regexp-list>
    <regexp value="[a-zA-Z-]{2,20}"/>
  </regexp-list>
</attribute>

The regex [a-zA-Z-]{2,20} does not include - and that is why is being filtered. As there are MANY ways of writting languages, the easiest (but safe) regex to use would be [a-zA-Z0-9-]{2,20}. Of course it allows languages like ------ but that is not a problem when looking for malicious HTML input. That change on avery policy file would do.

[davewichers]

@spassarop - Sebastian? Any thoughts here? I have no idea.

[LiuXing-R]

@spassarop You're right.

The regex [a-zA-Z-]{2,20} does not include - and that is why is being filtered. As there are MANY ways of writting languages, the easiest (but safe) regex to use would be [a-zA-Z0-9-]{2,20}. Of course it allows languages like ------ but that is not a problem when looking for malicious HTML input. That change on avery policy file would do.

@davewichers I'll change the regex to [a-zA-Z0-9-]{2,20} later

[davewichers]

@LiuXing-R / @spassarop - Isn't the trailing dash after the Z a dash? I tested the original regex at: https://www.freeformatter.com/java-regex-tester.html#ad-output, entering: [a-zA-Z-]{2,20} as the regex, and en-GB as the value and it matched just fine. I don't see that adding digits to the regex hurts anything but it shouldn't fix the issue you reported either.

[spassarop]

That was my mistake, the trailing dash is not part of the current policies (the PR shows that). I added it for testing (it worked) and left it when I copied that definition to explain.

The digits is because you can express languages like "es-419" too. I've found out about that yesterday, as well as more formats which include lots of letters and dashes. It's just to be more inclusive.