New URL validation breaks loading from jar files about antisamy

[davewichers]

What do you specifically suggest we do to fix this? And could you also provide a test case that shows the current failure?

[symposion]

I'm sorry for the redundant ticket, but after my initial confusion when upgrading broke our application, I've realised that I can simply open the InputStream on the URL myself and pass that in, so this may not be worth fixing. I guess the alternative is simply allowing jar:file: urls as well as file: .

[davewichers]

This seems reasonable to me. We would prefer not to force our users to have to rewrite their code to upgrade to new versions of AntiSamy. @spassarop @gerardocanedo - Either of you want to write a test case to show this failure and then implement an enhancement to allow this? Any risk/reason why we shouldn't allow URLs like this? Not urgent, given that @symposion has found a workaround for his app.

[kwwall]

Unless they have the AntiSamy policy file referenced in an external properties file, it would still require a code change I think. A minor one, but they'd still have to add the "jar:file:" to whatever they previously had hard-coded. -kevin

…

On Wed, Mar 24, 2021 at 10:56 AM Dave Wichers ***@***.***> wrote: This seems reasonable to me. We would prefer not to force our users to have to rewrite their code to upgrade to new versions of AntiSamy. @spassarop <https://github.com/spassarop> @gerardocanedo <https://github.com/gerardocanedo> - Either of you want to write a test case to show this failure and then implement an enhancement to allow this? Any risk/reason why we shouldn't allow URLs like this? Not urgent, given that @symposion <https://github.com/symposion> has found a workaround for his app. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#79 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAO6PG4XIY4VBVELIRFLH4LTFH4QRANCNFSM4ZU5W6GQ> .

-- Blog: https://off-the-wall-security.blogspot.com/ | Twitter: @KevinWWall | OWASP ESAPI Project co-lead NSA: All your crypto bit are belong to us.

[symposion]

Sorry, I will get round to creating a test case when I have a moment. More or less what we're talking about is this:

 URL policyUrl = Thread.currentThread().getContextClassLoader().getResource("style-filter-policy.xml");
 InternalPolicy.getInstance(policyUrl);

This used to work, but now it fails thanks to the code in Policy.resolveEntity:

if (!"file".equals(baseUrl.getProtocol())) {
               throw new MalformedURLException(
                   "Only local files can be accessed with the baseURL. Illegal value supplied was: " + baseUrl);
           }

I think that check could just be slightly more permissive and it would avoid breaking backwards compatibility. No code changes would be required by users of antisamy, people packaging their policy files in their jars/wars/etc will already be getting the jar:file: url from the getResource call

[spassarop]

I thought this was a recent change that I was not aware of, but it dates to v1.5.11 release.

Should we address this issue, the test case would be something like adding a JAR file to the test resources with a policy inside and loading it to get that jar:file: protocol?

[kwwall]

Or... you maybe you could package a reasonable default AntiSamy policy file in the AntiSamy jar and try a test that accesses it via a "jar:file:" URL from there??? Not sure if that would would, but it might be worth a shot. I would only do that if you package a really strict policy file there though.

…

-kevin

On Wed, Mar 24, 2021, 7:11 PM Sebastián Passaro ***@***.***> wrote: I thought this was a recent change that I was not aware of, but it dates to v1.5.11 release. Should we address this issue, the test case would be something like adding a JAR file to the test resources with a policy inside and loading it to get that jar:file: protocol? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#79 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAO6PG4LDM3JA3YK4FOHV4DTFJWQ7ANCNFSM4ZU5W6GQ> .

[davewichers]

@spassarop - I think adding a test jar with a test policy in it is the right way to add a test case for this.

[spassarop]

I guess we need a 1.6.3 branch then.

[davewichers]

@spassarop - created. Can you update the version in the pom in your pull request?

[spassarop]

No problem, should that be 1.6.3-SNAPSHOT?

[spassarop]

I added a JAR with a dummy main class and the default policy as resource. It gets loaded and I can see the jar:file: schema. What happens when calling getProtocol() is that it is just jar.

So my question is: should I check that ir is jar and then with the rest build another URL and repeat to check if the protocol is file?

It can be done in a "readable" but "long" and repetitive way:

if ("jar".equals(baseUrl.getProtocol())) {
    String remainingUrlString = baseUrl.getFile();
    URL remainingUrl = new URL(remainingUrlString);

    if (!"file".equals(remainingUrl.getProtocol())) {
        throw new MalformedURLException(
                "Only local files can be accessed with the baseURL. Illegal value supplied was: " + baseUrl);
    }
    return;
}

if (!"file".equals(baseUrl.getProtocol())) {
    throw new MalformedURLException(
            "Only local files can be accessed with the baseURL. Illegal value supplied was: " + baseUrl);
}

Or in a shorter but maybe harder-to-understand way:

if (!("jar".equals(baseUrl.getProtocol()) && "file".equals(new URL(baseUrl.getFile()).getProtocol())
        || "file".equals(baseUrl.getProtocol()))) {
    throw new MalformedURLException(
            "Only local files can be accessed with the baseURL. Illegal value supplied was: " + baseUrl);
}

Of course, both work. I'll let you comment before pushing anything to save us some trouble.

I still have to figure out how does the JAR get referenced and loaded just for tests. I added it with IntelliJ but no uploadable project config was changed :/

[davewichers]

@spassarop - I suspect just checking for JAR is sufficient. To verify though, you could write a test case with a hand crafted URL like:

jar:https://somebadsite.com/foo.xml and then verify that this functionally fails/throws an exception, even though it passes the 'is it a JAR URL check'.

[spassarop]

Some stuff to review:

1. Ended up loading the test JAR this way to avoid changing any other project file:

   java.net.URLClassLoader child = new java.net.URLClassLoader(
           new URL[] {Thread.currentThread().getContextClassLoader().getResource("policy-in-external-library.jar")},
           this.getClass().getClassLoader()
   );
   URL policyUrl = Class.forName("org.owasp.antisamy.test.Dummy", true, child).getClassLoader().getResource("policyInsideJar.xml");

   Is this OK?

2. Tried loading a policy with jar:https://somebadsite.com/foo.xml but the URL API just fails before even calling AntiSamy, in the URL object constructor. The filename constructor does not take an absolute URL so I don't see a way to test that case.

[davewichers]

@spassarop - I updated the pom in the 1.6.3 branch so please rebase before you do a pull request. I was expecting your 1. to look basically like the provided example:

URL policyUrl = Thread.currentThread().getContextClassLoader().getResource("style-filter-policy.xml"); InternalPolicy.getInstance(policyUrl);

Can't you add a test resource .jar file so something basically like this works?

For 2. - To me: "Tried loading a policy with jar:https://somebadsite.com/foo.xml but the URL API just fails before even calling AntiSamy" proves that you can't embed an http[s] URL inside a jar: URL, proving that all you need to verify is that the URL protocol is file or jar. So, I'd write the test case as if the URL API invocation works, and then wrap all the code in a catch Exception block and if any exception is thrown, the test passes.

[spassarop]

I already placed the JAR in src/test/resources/policy-in-external-library.jar, but that alone does not load it for the class loader to get it. So getResource returns null. That's why I loaded it by code, to avoid changing project config (besides, I don't know what to change in project config either, Google didn't want to help me).

About the exception stuff, I'm on it.

[davewichers]

@symposion - @spassarop added a test case to verify this JAR URL limitation, and then a fix to the code to allow JAR URLs (again). This fix is in the 1.6.3 branch. Can you test this fix to verify it works they way it used to work for you?

[davewichers]

@symposion - It's been 5 days since I asked you to verify/test our fix. You going to be able to get to this soon?

[symposion]

Hi, really sorry, will get to it later today.

[symposion]

Ok, I have tested this with 1.6.3-SNAPSHOT and the problem I had before has indeed been fixed. It's worth bearing in mind that there are a lot of custom URL schemes out there and it'd be pretty hard to create an exhaustive list of which ones are "safe" and which aren't. E.g. Spring has classpath: and Wildfly/Jboss use vfs: . The jar: scheme is built into the JVM so I'd say it's good to have added support for that one, but it's probably worth documenting this somewhere obvious in the release notes because I think people will hit this problem with a bunch of other common custom schemes and the fix is very simple (stick .openStream() on the end of your URL before passing it in.) Thanks!

[davewichers]

@symposion - I reopened this. I'll keep it open until we actually push out the 1.6.3 release. But it's good to know the fix worked. And I'll add the info you requested. Should I add it to the README? Maybe to the JavaDoc for the method that will reject these other URLs? Seems like the JavaDoc is the best place for that.

[kwwall]

Definitely add it to the Javadoc.

…

-kevin

On Tue, Apr 6, 2021, 2:58 PM Dave Wichers ***@***.***> wrote: @symposion <https://github.com/symposion> - I reopened this. I'll keep it open until we actually push out the 1.6.3 release. But it's good to know the fix worked. And I'll add the info you requested. Should I add it to the README? Maybe to the JavaDoc for the method that will reject these other URLs? Seems like the JavaDoc is the best place for that. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#79 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAO6PG6XGQKKBEKKUYYRFFTTHNKU7ANCNFSM4ZU5W6GQ> .

[davewichers]

@spassarop @symposion - Does the JavaDoc I added here: https://github.com/nahsra/antisamy/blob/1.6.3/src/main/java/org/owasp/validator/html/Policy.java#L315 look right to you? Did I put it in the right place and explain it properly? I want to make sure before I push out this release.

[spassarop]

Looks good to me :)

…

On Wed, 7 Apr 2021 at 21:37 Dave Wichers ***@***.***> wrote: @spassarop <https://github.com/spassarop> @symposion <https://github.com/symposion> - Does the JavaDoc I added here: https://github.com/nahsra/antisamy/blob/1.6.3/src/main/java/org/owasp/validator/html/Policy.java#L315 look right to you? Did I put it in the right place and explain it properly? I want to make sure before I push out this release. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#79 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHL3BMPYSR6RV2S2FEWGXZDTHT3F3ANCNFSM4ZU5W6GQ> .

[davewichers]

Fixed in release 1.6.3 that just went out.