IOException on Policy creation from InputStream when schema validation is disabled about antisamy

[spassarop]

@jeremiahjstacey - Thanks for the suggestion. On one hand it may make the code cleaner (@davewichers you relocated logging statements, don't you? This looks fine to you?). On the other hand, re running the validation changing the boolean setting still needs to reset the stream anyway to accomplish that goal. I really could not find a way yet to do it with the specific InputStream used in the test cases, I need to make the stream rewindable somehow (cloning was not an option in the past).

Sadly I can’t keep looking at this today but I'll see what I can do tomorrow.

[davewichers]

@jeremiahjstacey @gerardocanedo - Standby. I found the problem. I just have to figure out how to fix it. Hopefully won't take too long. This looks fishy: "Setting AntiSamy policy schema validation to 'false' because 'owasp.validator.validateschema' system property set to: 'true'." :-)

Apparently Boolean.parseBoolean() and Boolean.getBoolean() do different things ... :-). Fixed now, just cleaning up.

[davewichers]

Fixed in v1.6.2 that was just released.

[davewichers]

@spassarop - Hey Sebastian - can you write a test case to verify this, and then fix it, if verified?

[kwwall]

See the attached gzipped tarball. It already has a test case in it the you can either run standalone pull into your AntiSamy test suite.

…

-kevin

On Fri, Mar 12, 2021, 9:27 AM Dave Wichers ***@***.***> wrote: @spassarop <https://github.com/spassarop> - Hey Sebastian - can you write a test case to verify this, and then fix it, if verified? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#75 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAO6PGYB6XH2RRUILLWO2BDTDIJCJANCNFSM4ZCERNFA> .

[kwwall]

@davewichers - BTW, this is a showstopper for the planned 2.2.3.0 ESAPI release. These tests still fail with the same "Stream closed" IOException as their root cause even if this test is run as:

        mvn -Dowasp.validator.validateschema=false test

so I am not confident that this leaves any workaround except for correcting the application's AntiSamy policy file that otherwise doesn't pass schema validation. And since it the invalid '<html-entities>' node was in every single release of ESAPI's antisamy-esapi.xml file dating back to as far as 10 years or so, I think this will affect a lot of ESAPI users even though our new release will be removing that unused, invalid node. And yes, I will surely draw attention to this in our accompanying release notes, but there is significant evidence that people don't pay much attention to them even after they encounter problems. (They just figure it is easier to create a GitHub issue or to privately email us instead.)

As @jeremiahjstacey pointed out, I think the fix is just to drop in a call to InputStream.reset() before the second call to getDocumentElementFromSource() to reset the input stream. (That ought to work in most cases unless the input stream is associated with something like a pipe or stdin.) Maybe that is what the call

                source = getResetSource.call();

is intended to do, I don't know. I do know that @jeremiahjstacey spent a long time in looking at this and I do totally trust his judgement though.

[davewichers]

@kwwall - I figured as much. Hopefully @spassarop can get to this quickly as he implemented the enable/disable feature, but apparently missed this specific use case. I'd expect this would be an easy fix that we can get done in the next couple of days.

[spassarop]

Well that's a deep analysis indeed. To give some context:

This was potential issue since the implementation of validation warnings and retrying validation. The latter is the conflictive one because when it comes to stream reading, that gave me a heachache.

My solution to re read a stream is the one you wrote, resetting it:

private static Element getTopLevelElement(InputStream is) throws PolicyException {
    final InputSource source = new InputSource(is);
    source.getByteStream().mark(0);
    return getTopLevelElement(source, new Callable<InputSource>() {
        @Override
        public InputSource call() throws IOException {
            source.getByteStream().reset();
            return  source;
        }
    });
}

As there are two main ways of creating a policy (InputStream or URL) to avoid repeating LOTS of code I just encapsulated the reset behavior in a callable which is the getResetSource argument. In the case of InputStream, reset failed if I didn't use .mark(0) before.

Maybe the big problem here is that the InputStream is wrapped in an InputSource object, which I sadly could not avoid when implementing this. Avoiding re validation is an absolute solution but it wouldn't be so friendly for users I guess. However, I'm telling you all this without having executed the tests yet. I'll tell you what I get from all this but I wanted to show you the whole picture before.

[jeremiahjstacey]

@spassarop, I was looking at it and I wonder if the logging can be pushed down another level? I'm not overly familiar with the API, but I think this might work:

 protected static Element getTopLevelElement(InputSource source, Callable<InputSource> getResetSource) throws PolicyException {
        try {
            return getDocumentElementFromSource(source, validateSchema);
        }  catch (SAXException | ParserConfigurationException | IOException e) {
            throw new PolicyException(e);
        }
    }

 
    private static Element getDocumentElementFromSource(InputSource source, boolean schemaValidationEnabled)
            throws ParserConfigurationException, SAXException, IOException {
        Element returnElement;
        DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();

        /**
         * Disable external entities, etc.
         */
        dbf.setFeature(EXTERNAL_GENERAL_ENTITIES, false);
        dbf.setFeature(EXTERNAL_PARAM_ENTITIES, false);
        dbf.setFeature(DISALLOW_DOCTYPE_DECL, true);
        dbf.setFeature(LOAD_EXTERNAL_DTD, false);

        // Build the non-validaiton implementation before assigning the schema.
        DocumentBuilder nonValidatingBuilder = dbf.newDocumentBuilder();
        nonValidatingBuilder.setErrorHandler(new SAXErrorHandler());
       
        
        getPolicySchema();
        dbf.setNamespaceAware(true);
        dbf.setSchema(schema);

        DocumentBuilder validatingBuilder = dbf.newDocumentBuilder();
        validatingBuilder.setErrorHandler(new SAXErrorHandler());
        try {
            Document validDoc = validatingBuilder.parse(source);
            returnElement = validDoc.getDocumentElement();
            if (!validateSchema) {
                // We warn when the policy has a valid schema, but schema validation is disabled.
                logger.warn("XML schema validation is disabled for a valid policy. Please reenable policy validation.");
            }
        } catch (SAXParseException spe) {
            if (schemaValidationEnabled) {
                throw spe;
            }
        }
        //Only reaches here if the schema is valid or we have schemaValidationEnabled set to false
        if (!schemaValidationEnabled && returnElement == null) {
            Document nonvalidDoc = nonValidatingBuilder.parse(source);
            returnElement = nonvalidDoc.getDocumentElement();
             // We warn when the policy has an invalid schema, but schema validation is disabled.
             logger.warn("Invalid policy file: " + e.getMessage());
        }
        
        return returnElement;
    }

The stream may still need to be reset on some scale, I'm honestly not certain on that. It does reduce the nesting calls in the exception blocks, so I figured I'd throw it out in case the idea would be useful to you.

[davewichers]

@spassarop - Lets focus on getting this scenario to functionally work, and not break any of the other scenarios that currently work. And then we can figure out the best way to handle logging for all scenarios, with this one additional new one now working.

[jeremiahjstacey]

@spassarop, it may be a little cheaty -- ByteArrayInputStream does not implement close. If we read the original inputStream into a byte[], then build a ByteArrayInputStream to feed to the InputSource then the close behavior may not actually matter.

I have not tested this. In theory, it might work.

 private static Element getTopLevelElement(InputStream is) throws PolicyException {
        ByteArrayOutputStream buffer = new ByteArrayOutputStream();
        try {
        int nRead;
        byte[] data = new byte[1024];
        while ((nRead = is.read(data, 0, data.length)) != -1) {
            buffer.write(data, 0, nRead);
        }

        buffer.flush();
        } catch (IOException e) {
            logger.warn("Unable to read contents from stream");
           throw new PolicyException( e);
        }
        byte[] byteArray = buffer.toByteArray();
        
        final InputSource source = new InputSource(new ByteArrayInputStream(byteArray));
        source.getByteStream().mark(0);
        return getTopLevelElement(source, new Callable<InputSource>() {
            @Override
            public InputSource call() throws IOException {
                source.getByteStream().reset();
                return  source;
            }
        });
    }

References:
https://www.baeldung.com/convert-input-stream-to-array-of-bytes
https://docs.oracle.com/javase/7/docs/api/java/io/ByteArrayInputStream.html#close()

[gerardocanedo]

Something straight forward but it may be the solution

Instead of setting always true to validate schema, I'm setting the validateSchema variable.
It works for the direct config.. There may be something else with the system property.
I'm working on it

protected static Element getTopLevelElement(InputSource source, Callable getResetSource) throws PolicyException {
// Track whether an exception was ever thrown while processing policy file
Exception thrownException = null;
try {
return getDocumentElementFromSource(source, true);
...
}

protected static Element getTopLevelElement(InputSource source, Callable getResetSource) throws PolicyException {
// Track whether an exception was ever thrown while processing policy file
Exception thrownException = null;
try {
return getDocumentElementFromSource(source, validateSchema);
...
}

[jeremiahjstacey]

@gerardocanedo, I thought about that too. The cost there is that you'll lose the distinct logging to call out that schema validation is disabled. If not having the logging is an acceptable change, then that does cleanly remove the issues with the InputSource.

[davewichers]

I'd prefer to have the same logging behavior in this scenario as we've implemented for the other ones already. i.e., always warn one way or the other when validation is disabled.

[gerardocanedo]

I'm getting the following output from both test cases:

[main] WARN org.owasp.validator.html.Policy - Attempting to load AntiSamy policy from an input stream.
[main] WARN org.owasp.validator.html.Policy - XML schema validation is disabled for a valid AntiSamy policy. Please reenable policy validation.
[main] WARN org.owasp.validator.html.Policy - Attempting to load AntiSamy policy from an input stream.
[main] WARN org.owasp.validator.html.Policy - XML schema validation is disabled for a valid AntiSamy policy. Please reenable policy validation.

I did some changes to the static code using the getter and setter
static {
String validateProperty = System.getProperty(VALIDATIONPROPERTY);
if (validateProperty != null) {
setSchemaValidation( Boolean.getBoolean(validateProperty));
logger.warn("Setting AntiSamy policy schema validation to '" + getSchemaValidation()
+ "' because '" + VALIDATIONPROPERTY + "' system property set to: '" + validateProperty + "'");
}
}

[jeremiahjstacey]

I had to appease my curiosity on the ByteArrayInputStream, and wanted to share the results. I duplicated my original tests and rebuilt the streams to pass into Policy.instance(InputStream). (File Attached below)

Using the ByteArrayInputStream appears to resolve the issue without requiring any other code changes.

Downside is having the file in memory to support the implementation. Significantly large files may have an impact on overall application performance.

InvalidPolicyTest.java

(Failed tests are the same ones listed in my original comment)

Tests in error:
  testSystemProp(antisamy_tests.InvalidPolicyTest): java.io.IOException: Stream closed
  testDirectConfig(antisamy_tests.InvalidPolicyTest): java.io.IOException: Stream closed
Tests run: 4, Failures: 0, Errors: 2, Skipped: 0

[gerardocanedo]

IMHO, changing the true constant to validateSchema does the trick. It does log a warning
[main] WARN org.owasp.validator.html.Policy - XML schema validation is disabled for a valid AntiSamy policy. Please reenable policy validation.

Also, there's a test case that sets a system property.
System.setProperty(ANTISAMY_PROJECT_PROP_SCHEMA_VALIDATION, "false");

If the Policy class is already loaded, setting the property has no effect.
Is this expected behavior ? I believe so

[kwwall]

@gerardocanedo wrote

If the Policy class is already loaded, setting the property has no effect.
Is this expected behavior ? I believe so

In the AntiSamy world, it might not be the correct behavior because you can an application that is using multiple policy files at once. In the ESAPI world--the way AntiSamy us is currently used there, it should be fine, as you can only use a single AntiSamy policy file that you need to have selected before AntiSamy is loaded.

The real question is: what effect should it have in AntiSamy if one first sets the owasp.validator.validateschema property to "false" and processes a policy where they want to ignore the schema, and then either clears that property using System.clearProperty() or instead, subsequently sets that property to "true" and then processes a 2nd policy file? Do you want the use of any subsequent policy files to skip XML schema validation? I don't think so.

Now obviously, if an application wants to use AntiSamy in that manner, it would be better off to use Policy.setSchemaValidation(), but people don't always do the obvious thing, especially when you give them multiple options to shoot themselves in the foot. Also keep in mind that to some degree we are trying not to break client applications here which is one reason why the System property approach was proposed, especially for ESAPI clients. (ESAPI had a bogus XML node called <html-entities> going back 10+ years or so, so @davewichers added it partly for them. If he hadn't, I would have and just would done something similar because I don't want to break production code.) For ESAPI clients though, it isn't a concern (unless they are also using AntiSamy separately) since ESAPI only can currently only ever use a single AntiSamy policy file.

So to get back to answering your question, I'm not sure there is a clear cut answer in all the edge cases here. @davewichers, what do you think? How do you want it to behave in the scenario I've portrayed?

[davewichers]

@kwwall - To me, system properties are once and done. If you change a system property while its already loaded, it won't change the policy in affect. That is what the individual set methods are for, if you want to change validation between one policy and another. If the property is the easiest thing for you, then disable it if you have even 1 non-compliant policy. Once you get all the policies fixed to be valid, then you can reenable it. So - the 'as is' implementation of the system property static initializer is fine with me.

[gerardocanedo]

Ok. I'm working on a test case with it's own classloader to verify the property.
Once it's done, i'll commit the fix

[kwwall]

Keep in mind that the new static method in Policy is not perfect either--at least not in the scenario that I outlined--as it is not really thread-safe. But I agree that it is way better than using a System property.

…

-kevin

On Sat, Mar 13, 2021, 10:15 AM Dave Wichers ***@***.***> wrote: @kwwall <https://github.com/kwwall> - To me, system properties are once and done. If you change a system property while its already loaded, it won't change the policy in affect. That is what the individual set methods are for, if you want to change validation between one policy and another. If the property is the easiest thing for you, then disable it if you have even 1 non-compliant policy. Once you get all the policies fixed to be valid, then you can reenable it. So - the 'as is' implementation of the system property static initializer is fine with me. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#75 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAO6PG3IAXNNSYT752FXHFDTDNXRFANCNFSM4ZCERNFA> .

[spassarop]

@spassarop, it may be a little cheaty -- ByteArrayInputStream does not implement close. If we read the original inputStream into a byte[], then build a ByteArrayInputStream to feed to the InputSource then the close behavior may not actually matter.

Honestly, independently of any logging behavior changes, this may be the best solution regarding re executing the parser. About the files being in memory... I don't know if a policy XML will ever be large enough to provoke a performance issue, but if it does, is only while parsing de Policy which is a really small ammount of time (at least from what I've seen).

Besides, I'm not expert here, but if you reset a stream, doesn't that mean that the stream must be somehow in memory already to return to the 0 index?

[jeremiahjstacey]

if you reset a stream, doesn't that mean that the stream must be somehow in memory already to return to the 0 index?

That's my understanding. The Stream needs to hold a certain set of bytes in order to read ahead or reset back. I've seen this used for stream processing where you can read a chunk of a file for processing. I think it would work for an entire file too.

You are right. It will need to be entirely in memory either way in this case.

I also think that using the reset behavior requires the stream to be open, so I think the code would have to rebuild the input stream anyway in order to process the file more than once since something is closing the stream in the document generation process.

[spassarop]

Well in that case... if everyone agrees, that stream pre-reading could be impacted on 1.6.2. It should be independent from Gerardo's changes as they do not touch the reset logic itslef.

Off-topic: My fist words remind me of this:

[jeremiahjstacey]

@spassarop Looked a bit more at the method I provided for the ByteArrayInputStream (above) and determined there is a possible resource leak since the original stream is never closed.
Made some updates to wrap the InputStream in an InputReader, which allows the use of try-with-resources to pull the file into memory and should guarantee the stream is cleaned up after use.

Updated method below, if this is useful to you.

  private static InputStream toByteArrayStream(InputStream in) throws IOException {
        byte[] byteArray;
        try (Reader reader = new InputStreamReader(in)) {
            char[] charArray = new char[8 * 1024];
            StringBuilder builder = new StringBuilder();
            int numCharsRead;
            while ((numCharsRead = reader.read(charArray, 0, charArray.length)) != -1) {
                builder.append(charArray, 0, numCharsRead);
            }
            byteArray = builder.toString().getBytes();
        }
                
        return new ByteArrayInputStream(byteArray);        
    }

[davewichers]

@spassarop - After merging I reviewed/tested your changes, and found some problems. I added 4 test cases to the new ESAPI related test cases, 1 or 2 of which fail when using either the old or new version of getTopLevelElement(InputSource source, Callable getResetSource).

If you start reading at: https://github.com/nahsra/antisamy/blob/1.6.2/src/main/java/org/owasp/validator/html/Policy.java#L370 I added comments in the file itself describing the problems I found. I also added back in the old version of this method and made a few improvements/fixes to your new version.

However, both methods still have problems and I'm not sure if the best course of action is to fix the old one, or the new one, which is why they are both in there now. The new version is problematic because the invalid policy error messages don't really explain the real validation problem. As such, I suspect it is better to fix the old method to work properly, pass all the test cases, and not fail when trying to reset different kinds of streams.

Maybe the very helpful @jeremiahjstacey can help us figure out the best approach. Jeremiah - can you review my comments inside the updated Policy file and let us know if you have any brilliant ideas?

[jeremiahjstacey]

My current thought is that if the original implementation provided a more accurate workflow for the desired logic blocks, then it seems like a shorter path to restore that logic. From conversations in this thread with @spassarop, I believe that we can use the resource-safe ByteArrayInputStream wrapper (defined here) and use that for the stream fed to the InputSource. At that point, the original issue with the closed streams should be resolved, and the original logic flow will be restored.

At that point, the test cases that @gerardocanedo has provided will need to be updated to reflect these expectations. Tests should also be added for the explicit logging expectations of the behavior to capture the requirements in future efforts. (my opinion)

From a recommendation standpoint, I think that makes the most sense to me.

[jeremiahjstacey]

If we wanted to keep the updates to Policy.java, then pushing the logging down another level may also provide a path to resolution.

We touched on that possible update in this comment

Test updates may still be needed. Tests to assert the logging behavior would still be recommended.

[davewichers]

Regarding the use of ByteArrayInputStream wrapper, I think we'd have to do that inside of the AntiSamy methods as the caller should be blissfully unaware that the stream needs to be resettable. As such, I don't think we should have to change the test cases at all. Is there a way to 'test' whether a given stream is resettable, and if not, wrap it. But if it already is, you avoid double wrapping it for no reason?

I already have test cases that test the logging behavior, but they don't just pass/fail. Rather you have to look at them to make sure they look right. That's how I implemented/tested the logging behavior in the first place and noticed the problem this time. However, if there was a way to grab the 'most recent' log messages and make sure they contain certain strings, we could add pass/fail type test cases so human review wasn't required anymore. However, we can deal with that later. Let's get it working as desired, and then we can see if we can figure out a way to make the desired logging behavior automatically testable.

[gerardocanedo]

Can the stream be consumed once, and after that all validations made with the DOM?

[davewichers]

@gerardocanedo - test it yourself. Without reparsing the original stream, the error messages returned are completely different, and not helpful.

[jeremiahjstacey]

@davewichers, `InputStream.markSupported' may tell us whether or not an instance will support the reset behavior. TBH, I don't think the underlying issue isn't that the stream supports the reset function- it's an expectation declared in the documentation of the InputSource class:

 *
 * <p>An InputSource object belongs to the application: the SAX parser
 * shall never modify it in any way (it may modify a copy if
 * necessary).  However, standard processing of both byte and
 * character streams is to close them on as part of end-of-parse cleanup,
 * so applications should not attempt to re-use such streams after they
 * have been handed to a parser.  </p>

http://developer.classpath.org/doc/org/xml/sax/InputSource-source.html

I have not tested it, but I would assume that calling reset() on a closed InputStream would result in an IOException. The documentation on InputStream does not explicitly define this behavior, so it would need to be verified.

To get a more granular solution, I think we'd need to dig into the parsing behavior and figure out where the stream is being closed at. That may provide insight into ways we can influence the parser or the InputSource to change the behavior.

[davewichers]

I don't expect AntiSamy policies to get very long (a few K at most). As such, I suggest we make a copy in memory of the entire thing at the very start (e.g., a StringBuffer or whatever). And then turn that into a standard input stream we use. We then parse it, and if we have to parse it again, we create another input stream for it, and do it again. We then make sure we always close our internal streams. And then also close the original stream provided to use by the caller. I think this will simplify things.

[davewichers]

@gerardocanedo - please rebase again before issuing a pull request as I've checked in a number of changes, including more just now. This will minimize merge pain.
@kwwall - I added some info in the 1.6.2 branch to the README and the Policy methods related to disabling schema validation to indicate they are immediately deprecated and we plan to make them 'go away' in mid/late 2022 in the 1.7 release. Let me know what you think about how I've described this.

[jeremiahjstacey]

@davewichers , your suggestion on storing it in memory is effectively what the ByteArrayInputStream allows. By leveraging the fact that it does not implement close we can reuse the in-memory copy of the file as the byte array that is capture.
It has the added benefit of already being an InputStream, and allows the InputSource to be reused.
The only cost is to update the InputStream passed to the InputSource and the problem with the "reuse" of a Stream is mitigated.

private static Element getTopLevelElement(InputStream is) throws PolicyException {
        final InputSource source = new InputSource(toByteArrayStream(is));

        return getTopLevelElement(source, new Callable<InputSource>() {
            @Override
            public InputSource call() throws IOException {
                source.getByteStream().reset();
                return  source;
            }
        });
    }
    
    private static InputStream toByteArrayStream(InputStream in) throws PolicyException {
        byte[] byteArray;
        try (Reader reader = new InputStreamReader(in)) {
            char[] charArray = new char[8 * 1024];
            StringBuilder builder = new StringBuilder();
            int numCharsRead;
            while ((numCharsRead = reader.read(charArray, 0, charArray.length)) != -1) {
                builder.append(charArray, 0, numCharsRead);
            }
            byteArray = builder.toString().getBytes();
        } catch (IOException ioe) {
            throw new PolicyException(ioe);
        }
                
        return new ByteArrayInputStream(byteArray);        
    }

Note the use of the Java7 try-with-resources and the wrapping of the original input stream into an InputStreamReader. This will close the original stream once all the bytes are read.

[davewichers]

Thanks @jeremiahjstacey. I changed these methods exactly as suggested and now my original method has exactly the same 2 test case failures as the new method. But at least the error messaging around invalid schema's retains its 'better' info. We now need to track down why those two test cases are failing. Do you or @gerardocanedo have time to try to track down the cause/fix for these last two failing test cases??

[kwwall]

@davewichers - I've reviewed your README.md and I think it is reasonable. OTOH, if your mid-late 2022 date becomes mid-2021, then ESAPI might just decide not to update to your latest version.

…

-kevin

On Tue, Mar 16, 2021, 4:40 PM Dave Wichers ***@***.***> wrote: @gerardocanedo <https://github.com/gerardocanedo> - please rebase again before issuing a pull request as I've checked in a number of changes, including more just now. This will minimize merge pain. @kwwall <https://github.com/kwwall> - I added some info to the README and the Policy methods related to disabling schema validation to indicate they are immediately deprecated and we plan to make them 'go away' in mid/late 2022. Let me know what you think about how I've described this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#75 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAO6PG56FAGS3WVTJCALXCDTD662XANCNFSM4ZCERNFA> .

[davewichers]

I agree. That's why I don't intend to screw over ESAPI by changing this behavior sooner than that, unless you think we should because the ESAPI community is ready for that type of change.

[jeremiahjstacey]

I won't be able to do it tonight, I apologize. I may be able to take a look tomorrow morning. Can you confirm which branch I should clone to see the test failures?
I'm assuming this is all in the 1.6.2 branch on the project, just want to verify.

[davewichers]

OK. Done! I think. The failing test cases were caused by MY bug that I introduced back in 1.6.0 that I found by adding more test cases. So my fault. ESAPI team, can you test with this 1.6.2-SNAPSHOT version from the 1.6.2 branch to make sure you are happy with how everything works before I push 1.6.2 out? Hopefully we can avoid the need for a 1.6.3 in the next few days at least :-)

[jeremiahjstacey]

All tests pass.

1. Cloned the antisamy repository (https://github.com/nahsra/antisamy.git)
2. updated to the 1.6.2 branch: antisamy]$ git checkout 1.6.2
3. Ran antisamy]$ mvn clean install locally to stage artifacts in my .m2
4. Updated the antisamy dependency in the esapi/pom.xml on my fork to the locally-staged 1.6.2-SNAPSHOT

       <dependency>
            <groupId>org.owasp.antisamy</groupId>
            <artifactId>antisamy</artifactId>
            <version>1.6.2-SNAPSHOT</version>
        </dependency>

5. Updated the HTMLValidationRule class implementation to remove pre-wrapping of ByteArrayInputStream
6. Ran esapi]$ mvn clean test

[davewichers]

Thanks @jeremiahjstacey. Anyone else need/want to review or provide feedback before I hit publish (again)? I can't do so until tonight so you have all day before I can even get to it.

[kwwall]

@davewichers - Bottom line, I'm okay if you want to deploy to Maven Central as is, but see below. *GOOD NEWS* - All my tests execute as expected. *SURPRISING NEWS* - The only possible discrepancy that I see (and I'm fine with it personally) is that if I run a test where I've explicitly set the AntiSamy system property "owasp.validator.validateschema" to "true" (so that schema checking *is* enabled) I see the following *warning* logged: [main] WARN org.owasp.validator.html.Policy - Setting AntiSamy policy schema validation to 'true' because 'owasp.validator.validateschema' system property set to: 'true'. Note: this feature is temporary and will go away in AntiSamy v1.7.0 (~mid/late 2022) when validation will become mandatory. I thought that if you weren't going to disable the schema checking that all the logged messages that were not actually errors would come out as *INFO* rather than as *WARN*. And this isn't really an error. OTOH, I'm fine with it as-is because I guess the real unstated message here is "You shouldn't expect to be able to enable / disable the AntiSamy XML schema validation via the 'owasp.validator.validateschema' system property and not so much whether or not you have just explicitly disabled it." and taken that way, WARN is appropriate. I was just surprised based on your original logging code to see this one wasn't INFO rather than WARN. *BAD NEWS* - Using slf4j-simple with no additional explicit configuration, your logging goes to stdout (System.out) rather than stderr (System.err). Stdout is buffered, which means that your log messages fail when AntiSamy throws an PolicyException you may lose the SLF4J log output. This in fact occurs in ESAPI where, in a static initializer block in ESAPI (in HTMLValidationRule), as the result of AntiSamy's PolicyException, it results in an java.lang.ExceptionInInitializerError being thrown from ESAPI and unless you are catching Throwable or Error, you lose the whole reason for the original error. If you do catch it and print the exception stack trace, it has all the relevant information, but there is no guarantee that people are going to think they need to catch ExceptionInInitializerError in ESAPI. (Note: I haven't tested this with @jeremiahjstacey's PR so maybe things are better there after his code clean-up.) The good news (not verified) is that most JavaEE application servers will catch Throwable and at least try to log a complete exception stack trace (although it often gets truncated) before spinning off a new thread or whatever. (I recall writing code to test this way back in BEA WebLogic Server. I couldn't kill WebLogic from a web application using System.exit(1) or by throwing an OutOfMemoryError, but did get a complete stack trace for each. Not sure how they managed that, but it was impressive.) For the record, if you do catch ExceptionInInitializerError or some base class and print the stack trace, it will look something like this (with the important parts shown in red): === Exception stack trace follows === java.lang.ExceptionInInitializerError at org.owasp.esapi.reference.validation.HTMLValidationRuleBogusAntiSamyPolicyFileTest.testBogus(HTMLValidationRuleBogusAntiSamyPolicyFileTest.java:111) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:258) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.ParentRunner.run(ParentRunner.java:413) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) Caused by: org.owasp.esapi.errors.ConfigurationException: Couldn't parse antisamy-InvalidPolicy.xml at org.owasp.esapi.reference.validation.HTMLValidationRule.<clinit>(HTMLValidationRule.java:150) ... 32 more Caused by: org.owasp.validator.html.PolicyException: org.xml.sax.SAXParseException; lineNumber: 23; columnNumber: 19; cvc-complex-type.2.4.a: Invalid content was found starting with element 'notSupportedTag'. One of '{allowed-empty-tags}' is expected. at org.owasp.validator.html.Policy.getTopLevelElement(Policy.java:399) at org.owasp.validator.html.Policy.getTopLevelElement(Policy.java:373) at org.owasp.validator.html.Policy.getInstance(Policy.java:253) at org.owasp.esapi.reference.validation.HTMLValidationRule.<clinit>(HTMLValidationRule.java:148) ... 32 more Caused by: org.xml.sax.SAXParseException; lineNumber: 23; columnNumber: 19; cvc-complex-type.2.4.a: Invalid content was found starting with element 'notSupportedTag'. One of '{allowed-empty-tags}' is expected. at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source) at org.apache.xerces.util.ErrorHandlerWrapper.error(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.xs.XMLSchemaValidator$XSIErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.xs.XMLSchemaValidator.reportSchemaError(Unknown Source) at org.apache.xerces.impl.xs.XMLSchemaValidator.handleStartElement(Unknown Source) at org.apache.xerces.impl.xs.XMLSchemaValidator.startElement(Unknown Source) at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at org.apache.xerces.parsers.DOMParser.parse(Unknown Source) at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source) at org.owasp.validator.html.Policy.getDocumentElementFromSource(Policy.java:456) at org.owasp.validator.html.Policy.getTopLevelElement(Policy.java:386) ... 35 more Anyway, as I mentioned, I'm okay with you deploying this as-is.

…

-kevin -- Blog: https://off-the-wall-security.blogspot.com/ | Twitter: @KevinWWall | OWASP ESAPI Project co-lead NSA: All your crypto bit are belong to us.