pshtt scan exception case drops record from report about domain-scan

[konklone]

@egyptiankarim I'm closing this because I think it's made moot on the domain-scan side by the refactor in #155.

Try using the --meta flag to capture local errors in scan output, and let me know if you're still seeing issues with this.

You may also want to try the Lambda pipeline, which definitely returns stack traces when errors are observed.

[konklone]

@egyptiankarim Want to email me a live example so I can test? ([email protected])

The reason I'm bringing it up here, is because pshtt will produce a report with these exception cases properly reflected (i.e., as failing), but domain-scan just ends up dropping them from the report all together, which can be confusing as anything when your target list of 12K domains only results in a results.csv of 11,999 rows (gah!).

Totally get this. What do you think should be in the 12,000th row (the row representing the failed run) for such a scan?

[egyptiankarim]

Want to email me a live example so I can test?

@konklone Will do. Email on its way.

What do you think should be in the 12,000th row (the row representing the failed run) for such a scan?

Well, just generally speaking, I think domain-scan ought to mimic whatever a regular run of the underlying scanner would give you. For the exception cases in question here, a regular pshtt scan eventually gives us a "Live" True with a failure on "Valid HTTPS" and related attributes, which is a fair enough glimpse of the situation.

I think there's probably room for improvement on how we handle RequestExceptions (especially weird code 500 and redirect loop situations), and I'll puzzle over it, but for now I know that I get some result back running a regular pshtt scan, so whatever domain-scan gives me should approximate that rather than just dropping it.

Also, I'll emphasize again that I think the "fix" for all of this is in revisiting the logic in pshtt, and I'll be working a pull request there. I just wanted to post an issue on domain-scan so that people would be aware of the gap in reports produced by each.