investigate using alternate scanner about concourse-compliance-testing

[psiinon]

What problems have you encountered?
And how can we, the ZAP core team, help?
Thats what we're here for :)

[afeld]

Hi! I kind of felt bad about posting this, because you have been super helpful and supportive in the millions of times you've come up in Google searches 😄 The work I'm doing is here, if you're interested: #100

I've been wanting to post issues as I came across them, but have been in hustle mode, so didn't get a chance to (yet). For context: we have a bunch of sites we scan using ZAP via Concourse CI, and that number is growing all of the time. We have a number that use single-sign on, so I'm trying to automate getting through that so the sites can be fully tested. The high-level issues I've run into with ZAP in working on this feature:

• I had trouble finding complete examples of automated ZAP workflows. It's taken a lot of trial and error.
• It's not always easy to find the API calls that correspond to actions in the GUI.
• The API is inconsistent, e.g. using contextName in some places, vs. contextId in others.
• The API results don't always give the information I want, e.g. JSON/context/view/contextList/ gives the list of names (as a stringified array?), but not the IDs.
• Some of the fields that aren't marked as "required" in the API UI for various endpoints are in fact required, even if they are empty query parameters.
• Even after reading the docs and watching the videos multiple times, it still isn't entirely obvious to me how auth scripts get triggered.
• When using script-based auth via the API, it's not clear how to capture the output, which makes debugging challenging. I ended up switching to using ZAP as a proxy from Python so I had more control.
• The ZAP GUI on Mac provides the Nashorn JS engine, whereas zap2docker-stable provides Rhino. This was a not-pleasant surprise (I couldn't figure out why my script was working in one place but not the other), and it doesn't seem to be documented. Would be nice to pick one and use it in both places.
• Since some of our sites that require auth immediately redirect to the single-sign on provider (which lives at a separate domain), I'm needing to juggle page visits and adding sites to the Context in order for the redirects to be followed by ZAP. Feels like there should be a simpler way. I think I'm close to getting this to work, though.
• The Python client documentation isn't hosted anywhere, and I've had to do a lot of digging through the source to figure out how to use it.
• The Python client doesn't give the full errors (unless there's some configuration I missed?), so I ended up switching to doing HTTP calls directly so I can see what's happening.
• Scripting with the Python library requires a lot of sleeping...would be nice to have that built in as an option to make users' lives easier. Also, it's not obvious to me when something's not working as expected, or if I just didn't sleep long enough, or what.
• I got a handful of inexplicable errors through the API (e.g. Unable to persist User), which led me to digging through the source of ZAP itself.
• The GUI seems to have some state bugs where buttons/menus become unclickable or dropdowns are empty when they shouldn't be, so I moved to scripting it via Docker earlier than I would have otherwise because then at least it's starting from a clean slate each time.
• The possible values for the API endpoints aren't always clear, e.g. authentication/action/setAuthenticationMethod/ taking authMethodConfigParams.
• Pressing ENTER in the forms through the API UI doesn't submit the form.

Will add to this list if I remember others. I really want to like ZAP, but it just feels like I've had to jump through a lot of hoops to do setup that doesn't seem like it should be super complicated. I know that's a lot all at once...thanks in advance for any response!

[psiinon]

Er ... best advice - talk to us as and when you hit issues rather than waiting until you feel overwhelmed :P
They all look like completely understandable problems, but what you're trying is completely in line with what we want to support so we want to fix them. And if you're ok with using the weekly releases we can get any code fixes required to you relatively quickly.
My advice - take things one step at a time.
Whats the first thing that you want to get working - one that will show you that you've made a start? And what problems have you encountered with ZAP that prevent you from achieving that?
Thats what we'll help you with :)
If you're ok with that then are you ok moving this discussion to the ZAP User Group? https://groups.google.com/forum/#!forum/zaproxy-users
I'm sure lots of other ZAP users can learn from this, and some might even have good solutions for some of your problems.
Many thanks

[afeld]

talk to us as and when you hit issues rather than waiting until you feel overwhelmed

Heh, yeah, definitely hear that. I was trying to do this in a rush so was trying to get it working however I could, but will hopefully have time over the coming week or so to step back and open bug reports or start discussions where appropriate.

if you're ok with using the weekly releases we can get any code fixes required to you relatively quickly

👍

If you're ok with that then are you ok moving this discussion to the ZAP User Group?

Yep! Will try and post an overall "how would you approach this?" discussion there soon.