Add support for scanning more than just the root document about http-observatory

Hey @globau and @marumari, I've been doing a pile of work to add support for this over on my fork. It's a fairly large change that touches so much of the codebase, could both of you give me any early feedback before I start raising pull requests?

[april]

Oops, I thought I had commented about it. I'll get to it today. Thanks! :)

@marumari did you have a chance to take a look at it? I'm more than happy to open a "I'm not ready" pull request to take the discussion somewhere or we can chat on IRC.

[april]

Grr, I had totally written up a comment on it, and thought I had sent submit. I must have lost it when my browser crashed the other day. :(

I think it's probably a bit more invasive than what I was thinking about doing. Here's what I was thinking:

• Add an extra column to "scans" that is the path (varchar, allow null)
• If no path is set, leave it as null
• If a path is set, also mark the scan as hidden, so that you can't game the leaderboards

I think that's probably the simplest way to get it going. I'm trying to avoid having the database be too normalized, perhaps you can explain your reasoning behind having an extra table for the paths?

Thanks for all your hard work, and sorry for dropping the ball on this. I've been (and am) out of town, and things have been a bit crazy with the release.

Thanks for the reply.

To understand why I went with a separate table - my employer only has a few domains but hundreds of top level directories to segregate products and with a desire to use the observatory with less emphasis on scoring at a domain level, but at a per domain and path.

I do like the simpler approach and it's probably the best way to do things for now, I'll try to make some time over the next few days to make the changes. Did you have any thoughts on how schema migrations might be dealt with?

[april]

Should just be able to alter the table to add a column of varchar(2048) allowing nulls with a default value of null. I think that should be all that's needed. I can probably work that out though and I'll post it in this thread when I update the schema file.

I was more thinkng about how this might be tooled as now there are deployments of it out in the wild, for example maybe something like this?

So I've started making the changes as per your feedback @marumari, is this closer to what you had in mind? I have a few other pieces to do before I think this'll be ready for a pull request.

[jarondl]

Hi @marumari and @TigerToes, any news about this feature? It could be really useful for me.

[april]

It'll be coming this quarter for sure. :)

[jarondl]

Great! Anything I can do to help?

[jarondl]

Since this change is rather intricate, and has to check plenty of implicit assumptions,
maybe we should use the chance to allow testing of different ports? (see #77)
After all, different paths and different ports sound vary similar, although the top domain resources (/robots.txt etc) are a place where these paths will differ.

[april]

I believe the port thing should be another bug. It's coming eventually, I promise!

[april]

Note that this is now available with local-only scans. It's still something that needs to be worked on for the public site. :)

[tlmii]

Do you have any ETA on this for the public side? We are doing an upgrade and would like to use observatory to verify if it will be ready soon.

[april]

I don't. I hope to get it done this quarter, but I'm the sole developer. :)

Does the local scan not work for you? It should be very easy to do, with a simple command line tool if needed.

[floatingatoll]

@globau The (relatively new) local scanner accepts http port, https port, and path to scan as the three arguments you'd need to specify to scan any random site. It's not apparent whether the public scanner will accept custom port/path combinations, due to its nature as a public site. Does the local scanner (search for https_port in README.md) meet your needs for the purposes of this issue?