Control search engine indexing about st2docs

[arm4b]

👍
Something to discuss with @alexmakarski

[LindsayHill]

Agreed. Need to have banner across top of old docs saying "THESE ARE OLD" or something like that - see the way http://docs.influxdata.com/influxdb/v1.1/ does it.

Probably needs robots.txt or similar updated to stop google indexing old versions?

[Mierdin]

Personally I would prefer to make it difficult to even land on an old page. So I'd do both - put up a big banner saying "this is old", similar to how influx does it, but in addition, force the user to go out of their way to get to an old doc. So they won't even get there unless they're meaningfully doing it.

Even worse - and I haven't had time to dig into why this is - our docs will display an old version of the docs even if the URL says "latest":

I got to this page simply by googling "stackstorm actions". That definitely shouldn't happen

[alexmakarski]

@LindsayHill I wouldn't hastily de-index the old stuff, especially if it's bringing in traffic. I think the Influx approach is more sound.

[LindsayHill]

@Mierdin I think that you got those old versions because of some page re-arrangement we did. What was /runners.html got moved to /reference/runners.html. It looks like the build processes are only additive though - they're not deleting previously built content.

[Mierdin]

If the old stuff is bringing in traffic, is that really traffic that we want? At best, those situations are providing two google results - one with the correct version, and one with the old version - forcing the user to choose. At worst, users are finding references to concepts that have either radically changed or simply don't exist anymore. I don't think that traffic is what we're after.

@LindsayHill is probably right, and we just need to clean up the old stuff somehow

[LindsayHill]

I don't know if the cleanup is something easily automated (e.g. do a make clean; make docs). Depends how the sync with S3 step is done I guess - I don't know the details of that workflow

[arm4b]

This robots.txt should work to disallow Google indexing versioned doc URLs:

User-agent: *
Disallow: /0.*/
Disallow: /1.*/
Disallow: /2.*/
Disallow: /3.*/
Disallow: /4.*/
Disallow: /5.*/

[LindsayHill]

We could look at using the <priority> tags in sitemap.xml - see https://www.sitemaps.org/protocol.html

[LindsayHill]

I think there's 3 separate, but related issues here. These are:

1. Some doc pages have been moved/deleted over time, especially around v1.6->2.0 transition. The docs build & deployment process does not delete those pages, so we end up with things like an old version of docs.stackstorm.com/policies.html and the updated content at docs.stackstorm.com/reference/policies.html
2. On our old docs pages, we don't give users any indication that they are looking at outdated content.
3. Google is indexing version-specific docs, e.g. docs.stackstorm.com/1.6/actions.html, rather than docs.stackstorm.com/actions.html. We need to get Google to prefer to use the current docs, or possibly /latest (see more below).

This is what I think we need to do:

• Create placeholder pages containing the equivalent of <meta http-equiv="Refresh" content="0; url=http://www.example.com/" /> for those pages that have been moved. Ideally do this in Restructuredtext, so output gets auto-generated. Or we could do it via AWS API for S3 - see https://medium.com/cloud-io/put-redirect-301-in-s3-cloudfront-hosted-website-39bd9104ddbc. This would be a one-off manual task. Would require doing again in future if any other pages get moved.
  By putting in the 301 redirects, we'll steer Google & users to new content, and not lose rank. #453

• Figure out some way to auto-insert a 'this doc is old, go to location instead' banner in old docs branches. Needs to be something that gets done when we go through release branch process, as last change made to old version. - #455

• I think there's 3 options here for steering Google towards latest docs:

  • Configure robots.txt as above, to disallow searching under version-specific branches
  • Configure Canonical URL in our docs, point that to docs.stackstorm.com/. Should be able to do that with our theme.
  • Configure priority in sitemap.xml. Not sure if I can auto-generate that though.

  I think we should start with the second option. Relatively low impact. I'm not against also configuring robots.txt.

#443

Related: I've been wondering if we should use /latest as the latest stable docs branch, not the dev branch. Then we could have canonical URLs, etc pointing to /latest. Not sure on that.

[LindsayHill]

PR submitted to address item 2, redirects #453

Now I just need to figure out banners on old content...

[LindsayHill]

Bunch of Pull requests have gone in to address this, and things are looking much better. Haven't done final testing with version picker, but this looking much happier. Google searches are now leaning towards the current docs, and redirects are in place.