Comments (8)
Which user agents did you try? Using the Googlebot user agent should not cause a prerendered page because that would be cloaking. We use the _escaped_fragment_
protocol for Googlebot.
Here's the list of user agents we check for: https://github.com/prerender/prerender-node/blob/master/index.js#L37
Let me know if there are any user agents that we should add.
from prerender-node.
Using my Chrome user-agent changing extension I tried Googlebot, Yahoo, and BingBot. That would explain why all three didn't work. Sorry to trouble you, and good to know you're looking out for us with cloaking.
I guess I would have to ask, is it guaranteed that Googlebot will always use escaped_fragment when querying? We had prerender in production for about a month and didn't see much happen to our indexes in google. For example, we still had angular curleys in our meta description.
Edit: I was just observing our logs and this came through: http://i.imgur.com/7C2Mjhk.png
Is it correct to say this is Googlebot crawling our site? If so, the prerendered version wasn't served back, as we log when that happens too and it wasn't in the log. At this point I don't expect the problem to be with the library, just me.
from prerender-node.
If you are using html5 push state (or any URL without the #!), you need to have <meta name="fragment" content="!">
in the <head>
of the page. That is the signal to Google and Bing that you want them to use the _escaped_fragment_
when querying. They will then see that meta tag and re-request the page with the escaped fragment URL, which will then be matched by the middleware and proxied to our service.
So make sure you have that meta tag in your pages and then they should be requesting the escaped fragment URL.
from prerender-node.
Ah! That was it! I've never come across that in my reading on the internet - how odd...you've saved me. Thank you.
I was just reading the code to see if there was a way to add our own bot user agents to the crawlerUserAgents array at runtime. Is this a planned feature?
from prerender-node.
It would be something like this:
var prerender = require('prerender-node');
prerender.set('crawlerUserAgents', ['facebook', 'twitter']);
prerender.set('prerenderToken', 'YOUR_TOKEN');
app.use(prerender);
That will override the crawlerUserAgents list so make sure you add all of them to that list.
from prerender-node.
Ahh, nice convenience method. Didn't realise until I looked into the code how it worked. Thanks for that!
from prerender-node.
I'd recommend prerender.crawlerUserAgents.push('userAgent');
(use push, not set)
from prerender-node.
I agree! This is an old comment :) It should be:
var prerender = require('prerender-node');
prerender.crawlerUserAgents.push('userAgent'); //adds a user agent to the list instead of overriding
prerender.set('prerenderToken', 'YOUR_TOKEN');
app.use(prerender);
from prerender-node.
Related Issues (20)
- how can we judge that the bot is not requesting a resource?
- Add msnbot for results with "Fetch as bingbot"
- Ungzipping gzipped response from prerender service produces corrupted unicode strings HOT 1
- Cloudflare caches pre-rendered response HOT 1
- request dependency has been deprecated HOT 2
- Add hacktoberfest topic
- webp is missing in extensionsToIgnore HOT 1
- Circular redirect while page prerendering HOT 1
- Prerender.io's cached html header getting replaced by static header
- Access denied or empty body
- fastify missing middleware
- How to check for X-Prerender:1 using Express middleware
- [Feature Request] May include a bot detection library rather than a self maintained list? HOT 1
- Add new user agents to every known middleware and docs HOT 7
- Request for changelogs HOT 1
- Provide typescript type definitions HOT 2
- Not working under Meteor 2.5.3 HOT 5
- `req.connection.encrypted` is `undefined` in node:14 HOT 1
- Fastify support?
- Could not find a declaration for module "prerender-node" HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from prerender-node.