please add Host: to robots.txt about the-seo-framework

[sybrew]

Hi @antonmarin,

This is already done through the Canonical URL. See Use canonical URLs.

I quote:

[...] With content syndication, it's also easy for content to be distributed to different URLs and domains entirely. For example:

Your server is configured to serve the same content for the www subdomain or the http protocol.
a) http://example.com/green-dresses (http)
b) https://example.com/green-dresses (https)
c) http://www.example.com/green-dresses (http + www)

In the next update you'll have more control over the HTTP/HTTPS hosts directive. www/non-www is already completely supported through the Site Address (URL) setting set in http://example.com/wp-admin/options-general.php.

I hope this explains it :)

If this wasn't what you were looking for, let me know! Cheers!

[antonmarin]

I see.
As i understand it is not absolutely same for different search engines

[sybrew]

The canonical URL certainly works the same for all popular search engines:

https://yandex.com/support/webmaster/controlling-robot/html.xml#canonical
https://blogs.bing.com/webmaster/2011/10/06/managing-redirects-301s-302s-and-canonicals/

The hosts addition, alas, is currently only Yandex specific.

[antonmarin]

ok.
How add it manually? I searched for a file but not found (( File exists? Or it's a redirect?

[sybrew]

Hi @antonmarin,

The robots.txt "file" is virtual, and it's outputted by WordPress and overwritten by The SEO Framework.
If an actual robots.txt file is found in the root folder of your website, you're in full control of it 😄. You can add your own manually through FTP.

I hope this helps! Good luck!

[antonmarin]

So how to add Host: to robots.txt if file doesn't exist?

[sybrew]

You can create files through FTP.
Please advice on this with your hosting provider :)

[sybrew]

Hi Anton,

I hope you're doing well, and that you've been able to add your own robots.txt file.
If you still have any questions regarding this subject, feel free to ask!

I'm also closing this topic; because this feature has been included in other forms already.

Cheers!