This best practices gives us a pretty good breakdown of how we can guide search engines more, including linking to a sitemap

https://www.searchenginewatch.com/2019/03/21/robotstxt-best-practice-guide/

https://www.advancedhtml.co.uk/robots-sitemaps.htm/

I suspect that we can gain a lot of power by setting this up. It will also be work looking into how Facebook publishes its "Public preview"

added to epic &114

changed title from Add a robots.txt to Add a robots.txt

changed the description

We should add SEO as a label. Not sure what category it would go under.

added scoped label

assigned to @brianhatchet and unassigned @ottman and @jotto141

added scoped labels

changed due date to January 28, 2020

changed milestone to %Sprint::01/15 - Youthful Yabby

added scoped label and automatically removed label

added scoped label and automatically removed label

assigned to @eiennohi and unassigned @markeharding

changed time estimate to 4h

changed weight to 4

unassigned @brianhatchet

How about this for the robots.txt?

User-Agent: *
Disallow: /admin
Disallow: /search
Disallow: /groups/create
Disallow: /blog/edit/new
Disallow: /notifications

changed milestone to %Sprint:01/29 Abatic Aenome

added scoped label and automatically removed label

assigned to @markeharding and unassigned @eiennohi

added scoped label and automatically removed label

@markeharding is going to take care of this one. We don't need to worry about the robots.txt, but he'll generate a new site map within SSR

removed due date

moved to engine#1348