This best practices gives us a pretty good breakdown of how we can guide search engines more, including linking to a sitemap

https://www.searchenginewatch.com/2019/03/21/robotstxt-best-practice-guide/

https://www.advancedhtml.co.uk/robots-sitemaps.htm/

I suspect that we can gain a lot of power by setting this up. It will also be work looking into how Facebook publishes its "Public preview"

added to epic &114

changed title from Add a robots.txt to Add a robots.txt

changed the description