Refactoring Proposal

Update our robots.txt to avoid allowing official bots to access our API/exhibit pages. Bots crawling through the site are generating load and errors by crawling non existent exhibits and navigating API pages. This includes dev.jandig.app which should be excluded from indexing if possible.

Describe the design issue(s) you have identified in the code
Update the robots txt to only allow our blog/collection pages, to filter requests