Currently the same robots.txt is in use on both Prod and Non-Prod sites, which means that we are seeing Staging URL's listed in Google. The Robots.txt needs to be changed for Staging vs Prod to disallow crawling for non-production environments.
Currently the same robots.txt is in use on both Prod and Non-Prod sites, which means that we are seeing Staging URL's listed in Google.
The Robots.txt needs to be changed for Staging vs Prod to disallow crawling for non-production environments.