I client of mine resently ran into a wierd problem.
I had a robots.txt on a server but it didn’t get send to the client.
It seemed that the server didn’t allow txt files to be send.
This is easy to solve thou.
Simply add the settings below to the web.config (/configuration/system.webServer/security/requestFiltering/fileExtensions)
<fileExtensions> <removed fileExtension=".txt" /> <add fileExtension=".txt" allowed="true" /> </fileExtensions>