Why does the browser not robots.txt file from the server?

Server didn't send robots.txt to the client

Posted by Michael Sundstrøm on 2014-07-02
Tags

I client of mine resently ran into a wierd problem.

I had a robots.txt on a server but it didn’t get send to the client.

 

It seemed that the server didn’t allow txt files to be send.

This is easy to solve thou.

Simply add the settings below to the web.config (/configuration/system.webServer/security/requestFiltering/fileExtensions)

<fileExtensions>
  <removed fileExtension=".txt" />
  <add fileExtension=".txt" allowed="true" />
</fileExtensions>