Robots.txt for different sites

Chandra Shekar

Vote:

Hi All,

I have a EPi server web application. The same web application is hosted under 3 domains. Now my requirement is to have diffrenet Robots.txt for different domains. But since all the domains are pointing to sane folder I can have only file placed in that folder. Is there any workaround for placing seperate Robots.txt file under each domain.

Thanks in Advance.

#33163

Oct 03, 2009 11:03

Maras M

Vote:

You can create 3 files:

robotsDomain1.txt

robotsDomain2.txt

robotsDomain3.txt

and then define your own handler, assign it to robots.txt in web.config

<httpHandlers>
<add verb="*" path="robots.txt" type="Project.RobotsHandler"/>
</httpHandlers>

and inside the handler check Request.Url.DnsSafeHost and return the content of one of the files. If you use IIS7 you define handlers in section

<system.webServer>
<handlers>
<add name="robots" path="robots.txt" verb="*" type="Project.RobotsHandler, Project" />
</handlers>
</system.webServer>

#33165

Oct 03, 2009 11:14

Chandra Shekar

Vote:

Thanks for your reply.

Can you point me to any URL that explains the code to be written in the Handler.

#33166

Oct 03, 2009 11:39

Maras M

Vote:

http://www.kleenecode.net/2007/11/17/dynamic-robotstxt-with-aspnet-20/

#33167

Oct 03, 2009 12:11

Chandra Shekar

Vote:

Thanks Maras. Below link is also usefull

http://www.atlasbay.com/2008/11/06/robotstxt-httphandler-in-iis7/

Thanks for your suggestion.

#33168

Oct 03, 2009 12:25

Try our conversational search powered by Generative AI!

Robots.txt for different sites