Currently the NYC Department of Education website (http://schools.nyc.gov) is preventing Yahoo and MSN search engines from including its webpages in their search indices. If any parent tries to use Yahoo.com or MSN.com to search for information found on http://schools.nyc.gov, Yahoo and MSN will not be allowed to find it for the parent. The parent will be forced to use Google.com which is the only major search engine allowed to index content on http://schools.nyc.gov.

The technical reason why Yahoo and MSN is not allowed on the NYC DOE website is that there is a robots.txt file which is disallowing Yahoo’s Slurp bot and MSN’s msnbot but allowing Google’s googlebot. Combined Yahoo and MSN represent approximately 30% of all internet searchers. I think it is not a good thing that the New York City Department of Education website is blocking 30% of internet searchers from finding information.

It is very simple to allow Yahoo and MSN include the pages from NYC DOE website. The robots.txt file which is located at (http://schools.nyc.gov/robots.txt) simply needs the four lines that reference Slurp and msnbot to be deleted. Thank you for your time and I look forward to having this website available to all internet searchers.

Postnote : I no longer offer consulting but I will be happy to write the new robots.txt file and upload it for free. If you are worried about bandwidth costs I will also be happy to help with hosting. I’d be happy to help make this website a better resource for all.