# /robots.txt file for http://webcrawler.com/ # mail webmaster@webcrawler.com for constructive User-agent: * Disallow: /tmp Disallow: /temp Disallow: /logs Disallow: /db Disallow: /errors Disallow: /nukeALT Disallow: /admin Disallow: /catalog Disallow: /include Disallow: /lib Disallow: /maillist Disallow: /mysqladmin Disallow: /newsletter Disallow: /PHPchat Disallow: /search Disallow: /usage Disallow: /albums Disallow: /themes Disallow: /Album Disallow: /video Disallow: /nuke/html/modules/PostCalendar Disallow: /html/webcam