Robots Generator

directory secured by: index.html, index.php, .htpasswd,
robots.txt found and read
    /
    ..
 broken_link_checker
 cgi-bin
 classes
 download
 images
 imgo
 includes
 uploads
 visual

comment:



USAGE: Check items for being allowed, uncheck items for being disallowed for user-agent *