# Shows exclusion info so crawlers know what to ignore # EXCLUSION FOLDERS User-agent: * Disallow: /bin/ Disallow: /images/ Disallow: /includes/ User-Agent: Mediapartners-Google Disallow: / User-Agent: Googlebot-Image Disallow: / User-Agent: Googlebot-Mobile Disallow: / # This is the State Library Spider User-Agent: Ultraseek Allow: