Register
Login
Resources
Docs Blog Datasets Glossary Case Studies Tutorials & Webinars
Product
Data Engine LLMs Platform Enterprise
Pricing Explore
Connect to our Discord channel

robots.txt 492 B

You have to be logged in to leave a comment. Sign In
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
  1. User-agent: Slurp
  2. Crawl-delay: 100
  3. Disallow:
  4. User-agent: gsa-crawler-www
  5. Crawl-delay: 100
  6. User-agent: Googlebot
  7. Crawl-delay: 100
  8. User-agent: Mediapartners-Google
  9. Disallow:
  10. User-agent: Yahoo-NewsCrawler
  11. Disallow:
  12. User-Agent: msnbot
  13. Crawl-delay: 100
  14. Disallow:
  15. User-Agent: *
  16. Disallow: /config/
  17. Disallow: /handlers/
  18. Disallow: /includes/
  19. Disallow: /interceptors/
  20. Disallow: /layouts/
  21. Disallow: /logs/
  22. Disallow: /models/
  23. Disallow: /modules/
  24. Disallow: /modules_app/
  25. Disallow: /views/
  26. Allow: /
Tip!

Press p or to see the previous file or, n or to see the next file

Comments

Loading...