Where Can My Robot Go?
30 points Easy

Where do robots find what pages are on a website?

Hint:

What does disallow tell a robot?

Flag
Rating 4.56
5
4
3
2
1

Discussion

in 2025 the rise of AI waiting for SKYNET

0

dayummm

0

Damn this one was smooth

0

URL Enumeration - This is all i can say for now...

0

robots.txt is a file which tells search engines to not to crawl the page and show in search engine search results. Website puts sensitive pages in this list to make secure. We can visit those links to see the pages.

0

easy bro! just use your damn brain!!

0
Protected
0
Protected
0

I enjoyed this one a lot!

0

Guys its simple, change the url to ctflear.com/robots.txt 70r3hnanldfspufdsoifnlds.html, change the url to ctflearn.com/(what you copied)

0

whatever

0