Where Can My Robot Go?
30 points Easy

Where do robots find what pages are on a website?

Hint:

What does disallow tell a robot?

Flag
Miscellaneous ยท intelagent
26786 solves
Rating 4.56
5
4
3
2
1
Must solve to rate

Discussion

avatar
izanagi13

1 week ago

Reply Share

URL Enumeration - This is all i can say for now...

0
avatar
0xveczif

1 month ago

Reply Share

robots.txt is a file which tells search engines to not to crawl the page and show in search engine search results. Website puts sensitive pages in this list to make secure. We can visit those links to see the pages.

0
avatar
minstersdaf

1 month ago

Reply Share

easy bro! just use your damn brain!!

0
avatar
22786

1 month ago

Reply Share
Protected
0
avatar
buggerlugs

1 month ago

Reply Share
Protected
0
avatar
Patapam2

2 months ago

Reply Share

I enjoyed this one a lot!

0
avatar
Pctopus929w

2 months ago

Reply Share

Guys its simple, change the url to ctflear.com/robots.txt 70r3hnanldfspufdsoifnlds.html, change the url to ctflearn.com/(what you copied)

0
avatar
DoctorHorse

2 months ago

Reply Share

whatever

0
avatar
themtyhulk

3 months ago

Reply Share

internal server error !!!!!

0
avatar
Jaymet1

3 months ago

Reply Share

not much to go on for the robot go ?

2