Where Can My Robot Go?
30 points Easy

Where do robots find what pages are on a website?

Hint:

What does disallow tell a robot?

Flag
Miscellaneous ยท intelagent
27204 solves
Rating 4.56
5
4
3
2
1
Must solve to rate

Discussion

avatar
pA4_r0t

4 days ago

Reply Share

nice :)

0
avatar
mikeAl

1 week ago

Reply Share

Robots like reading text files :)

0
avatar
waladikoalamYA

2 weeks ago

Reply Share

in 2025 the rise of AI waiting for SKYNET

0
avatar
ninjabubble3

3 weeks ago

Reply Share

dayummm

0
avatar
EN4EX

3 weeks ago

Reply Share

Damn this one was smooth

0
avatar
izanagi13

1 month ago

Reply Share

URL Enumeration - This is all i can say for now...

0
avatar
0xveczif

2 months ago

Reply Share

robots.txt is a file which tells search engines to not to crawl the page and show in search engine search results. Website puts sensitive pages in this list to make secure. We can visit those links to see the pages.

0
avatar
minstersdaf

2 months ago

Reply Share

easy bro! just use your damn brain!!

0
avatar
22786

2 months ago

Reply Share
Protected
0
avatar
buggerlugs

3 months ago

Reply Share
Protected
0
avatar
TheImmortal2005

4 years ago

Reply Share
Protected
0
avatar
denzven

4 years ago

Reply Share
Protected
0