Where Can My Robot Go?
30 points Easy

Where do robots find what pages are on a website?

Hint:

What does disallow tell a robot?

Flag
Miscellaneous · intelagent
23396 solves
Rating 4.55
5
4
3
2
1
Must solve to rate

Discussion

avatar
mohiuddinlithu

4 weeks ago

Reply Share
Protected
-2
avatar
catt404

4 weeks ago

Reply Share

clue: robot.txt

0
avatar
wewe360

1 month ago

Reply Share

That is a good challenge

0
avatar
Itrajwa

1 month ago

Reply Share

Wow it’s nice

0
avatar
KL4T

2 months ago

Reply Share

Smart for Starters.

1
avatar
AMEERX

2 months ago

Reply Share

thats really smart , i like it , but still ... its not "easy" for many ppl .

0
avatar
otanazar

2 months ago

Reply Share

i answered "robots.txt" but it is not correct, why?

2
avatar
Itrajwa

1 month ago

Reply Share

write it in the link

-1
avatar
Recon82

2 months ago

Reply Share

robots.txt

-1
avatar
root66

2 months ago

Reply Share
Protected
0
avatar
v544

4 years ago

Reply Share

I learned a lot about how webcrawlers work in the course of doing this challenge. It was very much worth it to me. On one of the sites it said "of course if you are a hacker you will want to see which pages are disallowed" which makes sense to me.

0