I learned a lot about how webcrawlers work in the course of doing this challenge. It was very much worth it to me. On one of the sites it said "of course if you are a hacker you will want to see which pages are disallowed" which makes sense to me.
I may be dumb, I don't know, but I really am struggling with this. I am new to hacking in general, yet I understand this really isn't that hard. I've been struggling for like an hour so can anyone please help me.