File tree Expand file tree Collapse file tree 1 file changed +8
-11
lines changed
Expand file tree Collapse file tree 1 file changed +8
-11
lines changed Original file line number Diff line number Diff line change @@ -91,16 +91,13 @@ class::
9191
9292 >>> import urllib.robotparser
9393 >>> rp = urllib.robotparser.RobotFileParser()
94- >>> rp.set_url("http ://www.musi-cal.com /robots.txt")
94+ >>> rp.set_url("https ://www.python.org /robots.txt")
9595 >>> rp.read()
96- >>> rrate = rp.request_rate("*")
97- >>> rrate.requests
98- 3
99- >>> rrate.seconds
100- 20
101- >>> rp.crawl_delay("*")
102- 6
103- >>> rp.can_fetch("*", "http://www.musi-cal.com/cgi-bin/search?city=San+Francisco")
104- False
105- >>> rp.can_fetch("*", "http://www.musi-cal.com/")
96+ >>> print(rp.request_rate("*"))
97+ None
98+ >>> print(rp.crawl_delay("*"))
99+ None
100+ >>> rp.can_fetch("*", "https://www.python.org/psf/")
106101 True
102+ >>> rp.can_fetch("*", "https://www.python.org/webstats/")
103+ False
You can’t perform that action at this time.
0 commit comments