python学习模块系列(urllib, urllib2)

python admin 10℃ 0评论

urllib, urllib2应用比较广泛的地方在爬虫

#coding:utf-8
import urllib2
import random

ua_list = [
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
    "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"
]
user_agent = random.choice(ua_list)
url = 'http://www.molis.cc'

req = urllib2.Request(url)
req.add_header('User-Agent',user_agent)
response = urllib2.urlopen(req)
#print response.read()

print response.getcode()
print response.geturl()

需要更多的请求参数可参考如下代码,请求头包括proxy, ua, cookies

#coding:utf-8
import urllib, urllib2

url = 'http://www.molis.cc'
formdata = {'start':'20','limit':'100'}
data = urllib.urlencode(formdata)
headers = {'User-Agent':'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36'}
request = urllib2.Request(url,data = data,headers=headers)

req = urllib2.Request(url)
response = urllib2.urlopen(req)
#print response.read()

print response.getcode()
print response.geturl()

 

转载请注明:朋克网 » python学习模块系列(urllib, urllib2)

喜欢 (0)
发表我的评论
取消评论
表情

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址