Scrapy 中设置随机 User-Agent

安装scrapy-fake-useragent包

scrapy-fake-useragent包官网:https://pypi.org/project/scrapy-fake-useragent/
安装
pip install scrapy-fake-useragent

使用

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None, # 关闭默认方法
    'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400, # 开启
}

这里选用 http://httpbin.org/get 作为测试网站
测试结果:
scrapy-fake-useragent

本文参考自高级农名工:https://www.makcyun.top/web_scraping_withpython14.html

留下评论