Skip to main content
 首页 » 编程设计

python之试图伪造和轮换用户代理

2024年11月24日8qq78292959

我正在尝试伪造用户代理并在 Python 中轮换它们。
我在网上找到了一个关于如何使用 scrapy-useragents 使用 Scrapy 执行此操作的教程。包裹。
我抓取网页,https://www.whatsmyua.info/ ,为了检查我的用户代理,看看它是否与我的不同,以及它是否轮换。它是否与我的实际用户代理不同,但它不会轮换,它每次都返回相同的用户代理,我无法弄清楚出了什么问题。

设置.py

BOT_NAME = 'scrapy_javascript' 
 
SPIDER_MODULES = ['scrapy_javascript.spiders'] 
NEWSPIDER_MODULE = 'scrapy_javascript.spiders' 
 
 
# Crawl responsibly by identifying yourself (and your website) on the user-agent 
#USER_AGENT = 'scrapy_javascript (+http://www.yourdomain.com)' 
 
# Obey robots.txt rules 
ROBOTSTXT_OBEY = True 
DOWNLOADER_MIDDLEWARES = { 
        'scrapy_splash.SplashCookiesMiddleware': 723, 
        'scrapy_splash.SplashMiddleware': 725, 
        'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810, 
} 
 
# ----------------------------------------------------------------------------- 
# USER AGENT 
# ----------------------------------------------------------------------------- 
 
DOWNLOADER_MIDDLEWARES = { 
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None, 
    'scrapy_useragents.downloadermiddlewares.useragents.UserAgentsMiddleware': 500, 
} 
 
 
USER_AGENTS = [ 
    ('Mozilla/5.0 (X11; Linux x86_64) ' 
     'AppleWebKit/537.36 (KHTML, like Gecko) ' 
     'Chrome/57.0.2987.110 ' 
     'Safari/537.36'),  # chrome 
    ('Mozilla/5.0 (X11; Linux x86_64) ' 
     'AppleWebKit/537.36 (KHTML, like Gecko) ' 
     'Chrome/61.0.3163.79 ' 
     'Safari/537.36'),  # chrome 
    ('Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:55.0) ' 
     'Gecko/20100101 ' 
     'Firefox/55.0'),  # firefox 
    ('Mozilla/5.0 (X11; Linux x86_64) ' 
     'AppleWebKit/537.36 (KHTML, like Gecko) ' 
     'Chrome/61.0.3163.91 ' 
     'Safari/537.36'),  # chrome 
    ('Mozilla/5.0 (X11; Linux x86_64) ' 
     'AppleWebKit/537.36 (KHTML, like Gecko) ' 
     'Chrome/62.0.3202.89 ' 
     'Safari/537.36'),  # chrome 
    ('Mozilla/5.0 (X11; Linux x86_64) ' 
     'AppleWebKit/537.36 (KHTML, like Gecko) ' 
     'Chrome/63.0.3239.108 ' 
     'Safari/537.36'),  # chrome 
] 
 
SPLASH_URL = 'http://199.89.192.74:8050' 
 
 
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter' 
HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage' 

请您参考如下方法:

在这里您可以找到一个 API,它以 JSON 格式返回最常见的用户代理:
http://51.158.74.109/useragents/?format=json

我已经使用了这个工具,它会让你的用户代理列表始终更新为最新和最常用的用户代理:https://pypi.org/project/shadow-useragent/

     from shadow_useragent import ShadowUserAgent 
     shadow_useragent = ShadowUserAgent() 
 
     print(shadow_useragent.firefox) 
     # Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0 
     print(shadow_useragent.chrome) 
     # Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36 
     print(shadow_useragent.safari) 
     # Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15 
     print(shadow_useragent.edge) 
     # Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134 
     print(shadow_useragent.ie) 
     # Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko 
     print(shadow_useragent.android) 
     # Mozilla/5.0 (Linux; U; Android 4.3; en-us; SM-N900T Build/JSS15J) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30 
     print(shadow_useragent.ipad) 
     # Mozilla/5.0 (iPad; CPU OS 12_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Mobile/15E148 Safari/604.1 
     print(shadow_useragent.random) 
     # Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0 
 
     print(shadow_useragent.random_nomobile) 
     # Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.90 Safari/537.36 
 
     # and the best one, random via real world browser usage statistic 
     print(ua.random) 
     # Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.90 Safari/537.36 
 
     # if you want to excluse mobiles (some websites will display different pages) 
     print(shadow_useragent.random_nomobile) 
     # Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.90 Safari/537.36