Skip to main content
 首页 » 编程设计

Python、请求、线程、python 请求关闭其套接字的速度有多快

2025年04月02日10zhenyulu

我正在尝试使用 Python 请求执行操作。这是我的代码:

import threading 
import resource 
import time 
import sys 
 
#maximum Open File Limit for thread limiter. 
maxOpenFileLimit = resource.getrlimit(resource.RLIMIT_NOFILE)[0] # For example, it shows 50. 
 
# Will use one session for every Thread. 
requestSessions = requests.Session() 
# Making requests Pool bigger to prevent [Errno -3] when socket stacked in CLOSE_WAIT status. 
adapter = requests.adapters.HTTPAdapter(pool_maxsize=(maxOpenFileLimit+100)) 
requestSessions.mount('http://', adapter) 
requestSessions.mount('https://', adapter) 
 
def threadAction(a1, a2): 
    global number 
    time.sleep(1) # My actions with Requests for each thread. 
    print number = number + 1 
 
number = 0 # Count of complete actions 
 
ThreadActions = [] # Action tasks. 
for i in range(50): # I have 50 websites I need to do in parallel threads. 
    a1 = i 
    for n in range(10): # Every website I need to do in 3 threads 
        a2 = n 
        ThreadActions.append(threading.Thread(target=threadAction, args=(a1,a2))) 
 
 
for item in ThreadActions: 
    # But I can't do more than 50 Threads at once, because of maxOpenFileLimit. 
    while True: 
        # Thread limiter, analogue of BoundedSemaphore. 
        if (int(threading.activeCount()) < threadLimiter): 
            item.start() 
            break 
        else: 
            continue 
 
for item in ThreadActions: 
    item.join() 

但问题是,在我启动 50 个线程后,线程限制器 开始等待某个线程完成其工作。这就是问题所在。在 scrit 进入 Limiter 后,lsof -i|grep python|wc -l 显示事件连接远少于 50 个。但在 Limiter 之前,它显示了所有 <= 50 个进程。为什么会这样?或者我应该使用 requests.close() 而不是 requests.session() 来防止它使用已经打开的套接字?

请您参考如下方法:

您的限制器是一个紧凑的循环,占用了您的大部分处理时间。改为使用线程池来限制工作人员的数量。

import multiprocessing.pool 
 
# Will use one session for every Thread. 
requestSessions = requests.Session() 
# Making requests Pool bigger to prevent [Errno -3] when socket stacked in CLOSE_WAIT status. 
adapter = requests.adapters.HTTPAdapter(pool_maxsize=(maxOpenFileLimit+100)) 
requestSessions.mount('http://', adapter) 
requestSessions.mount('https://', adapter) 
 
def threadAction(a1, a2): 
    global number 
    time.sleep(1) # My actions with Requests for each thread. 
    print number = number + 1 # DEBUG: This doesn't update number and wouldn't be 
                              # thread safe if it did 
 
number = 0 # Count of complete actions 
 
pool = multiprocessing.pool.ThreadPool(50, chunksize=1) 
 
ThreadActions = [] # Action tasks. 
for i in range(50): # I have 50 websites I need to do in parallel threads. 
    a1 = i 
    for n in range(10): # Every website I need to do in 3 threads 
        a2 = n 
        ThreadActions.append((a1,a2)) 
 
pool.map(ThreadActons) 
pool.close()