Skip to main content
 首页 » 编程设计

python之如何压缩根据输入选择多少并发任务的代码

2024年11月24日9bjzhanghao

我有一个 scraper 项目,可以处理异步请求请求库和三重奏。 我想根据输入选择多少个并发任务,但我的代码又长又原始

我使用 trio 的 spawning 和 nursery 对象来处理并发任务(文档:https://trio.readthedocs.io/en/latest/reference-core.html)

这是我草率的代码:

import trio 
import asks 
Number_of_workers = input("how many workers do you want?: ") #How many tasks I want between 1 and 5 
 
async def child1(s): 
    r = await s.get("https://example.com", params={"example":"example"}) 
    print("do something with", r.text) 
 
async def child2(): 
    r = await s.get("https://example.com", params={"example":"example"}) 
    print("do something with", r.text) 
 
async def child3(): 
    r = await s.get("https://example.com", params={"example":"example"}) 
    print("do something with", r.text) 
 
async def child4(): 
    r = await s.get("https://example.com", params={"example":"example"}) 
    print("do something with", r.text) 
 
async def child5(): 
    r = await s.get("https://example.com", params={"example":"example"}) 
    print("do something with", r.text) 
 
async def parent():  
    s = Session(connections=5) 
    async with trio.open_nursery() as nursery: 
        if int(Number_of_workers) == 1: 
            nursery.start_soon(child1, s) 
 
        elif int(Number_of_workers) == 2: 
            nursery.start_soon(child1, s) 
            nursery.start_soon(child2, s) 
 
        elif int(Number_of_workers) == 3: 
            nursery.start_soon(child1, s) 
            nursery.start_soon(child2, s) 
            nursery.start_soon(child3, s) 
 
        elif int(Number_of_workers) == 4: 
            nursery.start_soon(child1, s) 
            nursery.start_soon(child2, s) 
            nursery.start_soon(child3, s) 
            nursery.start_soon(child4, s) 
 
        elif int(Number_of_workers) == 5: 
            nursery.start_soon(child1, s) 
            nursery.start_soon(child2, s) 
            nursery.start_soon(child3, s) 
            nursery.start_soon(child4, s) 
            nursery.start_soon(child5, s) 
trio.run(parent) 
 

我想您能理解我的意思,这个示例代码在理论上是可行的,但对于可能可以减少到更少代码行的内容来说它太长了。

这种方案在处理 10 或 20 名 worker 时会变得特别长,并且总是限于预定义的数量。

在其内部,每个 child 都是相同的,相同的代码,它只是从带有 importlib 的外部模块 .py 文件中获取不同的数据(例如参数和 url)。

有没有办法将其缩减为更优化的代码?

请您参考如下方法:

你可以使用循环!

async def child(s): 
    r = await s.get("https://example.com", params={"example":"example"}) 
    print("do something with", r.text) 
 
async def parent():  
    s = Session(connections=5) 
    async with trio.open_nursery() as nursery: 
        for i in range(Number_of_workers): 
            nursery.start_soon(child, s) 

编辑:这是一个独立的演示,您可以运行它来说服自己这确实在运行并发任务。它还演示了如何将不同的参数值传递给不同的任务,以便它们执行不同的操作——在本例中,打印不同的消息:

import trio 
 
Number_of_workers = 10 
 
async def child(i): 
    print("child {}: started".format(i)) 
    await trio.sleep(5) 
    print("child {}: finished".format(i)) 
 
async def parent(): 
    async with trio.open_nursery() as nursery: 
        for i in range(Number_of_workers): 
            nursery.start_soon(child, i) 
 
trio.run(parent) 

试试看!