我有一个 scraper 项目,可以处理异步请求请求库和三重奏。 我想根据输入选择多少个并发任务,但我的代码又长又原始
我使用 trio 的 spawning 和 nursery 对象来处理并发任务(文档:https://trio.readthedocs.io/en/latest/reference-core.html)
这是我草率的代码:
import trio
import asks
Number_of_workers = input("how many workers do you want?: ") #How many tasks I want between 1 and 5
async def child1(s):
r = await s.get("https://example.com", params={"example":"example"})
print("do something with", r.text)
async def child2():
r = await s.get("https://example.com", params={"example":"example"})
print("do something with", r.text)
async def child3():
r = await s.get("https://example.com", params={"example":"example"})
print("do something with", r.text)
async def child4():
r = await s.get("https://example.com", params={"example":"example"})
print("do something with", r.text)
async def child5():
r = await s.get("https://example.com", params={"example":"example"})
print("do something with", r.text)
async def parent():
s = Session(connections=5)
async with trio.open_nursery() as nursery:
if int(Number_of_workers) == 1:
nursery.start_soon(child1, s)
elif int(Number_of_workers) == 2:
nursery.start_soon(child1, s)
nursery.start_soon(child2, s)
elif int(Number_of_workers) == 3:
nursery.start_soon(child1, s)
nursery.start_soon(child2, s)
nursery.start_soon(child3, s)
elif int(Number_of_workers) == 4:
nursery.start_soon(child1, s)
nursery.start_soon(child2, s)
nursery.start_soon(child3, s)
nursery.start_soon(child4, s)
elif int(Number_of_workers) == 5:
nursery.start_soon(child1, s)
nursery.start_soon(child2, s)
nursery.start_soon(child3, s)
nursery.start_soon(child4, s)
nursery.start_soon(child5, s)
trio.run(parent)
我想您能理解我的意思,这个示例代码在理论上是可行的,但对于可能可以减少到更少代码行的内容来说它太长了。
这种方案在处理 10 或 20 名 worker 时会变得特别长,并且总是限于预定义的数量。
在其内部,每个 child 都是相同的,相同的代码,它只是从带有 importlib 的外部模块 .py 文件中获取不同的数据(例如参数和 url)。
有没有办法将其缩减为更优化的代码?
请您参考如下方法:
你可以使用循环!
async def child(s):
r = await s.get("https://example.com", params={"example":"example"})
print("do something with", r.text)
async def parent():
s = Session(connections=5)
async with trio.open_nursery() as nursery:
for i in range(Number_of_workers):
nursery.start_soon(child, s)
编辑:这是一个独立的演示,您可以运行它来说服自己这确实在运行并发任务。它还演示了如何将不同的参数值传递给不同的任务,以便它们执行不同的操作——在本例中,打印不同的消息:
import trio
Number_of_workers = 10
async def child(i):
print("child {}: started".format(i))
await trio.sleep(5)
print("child {}: finished".format(i))
async def parent():
async with trio.open_nursery() as nursery:
for i in range(Number_of_workers):
nursery.start_soon(child, i)
trio.run(parent)
试试看!
