Skip to main content
 首页 » 编程设计

python之dask:如何避免任务超时

2024年12月31日7mfryf

在我基于 dask 的应用程序(使用 分布式 调度程序)中,我看到以以下错误文本开头的故障:

tornado.application - ERROR - Exception in Future <Future cancelled> after timeout 
Traceback (most recent call last): 
  File "/miniconda/envs/flyem/lib/python3.6/site-packages/tornado/gen.py", line 970, in error_callback 
    future.result() 
concurrent.futures._base.CancelledError 

它们之后是第二个回溯(我认为)指示发生超时时我的任务正在运行哪一行。 (我不清楚 distributed 是如何做到这一点的——也许是通过信号?)

这是第二个回溯的粗略部分:

  ... my code... 
 
  File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/dask/base.py", line 156, in compute 
    (result,) = compute(self, traverse=False, **kwargs) 
  File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/dask/base.py", line 397, in compute 
    results = schedule(dsk, keys, **kwargs) 
  File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/distributed/client.py", line 2308, in get 
    direct=direct) 
  File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/distributed/client.py", line 1647, in gather 
    asynchronous=asynchronous) 
  File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/distributed/client.py", line 665, in sync 
    return sync(self.loop, func, *args, **kwargs) 
  File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/distributed/utils.py", line 277, in sync 
    six.reraise(*error[0]) 
  File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/six.py", line 693, in reraise 
    raise value 
  File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/distributed/utils.py", line 262, in f 
    result[0] = yield future 
  File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run 
    value = future.result() 
  File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/tornado/gen.py", line 1141, in run 
    yielded = self.gen.throw(*exc_info) 
  File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/distributed/client.py", line 1492, in _gather 
    traceback) 
  File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/six.py", line 692, in reraise 
    raise value.with_traceback(tb) 
  File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/dask/bag/core.py", line 1562, in reify 
    seq = list(seq) 
  File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/dask/bag/core.py", line 1722, in map_chunk 
    yield f(*a) 
 
  ... my code .... 
  1. after timeout 是否表示任务花费的时间太长,或者是否有其他一些超时触发了取消,例如保姆或心跳超时? (据我所知,dask 中的任务长度没有明确的超时,但也许我很困惑。)

  2. 我看到任务已取消。但我想知道为什么。有什么简单的方法可以找出哪一行代码(在 dask 或 distributed 中)正在取消我的任务,为什么?

  3. 我预计这些任务需要很长时间——它们正在将大缓冲区上传到云存储。如何在 dask 中增加特定任务的超时时间?

请您参考如下方法:

默认情况下,Dask 不会对任务施加超时。

您看到的取消的 future 不是 Dask 的 future ,而是 Tornado 的 future (Tornado 是 Dask 用于网络通信的库)。所以不幸的是,这一切都在说一些事情失败了。

后续的回溯有望包含有关失败代码的确切信息。理想情况下,这指向函数中发生故障的一行。也许这有帮助?

一般来说,我们建议在调试通过 Dask 运行的代码时执行以下步骤:http://docs.dask.org/en/latest/debugging.html