在我基于 dask 的应用程序(使用 分布式 调度程序)中,我看到以以下错误文本开头的故障:
tornado.application - ERROR - Exception in Future <Future cancelled> after timeout
Traceback (most recent call last):
File "/miniconda/envs/flyem/lib/python3.6/site-packages/tornado/gen.py", line 970, in error_callback
future.result()
concurrent.futures._base.CancelledError
它们之后是第二个回溯(我认为)指示发生超时时我的任务正在运行哪一行。 (我不清楚 distributed 是如何做到这一点的——也许是通过信号?)
这是第二个回溯的粗略部分:
... my code...
File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/dask/base.py", line 156, in compute
(result,) = compute(self, traverse=False, **kwargs)
File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/dask/base.py", line 397, in compute
results = schedule(dsk, keys, **kwargs)
File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/distributed/client.py", line 2308, in get
direct=direct)
File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/distributed/client.py", line 1647, in gather
asynchronous=asynchronous)
File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/distributed/client.py", line 665, in sync
return sync(self.loop, func, *args, **kwargs)
File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/distributed/utils.py", line 277, in sync
six.reraise(*error[0])
File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/distributed/utils.py", line 262, in f
result[0] = yield future
File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/tornado/gen.py", line 1141, in run
yielded = self.gen.throw(*exc_info)
File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/distributed/client.py", line 1492, in _gather
traceback)
File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/six.py", line 692, in reraise
raise value.with_traceback(tb)
File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/dask/bag/core.py", line 1562, in reify
seq = list(seq)
File "/groups/flyem/proj/cluster/miniforge/envs/flyem/lib/python3.6/site-packages/dask/bag/core.py", line 1722, in map_chunk
yield f(*a)
... my code ....
after timeout是否表示任务花费的时间太长,或者是否有其他一些超时触发了取消,例如保姆或心跳超时? (据我所知,dask 中的任务长度没有明确的超时,但也许我很困惑。)我看到任务已取消。但我想知道为什么。有什么简单的方法可以找出哪一行代码(在
dask或distributed中)正在取消我的任务,为什么?我预计这些任务需要很长时间——它们正在将大缓冲区上传到云存储。如何在 dask 中增加特定任务的超时时间?
请您参考如下方法:
默认情况下,Dask 不会对任务施加超时。
您看到的取消的 future 不是 Dask 的 future ,而是 Tornado 的 future (Tornado 是 Dask 用于网络通信的库)。所以不幸的是,这一切都在说一些事情失败了。
后续的回溯有望包含有关失败代码的确切信息。理想情况下,这指向函数中发生故障的一行。也许这有帮助?
一般来说,我们建议在调试通过 Dask 运行的代码时执行以下步骤:http://docs.dask.org/en/latest/debugging.html
