我想“ fork ”大量数据流,以便仅向前看几个元素。
我希望写这样的东西:
from itertools import tee
stream = # a generator of a very large data stream
while True:
try:
element= stream.next()
process_element( element )
if some_condition( element ):
stream, fork= tee(stream)
process_fork( fork )
except StopIteration:
break
阅读 the documentation for tee ,不过,给我的印象是 fork 的 deque 会继续增长,即使 fork 超出范围。
是这样吗?如果是这样,有没有办法告诉 tee “丢弃” fork ?或者还有其他更明显的方法吗?
请您参考如下方法:
这是一个简单的测试脚本:
from itertools import tee
def natural_numbers():
i=0
while True:
yield i
i+=1
stream = natural_numbers() #Don't use xrange, cpython optimizes it away
stream, fork= tee(stream)
del fork
for e in stream:
pass
看起来,至少在 CPython 中,进程的内存不会持续增长。 There seems to be a mechanism that detects this situation .
但是,如果您用 tee 代替 python 代码 the documentation states 是等价的...
def tee(iterable, n=2):
it = iter(iterable)
deques = [collections.deque() for i in range(n)]
def gen(mydeque):
while True:
if not mydeque: # when the local deque is empty
newval = next(it) # fetch a new value and
for d in deques: # load it to all the deques
d.append(newval)
yield mydeque.popleft()
return tuple(gen(d) for d in deques)
...正如预期的那样,内存确实在不断增长。
因此,我的猜测是这将是依赖于实现的行为
