我正在尝试使用 python 字典中的 cStringIO 读取一个非常大的字符串流:
def stream_read(self, path):
try:
# create a string stream from the contents at 'path'
# note: the string at self._my_dict[path] is 7MB in size
stream = StringIO.StringIO(self._my_dict[path])
while True:
# buffer size is 128kB, or 128 * 1024
buf = stream.read(self.buffer_size)
if buf != '':
yield buf
else:
raise StopIteration
except KeyError:
raise IOError("Could not get content")
在我的测试套件中,我通过首先测试 stream_write 来测试此函数,断言数据存在于该路径,然后调用 stream_read:
def test_stream(self):
filename = self.gen_random_string()
# test 7MB
content = self.gen_random_string(7 * 1024 * 1024)
# test stream write
io = StringIO.StringIO(content)
self._storage.stream_write(filename, io)
io.close()
self.assertTrue(self._storage.exists(filename))
# test read / write
data = ''
for buf in self._storage.stream_read(filename):
data += buf
self.assertEqual(content, data)
然而在我的测试套件中,我发现了一个 AssertionError:
======================================================================
FAIL: test_stream (test_swift_storage.TestSwiftStorage)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/bacongobbler/.../test/test_local_storage.py", line 44, in test_stream
self.assertEqual(content, data)
AssertionError: '[squelched]' != '<cStringIO.StringI object at 0x3148e70>'
----------------------------------------------------------------------
Ran 28 tests in 20.495s
FAILED (failures=1)
它看起来与 an issue I posted last week 有关,但我仍然不太确定我理解为什么在这种情况下 stream 被设置为 Generator 作为字符串。
如果有人想仔细查看源代码,都在https://github.com/bacongobbler/docker-registry/blob/106-swift-storage/test/utils/mock_swift_storage.py。
请您参考如下方法:
在调用 self._storage.stream_write(filename, io) 时,您只存储 StringIO 对象:
def put_content(self, path, content, chunk=None):
path = self._init_path(path)
try:
self._swift_container[path] = content
except Exception:
raise IOError("Could not put content")
其中 content 是您传入的 io 对象。
稍后,您再次将该文件对象传递给 StringIO:
stream = StringIO.StringIO(self.get_content(path))
这会在 self.get_content(path) 上调用 str(),存储 cStringIO.StringI() 实例的字符串表示形式:
>>> from cStringIO import StringIO
>>> str(StringIO('test data'))
'<cStringIO.StringI object at 0x1074ea470>'
您的读取 代码运行良好,您的编写模拟需要实际从 StringIO 对象中取出数据。 p>
.read() 调用将在此处执行:
def put_content(self, path, content, chunk=None):
path = self._init_path(path)
try:
self._swift_container[path] = content.read()
except Exception:
raise IOError("Could not put content")
