给定一个字符串列表列表,如:
test_array = [ ['a1','a2'], ['b1'], ['c1','c2','c3','c4'] ]
我想使用 h5py 存储它:
f['test_dataset'][0] = ['a1','a2']
f['test_dataset'][0][0] = 'a1'
etc.
按照线程中的建议 H5py store list of list of strings ,我尝试了以下方法:
import h5py
test_array = [ ['a1','a2'], ['b1'], ['c1','c2','c3','c4'] ]
with h5py.File('test.h5','w') as f:
string_dt = h5py.special_dtype(vlen=str)
f.create_dataset('test_dataset',data=test_array,dtype=string_dt)
然而,这会导致每个嵌套列表都存储为字符串,即:
f['test_dataset'][0] = "['a1', 'a2']"
f['test_dataset'][0][0] = '['
如果使用 h5py 或任何其他基于 hdf5 的库无法做到这一点,我很乐意听到有关我可用于存储数据的其他可能格式/库的任何建议。
我的数据由多维 numpy 整数数组和嵌套字符串列表组成,如上例所示,大约有 >1 亿行和约 8 列。
谢谢!
请您参考如下方法:
在Saving with h5py arrays of different sizes
我建议将可变长度数组列表保存为多个数据集。
In [19]: f = h5py.File('test.h5','w')
In [20]: g = f.create_group('test_array')
In [21]: test_array = [ ['a1','a2'], ['b1'], ['c1','c2','c3','c4'] ]
In [22]: string_dt = h5py.special_dtype(vlen=str)
In [23]: for i,v in enumerate(test_array):
...: g.create_dataset(str(i), data=np.array(v,'S4'), dtype=string_dt)
...:
In [24]: for k in g.keys():
...: print(k,g[k][:])
...:
0 ['a1' 'a2']
1 ['b1']
2 ['c1' 'c2' 'c3' 'c4']
对于许多小的子列表,这可能会很困惑,但我不确定它是否有效。
使用列表连接“展平”可能有效
In [27]: list1 =[', '.join(x) for x in test_array]
In [28]: list1
Out[28]: ['a1, a2', 'b1', 'c1, c2, c3, c4']
In [30]: '\n'.join(list1)
Out[30]: 'a1, a2\nb1\nc1, c2, c3, c4'
可以通过几次拆分重新创建嵌套列表。
另一个想法 - 腌制一个字符串并保存。
来自h5py介绍
An HDF5 file is a container for two kinds of objects: datasets, which
are array-like collections of data, and groups, which are folder-like
containers that hold datasets and other groups. The most fundamental
thing to remember when using h5py is:
Groups work like dictionaries, and datasets work like NumPy arrays
pickle 不起作用
In [32]: import pickle
In [33]: pickle.dumps(test_array)
Out[33]: b'\x80\x03]q\x00(]q\x01(X\x02\x00\x00\x00a1q\x02X\x02\x00\x00\x00a2q\x03e]q\x04X\x02\x00\x00\x00b1q\x05a]q\x06(X\x02\x00\x00\x00c1q\x07X\x02\x00\x00\x00c2q\x08X\x02\x00\x00\x00c3q\tX\x02\x00\x00\x00c4q\nee.'
In [34]: f.create_dataset('pickled', data=pickle.dumps(test_array), dtype=string
...: _dt)
....
ValueError: VLEN strings do not support embedded NULLs
json
In [35]: import json
In [36]: json.dumps(test_array)
Out[36]: '[["a1", "a2"], ["b1"], ["c1", "c2", "c3", "c4"]]'
In [37]: f.create_dataset('pickled', data=json.dumps(test_array), dtype=string_d
...: t)
Out[37]: <HDF5 dataset "pickled": shape (), type "|O">
In [43]: json.loads(f['pickled'].value)
Out[43]: [['a1', 'a2'], ['b1'], ['c1', 'c2', 'c3', 'c4']]
