我有一个非常大的 CSV 文件,其中第一列是 unix 时间戳 已经从低到高排序。
1461568570,2977.320000000000,0.032000000000
1461568570,2977.320000000000,0.076000000000
1461568570,2977.320000000000,0.076000000000
1461568569,2977.050000000000,0.050000000000
1461568569,2977.050000000000,0.050000000000
1461568569,2977.300000000000,0.021900000000
1461568569,2977.310000000000,0.021900000000
1461568569,2977.320000000000,0.050000000000
1461568423,2978.510000000000,0.500000000000
1461568421,2977.920000000000,0.023300000000
1461568421,2977.920000000000,0.010900000000
1461568421,2977.910000000000,0.165800000000
我想将数据导入到 pandas 数据框中,但我想将其限制为数据的一个子集。
现在,pandas read_csv 有 skiprows 和 skipfooter 选项,我可以告诉他只在特定点行之后检索数据CSV 文件。但是我想指定开始读取的行号,以便只捕获某个 unix timestamp 之后的行(所以基本上是以 unix timestamp< 开头的第一行的行号 等于或高于 1461568423 例如)。
怎样才能有效地做到这一点?
请您参考如下方法:
IIUC 然后你可以做类似下面的事情:
In [47]:
line=0
chunksz=3
for chunk in pd.read_csv(io.StringIO(t), header=None, names = ['timestamp','val1','val2'], chunksize=chunksz):
if len(chunk[chunk['timestamp'] == 1461568423]) > 0:
line += chunk[chunk['timestamp'] == 1461568423].index[0]
break
else:
line += chunksz
pd.read_csv(io.StringIO(t), header=None, names = ['timestamp','val1','val2'], skiprows=line)
Out[47]:
timestamp val1 val2
0 1461568423 2978.51 0.5000
1 1461568421 2977.92 0.0233
2 1461568421 2977.92 0.0109
3 1461568421 2977.91 0.1658
这里我们将行计数器设置为 0 和标称 block z,我们遍历 block 直到找到匹配项,然后将其用作 skiprows 的参数值。这应该很快,因为我们可以设置一个大的 block 大小并继续跳过找不到该行的 block
