Skip to main content
 首页 » 编程设计

python之为什么转储 `pickle` 比 `json` 快得多

2025年02月15日7zhujiabin

这是针对 Python 3.6 的。

编辑并删除了很多无关紧要的内容。

我曾认为 json 比 pickle 更快,Stack Overflow 上的其他答案和评论让很多其他人似乎也相信这一点。

我的测试符合犹太洁食标准吗?差异比我预期的要大得多。我在非常大的物体上测试得到了相同的结果。

import json 
import pickle 
import timeit 
 
file_name = 'foo' 
num_tests = 100000 
 
obj = {1: 1} 
 
command = 'pickle.dumps(obj)' 
setup = 'from __main__ import pickle, obj' 
result = timeit.timeit(command, setup=setup, number=num_tests) 
print("pickle: %f seconds" % result) 
 
command = 'json.dumps(obj)' 
setup = 'from __main__ import json, obj' 
result = timeit.timeit(command, setup=setup, number=num_tests) 
print("json:   %f seconds" % result) 

和输出:

pickle: 0.054130 seconds 
json:   0.467168 seconds 

请您参考如下方法:

我已经根据您的代码片段尝试了几种方法,发现使用 cPickle 并将转储方法的协议(protocol)参数设置为:cPickle.dumps(obj, protocol=cPickle.HIGHEST_PROTOCOL) 是最快的转储方法。

import msgpack 
import json 
import pickle 
import timeit 
import cPickle 
import numpy as np 
 
num_tests = 10 
 
obj = np.random.normal(0.5, 1, [240, 320, 3]) 
 
command = 'pickle.dumps(obj)' 
setup = 'from __main__ import pickle, obj' 
result = timeit.timeit(command, setup=setup, number=num_tests) 
print("pickle:  %f seconds" % result) 
 
command = 'cPickle.dumps(obj)' 
setup = 'from __main__ import cPickle, obj' 
result = timeit.timeit(command, setup=setup, number=num_tests) 
print("cPickle:   %f seconds" % result) 
 
 
command = 'cPickle.dumps(obj, protocol=cPickle.HIGHEST_PROTOCOL)' 
setup = 'from __main__ import cPickle, obj' 
result = timeit.timeit(command, setup=setup, number=num_tests) 
print("cPickle highest:   %f seconds" % result) 
 
command = 'json.dumps(obj.tolist())' 
setup = 'from __main__ import json, obj' 
result = timeit.timeit(command, setup=setup, number=num_tests) 
print("json:   %f seconds" % result) 
 
 
command = 'msgpack.packb(obj.tolist())' 
setup = 'from __main__ import msgpack, obj' 
result = timeit.timeit(command, setup=setup, number=num_tests) 
print("msgpack:   %f seconds" % result) 

输出:

pickle         :   0.847938 seconds 
cPickle        :   0.810384 seconds 
cPickle highest:   0.004283 seconds 
json           :   1.769215 seconds 
msgpack        :   0.270886 seconds