Repository navigation
Speed up pickling of lists in cPickle #49921
Description
| BPO | 5671 |
|---|---|
| Nosy | @rhettinger, @pitrou, @avassalotti |
| Superseder | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/avassalotti'
closed_at = <Date 2010-08-05.07:28:00.605>
created_at = <Date 2009-04-02.19:28:16.001>
labels = ['extension-modules', 'performance']
title = 'Speed up pickling of lists in cPickle'
updated_at = <Date 2010-08-05.07:28:00.603>
user = 'https://bugs.python.org/collinwinter'bugs.python.org fields:
activity = <Date 2010-08-05.07:28:00.603>
actor = 'alexandre.vassalotti'
assignee = 'alexandre.vassalotti'
closed = True
closed_date = <Date 2010-08-05.07:28:00.605>
closer = 'alexandre.vassalotti'
components = ['Extension Modules']
creation = <Date 2009-04-02.19:28:16.001>
creator = 'collinwinter'
dependencies = []
files = ['13585', '13595']
hgrepos = []
issue_num = 5671
keywords = ['patch', 'needs review']
message_count = 9.0
messages = ['85250', '85263', '85265', '85295', '85301', '85312', '97492', '101461', '112953']
nosy_count = 6.0
nosy_names = ['collinwinter', 'rhettinger', 'jafo', 'pitrou', 'alexandre.vassalotti', 'Garen']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '9410'
type = 'performance'
url = 'https://bugs.python.org/issue5671'
versions = ['Python 3.1', 'Python 2.7']
Activity
collinwinter commentedon Apr 2, 2009
The attached patch adds another version of cPickle.c's batch_list(),
batch_list_exact(), which is specialized for "type(x) is list". This
provides a nice performance boost when pickling objects that use
lists. This is similar to the approach taken in bpo-5670, though the
performance boost on our benchmark is smaller:
Pickle:
Min: 2.231 -> 2.200: 1.39% faster
Avg: 2.266 -> 2.227: 1.72% faster
Significant (t=10.994064, a=0.95)
Benchmark is at
http://code.google.com/p/unladen-swallow/source/browse/tests/performance/macro_pickle.py
(driver is ../perf.py; perf.py was run with "--rigorous -b pickle").
Workloads involving more lists will benefit more.
This patch passes all the tests added in bpo-5665. I would recommend
reviewing that patch first. I'll port to py3k once this is reviewed for
trunk.
rhettinger commentedon Apr 2, 2009
Out of curiousity, are you also benchmarking against marshal and json?
ISTM, that there is always one of them that will be the fastest and that
the others should mimic that approach.
collinwinter commentedon Apr 2, 2009
No, we haven't started looking at other serialization formats yet.
Marshal will probably be my next target, with json being a lower
priority. There were enough instances of low-hanging fruit in cPickle
that I didn't go looking at the other implementations for ideas.
avassalotti commentedon Apr 3, 2009
Here's a patch for py3k. I also added a special-case for 1-item lists.
pitrou commentedon Apr 3, 2009
A micro-benchmark of Collin's patch:
python -m timeit -s "import cPickle; l=range(150)" "cPickle.dumps(l,
protocol=-1)"
=> 15% faster on a favorable case
collinwinter commentedon Apr 3, 2009
I've added a microbenchmark to perf.py called pickle_list. Running that
on this change (perf.py -r -b pickle_list):
pickle_list:
Min: 1.126 -> 0.888: 26.86% faster
Avg: 1.154 -> 0.906: 27.43% faster
Significant (t=115.404547, a=0.95)
That's probably the upper bound on the performance improvement to be
realized from this patch.
smontanaro commentedon Jan 10, 2010
Still applies cleanly (with a little fuzz) to the trunk after applying
the bpo-5683 patch. Tests all still pass (including xpickle w/ 2.4,
2.5, 2.6 available).
jafo commentedon Mar 21, 2010
pickle_batch_list_exact_py3k.diff applies cleanly on current py3k trunk and passes tests. cpickle_list.patch applies cleanly against 2.x trunk and passes "make test".
I don't see any objections brought up about this set of patches, so we can get these applied?
avassalotti commentedon Aug 5, 2010
It is too late now for the 2.x version. And, the huge patch in bpo-9410 includes an updated version of this patch for 3.x.