encoding package's normalize_encoding() function is too slow #55531
Description
| BPO | 11322 |
|---|---|
| Nosy | @malemburg, @gpshead, @jcea, @abalkin, @vstinner, @ezio-melotti, @serhiy-storchaka |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2011-02-25.15:55:31.555>
labels = ['3.7', 'expert-unicode', 'performance']
title = "encoding package's normalize_encoding() function is too\tslow"
updated_at = <Date 2022-01-24.23:38:06.218>
user = 'https://github.com/malemburg'bugs.python.org fields:
activity = <Date 2022-01-24.23:38:06.218>
actor = 'gregory.p.smith'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Unicode']
creation = <Date 2011-02-25.15:55:31.555>
creator = 'lemburg'
dependencies = []
files = ['45909']
hgrepos = []
issue_num = 11322
keywords = ['patch']
message_count = 10.0
messages = ['129386', '129389', '129460', '129463', '165517', '220630', '220633', '283266', '283271', '283333']
nosy_count = 8.0
nosy_names = ['lemburg', 'gregory.p.smith', 'jcea', 'belopolsky', 'vstinner', 'ezio.melotti', 'sdaoden', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue11322'
versions = ['Python 3.7']
Activity
malemburg commentedon Feb 25, 2011
I don't know who changed the encoding's package normalize_encoding() function (wasn't me), but it's a really slow implementation.
The original version used the .translate() method which is a lot faster and can be adapted to work with the Unicode variant of the .translate() method just as well.
abalkin commentedon Feb 25, 2011
I don't think the normalize_encoding() function was the culprit for bpo-11303 because I measured timings with timeit which averages multiple runs while normalize_encoding() is called only the one time per encoding spelling due to caching.
vstinner commentedon Feb 25, 2011
We should first implement the same algorithm of the 3 normalization functions and add tests for them (at least for the function in normalization):
normalize_encoding() in encodings is more laxist than the two other functions: it normalizes " utf 8 " to 'utf_8'. But it doesn't convert to lowercase and keeps non-ASCII letters: "UTF-8é" is normalized "UTF_8é".
I don't know if the normalization functions have to be more or less strict, but I think that they should all give the same result.
malemburg commentedon Feb 25, 2011
STINNER Victor wrote:
Please see this message for an explanation of why we have those
three functions, why they are different and what their application
space is:
http://bugs.python.org/issue5902#msg129257
This ticket is just about the encoding package's codec search
function, not the other two, and I don't want to change
semantics, just its performance.
[-]encoding package's normalize_encoding() function is too slow[/-][+]encoding package's normalize_encoding() function is too slow[/+]serhiy-storchaka commentedon Jul 15, 2012
See changeset 54ef645d08e4 (Git commit ad5b9de).
BreamoreBoy commentedon Jun 15, 2014
What's the status of this issue, as we've lived with this really slow implementation for well over three years?
malemburg commentedon Jun 15, 2014
On 15.06.2014 15:02, Mark Lawrence wrote:
I guess it just needs someone to write a patch.
Note that encoding lookups are cached, so the slowness only
becomes an issue if you lookup lots of different encodings.
malemburg commentedon Dec 15, 2016
Thanks for the patch.
Victor has implemented the function in C, AFAIK, so an even better approach would be to expose that function at the Python level and use it in the encodings package.
vstinner commentedon Dec 15, 2016
It seems like encodings.normalize_encoding() currently has no unit test! Before modifying it, I would prefer to see a few unit tests:
Since we are talking about an optimmization, I would like to see a benchmark result before/after. I also would like to test Marc-Andre's idea of exposing the C function _Py_normalize_encoding().
_Py_normalize_encoding() works on a byte string encoded to Latin1. To implement encodings.normalize_encoding(), we might rewrite the function to work on Py_UCS4 character, or have a fast version on char*, and a more generic version for UCS2 and UCS4?
vstinner commentedon Dec 15, 2016
Oh, while reading Mercurial history, I found a note that I wrote:
"It's not exactly the same than encodings.normalize_encoding(): the C function also converts to lowercase."
IHMO it's fine to modify encodings.normalize_encoding() to also convert to lower-case.
12 remaining items
StanFromIreland commentedon Jul 22, 2025
Note: It is then called for all aliases.
serhiy-storchaka commentedon Jul 22, 2025
It is called once for every encoding name. How many different encoding names do you use in a program? Current
normalize_encoding('utf-8')takes 0.5 microseconds on my computer. You need to look up millions different encoding names to make it burdensome, and you will have memory issue (due to the growing cache) before this.malemburg commentedon Jul 27, 2025
Yes, it is slow (and it was much faster before until someone changed the function) and yes, if the encoding is found, the information is cached by the codec registry.
But that's not the point. We should not have multiple different variants of encoding name normalization in the stdlib.
It is better to stick with the C implementation
_Py_normalize_encoding()as the single source or truth.malemburg commentedon Jul 27, 2025
I've also had a look at the flow of things:
The search functions in the encodings package are called by the codec lookup function (in
Python/codecs.c) with an already normalized encoding name.The extra normalization happening in the encodings package is not really needed, AFAICT.
I think we can remove the call from the search function and perhaps only leave the
normalize_encoding()function as an alias for the new C function. We should then document that the search function in the encodings package needs to be called with a normalized encoding name.I doubt this would cause much breakage and it does make lookups a tad faster for applications that have to deal with lots of different encodings (e.g. mail processors or data import functions).
serhiy-storchaka commentedon Jul 28, 2025
I think there were good reasons to have different normalizations because they have different purpose.
Initially
codecs.lookup()only converted the encoding name to lower case (ASCII-only letters). It was pretty safe, because encoding names are case-insensitive. In 9e896b3 is started also replacing spaces with hyphens. I do not know reasons for this, perhaps because of "US ASCII"?Then the name is passed to search functions.
encodings.search_function()is one of the implementations of a search function. It needs to convert the encoding name to the module name, it does it by callingencodings.normalize_encoding(). Any filesystem specific characters (/,\,:, non-ASCII and control characters, etc) should be replaced. Safe bet is to leave only ASCII letters, digits, underscore and a dot (which is a separator between module name components). This normalization is only needed for that particular implementation, which reads encodings from Python modules. Other implementations, likeiconv, does not need it (see #88886)._Py_normalize_encoding()was used to normalize few builtin encoding names ("utf-8", "iso-8859-1", etc) to fasten the check. In general, it should combine two above transformations (convertion to lower case and replacing some separators), because it bypasses them, but this is not necessary. In worst case you will use a slow path for unusual spelling of builtin encoding name.Transformation in
codecs.lookup()should be fast, and it should not remove meaningful parts from the name. I am not even sure that any transformation is needed here._Py_normalize_encoding()should be super fast, but it only need to work for builtin encoding names and their common variations. So "ISO_8859-1" should be accepted as well as "ISO-8859-1", but accepting "ISO/8859:1" is an undesirable side effect.encodings.search_function()needs to convert encoding name to module name. It should normalize any common separators, but I think that it is better to return None for non-ASCII names or control characters.malemburg commentedon Jul 28, 2025
There are two parts to all this:
The codec lookup function used to do only partial normalization, to make sure that search functions don't get to see weird encoding names. The encoding package search function additionally had to make sure that the encoding names are converted to valid Python module names (and additionally uses an alias table helping with this).
Other search functions are free to apply different normalizations.
More recently, the encoding package search function was converted to an internal C function by Victor:
_Py_normalize_encoding(). This function is used in the codec lookup function for normalization, replacing the previous version. The change hasn't caused any issues as far as I am aware.The reason I am pushing for having just a single normalization function in the stdlib is to simplify the setup and have a central place to define what encoding normalization means. By exposing the
_Py_normalize_encoding()function as a Python callable C function, we can achieve this.And going one step further, we can even remove the normalization call in the encoding package's search function to the normalization function, since it the encoding name will already have been normalized in the same way by the calling codec lookup function. It doesn't make sense to call the same normalization function twice every time you want to find a codec in the encodings package.
Note that this is a new feature and not something we can or should backport.
serhiy-storchaka commentedon Jul 28, 2025
It caused. See #88886.
I am inclined to the opposite opinion -- remove normalization in the codec lookup function, leave this to the search function. So the third-party search function should see weird encoding names and handle them as they prefer. "foo-bar" and "foo+bar" can be different encodings, or only one of them is correct and we need to know the original name.
Also, non-ASCII characters and control codes should not be silently ignored. They should either cause immediate error or simply be passed to the search functions as is. If they not found matching encoding, we will get a standard error. There are also encodings with official non-ASCII name (e.g. https://en.wikipedia.org/wiki/KOI-7).
malemburg commentedon Jul 28, 2025
You do have a point there, but this is starting to turn into a partial redesign of the way codec lookups were handled in recent Python versions.
The alternative approach would be to:
The internal C function can continue to be used in short cuts to the resp. codec C APIs as is done now.
This should then also resolve the python-iconv package error, I guess.
malemburg commentedon Jul 29, 2025
Ok, so once #137167 has landed, we should then expose the internal
_Py_normalize_encoding()function in Python and use it in the encodings package'snormalize_encoding()function.StanFromIreland commentedon Jul 29, 2025
Ok I will update the PR once it is merged.
pythongh-55531: Implement `normalize_encoding` in C (python#136643)
pythongh-55531: Implement `normalize_encoding` in C (python#136643)