Repository navigation
Raise non-silent warning for invalid escape sequences #77093
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2018-02-22.18:34:57.749>
labels = ['interpreter-core', 'type-bug', '3.9']
title = 'Raise non-silent warning for invalid escape sequences'
updated_at = <Date 2022-01-13.16:38:26.080>
user = 'https://github.com/Vgr255'bugs.python.org fields:
activity = <Date 2022-01-13.16:38:26.080>
actor = 'dansebcar'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2018-02-22.18:34:57.749>
creator = 'abarry'
dependencies = []
files = []
hgrepos = []
issue_num = 32912
keywords = ['patch']
message_count = 34.0
messages = ['312575', '312610', '312760', '326766', '328044', '344312', '344764', '345557', '345558', '345639', '345675', '345703', '346871', '348205', '348269', '348270', '348281', '348347', '348348', '348349', '348354', '348357', '348375', '349027', '349103', '349106', '349318', '349320', '349321', '349322', '349323', '349328', '349335', '410496']
nosy_count = 17.0
nosy_names = ['rhettinger', 'terry.reedy', 'gregory.p.smith', 'mark.dickinson', 'ncoghlan', 'r.david.murray', 'njs', 'lukasz.langa', 'serhiy.storchaka', 'steve.dower', 'abarry', 'CuriousLearner', 'miss-islington', 'asmeurer', 'xtreak', 'Eric Wieser', 'dansebcar']
pr_nums = ['5849', '9652', '15142', '15195']
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32912'
versions = ['Python 3.9']
Activity
Vgr255 commentedon Feb 22, 2018
This is a follow-up to bpo-27364.
Back in Python 3.6, a silent warning was added for all invalid escape sequences in str and bytes. It was suggested that it would remain a silent warning (which still impacts tests, while not visually annoying the average user) for two releases (3.6 and 3.7), then would be upgraded to a non-silent warning for two subsequent releases (3.8 and 3.9) before becoming a full-on syntax error. With the 3.7 feature freeze on and going, I think it's time to evaluate the approach we take for 3.8 :)
I suggest upgrading the DeprecationWarning to a SyntaxWarning, which is visible by default, for 3.8 and 3.9. I have cross-linked bpo-27364 to this issue as well.
-Em
ncoghlan commentedon Feb 23, 2018
+1 for the DeprecationWarning->SyntaxWarning->SyntaxError approach from me (especially as 3.7 will make the existing deprecation warning visible in interactive shells and __main__ modules by default).
serhiy-storchaka commentedon Feb 24, 2018
PR 5849 changes not only the Python parser, but codecs. It shouldn't. The Python parser and codecs will go different ways. The warning in the Python parser will finally be upgraded to SyntaxError (it is already replaced with SyntaxError if the warning is raised as error). The warning in codecs will become UnicodeDecodeError or ValueError. This is why the code for emitting the warning is not shared between the parser and codecs at first place.
serhiy-storchaka commentedon Oct 1, 2018
Since the author of PR 5849 had not the time to work on it, I have created PR 9652 which properly replaces a DeprecationWarning with a SyntaxWarning. It also updates the documentation.
serhiy-storchaka commentedon Oct 19, 2018
New changeset 6543912 by Serhiy Storchaka in branch 'master':
bpo-32912: Replace a DeprecationWarning with a SyntaxWarning (GH-9652)
6543912
rhettinger commentedon Jun 2, 2019
I'm seeing these warnings pop up from time to time with various third party packages (such as pyparsing which is invoked by ensurepip).
Am wondering whether this should be deferred for another version or so until be have good confidence that the major third party packages have all had a chance to adapt. Otherwise, this will mostly be noise for end-users. Also, it gets in the way of the end-user strategy of "backslash anything that looks special" as preferred over "remember exactly which things are special". This may be a case of "the cure is worse than the disease".
asmeurer commentedon Jun 5, 2019
I agree with Raymond that third party libraries are not ready for this.
My biggest issue is that the way Python warns about this makes it very difficult for library authors to fix this. Most won't even notice. The problem is the warnings are only shown once, when the file is compiled. So if you run the code in any way that causes Python to compile it, a further run of 'python -Wd' will show nothing. I don't know if it's reasonable, but it would be nice if Python recompiled when given -Wd, or somehow saved the warning so it could be shown if that flag is given later.
As an anecdote, for SymPy's CI, we went through five (if I am counting correctly) iterations of trying to test this. Each of the first four were subtly incorrect, until we finally managed to find the correct one (for reference, 'python -We:invalid -m compileall -f -q module/'). So most library authors who will attempt to add tests against this will get it wrong. Simply adding -Wd as you would expect is wrong. If the code is already compiled (which it probably is, e.g., if you ran setup.py), it won't show the warnings. At the very least the "correct" way to test this should be documented.
Things would probably be improved if the warnings were always shown, as at least then devs will see the error once (although most will probably be confused when the warning doesn't repeat).
Another problem is the information in the warnings. It seems the line number of the string is now shown, which is an improvement (https://bugs.python.org/issue28128). It would be nice if it showed the actual line and column number in the file of the invalid escape. This is especially annoying when an escape appears in a docstring. It just shows """ as the offending line.
We have a lot of LaTeX in our docstrings in SymPy, so we had quite a few of these to fix. SymPy doesn't have invalid escapes anymore because I was proactive about it, but from what I've seen, most library authors haven't been.
By the way, this looks like a bug (python 3.8b1):
It might not be possible to see in what I pasted, but it filled my terminal with spaces.
serhiy-storchaka commentedon Jun 14, 2019
In the particular case of pyparsing there was a bug in the docstring.
\1 and \2 were interpreted as control characters U+0001 and U+0002, so the user could see
or
or
depending on the output device.
So this examples proves the usefulness of the new warning.
serhiy-storchaka commentedon Jun 14, 2019
Aaron, you have reported several different issues in your message:
compiler warnings are emitted only once, therefore it is easy to miss them and hard to reproduce.
warnings for multiline expressions (and multiline string literals in particularly) could be better.
strange output of spaces in some cases.
And maybe more.
I think all of them deserve opening separate issues.
asmeurer commentedon Jun 14, 2019
I agree. Please someone else do that. I don't know what already has issues and I unfortunately don't have time right now to help out with any of this.
I simply mentioned all these things as arguments why Python should not (yet) make these warnings errors, which is the point of this issue.
Also, for the pyparsing example, you would have gotten lucky because it also contained \w, which is not a valid escape. If it didn't, you wouldn't be warned. So clearly this will help things, but it will also be good to have linting tools that, for example, warn about any escape sequences inside docstrings.
22 remaining items
asmeurer commentedon Aug 9, 2019
Are there issues tracking the things I mentioned, which should IMO happen before this becomes a hard error (making the warnings reproduce even if the file has already been compiled, and making warning message point to the correct line in multiline strings)? And is it too late to potentially get some of those things in 3.8?
gpshead commentedon Aug 10, 2019
I haven't looked, so not that i'm aware of. I suggest filing one for each of those.
The warning not pointing to the right line in a multiline literal sounds like a bug to me so that one, if fixed, seems reasonable for 3.8. The release manager gets to decide.
Making the syntax warnings happen upon import of an already compiled pyc is more feature-ish, likely to be 3.9+.
rhettinger commentedon Aug 10, 2019
One possibility is to restrict the warning to a backslash followed by an alphabetic character or backslash, and that we define backslash followed by any other printable character as specifically allowed. This would catch the likely sources of errors without breaking the likes of \, or \-. Also, multiline strings have less of a need for a warning than a single line string.
miss-islington commentedon Aug 10, 2019
New changeset b4be87a by Miss Islington (bot) (Gregory P. Smith) in branch 'master':
bpo-32912: Revert SyntaxWarning on invalid escape sequences. (GH-15195)
b4be87a
dansebcar commentedon Jan 13, 2022
I'm not sure if this is an issue or by design, but this DeprecationWarning behaves differently to other DeprecationWarnings.
A normal DeprecationWarning triggered by code in __main__ is printed by default:
But this one is silent:
$ python -c '"\,"' [no output]To see this DeprecationWarning at all, I need to type:
But that enables this DeprecationWarning for all modules, not just __main__ .
I've tested this with Python 3.9 on debian bullseye and the 3.10 docker image.
CUDA stubs docstring: Replace illegal escape sequence
vstinner commentedon Nov 2, 2022
New attempt to replace DeprecationWarning with SyntaxWarning:
vstinner commentedon Nov 3, 2022
The regex was fixed in the meanwhile. With Bottle 0.12.23, there is no warning anymore:
vstinner commentedon Nov 3, 2022
I confirm that this issue is fixed in docutils 0.19. I tested Sphinx 5.3.0. There is no warning anymore:
serhiy-storchaka commentedon Jul 3, 2026
The SyntaxWarning was landed again in 3.12 (gh-98401). Closing as completed.