Repository navigation
Conversation
The maxval, poll, and currval deprecation warnings used stacklevel=1, so they named progressbar's own source instead of the line that used the old name. A fixed stacklevel=2 would only be right for currval: _apply_deprecated_aliases is reached through ProgressBar.__init__ and any subclass __init__ chain above it, so the depth varies per caller. _caller_stacklevel() walks the frames from the warning site outward until the code is no longer inside the progressbar package and returns that depth. Five tests record the warning and assert its filename and line are the caller's, for a plain bar, a DataTransferBar, a user subclass with its own __init__, and the currval property.
WrappingIO.write iterated the shared listener set while start_capturing and stop_capturing on another thread added or removed bars, which on 3.14t and 3.15t raised "Set changed size during iteration" tens of thousands of times per run and dropped bar updates. The listener set is now guarded by a small dedicated StreamWrapper.listeners_lock that every WrappingIO shares, held only around the add, remove and the snapshot in write(), never while calling out, so it cannot take part in a lock-order cycle with a logging handler or a locking target stream. The wider StreamWrapper.lock (reentrant) guards wrap, unwrap and capture transitions. Both classes pickle and deepcopy again by dropping the locks in __getstate__ and recreating them on load. StreamWrapper has class-level default locks so a subclass that skips super().__init__() keeps working. Unwrapping delivers the dropped wrapper's buffered text to the original stream and never raises for a stream that cannot take it (logged at DEBUG). What working code can notice: WrappingIO gains an optional lock= keyword (api_surface_snapshot regenerated), its default lock is a plain threading.Lock, and text that was buffered in a wrapper at unwrap time now reaches the original stream instead of being lost.
The new-style widget formatter resolved any attribute path a template
named, so a template taken from user input could walk {variables.x.__class__}
and further into the interpreter. Rendering goes through a string.Formatter
subclass that refuses attribute segments starting with two underscores
and raises ValueError; single-underscore names and everything else render
exactly as in 4.6.0. Non-str templates with a callable format() (lazy
translation proxies) are rendered through str(); None, lists and numbers
raise TypeError as before.
ProgressBar(desc=...) is rendered literally as f'{desc}: ' and is never
a format template; a prefix assigned afterwards is a template as before.
PROGRESSBAR_MINIMUM_UPDATE_INTERVAL that is not a non-negative finite
number is ignored with one WARNING per distinct value instead of raising
at construction.
What working code can notice: templates addressing __dunder__ paths now
raise ValueError, a desc containing braces renders them literally, and a
bad interval variable logs instead of raising.
Validate every size option through size_to_bytes with a minimum and a parser error (status 2) instead of a traceback, reject a missing input, a directory input, a directory output and an output path that is also an input before the output is truncated, and close every input after its copy. An early-closing reader (`... | head`) ends the copy quietly with status 0: the BrokenPipeError is caught and, when the broken stream is the process's real stdout (fd 1), the descriptor is pointed at os.devnull so the interpreter's exit flush cannot raise again. A replacement sys.stdout that owns another descriptor is left alone. Ctrl-C exits 130. main() is split into small helpers (_parse_size_option, _collect_inputs, _check_output_path, _open_input, _copy_stream) with no behaviour change beyond the above. What working code can notice: invalid -s/-B/-L values exit 2 with a message instead of a traceback, `-o` onto an input file is refused, and `... | head` exits 0 where 4.6.0 exited 120 with "Exception ignored".
…play
Batch item labels are data, not terminal markup: every ESC-introduced
sequence and every C0/C1 control is stripped before the label becomes a
MultiBar key, so an item cannot move the cursor, retitle the terminal or
hide output from the other rows. Labels that are not str are rendered
through str() first. A finished task's bar is finished (end='') before
it is dropped so its stream listeners, capture refcounts and resize
handler return to baseline instead of leaking per task.
In the shell runner the {} and {item} placeholders are substituted in a
single regex pass, so placeholder text inside an item is never expanded
again, and shell=True with a list or callable template is a TypeError:
subprocess would run only the first element through the shell and drop
the item.
The ANSI detection called the console-mode probe on Windows for every stream, including pipes and files, where it reports on the console the process happens to own rather than on the stream being asked about. The probe now runs only when the stream is a tty, matching the other platforms.
The publish job runs only for a v* tag whose name matches __version__ and whose commit is the checked-out commit, so a manual dispatch on a branch or a mistyped tag cannot publish. The built sdist and wheel are listed and refused when they contain sync-conflict, .orig, .rej or .bak files. The workflow token is limited to contents: read. Local secret and environment files are ignored by git.
Two adversarial reviews of the previous commit found that the
transition lock was still held while logging ("wrapping excepthook")
and while flushing captured text into the original streams. A logging
handler that writes into a wrapper, or a target stream with its own
lock, could then wait on a thread that was waiting for the transition
lock: a deadlock 4.6.0 could not have, because it held no lock at all.
Moving the call-outs outside the lock naively made two overlapping
stop_capturing() calls deliver the same text twice.
Now every transition takes the buffered text under the lock
(WrappingIO._take) and writes it to the original stream after
releasing it (WrappingIO._deliver), so nothing blocks while the lock is
held and no text is delivered twice. The excepthook installation logs
after the lock is released.
The locks live in a small shared box, _SharedLock, that a StreamWrapper
and the wrappers it installs hold in common. The box keeps its identity
through pickle and deepcopy whichever object is the root, so restored
wrappers share the restored lock, subclass slots survive (default
pickling again, no __getstate__), subclasses that define __eq__ without
__hash__ construct, and a forked child gets fresh unlocked locks for
every live box through os.register_at_fork, standalone wrappers
included. Lock annotations evaluate on 3.10 to 3.12, where
threading.Lock is a factory function.
What working code can notice: StreamWrapper.lock and listeners_lock
are read-only properties (both were introduced on this branch, 4.6.0
had neither), and get_type_hints(WrappingIO.__init__) works again on
3.10 to 3.12.
ProgressBar(desc=...) rendered the literal text by inserting the plain
str into the widget list, so bar.widgets[0] was a str where 4.6.0 (and
prefix=) had a FormatLabel; a subclass that rewrites that label after
start() silently missed it. The desc-derived prefix is now escaped and
wrapped in a FormatLabel when the widgets are built, while bar.prefix
keeps the raw f'{desc}: ' value that the fast path renders verbatim.
The safe formatter also refuses the non-dunder attributes that lead
from a generator, coroutine, traceback or frame stored as a bar
variable to a frame's globals (f_globals, f_locals, f_builtins,
f_back, gi_frame, cr_frame, ag_frame, tb_frame).
The once-per-value warning for a bad PROGRESSBAR_MINIMUM_UPDATE_INTERVAL
records the value under a lock before logging, so concurrent first
constructions produce one record, remembers every distinct value, and
its lock is reset in a forked child.
Refuse an -o path that is the file stdin is open on, an unreadable input and a closed stdin before the output is opened, so the output is never truncated for an invocation that then fails. Reject a size option above sys.maxsize instead of raising OverflowError from read(). Flush the output inside the guarded block, so a broken pipe that is only noticed when a small copy's buffer reaches the pipe exits 0 like a large one.
The render thread snapshots the bars and a worker may delete one (task_finished) before the finished-bar path runs. That path now returns early for a bar that is no longer registered, undoes its timestamp if the bar vanished while it was stamped, and ignores a concurrent removal when expiring it, instead of leaking the entry or raising KeyError from the render thread.
…ames The version check accepted only `__version__ = '...'`; the annotated spelling `__version__: str = '...'` failed the guard although the package built. The stray-file pattern matched any path containing .orig, .rej or .bak; it now matches those as file extensions only.
The test relied on 30 ms sleep differences between five workers and failed twice on a loaded machine. Only one item can now complete until a gate opens, so the first yielded pair is deterministic.
test_wrapping_io_and_a_redirecting_bar_deepcopy_and_pickle deep-copied a started, redirecting bar and a StreamWrapper holding one, and dropped the copies. A copy of a started bar is itself a started bar, so its __del__ ran finish() whenever the gc got to it, usually inside a later test, where it dropped the singleton's capture count and restored sys.stdout under that test's feet. That is what made test_redirected_stderr_lines_are_flushed_above_bar fail now and then under coverage. The copies are marked finished before they go.
A flush takes the buffered text of both wrapped streams and then writes it to the two original streams. When the stdout write raised, the exception left the loop before the stderr text was written, and that text was already gone from the buffer, so a broken stdout cost the stderr lines of the same round. Every taken text is now written before the first exception is raised. Delivery stays as 4.6.0 did it: each caller writes what it took, at once, with no lock held during the writes. A single ordered delivery queue was tried first and taken back again, in both a non-blocking shape (a caller returned with its own text still queued behind another thread, so text written right after a stop landed above it, and a flush during a render thread's delivery wrote nothing) and a blocking shape (a lock-order cycle with a logging handler whose stream is the wrapper, when the original stream itself logs). Working code notices: an exception from the stdout target no longer drops the stderr text flushed in the same call.
The shared resize registry saved the previous handler when its set of bars was empty and restored it from the last uninstall(). Three ways out of a bar skipped that restore: a bar collected without finish() left the registry's handler installed for the rest of the process, a finish() off the main thread hit the ValueError from signal.signal() with the bar already discarded, after which the next bar saved the registry's own handler as the "previous" one and the real one was lost for good, and a handler installed from C (getsignal() returns None) made the restore a TypeError. One bar whose _handle_resize() raised also skipped the other bars and propagated out of the signal handler into unrelated code. Whether the shared handler is installed is now read from getsignal() instead of inferred from the set, so a handler that other code set over ours is never clobbered. The saved handler is cleared only after a restore succeeds, a restore that fails off the main thread is retried at the next registration change or the next resize signal, None restores SIG_DFL (SIGWINCH is ignored by default), every registered bar gets a weakref finalizer that restores once no live bar is left, and a failing bar is logged at WARNING on the progressbar.bar logger and skipped. What working code can notice: a resize handler that raised now logs instead of raising, and the previous handler comes back after bars that were dropped without finish() or finished from a worker thread.
WrappingIO.write() appended to the StringIO buffer without a lock while _take() read it, then truncated it. A write landing between the read and the truncate was gone: deterministic with a buffer that pauses in truncate(), and a handful of tokens per run in a six-thread stress on the free-threaded builds. 4.6.0 has the same race. Each WrappingIO now holds a reentrant lock in a shared-lock box that write() and _take() take around the StringIO write, getvalue, seek and truncate calls only. It is never held while a listener, the target stream or the buffer's own flush() is called (a review found that a replacement buffer whose flush() logs through a handler on the wrapper deadlocked when the flush ran inside the lock), so it cannot take part in a lock-order cycle with a logging handler. The box survives pickle, deepcopy and fork like the listeners lock does, and the class-level default keeps a subclass that skips super().__init__() working.
`progressbar FILE -o /dev/stdout | head -c1` exited 1. The file opened for -o is not sys.stdout, so the quiet exit for a reader that went away did not apply, and the interpreter reported the failing flush at exit. _output_is_stdout() now also accepts a file whose fstat() reports the same device and inode as sys.stdout, and _discard_broken_stdout() points both descriptor 1 and the file's own descriptor at os.devnull so neither the exit flush nor the exit stack's close raises again. Nothing changes unless descriptor 1 is involved: a replacement stdout on another descriptor, a regular output file and a FIFO given by path are left alone.
python-utils before 4.1.4 scaled a value between 0 and 1/1024 to a
negative power, and DataSize and format_unit_value() used that power as
an index into their prefix tuple, so a fraction of a byte rendered with
the largest prefix ('0.1 YiB', '0.1 PiB'). 4.1.4 returns power 0 for
those values. The floor moves from 3.8.1 to 4.1.4, the development lock
follows, and two tests pin the rendering of a value below one byte.
What working code can notice: installing progressbar2 upgrades a
python-utils older than 4.1.4. Both packages already require Python
3.10 or newer.
| def test_subclass_without_super_init_can_wrap( | ||
| monkeypatch: pytest.MonkeyPatch, | ||
| ) -> None: | ||
| class NoSuper(utils.StreamWrapper): |
|
|
||
|
|
||
| def test_wrapping_io_subclass_without_super_init_can_write() -> None: | ||
| class NoSuper(utils.WrappingIO): |
| self.needs_clear = False | ||
|
|
||
| wrapper: NoSuper = NoSuper(io.StringIO()) | ||
| assert wrapper.write('spam\n') == 5 |
|
|
||
| import pytest | ||
|
|
||
| import progressbar |
| import pytest | ||
| from test_stream import reset_wrapped_streams | ||
|
|
||
| import progressbar |
| # Never flushes here: the count just went above zero | ||
| try: | ||
| self._apply_capturing() | ||
| except BaseException as exception: # noqa: BLE001 |
| self.capturing = max(self.capturing - 1, 0) | ||
| try: | ||
| pending = self._apply_capturing() | ||
| except BaseException as exception: # noqa: BLE001 |
| if failure is None: | ||
| try: | ||
| self._deliver(pending) | ||
| except BaseException as exception: # noqa: BLE001 |
| elif operation == 'pop': | ||
| assert multibar.pop(key) is progress | ||
| elif operation == 'setdefault': | ||
| assert multibar.setdefault(key) is progress |
| assert multibar.pop('none') is None | ||
| assert multibar.pop('missing', 42) == 42 | ||
| with pytest.raises(KeyError): | ||
| multibar.pop('missing') |
| assert multibar.setdefault('none') is None | ||
| assert multibar.pop('none') is None | ||
| assert multibar.pop('missing', 42) == 42 | ||
| with pytest.raises(KeyError): |
| if removal == 'pop': | ||
| assert multibar.pop('task') is progress | ||
| elif removal == 'popitem': | ||
| assert multibar.popitem() == ('task', progress) |
| self._finished_at.clear() | ||
| self._labeled.clear() | ||
| # Release the last references after leaving the membership guard. | ||
| del removed |
| list(multibar._render_bar(old, now=10, expired=None)) | ||
| except BaseException as error: | ||
| errors.append(error) | ||
| finally: |
|
|
||
| def __hash__(self) -> int: | ||
| raise AssertionError('hash callback') | ||
|
|
| class EqualBar(progressbar.ProgressBar): | ||
| def __hash__(self) -> int: | ||
| raise AssertionError('tracking called hash') | ||
|
|
| ) | ||
| except BaseException as error: | ||
| errors.append(error) | ||
| finally: |
| """Deliver one wrapper's text, returning what it raised.""" | ||
| try: | ||
| self._deliver_one(name, wrapper, text) | ||
| except BaseException as exception: # noqa: BLE001 |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found by a security audit and three adversarial hunt rounds over the
streams, widgets, CLI, parallel and release surface of 4.6.0, followed by
five independent review passes (two fresh Claude reviewers, three Codex
runs) over the fixes themselves. Every fix started as a failing test. Only
errors whose fix cannot break working code are in here, plus the contract
changes the maintainer decided on, listed at the bottom with the decision.
Streams (progressbar/utils.py)
StreamWrapper and every WrappingIO it installs, held only around the
add, remove and snapshot, never while calling out.
reentrant lock and write it to the original streams after releasing
it, so a logging handler or a locking target stream cannot form a
lock-order cycle, and two overlapping flushes never deliver text twice.
still written before the exception propagates.
a per-wrapper lock covers the StringIO write, getvalue, seek and
truncate calls only, never a call-out.
deepcopy and is reset in a forked child, so wrappers restored together
share one lock and nothing stays locked after fork.
instead of being lost.
Widgets and bar (progressbar/widgets.py, progressbar/bar.py)
refuses attribute segments starting with two underscores and the frame
attributes (f_globals, gi_frame, tb_frame and friends), so a template
taken from user input cannot walk from a bar variable into the
interpreter. Everything else renders as in 4.6.0.
render through str(). None, lists, numbers and widgets raise TypeError
as before.
bar.widgets[0] stays a FormatLabel.
number is ignored with one WARNING per distinct value instead of
raising at construction.
collected without finish(), finished from a worker thread (retried on
the main thread at the next change or signal), or over a handler that
was installed from C. A bar whose resize handler raises is logged and
skipped instead of aborting the dispatch for the other bars. A handler
that other code set over ours is never clobbered.
finished frame no longer raises KeyError or leaks its entry.
Command line (progressbar/main.py)
of a traceback, including values above sys.maxsize.
that is also an input or the file stdin is open on, and a closed stdin
are refused before the output is truncated.
... | head) ends the copy with status 0 andno "Exception ignored" at exit, for a large copy, a small one whose
buffer only reaches the pipe at the final flush, and an
-o /dev/stdoutthat aliases descriptor 1.
Parallel display (progressbar/_parallel)
characters before they become MultiBar keys, so an item cannot move the
cursor or retitle the terminal.
capture counts and the resize registration return to baseline.
with a list or callable template is a TypeError instead of running only
the first element.
Release and environment
(plain or annotated) on the checked-out commit, refuses sdists and
wheels with stray files, and has a read-only token.
scale_1024 power for a fraction of a unit and DataSize rendered
'0.1 YiB' for it.
Compatibility
How it was checked: three hunt rounds against the released wheel (every
reproduction re-run under both trees, template sweeps of 47802 cases, the
released 4.6.0 test suite unchanged against the branch), then five review
passes with probes on 3.13, 3.14t and 3.15t. Final tree: 1226 tests pass
with 100% line and branch coverage, the released 4.6.0 suite passes (896
passed, 13 skipped, two deselected tests pin the surface changes below),
the touched test files pass on 3.10, 3.12, 3.15, 3.14t and 3.15t, and a
six-writer two-flusher stress of 27000 tokens per interpreter loses and
duplicates nothing where 4.6.0 lost 5670 and duplicated 1462 on 3.14t.
What working code can notice:
api_surface_snapshot test reports the wider signature (regenerated).
existed in 4.6.0). A WrappingIO subclass declaring slots = ('lock',)
shadows the property.
-oonto an input isrefused,
... | headexits 0 where 4.6.0 exited 120.Pre-existing, left as in 4.6.0
Found by the final review pass and reproduced identically on 4.6.0:
capturingjust before the wrapper was unwrappedlands in the detached buffer and is lost. The decision and the write
are not atomic with the unwrap.
progressbar.utils.streamswhile it wraps fails on thereplaced
sys.excepthookbuiltin. Standalone wrappers and deepcopywork.
share one entry, and an unhashable bar leaves the handler installed.
For a decision (decided)
progressbar before its own write lands sees later text first. Identical
to 4.6.0, kept (an ordered delivery queue was tried and taken back: it
stranded text behind other threads in one shape and deadlocked against
a logging handler in the other).
str(). An object with only str stays a TypeError as in 4.6.0.