Pewarta Proxy

📋 Server Request & Proxy Logs

Loading active proxy settings...
Developed by Khairul Adha (adhexa.id)
Mengambil log server...

⚙️ Pengaturan Proxy Target

Skip to content

Fix the security audit and adversarial hunt findings against 4.6.0 - #330

Open
wolph wants to merge 25 commits into
developfrom
fix/security-audit-hardening
Open

wolph wants to merge 25 commits into
developfrom
fix/security-audit-hardening

Conversation

@wolph

@wolph wolph commented Oct 28, 2026

Copy link
Copy Markdown
Owner

Found by a security audit and three adversarial hunt rounds over the
streams, widgets, CLI, parallel and release surface of 4.6.0, followed by
five independent review passes (two fresh Claude reviewers, three Codex
runs) over the fixes themselves. Every fix started as a failing test. Only
errors whose fix cannot break working code are in here, plus the contract
changes the maintainer decided on, listed at the bottom with the decision.

Streams (progressbar/utils.py)

# 4.6.0, two threads, 3.14t                   # branch
RuntimeError: Set changed size during ...     bars updated, no error
stdout write fails -> stderr text lost        stderr text still delivered
write during flush -> token gone (5670/27000) 0 missing, 0 duplicated
  • The listener set is guarded by a small lock shared between the
    StreamWrapper and every WrappingIO it installs, held only around the
    add, remove and snapshot, never while calling out.
  • Wrap, unwrap and capture transitions take the buffered text under a
    reentrant lock and write it to the original streams after releasing
    it, so a logging handler or a locking target stream cannot form a
    lock-order cycle, and two overlapping flushes never deliver text twice.
  • When the stdout write raises, the stderr text of the same round is
    still written before the exception propagates.
  • A write that lands between the read and the truncate of a drain is kept:
    a per-wrapper lock covers the StringIO write, getvalue, seek and
    truncate calls only, never a call-out.
  • Locks live in a shared box that keeps its identity through pickle and
    deepcopy and is reset in a forked child, so wrappers restored together
    share one lock and nothing stays locked after fork.
  • Text buffered in a wrapper at unwrap time reaches the original stream
    instead of being lost.

Widgets and bar (progressbar/widgets.py, progressbar/bar.py)

  • New-style templates render through a string.Formatter subclass that
    refuses attribute segments starting with two underscores and the frame
    attributes (f_globals, gi_frame, tb_frame and friends), so a template
    taken from user input cannot walk from a bar variable into the
    interpreter. Everything else renders as in 4.6.0.
  • Non-str templates with a callable format() (lazy translation proxies)
    render through str(). None, lists, numbers and widgets raise TypeError
    as before.
  • ProgressBar(desc=...) is rendered literally and is never a template.
    bar.widgets[0] stays a FormatLabel.
  • PROGRESSBAR_MINIMUM_UPDATE_INTERVAL that is not a non-negative finite
    number is ignored with one WARNING per distinct value instead of
    raising at construction.
  • The shared SIGWINCH handler is restored however the last bar goes away:
    collected without finish(), finished from a worker thread (retried on
    the main thread at the next change or signal), or over a handler that
    was installed from C. A bar whose resize handler raises is logged and
    skipped instead of aborting the dispatch for the other bars. A handler
    that other code set over ours is never clobbered.
  • A MultiBar bar removed by a worker while the render thread draws its
    finished frame no longer raises KeyError or leaks its entry.

Command line (progressbar/main.py)

  • Every size option is validated with a parser error (status 2) instead
    of a traceback, including values above sys.maxsize.
  • A missing, unreadable or directory input, a directory output, an output
    that is also an input or the file stdin is open on, and a closed stdin
    are refused before the output is truncated.
  • A reader that goes away (... | head) ends the copy with status 0 and
    no "Exception ignored" at exit, for a large copy, a small one whose
    buffer only reaches the pipe at the final flush, and an -o /dev/stdout
    that aliases descriptor 1.

Parallel display (progressbar/_parallel)

  • Batch item labels are stripped of escape sequences and control
    characters before they become MultiBar keys, so an item cannot move the
    cursor or retitle the terminal.
  • A finished task's bar is finished before it is dropped, so listeners,
    capture counts and the resize registration return to baseline.
  • Shell templates substitute {} and {item} in one pass, and shell=True
    with a list or callable template is a TypeError instead of running only
    the first element.

Release and environment

  • The publish workflow runs only for a v* tag that matches version
    (plain or annotated) on the checked-out commit, refuses sdists and
    wheels with stray files, and has a read-only token.
  • The Windows console-mode probe runs only for a tty.
  • python-utils floor 3.8.1 -> 4.1.4: earlier versions returned a negative
    scale_1024 power for a fraction of a unit and DataSize rendered
    '0.1 YiB' for it.

Compatibility

How it was checked: three hunt rounds against the released wheel (every
reproduction re-run under both trees, template sweeps of 47802 cases, the
released 4.6.0 test suite unchanged against the branch), then five review
passes with probes on 3.13, 3.14t and 3.15t. Final tree: 1226 tests pass
with 100% line and branch coverage, the released 4.6.0 suite passes (896
passed, 13 skipped, two deselected tests pin the surface changes below),
the touched test files pass on 3.10, 3.12, 3.15, 3.14t and 3.15t, and a
six-writer two-flusher stress of 27000 tokens per interpreter loses and
duplicates nothing where 4.6.0 lost 5670 and duplicated 1462 on 3.14t.

What working code can notice:

  • WrappingIO gains an optional lock= keyword, so the released
    api_surface_snapshot test reports the wider signature (regenerated).
  • StreamWrapper.lock and listeners_lock are read-only properties (neither
    existed in 4.6.0). A WrappingIO subclass declaring slots = ('lock',)
    shadows the property.
  • Templates addressing dunder or frame attributes raise ValueError.
  • A desc containing braces renders them literally.
  • A bad interval variable logs once per value instead of raising.
  • A resize handler that raises is logged, not propagated.
  • CLI: invalid sizes exit 2 instead of a traceback, -o onto an input is
    refused, ... | head exits 0 where 4.6.0 exited 120.
  • Installing progressbar2 upgrades python-utils to 4.1.4 or newer.

Pre-existing, left as in 4.6.0

Found by the final review pass and reproduced identically on 4.6.0:

  • A write that read capturing just before the wrapper was unwrapped
    lands in the detached buffer and is lost. The decision and the write
    are not atomic with the unwrap.
  • Pickling progressbar.utils.streams while it wraps fails on the
    replaced sys.excepthook builtin. Standalone wrappers and deepcopy
    work.
  • The resize registry keys bars by equality and hash: two equal bars
    share one entry, and an unhashable bar leaves the handler installed.

For a decision (decided)

  • Reentrant target streams: a target whose write() calls back into
    progressbar before its own write lands sees later text first. Identical
    to 4.6.0, kept (an ordered delivery queue was tried and taken back: it
    stranded text behind other threads in one shape and deadlocked against
    a logging handler in the other).
  • Lazy templates: only objects with a callable format() render through
    str(). An object with only str stays a TypeError as in 4.6.0.
  • streams.lock stays read-only, no setter.
  • The slots = ('lock',) subclass shape is accepted as is.

wolph added 20 commits September 9, 2026 01:12
The maxval, poll, and currval deprecation warnings used stacklevel=1, so they named progressbar's own source instead of the line that used the old name. A fixed stacklevel=2 would only be right for currval: _apply_deprecated_aliases is reached through ProgressBar.__init__ and any subclass __init__ chain above it, so the depth varies per caller. _caller_stacklevel() walks the frames from the warning site outward until the code is no longer inside the progressbar package and returns that depth. Five tests record the warning and assert its filename and line are the caller's, for a plain bar, a DataTransferBar, a user subclass with its own __init__, and the currval property.
WrappingIO.write iterated the shared listener set while start_capturing
and stop_capturing on another thread added or removed bars, which on
3.14t and 3.15t raised "Set changed size during iteration" tens of
thousands of times per run and dropped bar updates. The listener set is
now guarded by a small dedicated StreamWrapper.listeners_lock that every
WrappingIO shares, held only around the add, remove and the snapshot in
write(), never while calling out, so it cannot take part in a lock-order
cycle with a logging handler or a locking target stream. The wider
StreamWrapper.lock (reentrant) guards wrap, unwrap and capture
transitions.

Both classes pickle and deepcopy again by dropping the locks in
__getstate__ and recreating them on load. StreamWrapper has class-level
default locks so a subclass that skips super().__init__() keeps working.
Unwrapping delivers the dropped wrapper's buffered text to the original
stream and never raises for a stream that cannot take it (logged at
DEBUG).

What working code can notice: WrappingIO gains an optional lock=
keyword (api_surface_snapshot regenerated), its default lock is a plain
threading.Lock, and text that was buffered in a wrapper at unwrap time
now reaches the original stream instead of being lost.
The new-style widget formatter resolved any attribute path a template
named, so a template taken from user input could walk {variables.x.__class__}
and further into the interpreter. Rendering goes through a string.Formatter
subclass that refuses attribute segments starting with two underscores
and raises ValueError; single-underscore names and everything else render
exactly as in 4.6.0. Non-str templates with a callable format() (lazy
translation proxies) are rendered through str(); None, lists and numbers
raise TypeError as before.

ProgressBar(desc=...) is rendered literally as f'{desc}: ' and is never
a format template; a prefix assigned afterwards is a template as before.
PROGRESSBAR_MINIMUM_UPDATE_INTERVAL that is not a non-negative finite
number is ignored with one WARNING per distinct value instead of raising
at construction.

What working code can notice: templates addressing __dunder__ paths now
raise ValueError, a desc containing braces renders them literally, and a
bad interval variable logs instead of raising.
Validate every size option through size_to_bytes with a minimum and a
parser error (status 2) instead of a traceback, reject a missing input,
a directory input, a directory output and an output path that is also an
input before the output is truncated, and close every input after its
copy. An early-closing reader (`... | head`) ends the copy quietly with
status 0: the BrokenPipeError is caught and, when the broken stream is
the process's real stdout (fd 1), the descriptor is pointed at os.devnull
so the interpreter's exit flush cannot raise again. A replacement
sys.stdout that owns another descriptor is left alone. Ctrl-C exits 130.

main() is split into small helpers (_parse_size_option, _collect_inputs,
_check_output_path, _open_input, _copy_stream) with no behaviour change
beyond the above.

What working code can notice: invalid -s/-B/-L values exit 2 with a
message instead of a traceback, `-o` onto an input file is refused, and
`... | head` exits 0 where 4.6.0 exited 120 with "Exception ignored".
…play

Batch item labels are data, not terminal markup: every ESC-introduced
sequence and every C0/C1 control is stripped before the label becomes a
MultiBar key, so an item cannot move the cursor, retitle the terminal or
hide output from the other rows. Labels that are not str are rendered
through str() first. A finished task's bar is finished (end='') before
it is dropped so its stream listeners, capture refcounts and resize
handler return to baseline instead of leaking per task.

In the shell runner the {} and {item} placeholders are substituted in a
single regex pass, so placeholder text inside an item is never expanded
again, and shell=True with a list or callable template is a TypeError:
subprocess would run only the first element through the shell and drop
the item.
The ANSI detection called the console-mode probe on Windows for every
stream, including pipes and files, where it reports on the console the
process happens to own rather than on the stream being asked about.
The probe now runs only when the stream is a tty, matching the other
platforms.
The publish job runs only for a v* tag whose name matches __version__
and whose commit is the checked-out commit, so a manual dispatch on a
branch or a mistyped tag cannot publish. The built sdist and wheel are
listed and refused when they contain sync-conflict, .orig, .rej or .bak
files. The workflow token is limited to contents: read. Local secret
and environment files are ignored by git.
Two adversarial reviews of the previous commit found that the
transition lock was still held while logging ("wrapping excepthook")
and while flushing captured text into the original streams. A logging
handler that writes into a wrapper, or a target stream with its own
lock, could then wait on a thread that was waiting for the transition
lock: a deadlock 4.6.0 could not have, because it held no lock at all.
Moving the call-outs outside the lock naively made two overlapping
stop_capturing() calls deliver the same text twice.

Now every transition takes the buffered text under the lock
(WrappingIO._take) and writes it to the original stream after
releasing it (WrappingIO._deliver), so nothing blocks while the lock is
held and no text is delivered twice. The excepthook installation logs
after the lock is released.

The locks live in a small shared box, _SharedLock, that a StreamWrapper
and the wrappers it installs hold in common. The box keeps its identity
through pickle and deepcopy whichever object is the root, so restored
wrappers share the restored lock, subclass slots survive (default
pickling again, no __getstate__), subclasses that define __eq__ without
__hash__ construct, and a forked child gets fresh unlocked locks for
every live box through os.register_at_fork, standalone wrappers
included. Lock annotations evaluate on 3.10 to 3.12, where
threading.Lock is a factory function.

What working code can notice: StreamWrapper.lock and listeners_lock
are read-only properties (both were introduced on this branch, 4.6.0
had neither), and get_type_hints(WrappingIO.__init__) works again on
3.10 to 3.12.
ProgressBar(desc=...) rendered the literal text by inserting the plain
str into the widget list, so bar.widgets[0] was a str where 4.6.0 (and
prefix=) had a FormatLabel; a subclass that rewrites that label after
start() silently missed it. The desc-derived prefix is now escaped and
wrapped in a FormatLabel when the widgets are built, while bar.prefix
keeps the raw f'{desc}: ' value that the fast path renders verbatim.

The safe formatter also refuses the non-dunder attributes that lead
from a generator, coroutine, traceback or frame stored as a bar
variable to a frame's globals (f_globals, f_locals, f_builtins,
f_back, gi_frame, cr_frame, ag_frame, tb_frame).

The once-per-value warning for a bad PROGRESSBAR_MINIMUM_UPDATE_INTERVAL
records the value under a lock before logging, so concurrent first
constructions produce one record, remembers every distinct value, and
its lock is reset in a forked child.
Refuse an -o path that is the file stdin is open on, an unreadable
input and a closed stdin before the output is opened, so the output is
never truncated for an invocation that then fails. Reject a size option
above sys.maxsize instead of raising OverflowError from read(). Flush
the output inside the guarded block, so a broken pipe that is only
noticed when a small copy's buffer reaches the pipe exits 0 like a
large one.
The render thread snapshots the bars and a worker may delete one
(task_finished) before the finished-bar path runs. That path now
returns early for a bar that is no longer registered, undoes its
timestamp if the bar vanished while it was stamped, and ignores a
concurrent removal when expiring it, instead of leaking the entry or
raising KeyError from the render thread.
…ames

The version check accepted only `__version__ = '...'`; the annotated
spelling `__version__: str = '...'` failed the guard although the
package built. The stray-file pattern matched any path containing
.orig, .rej or .bak; it now matches those as file extensions only.
The test relied on 30 ms sleep differences between five workers and
failed twice on a loaded machine. Only one item can now complete until
a gate opens, so the first yielded pair is deterministic.
test_wrapping_io_and_a_redirecting_bar_deepcopy_and_pickle deep-copied a
started, redirecting bar and a StreamWrapper holding one, and dropped the
copies. A copy of a started bar is itself a started bar, so its __del__
ran finish() whenever the gc got to it, usually inside a later test,
where it dropped the singleton's capture count and restored sys.stdout
under that test's feet. That is what made
test_redirected_stderr_lines_are_flushed_above_bar fail now and then
under coverage. The copies are marked finished before they go.
A flush takes the buffered text of both wrapped streams and then writes
it to the two original streams. When the stdout write raised, the
exception left the loop before the stderr text was written, and that
text was already gone from the buffer, so a broken stdout cost the
stderr lines of the same round.

Every taken text is now written before the first exception is raised.
Delivery stays as 4.6.0 did it: each caller writes what it took, at
once, with no lock held during the writes. A single ordered delivery
queue was tried first and taken back again, in both a non-blocking
shape (a caller returned with its own text still queued behind another
thread, so text written right after a stop landed above it, and a
flush during a render thread's delivery wrote nothing) and a blocking
shape (a lock-order cycle with a logging handler whose stream is the
wrapper, when the original stream itself logs).

Working code notices: an exception from the stdout target no longer
drops the stderr text flushed in the same call.
The shared resize registry saved the previous handler when its set of
bars was empty and restored it from the last uninstall(). Three ways out
of a bar skipped that restore: a bar collected without finish() left the
registry's handler installed for the rest of the process, a finish() off
the main thread hit the ValueError from signal.signal() with the bar
already discarded, after which the next bar saved the registry's own
handler as the "previous" one and the real one was lost for good, and a
handler installed from C (getsignal() returns None) made the restore a
TypeError. One bar whose _handle_resize() raised also skipped the other
bars and propagated out of the signal handler into unrelated code.

Whether the shared handler is installed is now read from getsignal()
instead of inferred from the set, so a handler that other code set over
ours is never clobbered. The saved handler is cleared only after a
restore succeeds, a restore that fails off the main thread is retried at
the next registration change or the next resize signal, None restores
SIG_DFL (SIGWINCH is ignored by default), every registered bar gets a
weakref finalizer that restores once no live bar is left, and a failing
bar is logged at WARNING on the progressbar.bar logger and skipped.

What working code can notice: a resize handler that raised now logs
instead of raising, and the previous handler comes back after bars that
were dropped without finish() or finished from a worker thread.
WrappingIO.write() appended to the StringIO buffer without a lock while
_take() read it, then truncated it. A write landing between the read and
the truncate was gone: deterministic with a buffer that pauses in
truncate(), and a handful of tokens per run in a six-thread stress on the
free-threaded builds. 4.6.0 has the same race.

Each WrappingIO now holds a reentrant lock in a shared-lock box that
write() and _take() take around the StringIO write, getvalue, seek and
truncate calls only. It is never held while a listener, the target
stream or the buffer's own flush() is called (a review found that a
replacement buffer whose flush() logs through a handler on the wrapper
deadlocked when the flush ran inside the lock), so it cannot take part
in a lock-order cycle with a logging handler. The box survives pickle,
deepcopy and fork like the listeners lock does, and the class-level
default keeps a subclass that skips super().__init__() working.
`progressbar FILE -o /dev/stdout | head -c1` exited 1. The file opened
for -o is not sys.stdout, so the quiet exit for a reader that went away
did not apply, and the interpreter reported the failing flush at exit.

_output_is_stdout() now also accepts a file whose fstat() reports the
same device and inode as sys.stdout, and _discard_broken_stdout() points
both descriptor 1 and the file's own descriptor at os.devnull so neither
the exit flush nor the exit stack's close raises again. Nothing changes
unless descriptor 1 is involved: a replacement stdout on another
descriptor, a regular output file and a FIFO given by path are left
alone.
python-utils before 4.1.4 scaled a value between 0 and 1/1024 to a
negative power, and DataSize and format_unit_value() used that power as
an index into their prefix tuple, so a fraction of a byte rendered with
the largest prefix ('0.1 YiB', '0.1 PiB'). 4.1.4 returns power 0 for
those values. The floor moves from 3.8.1 to 4.1.4, the development lock
follows, and two tests pin the rendering of a value below one byte.

What working code can notice: installing progressbar2 upgrades a
python-utils older than 4.1.4. Both packages already require Python
3.10 or newer.
def test_subclass_without_super_init_can_wrap(
monkeypatch: pytest.MonkeyPatch,
) -> None:
class NoSuper(utils.StreamWrapper):


def test_wrapping_io_subclass_without_super_init_can_write() -> None:
class NoSuper(utils.WrappingIO):
self.needs_clear = False

wrapper: NoSuper = NoSuper(io.StringIO())
assert wrapper.write('spam\n') == 5
Comment thread progressbar/utils.py Fixed

import pytest

import progressbar
import pytest
from test_stream import reset_wrapped_streams

import progressbar
Comment thread progressbar/utils.py
# Never flushes here: the count just went above zero
try:
self._apply_capturing()
except BaseException as exception: # noqa: BLE001
Comment thread progressbar/utils.py
self.capturing = max(self.capturing - 1, 0)
try:
pending = self._apply_capturing()
except BaseException as exception: # noqa: BLE001
Comment thread progressbar/utils.py
if failure is None:
try:
self._deliver(pending)
except BaseException as exception: # noqa: BLE001
Comment thread tests/test_multibar.py
elif operation == 'pop':
assert multibar.pop(key) is progress
elif operation == 'setdefault':
assert multibar.setdefault(key) is progress
Comment thread tests/test_multibar.py
assert multibar.pop('none') is None
assert multibar.pop('missing', 42) == 42
with pytest.raises(KeyError):
multibar.pop('missing')
Comment thread tests/test_multibar.py
assert multibar.setdefault('none') is None
assert multibar.pop('none') is None
assert multibar.pop('missing', 42) == 42
with pytest.raises(KeyError):
Comment thread tests/test_multibar.py
if removal == 'pop':
assert multibar.pop('task') is progress
elif removal == 'popitem':
assert multibar.popitem() == ('task', progress)
Comment thread progressbar/multi.py
self._finished_at.clear()
self._labeled.clear()
# Release the last references after leaving the membership guard.
del removed
Comment thread tests/test_multibar.py
list(multibar._render_bar(old, now=10, expired=None))
except BaseException as error:
errors.append(error)
finally:
Comment thread tests/test_multibar.py

def __hash__(self) -> int:
raise AssertionError('hash callback')

Comment thread tests/test_multibar.py
class EqualBar(progressbar.ProgressBar):
def __hash__(self) -> int:
raise AssertionError('tracking called hash')

Comment thread tests/test_multibar.py
)
except BaseException as error:
errors.append(error)
finally:
Comment thread progressbar/utils.py
"""Deliver one wrapper's text, returning what it raised."""
try:
self._deliver_one(name, wrapper, text)
except BaseException as exception: # noqa: BLE001

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants