Repository navigation
Whole-VM wedge recurred on 2.2.3: blocking open() still on the vCPU thread #2659
Description
Follow-up to #2593, which was closed as fixed in v2.2.2. Same failure on 2.2.3 — the blocking open() is still on the vCPU thread.
OrbStack 2.2.3 (2020300) c83556b0 / macOS 26.5.2 (25F84) / M4, 16 GiB / guest kernel 7.0.14-orbstack-00380-ga7e0a2dc9535, 10 vCPUs.
Wedged 18:52:29–19:37:11 local on 2026-08-19, ~45 min. docker and orb status hang, vmgr at ~790% CPU, 37 hang samples. Host side, vcpu7, 689 of 689 samples:
689 Thread_4415934: vcpu7
+ ... 9 frames in OrbStack Helper ...
+ 689 open (in libsystem_kernel.dylib) + 64
+ 689 __open (in libsystem_kernel.dylib) + 8
Guest side, all 10 stall reports on CPU 7 in one grace period, counters frozen from first to last:
[644184.816960] rcu: 7-...0: (16 ticks this GP) idle=0834/1/0x4000000000000000 softirq=439232/439234 fqs=10032
[644184.817007] rcu: (detected by 8, t=90007 jiffies, g=1593677, q=55228 ncpus=10)
[646705.470882] rcu: (detected by 8, t=2610649 jiffies, g=1593677, q=71448 ncpus=10)
The sampler caught the stuck CPU's guest stack this time, which #2593 was missing:
virtio_fs_enqueue_req+2008
virtio_fs_send_req+95
__fuse_simple_request+547
fuse_file_open+279
fuse_do_open+31
fuse_dir_open+75
vfs_open+279
path_openat+1051
__arm64_sys_openat+287
fuse_dir_open, so the host open() is on a directory. My FIFO guess in #2593 was wrong.
Seccomp escalates one stuck CPU into a full spin: container starts JIT a filter, which reaches __text_poke → kick_all_cpus_sync and waits for every CPU to ack. CPU 7 never does, so each start parks another vCPU in smp_call_function_many_cond — ~240 of 334 guest stack dumps sit there. That's the 790% versus ~200% in #2593, and it means the VM degrades progressively under restart policies rather than losing one CPU.
Separately, the disk warning misdiagnoses this. The health check fails 27 times with Post "http://vcontrol/disk/report_stats": context deadline exceeded, and OrbStack put up a dialog asking whether my disk was full. It wasn't — 45 GiB free on the host volume, 41 GiB on the OrbStack volume. That timeout is a symptom of the wedged vCPU, so the dialog sends you looking at storage. I dismissed it and the VM recovered within seconds; previous incidents stayed wedged for hours untouched, so that path may be worth checking.
Trigger unchanged: a throwaway container joined the bridge at 18:51:09 and was destroyed moments later, first stall detection 80 s after. Checked after recovery — no nested mounts under the shared trees, no dataless directories, no leftover FIFOs or sockets, no symlink into the OrbStack NFS mount at ~/OrbStack.
Redacted orb report: https://gist.githubusercontent.com/owahltinez/e4fd6f49b65afc75f1b907d2646f78c6/raw/orbstack-diagreport-redacted.zip. Dropped netstat_rn.txt and scutil_dns.txt, replaced my username and container names, collapsed pre-hang krpc: dref failed paths. Kernel logs, stalls, both sample captures and the .orbprof files unmodified.
Activity
danrush777 commentedon Sep 1, 2026
Can confirm this on a second, independent setup — five freezes in six days, same signature.
Setup: Mac mini, macOS 26.6.2, OrbStack 2.2.3, guest kernel
7.0.14-orbstack-00380, 5 containers, heavy bind-mount usage.Signature (sampled during a freeze): One vCPU host thread stuck in
open()(662/662 samples) while the others idle or spin. Guest NMI backtrace on the corresponding CPU showsfindinfuse_dir_open → virtio_fs_send_reqon/mnt/macpaths. Every subsequentdocker execthen piles up indo_seccomp → bpf_int_jit_compile → __text_poke → kick_all_cpus_sync → smp_call_function_many_cond, so 6-7 of 8 vCPUs end up spinning (~580-690% CPU on the OrbStack Helper).docker psandorbctl statusblock indefinitely.Incidents: Aug 26 (17 h), Aug 30 (5 h), Aug 31 03:00 (3.5 h), Aug 31 15:14 (50 min, self-resolved — matches the ~45 min self-recovery reported above), Sep 1 03:01 (5+ h). Frequency increased over the week. All but the self-resolving one required
orbctl stop/start.Triggers observed (not deterministic): a 341 MB database export written through a bind mount, and recursive
findruns over large mounted trees (incl.node_modules). The same operations complete fine on other days, so it looks like a race rather than a deterministic bug — consistent with #2471/#2593 resurfacing.Since the reporter is on macOS 26.5.2 and we are on 26.6.2, this does not look like a macOS-version regression.
Happy to provide vmgr logs, spindumps, or run a debug build if that helps.
nick-inkeep commentedon Oct 4, 2026
Same wedge on macOS 27.0 (26A428), Apple M5 Max, OrbStack 2.2.3 (2020300, c83556b0), kernel
7.0.14-orbstack-00380-ga7e0a2dc9535, an 18-vCPU / 16 GiB VM, Docker 29.4.0.It began after about 5h20m of VM uptime and lasted two hours, until a restart:
g=622501.A host sample at the end shows vcpu13 in
open()for all 164 samples:The seccomp JIT path described above was stuck too. In addition, BPF program freeing was stuck in the same cross-CPU sync on another CPU, in all 28 waves:
orbctl stopforced the VM down after its graceful-stop timeout, and a fresh start recovered. The path passed toopen()wasn't captured, and I don't have a deterministic repro.nick-inkeep commentedon Oct 6, 2026
A likely trigger for this wedge, and a workaround that has held for 24 h so far (OrbStack 2.2.3, macOS 27.0).
Three of four wedges here on Oct 4–5 began within minutes of a container bind-mounting a path on an external APFS volume (USB4 SSD under
/Volumes). In the clearest one,docker eventsshows a container created with such a bind, then an exec runningchown -Rover it. The first RCU stall followed about 70 s later, the exec never returned, and the screen was locked at the time.Inferred, not proven: an
open()of a removable-volume path waits on a macOS privacy (TCC) decision that never arrives, so the vCPU thread blocks there (vcpu13 inopen()above). On the same Mac, a launchd-started process reading that volume also blocks in__open(908/908 samples).After granting OrbStack Full Disk Access and restarting it, the same pattern completes in about a second: bind-mount a 300-file folder from that volume, then
cat,chown -Randtouchover it. The VM has since run 24 h with no RCU stall. Bind mounts of that volume were otherwise avoided during those 24 h, so treat Full Disk Access as a workaround, not a confirmed fix.dwaipayanbanerjee commentedon Oct 10, 2026
Independent reproduction and request for a supported fix/debug build on OrbStack 2.2.3 (2020300, c83556b0), guest 7.0.14-orbstack-00380-ga7e0a2dc9535, Docker 29.4.0, macOS 27.0.1, arm64, 8 vCPUs.
Historical VM-wide stalls show one missing guest CPU while other CPUs wait in
__text_poke -> kick_all_cpus_sync -> smp_call_function_many_cond. A host sample puts the corresponding vCPU ingetattrlistbulk; a later guest stack isvirtio_fs_enqueue_req -> virtio_fs_send_req -> __fuse_simple_request -> fuse_readdir -> getdents64. Older task backtraces identify NZBGet on this directory-read path across different CPUs.A bounded privileged replay on the same helper identified an external APFS directory with 5,617 entries. The scan took 7.83 seconds and issued 80 successive
getattrlistbulkcalls totaling 7.62 seconds. These generated 1,586 small APFS metadata reads totaling 7.56 seconds. No individual metadata read exceeded 36 ms; the longest bulk call was 0.79 seconds. Kernel samples show APFS B-tree/object reads andbuf_biowaiton the vCPU host thread. Identical sampled stacks therefore do not establish one continuously blocked syscall.An ordinary concurrent container exec took 7.18 seconds versus a 0.041-second baseline, while all 30 Docker pings succeeded below 3 ms. The short replay lacks guest kernel stacks and did not reproduce the historical minutes-long outage. Moving active metadata to Linux volumes avoids the workload but leaves the underlying execution behavior unchanged.
Disk sleep was already disabled, Full Disk Access was granted before the incidents, and examined device counters showed no transport timeout/reset/retry evidence. We have no basis for changing those settings or disabling seccomp.
Is there a supported release/debug build that moves slow host filesystem work off vCPU execution, or another supported correction for this failure class? For the minutes-long case, what targeted capture would distinguish a single blocked syscall from uninterrupted processing of successive directory calls? We can test a candidate against the same workload and verify guest CPU progress and concurrent task start while retaining normal security settings.
No raw diagnostics are attached; they contain private paths and would require separate review before sharing.
jameslai-sparkofy commentedon Oct 14, 2026
Another reproduction on 2.2.3, plus a TCC data point that matches the Full Disk Access workaround above.
Setup: Mac mini M4 Pro, 64 GiB, macOS 26.6.2, OrbStack 2.2.3, guest kernel
7.0.14-orbstack-00380-ga7e0a2dc9535, Docker 29.4.0, 12 vCPU / 32 GiB VM, ~160 containers. The OrbStack data image (data_dir) and the bind mounts of 45 containers live on an external APFS SSD under/Volumes.Wedges: 2026-10-07 (twice, on 2.1.3, the
do_seccomp → bpf_int_jit_compile → __text_poke → kick_all_cpus_syncstack from #2471), then on 2.2.3 on 2026-10-12 ~04:00 and 2026-10-13 00:32 UTC (~12–18 min,sampling stacks due to VM hang). Host-sidevmgrsample during the 10-13 wedge: 8 of 12 vCPU threads were inopen(libsystem_kernel open+64 → __open+8, every sample). During the manual stop/start right after, the helper also sat ~10 min inopen()ofdata.img.rawon the same external volume.TCC state at the time:
kTCCServiceSystemPolicyRemovableVolumes= allowed fordev.kdrag0n.MacVirt, butkTCCServiceSystemPolicyAllFiles(Full Disk Access) = denied (auth_value 0). That fits the theory that anopen()on a removable-volume path is waiting on a TCC decision on the vCPU thread. We are about to grant Full Disk Access and restart; I'll report back if it recurs.+1 to moving blocking host filesystem calls off the vCPU thread. A single slow
open()should not be able to wedge the whole VM through the seccomp IPI.TyceHerrman commentedon Oct 30, 2026
Potentially related: #2659 describes whole-VM hangs with host filesystem operations blocking vCPU threads.
The symptoms are different but cross-linking in case the connection is useful for investigation
TyceHerrman commentedon Oct 30, 2026
Potentially related incident on OrbStack 2.2.5, macOS 27.2.
The VM became unresponsive during a NixOS development workload. A guest command timed out after 480 seconds, independent machine-info/status requests hung, and the existing SSH listener accepted a connection but sent no greeting. The helper remained alive.
Preserved logs show:
may not have been same cause as this issue. the host sample doesn't show a vCPU blocked in open() or getattrlistbulk, and the captured guest stacks don’t show the seccomp/BPF JIT path described here. The host sample is sparse though.
I preserved the VM logs and existing Helper sample. and could send the raw diagnostics in private channel after redacting private paths and identifiers