Repository navigation
SCT fails with OSError: [Errno 98] Address already in use #6345
Description
Issue description
- This issue is a regression.
- It is unknown if this issue is a regression.
The artifacts-ubuntu2204-arm-test job failed with the error:
14:36:24 < t:2023-07-09 11:36:24,227 f:test_config.py l:296 c:sdcm.test_config p:INFO > Initializing Argus connection...
14:36:24 < t:2023-07-09 11:36:24,703 f:tester.py l:390 c:ArtifactsTest p:INFO > test_id 707e495b-0757-47c6-966e-4a613fa62c4f already exists in Argus with status: created
14:36:25 < t:2023-07-09 11:36:24,849 f:tester.py l:405 c:ArtifactsTest p:INFO > sct_runner info in Argus TestRun is updated
14:36:26 Process SyncManager-3:
14:36:26 Traceback (most recent call last):
14:36:26 File "/usr/local/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
14:36:26 self.run()
14:36:26 File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run
14:36:26 self._target(*self._args, **self._kwargs)
14:36:26 File "/usr/local/lib/python3.10/multiprocessing/managers.py", line 591, in _run_server
14:36:26 server = cls._Server(registry, address, authkey, serializer)
14:36:26 File "/usr/local/lib/python3.10/multiprocessing/managers.py", line 156, in __init__
14:36:26 self.listener = Listener(address=address, backlog=16)
14:36:26 File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 453, in __init__
14:36:26 self._listener = SocketListener(address, family, backlog)
14:36:26 File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 596, in __init__
14:36:26 self._socket.bind(address)
14:36:26 OSError: [Errno 98] Address already in use
14:36:26
14:36:26 Aborted!
The error happened at the very beginning of the execution of Run SCT Test () stage of the pipeline, when ClusterTester was being initialized - the test didn't really start.
How frequently does it reproduce?
It reproduced in many other artifacts jobs
scylla-master:
- artifacts-rocky9-test
- artifacts-ubuntu2204-nonroot-test
- artifacts-debian11-test
- artifacts-ubuntu2204-arm-test
- artifacts-debian11-nonroot-test
- artifacts-rocky9-nonroot-test
scylla-enterprise:
- artifacts-ubuntu2204-test
- artifacts-rocky8-test
- artifacts-centos8-arm-test
- artifacts-ubuntu2204-arm-test
and more
Installation details
Cluster size: 1 nodes (im4gn.xlarge)
Scylla Nodes used in this run:
No resources left at the end of the run
OS / Image: ami-022c8ce295ce9ac4c (aws: undefined_region)
Test: artifacts-ubuntu2204-arm-test
Test id: 707e495b-0757-47c6-966e-4a613fa62c4f
Test name: scylla-master/artifacts/artifacts-ubuntu2204-arm-test
Test config file(s):
Logs and commands
- Restore Monitor Stack command:
$ hydra investigate show-monitor 707e495b-0757-47c6-966e-4a613fa62c4f - Restore monitor on AWS instance using Jenkins job
- Show all stored logs command:
$ hydra investigate show-logs 707e495b-0757-47c6-966e-4a613fa62c4f
Logs:
- sct-runner-events-707e495b.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/707e495b-0757-47c6-966e-4a613fa62c4f/20230709_113636/sct-runner-events-707e495b.tar.gz
- sct-707e495b.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/707e495b-0757-47c6-966e-4a613fa62c4f/20230709_113636/sct-707e495b.log.tar.gz
Metadata
Metadata
Assignees
Labels
No labels
Activity
soyacz commentedon Jul 11, 2023
@aleksbykov isn't it related to manager you added recently in #6302?
Maybe we can postpone starting it when needed only?
aleksbykov commentedon Jul 11, 2023
@soyacz , yes looks it is related to that code. It failed when try to initialize the new process event for counter. But it seems internal python errro, so event if it will be initialized lately, it will get same error, because addr will in used anyway. May be i am wrong. As workaround, i can add try except on launching this event-process, so start of it doesn't fail the test? WDYT?
with perf and longevities i didn't catch such error.
soyacz commentedon Jul 11, 2023
We don't count events in ArtifactTests, so this could be a workaround.
Because we run only one test at a time in sct runners.
How about resigning from using manager at all and use Queue/Pipe as a bus for add/get/remove counter (or even don't get counters, just get_stats straight from
EventsCounter) and make counters register internal toEventsCounter. WDYT?fruch commentedon Jul 11, 2023
one thing is for sure in artifacts we could get into a situation we would be running multiple jobs on the same builder as the same time.
so the failure of address is already in use, sounds related to two test running at the same time.
we shouldn't have such a limit, it cloud backfire in multiple places.
fruch commentedon Jul 11, 2023
sounds like it's a bug in python 3.9 and 3.10
and we are pinning to
3.10.0-slim-bullseye, we should move to use3.10.12-slim-bullseye(and also start looking at moving to python 3.11)
fruch commentedon Jul 11, 2023
I'm opening a PR with rebuild of the image, I think it would solve this one.
fix(hydra): switch docker base to 3.10.12-slim-bullseye
fix(hydra): switch docker base to 3.10.12-slim-bullseye
fix(hydra): switch docker base to 3.10.12-slim-bullseye
fix(hydra): switch docker base to 3.10.12-slim-bullseye