mirror of
https://github.com/bentoml/BentoML.git
synced 2026-10-12 02:11:27 +00:00
d107ab840c
`bento_new_sdk-e2e-tests` intermittently hangs in `test_custom_command.py::test_command_in_method` until the job's 90-minute cap kills it, which marks the whole CI run cancelled and skips report-coverage and evergreen. It happened on the merge commit for #5689 (run 32459681347: 42 jobs green, that one job 07:40:41 -> 09:10:57, then "Terminate orphan process: nox / pytest / python") and once more in the previous 40 CI runs. Root cause is in this fixture, not in the product code. Both services run `python -m http.server` on a hard-coded port, and neither declared `workers`, so they inherit the custom-command default of `min(16, cpu/2)` workers (_bentoml_sdk/service/factory.py). Reproduced locally: one `bentoml.serve()` of `StaticHTTP1` starts 12 workers on a 24-core box. Only the first worker starts the process (proxy.py's `should_start_process`); every other worker sits in `while proc is None ...: await anyio.sleep(0.5)` waiting for a health check that can never pass if that one process failed to bind. Those workers do not exit on SIGTERM (verified: two SIGTERMs ignored, SIGKILL required), so the test's `with bentoml.serve(...)` teardown blocks forever. On a 4-core runner this is 2 workers, which is why it only bites sometimes. Both fixture services now pin `workers: 1` — required rather than a tuning choice, since a command that binds its own fixed port cannot be replicated — and use separate ports (8000 / 8001, with a matching `http.proxy_port`) so that running the two tests back-to-back cannot collide on a child that has not been reaped yet. That second failure mode was also reproducible: at `workers: 1` but a shared port, the run failed with `OSError: [Errno 98] Address already in use` surfacing as `httpx.RemoteProtocolError: illegal status line`. Before: hangs indefinitely, or ~179 s of readiness timeouts. After: `2 passed in 2.7s`, five consecutive runs, no orphan processes left. `test_asgi.py` in the same suite still passes. The product-side hardening this exposes — `serve()` teardown escalating to SIGKILL after a grace period, so no user command that ignores SIGTERM can hang a run — is deliberately left out of this change. Co-authored-by: xianml <[email protected]> Co-authored-by: Claude Opus 5 <[email protected]>