tPj UdZddlZddlZddlZddlZddlZddlZddlmZddl m Z dZ ee d<dZ ee d<d Zee d <d Zee d <d efdZdeded eeefdzfdZd#deded eedffdZded efdZdeded dfdZdeedfded eedffdZefdeded dfdZefdeedfded dfdZd#deded eedffdZded dfdZd Zee d!<d ej@dzfd"Z!y)$aSupervisor-side reaper for orphaned Prisma query-engine processes. Each proxy worker owns a Prisma query-engine subprocess whose only cleanup hook is an in-process ``atexit`` handler. When a multi-worker supervisor (uvicorn's multiprocess manager, the gunicorn arbiter) force-kills a hung or crashed worker, that handler never runs: the engine reparents to the nearest subreaper (PID 1 in a container, which is the supervisor itself under the standard docker entrypoint) and keeps its database connection pool established forever, while the replacement worker opens a fresh pool. Over repeated worker deaths the active database connections grow without bound. The reaper runs only in the supervisor process, where a query-engine process can never be a legitimate direct child: workers own their engines, and the supervisor never starts one. Any direct child whose command name begins with ``query-engine`` is therefore an adopted orphan and is terminated (SIGTERM, bounded grace, SIGKILL) and reaped. On Linux the supervisor also marks itself a child subreaper so orphans reparent to it even when it is not PID 1. Linux-only by construction (``/proc`` scan, ``prctl``); a no-op elsewhere. N)Final)verbose_proxy_loggerz query-engineQUERY_ENGINE_COMM_PREFIX@REAPER_SCAN_INTERVAL_SECONDSg$@SIGTERM_GRACE_SECONDS$PR_SET_CHILD_SUBREAPERreturnctjjdsy tjdd}|j t dddd}|dk(S#ttf$rYywxYw)zMark this process as a child subreaper so orphaned descendants reparent to it instead of PID 1. Best-effort: when it fails (or on non-Linux) the reaper still covers the containerized case where the supervisor already is PID 1.linuxFNT) use_errnor) sysplatform startswithctypesCDLLprctlr OSErrorAttributeError)libcresults i/home/oem/CyberBrain26-unified/.venv/lib/python3.12/site-packages/litellm/proxy/db/query_engine_reaper.pyset_child_subreaperr'sj << " "7 +kk$$7jj "Aq!Q { ^ $s4AA)(A)pid proc_rootc t|d|ddd5}|j}dddj d}|jd}|dk(s |dk(s||kry||d z|}||d zdj}t|d kry t|d }||fS#1swY{xYw#tttt f$rYywxYw#t$rYywxYw) N/z/statasciireplace)encodingerrors()r) openreadFileNotFoundErrorProcessLookupErrorPermissionErrorrfindrfindsplitlenint ValueError) rr stat_filedatalparenrparencommfieldsppids r_read_comm_and_ppidr:8s YKqU+gi P +T]#..*D +IIcNFJJsOF |v|vvzF+D!&,,.F 6{Q&)n :! + + 1?G L s9B-B!B-C !B*&B--C C  CC parent_pid.c tj}d|D}t fd|DS#ttf$rYywxYw)zPIDs of direct children of ``parent_pid`` whose command name marks them as Prisma query engines. In the supervisor these are always adopted orphans: live engines are children of workers, not of the supervisor.c3TK|] }|jst|"ywN)isdigitr1).0entrys r z,list_orphaned_engine_pids..VsPE SZPs((c3~K|]4}t|x$dk(rdjtr|6yw)Nrr)r:rr)rArinfor;rs rrCz,list_orphaned_engine_pids..WsJ 'Y7 7D D Gz ! G  7 8 s:=)oslistdirr*rtuple)r;rentriescandidate_pidsrEs`` @rlist_orphaned_engine_pidsrKMsV I.QWPN !  w 's9A  A c tj|tj\}}||k(S#t$rYyt$rYywxYw)NT)rFwaitpidWNOHANGChildProcessErrorr)r reaped_pid_s r _try_reaprR`sK 3 3 A    s'. AAAsignumcf tj||y#tttf$rYywxYwr?)rFkillr+r,r)rrSs r _send_signalrVjs.  V  9   s 00pidstimeout_secondsctj|z}|}|rZtj|krCtd|D}|rtjd|rtj|krC|S)zoPoll until every PID is reaped or the shared deadline passes. Returns the PIDs still alive at the deadline.c38K|]}t|r|ywr?)rR)rArs rrCz _await_reaped..wsI#)C.#Isg?)time monotonicrHsleep)rWrXdeadline remainings r _await_reapedr`qsdnn&8HI (83III  JJsO (83  grace_secondsc t|f|y)z{SIGTERM the orphaned engine, escalate to SIGKILL after the grace period, and reap it so it does not linger as a zombie.)rbN)terminate_and_reap_all)rrbs rterminate_and_reapre}sC6?racR|D]2}tjd|t|tj4t ||}|sy|D]3}tjd||t|tj 5t |d}|D]}tjd|y)aTerminate a batch of orphaned engines concurrently: SIGTERM all of them, share one grace period, SIGKILL the stragglers, and reap. The shared deadline keeps cleanup time bounded when several workers die at once instead of paying the grace period once per orphan.zXReaping orphaned prisma query-engine PID %s (its worker process exited without cleanup).NzZOrphaned prisma query-engine PID %s did not exit within %.1fs of SIGTERM; sending SIGKILL.rzROrphaned prisma query-engine PID %s survived SIGKILL; will retry on the next scan.)rwarningrVsignalSIGTERMr`SIGKILLerror)rWrbr survivors unkillables rrdrds*$$ f   S&..) * %T=9I *$$ h   S&..) *&i5J "" `   rac:t||}|r t||S)z5One scan-and-reap pass. Returns the PIDs it acted on.)r)rKrd)r;r orphaned_pidss rreap_orphaned_enginesrps4Z9UM}- rac t|t j t &#t$r }tjd|Yd}~>d}~wwxYw)Nz%Orphaned query-engine scan failed: %s)rp Exceptionrdebugr[r]r)r; scan_errors r _reaper_looprusP  \ !* - /0  \ & &'NPZ [ [ \s ( AA  Az"litellm-orphan-query-engine-reaperREAPER_THREAD_NAMEctjjdsytdt j Dd}||St t jttjfdt}|jtjdtj|S)aStart the reaper daemon thread in the supervisor process. Must only be called from a process that never hosts the proxy app itself (uvicorn with ``workers > 1``, the gunicorn arbiter): with a single in-process uvicorn worker the query engine is a legitimate direct child and must not be touched. Idempotent: a reaper already running in this process is returned instead of starting a second one. r Nc3HK|]}|jtk(s|ywr?)namerv)rAthreads rrCz,start_query_engine_reaper..sYFv{{FX7XYs""T)targetargsdaemonryzEStarted orphaned prisma query-engine reaper in supervisor process %s.)rrrnext threading enumeraterThreadrurFgetpidrvstartrrE)existing reaper_threads rstart_query_engine_reaperrs << " "7 +Yi113Y H$++iik^  M O  ra)z/proc)"__doc__rrFrhrrr[typingrlitellm._loggingrr__annotations__rrr boolrr1strrHr:rKrRrVfloatr`rerdrprurvrrr=rarrs,  1"0%0&)e)#u# ""T"SSU38_t5K*##ERUWZRZO&34 c 3 4  c3h % E#s(O 9N@C@@RV@1 S/    @cccSVh1S1T1AE@9#3#3d#:ra