uPj,UdZddlZddlZddlZddlZddlZddlZddlmZddl m Z m Z m Z ddl mZmZddlmZddlmZmZdZeed <d Zeed <eezZed d GddZdedefdZdedzfdZdede gefdzfdZdedefdZ dede!fdZ"dede!dzfdZ#dede!fdZ$de eddfdZ%GddZ&e&Z'eed <y)!aA Deferred close of HTTP/SDK clients that the LLM client cache has evicted. Eviction only drops the cache's reference to a client. Every OpenAI/Azure SDK client is a reference cycle (each resource namespace holds the client back), so an evicted client and its pooled TCP connections survive until a generational collection runs, which under load is thousands of requests later. Closing at eviction time is not an option: a request that was handed the client just before it was evicted is still using it, and closing it underneath that request raises ``RuntimeError: Cannot send a request, as the client has been closed.`` So an evicted client is closed once two conditions hold. A grace window must have passed since its eviction, which covers a request that holds the client but is momentarily not on the wire, and the client must report no connection in flight. The second condition is what keeps the first honest: a request may run for ``litellm.request_timeout`` seconds, 6000 by default, and a streaming response is bounded only by how long the upstream keeps sending, so no deadline on its own can promise that a request has finished. Only clients litellm itself created are closed; a client the caller supplied is left alone because litellm does not own its lifecycle. A client that closes synchronously is closed from wherever the cache is next used. One whose close is a coroutine needs the event loop it was evicted on, so it waits for a call from that loop rather than having work scheduled onto a loop it does not belong to. Queued clients are therefore bucketed by what it takes to close them, and each bucket is ordered by deadline, so a reap walks the entries that are due rather than the whole queue. The queue holds its clients weakly, so waiting out a grace window never keeps alive anything the collector would have reclaimed first. N)deque) AwaitableCallableIterator) dataclassreplace)Final)&EVICTED_LLM_CLIENT_CLOSE_GRACE_SECONDS$EVICTED_LLM_CLIENT_CLOSE_MAX_PENDINGzclosable-anywhere_CLOSABLE_ANYWHEREzclosable-on-any-loop_CLOSABLE_ON_ANY_LOOPT)frozenslotsc@eZdZUdZded<edzed<eed<eed<y) _PendingClosea~A queued close. The client is held weakly, so queueing one never keeps alive anything the collector would otherwise have reclaimed first. ``needs_loop`` is set for a client whose close is a coroutine; those can only be closed from the event loop they were evicted on, recorded in ``loop_id``. A client that closes synchronously carries neither constraint. zweakref.ref[object] client_refNloop_id needs_loop close_after)__name__ __module__ __qualname____doc____annotations__intboolfloatj/home/oem/CyberBrain26-unified/.venv/lib/python3.12/site-packages/litellm/caching/evicted_client_closer.pyrr:s%&% 4Zrrpendingreturncb|jstS|jtS|jS)zPWhich reaps can close this entry: any at all, any running a loop, or one loop's.)rr rr )r!s r _bucket_keyr$Ls+   !!$$ ??rc\ ttjS#t$rYywxYwN)idasyncioget_running_loop RuntimeErrorrrr _running_loop_idr+Us-'**,-- s  ++clientc>t|ddxs t|dd}|S)Nacloseclosegetattr)r,close_fns r _close_functionr3\s'3:68T3R3tV]^dfmosVtH Orcjt|ddt|dd|fD]}t|dd}||cSy)zOThe httpx transport behind an SDK wrapper, a litellm handler, or a bare client._clientNr, _transportr0)r,holder transports r _transport_ofr9asJ69d3WVXt5TV\]#FL$?    r connectioncVt|dd}t|rt|SdS)z=A pooled connection is idle unless it is servicing a request.is_idleNT)r1callabler)r:r<s r _connection_is_idler>js($ZDAG&w/4 ?9T9rr8ctt|dddd}t|ttfsyt d|DS)zWhether the httpcore pool behind the transport is servicing a request. ``None`` when there is no such pool, so the caller can ask the other backend. _poolN connectionsc34K|]}t| ywr&)r>).0r:s r z,_pool_has_busy_connection..xs   ++s)r1 isinstancelisttupleany)r8pooleds r _pool_has_busy_connectionrJpsG $GIw$E}VZ[F ftUm ,   rc  t|}t|}||St|dd}ttt|ddddS#t$rYywxYw)a?Whether the client is servicing a request right now. Both connection backends litellm uses already account for the connections they have handed out, so this reads the client's own lease accounting rather than inferring it from elapsed time: httpcore reports a non-idle connection for the whole of a response including a stream, and aiohttp holds the connection in ``_acquired`` over the same span. A client that cannot answer is reported as idle, which leaves the grace window as the only guard, exactly as it was before this check existed. Nr, connector _acquiredF)r9rJr1r Exception)r,r8 pooled_busysessions r _has_connection_in_flightrQ~sd(0 6yA  " !(Hd!CGGG[$?dSTT sA -A AAclosingcKtjt5|d{dddy7 #1swYyxYwwr&) contextlibsuppressrN)rRs r _close_quietlyrVs8   Y ' s >202 >2;>c eZdZdZeeejfdede de gefddfdZ de ddfd Z de defd Zde ddfd Zdd Zede fd ZdeddfdZde dzdedeedffdZdededeefdZde ddfdZy)EvictedClientCloserzJCloses evicted, litellm-owned clients once they are idle and out of grace. grace_seconds max_pendingclockr"Nc||_||_||_tj|_i|_d|_tj|_ t|_ y)Nr) _grace_seconds _max_pending_clockweakrefWeakSet_owned_buckets_pending_count threadingLock _queue_lockset _close_tasks)selfrYrZr[s r __init__zEvictedClientCloser.__init__sQ ,' /6/@ @B $>>+58Urr,cZ |jj|y#t$rYywxYw)zIRecord that litellm created this client, so it may be closed on eviction.N)rbadd TypeErrorrjr,s r mark_ownedzEvictedClientCloser.mark_owneds)  KKOOF #   s  **c> ||jvS#t$rYywxYw)NF)rbrnros r _is_ownedzEvictedClientCloser._is_owneds( T[[( (  s  c J||j|syt|}|y|j|jk\ry|j t t j|ttj||j|jzy)a~Queue an evicted client for closing once it is idle and out of grace. Past ``max_pending`` the client is left to the collector instead, so a workload that churns the cache cannot grow this queue without bound. Every queued entry comes due within one grace window, so the capacity it occupies is returned within that window rather than held. N)rrrr) rrr3rdr^_enqueuerr`refr+inspectiscoroutinefunctionr_r])rjr,r2s r schedulezEvictedClientCloser.schedules >!7 )&1      $"3"3 3  ";;v.(*"66x@ KKMD,?,??   rc.|jsy|j}|jt|D][}|j }|t |r*|j t|||jzK|j|]y)zClose every queued client that is due, idle, and closable from here. Called from the cache's read path, so the empty-queue exit comes first and the work done past it is proportional to what is due, not to the queue. N)r) rdr_ _take_duer+rrQrtrr]_close)rjnowr!r,s r reapzEvictedClientCloser.reaps "" [[]~~&6&8#> G'')F~(0 gg3ATAT;TUV KK  rc|jSr&)rd)rjs r pending_countz!EvictedClientCloser.pending_counts"""rr!c|j5|jjt|t }|rN|dj ;|j |xjdzc_|r|dj ;|j||xjdz c_dddy#1swYyxYw)a-Append to the entry's bucket, dropping any dead entries it queues behind. Deadlines only ever move forward, so appending keeps each bucket ordered by deadline, and entries whose client the collector already took sit at the front rather than having to be searched for. rN) rgrc setdefaultr$rrpopleftrdappend)rjr!buckets r rtzEvictedClientCloser._enqueues   % MM44[5I57SFVAY113; ##q(#VAY113; MM' "   1 $   % % %sA=B: &B::Crr|.c|tfn tt|f}j5tfd|DcdddS#1swYyxYw)Nc3PK|]}j|D]}|ywr&) _drain_locked)rCkeyr!r|rjs r rDz0EvictedClientCloser._take_due..s,`S4CUCUVY[^C_```s#&)r r rgrG)rjrr|bucketss` ` r rzzEvictedClientCloser._take_duesL+2?%'ASUjls@t    a`G`` a a as AA rc#K|jj|}|y|rN|dj|kr<|xjdzc_|j |r|dj|kr<|s|j|=yyw)Nrr)rcgetrrdr)rjrr|rs r rz!EvictedClientCloser._drain_lockeds ))#. > ..#5   1 $ .." "..#5 c"s A.B1BcRt|}|y |}tj|syt j j t|}|jj||j|jjy#t$rYywxYwr&) r3rNrv isawaitabler(r) create_taskrVrirmadd_done_callbackdiscard)rjr,r2rRtasks r r{zEvictedClientCloser._closes)&1    %ZG""7+ ..0<<^G=TU d# t00889    sB B&%B&)r"N)rrrrr r time monotonicrrrrkobjectrprrrrxr}propertyrrrtrGrz _BucketKeyrrr{rrr rXrXsT F?%)^^ ; ; ;E " ;  ;D4  v $ 0 $#s## % %$ %ata%aE-QTBTrJrQrVrXrrrr rs=!F 99* 0E/5u5 3Y  $d#$":#*FxF ';d'B &V:F:t:  D4K f.)F"3 w:w:t(;'<u