uPj&UdZddlZddlmZmZmZddlmZerddlm Z GddeZ Gdd eZ Gd d eZ ed hZeed < ddedzdedfdZy)aBounds the blast radius of a single node's transient connection error on the async Redis Cluster client. redis-py's ``RedisCluster._execute_command`` responds to a ``ConnectionError`` or ``TimeoutError`` on ANY one node by tearing down every node's connections and flipping the client into "needs reinitialization", which forces every other concurrent caller sharing this client through one reinit lock until the whole cluster topology is re-walked. Under real proxy load, a client-side socket timeout on a single node is a routine event (the event loop was too busy to read the response before ``socket_timeout`` elapsed) and does not mean the cluster's topology moved, so treating it as a full-cluster event turns one slow node into a proxy-wide latency spike while Redis itself stays healthy -- confirmed live: pausing one of three local cluster nodes made every concurrent command against the other two, untouched nodes stall for the full pause duration too. ``get_litellm_async_redis_cluster_class`` returns a ``RedisCluster`` subclass that resets only the node that actually failed (mirroring what a plain, non-cluster Redis client already does when one of its pooled connections errors), leaving every other node's connections untouched. Every other branch (MOVED, ASK, CLUSTERDOWN, slot-not-covered, retry-exhaustion) is unchanged from upstream, since those already carry real evidence the topology changed. redis-py 8.x fixed this upstream with gentler machinery than this override's ``node.disconnect()`` (which also kills connections other coroutines are mid-operation on, so one timeout cascades into a reconnect storm and, with TLS, a fresh handshake per killed connection): it marks in-use connections for reconnect only after their current operation completes, disconnects only the idle pooled ones, and defers reinitialization to the outer retry loop. When the installed ``ClusterNode`` has that per-connection recovery API, the factory returns the base ``RedisCluster`` unmodified. N) TYPE_CHECKINGFinalProtocol)verbose_logger RedisClusterc,eZdZdZdededefdZddZy) _ClusterNodeAttrsaThe subset of ``redis.asyncio.cluster.ClusterNode`` this override reads. redis-py ships no resolvable stub for these members under the repo's current types-redis pin, so a plain attribute access resolves every downstream use to ``Unknown`` under strict mode; typing ``target_node`` as this Protocol at the one boundary keeps the override's own logic fully typed without a banned ``typing.cast``.argskwargsreturnc KywN)selfr r s q/home/oem/CyberBrain26-unified/.venv/lib/python3.12/site-packages/litellm/caching/redis_cluster_node_isolation.pyexecute_commandz!_ClusterNodeAttrs.execute_command/s Nc Kywrrrs r disconnectz_ClusterNodeAttrs.disconnect4rr N)__name__ __module__ __qualname____doc__objectrrrrr r (s-?    ,rr c0eZdZUeed<dedededefdZy)_NodesManagerAttrs_moved_exceptionslotread_from_replicasload_balancing_strategyr cyrr)rr#r$r%s rget_node_from_slotz%_NodesManagerAttrs.get_node_from_slot:srN) rrrr__annotations__intboolr r'rrrr!r!7s-  -1 LR  rr!creZdZUeed<eed<eed<eed<eed<eed<dede fd Z d edefd Z dd Z y ) _ClusterAttrsRedisClusterRequestTTLreinitialize_counterreinitialize_stepsr$r% nodes_manager node_namer cyrr)rr1s rget_nodez_ClusterAttrs.get_nodeGsrr c Kywrr)rr s r_determine_slotz_ClusterAttrs._determine_slotHrrNc Kywrrrs raclosez_ClusterAttrs.acloseIrrr) rrrr)r(r*rr!strr r3r5r7rrrr,r,?sD##%%@#@*;@>6>c>'rr,z5.3.1_VERIFIED_REDIS_VERSIONScluster_node_classr _AsyncRedisClusterTypec ddl}ddlm}ddlm}ddlmddlm ddlm m m m m m m mddlmdd lm||n|}t'|d r"t)j*d |j,|S|j,t.vr.t)j0d |j,t3t.G f d d|}|S)aReturns the base ``RedisCluster`` when the installed redis-py already recovers a node-level connection error per-connection (8.x+), else builds the ``RedisCluster`` subclass with the per-node isolation fix for older versions whose upstream branch tears down the whole cluster client. ``cluster_node_class`` exists for dependency injection in tests; production callers leave it unset and the installed ``ClusterNode`` is used. Imported lazily because this module is reachable from a base ``import litellm`` while redis is not a base dependency. Cheap to call repeatedly: the underlying redis submodules are cached in ``sys.modules`` after the first import. rN) ClusterNoder) get_node_name) READ_COMMANDS)AskErrorBusyLoadingErrorClusterDownError ClusterErrorMaxConnectionsError MovedErrorSlotNotCoveredError TryAgainError)ConnectionError) TimeoutError'update_active_connections_for_reconnectzredis-py %s recovers a node-level connection error per-connection upstream; using the base RedisCluster without litellm's node-isolation override.zredis-py %s is not in the set this cluster-teardown-storm fix was verified against (%s). The per-node-isolation override may not match the installed library's real _execute_command behavior.c @ eZdZdedededef f d Zy)Gget_litellm_async_redis_cluster_class..LiteLLMAsyncRedisCluster target_noder r r cj K|}|}dx}}d}|j} | dkDr| dz} |r2|J|j|}|jdd{d}n^|r\|j|d{} |jj | |j xr|dv|dvr |jnd}d}|j|i|d{Sd 77m7# f$rf$r|jd{7f$r9|jd{7tjdd{7$r{} |xjdz c_ |jr=|j|jzdk(r!|jd{7d|_ n| |j_d}Yd} ~ nkd} ~ w $r*} | j| j }d}Yd} ~ n>d} ~ w$r3| |jd z krtjd d{7YnwxYw| dkDrbw) NFr)r1ASKINGg?T)hostportg?zTTL exhausted.)r-r3rr5r0r'r$r%rr7asynciosleepr.r/r"rQrR)rrMr r clusternodeaskingmoved redirect_addrttlr#er@rArBrCrDrEr?rFrG_RedisConnectionError_RedisTimeoutErrorr>s r_execute_commandzXget_litellm_async_redis_cluster_class..LiteLLMAsyncRedisCluster._execute_commandsP &*GD" "FU(,M00C'q)2,888&//-/H"228<<-/AB//+++(*=>!..***!-----!!00A5011g6R6RU\UoUo6ost6t%nn...784AB--> E"$1qvvAFF$KM!F$2W;;a??%mmD1112S's!H3,CCC-C.AC C C H3CCC!H'DH' D#!H'>E? H' AF?F F?:H3?H' G,'H3,2H'H!H'$H3&H''H3N)rrrr rr_) r@rArBrCrDrEr?rFrGr]r^r>s rLiteLLMAsyncRedisClusterrLs8: 1*: 1: 1 : 1  : 1: 1: 1rr`)redisredis.asyncio.clusterr=r redis.clusterr>redis.commandsr?redis.exceptionsr@rArBrCrDrErFrGrHrIhasattrrdebug __version__r9warningsorted)r:ra_AsyncClusterNode_BaseAsyncRedisCluster node_classr`r@rArBrCrDrEr?rFrGr]r^r>s @@@@@@@@@@@@r%get_litellm_async_redis_cluster_classrnSs,,   JC.@.L*RcJzDE U    &%  88 .    + ,  =1=1=1=1~ $#rr)rrTtypingrrrlitellm._loggingrrbrr;r r!r, frozensetr9r(typernrrrrss}<11+L , ,   (H ("#,WI"6%6'+v$t v$ "#v$r