uPj+_UdZddlZddlZddlmZddlmZddlmZm Z m Z ddl Z ddl m Z ddlmZmZmZmZddlmZd d lmZmZerdd lmZdd lmZeZee zZne Ze Zd Ze e d<de!ddfdZ"de!dzde#fdZ$e jJdfe jLdfe jNdfe jPdfe jRdfe jTdfe jVdfe jXdfe jZdff Z.e e/e/e0e1fdfe d<dee1e fdzde1de2e3zdzfd Z4d!ed"e1de/ee1e2fdze2dzffd#Z5d$ee1e2fdzde6de2dzfd%Z7d!ed"e1d&e6d'ee1e2fdzd(e2dzd)e#de#fd*Z8d!ed"e1dzd&e6de#fd+Z9 d?d!ed,e1d-e1e2zd.e1dzde#f d/Z: d?d!ed"e1dzd-e1e2zd&e d0e3dzde#f d1Z; d?d!ed"e1d-e1e2zd&e d2e1dzde#f d3Z< d@d!ed&e d-e1e2zd"e1dzd0e3dzd2e1dzde#fd4Z=d!ed5edzde>e1fd6Z?d!ed5edzde>e/fd7Z@d!ed5edzde>e1fd8ZA d@d!ed"e1d&e d9e2dzd:e3dzd;e1dzde#fd<ZBd!ede#fd=ZCd-e1e2zde2fd>ZDy)Aa Router cooldown handlers - _set_cooldown_deployments: puts a deployment in the cooldown list - get_cooldown_deployments: returns the list of deployments in the cooldown list - async_get_cooldown_deployments: ASYNC: returns the list of deployments in the cooldown list N)Mapping)MappingProxyType) TYPE_CHECKINGAnyFinal)verbose_router_logger)DEFAULT_COOLDOWN_TIME_SECONDS*DEFAULT_FAILURE_THRESHOLD_MINIMUM_REQUESTS!DEFAULT_FAILURE_THRESHOLD_PERCENT+SINGLE_DEPLOYMENT_TRAFFIC_FAILURE_THRESHOLD)router_cooldown_event_callback)*get_deployment_failures_for_current_minute+get_deployment_successes_for_current_minute)Span)Router&_litellm_advisor_orchestration_failure#_ADVISOR_ORCHESTRATION_FAILURE_ATTR exceptionreturnc&t|tdy)aKTag an exception as originating from advisor orchestration rather than the health of the router-selected deployment. Advisor orchestration failures (an advisor sub-call that targets different provider/credentials, or the orchestration loop exceeding max_uses) are not caused by the selected deployment, so they must not be attributed to (and cool down) that otherwise-healthy deployment. The exception object is tagged rather than wrapped so its type is preserved and the router's retry/fallback classification and the client-facing error are unchanged. TN)setattrrrs k/home/oem/CyberBrain26-unified/.venv/lib/python3.12/site-packages/litellm/router_utils/cooldown_handlers.py"mark_advisor_orchestration_failurer,s I:DAc6tt|tdS)zKWhether ``exception`` was tagged by ``mark_advisor_orchestration_failure``.F)boolgetattrrrs r is_advisor_orchestration_failurer :s  #FN OOr'ContentPolicyViolationErrorAllowedFailsBadRequestErrorAllowedFailsAuthenticationErrorAllowedFailsTimeoutErrorAllowedFailsRateLimitErrorAllowedFailsInternalServerErrorAllowedFails#ServiceUnavailableErrorAllowedFailsBadGatewayErrorAllowedFailsNotFoundErrorAllowedFails._EXCEPTION_POLICY_FIELDSsourceskeycF|D]}||j|}||cSy)zReturn *key* from the first source mapping where it's set, so callers can support a setting living in more than one deployment config location. Sources are checked in order from most to least specific to that setting.N)get)r,r+sourcevalues r_first_presentr1Ms8 >  3  L  rlitellm_router_instance deploymentc|j|}|y|jdxs ti}|jd}t|tr|nd}|jd}||fS)aReturn (allowed_fails_policy, allowed_fails) from deployment model_info, or (None, None). `model_info` is the only supported location for these two fields (unlike `cooldown_time`, they have no pre-existing `litellm_params` precedent): `litellm_params` gets copied wholesale into the actual provider call kwargs (see e.g. Router._image_generation's `data = deployment["litellm_params"].copy()`), so a new field placed there would leak into the outgoing LLM request instead of staying router-internal. idN)NN model_infoallowed_fails_policy allowed_fails)get_model_infor.r isinstancedict)r2r3depmirawpolicyalloweds r_get_deployment_cooldown_policyrBZsp)77:7FC {#&77<#8#P-JOo&?%J!Pa % 0  )),8id )77:7FC ? sww|,cgg6F.G_]  9 7-55)  rcB|yt||\}}t||duS)aTrue if this deployment has an explicit, deployment-level allowed_fails_policy entry matching *original_exception*'s type. `_is_cooldown_required` skips cooldown evaluation for most 4XX errors (BadRequestError, ContentPolicyViolationError) by default, since a generic client error is usually not the deployment's fault. A deployment-level allowed_fails_policy entry naming that exact exception type is this PR's own per-deployment opt-in, so it overrides that default. Deliberately scoped to the deployment level only, and to the named-exception-type policy dict rather than a plain `allowed_fails` integer: a pre-existing router-wide `allowed_fails_policy` (or a deployment's generic `allowed_fails`) predates this feature and must keep its existing behavior for 4XX types `_is_cooldown_required` already excludes, rather than silently start cooling down deployments whose configs never opted into this specific override. NF)rBrF)r2r3rGrH_s r0_has_explicit_allowed_fails_policy_for_exceptionrZs2(34KZXMJ -j:L MUY YYrmodel_idexception_status exception_strc dg}| |D]}||vsyt|trt|dk(ryt|}|dk\r|dkr|dk(ry|dk(ry|d k(s|d k(ryyy#t$rYywxYw) a> A function to determine if a cooldown is required based on the exception status. Parameters: model_id (str) The id of the model in the model list exception_status (Union[str, int]): The status of the exception. Returns: bool: True if a cooldown is required, False otherwise. APIConnectionErrorFriTiii)r;strlenint Exception)r2r[r\r]ignored_stringsignored_strings r_is_cooldown_requiredrhs ""6!7  $"1 !!]2  ! & ,#$)"#34  s "'7#'=3&!S(!S(,<,C s,A!A!A!A!A! A!! A-,A-time_to_cooldownc||j|tjdy|.tj|ddrtjdy|j rtjdy|tjd yt |||t| s$t||| stjd y||jvrtjd yy)a Helper that decides if cooldown logic should be run Returns False if cooldown logic should not be run Does not run cooldown logic when: - router.disable_cooldowns is True - deployment is None - _is_cooldown_required() returns False - deployment is in litellm_router_instance.provider_default_deployment_ids - exception_status is not one that should be immediately retried (e.g. 401) r5zSShould Not Run Cooldown Logic: deployment id is none or model group can't be found.Fg& .>)ababs_tolz@Should Not Run Cooldown Logic: time_to_cooldown is effectively 0z8Should Not Run Cooldown Logic: disable_cooldowns is Truez1Should Not Run Cooldown Logic: deployment is None)r2r[r\r]r2r3rGzCShould Not Run Cooldown Logic: _is_cooldown_required returned FalsezOShould Not Run Cooldown Logic: deployment is in provider_default_deployment_idsT) get_model_grouprdebugmathisclosedisable_cooldownsrhrbrZprovider_default_deployment_idsr2r3r\rGris r_should_run_cooldown_logicrws$4DD DS[## a  # 7G3X\(]##$fg00##$^_##$WX 7),-   ? 7- ##$ij,LLL##$uv rrequested_model_groupcF|j|}d}| t|dk(r|j| }t||\}}||t ||||||S|j t |durt||} t||} | | z} d} | dkDr| | | zz } tjd|| | | t|} | d k(r|sy | d k(r | tk\ry | tkDr | tk\r|sy tj t| dury yt#||| S)a_ Helper that decides if a deployment should be put in cooldown Returns True if the deployment should be put in cooldown Returns False if the deployment should not be put in cooldown Deployment is put in cooldown when: - v2 logic (Current): cooldown if: - got a 429 error from LLM API - if %fails/%(successes + fails) > ALLOWED_FAILURE_RATE_PER_MINUTE - got 401 Auth error, 404 NotFounder - checked by litellm._should_retry() - v1 logic (Legacy): if allowed fails or allowed fail policy set, coolsdown if num fails in this minute > allowed fails r5Frr2)r2 deployment_idrkrzYpercent fails for deployment = %s, percent fails = %s, num successes = %s, num fails = %sraTg?) status_codero)rprcrouting_group_has_alternativesrBrWr8_is_allowed_fails_set_on_routerrrrrqcast_exception_status_to_intr r r litellm _should_retryrU)r2r3r\rGrx model_grouprJrHrInum_successes_this_minutenum_fails_this_minutetotal_requests_this_minute percent_failsexception_status_ints r_should_cooldown_deploymentr=s21@@J@OK',$3{#3q#8/F/e/e !0 , ( %DD[]g$h!J!!2!>: #     ,    44< +D[ \`e e+V$;:, !(R$;:( -FH],]" % )15NQf5fgM## g   % !  'CCS&T 3 &/O c !&@Do&o = =*.XX4  " "/KL\/] ^bg g<$;!1  rchtjdt|||||dus|tjdyt|}tjd|t |||||rB|j j ||||tjt||||y y) aL Add a model to the list of models being cooled down for that minute, if it exceeds the allowed fails / minute or the exception is not one that should be immediately retried (e.g. 401) Returns: - True if the deployment should be put in cooldown - False if the deployment should not be put in cooldown z"checks 'should_run_cooldown_logic'rvFz(should_run_cooldown_logic returned Falsez%Attempting to add %s to cooldown list)r2r3r\rGrx)r[rGr\rN)r2r{r\rNT) rrqrwrrcooldown_cacheadd_deployment_to_cooldownasyncio create_taskr )r2rGr\r3rirxrs r_set_cooldown_deploymentsrs& DE #$;!-1-      ##$NO">?O"P GT" 7)-3  ..II11* J   *(?(!1.    rparent_otel_spanc@K|j}|jj||d{}g}|Ct|tr3t |dkDr%t|dt r|Dcgc]}|d }}tjd||S7ccc}ww)= Async implementation of '_get_cooldown_deployments'  model_idsrNrretrieve cooldown models: %s) get_model_idsrasync_get_active_cooldownsr;listrctuplerrqr2rrcooldown_modelscached_value_deployment_idscvs r_async_get_cooldown_deploymentsrs/<<>I#:#I#I#d#d)$e$O #%#  -  1 $ q)5 17F&Gr!u&G#&G >P &&'Hs!1BB=B1 B=BBcK|j}|jj||d{}tjd||S7w)rrNr)rrrrrq)r2rrrs r/_async_get_cooldown_deployments_with_debug_infors[/<<>I#:#I#I#d#d.>$e$O >P  s1AAAc|j}|jj||}g}|Ct|tr3t |dkDr%t|dt r|Dcgc]}|d }}|Scc}w)zB Get the list of models being cooled down for this minute rr)rrget_active_cooldownsr;rrcrrs r_get_cooldown_deploymentsrs/<<>I4CCXX.>YO#%#  -  1 $ q)5 17F&Gr!u&G#&G &&'Hs' A7rPrQrRc(|j|}||n||n |j}||n|jxst}|r|d|n|} |jj | xsd} | dz} | |kDry|jj | | |y) aO Check if fails are within the allowed limit and update the number of fails. When *allowed_fails_override* / *cooldown_time_override* are supplied they take precedence over the router-level values (used by deployment-level overrides). When *cache_key_suffix* is supplied the fail counter is keyed as ``{deployment}:{cache_key_suffix}`` so that different exception types are tracked independently per deployment. Returns: - True if fails exceed the allowed limit (should cooldown) - False if fails are within the allowed limit (should not cooldown) r:rOrrT)r,r0ttlF)get_allowed_fails_from_policyr9rNr failed_calls get_cache set_cache) r2r3rGrPrQrRrVr9rN cache_key current_fails updated_failss rrUrU#s,(?'\'\$(]( " - )4 &(66  " - %33T7T >N*Q'7&89S]I2??IIiIX]\]M(1,M}$,,669M_l6m rcZ|jy|jtjk7ryy)z Check if Router.allowed_fails is set or is Non-default Value Returns: - True if Router.allowed_fails is set or is Non-default Value - False if Router.allowed_fails is None or is Default Value FT)r9rrzs rr~r~Ws-,,4,,0E0EE rct|tr t|}|S|S#t$rt j d|d}Y|SwxYw)NzDUnable to cast exception status to int %s. Defaulting to status=500.r`)r;rbrdrerrq)r\s rrrhs_"C( #"#34     # ! ' 'VXh  #    #s !!AA)N)NNN)E__doc__rrrcollections.abcrtypesrtypingrrrrlitellm._loggingrlitellm.constantsr r r r 'litellm.router_utils.cooldown_callbacksr )router_callbacks.track_deployment_metricsrropentelemetry.tracer_Spanlitellm.routerr_Router LitellmRouterr__annotations__ BaseExceptionrrr ContentPolicyViolationErrorBadRequestErrorAuthenticationErrorTimeoutRateLimitErrorInternalServerErrorServiceUnavailableErrorBadGatewayError NotFoundErrorr*rrSrbrdfloatr1rBrerFrWrZrhrwrrrrrrrUr~rrrrs #",,2 S 10M 3;DM D-U#UU B- BD BP 0DPP ((*ST ;<   "CD __01 9:   "CD $$&KL ;< 78 A%eD#I&6&; <=  WS#X.5 C C%KRVDV * 738 t #S4Z /0.  CH  $   4Z 0*00"0S!D( 0 Tz 0 '+ 0 0fZ*Zd Z"Z Z<!% 2*22Ci2: 2  2t&* 8*8d 8Ci8 8 dl 8  8@)- ]*]]Ci] ] : ]  ]H"%)(, >*>>Ci>d > dl > : > >B'*'Tk' #Y'4 * Tk  %[  '}'X\_cXc'hlmphq'>*.+/#' 1*111 $J 1 "DL 1 Dj 1 1h* " 39  r