uPj_.UddlZddlZddlmZddlmZddlmZddlm Z m Z m Z ddl Z ddl mZddlmZddlmZdd lmZmZdd lmZdd lmZmZmZmZdd lmZdd lm Z e r ddl!m"Z#e#Z$ne Z$e%dZ&e e'd<de$dee(e fde)ddfdZ*de+de(dzfdZ,edGddZ-de(de(de.fdZ/de0e de(de1e0e(dze2dzffd Z3d!Z4e e'd"<e%hd#Z5e e'd$<e%d%hZ6e e'd&<d'e(ee(e+fzde(dzfd(Z7de$d'e(ee(e+fzd)e(dee(e+fde.f d*Z8dee(e+fde.fd+Z9dee(e+fde.fd,Z:d-d.d/e1e de$d0e0e(d)e(d1e)d2e2d3e2d4e.de fd5Z;d)e(dede0e dzde.fd8Z?de0e(e0eB^^ ! ' 'O    ::+ ,1MN^1_cf1f ! ' 'L  ,39>TVZ+[  ! ' '(t u "0!>!>-!>!P * ##L1?3F3FGW3X^m  $+#=#=#U#U#k#k($l$  MM @ @ EVDa .? @gk  #.3F!3K #.?a3G #11  9$2'  "$2-($-   ##$SUbc `##$[]^__`s0 F(AF()4F($F(C$F(( G1G  Gfallback_targetcDt|tr|St|tsy|jd}t |dk(rt|tr|St j |dt}tj|jjS)a Identity of one fallback attempt, so the same attempt is never made twice per request. A bare model group name and a `{"model": name}` entry describe the same attempt. An entry carrying anything else describes a different one and keeps its own identity: a client-side fallback list overrides request params such as `messages`, and the router re-targets the group that just failed by attaching `_target_order` or `_excluded_deployment_ids` to select a different set of deployments inside it. The payload is hashed rather than kept, so a large `messages` override does not make the request hold a second copy of itself. Returns None for a shape with no usable identity, which is never skipped. Nmodel)rBT) sort_keysdefault) isinstancestrdictr/tuplejsondumpshashlibsha256encode hexdigest)r@rB serializeds r>fallback_attempt_keyrPs/3' ot ,"&&w/E _+ 5#0F  ?dCPJ >>*++- . 8 8 ::T)slotscPeZdZUdZeZeeed<dedefdZ deddfdZ y)AttemptedFallbackTargetsa The fallback attempts a single request has already made. One instance is created on the first fallback hop and shared by reference for the rest of the walk, so an attempt made in one branch is not repeated in a sibling branch. Without it the walk enumerates paths rather than attempts: a fallback graph containing a cycle retries one deterministic failure once per path through the cycle, and a client-side fallback list is re-walked at every level of the recursion. keysr%rc||jvSN)rUselfr%s r> __contains__z%AttemptedFallbackTargets.__contains__sdiirQNc@|jt|fz|_yrW)rU frozensetrXs r>recordzAttemptedFallbackTargets.recordsII 3& 11 rQ) __name__ __module__ __qualname____doc__r\rUrF__annotations__boolrZr]rQr>rTrTs?%;D)C.&   2#2$2rQrT model_group fallback_keyctjD]R}t|tr |j}n|}|j |ds7|j |dd}||k(sRyy)z Handles wildcard routing scenario where fallbacks set like: [{"gpt-3.5-turbo": ["claude-3-haiku"]}] but model_group is like: "openai/gpt-3.5-turbo" Returns: - True if the stripped model group == fallback_key /rTF)r1 provider_listrErvalue startswithreplace)rerfprovider _providerstripped_model_groups r>_check_stripped_model_grouprpsn)) h % I I  ! !YKq/ 2#.#6#6)A#K #|3 rQ fallbackscd}d}d}t|D]\}}t|trt|j d|k(r||}nt |t|j dr |t|j d}t|j ddk(s|}t|t s|g}|||}||fS|||d}||fS)a) Returns: - fallback_model_group: List[str] of fallback model groups. example: ["gpt-4", "gpt-3.5-turbo"] - generic_fallback_idx: int of the index of the generic fallback in the fallbacks list. Checks: - exact match - stripped model group match - generic fallback Nr)rerf*) enumeraterErGlistrUrprF)rqregeneric_fallback_idxstripped_model_fallbackfallback_model_groupidxitems r>get_fallback_model_groupr{s(,04-1y) * T dD !DIIK #{2'+K'8$,'d499;6G6J+/tDIIK/@/C*D'diik"1%,'*$ c "$(6  *# " .#:  !5 55" -#,-A#B3#G !5 55rQ) input_file_id training_filebatch_idfile_idfine_tuning_job_idPROVIDER_SCOPED_RESOURCE_KEYS> afile_delete afile_content_acancel_batch_acreate_batchacancel_fine_tuning_jobacreate_fine_tuning_jobaretrieve_fine_tuning_job'PROVIDER_SCOPED_RESOURCE_FUNCTION_NAMES _acreate_file'PROVIDER_SCOPED_CREATION_FUNCTION_NAMESfallback_entrycpt|tr|S|jd}t|tr|SdS)NrB)rErFr/)rtargets r> _get_fallback_target_model_grouprs6.#&"&&w/F,66$6rQoriginal_model_groupcK|j}t|}||||k(ry||||d{rytjdt |y7%w)NT)rBrequest_kwargs llm_routerzJSkipping fallback to model_group = %s: caller is not authorized to call itF)fallback_access_checkrr infor )rrrr access_checkrs r>_is_fallback_target_authorizedrsk )>>L4^DFv~;O1O v. YYYT 0  Zs0AA&ActfddD}td|DrytfdtDS)a True when a file, batch, or fine-tuning job operation names an id that only exists under one provider's credentials. Each of those ids lives in the account of the deployment that issued it. Handing it to a different model group asks a provider about an id it never issued, which costs an extra round trip that can only answer not-found. Generic calls dispatched through `Router._ageneric_api_call_with_fallbacks` carry the real handler in `original_generic_function`, so both slots are checked. Gating on the handler name keeps completion-style requests eligible for cross-group fallback even when a caller passes a stray extra body field that happens to share one of these key names. c3VK|] }tj|dd"yw)r^N)r.r/).0 function_keyrs r> z6references_provider_scoped_resource..5s+!   <(*d;!s&))original_functionrc3,K|] }|tvywrW)r)rnames r>rz6references_provider_scoped_resource..9s Y44> > YsFc3@K|]}j|ywrW)r/)rr%rs r>rz6references_provider_scoped_resource..;sH3vzz#Hs)rHallanyr)r handler_namess` r>#references_provider_scoped_resourcer(sA!!N!M Y= YY H*GH HHrQcFt|jdddtvS)a True when the request creates a resource that will live under one provider's credentials. A file uploaded for batches or fine-tuning is stored in the account of the deployment that handled it, and its id is only usable against the model group the caller named. Letting the upload fall back to a different model group silently stores the file with the wrong provider, and every later use of the returned id fails. rr^N)r.r/r)rs r> creates_provider_scoped_resourcer>s# 6::12J EIp pprQF)include_fallback_errorsargsrxr& max_fallbacksfallback_depthrc&K||k\r||} t|f} tt|jddd} t |xs t |} |jd} t | tr| n t}|j||D]}||k(r | r/t||k7r!tjdt||;t||||d{sRt|}|5||vr tjdt||j| |j||}tjd t|t |t r||d <n!t |t"r|j%||d z}t#|j| xsi}|j'd |}|j'd d|j'dd||d <|jd d|d <||d<||| <||d<||d<||d<|r||d<|j(|i|d{}tjdt+|||r t-| nd}t/|||d{|cS| 77T7#t0$rp}|} | t|fz} t3|||d{7|jd}|*|j4jddrt7|||Yd}~sd}~wwxYww)a Loops through all the fallback model groups and calls kwargs["original_function"] with the arguments and keyword arguments provided. If the call is successful, it logs the success and returns the response. If the call fails, it logs the failure and continues to the next fallback model group. If all fallback model groups fail, it raises the most recent exception. Args: litellm_router: The litellm router instance. *args: Positional arguments. fallback_model_group: List[str] of fallback model groups. example: ["gpt-4", "gpt-3.5-turbo"] original_model_group: The original model group. example: "gpt-3.5-turbo" original_exception: The original exception. **kwargs: Keyword arguments. `attempted_targets` carries the fallback attempts already made for this request, created on the first hop and shared by reference for the rest of the walk. A target already in it is skipped, so neither a fallback graph that loops back on itself nor a client-side fallback list re-walked at each level can repeat an attempt that has already failed. Identity comes from `fallback_attempt_key`, so an entry that overrides request params or re-targets the failed group with a different deployment selection stays distinct from a bare name. Returns: The response from the successful fallback model group. Raises: The most recent exception if all fallback model groups fail. rr^N) function_nameattempted_targetszYSkipping fallback to model_group = %s: request names a resource owned by model_group = %szISkipping fallback to model_group = %s, already attempted for this request)rr=z Falling back to model_group = %srBrreattempted_fallbacksrrrzSuccessful fallback b/w models.)responserfallback_errorsrrr&litellm_logging_objhas_logged_async_failureF)rrr)r rr.r/rrrErTr]rr rr rrP log_retryrFrGupdatepopasync_function_with_fallbacksr rulog_success_fallback_eventr7log_failure_fallback_eventmodel_call_detailsr?)rrxrr&rrrrrerror_from_fallbacksrmetadata_variable_namesame_model_group_onlycarried_targets attemptedmg attempt_key _hop_metadata_original_model_group_stamprr= logging_objs r>run_async_fallbackrJsyP&  -./ABDO$Ffjj)<=z4P%$Gv#N$RrS $ZZ(;rr$<<XXYefN#1V V)DD%9#5E  V   V ! ' '(QST U U V@)A9A A A A9 A  A6A1,A91A66A9cKtjjt}|D]} |j |||d{!y7#t $r }t jd|Yd}~Id}~wwxYww)a Log a failed fallback event to all registered callbacks. Uses LoggingCallbackManager.get_custom_loggers_for_type() to get deduplicated CustomLogger instances from all callback lists. Args: original_model_group (str): The original model group before fallback. kwargs (dict): kwargs for the request Note: Errors during logging are caught and reported but do not interrupt the process. rNz'Error in log_failure_fallback_event: %s)r1rrr rr7r rrs r>rrrrc|t|trt|dk(rytd|Drytd|Dr7|D]2}tj D]}||vst||try4y)ar Checks if the fallbacks list is a list of strings or a list of dictionaries. If - List[str]: e.g. ["claude-3-haiku", "openai/o-1"] - List[Dict[, Any]]: e.g. [{"model": "claude-3-haiku", "messages": [{"role": "user", "content": "Hey, how's it going?"}]}] If [{"gpt-3.5-turbo": ["claude-3-haiku"]}] then standard format. rFc3<K|]}t|tywrW)rErFrrzs r>rz6_check_non_standard_fallback_format..s 7T:dC 7Tc3<K|]}t|tywrW)rErGrs r>rz6_check_non_standard_fallback_format..s :Zd # :r)rErulenrrrb)rqrzr%s r>#_check_non_standard_fallback_formatr s 9d ;s9~QR?R 7Y 77 : : : $D-== $$;&d3i6#  $ $ rQcyrWrd)rqres r> run_non_standard_fallback_formatr#srQ)ArKrIcollections.abcr dataclassesrenumrtypingrrrr1litellm._loggingr "litellm.integrations.custom_loggerr 0litellm.litellm_core_utils.sensitive_data_maskerr /litellm.router_utils.add_retry_fallback_headersr r litellm.router_utils.batch_utilsr&litellm.router_utils.cooldown_handlersrrrr>litellm.router_utils.router_callbacks.track_deployment_metricsrlitellm.types.routerrlitellm.routerr_Router LitellmRouterr\rrbrFr7r?objectrPrTrcrprurHintr{rrrrrrrrrGrrrrrdrQr>rs #!,,2;UP 80MM'0&7e7g`!g` CH g`g` g`T;&;S4Z;2 222(S2#6S #6#6dSViZ^N^`cfj`jNjHk#6L(wuv1: 2' 2;O;L1M'M7S73;;O5O7TWZ^T^7!'#v+.. CK   &IV 0DII, qWS&[-A qd q(%*D :D!Ds)D D " D  DD"D DNV3VVbkV8V3VVbkV849t3C4 S Dc3h