tPj(UdZddlZddlmZmZmZddlmZddlm Z m Z m Z m Z m Z mZddlmZddlmZmZmZddlZddlmZdd lmZmZmZmZdd lmZdd lm Z dd l!m"Z"m#Z#m$Z$m%Z%dd l&m'Z'ddl(m)Z)m*Z*ddl+m,Z,m-Z-m.Z.m/Z/ddl0m1Z1m2Z2m3Z3ddl4m5Z5e r4ddl6m7Z8ddl0m9Z:ddl0m;Z<ddl0m=Z>ddl?m@ZAddlBmCZDe8e zZ7eAZ@eDZCe>ZEem$L % %!CM!R (G(X !)!,,rBcx|j|}|r|S|jd}t|tr|r|Sy)aResolve the deployment/model used for this batch from request data. Mirrors ``create_batch`` routing precedence: a model bound to the input file id wins over the top-level ``model``, because the batch endpoint ignores the top-level model for file-bound ids. Resolving the provider skip from the top-level model first would let a caller point ``model`` at a skip-listed provider while the file routes a rate-limited one. modelN)rYrSrTr@)rJrLfile_bound_modelr[s rC_get_batch_routing_modelz0_PROXY_BatchRateLimiter._get_batch_routing_models@#'"B"B4"H # #xx( eS !eLrB batch_modelc|syddlm}ddlm}|y |||d}|j d}t |tr|r|SdS#t$rYywxYw)a0Resolve the provider from the deployment that serves ``batch_model``. The provider is read from trusted router credentials rather than the user-supplied ``custom_llm_provider`` request field, so a caller cannot spoof a skip-listed provider to bypass batch rate limiting. Nr)get_credentials_for_model llm_router%batch input file read (rate limiting)rbmodel_idoperation_contextcustom_llm_provider)rUr`litellm.proxy.proxy_serverrbr rSrTr@)rJr^r`rb credentialsproviders rC_resolve_batch_providerz/_PROXY_BatchRateLimiter._resolve_batch_providersp  :   !:%$"I"K&//*?@%h4xKtK   s A AAuser_api_key_dictr(cB|jj||dddS)aBuild the standard key/user/team/model descriptor list a batch is charged against. Deliberately excludes the project-scoped ITPM/OTPM descriptors: those are charged per the JSONL row's own `body.model` once the file is parsed (`_create_project_io_descriptors_for_models`), not the file-bound/top-level routing model this function resolves. Charging project quotas here would let a caller bind the file to a model without a quota while rows execute against a quota-limited model. NF)rlrLrpm_limit_typetpm_limit_typemodel_has_failures)rG_create_rate_limit_descriptors)rJrlrLs rC$_create_batch_rate_limit_descriptorsz<_PROXY_BatchRateLimiter._create_batch_rate_limit_descriptorss2,,KK/$ L  rBcx|jytt|ddxstt|ddS)aTrue when the project has any per-model ITPM/OTPM quota configured. Used to stop the "skip batch input file processing" fast path from bypassing a project quota configured for a model other than the batch's file-bound/top-level routing model: the row models that actually drive execution and billing aren't known until the JSONL is parsed, so the file must be read whenever *any* model could be quota-limited, not only when the routing model itself is. Fproject_metadatamodel_itpm_limitmodel_otpm_limit) project_idboolr)rls rC _project_has_any_io_token_limitsz8_PROXY_BatchRateLimiter._project_has_any_io_token_limitssK  ' ' / ./@BTVh i q 45FHZ\no p qrBr8c 6g}g}|jD]\}}g}|jj||||D]U}|j||jd|dtk(r|j ddn|j dddW||fS)aBuild project ITPM/OTPM descriptors charged against each row's own model. One descriptor pair per distinct `body.model` found in the JSONL, each incremented only by that model's own counted usage -- never the whole-batch total -- so a quota-limited model can't hide behind an unlimited routing model, and an unrelated model's rows can't inflate a different model's counter. )rlrequested_model descriptorsrkeyr6r4r.)itemsrG9add_project_io_token_rate_limit_descriptors_from_metadataappendr$rS) rJrlr8extra_descriptorsextra_incrementsr[usagemodel_descriptors descriptors rC)_create_project_io_descriptors_for_modelszA_PROXY_BatchRateLimiter._create_project_io_descriptors_for_modelss?A:<+113 LE5;=   ) ) c c"3 %- d  0  !((4 ''$%%e,0KK#())OQ"?"YY~q9   "!"222rBhas_enqueued_scopescddlm}|j||j|ry|j ddury|j dxsg}|r=|j |j |}|r||vrtjd|y|j|| }|s8|j|s'|j|stjd yd |fS) a Skip downloading batch input files when the operator disabled batch input-file rate limiting, when the batch runs entirely on a skip-listed provider, or when there is nothing to enforce (no applicable rate limits). A skip is only honored for keys with unrestricted model access. When the key has a model allowlist, the JSONL must still be downloaded so ``_enforce_batch_file_model_access`` can validate every ``body.model`` entry, otherwise a restricted key could smuggle unauthorized models into the file via an admin-configured skip. The skip is never keyed on a specific model name. The models a batch actually runs are its JSONL ``body.model`` entries, and any model identifier the caller can influence (the top-level ``model`` or the unsigned model embedded in a ``file-...`` id) can be pointed at a skip-listed deployment while the file routes a different, rate-limited model. The provider skip is safe because the provider is read from the routing deployment's trusted credentials and the batch is constrained to run on that provider. The no-limits check also treats any project-configured ITPM/OTPM quota as an applicable limit, even when it isn't scoped to the routing model: a row can target a different, quota-limited model, and that isn't knowable without parsing the JSONL. Returns ``(should_skip, descriptors)`` where ``descriptors`` is the rate-limit descriptor list computed for the no-limits check, so the caller can reuse it for counter enforcement without recomputing. r)general_settings)FN&disable_batch_input_file_rate_limitingT)TN1skip_batch_input_file_rate_limiting_for_providersz4Skipping batch input file processing for provider=%srlrLz?Skipping batch input file processing: no rate limits configuredF) rhr*_warn_if_unsupported_model_skip_configured&_key_requires_batch_model_access_checkrSrkr]rdebugrr!_has_applicable_batch_rate_limitsry)rJrLrlrrskip_providersbatch_providerr|s rC(_should_skip_batch_input_file_processingz@_PROXY_BatchRateLimiter._should_skip_batch_input_file_processing)sH @ 778HI  6 67H I    H IT Q 0 4 45h i omo $($@$@A^A^_cAd$eN.N"B$**+acqr!!FF/G $::;G99:KL & &'h ik!!rBrcz|jry|jdrd|_tjdyy)a;Warn once that ``skip_batch_input_file_rate_limiting_for_models`` is a no-op. A per-model skip is intentionally not honored because the model a batch runs on is caller-influenced and can be pointed at a skip-listed deployment while the JSONL routes a different, rate-limited model. N.skip_batch_input_file_rate_limiting_for_modelsTzgeneral_settings.skip_batch_input_file_rate_limiting_for_models is not supported and has no effect. Use skip_batch_input_file_rate_limiting_for_providers or disable_batch_input_file_rate_limiting instead.)rIrSrwarning)rJrs rCrzB_PROXY_BatchRateLimiter._warn_if_unsupported_model_skip_configuredlsA  . .     P Q26D / ( (B  RrBc|jxsg}d|vrytjj|vry|jry|syy)zKTrue when the key may only call a subset of models (JSONL must be checked).*FT)modelsrall_proxy_modelsvalueaccess_group_ids)rlrs rCrz>_PROXY_BatchRateLimiter._key_requires_batch_model_access_check~sJ *006B &=  - - 3 3v =  - -rBentrymin_configured_otpm_limitcD|jd}t|trd|vry|jd}t|trt t j |n t i}td|jd|jd|jdfDd }|jj|}| tdt||zS|jj||zS#tttf$rY4wxYw) ahConservative per-row output-token estimate for the project OTPM reservation. Batch completion never reconciles actual usage back into the rate limiter, so this pre-call estimate is the only OTPM enforcement a batch gets. Mirrors the real-time no-``max_tokens`` floor so a row that omits an output cap can't be used to bypass OTPM the way an unbounded streaming request could. Embeddings rows are identified by the row's own ``url`` (the OpenAI batch schema puts the target route there, e.g. ``/v1/embeddings``), never by body shape: a `/v1/responses` row also carries `body.input` with no `messages`/`prompt`, so guessing from body shape alone would misclassify a token-generating Responses row as a zero-output embeddings row and let it skip the OTPM reservation entirely. url embeddingsrbodyc3$K|]}|| ywNrA.0vs rC zH_PROXY_BatchRateLimiter._estimate_entry_output_tokens..s  =  s max_tokensmax_completion_tokensmax_output_tokensN)rSrTr@rrr-validate_pythonnextrGget_output_candidate_countmaxr= TypeError ValueError OverflowErrorno_max_tokens_output_floor)rJrrrraw_bodyr explicit_capcandidate_counts rC_estimate_entry_output_tokensz5_PROXY_BatchRateLimiter._estimate_entry_output_tokenss!(YYu% c3 LC$7))F+(G, 0@@J K!"% # HH\*HH45HH01    "&!>!>!Y!YZ^!_  # 1c,/0?BB,,GGHabetttz=9  sDDDr|c|D]L}|jdxsi}|jd#|jd|jdLyy)N rate_limitrequests_per_unittokens_per_unitmax_parallel_requestsTFrS)r|rrs rCrz9_PROXY_BatchRateLimiter._has_applicable_batch_rate_limitss^& J# 5;J23?>>"34@>>"9:F rBfile_idrgcddlm}m}m}ddlm}d|i}||} | rO|C ||| d} |j t| | |d<| jd} | r| |d<|||fS|jd} t| trK| rI|G ||| d} |j t| | |d<| jd} | r| |d<||fS||fS#t$rYwxYw#t$rY||fSwxYw)a! Map proxy-facing file IDs to provider file IDs and credentials. Model-embedded IDs (``file-``) are not unified managed-file IDs; without decoding them, ``afile_content`` is called with the encoded ID and the upstream provider returns 404. r)rQr`get_original_file_idrargrcrdr[) rUrQr`rrhrbupdaterrSr rTr@) rJrrgrLrQr`rrb fetch_kwargsrVrirj request_models rC&_resolve_batch_input_file_fetch_paramsz>_PROXY_BatchRateLimiter._resolve_batch_input_file_fetch_paramssZ  : "#6/ %>g$F % ";#-!3*Q#K !''(H(UV,>L)*/DEH>F %:;(0,> >#xx0 mS )m @V 7)*&M ##$D[$QR(5 W%&??+@A:BL!67 $$w $$)%"!  $$ s%ACAC- C*)C*- C<;C< batch_usagescopes.cK|jjj|j||jd{}|xt dx\|j |||ytdx\|t_ yy7Ew)aReserve the batch's estimated tokens against the caller's enqueued-token allowance. Runs instead of the per-minute counter charge when the key or team opted in via ``batch_enqueued_token_limit`` metadata. The reservation is stashed on the request so the v3 limiter's post-call hooks can persist it (keyed by the provider batch id) and refund it when the batch reaches a terminal state. )r0rlitellm_parent_otel_spanNrA) over_limitrLr) rGbatch_enqueued_token_storereserver4parent_otel_spanr_raise_enqueued_limit_errorr r%batch_enqueued_reservation)rJrlrLrroutcomes rC_reserve_batch_enqueued_tokensz6_PROXY_BatchRateLimiter._reserve_batch_enqueued_tokens s $<<WW__++%6%G%G `   .,.00G$\g0h/.0KR+-H1 sAB BAB rc |j}td|j|jz }d|jd|j d|j d|d|jd }|jd}tt|tr|nd\}} t|td d itjtd || ) Nrz(Batch enqueued token limit exceeded for : z. Batch requires  tokens but only z" enqueued tokens remaining out of zS enqueued token limit. Tokens free up as running batches complete or are cancelled.r[rate_limit_typer0detailheaderscategoryrr[ llm_provider)scoperlimitenqueuedr}rr4rSr&rTr@rrrLITELLM_BATCH_RATE_LIMITr) rJrrLrr remainingr raw_modelresolved_modelrs rCrz3_PROXY_BatchRateLimiter._raise_enqueued_limit_error&s "''q%++ 0C0C"CD 6uyykEKK=Q)6677H Tkk]#K L   88G, 'J#Is3I( $ "$&7%BC+DD28< %   rBstatusr) limit_typer{c ddlm}jdtfdt|Dd}|r||ndddd}|j j } |j j} | | z} |j| jd} tdd } d }|d k(r@d |jd dd|jddd|jd| d|d| }n|jd ttfvr%|jddjdddnd}|r|jj|nd}|jd tk(r |xsijd|j nC|jd tk(r |xsijd|j"n |j"}d |jd dd|jddd|d| d|d| }t%|\}}t'|t)| || dt*j,t/|||)zIRaise :class:`ProxyRateLimitError` (a 429) for batch rate limit exceeded.r)datetimedescriptor_valuec3K|]B\}}|jdjdk(r|jdk(r|Dyw)r}descriptor_keyNrr)ridrstatus_descriptor_values rCrzB_PROXY_BatchRateLimiter._raise_rate_limit_error..VsM Aq55<6::.>#??,4gJa8a sAA N)r}rrz%Y-%m-%d %H:%M:%S UTClimit_remaining current_limitr/zBatch rate limit exceeded for r}unknownrrz. Batch contains z requests but only z requests remaining out of z RPM limit. Limit resets at: :r6r4rz tokens remaining out of z TPM limit. Limit resets at: )z retry-afterrreset_atr)rrSr enumeratenow timestamprG window_size fromtimestampstrftimerr5r#r$splitr8r6r4r&rr@rrr)rJrr|rrr{rdescriptor_indexrrr reset_timereset_time_formattedremaining_displayrrdescriptor_model model_usagebatch_token_countrrrs ` @rC_raise_rate_limit_errorz/_PROXY_BatchRateLimiter._raise_rate_limit_errorAs &*04F)G"& %k2  # .9K( )bSUei>j \\^--/!::FF +- &.&<&>%(-HJe,ffw+11#q9"=  Wg!>%(,GG"''9R9RS>>%(,GG"'R,,^[=U=UV -- 1y1Q0RRTU_UcUcdkmvUwTxy""3!44EFWEXY')$$8#9; (K?'[$ !";/#-0 ,DD2:> %  rBcK||j||}|Dcgc]}|j|jd}}|j||j\}}g||}g||}|j j |||jd{} | ddk(rB|r|jdnd} | d D]$} | d dk(s |j| ||| d | &yycc}w7Tw) a Atomically check + increment rate-limit counters by the batch amounts. Raises HTTPException if any descriptor would exceed its limit; in that case no counter is modified. Backed by `atomic_check_and_increment_by_n` which uses a Redis Lua script when available (multi-process atomic) and falls back to a per-process asyncio.Lock + in-memory operation. ``descriptors`` may be passed in by the pre-call hook to reuse the list already computed when deciding whether to skip file processing. It never contains project ITPM/OTPM descriptors (those are model-specific and only knowable once ``batch_usage.per_model_usage`` is populated by parsing the JSONL), so this always builds and appends them here. Nrr.)rlr8)r| incrementsr overall_code OVER_LIMITr[statusescoder)r{) rrr5r4rr8rGatomic_check_and_increment_by_nrrSr) rJrlrLrr|_drproject_io_descriptorsproject_io_incrementsrate_limit_responser{rs rC#_check_and_increment_batch_countersz;_PROXY_BatchRateLimiter._check_and_increment_batch_counterssU*  CC"3DK" . (55%22 . . 9=8f8f/'779g9 5 5> =&<= :z:$9: +/+H+H+h+h#!.??,i, &  ~ ., >:>TXXg%6DO-j9 &>\100##01(7 1  ?+. & s"C, C%AC,C*1C,#C,)openaiazure vertex_aicKtd|xsdD}|/t|ddx}r td|jDnd}tg||d} dd lm} m} | |} | r| | ng} | r||j|| d{}n;|j|||xsi \}}tjd||d |d{}t|d d}t|tstd |dt|t!}i}d}d}d}t#|D]}|dz } t%j&|}t|t0r$|j3dxsij3dnd}|r|j5|t|t0r|j7|| }n|j,j/| }||z } t9|}||z }|s|j;|dddd}|dxx|z cc<|dxx|z cc<|dxxdz cc<| |j=||| xsdd{t?||||S77#t($r4t+|}|j,j/| }||z }||z }YpwxYw#t($rt+|}YwxYw7s#t@$rc}|jBdk(r"tEjFd||jHtEjJd||jB|jHd}~wt($r}tEjJd||d}~wwxYww)a Count number of requests and tokens in a batch input file. Args: file_id: The file ID to read custom_llm_provider: The custom LLM provider to use for token encoding user_api_key_dict: User authentication information for file access (required for managed files) descriptors: Rate limit descriptors already computed for this batch, so the configured project OTPM limit can scale the no-``max_tokens`` output floor Returns: BatchFileUsage with total_tokens, output_tokens, request_count, and per_model_usage (each row's own totals, keyed by its `body.model`) c3K|]O}|jdtk(r5|jd}||jdnd}| t|Qyw)r}rNr)rSr$r=)rrrrs rCrzA_PROXY_BatchRateLimiter.count_input_file_usage..sW. uuU|:: uu\2 ;E;Qjnn%67W[} F. sAArANrtrvc32K|]}t|ywr)r=rs rCrzA_PROXY_BatchRateLimiter.count_input_file_usage..sBQ#a&Bs)defaultr)rPrR)rrl)rrgrLcontentz/Expected bytes content from file retrieval for z, got rrr[)r4r6r5r4r6r5)rlrrX)r4r5r6r8zABatch rejected: caller not authorized for a model named in %s: %sz5Batch input file rejected for %s: status=%s detail=%sz*Error counting input file usage for %s: %s)&tuplervaluesminrUrPrR_fetch_managed_file_contentrlitellm afile_contentgetattrrTbytesrtypesetrjsonloads ExceptionrrGrr?rSaddrr setdefault _enforce_batch_file_model_accessr3r status_coderrrerror)rJrrgrlrLr|descriptor_otpm_limitsproject_otpm_limit_mapproject_otpm_limitsrrPrRis_managed_filerX file_contentprovider_file_idrfile_content_bytesrr8r4r6r5raw_linerentry_total_tokensentry_output_tokensr[res rCcount_input_file_usagez._PROXY_BatchRateLimiter.count_input_file_usages",).. !'R. ) !,*L%'9;M+& B"8"?"?"AB B ,//^1G/^J]/^hl+m!C  &H%PO]l(G(Xqs #4#@%)%E%E#&7&F&  261\1\#(;2]2. , &-%:%:&,&7&#&  )0 i(N 0%8 EgYfUYZlUmTno""%FACOLMM34FG% 6"   JJx0EOYY^`dNeUYYv%6%<"$A$A'$JkoJJu%eT**.*L*LUTm*n'*.*G*G*b*b1+'!44 P)}|j,t.j0k7r|||| | d{7sYd}~d}~wwxYw77#t"$rt$$r}t#ddd|d|i d}~wwxYww)aReject the batch if the caller is not authorized for the upload target. For managed files, ``target_model_names`` (from the unified file id) is the proxy alias the file was uploaded for and is checked directly. Otherwise the ``body.model`` values collected from the JSONL (``models``) are checked. Reuses standard auth helpers so the same model access rules the proxy enforces on `/chat/completions` apply here. r)_check_team_member_model_access_key_access_group_grants_modelcan_key_call_modelcan_team_access_modelget_team_object)rb prisma_clientproxy_logging_objuser_api_key_cacheN)team_idr;r=rr<rr(zNBatch input file model access could not be validated against the current team.)r'r)r[ team_objectrbteam_model_aliases)r[ valid_tokenr?rb)r[r?rArbr;r=r<)r[llm_model_listrArbzOBatch input file references a model the caller is not authorized to use: model=z , reason=)litellm.proxy.auth.auth_checksr6r7r8r9r:rhrbr;r<r=rall_team_modelsrrr>rr r# model_listr@rrrteam_model_access_denied)rJrlrrXr6r7r8r9r:rbr;r<r=r?r3rBr[model_to_check team_denials rCr&z8_PROXY_BatchRateLimiter._enforce_batch_file_model_accesss&    po 'F   - - 3 38I8P8P8VTV W!))5) $3-55"/'9%6%G%G&7 % ":D9O 5 5UY0 E"N. *"3"0(3'1/@/S/S  :,$/$5#-&3+=*;-,'5$5#- ?0 %!  # #"r    * "&++/W/WW!%C"0(9(3'1 &   "   " !  # #<FFFcZKddlm} ddlm}m}| t d|jd}| t d t||s t d | t d |j||j| d{}|S#t $r}t d|dd}~wwxYw7&w) a Fetch file content from managed files hook. This is needed for managed files because they require proper user context to verify file ownership and access permissions. Args: file_id: The managed file ID (base64 encoded) user_api_key_dict: User authentication information Returns: HttpxBinaryResponseContent with the file content r)BaseFileEndpoints)rbr<z)Cannot import proxy_server dependencies: z7. Managed files require proxy_server to be initialized.NzBproxy_logging_obj not available. Cannot access managed files hook. managed_filesz9Managed files hook not found. Cannot access managed file.z7Managed files hook is not a BaseFileEndpoints instance.z6llm_router not available. Cannot access managed files.)rrrb) *litellm.llms.base_llm.files.transformationrJrhrbr< ImportErrorrget_proxy_hookrTrr) rJrrlrJrbr<r3managed_files_objr-s rCrz3_PROXY_BatchRateLimiter._fetch_managed_file_contents$ Q  P  $ab b#4#C#CO#T  $XY Y+->?VW W  UV V%6$C$C%6%G%G!%D%  7 ;A3>uv  * s4B+B A0B+B)B+ B&B!!B&&B+cache call_typecK|dk7rtjd||Stjd |jd}|stjd|St|}|j ||t |\}}|r|S|jdd} tjd ||j || ||| d {} tjd | j| j| j|d <| j|d<|r4|j||| |d {tjd|S|j||| |d {tjd|S77Q7#t$rt$r$} tjd| d|cYd } ~ Sd } ~ wwxYww)aJ Pre-call hook for batch operations. Only handles batch creation (acreate_batch): - Reads input file - Counts tokens and requests - Reserves rate limit capacity via parallel_request_limiter Args: user_api_key_dict: User authentication information cache: Cache instance (not used directly) data: Request data call_type: Type of call being made Returns: Modified data dict or None Raises: HTTPException: 429 if rate limit would be exceeded acreate_batchzOBatch rate limiter: Not handling batch creation rate limiting for call type: %sz9Batch rate limiter: Handling batch creation rate limitingrOz9No input_file_id in batch request, skipping rate limiting)rLrlrrgrz)Counting tokens from batch input file: %s)rrgrlrLr|Nz1Batch input file usage - Tokens: %s, Requests: %s_batch_token_count_batch_request_count)rlrLrrz*Batch enqueued-token reservation succeeded)rlrLrr|z3Batch rate limit check passed, counters incrementedz Error in batch rate limiting: %sT)exc_info)rrrSr"rrxr4r4r5rr r r#r() rJrlrPrLrQrOenqueued_scopes should_skipbatch_rate_limit_descriptorsrgrr3s rCasync_pre_call_hookz+_PROXY_BatchRateLimiter.async_pre_call_hook&s8  ' & &acl K""#^_? #'88O#TXYhTi9f9 5K5 *.2G)R  ! & &'RTa b'+'B'B%$7"38 (C("K ! & &C[E]E]_j_x_x  *5)A)AD% &+6+D+DD' (99&7 +* : %**+WX ::"3'8 ;   ! & &'\ ]KI""     & &'I1W[ \K s3G)FG .FGAFFA'F8F 9FGF,F -FGF F FG F?9G:G?GG)Fr)rNNN)NN)/r9r:r;r<r+ParallelRequestLimiterrKr?r@rYr]rkrlistrr staticmethodrxryrr=rr1rrrrobjectrrrrr3r!rrr rrr r rr4rr&rr#rZrArBrCrErEtsg404#94&t>TcDj&L3:L#*L< )   # $  ,qNqtqq 3) 3!gc3h&7!78 3 t)*D1A,BB C 3L%* A"A"*A"" A" tT/0477 8 A"F4TX$ )    2usF{#2u$':2u 2uh /0    :%:%!:% :% sDcN" # :%xS)Sc6k"S$ S -s23 S  S4 / c6k" $    B'+ V !V /0V $ V  V t V  V z;? :)::$ : /047 :  :~HP37 >B qq%%CDq*D0 q Tk q 34t; q ql(,/3 h)h $h!I, h  hT22*2 2hc)cc c  c S4 $ & crBrE)Or<r!collections.abcrrrtypesrtypingrrr r r r fastapir pydanticrrrrlitellm._loggingrlitellm.batches.batch_utilsrrrrlitellm.exceptionsr"litellm.integrations.custom_loggerrlitellm.proxy._typesrrrrlitellm.proxy.auth.auth_utilsr1litellm.proxy.common_utils.proxy_rate_limit_errorrr)litellm.proxy.hooks.batch_enqueued_tokensrr r!r"/litellm.proxy.hooks.parallel_request_limiter_v3r#r$r%&litellm.proxy.hooks.rate_limiter_utilsr&opentelemetry.tracer'_Spanr(_RateLimitDescriptorr)_RateLimitStatusr*_ParallelRequestLimiterlitellm.proxy.utilsr+_InternalUsageCachelitellm.routerr,_Routerr[r?r@r^r-r>r=r1r3rErArBrCrws2& 77"JJ!221 6; M  W1N0 3;D, F4&O. D F 38nOsCx.)c6k):;U;"7+?#@##EF)FY&UlUrB