uPjpUddlZddlmZmZmZddlmZddlmZddl m Z ddl m Z m Z mZddlZddlmZddlmZdd lmZdd lmZdd lmZmZmZdd lmZed d GddZ dGdee dedde!dzdedzdef dZ" dHdededde!dzde dzdedzdef dZ#Gdde Z$ed d GddZ%dee ded de!dzdedzdee%e$zf d!Z&d"ee!e fded de!dzdedzde%dzf d#Z'd"ee!e fded de!dzdedzde%f d$Z(d%ee!e fd&eded de!dzd'e!dzdedzde)fd(Z* dGdee ded de!dzdedzdef d)Z+ dId*ee de!dzdefd+Z,d,e!de!fd-Z- dJd.e!dedde dzde.fd/Z/ dJdededde dzde.fd0Z0dededde dzde1fd1Z2de dzde fd2Z3d3e.dee fd4Z4d3e.dee.fd5Z5d3e.dee fd6Z6d7e.de dzfd8Z7d9Z8e e9d:<d;e.de1fd<Z: dId"e de!dzde1fd=Z;d>e!d?e de1fd@Z< dKd%ee!e fde!defdAZ=dBee!e fdee!e ffdCZ> dKdDee!e fde!dee!e ffdEZ? dKdDee!e fde!de@fdFZAy)LN)IterableIteratorMapping) dataclass)replace)Enum)AnyFinalLiteral)verbose_loggerAWS_CREDENTIAL_KWARGS_KEYS)parse_prompt_tokens_details)Batch) CallTypes ModelInfoUsage) token_counterT)frozenslotscJeZdZUdZeed<eed<eeed<e ed<e ed<y)BatchCostUsageResultzKAggregate cost, usage, and per-line pass/fail counts for a completed batch.costusagemodelssuccessful_requestsfailed_requestsN) __name__ __module__ __qualname____doc__float__annotations__rliststrint`/home/oem/CyberBrain26-unified/.venv/lib/python3.12/site-packages/litellm/batches/batch_utils.pyrrs%U K L Ir(rfile_content_dictionarycustom_llm_provider)openaiazure vertex_ai hosted_vllm anthropic model_name model_inforeturncpK|dk(r|rttddr t||St||||Sw)aO Calculate the cost and usage of a batch. Args: model_info: Optional deployment-level model info with custom batch pricing. Threaded through to batch_cost_calculator so that deployment-specific pricing (e.g. input_cost_per_token_batches) is used instead of the global cost map. r.*disable_vertex_batch_output_transformationFentriesr+r1r2)getattrlitellm(calculate_vertex_ai_batch_cost_and_usage"_aggregate_batch_cost_usage_models)r*r+r1r2s r)calculate_batch_cost_and_usager<sH {*  GI5 Q78OQ[\\ -'/  s46batchlitellm_paramsc K|j0tdtdddgdt|||d{St |||d{}t|||d{}|dk(r(|r&t t d d rtt||ntt|||| }|s|St||j|z S777kw) a]Fetch a completed batch's output file and aggregate its cost, usage, and models in a single pass over the JSONL lines, so the parsed file content is never materialized in memory. Args: batch: The batch object custom_llm_provider: The LLM provider model_name: Optional model name litellm_params: Optional litellm parameters containing credentials (api_key, api_base, etc.) model_info: Optional deployment-level model info with custom pricing, threaded through so a deployment's configured rates win over the global cost map. Nr prompt_tokenscompletion_tokens total_tokensr+r>rrrrr)r>r.r5Fr6)r) output_file_idrr count_error_file_failed_requests _fetch_batch_output_file_contentr8r9r:_get_file_content_as_dictionaryr;_iter_batch_output_entriesdataclasses_replacer)r=r+r1r>r2 file_contenterror_file_failed_requestsoutput_file_results r)_handle_completed_batchrP:s8 ##a11M !"B+>~#   :%ATesttL.N #6~/) ; .!MuU 11PQ]1^`jk 0.|< 3!!   &!! ,>,N,NQk,k 7 u)s43CCCCC&C'A(CCCceZdZdZdZdZy) _LineOutcomez3A batch output line that yielded no billable stats.provider_failed uncostableN)rrr r!PROVIDER_FAILED UNCOSTABLEr'r(r)rRrR|s='OJr(rRcdeZdZUeed<eed<eed<eed<eed<eed<eed<edzed <y) _BatchOutputLineStatsrrBrCrDcache_read_tokenscache_creation_tokensreasoning_tokensNmodel)rrr r"r#r&r%r'r(r)rXrXs5 K :r(rXr7)r,r-r.r/r0bedrockc#K|D]E}t||stj"t||||}||ntjGyw)aClassify every output line in a single pass, so counting failures never needs a second read of a potentially huge output file. A line the provider reported as failed yields ``PROVIDER_FAILED``; a successful line litellm could not price yields ``UNCOSTABLE`` and still counts as a successful request billed at $0, so the counts stay reconcilable with the provider's own ``request_counts``.N)_batch_response_was_successfulrRrU_safe_output_line_statsrV)r7r+r1r2entrystatss r)_classify_output_line_statsrcsXF-e5HI.. . '/BJPZ[(el.E.EE FsA Aract|tr|jdnd} t||||S#t$r*}t j d|t|Yd}~yd}~wwxYw)zReturn the stats for one provider-successful batch output line, or None when it cannot be costed, so a single bad line never aborts the whole batch's cost accounting. custom_idNz}batch output line could not be costed, so it is billed at $0 and the rest of the batch is still billed. custom_id=%s error=%s) isinstancedictget_compute_output_line_stats Exceptionr warningr%)rar+r1r2rees r)r`r`sg2N$((1I)3Is)C IW[N % ? ? ' 3!)!  ))11''();<,-DEAS,==Y]cbc r(rnrrocddlm}|4|dvr0tj||tj j S|dk(r|r|n |xs|xsd}|||||\}} || zS)Nrbatch_cost_calculatorr0r])completion_responser+ call_typer])rr\r+r2)litellm.cost_calculatorrzr9completion_costraretrieve_batchvalue) rnrr+r1ror2rz cost_model prompt_costrs r)rtrts>19QQ&& - 3//55  *Y6: >Km]gKmkm$9/ $ K  ((r(c tt||||}td|D}td|D}t||z }dtd|Dfdtd|DffD cic] \}} | dkDr|| } }} t dtd|Dtd |Dtd |Dtd |Dd | } |r|gn&|D cgc]} | j s| j c} } td |Dd}t jd|| | ||t|| | ||Scc} }wcc} w)zAggregate cost, usage, models, and pass/fail counts from batch output entries in a single pass, holding one small stats record per line instead of the parsed file.c3BK|]}t|ts|ywN)rfrX.0results r) z5_aggregate_batch_cost_usage_models..sl*VUjBkflsc3FK|]}|tjusdywN)rRrUrs r)rz5_aggregate_batch_cost_usage_models..s gv,JfJf@f gs!!cache_read_input_tokensc34K|]}|jywr)rYrrbs r)rz5_aggregate_batch_cost_usage_models..s+\E,C,C+\cache_creation_input_tokensc34K|]}|jywr)rZrs r)rz5_aggregate_batch_cost_usage_models..s/dPU0K0K/drrc34K|]}|jywr)rDrs r)rz5_aggregate_batch_cost_usage_models.. sD++Drc34K|]}|jywr)rBrs r)rz5_aggregate_batch_cost_usage_models.. sF%%--Frc34K|]}|jywr)rCrs r)rz5_aggregate_batch_cost_usage_models.. sN%e55Nrc34K|]}|jywr)r[rs r)rz5_aggregate_batch_cost_usage_models.. sLU33Lr)rDrBrCr[c34K|]}|jywr)rrs r)rz5_aggregate_batch_cost_usage_models..s@EUZZ@rr@zJbatch output aggregate: cost=%s usage=%s models=%s successful=%d failed=%drFr') tuplercsumlenrr\r debugr)r7r+r1r2 all_results line_statsrrkeytokenscache_token_params batch_usagerb batch_models total_costs r)r;r;s|:7DWYceopqKl;llJ g ggO!$[!1O!C '+\Q[+\(\ ] *C/dYc/d,d e ! C A: V !!DDDF:FFN:NNLLL   K+5:,T^:n5bgbmbm5;;:nL@Z@#FJT  /'  3!;os5E#E 5E vertex_ai_batch_responsesc lddlm}d}d}d}d}d}d}|xsd} |D]} | jd} | |dz }|dz }| jdi} | jd dxsd} | jd dxsd}| jd dxs| |z}t| || } ||| d \}}|||zz }|| z }||z }||z }t jd||||||t|t|||| g||S#t$r)}t j dt|Yd}~vd}~wwxYw)a Calculate both cost and usage from raw Vertex AI batch responses. Used only when ``litellm.disable_vertex_batch_output_transformation = True``. In that case the GCS predictions.jsonl is returned as-is, with each line in the native Vertex format: {"request": ..., "response": {"candidates": [...], "usageMetadata": {...}}} usageMetadata contains promptTokenCount, candidatesTokenCount, totalTokenCount. A row with no ``response`` is counted as failed - the same signal already used to skip it from cost/usage aggregation, since Vertex batch prediction output doesn't establish a distinct error shape in this (non-default) path. rryr@zgemini-2.0-flash-001responseNr usageMetadatapromptTokenCountcandidatesTokenCounttotalTokenCountrAr.)rr\r+z3vertex_ai batch cost calculation error for line: %sz[vertex_ai batch cost: cost=%s, prompt=%d, completion=%d, total=%d, successful=%d, failed=%d)rDrBrCrF) rrzrhrrjr rr%infor)rr1rzrrDrBrCrractual_model_namerrnusage_metadata_prompt _completion_total line_usagep_costc_costrls r)r:r:#s&>JLMO)C-C- Z0  q O q &**?B? $$%7;@q$(()?CHq ##$5q9Tg >S!)  `2 '$/NFF &6/ )J  [( =@e %'/ ""/'  # `  !VX[\]X^ _ _ `s"D D3 D..D3rGcddlm}m}||}|s||S |jddjdd}t jd||S#tt f$r#}t jd|||cYd}~Sd}~wwxYw) z Resolve the file id the provider actually knows: unified ids yield their embedded llm_output_file_id, model-encoded ids decode to the raw provider id, raw ids pass through. r)"_is_base64_encoded_unified_file_idget_original_file_idzllm_output_file_id,r;zHFailed to extract LLM output file ID from unified file ID: %s, error: %sNz5Extracted LLM output file ID from unified file ID: %s) 1litellm.proxy.openai_files_endpoints.common_utilsrrsplit IndexErrorAttributeErrorr errorr)rGrrunified_file_id extractedrls r)_provider_output_file_idrws  @OO #N33*001FGJPPQTUVWX PR[\   ' V    s&AB )BB B file_idcKddlm}t||d}t|}|j ||di|d{}|j S7w)a Fetch a batch's output or error file and return its raw JSONL bytes. Args: file_id: The provider or unified (litellm-managed) file id to fetch custom_llm_provider: The LLM provider litellm_params: Optional litellm parameters containing credentials (api_key, api_base, etc.) Required for Azure and other providers that need authentication r) afile_content)rr+Nr')litellm.files.mainrr _extract_file_access_credentialsupdatecontent)rr+r>rfile_content_kwargs credentials _file_contents r)!_fetch_batch_managed_file_contentrs]1,G42" :.IK{+!.!E1D!EEM   Fss r)rIrIsF #NOO2 2EVd  s 3<:<cK|jy t|j||d{}t dt |DS7#t$r+}tjd|j|Yd}~yd}~wwxYww)asCount failed requests reported only in the batch's separate error file. OpenAI-shaped batch providers write successful lines to ``output_file_id`` and per-request failures (e.g. a rejected param) to a distinct ``error_file_id`` - they never appear in the output file at all, so counting failures from the output file alone silently undercounts them. NrrEz'Failed to fetch batch error file %s: %sc3 K|]}dywrr')r_s r)rz3count_error_file_failed_requests..sFQqFs ) error_file_idrrjr rr_iter_batch_input_lines)r=r+r>error_file_contentrls r)rHrHs "#D   5HYg$   F12DEF FF   FH[H[]^_s=BAA AB A B!A=8B=BBcdi}|r+ddddddddd d d d d dgt}|D]}||vs||||<|S)a Extract credentials from litellm_params for file access operations. This method extracts relevant authentication and configuration parameters needed for accessing files across different providers (Azure, Vertex AI, etc.). Args: litellm_params: Dictionary containing litellm parameters with credentials Returns: Dictionary containing only the credentials needed for file access api_keyapi_base api_version organizationazure_ad_tokenazure_ad_token_providervertex_projectvertex_locationvertex_credentialsgcs_bucket_name bucket_nametimeout max_retries#_litellm_internal_model_credentialsr )r>rcredential_keysrs r)rrsyK      %       1" (" "# 7Cn$#1##6 C  7 r(rMc*tt|S)zm Get the file content as a list of dictionaries from JSON Lines format, skipping malformed lines )r$rK)rMs r)rJrJs *<8 99r(c#Kdt|td}}}||krF|j||}|dk(r||d|}}n ||||dz}}|j}|r|||krEyyw)z Yield non-empty JSONL lines (unparsed) one at a time, so a caller can parse each row in its own try/except and a single malformed line cannot abort the whole pass. Peak memory stays bounded for large batch files. r Nr)rordfindstrip)rMstartlengthnewlineidxchunklines r)rrs~ L 13t976E &./ "9'/5E'c2C!G5E{{} J &.s A"A'%A'c#PKt|D]}t|}||yw)a Yield parsed batch output JSONL entries one at a time without materializing the whole file as a list, so peak memory stays bounded. A malformed or non-object line is skipped with a warning so one bad line never aborts the whole batch's cost accounting. N)r_parse_batch_output_line)rMrras r)rKrK s1( 5(.  Ks&&rc tj|}t |tr|Stjdt|jy#t$r)}tjdt |Yd}~yd}~wwxYw)Nz(skipping malformed batch output line: %sz0skipping non-object batch output line of type %s) jsonloadsrr rkr%rfrgtyper)rparsedrls r)rr-so 4(&$ MtTZ|OdOde  I3q6RsA BBB%_BATCH_TOKEN_ESTIMATE_BYTES_PER_TOKENraw_linec:tdt|tzS)zConservative token estimate for a batch row the token counter cannot measure (or that cannot be parsed). Keeps the batch token total non-zero so a crafted row cannot evade the TPM limit, without hard-rejecting a legitimate batch.r)maxrr)rs r)_estimate_batch_entry_tokensr>s q#h-#HH IIr(c|jdixsi}|jd|xsd}|jd}|r t||S|jd}|r t||S|jd}|r t||Sy ) zHToken-count a single batch input entry's body (chat / text / embedding).bodyr\r~messages)r\rprompt)r\rinputr)rhr_count_prompt_or_input_tokens)rar1rr\rr input_datas r)_count_entry_tokensrEs ))FB'-2D88GZ%526Ehhz*H58<<HHX&F ,5GG)J,5 KK r(r\rc^t|tr t|St|trd}|D]v}t|tr|t|z }$t|tr|dz }:t|tsK|t d|Dz }|t fd|Dz }x|Sy)aToken-count a ``prompt`` / ``input`` field that the OpenAI batch schema allows in four shapes: - ``str``: a single text prompt. - ``list[str]``: multiple text prompts. - ``list[int]``: a pre-tokenized prompt (each int counts as 1 token). - ``list[list[int]]``: multiple pre-tokenized prompts. Pre-fix only the string shapes were counted, so a caller could send a large ``list[list[int]]`` payload and slip past TPM rate limits with a recorded cost of zero tokens. r\textrrc3DK|]}t|trdndyw)rrN)rfr&)rts r)rz0_count_prompt_or_input_tokens..wsK*Q"4Q!;Ks c3ZK|]"}t|tst|$yw)rN)rfr%r)rr r\s r)rz0_count_prompt_or_input_tokens..xs'eAR\]^`cRd]Q??es++)rfr%rr$r&r)r\rtotalrs` r)rr\s%5u55% fE%%U??E3' E4(KUKKKeeee f r(c|dvrddlm}ddlm}|j ddxsi}|dk(r'|j |r|j |S|j|d}|r=|jdk(r.tjd |t|j|Sdd l m}|j ddxsi}|j|r|j!|St#d i|}|S) z> Get the tokens of a batch job from the response body r{r)AnthropicConfig)AmazonConverseConfigrNr]) usage_objectreasoning_contentzwbatch output line reported usage this parser does not understand, so it will be billed at $0. provider=%s usage_keys=%s)ResponseAPILoggingUtilsr')*litellm.llms.anthropic.chat.transformationr1litellm.llms.bedrock.chat.converse_transformationrrhis_converse_usage_shapeusage_from_batch_outputcalculate_usagerDr rksortedkeyslitellm.responses.utilsr_is_response_api_usage+_transform_response_api_usage_to_chat_usager) rnr+rrranthropic_usager _usage_dictrs r)rrrr}s 66NZ+//>D" ) +0D0\0\]i0j')AA,O O!0!2!B!B%""C"  O88A=  " ",#|((*+   ?&**7D9?RK55kB&RRS^__.+.E Lr(batch_results_linec.|jddxsiS)z Get the ``result`` object from a line of an Anthropic message batch results JSONL file. Anthropic batch results lines look like: ``{"custom_id": ..., "result": {"type": "succeeded", "message": {..., "usage": {...}}}}`` rN)rh)r s r)-_get_anthropic_result_from_batch_results_liner"s  ! !(D 1 7R7r(batch_job_output_filec|dk(rt|jddxsiS|dk(r|jddxsiS|jddxsi}|jddxsi}|S)z9 Get the response from the batch job output file r0messageNr] modelOutputrrr"rh)r#r+ _response_response_bodys r)rqrqs k)<=RSWWXacghnlnni'$((=CC266z4HNBI%MM&$7=2N r(c|dk(rt|jddk(S|dk(r(|jdduxr|jdduS|jddxsi}|jd dd k(S) a Check if the batch job response was successful OpenAI-shaped output rows report ``response.status_code == 200``; Anthropic message batch results lines report ``result.type == "succeeded"``; Bedrock batch output lines report ``modelOutput`` (and no ``error``). r0r succeededr]r&Nrr status_coder')r#r+r(s r)r_r_sk)<=RSWWX^_cnnni'$((7tCrH]HaHabiHjnrHrr266z4HNBI == - 44r()NN)NNNr)r,N)r,)Brcollections.abcrrr dataclassesrrrLenumrtypingr r r r9litellm._loggingr -litellm.litellm_core_utils.get_litellm_paramsr.litellm.litellm_core_utils.llm_cost_calc.utilsrlitellm.types.llms.openairlitellm.types.utilsrrr litellm.utilsrrr$rgr%r<rPrRrXrcr`rir"rtr;r:rbytesrrIr&rHrrJrrKrrr#rrrrrr"rqboolr_r'r(r)r:s\ 77!6&&+TV+;;' $d#$"#' !$Z ![\d D   B""&#' ? ? ![\?d ?4K ? D ?  ?D4 $d#$F d^F !fgFd FD F #l23 F& 38  !fgd D  T! , 38  !fgd D   >)38$) )!!fg)d ) $J ) D ) )>"#' - d^- !fg-d -D -  -d"Q#DzQd QQhSS8`h"&! ! ![\!4K! !@`h"&  ![\4K ,G G ![\G4KG G0&TD[&T&R:%:DJ:%HUO$ U x~  5 TD[ 01%u0J5JSJ"  d    .  S S DBJ38$;> D8gcSVhFW8\cdgildl\m8JR "38, CF  S#X  JR5"38,5CF5 5r(