uPj)"ddlmZmZmZddlmZddlZddlmZddl m Z ddl m Z gdddddddddddddddd fd ed ed edzd edzdedzdedzdedzdedzdedzdedzdedzdedzdedzdedzdedzdedzf dZdZdZy))FIRST_COMPLETEDThreadPoolExecutorwait)FinalN) print_verbose)get_optional_params)handleriXdmodelmessages functions function_call temperaturetop_pnstream max_tokenspresence_penaltyfrequency_penalty logit_biasuserrequest_timeouttimeout max_workersc Tt}|}g}|}d}|jdddtjvr*|jddd}|jddd}|dk(rFt ||||||xsd|| | | | | ||}t j ||tj|}|Sd }t| 5}||d D]z}|D]s}|j}|jd ||d <i}d|vr|jd}|jtjfi||}|j|u| dddg}|D]"} |j|j$|S#1swY4xYw#t$r} |j| Yd} ~ Rd} ~ wwxYw)a  Batch litellm.completion function for a given model. Args: model (str): The model to use for generating completions. messages (List, optional): List of messages to use as input for generating completions. Defaults to []. functions (List, optional): List of functions to use as input for generating completions. Defaults to []. function_call (str, optional): The function call to use as input for generating completions. Defaults to "". temperature (float, optional): The temperature parameter for generating completions. Defaults to None. top_p (float, optional): The top-p parameter for generating completions. Defaults to None. n (int, optional): The number of completions to generate. Defaults to None. stream (bool, optional): Whether to stream completions or not. Defaults to None. stop (optional): The stop parameter for generating completions. Defaults to None. max_tokens (float, optional): The maximum number of tokens to generate. Defaults to None. presence_penalty (float, optional): The presence penalty for generating completions. Defaults to None. frequency_penalty (float, optional): The frequency penalty for generating completions. Defaults to None. logit_bias (dict, optional): The logit bias for generating completions. Defaults to {}. user (str, optional): The user string for generating completions. Defaults to "". deployment_id (optional): The deployment ID for generating completions. Defaults to None. request_timeout (int, optional): The request timeout for generating completions. Defaults to None. max_workers (int,optional): The maximum number of threads to use for parallel processing. Returns: list: A list of completion results. N/rvllmF)rrrrrrstoprrrrrr custom_llm_provider)r r custom_prompt_dictoptional_paramsc3VKtdt||D] }||||zyw)z)Yield successive n-sized chunks from lst.rN)rangelen)lstris b/home/oem/CyberBrain26-unified/.venv/lib/python3.12/site-packages/litellm/batch_completion/main.pychunksz batch_completion..chunks_s21c#h* %!a!en$ %s')rr rr kwargs)localssplitlitellm provider_listr vllm_handlerbatch_completionsr"rcopypopsubmit completionappendresult Exception)!r r rrrrrrr rrrrr deployment_idrrrr,argsbatch_messages completionsr!r#resultsr*executor sub_batch message_listkwargs_modifiedoriginal_kwargsfutureexcs! r)batch_completionrF s`(D$NK E {{31!6!66#kk#q1!4 C#A&f$!4'#?U!-/! 3" "00#&99+  H N7 %  K 8 /H#NC8 / $-/L&*iikO#'' 62>OJ/&(O?2*9*=*=h*G,X__W-?-?f?fVefF&&v./ / /! $F $v}}/ $ N- / /& $s## $s%z)batch_completion_models..sv||TUVWTXGY)keyN deployments model_listr,z waiting for next result ) return_whenz done list z3 got an exception, ignoring, removing from futuresz removing keyz new futuresz done looping through futures )r4rr&r5r/r6sorteditemsr8rrvaluesrr9)r;r,futuresr?r rDrO nested_kwargs deploymentrNdone_r8 new_futuresvaluerHs @r)batch_completion_modelsr\s&& 76x( 8 CK 8 +H c!01C1C!bd!bRW!b[a!b c"( =Y!Z + v==?.!==?*  + + +  +` S & #M2  =! < %zz(B7  C ,< =! ') ] !6C*,*0+ 369J8-8/>xw?Q?Q/\U[/\ 7+,  ]@Aw~~/_Ma D623"!F!!'% !! '! '!&DEg&3! 'F a +` %% !%&]^%g.&( *1--/9JC% - SE.B C (38 C 0 9 #.% G9&=>  !#! 'F sWA%G6.G6 G6)I*9A;I*5HI*I*6G?A"I'$I*&I''I**I3c ddl}d|vr|jdd|vr|jd}n tdt|tr|g}n-t|t t fr t |}n tdt|dk(rgSg}|jjt|5}|Dcgc](}|jtjg|d|i|*}}|D]&} |j} | |j| ( ddd|Scc}w#t$r} t!d| Yd} ~ Ud} ~ wwxYw#1swY|SxYw) a Send a request to multiple language models concurrently and return a list of responses from all models that respond. Args: *args: Variable-length positional arguments passed to the completion function. **kwargs: Additional keyword arguments: - models (str or list of str): The language models to send requests to. - Other keyword arguments to be passed to the completion function. Returns: list: A list of responses from the language models that responded. Note: This function utilizes a ThreadPoolExecutor to parallelize requests to multiple models. It sends requests concurrently and collects responses from all models that respond. rNr rHz'models' param not in kwargsz,'models' must be a string or list of stringsr+z=batch_completion_models_all_responses: model request failed: )concurrent.futuresr4r9 isinstancestrlisttuple TypeErrorr&rUrr5r/r6r8r7r) r;r, concurrentrH responsesr?r rUrDr8es r)%batch_completion_models_all_responsesrgs\$& 76H%677&# FT5M *fFGG 6{a I    . .3v; . G 8iop`e/(//'*<*<[t[5[TZ[pp F %$$V,    q  ]^_]`ab   sH3E8-D%E-#D"EE" E+D>9E>EEE)r^rrrtypingrr/litellm._loggingr litellm.utilsrllms.vllm.completionr r1r`rafloatintbooldictrFr\rgrMr)rqsHHH*-: ! $ $ !%)&*""&!'o ood{ o : o  o 4<o Tzo 4Kod odlot|ot o *o"4Z#o$4Z%o&t'ohIX5rM