tPjUdZddlmZddlZddlmZmZmZmZddl m Z ddl m Z ddl mZddlmZmZmZmZmZmZmZmZeZeed <ej5d d gee ge d ee fdedededzde fdZej;d d gee ge dddddd ee fdedededzdedzdedzdedzdedzdedzde fdZej;dd gee ge d ee fdededededzde f dZ ej5dd gee ge d ee fdededededzde f dZ!ejEdd gee ge d ee fdededededzde f dZ#ej5dd gee ge d ee fdededededzde f d Z$ej5d!d"gee ge d ee fdededededzde f d#Z%ej;d!d"gee ge ddddd ee fdededededzdedzdedzdedzdedzde fd$Z&ej;d%d"gee ge d ee fded&ededededzde f d'Z'ej5d%d"gee ge d ee fded&ededededzde f d(Z(ejEd%d"gee ge d ee fded&ededededzde f d)Z)y)*z( OpenAI Evals API endpoints - /v1/evals )FinalN) APIRouterDependsRequestResponse)UserAPIKeyAuth)user_api_key_auth)ProxyBaseLLMRequestProcessing)CancelEvalResponseCancelRunResponseDeleteEvalResponseEvalListEvalsResponseListRunsResponseRunRunDeleteResponserouterz /v1/evalszOpenAI Evals API)tags dependenciesresponse_modelopenaifastapi_responserequestcustom_llm_provideruser_api_key_dictcFKddlm}m}m}m}m}m} m} m} m } m } m }|jd{}|rtj|ni}|jdxs8|j jdxs|j"jd}|r||d<d|vr||d<t%|} |j'|||d||||||jd| | | | | | d{S77#t($r#}|j+|||| d{7d}~wwxYww) a Create a new evaluation. Model-based routing (for multi-account support): - Pass model via header: `x-litellm-model: gpt-4-account-1` - Pass model via query: `?model=gpt-4-account-1` - Pass model via body: `{"model": "gpt-4-account-1"}` Example usage: ```bash curl -X POST "http://localhost:4000/v1/evals" -H "Authorization: Bearer your-key" -H "Content-Type: application/json" -d '{ "name": "Test Eval", "data_source_config": {"type": "file", "file_id": "file-abc123"}, "testing_criteria": {"graders": [{"type": "llm_as_judge"}]} }' ``` Returns: Eval object with id, status, timestamps, etc. r general_settings llm_router proxy_configproxy_logging_objselect_data_generator user_api_baseuser_max_tokens user_modeluser_request_timeoutuser_temperatureversionNmodelx-litellm-modelrdata acreate_evalrrr route_typer!rrr r"r)r%r'r&r$r#r(err!r(litellm.proxy.proxy_serverrrr r!r"r#r$r%r&r'r(bodyorjsonloadsget query_paramsheadersr base_process_llm_request Exception_handle_llm_api_exception)rrrrrrr r!r"r#r$r%r&r'r(r4r,r) processorr1s s/home/oem/CyberBrain26-unified/.venv/lib/python3.12/site-packages/litellm/proxy/openai_evals_endpoints/endpoints.py create_evalr?sHD     &D(,&,,t$"D88G$s(<(<(@(@(IsW__M`M`arMsE W D(&9 "#5$?I 77-/%/!-%"7((7#!-!5+'!8   ' $  77// 8     sR.D!C.BD!63C2)C0*C2-D!0C22 D;DDDDD!limitafterbeforeorderorder_byc Kddlm} m} m} m} m} m}m}m}m }m }m }|jd{}|rtj|ni}d|vr|||d<d|vr|||d<d|vr|||d<d|vr|||d<d|vr|||d<|jd xs8|j jd xs|j"jd }|r||d <d |vr||d <t%| } |j'|||d | | | | | |jd ||||||d{S77#t($r#}|j+||| |d{7d}~wwxYww)a List evaluations with pagination. Model-based routing (for multi-account support): - Pass model via header: `x-litellm-model: gpt-4-account-1` - Pass model via query: `?model=gpt-4-account-1` - Pass model via body: `{"model": "gpt-4-account-1"}` Example usage: ```bash curl "http://localhost:4000/v1/evals?limit=10" -H "Authorization: Bearer your-key" ``` Returns: ListEvalsResponse with list of evaluations rrNrArBrCrDrEr)r*rr+ alist_evalsr.r0r2)rrrArBrCrDrErrrrr r!r"r#r$r%r&r'r(r4r,r)r=r1s r> list_evalsrHvsB     &D(,&,,t$"Ddu0W du0W t 2Xdu0W ("6#Z88G$s(<(<(@(@(IsW__M`M`arMsE W D(&9 "#5$?I 77-/$/!-%"7((7#!-!5+'!8   5'4 $  77// 8     sR.ED%B;E-3D) D'!D)$E'D)) E2E E  EEEz/v1/evals/{eval_id}eval_idcPKddlm}m}m}m}m} m} m} m} m } m }m }|jd{}|rtj|ni}||d<|jdxs8|j jdxs|j"jd}|r||d<d|vr||d<t%|} |j'|||d ||||| |jd| || | | | d{S77#t($r#}|j+|||| d{7d}~wwxYww) a Get a specific evaluation by ID. Model-based routing (for multi-account support): - Pass model via header: `x-litellm-model: gpt-4-account-1` - Pass model via query: `?model=gpt-4-account-1` - Pass model via body: `{"model": "gpt-4-account-1"}` Example usage: ```bash curl "http://localhost:4000/v1/evals/eval_123" -H "Authorization: Bearer your-key" ``` Returns: Eval object rrNrIr)r*rr+ aget_evalr.r0r2rIrrrrrrr r!r"r#r$r%r&r'r(r4r,r)r=r1s r>get_evalrMsR:     &D(,&,,t$"DDO88G$s(<(<(@(@(IsW__M`M`arMsE W D(&9 "#5$?I 77-/"/!-%"7((7#!-!5+'!8   #'" $  77// 8     R.D&C3B D&;3C7.C5/C72D&5C77 D#DDDD##D&cPKddlm}m}m}m}m} m} m} m} m } m }m }|jd{}|rtj|ni}||d<|jdxs8|j jdxs|j"jd}|r||d<d|vr||d<t%|} |j'|||d ||||| |jd| || | | | d{S77#t($r#}|j+|||| d{7d}~wwxYww) a Update an evaluation. Model-based routing (for multi-account support): - Pass model via header: `x-litellm-model: gpt-4-account-1` - Pass model via query: `?model=gpt-4-account-1` - Pass model via body: `{"model": "gpt-4-account-1"}` Example usage: ```bash curl -X POST "http://localhost:4000/v1/evals/eval_123" -H "Authorization: Bearer your-key" -H "Content-Type: application/json" -d '{"name": "Updated Name"}' ``` Returns: Updated Eval object rrNrIr)r*rr+ aupdate_evalr.r0r2rLs r> update_evalrQ3sR>     &D(,&,,t$"DDO88G$s(<(<(@(@(IsW__M`M`arMsE W D(&9 "#5$?I 77-/%/!-%"7((7#!-!5+'!8   #'" $  77// 8     rNcPKddlm}m}m}m}m} m} m} m} m } m }m }|jd{}|rtj|ni}||d<|jdxs8|j jdxs|j"jd}|r||d<d|vr||d<t%|} |j'|||d ||||| |jd| || | | | d{S77#t($r#}|j+|||| d{7d}~wwxYww) a Delete an evaluation. Model-based routing (for multi-account support): - Pass model via header: `x-litellm-model: gpt-4-account-1` - Pass model via query: `?model=gpt-4-account-1` - Pass model via body: `{"model": "gpt-4-account-1"}` Example usage: ```bash curl -X DELETE "http://localhost:4000/v1/evals/eval_123" -H "Authorization: Bearer your-key" ``` Returns: DeleteEvalResponse with deletion confirmation rrNrIr)r*rr+ adelete_evalr.r0r2rLs r> delete_evalrTR:     &D(,&,,t$"DDO88G$s(<(<(@(@(IsW__M`M`arMsE W D(&9 "#5$?I 77-/%/!-%"7((7#!-!5+'!8   #'" $  77// 8     rNz/v1/evals/{eval_id}/cancelcPKddlm}m}m}m}m} m} m} m} m } m }m }|jd{}|rtj|ni}||d<|jdxs8|j jdxs|j"jd}|r||d<d|vr||d<t%|} |j'|||d ||||| |jd| || | | | d{S77#t($r#}|j+|||| d{7d}~wwxYww) a Cancel a running evaluation. Model-based routing (for multi-account support): - Pass model via header: `x-litellm-model: gpt-4-account-1` - Pass model via query: `?model=gpt-4-account-1` - Pass model via body: `{"model": "gpt-4-account-1"}` Example usage: ```bash curl -X POST "http://localhost:4000/v1/evals/eval_123/cancel" -H "Authorization: Bearer your-key" ``` Returns: CancelEvalResponse with cancellation confirmation rrNrIr)r*rr+ acancel_evalr.r0r2rLs r> cancel_evalrXrUrNz/v1/evals/{eval_id}/runszOpenAI Evals API - RunscKddlm}m}m}m}m} m} m} m} m } m }m }|jd{}|rtj|ni}||d<|jj!dxsr|j"j!dxsU|j!dxsBt%|j!dt&r!|j!dij!dnd}|r||d<d|vr||d<t)| } |j+|||d ||||| |j!d| || | | | d{S77#t,$r#}|j/|||| d{7d}~wwxYww) a Create a new run for an evaluation. Model-based routing (for multi-account support): - Pass model via header: `x-litellm-model: gpt-4-account-1` - Pass model via query: `?model=gpt-4-account-1` - Pass model via body: `{"model": "gpt-4-account-1"}` - Pass model via completion.model: `{"completion": {"model": "gpt-4-account-1"}}` Example usage: ```bash curl -X POST "http://localhost:4000/v1/evals/eval_123/runs" -H "Authorization: Bearer your-key" -H "Content-Type: application/json" -d '{ "data_source": {"type": "dataset", "dataset_id": "dataset_123"}, "completion": {"model": "gpt-4", "temperature": 0.7} }' ``` Returns: Run object with id, status, timestamps, etc. rrNrIr*r) completionrr+ acreate_runr.r0)r3rrr r!r"r#r$r%r&r'r(r4r5r6r9r7r8 isinstancedictr r:r;r<rLs r> create_runr^BsF     &D(,&,,t$"DDO -. k    # #G , k 88G  k8B$(( list_runsrbs@@      D??&&'89^W=Q=Q=U=UV]=^E W D(&9 "#5$?I 77-/#/!-%"7,0HHW,=,I#dhhw'(t!-!5+'!8    $  77// 8     sIA7C?:AC C C C?C C<C71C42C77C<<C?z!/v1/evals/{eval_id}/runs/{run_id}run_idcKddlm}m}m}m} m} m} m} m} m }m }m }||d}|jjdxs|jjd}|r||d<d|vr|||d<t|} |j!|||d | |||| |jd| ||| | | d{S7#t"$r#}|j%||| | d{7d}~wwxYww) a Get a specific run by ID. Model-based routing (for multi-account support): - Pass model via header: `x-litellm-model: gpt-4-account-1` - Pass model via query: `?model=gpt-4-account-1` Example usage: ```bash curl "http://localhost:4000/v1/evals/eval_123/runs/run_456" -H "Authorization: Bearer your-key" ``` Returns: Run object with full details rr)rIrcr*r)rNr+aget_runr.r0)r3rrr r!r"r#r$r%r&r'r(r9r7r8r r:r;r<)rIrcrrrrrrr r!r"r#r$r%r&r'r(r,r)r=r1s r>get_runrfs':     D ??&&'89^W=Q=Q=U=UV]=^E W D(-@-L&9 "#5$?I 77-/!/!-%"7((7#!-!5+'!8    $  77// 8     sHA6C"93B3,B1-B30C"1B33 C<CCCCC"cZKddlm}m}m}m} m} m} m} m} m }m }m }|jd{}|rtj|ni}||d<||d<|jdxs8|j jdxs|j"jd}|r||d<d|vr||d<t%| } |j'|||d | |||| |jd| ||| | | d{S77#t($r#}|j+||| | d{7d}~wwxYww) a Cancel a running run. Model-based routing (for multi-account support): - Pass model via header: `x-litellm-model: gpt-4-account-1` - Pass model via query: `?model=gpt-4-account-1` Example usage: ```bash curl -X POST "http://localhost:4000/v1/evals/eval_123/runs/run_456/cancel" -H "Authorization: Bearer your-key" ``` Returns: CancelRunResponse with cancellation confirmation rrNrIrcr)r*rr+ acancel_runr.r0r2rIrcrrrrrrr r!r"r#r$r%r&r'r(r4r,r)r=r1s r> cancel_runrjY[:     &D(,&,,t$"DDODN88G$s(<(<(@(@(IsW__M`M`arMsE W D(&9 "#5$?I 77-/$/!-%"7((7#!-!5+'!8   %'$ $  77// 8     R.D+C8BD+3C<3C:4C<7D+:C<< D(D#D D##D((D+cZKddlm}m}m}m} m} m} m} m} m }m }m }|jd{}|rtj|ni}||d<||d<|jdxs8|j jdxs|j"jd}|r||d<d|vr||d<t%| } |j'|||d | |||| |jd| ||| | | d{S77#t($r#}|j+||| | d{7d}~wwxYww) a Delete a run. Model-based routing (for multi-account support): - Pass model via header: `x-litellm-model: gpt-4-account-1` - Pass model via query: `?model=gpt-4-account-1` Example usage: ```bash curl -X DELETE "http://localhost:4000/v1/evals/eval_123/runs/run_456" -H "Authorization: Bearer your-key" ``` Returns: RunDeleteResponse with deletion confirmation rrNrIrcr)r*rr+ adelete_runr.r0r2ris r> delete_runrorkrl)*__doc__typingrr5fastapirrrrlitellm.proxy._typesr$litellm.proxy.auth.user_api_key_authr 'litellm.proxy.common_request_processingr litellm.types.llms.openai_evalsr r r rrrrrr__annotations__postrar?r7intrHrMrQdeleterTrXr^rbrfrjror>r}s 99/BQ      +,-  '/(/0A(B R R R tR & R  R j  +,-$  &.(/0A(B\ \ \  :\  : \ $J \ : \ Dj\ t\ &\  \ ~  +,-  '/(/0A(B O O O O t O & O  O d  +,-  '/(/0A(B Q Q Q Q t Q & Q  Q h  +,-% '/(/0A(B O O O O t O & O  O d  +,-%  '/(/0A(B O O O O t O & O  O n # $+,-  '/(/0A(B Z Z Z Z t Z & Z  Z z # $+,-#  &.(/0A(BT T T T  : T : T $J T  :T tT &T  T n' # $+,-  '/(/0A(B N N N N  N t N & N  N b' # $+,-$  '/(/0A(B P P P P  P t P & P  P f' # $+,-$ '/(/0A(B P P P P  P t P & P  P r|