cQjO: dZddlZddlZddlZddlZddlZddlmZddlZddl Z ddl Z ddl m Z dZ de jfdZd"ded edzfd Zde jd ed efd Zde jd edefdZde jdede jfdZd ede j.defdZde jdededededef dZde jdedededef dZe j8ded edzdededef dZd Zed!k(rFeZ ee jBe jDe jFe jHe jJyy)#z Export LLM to onnx N)Path)nncdtjj_dtjj_dtjj_dtjj_dtjj_dtjj_dtjj_ dtjj_ y ) z1do not init model twice as it slow initializationc|SNxargskwargss r/home/oem/CyberBrain26-unified/.venv/lib/python3.12/site-packages/onnxruntime/transformers/large_model_exporter.pyz*disable_huggingface_init..sc|Srrr s r rz*disable_huggingface_init..src|Srrr s r rz*disable_huggingface_init..sqrc|Srrr s r rz*disable_huggingface_init..src|Srrr s r rz*disable_huggingface_init..qrc|Srrr s r rz*disable_huggingface_init..sarc|Srrr s r rz*disable_huggingface_init.. rrc|Srrr s r rz*disable_huggingface_init..!s1rN) torchrinitkaiming_uniform_uniform_normal_ constant_xavier_uniform_xavier_normal_kaiming_normal_ orthogonal_rrr disable_huggingface_initr"s&BEHHMM"9EHHMM8EHHMM:EHHMM$@EHHMM!#?EHHMM $@EHHMM! +>+@@@ foo'' ([(D047H Orhf_model cache_dircttjj|tj |d}||}tj j|}t|ddj}||fS)zU get the pretrained torch model from hugginface, and sample model-inputs T) torch_dtyper3trust_remote_codezHello, my dog is cutept)return_tensors) r" transformersAutoModelForCausalLMfrom_pretrainedrfloat16 AutoTokenizertuplevalues)r2r3 tokenizerr# sample_inputss r "initialize_model_and_sample_inputsrB4s|   - - = =emmyTX > E **::9EI)$;DQXXZ[M - rgpulistrAc d}fd|j}g}|j|j|dtt |j d}|j D]\}}|j D]%\}} |j| j|dt | tjjfvrtjt| t|z } t| D]s\} } |j| j|d|t| | zt|} | j| | | t!d|d|d| d| u| j|dt!d|d|d|d(tt#|j dk(sg|j|dt!d|d|dtj$5||d|d  d d d |S#1swY|SxYw) z/Make the model executable across multiple GPUs.c g}d}|D]}t|tjur|j|nFt |dr6|j|j |j jnt |drEt|j|j}|j|j |nt t|jddrL|j|j t|jj jnC|0|j|k7r!|j|j |n|j||||dj}|jD]5\}}t|tjus"|j |||<7t||fS)Nweightr&r) typerTensorappendhasattrtorFdevicenextr&childrenitemsr>) modinputsr modifyed_inputs first_dev layer_inputrLkeyvalues r input_gpu_device_hookz5auto_pipeline_parallel..input_gpu_device_hookJs ! 6KK  4&&{3h'&&{~~cjj6G6G'HIl+cnn. <CC&&{~~f'=>cllnd3X>&&{~~d3<<>6J6Q6Q6X6X'YZ&;+=+=+J&&{~~i'@A&&{3 +A.55  6 !,,. 2JCE{ell*#hhy1s  2o&//rcj|j||jD]}|d|y)N)rKnamed_children)rPdevlayermove_layer_to_device_rurcs r r]z9auto_pipeline_parallel..move_layer_to_device_rurccs4 s '') 5E %eAh 4 5rT with_kwargsrzmove .z to rYattention_maskN)halfrIregister_forward_pre_hookrMiterrZrGrr ModuleListmathfloorlen enumerateminrKprintlistno_grad)r#rCrArW all_hookspre_fixtop_name top_modulenamemodulenum_layers_on_each_gpuidx attn_layerto_devr]s @r auto_pipeline_parallelryGs+025 JJLEI U445JX\4]^4,,./03G % 4 4 66*&557 @LD&   V==>Sae=f gF| 3 344)-CK#g,4N)O&'0'8FOC$$Z%I%IJ_mq%I%rs$S0F)FG %UVFMM&)-j&AE'!D63%tF8DE F '!*%gYavT'!*>? @ tJ--/0 1Q 6 MM'!* % E(4 |4 5#6& A mA}Q/?@A LA Ls H44H> with_pastcgfd}|j|d}tj|jj}t |j }|Dcgc]}|j|j}}||d|d} |jd|} tdD]\} } d| | | <djD]\}} |j|} | | | <tt|| dD]]\} \}} t| tj ur| j#|j$d |vsD|| | <|r||d|d| n| } _|| | j&fScc}w) zn auto retrieve onnx inputs from torch model as we can't enumlate all possibilities for all models c4j||fdS)Nr)rI)_rQr user_inputss r hook_for_inputsz-retrieve_onnx_inputs..hook_for_inputss FF+,1~rTr^rrYraF)strict use_cache)rbr)rdinspect signatureforwardr&rmkeysgetdefaultremoverjrOindexziprGrrHrKrLpast_key_values)r#rArzr hook_handleforward_params input_keysrUdefault_valuesout onnx_inputsrv_valrVr~s @r retrieve_onnx_inputsrs K11/t1TK&&u}}5@@Nn))+,JAKL#n((-55LNL  a q1A BCa.K K{1~./ T&q>#. C/!!n**,! Us# C!'s:{5'QRv\c5 ;%,, & HHU\\ " # (K cl% a(q9IU^_ruC v {C$7$7 77#Ms""Fsample_inputs_tpreturnctjjdjdz dz }t dt |dz dt d|dz dt ||dzkDrt tjjDcgc]}tj|}}t|dkDr&t t|dt|||}|St d |jj}|St d |jj}|Scc}w) a  According to the model size, we will upload it to CPU if has no GPU or enough GPU memory, Single GPU if has only one GPU in local or model size is enough to fit one GPU Multiple GPU if there is more than one gpu in local and model is too large rr%z Model_Size = z GBztotal_mem_per_cpu = g?rYzk GPUs are used to export onnx, Please set CUDA_VISIBLE_DEVICES to use specific GPU groupz5!!!! convert model to float and export onnx using CPUzExport model on a single GPU)rcudaget_device_properties total_memoryrlr1range device_countrLrirycpufloatrc)r#rtotal_mem_per_cpuidevice_collections r move_to_appropriate_devicers8 88;HH4ORVV M259D@A EF !2T!9 :# >?&):T)AA6;EJJ|j|Pt |S)zmove inputs to device) isinstancerrHrIrKr>)rArLsample_inputs_ sample_ints r adapt_inputs_to_devicersVN#. j%,, /  ! !*--"7 8  ! !* - .   rrtorch_input_namesrinput_with_pastc d}ddi}|t|}tj|ddj|djdk(j j d}|j dk(sJdd|jdi}|s|jj}ttt|D cgc]%} t|| tjs!|| 'c} } d| vrd| vsJd d } ddd ddd d } t| D]>\} }|| vs t|| jD cic] } | | d |  }} || |<@|r4t|D]&} | d| dfz } | d| dfz } || | d<|| | d<(|s|r$t|D]} | d| dfz } | d| dfz } t|D]\} }|s |dk(r||| <|dk(r[|| }tj |tj"|jddf|j$|j&fd|| <t|dk(sz|| }|ddddf|| <| | | fScc} wcc} w)z"fetch onnx inputs and outputs namer batch_sizeNrYseq_len input_idsrbz6input_ids and attention_mask must be existed in inputs)logits)rrY)rrb__unknown_dims__zpast_key_values.z.keyz.valuezpresent.r)rLdtype)dim)rirtensorshapenonzeroviewnumelitemconfignum_hidden_layersr>rrrHrjrcatonesrLr)r#rrrrzrnum_of_past_key kv_cache_axis seq_indexronnx_inp_namesonnx_out_namesonnx_dynamic_axesrvrs unknown_dims attn_maskrs r fetch_onnx_inputs_outputs_namersO %M"o.\\/!"4Q"7"="=>+a.BVBVWYBZZccejjkmn  A%%%L)..*:IF ,,88',S1B-C'Dq! S^_`SachcocoHp 1 qN . (-=-O@ O!N%)4*y9 ~.3 T ( (DI+VYJZJ^J^J`DabqA#&6qc::bLb&2 d #3' BA !1!D9; ;N !1!F;= =N4A nR0 14A nR0 1  BO' 6A !D13 3N !F35 5N 601 5 T ((#2 C ))', #(99 IOOA,>+B9K[K[clcrcr st$ C $', #,QV#4 C  5 >+< <unlinkparentloadstr save_modelrilistdir) r#rrrronnx_model_namerrr tmpdirnametmp_onnx onnx_models r do_export_internalrsnnO8E5NN$5  $ $ & *77<< J7 {#&'*   D)   /y9 9AATARYYs8}-     N#&rzz*'=#>#B$('( 2# '   s C+DD$ onnx_path_strct||\}}t||}t|t|j j }t |||\}} } t|| || |d} d} t|j} | jdk7r| | z } t|| | | ||syt|| || |d} d} | j| z } t|| | | |y)z do export model: torch model onnx_path: where the onnx model saved to sample_inputs_tp: inputs for torch model Fz model.onnxz.onnxNTzmodel_with_past.onnx) rBrrrMr&rLrrrabsolutesuffixrr)r2r3rrzrr#rrArrpast_key_valuerrrs r export_onnxr7sA9UE  &u.> ?E*+;T%BRBRBT=U=\=\]M/C5-Yb.c+J ^25+zSaclnstM"O=)224I7"/ um[)UK 25+zSaclnrsM,O  ?2Ium[)UKrcJtj}|jdddtdgd|jddd td d |jd d td d|jddd d|jdd tdd|j S)zarguments parsing.z-mz--modelTzmeta-llama/Llama-2-70b-hfz+Pre-trained models in huggingface model hub)requiredrGrhelpz-sz --saved_pathFz./onnx_models/z"where the onnx model will be savedz --cache_dirNz[cache directly of huggingface, by setting this to avoid useless downloading if you have onez --with_past store_truez;The tool will export onnx without past-key-value by default)actionrrz--opsetzothe opset to save onnx model, try to increase it if this opset doens't have new features you want)argparseArgumentParser add_argumentrint parse_args)parsers r parse_argumentsr[s  $ $ &F   ,- :     1   k  K     S      r__main__r)&__doc__rrrgrrpathlibrrrr9rr"Moduler1rrBrmr>ryboolrrrLrrrrrnrr__name__r r#r3 saved_pathrzrrrr rs    = BII    t &:")):d:5:z 8 8% 8D 8Fbii5RYY8!%!!%!A= 99A=A=A= A=  A=  A=H" bii" " E" ^b" kn" J L# L#* LS LUY Lbe L LF+\ z  D DNNDOOT^^TZZXr