cQjddlmZddlZddlmZddlmZddlm Z m Z ddl m Z m Z mZmZddlmZeeZGdd ZGd d eZy) ) getLoggerN)Fusion)AttentionMaskFormat) FusionUtils NumpyHelper) NodeProto TensorProtohelper numpy_helper) OnnxModelcJeZdZdZdefdZdefdZdZdZ de d e d zfd Z y ) AttentionMask: Fuse Attention subgraph into one Attention node. modelc||_i|_i|_t||_t j |_|j|_ yN) r mask_indice mask_castedrutilsr MaskIndexEnd mask_formatget_opset_version opset_version)selfrs n/home/oem/CyberBrain26-unified/.venv/lib/python3.12/site-packages/onnxruntime/transformers/fusion_attention.py__init__zAttentionMask.__init__sD  ' .;;"446rc||_yr)r)rrs rset_mask_formatzAttentionMask.set_mask_format s &rcf||jvr||j|k(sJ||j|<yr)r)rmask mask_indexs rset_mask_indicezAttentionMask.set_mask_indice#s9 4## #!1!1$!77 77!+rcrt|jdkDsJtt|jSNr)lenrnextiter)rs rget_first_maskzAttentionMask.get_first_mask(s14##$q(((D))*++rmask_2dreturnNc |jtjk(ry||jvr|j|S|jj |r|j j|\}}n |j j|\}}d}|r||j|<|jtjk(r||j|<|S|jjd}|jdkr|tjd|g|g|jjdd}|jj!tj"ddgtj"d d gnd }|jj%|C|jj'tj(|t*j,dgdgd tjd||g|g|jjdd}|jj!tj"d d g|jj/|||j|<|S)NTr" ReduceSum MaskReduceSuminputsoutputsnameaxeskeepdimsrort_const_1_reduce_sum_axesFr3 data_typedimsvalsraw)rrNoMaskrrfind_graph_inputrcast_graph_input_to_int32cast_input_to_int32rrcreate_node_namerr make_node attributeextendmake_attributeget_initializeradd_initializer make_tensorr INT64add_node)rr*casted input_name _cast_node output_namemask_index_node axes_names r process_maskzAttentionMask.process_mask,s-   299 9 d&& &##G, , :: & &w /!%!E!Eg!N FJ%)ZZ%C%CG%L "J F (2D  W %   2@@ @(2D  W % jj11,?    "$.."|$ ZZ00oN O  % % , ,f.C.CFQC.PRXRgRghrtuRv-w x6Izz)))4< **&&&"-"3"3SS! %.."I.$ ZZ00oN O  % % , ,f.C.CJPQ.R-S T O,$/!r) __name__ __module__ __qualname____doc__r rrrr#r)strrQrrrrsA7i7'+>', ,8C8C$J8rrc(@eZdZdZdddddgfdededed edzd ed ed ee ffd Z de de eeffdZ de de eeffdZde fdZde fdZde de de fdZde de de fdZde de dzde dzd e de dzf d!Zd"e d#e d$e de de dzde dzde e e e ffd%Z d4d"e d#e e zdzd$e e zdzde de dzde dzdeded&e d'e de d(ede de d)e d*e d+ede dzf$d,Z d5d-e dzd"e d#e d$e de de de deded.e d&e d/e d0ede de d)e d*e d1edzde dzf&d2Zd3ZxZS)6FusionAttentionrNFSkipLayerNormalizationLayerNormalizationr hidden_size num_headsattention_maskuse_multi_head_attention!disable_multi_head_attention_biassearch_op_typesc|rdnd}t ||||||_||_|r|n t ||_||_||_d|_d|_ d|_ d|_ d|_ y)NMultiHeadAttention AttentionT) superrr\r]rr^r_r`mask_filter_valuenum_heads_warninghidden_size_warning shape_infershape_infer_done) rrr\r]r^r_r`raattention_op_name __class__s rrzFusionAttention.__init__ls5M0R]  1?C&"0>nMRWDX(@%1R.!%"&#'  $rconcatr+cdt|jdk(r|jj|jd}|jj|jd}t |t j r|jdk(rt |t j rm|jdk(r^|ddk(rF|jdkDr7|j|j|dzk(r|j|jfS|d|d|dzfS|j|jfS)aU Detect num_heads and hidden_size from Concat node in the following subgraph: SkipLayerNormalization or EmbedLayerNormalization / | MatMul Shape | | Add Gather(indices=0) | | | Unsqueeze | | | Concat (*, -1, 12, 64) | / Reshape | Transpose r5r) r&inputrget_constant_value isinstancenpndarraysizer]r\)rrmr] head_sizes r)get_num_heads_and_hidden_size_from_concatz9FusionAttention.get_num_heads_and_hidden_size_from_concats$ v||  ! 55fll1oFI 55fll1oFI9bjj1NNa'y"**5NNa'Q<2%$..1*<AQAQUYUcUcfopqfrUrAr>>4+;+;;; |Yq\IaL%@@@~~t////r reshape_qc|jj|jd}|y|jj|d}| |jdk(r|j |St jd|jd|j|jfSt|tjrt|dk7s|ddks|ddkr.t jd||j|jfS|d}|d}||z}|jdkDrC||jk7r4|jr(t jd |j|d |_|jdkDrC||jk7r4|j r(t jd |j|d |_||fS) zDetect num_heads and hidden_size from a reshape node. Args: reshape_q (NodeProto): reshape node for Q Returns: Tuple[int, int]: num_heads and hidden_size r5Concatz%s is not initializer.rorprrqzGq_shape_value=%s. Expected value are like [0, 0, num_heads, head_size].z>--num_heads is %d. Detected value is %d. Using detected value.Fz@--hidden_size is %d. Detected value is %d. Using detected value.)rrtrs get_parentop_typerzloggerdebugr]r\rurvrwr&rgwarningrh)rr{ q_shape_valuermr]ryr\s rget_num_heads_and_hidden_sizez-FusionAttention.get_num_heads_and_hidden_sizes 55iooa6HI  ZZ**9a8F!fnn&@EEfMM LL19??13E F>>4#3#33 3M2::6=!Q&a A%q)9Q)> LLbdq r>>4#3#33 3!!$ !!$ )+ >>A )t~~"=%%TVZVdVdfo*/&   a K43C3C$C''VX\XhXhju,1(+%%radd_qkc|js(|jjd|_d|_|jy|jj |j d}|jj |j d}||t jd|y||k7rt jd|y|j dS)NT)updaterr5zone of the inputs of %s is Nonez)the shape of two inputs of %s is not same)rjrinfer_runtime_shaperiget_edge_shapersrr)rr input_0_shape input_1_shapes rget_add_qk_strzFusionAttention.get_add_qk_strs$$#zz==T=JD $(D !    #((77 QH ((77 QH  M$9 LL:F C M ) LLDf M||Arc|dzttfd|j}t|dk(rSt|dk(sJ|jj d}t jdt|jDcgc]}|c}g|d}|jj||j|j|<Scc}w)N_maskc(|jdk(Sr%)output)nodemask_output_names rz0FusionAttention.reshape_add_qk..st{{1~AQ/Qrr5rr}r1r2r3axis) listfilter nodes_to_addr&rrAr rBranger]appendthis_graph_namenode_name_to_graph_name)rr concat_nodeconcat_node_name_concat_add_qk_fp32rs @rreshape_add_qkzFusionAttention.reshape_add_qks"G+6"QSWSdSdef { q # #;1$$$::66x@#-- $)$..$9:qF:%&!     !349=9M9M$$%56;s C past_kpast_vcF|jjd}|jjd}|dzjdd}|dzjdd}tjd|g|g|dg}tjd|g|g|dg}|j j ||j j ||j|j|<|j|j|<|jjd} |jdd jddjd d } tjd||g| g| d } |j j | |j|j| <| S) zConcatenate past_k and past_v inputs to create past_kv input. Args: past_k (str): name of past K value past_v (str): name of past V value Returns: kv_output_name (str): name of past KV value Unsqueeze_5d.rr)r1r2r3r4r}z.valuez.kv_value_kvr) rrAreplacer rBrrrr) rrrunsqueeze_k_nameunsqueeze_v_name k_5d_name v_5d_namek_5dv_5drkv_output_name concat_kvs rrzFusionAttention.concat_kvs ::66{C::66{Ce^,,S#6 e^,,S#6  8K!   8K!     &   &9=9M9M$$%569=9M9M$$%56 ::66x@%8@@cJRRS[]bc$$ y)#$!     +9=9M9M$$%56rpresent_k_namepresent_v_namekv_nodeczd\}}|jj|}|jj|}|Rtjt j dd|}|jj ||j|Rtjt j dd|}|jj ||j|jjd}|jjd} tjd||g|g|d } tjd||g|g| d } |jj| |jj| |j|j|<|j|j| <y) a?Split kv_node containing present KV values into separate present K and present V values. Args: present_k_name (str): name of output to store present K value in present_v_name (str): name of output to store present V value in kv_node (str): name of present KV values )index_0index_1Nrint64)dtype)r3r5Gatherr)rrFr from_arrayrvarrayrGrrAr rBrrr) rrrrk_indexv_indexk_dimv_dim gather_k_name gather_v_name present_k present_vs rsplit_kvzFusionAttention.split_kv:s~0 **73 **73 = ++BHHQg,FWUE JJ & &ud.B.B C = ++BHHQg,FWUE JJ & &ud.B.B C 33H=  33H= $$ W%#$  $$ W%#$     +   +6:6J6J$$]36:6J6J$$]3rq_addk_addv_add name_prefixc|jj|jdxs(|jj|jd}tj|}t j |}t j |}|g|jj|jdxs(|jj|jd} tj| }|g|jj|jdxs(|jj|jd} tj| }t j|||fd} dt j|jz} |dz} |j| |j| g| | S)Nr5rrrq _qkv_biasr3r9r:r;) rrFrsrto_arrayrv zeros_likestackprodshaperGr9)rrrrrq_biasqbkbvbk_biasv_biasqkv_bias qkv_bias_dim bias_names rcreate_combined_qkv_biasz(FusionAttention.create_combined_qkv_biasgsy++EKKN;itzz?Y?YZ_ZeZefgZh?i  ! !& ) ]]2  ]]2   ZZ// A?m4::C]C]^c^i^ijk^lCmF%%f-B  ZZ// A?m4::C]C]^c^i^ijk^lCmF%%f-B88RRLq1277288,, +-  &&  rq_matmulk_matmulv_matmulc6 |jjd}|jd|jdk(r|jd|jdk(sJ|jj|jd}|jj|jd} |jj|jd} t j |} t j | } t j | } | j | j k(r| j | j k(sJ| j d}tj| | | fdj|d|zf}|dz}|j||j|j d|j dg||dz}tjd|jd|g|g| }|j|j|<|g}|d z}|j|t j"dgdgd |d z}|j|t j"dg|gd |dz}|j|t j"dgd|zgd |dz}|j|t j"dgd|zgd |dz}|j|t j"dgdgd |dz}tjd||||g|g|jjd }|j|j|j$<|dz}tjd||||g|g|jjd }|j|j|j$<|dz}tjd||||g|g|jjd }|j|j|j$<|}|} |}!|j'|||g|j(rL||jj|jdrdnd}"tj*t j |jj|j|"rH||jd|"z <|}|j-||j|j|j$<||jj|jdrdnd}"tj*t j |jj|j|"rH||jd|"z <|} |j-||j|j|j$<||jj|jdrdnd}"tj*t j |jj|j|"rH||jd|"z <|}!|j-||j|j|j$<|j.j'||| |!fS)aCreate packed QKV MatMul node before MultiHeadAttention node. This is for the scenario where an Attention node should be created but cannot be created because past_key and past_value are separate inputs and not one concatenated input. Args: q_matmul (NodeProto): name of MatMul from Q path - (batch_size, sequence_length, hidden_size) k_matmul (NodeProto): name of MatMul from K path - (batch_size, sequence_length, hidden_size) v_matmul (NodeProto): name of MatMul from V path - (batch_size, sequence_length, hidden_size) q_add (NodeProto): name of Add from Q path k_add (NodeProto): name of Add from K path v_add (NodeProto): name of Add from V path Returns: q_output (NodeProto): Slice node for Q k_output (NodeProto): Slice node for K v_output (NodeProto): Slice node for V MatMulrr5rrq _qkv_weightr_qkv_outr0_q_start_indexFr8_k_start_index_v_start_indexrp_end_of_qkv_index_qkv_last_axisrr_q_outSlice_k_out_v_out)rrArsrFrrrrvrreshaperGr9r rBrrr rIr3rDr`anyrr)#rrrrrrrmatmul_node_nameq_weightk_weightv_weightqwkwvwd qkv_weightqkv_weight_nameqkv_matmul_output qkv_matmul qkv_nodes q_slice_name k_slice_name v_slice_nameend_of_qkv_nameqkv_last_axis_nameq_slice_outputq_slicek_slice_outputk_slicev_slice_outputv_sliceq_outputk_outputv_outputinitializer_inputs# rcreate_packed_qkv_matmul_nodez-FusionAttention.create_packed_qkv_matmul_nodes$4 ::66x@~~a HNN1$55(..:Kx~~^_O`:```::--hnnQ.?@::--hnnQ.?@::--hnnQ.?@  ! !( +  ! !( +  ! !( +xx288#BHH(<<< HHQKXXr2rl3;;QAJG *]:  ((""1%z'7'7':;  "-z9%% NN1%7&'!  :>9M9M$$%56L (*::  ,+:K:KSTRU]^\_ejk'*::  ,+:K:KSTRU]^\_ejk'*::  ,+:K:KSTRU]^ab]b\cino*-@@ /[=N=NVWUX`ade`e_flqr-0@@ "4 @Q@QYZX[cebflqr)H4"" %|\CUV#$,,W5   6:5I5I$$W\\2)H4"" %|\CUV#$,,W5   6:5I5I$$W\\2)H4"" %|_FXY#$,,W5   6:5I5I$$W\\2'7G45  1 1 )-)C)CEKKPQN)SAYZ!66+..tzz/I/I%++VgJh/ijk9GEKK$5 56$H$$U+?C?S?SD00< )-)C)CEKKPQN)SAYZ!66+..tzz/I/I%++VgJh/ijk9GEKK$5 56$H$$U+?C?S?SD00< )-)C)CEKKPQN)SAYZ!66+..tzz/I/I%++VgJh/ijk9GEKK$5 56$H$$U+?C?S?SD00<   +8++rrkey_padding_maskunidirectionalrr packed_qkvcT|dkDsJ|dkDr ||zdk7rtjd||y|jjjDchc]}|j }}|jj d}g}|rW|j||||||\}}}|j|jd|jd|jdgnt|trt|tr|jr<|j|jd|jd|jdgn|j|jd|jd|jdgnyt|trht|trX||vrT||vrP|jr"|j|jd||gn#|j|jd||gny|js&|j||||}|j|n|jd| r|r|j| | | |gn| s| r|j| | g| g}|r|r|j||gt!j"d|||}d|_|j&jt!j(d || r8|j&jt!j(d t+| |j-d|Scc}w) aCreate a MultiHeadAttention node. Args: q_matmul (NodeProto): name of MatMul from Q path - (batch_size, sequence_length, hidden_size) k_matmul (NodeProto): name of MatMul from K path - (batch_size, sequence_length, hidden_size) or (batch_size, num_heads, past_sequence_length, head_size) v_matmul (NodeProto): name of MatMul from V path - (batch_size, sequence_length, hidden_size) or (batch_size, num_heads, past_sequence_length, head_size) q_add (NodeProto): name of Add from Q path k_add (NodeProto): name of Add from K path v_add (NodeProto): name of Add from V path num_heads (int): number of attention heads. If a model is pruned, it is the number of heads after pruning. hidden_size (int): hidden dimension. If a model is pruned, it is the hidden dimension after pruning. output (str): output name of MHA key_padding_mask (str): name of key padding mask add_qk (str): name of add after Q x K' unidirectional (bool): whether to apply causal attention mask automatically or not past_k (str): name of past K value - (batch_size, num_heads, past_sequence_length, head_size) past_v (str): name of past V value - (batch_size, num_heads, past_sequence_length, head_size) present_k (str): name of present K value - (batch_size, num_heads, sequence_length, head_size) present_v (str): name of present V value - (batch_size, num_heads, sequence_length, head_size) packed_qkv (bool): whether to combine MatMuls from Q, K, V paths Note: This is for the scenario where an Attention node should be created but cannot be created because past_key and past_value are separate inputs and not one concatenated input. Returns: Union[NodeProto, None]: the node created or None if failed. r9input hidden size %d is not a multiple of num of heads %dNrdrcr0 com.microsoftr]r)rrrgraphrsr3rAr rDrrurr`rVrrr rBdomainrCrEintincrease_counter)rrrrrrrr]r\rr rrrrrrrrgraph_input_names mha_node_name mha_inputsrrrr mha_outputsmha_nodes rcreate_multihead_attention_nodez/FusionAttention.create_multihead_attention_nodes^1}} ? i 7A= LLTVacl m37::3C3C3E3K3KL4TYYLL 33K@  (,(J(J ) %GWg   w~~a0'..2CW^^TUEVW X ) ,Hi1P55!!5<<?HOOA4F UV"XY!!8??1#5xq7I8??[\K]"^_ x %8S)----55!!5<<?Hh"GH!!8??1#5x"JK 5555eUE=YI   i (   b ! f   /H I    /8 9h     95 6##    *!!&"7"7 Y"OP     % %f&;&;C|0j8j;t+j<dtA|j>g|0S)a>Create an Attention node. Args: mask_index (str | None): mask input q_matmul (NodeProto): MatMul node in fully connection for Q k_matmul (NodeProto): MatMul node in fully connection for K v_matmul (NodeProto): MatMul node in fully connection for V q_add (NodeProto): Add bias node in fully connection for Q k_add (NodeProto): Add bias node in fully connection for K v_add (NodeProto): Add bias node in fully connection for V num_heads (int): number of attention heads. If a model is pruned, it is the number of heads after pruning. hidden_size (int): hidden dimension. If a model is pruned, it is the hidden dimension after pruning. first_input (str): first input name output (str): output name add_qk_str (str): name of Add node after Q x K' causal: whether it is uni-directional mask. past_k (str): name of input for past K value past_v (str): name of input for past V value present_k (str): name of output to store present K value present_v (str): name of output to store present V value scale: scale before softmax Returns: Union[NodeProto, None]: the node created or None if failed. rrNTFr5)NNNzl is not an initializer. Please set do_constant_folding=True in torch.onnx.export to unblock attention fusionzInput hidden size (%d) is not same as weight matrix dimension of q,k,v (%d). Please provide a correct input hidden size or pass in 0rrqrdrrrzVMultiHeadAttention does not support relative_position_bias: cannot fuse the attention.rcr0rz.key_keyrrrr]rr!qkv_hidden_sizesrf)!rrrrFrsprintrrrrrvr concatenaterrAr_rGr9rrrr rBrrrrrrCrDrErffloat)5rr"rrrrrrr]r\rrrr rrrrr!has_biasrrrrrrrrr qw_in_size kw_in_size vw_in_sizeis_qkv_diff_dims qw_out_size kw_out_size vw_out_sizeqkv_weight_dimrrrrrr q_bias_shape k_bias_shape v_bias_shapeattention_node_nameattention_inputsattention_node past_existspast_kvattention_outputs present_kvs5 rcreate_attention_nodez%FusionAttention.create_attention_nodes\1}} ? i 7A= LLTVacl m =U]u}H::--hnnQ.?@::--hnnQ.?@::--hnnQ.?@!1 ZZ// A?m4::C]C]^c^i^ijk^lCmFZZ// A?m4::C]C]^c^i^ijk^lCmFZZ// A?m4::C]C]^c^i^ijk^lCmFf   >>!$%&gg   ! !( +  ! !( +  ! !( +xx288###XXa[ XXa[ XXa[ Z5:55555 ?{j8 NNJ  ! 88rxx #  ggbhhqrl+ ggbhhqrl+ ggbhhqrl+  R 1=J(;6DN2r2,Q7J_N &* %%f-B%%f-B%%f-B77288,L77288,L77288,L<>;> >> >>;. ..>>2r2,Q?+l:\I 88RRLq9 </ "jj99+F,,  (=8",, #n"56 !    (;6 **,'( !   ( ( uv"""#k1   % '' 3#--$'( N  ! !"6 7#m35=#k12   % '' 3 ''+ +VK..8 ''0"$++B/ '' 3!' Y&..vr:BB62NVVWZ\_` !((4 iJ?#--')( N  ! !+ . /  '')>)>{I)V(WX   $ $ + +V-B-BCSUV-W,X Y    $ $ + +V-B-B7E-R,S T   $ $ + +&&'9KVa;bcd   ! ! -  $ $ + +V-B-BCVX]^b^t^tXu-v,w xrc|}|}|jdk(rN|jj|dd}||}n,|jj|jd|}ny|jj |gdgd}d}| |\} } } } } n,|jj |gdgd}||\} }} } nyg} t |jD]L\}}||vr|jj|&||djdk(r<| j|Nt| dk7ry| d}|jj|d d}|h||jd}|3t|d k(r%|d}|jdk(r|jd}n[y|t|d k(r|jd}n:y|jd vr+||}|D]!}|jdk(s|jd}#||vr;||}|jd k(r't|jdk(r|jd}||}|Dcgc]}|j}}|jddk7ry|jj | gdgd}|tjdy|\} } }}d}d}d}d}gdgdfgdgdfgdgdfgdgdfgdgdfgdgdfd }d} |jD]K\}!}"|jj | |"d|"d} | ,|!d!k(rd"}n|!d#k(rd"}n|!d$k(rd"}n|!d%k(rd"}n| tjd&yd}#d}$d}%d}&|r| \} }%}$} n%|r| \} }#}%}$n|r| \} } }$n|r | \} }#}$}&} n| \} }#} }$|&xs|$}&|jj |&gdgd'}'|'9|jj |&gd(gd)}'|'tjd*y|'d+}(|'d,})|'d-}*|$}+|r>|jj |$d d.gddg},|,tjd/y|,\}+} |jj |+gd|rdnddddg}-|-9|jj |$gd0gd1}-|-tjd2y|-d,}.|-d-}/d}0d}1d}2d3}3|r7|jj|%gd4gdfgd5gdfgd6gd7fg|\} }0} n|r[|jj|%gd8gd7fgd5gdfg|\} }0} |#|j|#}3|3otjd9|#y|rnU|jj|#gd:gd;fgd<gd=fgd>gd?fgd@gdAfgdBgdAfgdCgdDfg|\}4}0} |4dk(}1|4d k(}2|s|2s|0tjdEy|s|2st|0dkDr|1s|jj!|0d\} }5|5<||jd};dI|;z}<|jAdJ|;ztBjDdgdd|8tG|9|8z gdK}=|jjItKjLdL|7jd|=j>g|r=r)r5rrNz&fuse_attention: failed to match v pathF)Softmaxr=Divr)rrNr)rCr=rAr)rCWhererrD)rrrpr)rCr=rEr)rrrrp)rCrDr)rrr)rCr=rrASqrt)rrNrr5)path1path2path3path4path5sdparITrJrKrLz'fuse_attention: failed to match qk path)rrrN)rDr?r>r=r)rrrrNz&fuse_attention: failed to match q pathrrrFz/fuse_attention: failed to match mul sqrt q path)r?r?r>r=r)r5rrrNz&fuse_attention: failed to match k pathr)Expandr>Equal)rPrr)CastrOr>rP)rrrr)rQrPrrz6fuse_attention: failed to verify shape inference of %s)rASubrQrr)Nrr5rr)rArRrr)Nrr5r)rErQrRrOrr)Nrrr5rr)rErQrRrQrOrr)Nrrr5rrr)rErOAndr>r>rFlatten) rErOGreaterOrEqualrrrRangerShape) Nrrrrrrr5rz)fuse_attention: failed to match mask pathizmFailed to detect num_heads and hidden_size for Attention fusion. Please specify those parameters in argument.) r"rrrrrrr]r\rrredge_modified_shape_modified_tensorr8r>reshape_modified_)+rr match_parentr>rsmatch_parent_path enumeraterrr&countrritemsmatch_parent_pathsrget_constant_inputrurvrwrxitemrfr^rQrrr;rrrr3rGr rIrrJr rBnodes_to_removerDr_ prune_graph)>rrinput_name_to_nodesoutput_name_to_nodenormalize_node start_nodeadd_before_layernormr einsum_noder reshape_qkv transpose_qkv matmul_qkv other_inputs_i node_input root_inputmul_before_layernorm mul_childrenlayernorm_nodechildrenchild parent_nodechildren_typesv_nodesadd_vmatmul_v is_distillis_distill_addis_no_mask_attentionis_sdpaqk_pathsqk_nodeskvr matmul_qkwhere_qkafter_qq_nodesr{add_qmatmul_qafter_k mul_k_nodesk_nodesadd_kmatmul_k mask_nodesis_transformers_5_maskis_transformers_5_no_maskr mask_pathmul_valr"attention_last_node q_num_heads q_hidden_sizenew_node unique_indexnew_edge shape_tensors> rfusezFusionAttention.fusexs1 #  ! !%9 9#'::#:#:>5RS#T #/1 ,,^-A-A!-DEQ , JJ00  ? !   =F :Q; z 44DoI$>G;K  ' (8(89 ,NB !44::..z:BYq\0033    + , |  ! !!_  $zz66z5!L  +./C/J/J1/MNL'C ,=,B!-a!))-AA!/!6!6q!9J)c,.?1.D188;   # #'W W*:6H! 1==$88!&aJ 1 , ,-j9K""&>>3{GYGYCZ^_C_(//2 &z25=>E%-->>    )Q . **..z;dfuv ? LLA B ")Auh $9?K9?K;\J;\J2I>@BTU  NN$ DAqzz33J!adKHG|! g!%g'+$f     LLB C   *2 'Q)Q /7 ,Q) ! ( Q9 19 .Q 7A(0 %Q9&Y**..w8acrs ?jj22@"G  EFBK  2; **66y5&/TUW[S\]K" NO&LWa**.. >gSTVWYZ\`@a  ?jj22F"G  EF 2; !&$)! #zz<<3Y?8)D;\J $  Az1#zz<<@,O8)D$   Az1!!008 %LL!Y[ab ! '+zz'D'DEGYZ=OQShiY[st^0 7 6$;( $Iz1>&/!^ "(1Q %#,E*J\ LLD E %-J!#*66z!}EJAw"7BJJ7GLLA>!  *x~~a/@J/NS[SaSabcSdhrSr,4M##00B1E1Ea1HI  2=1D+- )-)K)KI)V &Ka=A#5C11%!!!%)&*11!4%2 H    $ $X .:>:N:ND ( ( 7&*003 +l:#330<?)//Q S1L-MN 4   ##$$!,33A6 8I8IJ! +l:  (((0 !!!$  ' ')c&)rrFrrrrF)rFrrrrN)rRrSrTrUr rrboolrrVrrtuplerzrrrrrrr rr'r;r __classcell__)rls@rrYrYgs04).27&>@T%U%%% % &, % #' %,0%c%40 0eTWY\T\o0B,&y,&U3PS8_,&\Y* S 255S5S5n+Ks+KC+K#+KZ44    T  rsD .1== 8 SSlj$fj$r