cQjltddlZddlmZddlZddlmZddlm Z m Z eje Z GddZy)N) __version__) DynamicCacheEncoderDecoderCacheceZdZdZedZeddefdZedZedZ ede e e jfdZ edd e e jd efd Zedde e e jfd Zy )PastKeyValuesHelperzEHelper functions to process past key values for encoder-decoder modelct|dk(r|d|d|d|dfSt|dk(sJdt||S)Nrz(Expected to have four or six items. Got )len) present_layers i/home/oem/CyberBrain26-unified/.venv/lib/python3.12/site-packages/onnxruntime/transformers/past_helper.py _unpack_layerz!PastKeyValuesHelper._unpack_layersa }  " #]1%5}Q7GWXIYY Y=!Q&g*RSVWdSeRf(gg&presentcg}g}t|D]L}|j|r d|d|gn d|d|g|j|r d|d|gn d|d|gN||zS) Npresent_key_self_present_value_self_past_key_self_past_value_self_present_key_cross_present_value_cross_past_key_cross_past_value_cross_)rangeextend) num_layersrpast_self_namespast_cross_namesis rget_past_namesz"PastKeyValuesHelper.get_past_namessz" A  " "%QC(,?s*CD&qc*.>qc,BC   # #&aS)-A!+EF's+/@-DE  !111rcg}g}t|D]E\}}tj|\}}}}|j||g|j||gG||fS)aSplit present state from grouped by layer to grouped by self/cross attention. Before: (past_key_self_0, past_value_self_0, past_key_cross_0, past_value_cross_0), (past_key_self_1, past_value_self_1, past_key_cross_1, past_value_cross_1), ... After: (past_key_self_0, past_value_self_0, past_key_self_1, past_value_self_1, ...), (past_key_cross_0, past_value_cross_0, past_key_cross_1, past_value_cross_1, ...)  enumeraterrr) present_key_values present_self present_cross_ipresent_layer_ipresent_key_selfpresent_value_selfpresent_key_crosspresent_value_crosss rgroup_by_self_or_crossz*PastKeyValuesHelper.group_by_self_or_cross,s  #,-?#@ K B $11/B   "!#   !13E F G  "35H!I J K]**rc*tdzk(sJtfdtD}tjt tjdk\r0t d|D}t d|D}t||S|S)aReorder past state from grouped by self/cross attention to grouped by layer. Before: past_key_self_0, past_value_self_0, past_key_self_1, past_value_self_1, ..., past_key_cross_0, past_value_cross_0, past_key_cross_1, past_value_cross_1, ... After: (past_key_self_0, past_value_self_0, past_key_cross_0, past_value_cross_0), (past_key_self_1, past_value_self_1, past_key_cross_1, past_value_cross_1), r c3~K|]4}d|zd|zdzdzd|zzdzd|zzdzg6yw)r N).0r!rpasts r z5PastKeyValuesHelper.group_by_layer..Gsf  QU QUQYQ^a!e+,Q^a!e+a/0   s:=z5.0c30K|]}|d|dfyw)rr Nr3r4layers rr6z5PastKeyValuesHelper.group_by_layer..Qs/Yq580D/Yc30K|]}|d|dfyw)r2r Nr3r8s rr6z5PastKeyValuesHelper.group_by_layer..Rs0Z%%(E!H1E0Zr:)r tuplerpvVersiontransformers_versionrr)r5rlayersself_attention_cachecross_attention_caches`` rgroup_by_layerz"PastKeyValuesHelper.group_by_layer@s 4yA N*** :&   ::* +rzz%/@ @#//YRX/Y#Y $00ZSY0Z$Z !&';=RS S rpast_key_valuescd}t|dz}tt|dzD])}d|z}|||||dz|||z|||zdzffz }+|S)aCategorize present_key_values from self and cross attention to layer by layer. Reorder past state from grouped by self/cross attention to grouped by layer. Before: past_key_self_0, past_value_self_0, past_key_self_1, past_value_self_1, ..., past_key_cross_0, past_value_cross_0, past_key_cross_1, past_value_cross_1, ... After: (past_key_self_0, past_value_self_0, past_key_cross_0, past_value_cross_0), (past_key_self_1, past_value_self_1, past_key_cross_1, past_value_cross_1), Args: present_key_values: From past_key_values of a model (group by self and cross attention) Returns: past_tuples: present key and values grouped by layer. r3r2r r )r r)rD past_tupleshalf_idxr!idxs rback_group_by_layerz'PastKeyValuesHelper.back_group_by_layerVs  '1,s?+q01 Aa%C #C(#C!G,#HsN3#HsNQ$67  K rr&concatcg}g}t|D]E\}}tj|\}}}} |j||g|j|| gG|r||zS||fS)aCategorize present_key_values into self and cross attention. Split present state from grouped by layer to grouped by self/cross attention. Before: (past_key_self_0, past_value_self_0, past_key_cross_0, past_value_cross_0), (past_key_self_1, past_value_self_1, past_key_cross_1, past_value_cross_1), ... After: (past_key_self_0, past_value_self_0, past_key_self_1, past_value_self_1, ...), (past_key_cross_0, past_value_cross_0, past_key_cross_1, past_value_cross_1, ...) Args: present_key_values: From past_key_values of a model (group by layer) concat: If concat self attention with cross attention key/value to return Returns: present_self (Tuple[torch.Tensor]): present key and values from self attention present_cross (Tuple[torch.Tensor]): present key and values from cross attention r$) r&rJr'r(_r*r+r,r-r.s rgroup_by_self_and_crossz+PastKeyValuesHelper.group_by_self_and_crossts$,. ,. "+,>"? K A#11/B Y 02CEX   !13E F G  "35H!I J  K -/ /. .rcHg}|rt|dzn t|}|sdnd}t|D],}|jd|d|fDcgc]}||z c}.t|D],}|jd|d|fDcgc]}||z c}.|Scc}wcc}w)zProcess input names of model wrapper. Args: past_key_values: Consider `self` and `cross` past_key_values Returns: names (List[string]): input names r past_present_ key_self_ value_self_ key_cross_ value_cross_)r rr)rDencodernamesrprefixr!ss rget_input_namesz#PastKeyValuesHelper.get_input_namess29S)Q.s??S 'Zz" UA LL1#+aS@Q.RS&1*S T Uz" WA LLA3/?ris4 <F   8 $SSr