jl d Z ddlZddlZddlZddlZddlmZ ddlmZ ddl m
Z
ddl m Z m
Z
mZmZmZmZmZmZ ddlmZmZ d d
lmZmZmZ ej6 e Z e rddl Z dd edd dd d
ddee e
f dee de dee!e f deee e f de deee fdZ"dd edd dd dee df dee e
f dee de dee!e f deee e f de deee ddfdZ#ejH eddee df de dee!e f defd Z%d ddee!ee
d f f fd Z&d ddeed ee!ee
d f f e!f fd!Z'd dde!fd"Z( e d2d# Z)d ddee!ee
d f f fd$Z* d3dee df dee e f de deee fd%Z+d dde!fd&Z,d'eee dee df deee fd(Z-dee df deee fd)Z.d dde fd*Z/ddd+dee df d,eee d-eee dee ee f fd.Z0 e d/d0de!fd1 Z1y)4z"Contains pytorch-specific helpers. N) defaultdict) lru_cache)Path)
TYPE_CHECKINGAnyDictListOptionalSetTupleUnion ) constantslogging )MAX_SHARD_SIZEStateDictSplit$split_state_dict_into_shards_factoryT)filename_patternforce_contiguousmax_shard_sizemetadatasafe_serializationshared_tensors_to_discardmodelztorch.nn.Modulesave_directoryr r r r r r c
F t | j ||||||| y)a
Saves a given torch model to disk, handling sharding and shared tensors issues.
See also [`save_torch_state_dict`] to save a state dict with more flexibility.
For more information about tensor sharing, check out [this guide](https://huggingface.co/docs/safetensors/torch_shared_tensors).
The model state dictionary is split into shards so that each shard is smaller than a given size. The shards are
saved in the `save_directory` with the given `filename_pattern`. If the model is too big to fit in a single shard,
an index file is saved in the `save_directory` to indicate where each tensor is saved. This helper uses
[`split_torch_state_dict_into_shards`] under the hood. If `safe_serialization` is `True`, the shards are saved as
safetensors (the default). Otherwise, the shards are saved as pickle.
Before saving the model, the `save_directory` is cleaned from any previous shard files.
If one of the model's tensor is bigger than `max_shard_size`, it will end up in its own shard which will have a
size greater than `max_shard_size`.
If your model is a `transformers.PreTrainedModel`, you should pass `model._tied_weights_keys` as `shared_tensors_to_discard` to properly handle shared tensors saving. This ensures the correct duplicate tensors are discarded during saving.
Args:
model (`torch.nn.Module`):
The model to save on disk.
save_directory (`str` or `Path`):
The directory in which the model will be saved.
filename_pattern (`str`, *optional*):
The pattern to generate the files names in which the model will be saved. Pattern must be a string that
can be formatted with `filename_pattern.format(suffix=...)` and must contain the keyword `suffix`
Defaults to `"model{suffix}.safetensors"` or `pytorch_model{suffix}.bin` depending on `safe_serialization`
parameter.
force_contiguous (`boolean`, *optional*):
Forcing the state_dict to be saved as contiguous tensors. This has no effect on the correctness of the
model, but it could potentially change performance if the layout of the tensor was chosen specifically for
that reason. Defaults to `True`.
max_shard_size (`int` or `str`, *optional*):
The maximum size of each shard, in bytes. Defaults to 5GB.
metadata (`Dict[str, str]`, *optional*):
Extra information to save along with the model. Some metadata will be added for each dropped tensors.
This information will not be enough to recover the entire shared structure but might help understanding
things.
safe_serialization (`bool`, *optional*):
Whether to save as safetensors, which is the default behavior. If `False`, the shards are saved as pickle.
Safe serialization is recommended for security reasons. Saving as pickle is deprecated and will be removed
in a future version.
shared_tensors_to_discard (`List[str]`, *optional*):
List of tensor names to drop when saving shared tensors. If not provided and shared tensors are
detected, it will drop the first name alphabetically.
Example:
```py
>>> from huggingface_hub import save_torch_model
>>> model = ... # A PyTorch model
# Save state dict to "path/to/folder". The model will be split into shards of 5GB each and saved as safetensors.
>>> save_torch_model(model, "path/to/folder")
# Load model back
>>> from huggingface_hub import load_torch_model # TODO
>>> load_torch_model(model, "path/to/folder")
>>>
```
)
state_dictr r r r r r r N)save_torch_state_dictr )r r r r r r r r s q/home/oem/CyberBrain26-unified/dev-env/.venv/lib/python3.12/site-packages/huggingface_hub/serialization/_torch.pysave_torch_modelr! # s. d ##%))%-%"; r z torch.Tensorreturnc t | }|"|rt j nt j }|i }|r ddlm} t | ||| } nddlm } t j d t | || }
t j |j d
d z } t! j" | D ]a } | j% | s t j' d | d
t! j( t j* j- || c ddi}
|
j0 s|
j3 | |rd|
ini }|
j4 j7 D ]X \ } }|D ci c] }|| |
}} ||t j* j- || fi | t j' d| Z |
j0 r|j d
dz }i |
j8 ||
j: d}t= t j* j- || d 5 }t? j@ ||d ddd t jC d| d tE |
j4 d | d t jC d | d y# t $ r} t
d | d} ~ ww xY w# t. $ r'} t j d| d| d Y d} ~ d} ~ ww xY wc c}w # 1 sw Y xY w)!ae
Save a model state dictionary to the disk, handling sharding and shared tensors issues.
See also [`save_torch_model`] to directly save a PyTorch model.
For more information about tensor sharing, check out [this guide](https://huggingface.co/docs/safetensors/torch_shared_tensors).
The model state dictionary is split into shards so that each shard is smaller than a given size. The shards are
saved in the `save_directory` with the given `filename_pattern`. If the model is too big to fit in a single shard,
an index file is saved in the `save_directory` to indicate where each tensor is saved. This helper uses
[`split_torch_state_dict_into_shards`] under the hood. If `safe_serialization` is `True`, the shards are saved as
safetensors (the default). Otherwise, the shards are saved as pickle.
Before saving the model, the `save_directory` is cleaned from any previous shard files.
If one of the model's tensor is bigger than `max_shard_size`, it will end up in its own shard which will have a
size greater than `max_shard_size`.
If your model is a `transformers.PreTrainedModel`, you should pass `model._tied_weights_keys` as `shared_tensors_to_discard` to properly handle shared tensors saving. This ensures the correct duplicate tensors are discarded during saving.
Args:
state_dict (`Dict[str, torch.Tensor]`):
The state dictionary to save.
save_directory (`str` or `Path`):
The directory in which the model will be saved.
filename_pattern (`str`, *optional*):
The pattern to generate the files names in which the model will be saved. Pattern must be a string that
can be formatted with `filename_pattern.format(suffix=...)` and must contain the keyword `suffix`
Defaults to `"model{suffix}.safetensors"` or `pytorch_model{suffix}.bin` depending on `safe_serialization`
parameter.
force_contiguous (`boolean`, *optional*):
Forcing the state_dict to be saved as contiguous tensors. This has no effect on the correctness of the
model, but it could potentially change performance if the layout of the tensor was chosen specifically for
that reason. Defaults to `True`.
max_shard_size (`int` or `str`, *optional*):
The maximum size of each shard, in bytes. Defaults to 5GB.
metadata (`Dict[str, str]`, *optional*):
Extra information to save along with the model. Some metadata will be added for each dropped tensors.
This information will not be enough to recover the entire shared structure but might help understanding
things.
safe_serialization (`bool`, *optional*):
Whether to save as safetensors, which is the default behavior. If `False`, the shards are saved as pickle.
Safe serialization is recommended for security reasons. Saving as pickle is deprecated and will be removed
in a future version.
shared_tensors_to_discard (`List[str]`, *optional*):
List of tensor names to drop when saving shared tensors. If not provided and shared tensors are
detected, it will drop the first name alphabetically.
Example:
```py
>>> from huggingface_hub import save_torch_state_dict
>>> model = ... # A PyTorch model
# Save state dict to "path/to/folder". The model will be split into shards of 5GB each and saved as safetensors.
>>> state_dict = model_to_save.state_dict()
>>> save_torch_state_dict(state_dict, "path/to/folder")
```
Nr ) save_filezjPlease install `safetensors` to use safe serialization. You can install it with `pip install safetensors`.)r r )savea You are using unsafe serialization. Due to security reasons, it is recommended not to load pickled models from untrusted sources. If you intend to share your model, we strongly recommend using safe serialization by installing `safetensors` with `pip install safetensors`.r r z(-\d{5}-of-\d{5})?)suffixz(\.index\.json)?zRemoving existing file 'z' from folder.z&Error when trying to remove existing 'z' from folder: z. Continuing...formatptr zShard saved to z .index.json)r
weight_mapwr )indentz:The model is bigger than the maximum size per checkpoint (z#). Model weighs have been saved in z^ checkpoint shards. You can find where each parameters has been saved in the index located at .z$Model weights successfully saved to !)#strr SAFETENSORS_WEIGHTS_FILE_PATTERN PYTORCH_WEIGHTS_FILE_PATTERNsafetensors.torchr% ImportError!_clean_state_dict_for_safetensorstorchr& loggerwarning"split_torch_state_dict_into_shardsrecompiler) oslistdirmatchdebugremovepathjoin Exception
is_shardedupdatefilename_to_tensorsitemsr tensor_to_filenameopenjsondumpinfolen)r r r r r r r r save_file_fnestate_dict_splitexisting_files_regexfilenameper_file_metadatasafe_file_kwargstensorstensorshard
index_pathindexfs r r r s \ (N "
6 677 C 7 -&?
/
c
:%5n
::&6&=&=EZ&=&[^q&qrJJ~. u % %h /
u 7z PQ "'',,~x@A u "4( & & *:L
$56RT - A A G G I 3':ABF++BBUBGGLL BWFVW xj123 ""%,,B,7-G
A+44AA*==
"'',,~z:C
@ *A IIeQq ) * H HX Y//23C3W3W/X.Y ZYYcXdde
g
KK6~6FaHIy E
@
u!GzQ`ab`ccr stt
u C * *sB J
A J'$
K K
J$ J J$' K0 KK K(r' c 2 t | ||t t S )a>
Split a model state dictionary in shards so that each shard is smaller than a given size.
The shards are determined by iterating through the `state_dict` in the order of its keys. There is no optimization
made to make each shard as close as possible to the maximum size passed. For example, if the limit is 10GB and we
have tensors of sizes [6GB, 6GB, 2GB, 6GB, 2GB, 2GB] they will get sharded as [6GB], [6+2GB], [6+2+2GB] and not
[6+2+2GB], [6+2GB], [6GB].
To save a model state dictionary to the disk, see [`save_torch_state_dict`]. This helper uses
`split_torch_state_dict_into_shards` under the hood.
If one of the model's tensor is bigger than `max_shard_size`, it will end up in its own shard which will have a
size greater than `max_shard_size`.
Args:
state_dict (`Dict[str, torch.Tensor]`):
The state dictionary to save.
filename_pattern (`str`, *optional*):
The pattern to generate the files names in which the model will be saved. Pattern must be a string that
can be formatted with `filename_pattern.format(suffix=...)` and must contain the keyword `suffix`
Defaults to `"model{suffix}.safetensors"`.
max_shard_size (`int` or `str`, *optional*):
The maximum size of each shard, in bytes. Defaults to 5GB.
Returns:
[`StateDictSplit`]: A `StateDictSplit` object containing the shards and the index to retrieve them.
Example:
```py
>>> import json
>>> import os
>>> from safetensors.torch import save_file as safe_save_file
>>> from huggingface_hub import split_torch_state_dict_into_shards
>>> def save_state_dict(state_dict: Dict[str, torch.Tensor], save_directory: str):
... state_dict_split = split_torch_state_dict_into_shards(state_dict)
... for filename, tensors in state_dict_split.filename_to_tensors.items():
... shard = {tensor: state_dict[tensor] for tensor in tensors}
... safe_save_file(
... shard,
... os.path.join(save_directory, filename),
... metadata={"format": "pt"},
... )
... if state_dict_split.is_sharded:
... index = {
... "metadata": state_dict_split.metadata,
... "weight_map": state_dict_split.tensor_to_filename,
... }
... with open(os.path.join(save_directory, "model.safetensors.index.json"), "w") as f:
... f.write(json.dumps(index, indent=2))
```
)r r get_storage_sizeget_storage_id)r get_torch_storage_sizeget_torch_storage_id)r r r s r r: r: s! F 0%)/+ r" rW .c 2 ddl m} | r' j \ }}t fd|D S j
j dk( r+t r!ddl}|j j }|S t }|S # t $ r Y \w xY w)zReturns a unique id for plain tensor
or a (potentially nested) Tuple of unique id for the flattened Tensor
if the input is a wrapper tensor subclass Tensor
r is_traceable_wrapper_subclassc 3 H K | ] }t t | y wN)_get_unique_idgetattr.0attrrW s r z!_get_unique_id..t s Q4 (=>Q "xlaN) torch.utils._python_dispatchrc __tensor_flatten__tupler5 devicetypeis_torch_tpu_available torch_xla_XLAC_xla_get_tensor_id storage_ptr)rW rc attrs_rt unique_ids ` r rf rf h s
N ( 0 002HE1Q5QQ Q 1 }}U"'='?
OO66v> '
s 4B
BBz torch.devicec x | j j dk( ry| j t | t | fS )a,
Return unique identifier to a tensor storage.
Multiple different tensors can share the same underlying storage. This identifier is
guaranteed to be unique and constant for this tensor's storage during its lifetime. Two tensor storages with
non-overlapping lifetimes may have the same id.
In the case of meta tensors, we return None since we can't tell if they share the same storage.
Taken from https://github.com/huggingface/transformers/blob/1ecf5f7c982d761b4daaa96719d162c324187c64/src/transformers/pytorch_utils.py#L278.
metaN)rq rr rf r_ )rW s r r` r` s5 }}V#}}nV 46LV6TTTr" c ddl m} | r' j \ }}t fd|D S j j
S # t $ r Y *w xY w# t $ ro j j t j z cY S # t $ r+ j t j z cY cY S w xY ww xY w)z
Taken from https://github.com/huggingface/safetensors/blob/08db34094e9e59e2f9218f2df133b7b4aaff5a99/bindings/python/py_src/safetensors/torch.py#L31C1-L41C59
r rb c 3 H K | ] }t t | y wre )r_ rg rh s r rk z)get_torch_storage_size.. s W-gfd.CDWrl )rn rc ro sumr5 untyped_storagenbytesAttributeErrorstoragesize_get_dtype_sizedtypeNotImplementedErrornelementrW rc rx ry s ` r r_ r_ s
N ( 0 002HE1WQVWW W 1 E%%'..00
E E>>#((*_V\\-JJ J" E ??$v||'DD D E Es@ 4A A' A$#A$' C 13B'$C '/CC CC c t j j d | r ddlmc m} |j
}yyy# t $ r Y yw xY w)z
Checks if `torch_xla` is installed and potentially if a TPU is in the environment
Taken from https://github.com/huggingface/transformers/blob/1ecf5f7c982d761b4daaa96719d162c324187c64/src/transformers/utils/import_utils.py#L463.
rt Nr TF) importlibutil find_spectorch_xla.core.xla_modelcore xla_model
xla_device RuntimeError) check_devicexmry s r rs rs sQ ~~ ,8
55MMO
s ? A
A c ddl m} || r t | S | j j S # t $ r Y *w xY w# t $ r2 | j j cY S # t $ r Y Y yw xY ww xY w)z
Taken from https://github.com/huggingface/safetensors/blob/079781fd0dc455ba0fe851e2b4507c33d0c0d407/bindings/python/py_src/safetensors/torch.py#L11.
r rb ) rn rc rf r5 r data_ptrrD r r )rW rc s r rw rw s
N ( 0!&) ) 1 %%'0022
>>#,,. ." s> ; A
AA
B A41B4 B=B BBc t | | }|j D ] \ }}|D ] }|i }||vr|||< | |= |r1| j D ci c] \ }} || j } }} | S c c} }w )a6 Remove shared tensors from state_dict and update metadata accordingly (for reloading).
Warning: `state_dict` and `metadata` are mutated in-place!
Taken from https://github.com/huggingface/safetensors/blob/079781fd0dc455ba0fe851e2b4507c33d0c0d407/bindings/python/py_src/safetensors/torch.py#L155.
)
discard_names)_remove_duplicate_namesrH
contiguous)
r r r r
to_removes kept_nameto_remove_group to_removekvs
r r6 r6 s )C\]J&0&6&6&8 &" ?( &I (&/#9% && 4>4D4D4FGDAqa 'G
G Hs A2c | j r:| j d d j t | j z }|S | j }|S )z
Taken from https://github.com/huggingface/safetensors/blob/079781fd0dc455ba0fe851e2b4507c33d0c0d407/bindings/python/py_src/safetensors/torch.py#L23.
)r viewr r r )rW stops r _end_ptrr sQ {{2r"++- 0MM K Kr" rV c g }| D ] }t | dk r|j | #g }|D ]2 }|| }|j |j t | |f 4 |j |d \ }}} |j | h |dd D ]4 \ }
} }|
|k\ r|j |h n|d j | | }6 |S )z
Taken from https://github.com/huggingface/safetensors/blob/079781fd0dc455ba0fe851e2b4507c33d0c0d407/bindings/python/py_src/safetensors/torch.py#L44
r r r Nr )rN appendr r sortadd) rV r filtered_tensorssharedareasnamerW ry last_stop last_namestartr s r _filter_shared_not_sharedr s v;? # #F + FD %F LL&//+Xf-=tD E F
"'( 9i ,!&qr E4 ! ''/ $((. I ( r" c ddl }t t }| j D ]v \ }}|j |j d k7 s&t | dk7 s5t
| dk7 sD||j t | t
| f j | x t t |j }t || }|S )z
Taken from https://github.com/huggingface/safetensors/blob/079781fd0dc455ba0fe851e2b4507c33d0c0d407/bindings/python/py_src/safetensors/torch.py#L69.
r Nr| ) r7 r setrH rq rw r_ r listsortedvaluesr )r r7 tensors_dictr r rV s r _find_shared_tensorsr $ s s#L " W1 88|u||F+ + A!0CH^_`HaefHf !((KN4J14MN O S STU VW 6,--/01G'.> s Mt|GFD$9:Mrl ) rn rc ro allr5 r rw r r r r_ r s ` r r r 4 s
N ( 0 002HE1MuMM M 1 ?? F 3 3 (8IO M 9 '9( (
s 4B BB)preferred_namesr r r c
. |g }t | }|g }t | }t | }t t }|D ] }t |D cg c] }t | | s| c} } | st d| d t
t | d }
| j | } | rt
t | d }
|r*|j | } | rt
t | d }
t
| D ] }||
k7 s ||
j | |S c c}w )r zvError while trying to find names to remove to save state dict, but found no suitable name to keep for saving amongst: z. None is covering the entire storage. Refusing to save/load the model since you could be storing much more memory than needed. Please refer to https://huggingface.co/docs/safetensors/torch_shared_tensors for more information. Or open an issue.r )
r r r r r r r
difference intersectionr ) r r r unique_preferred_namesunique_discard_namessharedsr r r complete_names keep_name preferreds r r r H s= 1
}-":.GD!I 2 v Xt jQUFV9Wd XY ((.x 0
4/03 #--.BC tI/2I !.;;NKI "4 ? 3A 6 6N 2Dy )$++D1 2328 7 Ys D
D
r z torch.dtypec D ddl }t |dd }t |dd }|j d|j d|j d|j
d|j d|j d|j d|j d|j d|j d|d|di }|| S ) z
Taken from https://github.com/huggingface/safetensors/blob/08db34094e9e59e2f9218f2df133b7b4aaff5a99/bindings/python/py_src/safetensors/torch.py#L344
r N
float8_e4m3fn float8_e5m2 r r ) r7 rg int64float32int32bfloat16float16int16uint8int8boolfloat64)r r7 _float8_e4m3fn _float8_e5m2_SIZEs r r r y s
UOT:N5-6L
Q
q
Q
q
Q
Q
A
A
qa
E <r" )T)TN)2__doc__r rK r= r; collectionsr functoolsr pathlibr typingr r r r r
r r r
r+ r r _baser r r
get_logger__file__r8 r7 r1 r intr! r r2 r: rf r` r_ rs rw r6 r r r r r r r" r r s ) # N N N ! W W
H % '+ !&4)- #59[ [#t)$[ sm [
[ #s(O
[ tCH~&[ [ (S 2[D '+ !&4)- #59XJS.()XJ#t)$XJ sm XJ
XJ #s(O
XJ tCH~&XJ XJ (S 2XJ
XJ| & F F&4 IS.()I I #s(O I
IX > eCsCx 4H.I @U UHU>SXY\^cdgildl^mYmSnpsCs=t4u U"E> Ec E6
( 5eCHo1E+F 8 "59 S.()38n (S 2 6^ tCH~ 4^H[C\ aefijmfnao 8
T#~*=%>
4C>
( (D (. ,0)- .S.(). d3i(. DI& .
#tCy. .b = S
r"