rPj= UddlZddlZddlZddlZddlZddlmZddlmZm Z ddl m Z ddl m Z ddlmZddlmZmZmZddlmZdd lmZmZdd lmZmZmZdd lmZmZddlZddl Z dd l!m"Z"m#Z#m$Z$dd l%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,m-Z-m.Z.m/Z/ddl0m1Z1m2Z2ddl3m4Z4m5Z5ddl6m7Z7ddl8m9Z9m:Z:m;Z;mm?Z?m@Z@mAZAddlBmCZCmDZDmEZEmFZFmGZGddlHmIZImJZJmKZKmLZLmMZMmNZNddlOmPZPddlQmRZRmSZSddlTmUZUeGddZVeGddeVZWeGd d!eVZXeYejZ[ee\d"<d#eFeGze1ze2zd$e]d%e^e]effd&Z_d'Z`Gd(d)ZaGd*d+e[ZbGd,d-ejeb.ZcGd/d0ejjZfGd1d2ejjZgd3e]d%e]fd4Zhd5e]d%e]fd6Zid7e]d8ejdzd%efd9Zkd:ecd7e]d;e]dZmd?ZnecZoy)@N)deque)IterableIterator)ThreadPoolExecutor) ExitStack)deepcopy) dataclassfieldreplace)datetime)Path PurePosixPath)AnyNoReturnUnion)quoteunquote)_DEFAULT_CALLBACK NoOpCallback TqdmCallback) apply_config) isfilelike) constants)BUCKET_FOLLOWERSBUCKET_FOLLOWERS_LOCKBucketFileChangeBucketFollower) BucketFile BucketFolder)CommitOperationCopyCommitOperationDelete)_validate_relative_filename)BucketNotFoundErrorEntryNotFoundErrorHfHubHTTPErrorRepositoryNotFoundErrorRevisionNotFoundError) hf_hub_urlhttp_getxet_get)SPECIAL_REFS_REVISION_REGEXHfApiLastCommitInfoRepoFile RepoFolder)HFValidationError XetFileDatahf_raise_for_status http_backoffhttp_stream_backoff parse_hf_uri)is_xet_available) XetTokenTypexet_connection_info_refresh_url)md5c2eZdZUdZeed<eed<defdZy)HfFileSystemResolvedPathz_Top level Data structure containing information about a resolved Hugging Face file system path.rootpathreturncV|jd|jjdS)N/)r=r>rstripselfs c/home/oem/CyberBrain26-unified/.venv/lib/python3.12/site-packages/huggingface_hub/hf_file_system.py unresolvez"HfFileSystemResolvedPath.unresolve2s&))Adii[)0055N)__name__ __module__ __qualname____doc__str__annotations__rFrGrEr<r<+si I I636rGr<ceZdZUdZeed<eed<eed<eed<edZeed<edZeed <ed d Z ed zed <d Z y )"HfFileSystemResolvedRepositoryPathzLData structure containing information about a resolved path in a repository. repo_typerepo_idrevision path_in_repoFinitr=r>N)defaultrepr _raw_revisioncbtjj|jd|jz}|j r|d|j |_nD|jtjk7r |dt|j|_n||_|j|_ y)N@) rREPO_TYPES_URL_PREFIXESgetrQrRrYr=rSDEFAULT_REVISION safe_revisionrTr>)rD repo_paths rE __post_init__z0HfFileSystemResolvedRepositoryPath.__post_init__Ds5599$.."MPTP\P\\   $+Qt'9'9&:;DI ]]i88 8$+Q}T]]'C&DEDI!DI%% rG) rHrIrJrKrLrMr r=r>rYrbrNrGrErPrP6sTVN LM5!D#!5!D#!!&d ?M3:?&rGrPc>eZdZUdZeed<edZeed<dZy)HfFileSystemResolvedBucketPathzHData structure containing information about a resolved path in a bucket. bucket_idFrUr=c,d|jz|_y)Nbuckets/)rer=rCs rErbz,HfFileSystemResolvedBucketPath.__post_init__Vs/ rGN) rHrIrJrKrLrMr r=rbrNrGrErdrdOsRN5!D#!0rGrd _cached_base path_info cache_pathr?c t|trG||jd|j|j|j |j |jdSt|tr1||jd|j |j|jdSt|tr|dd|j|j dS|dd|jdS)zNConvert an API path info into the dict format used in `HfFileSystem.dircache`.filenamesizetypeblob_idlfsxet_hash last_commitsecurity)rnrorprsmtime uploaded_atr directoryrnrorptree_idrt)rnrorprw) isinstancer/rorqrrrsrtrurrvrwr0rz)rirjs rE_cache_info_from_path_infor|^s)X&NN ((==!**$00!**  )Z(NN!**__$00   )Z( (($00    ,,  rGceZdZdZdddededzfdZded ee e e ffd Z ded ed dfd Z ded ed dfd Zded ed dfdZde e efded efdZded ed dfdZded edeeezd dfdZy)_ParallelRepoTreeListingahList a repo tree recursively, listing independent directories in parallel. The Hub `tree` endpoint returns at most 1000 entries per request (50 when `expand=True`) and pagination is cursor-based: a page can only be fetched once the previous one has been received. Listing a repo with 800k files through a single request therefore requires ~800 *sequential* requests, which takes many minutes. To work around this, the listing is split into independent sub-trees listed concurrently by a thread pool. Since sub-directories can only be known by listing their parent, a sub-tree is split lazily while it is being paginated: - A sub-tree that fits in a single page is kept as-is: it costs one request, exactly like it would without parallelization. Small sub-trees are therefore never split (no request is wasted). - Once `_MIN_PAGES_BEFORE_SPLIT` pages have been fetched without reaching the end of the sub-tree, the sub-tree is split if (and only if) the sub-directories seen so far hold enough entries to "pay" for their own request (see `_is_worth_splitting`). In that case the directory is listed one level deep, and each of its sub-directories is dispatched to a new task applying the same logic recursively. If splitting is not worth it, the pagination is simply followed to its end. The number of requests stays close to the theoretical minimum (one request per 1000 entries) while independent directories are listed concurrently. Listing `datasets/allenai/objaverse` (~800k files spread in 160 sub-directories) goes from ~30 minutes to ~30 seconds. fs HfFileSystem expand_infomaxdepthNc||_||_||_t|jd|_t j|_d|_ d|_ g|_ i|_ y)Nz hf-ls-tree) max_workersthread_name_prefixr) rrrrrpool threading Conditionlock pending_taskserroroutlistings)rDrrrs rE__init__z!_ParallelRepoTreeListing.__init__sX&  &2>>Vbc '') +/ )+9; rG resolved_pathr?c|j|d |j5|jr'|jj|jr'ddd|j jdd|j |j |jS#1swYJxYw#t$r}|j xs||_d}~wwxYw#|j jddwxYw)zRList the full tree below `resolved_path` and return the path infos (in any order).r)depthNT)waitcancel_futures) _submitrrr BaseExceptionrrshutdownr)rDrexcs rErunz_ParallelRepoTreeListing.runs ]! , ? %((IINN$(( % II  D  > :: !** xx % % *sDJ   II  D  >s: B*3BB*B'#B** C3C  CCC0rc|j5|j dddy|xjdz c_ddd|jj |j ||y#1swY1xYw)z\Dispatch the listing of a sub-tree to the thread pool (no-op if the listing already failed).Nr)rrrrsubmit _run_task)rDrrs rErz _ParallelRepoTreeListing._submitsd YY $zz% $ $   ! #  $ >  $ $sA)A))A2cl |j5|jdu}ddds|j|||j5|xjdzc_|jj dddy#1swYbxYw#t$r?}|j5|j||_dddn #1swYnxYwYd}~d}~wwxYw#1swYyxYw#|j5|xjdzc_|jj dddw#1swYwxYwxYw)Nr)rr _list_subtreerr notify_all)rDrrfailedrs rErz"_ParallelRepoTreeListing._run_tasks ' 04/ 0""=%8  '""a'" $$& ' ' 0 0 % %::%!$DJ % % % %  ' ' '""a'" $$& ' ' 's{ B A>B 0C>BB C C B<3 C <C C C! CC!C! D3.0D' D3'D0 ,D3c^|j"||jk\r|j||yg}i}d}|jr|jdznd}|jjj |j |jxsdd|j|j|j}t|dD]\}\} } |jy|j| t|t| }| D]d} | jj!|s| jt|dj#d\} } }| sM|j%| ddz|| <f| r|t&k\s|j)||s|j||y|j+|||y) zList a single sub-tree, splitting it in sub-directories if it is worth it. `depth` is the depth (relative to the root of the listing) of the direct children of `resolved_path`. NrrAr[T recursiveexpandrSrQr)start)r_list_childrenr>r_api_iter_repo_tree_pagesrRrTrrSrQ enumeraterextendmaxlen startswith partitionr^_MIN_PAGES_BEFORE_SPLIT_is_worth_splitting_record)rDrrentriesentries_per_folder max_page_sizeprefixpagespage_nbpagefinishedrifoldersep_s rErz&_ParallelRepoTreeListing._list_subtrees == $$--)?    u 5 /1-/ -:-?-?##c)R 22  ! !  & & .$##"++#-- 3 *35)B  %G%dHzz% NN4  s4y9M! W  ~~008!*F !>!H!H!MQ1C1G1GPQ1RUV1V&v.  W66,,-?O##M59# ( ]E73rGr page_sizeclt|xr(t|j|t|zk\S)zNSplit only if sub-directories hold at least a full page of entries on average.)boolsumvaluesr)rDrrs rErz,_ParallelRepoTreeListing._is_worth_splitting s4&'sC0B0I0I0K,LPY\_`r\sPs,ssrGc t|jjj|j|j xsdd|j |j|j}|j||||j|dz|jkDry|D]=}t|ts|jt||j|dz?y)zList the direct children of a directory and dispatch each sub-directory to a new task. Sub-directories are only dispatched as long as their content is at a depth that is still requested (see `maxdepth`). NFrrrT)listrrlist_repo_treerRrTrrSrQrrr{r0rr r>)rDrrchildrenris rErz'_ParallelRepoTreeListing._list_childrens  GGLL ' '%%**2d''&//'11 (    ]E84 == $T]])B ! ]I)Z0 W]PRWZ[R[\ ]rG path_infosc|j}g}i}|D]}|jdz|jz}t||} ||k7r:|j |j j |gj| ||t|djdzdz } |j| |jks|j| |j5|jj||jD]0\} } |jj | gj| 2 dddy#1swYyxYw)zHStore freshly listed entries in the (shared) output and directory cache.rANr)rFr=r>r| setdefaultr_parentappendrcountrrrritemsr) rDrrrr>rrrirjcache_path_info entry_depthparent path_infos_s rErz _ParallelRepoTreeListing._record%s4&&($&46# ,I&++c1INNBJ8JOOT!##DGGOOJ$?DKKO\*SY["9"?"?"DDqHK}}$ t}}(D ?+ ,YY I HHOOC '/~~'7 I#  ((4;;KH I I I Is AEE)rHrIrJrKrintrrPrdictrLrrrrrrrrr/r0rrNrGrErrs$0 <> < <d <!CTRUWZRZ^H\&?%G?PS?X\? ''I 'RU 'Z^ '.4+M.4VY.4^b.4`td38ntQTtY]t],N]WZ]_c].I9IIX 23 I  IrGrc(eZdZdZfdZdZxZS)_Cacheda Metaclass for caching HfFileSystem instances according to the args. This creates an additional reference to the filesystem, which prevents the filesystem from being garbage collected when all *user* references go away. A call to the :meth:`AbstractFileSystem.clear_instance_cache` must *also* be made for a filesystem instance to be garbage collected. This is a slightly modified version of `fsspec.spec._Cached` to improve it. In particular in `_tokenize` the pid isn't taken into account for the `fs_token` used to identify cached instances. The `fs_token` logic is also robust to defaults values and the order of the args. Finally new instances reuse the states from sister instances in the main thread. c2t||i|i|_yN)superr_cache)clsargskwargs __class__s rErz_Cached.__init__Ms $)&) rGct||}|jdd}|j|tjg|i|}|j|tj j g|i|}|s0|jr$||jvr||_ |j|Stj|g|i|}|s\|jrP||jvrB|j|j}|jD]\}} t||| ||_||_||_|jr|s||_ ||j|<|S)Nskip_instance_cacheF)rpop _tokenizer get_ident main_threadidentcachabler_latestrp__call___get_instance_statersetattr _fs_token_ storage_argsstorage_options) rrrskipfs_tokenfs_token_main_threadobjinstance_stateattr state_values rErz_Cached.__call__UsHc6*zz/7 3==i&9&9&;MdMfM,s}}S)2G2G2I2O2OaRVaZ`a SZZ)?"CK::h' '--5d5f5CCLL-ASZZ-O!$,@!A!U!U!W)7)=)=)?4%D+C{34%CN#C "(C ||D& '* 8$JrG)rHrIrJrKrr __classcell__rs@rErr=s rGrc&eZdZdZdZdZddddddddedzdeezdzd edzd edzd ed edzf fd Z e dedefdZ dedededzde ee dzffdZdede ee dzffdZ dBdededzdeezfdZdBdedzddfdZdeddfdZdeddfdZdedeeddfdZdeddfdZ dCdeded edzdedzded f d!ZdBdededzddfd"Z dDded#ed$edzdedzddf d%Z dEded&ed'ededzdeeeee fzf d(Z! dFded#ed'ededzd edzd$edzf d)Z"deddfd*Z#ded+ed#edee$e%zfd,Z&dede'e eeeeefffd- Z(dBded$edzdeeffd. Z) dGded$edzd/ed&ed'ededzdeeeeeee ffzfd0Z*dBd1ed2ededzddfd3Z+dede,fd4Z-dHded'ededzdeee ffd5Z.d6Z/d7Z0d8Z1dedefd9Z2d:eezd;eee fde3dzfd<Z4de5dfdIfd= Z6e7d>Z8d?Z9d@Z:dAZ;xZ [!WARNING] > [`HfFileSystem`] provides fsspec compatibility, which is useful for libraries that require it (e.g., reading > Hugging Face datasets directly with `pandas`). However, it introduces additional overhead due to this compatibility > layer. For better performance and reliability, it's recommended to use `HfApi` methods when possible. The file system supports paths for the `hf://` protocol, which follows those URL schemes: * Models, Datasets and Spaces repositories: ``` hf://[@]/ hf://datasets/[@]/ hf://spaces/[@]/ ``` * Buckets (generic storage): ``` hf://buckets// ``` Note: when using the [`HfFileSystem`] directly, passing the `hf://` protocol prefix is optional in paths. Args: endpoint (`str`, *optional*): Endpoint of the Hub. Defaults to . token (`bool` or `str`, *optional*): A valid user access token (string). Defaults to the locally saved token, which is the recommended method for authentication (see https://huggingface.co/docs/huggingface_hub/quick-start#authentication). To disable authentication, pass `False`. block_size (`int`, *optional*): Block size for reading and writing files. expand_info (`bool`, *optional*): Whether to expand the information of the files. max_workers (`int`, *optional*): Maximum number of concurrent requests used to list large directories in parallel (see [`~HfFileSystem.find`]). Defaults to 16. Set it to 1 to disable parallel listings. live_follow (`bool`, *optional*): Whether to follow file changes in buckets as they happen on the Hub, so that cached listings stay fresh without manual [`HfFileSystem.invalidate_cache`] calls. When enabled, listing a bucket opens a background connection to the Hub event feed for that bucket. Has no effect on model, dataset and space repositories (the Hub does not feed changes for those). **storage_options (`dict`, *optional*): Additional options for the filesystem. See [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.__init__). Usage: ```python >>> from huggingface_hub import hffs >>> # List files >>> hffs.glob("my-username/my-model/*.bin") ['my-username/my-model/pytorch_model.bin'] >>> hffs.ls("datasets/my-username/my-dataset", detail=False) ['datasets/my-username/my-dataset/.gitattributes', 'datasets/my-username/my-dataset/README.md', 'datasets/my-username/my-dataset/data.json'] >>> # Read/write files >>> with hffs.open("my-username/my-model/pytorch_model.bin") as f: ... data = f.read() >>> with hffs.open("my-username/my-model/pytorch_model.bin", "wb") as f: ... f.write(data) ``` Specify a token for authentication: ```python >>> from huggingface_hub import HfFileSystem >>> hffs = HfFileSystem(token=token) ``` r[hfN)endpointtoken block_sizerr live_followrrrrrrct ||i||xstj|_||_t |||_||_||_ ||_ ||_ t|_ i|_i|_i|_y)N)rr)rrrENDPOINTrrr-rrrrrr_pending_bucket_changes_repo_and_revision_exists_cache_bucket_exists_cachedircache) rDrrrrrrrrrs rErzHfFileSystem.__init__s $2/2 6I$6$6  859 $&&&Y^Y`$rt,NP!:< rGthreading_identr?c(|jdxstj|d<|jd|d<t|Dcic]}||| }}||||f}t t |j }|jScc}w)zDeterministic token for cachingrr)r^rrsortedr:rLencode hexdigest)rrrrkey tokenize_argshs rErzHfFileSystem._tokenizes$ZZ 3Iy7I7Iz **W-w.4Vn=s#vc{"==otV< M"))+ ,{{} >s BrQrRrSc|||f|jvrR |jj|||tjd|j|||f<d|j||df<|j|||fS#t t f$r2}d|f|j|||f<d|f|j||df<Yd}~Nd}~wt$r0}d|f|j|||f<d|j||df<Yd}~d}~wwxYw)N)rSrQtimeoutTNF)rr repo_inforHF_HUB_ETAG_TIMEOUTr'r1r()rDrQrRrSes rE_repo_and_revision_existz%HfFileSystem._repo_and_revision_exists! w )1U1U U ^ ##h)YMjMj$Xb44i(5STS]44i$5OP33Y4RSS,->? \W\^_W_44i(5STSXZ[S[44i$5OP( ^W\^_W_44i(5STS]44i$5OP ^s#-A66C.(B22 C.>&C))C.rec||jvr+ |jj|d|j|<|j|S#t$r}d|f|j|<Yd}~.d}~wwxYw)Nr F)rr bucket_infor$)rDrers rE_bucket_existszHfFileSystem._bucket_existsst D55 5 B %%i08B)))4((33 ' @7c|j|}|s td|jddk(rtd|dt t j |}|jrM|j|j\}}|s t||t|j|jS||j|j|k7r|jddt jvr|jdddn|}|jd d\}}d|vr|jddn|d f\} } t!| } | |k7rtd | d |d |j#|j$||\} }| s t||t'|j$||| | S|j| |j}|j#|j$|j|\} }| s t||d} d |vr|jz|jddt jvr|jdddn|}|jd dd}|jr|dt)|jdz n|} ||nt j*}t'|j$|j||j| S)a Resolve a Hugging Face file system path into its components. Args: path (`str`): Path to resolve. revision (`str`, *optional*): The revision of the repo to resolve. Defaults to the revision specified in the path. Returns: [`HfFileSystemResolvedPath`]: Resolved path information containing `repo_type`, `repo_id`, `revision` and `path_in_repo`. Raises: `ValueError`: If path contains conflicting revision information. `NotImplementedError`: If trying to list repositories. zNrr\r[zRevision specified in path ("z ") and in `revision` argument ("z") are not the same.)rY)_strip_protocolNotImplementedErrorr ValueErrorr6r HF_PROTOCOL is_bucketrid_raise_file_not_foundrdrTrSsplitHF_URI_TYPE_PREFIXESrrrprPrr_)rDr>rSparsed bucket_existserrpath_without_typerRafter_atrevision_in_pathrTrevision_in_path_decodedrepo_and_revision_exist raw_revision raw_after_ats rE resolve_pathzHfFileSystem.resolve_paths*##D)%&de e ::c?a ?vFBC !6!6 7v>?   !%!4!4VYY!? M3 %dC01FIIFL_L_` `  FOO$?FOOW_D_ :>C9KyOmOm9m 3 21 5sw  1 7 7Q ? GXGJhX^^C-C]egi\j * l'./?'@ $'83 34L3MMmnvmwxLM,0+H+HV]_g+h ( #S*%dC05 Wh L\  ?? &8+;H'+'D'DV[[RXR[R[]e'f$& !$ ,$( $;6??69=C9KyOmOm9m 3 21 5sw ,223:1=LNTNaNa<(IC0C0C,Dq,H*IJgsL'389S9S1 KKHf.A.AQ]  rGc|sO|jj|jj|jjy |j |}|j}|j||jst|trq|jj|j|jdfd|jj|j|j|jfdy|jj|j dyy#t $r|jj Dcic]\}}|ds ||ncc}}wc}}|_|jj Dcic]\}}|ds ||ncc}}wc}}|_YywxYw)ac Clear the cache for a given path. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.invalidate_cache). Args: path (`str`, *optional*): Path to clear from cache. If not provided, clear the entire cache. rN)rclearrrr)FileNotFoundErrorrrF_invalidate_dircache_ancestorsr>r{rPrrQrRrSre)rDr>rkvs rEinvalidate_cachezHfFileSystem.invalidate_cacheWs MM   !  0 0 6 6 8  % % + + -  $ 1 1$ 7 !**,D  / / 5!%%m-OP88<<&00-2G2GNPT88<<&00-2G2GI_I_`bf--11-2I2I4P&% &*%I%I%O%O%Q8!QUVWXUYAqD884?C>W>W>]>]>_,hdacdefcgQT,h,h)  s6D::&G E4.E43-G F4.F43GGcj|r1|jj|d|j|}|r0yy)zZDrop the cached listing of a path (unresolved) and of every one of its parent directories.N)rrrrDr>s rEr-z+HfFileSystem._invalidate_dircache_ancestorss, MM  dD )<<%DrGcd|}|jDcgc]}||k(s|j|dzs| c}D]}|jj|d ycc}w)z0Drop every cached listing belonging to a bucket.rgrAN)rrr)rDrer=r> cached_paths rE_invalidate_bucket_treez$HfFileSystem._invalidate_bucket_trees[)%-1]]jTddldoo^beh^hNiDj 1K MM  k4 0 1js AAchangesc d|}|D]t}|d|j}|jDcgc]}|j|dzs|c}D]}|jj|d |j |vycc}w)a8 Apply a batch of live-follow changes to the dircache. Called by `_ls_tree`, on the thread owning the instance. A change is applied as an invalidation of the cached listings of every parent directory of the changed path (up to the bucket root), plus the subtree under that path in case it was itself listed as a directory. Dropping a cached listing is always safe: the next call to [`HfFileSystem.ls`] re-lists just that directory. Directories that were never listed need nothing, their first listing is fresh. rgrAN)r>rrrr-)rDrer6r=changer>cachedr4s rE_apply_bucket_changesz"HfFileSystem._apply_bucket_changess)% 6FV1V[[M*D59]]d6fFWFWX\_bXbFcd 5  !!+t4 5  / / 5  6ds BBc *|j|jf}t5|tvrDt|dxr6t |vr|t |<|j  dddyt jt ||idttdzddffd t|j|jfdfd}|ft|<|j|j dddy#1swYyxYw) zjSubscribe to the live-follow thread of a bucket, shared by all instances with the same endpoint and token.rNr6r?ct5tjD]>\}}t|jdk\r|="|jj |f@ dddy#1swYyxYw)Ni')rrrrrr)r6fs_idrre subscriberss rEpushz2HfFileSystem._ensure_bucket_follower..pushsv*T%)+*;*;*=%>T rr99:fD +E 266==y'>RS TTTTs AA--A6cdSrrN)r?srEz6HfFileSystem._ensure_bucket_follower..s T$ZrGc tdkDS)Nr)r)r>srErAz6HfFileSystem._ensure_bucket_follower..s[!1A!5rG)rrer on_changes on_reconcileis_alive) rrrrrr5weakrefWeakValueDictionaryrrrr)rDrerfollowerr?r>s ` @@rE_ensure_bucket_followerz$HfFileSystem._ensure_bucket_followers}}djj)4 " 4&&;KC;PQR;S,SK,Sd8;.,0K4)00;  4 4"55r$x6FGK Td#34t; T T&#jj/5 H&.{$; S ! NN   ( ( 3= 4 4 4sAD 0BD  Dmode)HfFileSystemFileHfFileSystemStreamFilec ||n |j}|||d<d|vr td|dk(rt||f||d|St||f||d|S)Nraz/Appending to remote files is not yet supported.r)rJrS)rrrLrK)rDr>rJrrSrs rE_openzHfFileSystem._opensr$.#9Zt  !#-F< $;%&WX X ?)$]4(]V\] ]#D$WTHWPVW WrGc |j||}t|tr3|jj |j |j gnr|jj|j|j|j|j|j|jd|jd|j|jy)NrSdeletecommit_messagecommit_description)rTrRrrQrSrTrUr>)r)r{rdrbatch_bucket_filesrer> delete_filerTrRrrQrSr^r0rF)rDr>rSrrs rE_rmzHfFileSystem._rms))$)B m%C D II ( ()@)@-J\J\I] ( ^ II ! !*77%--jj'11&//%zz*:;#)::.B#C "  =#:#:#<=rGrrc F|j||}|j||||}t|trb|Dcgc]/}|j |r|j|j 1}}|j j|j|n|Dcgc]/}|j |r|j|j 1} }| D cgc]} t| } } dd} | |rdndz } | |d |dndz } |j j|j|j|j| |j|jd | |jd  |j!|j#y cc}wcc}wcc} w)a Delete files from a repository. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.rm). > [!WARNING] > Note: When possible, use `HfApi.delete_file()` for better performance. Args: path (`str`): Path to delete. recursive (`bool`, *optional*): If True, delete directory and all its contents. Defaults to False. maxdepth (`int`, *optional*): Maximum number of subdirectories to visit when deleting recursively. revision (`str`, *optional*): The git revision to delete from. rQ)rrrSrRrzDelete  z recursively r[Nz up to depth rTrU)rRrQr operationsrSrTrUrV)r) expand_pathr{rdisdirr>rrWrer" create_commitrRrQrrSr^r0rF) rDr>rrrSrrpathsrS paths_in_reporTr\rTs rErmzHfFileSystem.rms6))$)B   XX` a m%C D?D]tDJJW[L\d''-22]F] II ( ()@)@ ( PFKddSWS]S]^bScT..t499dMd_lm|/\JmJm&tfA.N  nr AN H

[!WARNING] > Note: When possible, use `HfApi.list_repo_tree()` for better performance. Args: path (`str`): Path to the directory. detail (`bool`, *optional*): If True, returns a list of dictionaries containing file information. If False, returns a list of file paths. Defaults to True. refresh (`bool`, *optional*): If True, bypass the cache and fetch the latest data. Defaults to False. revision (`str`, *optional*): The git revision to list from. Returns: `list[Union[str, dict[str, Any]]]`: List of file paths (if detail=False) or list of file information dictionaries (if detail=True). rQ)rdrSNrnr)r)rF_ls_treer%r>rrr) rDr>rcrdrSrrros rElszHfFileSystem.lss4))$)B &&( 2$--SgSFSCs:c":1V9"::" 2 %%%dD1 #dmmDLL$6eRZe^de%  !7QvY$%6177C73x1}%dD1 2#;sMA C)!C&4%BC& B(%C&'B((C&/ C=C"C&%C&c : ||n|j |jnd}|j||}|j}|j}|r|dk(rd}|r|nd}|jrt |t r|j|j|jrf|jjx rdk(r\c} |j| n  rdk(r\} } |j| | n|jrfg} ||jvr|s|j|} | j| g} |rd}t| Dcgc]}|ddk(s ||fc}}|r|j\}}|||kru|d|jvr| j!|dnO|j|d} | j| |j| Dcgc]}|ddk(r|dz|fc}|rg}|r7t |t"r'| Dcgc]}|d |j%|d}}|r| s|r|rt'j(| |z}|"|t+|dj-d }||z}| Dcgc]}|dj/|d zr|} }t1|jD]3}|j/|d zs|jj3|d5|jj3|d| j|j5||d ||| | St |t r)|j7|j|j8| }n|r|j:dkDrvt=||| }|j?|} |jjA|jB| r|jE|| S|jBsg|j|<| S|jFjI|jJ|j8|||jL|jN}i}|D]}|d z|j8z}tQ||}||k7r0|jS|j%|gj!||t+|dj-d }|||ks{| j!||jjA|| r|jE|| Sg|j|<| Scc}wcc}wcc}wcc}w)NFrQrr}rprxrnrtrAT)rrdrSrr)rr)rrr)*rr)rFr=rr{rdrIrerpopleftr5r:rrrrrPr posixpath commonpathrrrrrrf_list_bucket_tree_with_foldersr>rrrupdater_drop_outdated_parent_listingrrrRrSrQr|r)rDr>rrdrSrrr root_pathrer6rcached_path_infosdirs_not_in_dircacherri dirs_to_visitdir_infodirs_not_expandedrg common_pathcommon_path_depthr4treelistingrrjrs rErfzHfFileSystem._ls_tree?s;'2KTM]M]Mi9I9Iot ))$)B &&(!&& QI(8a    =:X Y  ( ()@)@ A..22::<**)T*44Y?+--229gF. .. 4==  $ d 3  JJ( )#%  %9JoIiX^N_cnNneY'o! $&3&;&;&=OE8'5H+<#F+4==@0778HI04 hv>N0O-JJ'89)006G!"(1'0'8K'G&+QY $:!"$ !# z-9[\FI$fQ}M]MeT\\!F)%<$f!$f2 HY(223GJ[3[\ '(3CIK(@(F(Fs(K% 11H"%UQQvY-A-A+PSBS-TqUU#' #6=K"--kC.?@ ))+t<= !!+t4 MM#"+ $!)$/!) " r [-)GH::!++(--'; t//!334[[cdkk-0 $$W%5%5666t< !))*,DMM$' yy//!))!&&'&*33+55 09;H! 0 &_y~~= "s rE z=HfFileSystem._drop_outdated_parent_listing..s/fa& T0A/fs)rrall)rDr> parent_paths ` rEroz*HfFileSystem._drop_outdated_parent_listingsIll4( $-- 'C/f4==YdKe/f,f k*-g 'rGrc|jj|||}i}|rd|jdznd}g}|D]}|r.|j|k7r|jj |ds4|j ||sHt t|jjd| dz D]} tdt| |j} | j|vr!|j | | || j<S| j`|| j} | j| j| jks| j| _ !|r|std|d |d |S) zrwzFile not found in bucket 'z': '') rlist_bucket_treerr>rrrrparentsr rLrwr%) rDrerr bucket_filesbucket_folders min_depthr bucket_entryparent_bucket_folder_strparent_bucket_folder bucket_folders rErmz+HfFileSystem._list_bucket_tree_with_folderssyy11)Vy1Y 24-3A S)) /1( UL,++v5l>O>O>Z>Z^d]eef[g>h JJ| $-1|?P?P1Q1Y1Y,Z[k^g]gjk]k,l U('3$3/G+HVbVnVn($ (,,NBJJ34@TN#7#<#<=(33?$23G3L3L$MM$008%114H4T4TT4H4T4T 1# U UD #$'A)DQWPXXY%Z[ [ rGc/K|j||jdj}t||g|i|Ed{y7w)a Return all files below the given path. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.walk). Args: path (`str`): Root path to list files from. Returns: `Iterator[tuple[str, list[str], list[str]]]`: An iterator of (path, list of directory names, list of file names) tuples. rSrQN)r)r^rFrwalk)rDr>rrrs rErzHfFileSystem.walksL   :0F GQQS7<6t6v666sAA A Ac |j||jdj}t||fd|i|S)a Find files by glob-matching. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.glob). Args: path (`str`): Path pattern to match. maxdepth (`int`, *optional*): Maximum depth to descend into directories. By default, no limit. Returns: `list[str]`: List of paths matching the pattern. rSrQr)r)r^rFrglob)rDr>rrrs rErzHfFileSystem.glob sE   :0F GQQSw|D>8>v>>rGwithdirsc ,||dkr td|j||}|j} |j|fd||d|} |s| D cgc] } | ddk7s | } } n#|j|fi|} | ddk(r| g| zn| } | D cic]} | d|  } } t| } |s| S| D cic]} | | |  c} Scc} wcc} w#t $r9 |j|fd |i|} | dd k(r|| ini} n#t $ri} YnwxYwYnwxYwcc} w) a List all files below path. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.find). > [!TIP] > Large repositories are listed directory by directory, in parallel, to work around the > cursor-based pagination of the Hub `tree` endpoint. Tune this with the `max_workers` > argument of [`HfFileSystem`]. Args: path (`str`): Root path to list files from. maxdepth (`int`, *optional*): Maximum depth to descend into subdirectories. withdirs (`bool`, *optional*): Include directory paths in the output. Defaults to False. detail (`bool`, *optional*): If True, returns a dict mapping paths to file information. Defaults to False. refresh (`bool`, *optional*): If True, bypass the cache and fetch the latest data. Defaults to False. revision (`str`, *optional*): The git revision to list from. Returns: `Union[list[str], dict[str, dict[str, Any]]]`: List of paths or dict of file information. rzmaxdepth must be at least 1rQT)rrdrrprxrnrSrl)rr)rFrfinfor%r,r)rDr>rrrcrdrSrrrrgrinamesrns rEfindzHfFileSystem.findsjJ  HqL:; ;))$)B &&( .$--cgPXc\bcC"%BQ6k)AqBB&DIId5f5 +4V+< +KykC'QT),-A1V9a<-C-s L056D#d)O6 6C ."  %DIIdHXHH +4V+<+FtY'B$    &7sMC  C"C C2 D D#C:9D: DDDD Dpath1path2c |j||}|j||}t|tr?|j||}|ddk(ryt|trd|j} }n|j |j } }|jdx} 7|jj|j|| | |jfgna|j|d|5} | j} ddd|jj|j |jfg |j|j |j|j yt|tr td |j |j k(xr|j |j k(} | rd |d |}|jj!|j |j |j"|jd||jddt%|j&|j&|j"gn|j|d|j"5} | j} dddd |d |}|jj) |j&|j |j*|j |j"|jd||jd|j|j |j|j y#1swYWxYw#1swYxYw)a Copy a file within or between repositories, or into a bucket. > [!WARNING] > Note: When possible, use `HfApi.upload_file()` for better performance. Args: path1 (`str`): Source path to copy from. path2 (`str`): Destination path to copy to. revision (`str`, *optional*): The git revision to copy from. rQrprxNbucketrs)copyrbaddrVz6Copy from buckets to repositories is not available yetzCopy z to rTrUr[)src_path_in_reporT src_revision)rRrQrSrTrUr\path_or_fileobjrTrRrrQrSrTrU)r)r{rdrrerQrRr^rrWr>openreadr0rFrr_rSr!rT upload_filer)rDrrrSrresolved_path1resolved_path2 source_info source_type source_idrsfcontent same_reporTs rEcp_filezHfFileSystem.cp_file[s/ **58*D**58*D n&D E))EH)=K6"k1.*HI)1>3K3KY )7)A)A>CYCYY 'OOJ77D ,,",,[)XWeWjWj4k3l- YYudXY>'!ffhG' ,,^-E-EWVdViViLjKk,l  ! !~'?'?'A ! B  ! !~'?'?'A ! B  n&D E%&^_ _  $ $(@(@ @ u^E[E[_m_u_uEu  $UG4w7N II # #&..(22'00%zz*:NK#)::.BB#G')7)D)D%3%@%@%3%<%< $ 5$1H1HI #Q&&( #$UG4w7N II ! ! '+88&..jj(22'00%zz*:NK#)::.B#C "  >#;#;#=> >#;#;#=>_''B # #s)M9MMM%c |j|fii|ddi}d|vr#|dtd|d|djSd|vr |dr|dSd|vr |dr|dStd|d |d ) ak Get the last modified time of a file. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.modified). Args: path (`str`): Path to the file. Returns: `datetime`: Last modified time of the file. rTrtz9'modified' is not implemented for repository paths like 'rrvrwz'Cannot determined 'modified' for path 'z ' (info: ))rrdate)rDr>rrs rEmodifiedzHfFileSystem.modifiedstyyA!@F!@M4!@A D M"*),efjekkl*mnn &++ + _g= d "tM': & &%(OPTvU^_c^dde&fg grGc |j||}|j}|jd|j |jnd}|js|ddd}t |t rd|d<t |t r|r|jj|j|j|j d }i|dt|j|j|j d }nBt |t r^|j#|} |j%| | |j&| D cgc] } | d|k(s | } } | s t)|d| d}nd}|j#|} |s| |j&vr|j%| | |j&vr8|j&| D cgc] } | d|k(s | } } | s t)|d| d}|s| |rS|rP|dJ|jj+|j|j,||j|j} | s t)|d| d} t |j|j|jd|j.j}t | t0rX|dz| jz| j2d| j4| j6| j8| j:| j<d}n,|dz| jzdd| j>| j:d}|sdDcic]}||| }}|J|Scc} wcc} wcc}w)aS Get information about a file or directory. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.info). > [!WARNING] > Note: When possible, use `HfApi.get_paths_info()` or `HfApi.repo_info()` for better performance > (or `HfApi.get_bucket_paths_info()` or `HfApi.bucket_info()` for buckets) Args: path (`str`): Path to get info for. refresh (`bool`, *optional*): If True, bypass the cache and fetch the latest data. Defaults to False. revision (`str`, *optional*): The git revision to get info from. Returns: `dict[str, Any]`: Dictionary containing file information (type, size, commit info, etc.). rQrNFrrx)rnrorprt)rQrS)oidtitler)rzrt)rdrn)rrSrQr[)rTrYrArlrmry) r)rFr^rr>r{rPrlist_repo_commitsrRrQrSr. commit_idr created_atrdrrhrrget_paths_inforTrYr/rorqrrrsrtrurz)rDr>rdrSrrrrrtrrgout1 paths_inforirpr.s rErzHfFileSystem.infosX,))$)B &&(jj t/?/?/K4++QV !!#C -)KL%)M"-)KLQ\"ii99!))]5L5LWdWmWm: ##1'119J9JQ\QgQg$ 'E F,,t,K GGKG 1#}}[9O!QvY$=NAODO%dD1q'CC,,t,K;dmm#C $dmm+#'==#=Sa6dARSS)$51g#++##mBTB\!YY55!))!..&*33+55 6 ")$5&qM >!++!))!**!#"/"="=  )+ i2 )C).. @ ) &#,#4#4(}}$-$6$6'0'<'<$-$6$6 C!*C).. @ ! +#,#4#4'0'<'< C#.FG1c!f9GCG qPTTHs MM> M  M 0 Mc |jddr|j||j|fi|y#t$rYywxYw)a Check if a file exists. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.exists). > [!WARNING] > Note: When possible, use `HfApi.file_exists()` for better performance. Args: path (`str`): Path to check. Returns: `bool`: True if file exists, False otherwise. rdFT)r^r0rOSError)rDr>rs rEexistszHfFileSystem.exists8sK  zz)U+%%d+ DIId %f %  s69 AAcP |j|ddk(S#t$rYywxYw)ai Check if a path is a directory. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.isdir). Args: path (`str`): Path to check. Returns: `bool`: True if path is a directory, False otherwise. rprxFrrr2s rEr^zHfFileSystem.isdirQs1 99T?6*k9 9    %%cP |j|ddk(S#t$rYywxYw)a` Check if a path is a file. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.isfile). Args: path (`str`): Path to check. Returns: `bool`: True if path is a file, False otherwise. rprlFrr2s rEisfilezHfFileSystem.isfilecs1 99T?6*f4 4  rcx|j|}t|tr2|jd|jdt |j }nBt|j|j|j|j|j}|j|r|jddd}|S)z Get the HTTP URL of the given path. Args: path (`str`): Path to get URL for. Returns: `str`: HTTP URL to access the file or directory on the Hub. z /buckets/z /resolve/)rQrSrz/tree/r)r)r{rdrrerr>r)rRrTrQrSr^r )rDr>rurls rErzHfFileSystem.urlus))$/ m%C D]]O9]-D-D,EYuUbUgUgOhNijC%%**'11&// C ::d ++k8Q7C rGrrcHtr|jdx}yt|tr|jdd}}}n/|j |j }}t|jd}ttj||||j}t||S)zlBuild the Xet metadata of a file from its `info()` entry, or return None if it can't be downloaded with Xet.rsNrr[safe) token_typerRrQrSr) file_hash refresh_route)r7r^r{rdrerRrQrrSr9r8READrr2)rDrrrsrRrQrSrs rE_get_xet_file_datazHfFileSystem._get_xet_file_datas !$((:2F&Fh%O m%C D+8+B+BHdYG"/!6!6 8O8OYG]33"=H7#((]]  X]KKrGc |jd}|j||}t|jt |j dhz }t |ttfrt|dkDrt|,||f||d|S|j||} d} || tdt|r|}n|j|rt!j"|d yt!j"t jj%|xst j&d |j)|| x} l|j+| d t-t/|| |j0j3| d |t |tr|j4 yd yt7|d }d} |j9} | d } |j+|  t;|j=|j?||| d|j0j3t |tr |j4nd |jA| | r|jCyy#| r|jCwwxYw)a Copy single remote file to local. > [!WARNING] > Note: When possible, use `HfApi.hf_hub_download()` or `HfApi.download_bucket_files` for better performance. Args: rpath (`str`): Remote path to download from. lpath (`str`, *optional*): Local path to download to. Can be omitted if `outfile` is provided. callback (`Callback`, *optional*): Optional callback to track download progress. Defaults to no callback. outfile (`IO`, *optional*): Optional file-like object to write to. If provided, `lpath` is ignored. rSrQr)callbackoutfileFNz-Either `lpath` or `outfile` must be provided.T)exist_okro)incomplete_path xet_file_dataheaders expected_sizedisplayed_filename _tqdm_barwb)r temp_filerr resume_sizerr)"r^r)r#r>setkeysr{rrrrget_filerrrr^osmakedirsdirnamecurdirrset_sizer+r r_build_hf_headerstqdmrtellr*rrFseekclose)rDrpathlpathrrrrSresolve_remote_pathunhandled_kwargsr close_filer initial_posrrs rErzHfFileSystem.get_filesE$::j)"///I $$7$<$<=v{{}- <(\<$@ASIYEZ]^E^7#E5_8W_X^_ _yyy2 ?} !PQQ% E" ED1 BGGOOE2?bii$O%)%<%<=PRV%WWMd%%d6l3(,U &3 $ ; ; =&*6l+03=h 3U(--  \`  ud+! lln V -( HH0::<=!#(+ 335+5h +M(--SW  LL % z s 'A4I//Jctd)zA context within which files are committed together upon exit Requires the file class to implement `.commit()` and `.discard()` for the normal and exception cases. (Transactional commits are not supported.rrCs rE transactionzHfFileSystem.transactions""LMMrGctd)z@Begin write transaction for deferring files, non-context versionrrrCs rEstart_transactionzHfFileSystem.start_transactions""LMMrGcptt||j|j|j ffSr) make_instancerprrrrCs rE __reduce__zHfFileSystem.__reduce__s9 J      $ $ &    rGct|jt|jt|jdS)N)rrr)rrrrrCs rErz HfFileSystem._get_instance_state s4 //78\8\/]$,T-F-F$G  rGr)rNN)FNN)TFN)FFNNN)NFFFN)FNr?N)=rHrIrJrK root_markerprotocolrLrrr classmethodrtuple ExceptionrrrPrdr)r0r-r5rrr:rIrrOrYrbrrrrhrfrorr rmrrrrrr rrrr^rrr2rrrpropertyrrrrrrs@rErrrs4HTKH $#'!%#'#'<*<czD < $J < D[ <<D[<>    TT'*T69DjT tY%% &T&44dI>sTz>t>& ## 0>0>0>* 0> * 0> 0>f]a););!%);7;);ORUYz); cDcN" #);\ ##'#CCC C * C D[ C*CJ+#+$+ --&)-6:- *|+ ,-^77(5d3iQUVYQZAZ;[2\7 ??d ?S ?* $#>7>7*>7 >7  >7  >7*>7 cT#tCH~-. .>7@T?ST?T?d T?X\T?lhShxh2lltlsTzl_cdgildl_ml\2$$4L?B``LhlmprumuhvL t L*%)3DdL \NNN   rGr) metaclassceZdZddedededzffd ZfdZdeded efd Z dd Z dd e d dfd Z dfd Z d efdZxZS)rKNrr>rSc  |j|||_t |||jj fi||y#t$r)}d|jddvrt|d|d}~wwxYw)NrQwrJr[zB. Make sure the repository and revision exist before writing data.)r)rr,r^rrrF)rDrr>rSrrrs rErzHfFileSystemFile.__init__s !#!ID  T//99;FvF ! fjj,,'c\]   sA A9$A44A9c:t|dsyt| SNr)hasattrr__del__rDrs rErzHfFileSystemFile.__del__!st_- w  rGrendr?cdd|d|dz i|jjj}|j}t d||t j }t||jS)Nrangebytes=-rGETrr ) rrrrr4rHF_HUB_DOWNLOAD_TIMEOUTr3r)rDrrrrrs rE _fetch_rangezHfFileSystemFile._fetch_range'sm veWAcAgY/ ggll,,. hhj Wi>_>_ `AyyrGc<tjdd|_y)Nzhffs-F)rrS)tempfileNamedTemporaryFilerrCs rE_initiate_uploadz!HfFileSystemFile._initiate_upload1s!44GERrGfinalc J|jjd|jj}|jj ||r|jj t |jtrg|jjj|jj|jj|jjfgn|jjj|jj|jj |jj"|jj$|jj&|jj(|j*j-d|j*j-dt/j0|jj|jj3|jj5yy)NrrrTrUrrV)bufferrrrwriterr{rrdrrrWrernr>rrTrRrrQrSrr^rremover0rF)rDrblocks rE _upload_chunkzHfFileSystemFile._upload_chunk4s    " U#  NN "$,,.LM //&&008K8KTM_M_MdMd7e6f0 (($(NN$7$7!%!3!3!@!@ ..66''--"00::!//88#';;??3C#D'+{{7K'L)  IIdnn)) * GG $ $''113 % % rGcH|jdk(rx||dk(rq|jdk(rb|jj|jdd5}|j }|xjt |z c_|cdddSt||S#1swYxYw)zRead remote file. If `length` is not provided or is -1, the entire file is downloaded and read. On POSIX systems the file is loaded in memory directly. Otherwise, the file is downloaded to a temporary file and read from there. rNrr)r)rJlocrrr>rrr)rDlengthrrrs rErzHfFileSystemFile.readNs 99 &.FbLdhhRSmdii!< ffhCH$  w|F##   s 0BB!cL|jj|jSrrrr>rCs rErzHfFileSystemFile.url[ww{{499%%rGrr)Fr)rHrIrJrrLrrrbytesrrrrrrrrs@rErKrKsj < s cDj ! #CES4D4 $&S&rGrKc eZdZ ddededededzdedef fd Zdd ed efd Zdd efd Zdde e d ede fdZ defdZ fdZ dZdZxZS)rLNrr>rJrSr cache_typec |dk7rtd||dk7rtd|d|vrtd|d|j|||_|jjdd |_t |||jjf|||d |d|_|t|_ d|_ t|_ y) Nrz:HfFileSystemStreamFile only supports block_size=0 but got nonez?HfFileSystemStreamFile only supports cache_type='none' but got rz;HfFileSystemStreamFile only supports reading but got mode='rrQ)rnrorJrr$) rr)rrFdetailsrrresponser _exit_stack_stream_iterator bytearray_stream_buffer) rDrr>rJrSrr$rrs rErzHfFileSystemStreamFile.__init__`s ?YZdYefg g  ^_i^jkl l $;Z[_Z``abc c __TH_E $ 2 2 < < >M   "",,. 59j]g kq 15  $;8<'krGrwhencecX|dk(r|dk(ry||jk(r|dk(rytd)NrrzCannot seek streaming HF file)rr)rDrr.s rErzHfFileSystemStreamFile.seeks2 !8!  $((?v{ 899rGrc|j|jd} |j |jy|j|j|}|xjt |z c_|S#t $r>|j|jj|r|jd}YnwxYw)zRead the remote file. If the file is already open, we reuse the connection. Otherwise, open a new connection and read from it. If reading the stream fails, we retry with a new connection. FTrG)r)_open_connectionr+_read_from_streamrrrr)rDr retried_oncers rErzHfFileSystemStreamFile.reads ==  ! ! #  $==(D,A,A,I,,T-B-BFKCH$  $==,MM'')%%'#  $sA6;A66AB=<B=iteratorr?c|dk(ry|dkrRt|j}|jj|D]}|j|t |S|t |jkr)t |jd|}|jd|=|St|j}|jj|D]m}|t |z }|t |kDr|j|1|j|d||jj||dt |St |S)zRead up to `length` bytes from stream buffer and stream. If length < 0, read until EOF. If EOF is reached before length, fewer bytes may be returned. rrGN)r,r-r+rr"r)rDr4rbufchunkresultneeds rEr2z(HfFileSystemStreamFile._read_from_streams9 Q; A:D//0C    % % '! " 5! ":  S,,- -4..w78F##GVG,M++, !!# ECH$Dc%j  5! 5$<(##**5<8Sz SzrGcL|jj|jSrrrCs rErzHfFileSystemStreamFile.urlr rGcnt|dsy|jjt|Sr)rr*rrrrs rErzHfFileSystemStreamFile.__del__s/t_-   w  rGct|j|j|j|j|j j ffSr)reopenrr>rJ blocksizecachernrCs rErz!HfFileSystemStreamFile.__reduce__s0DIIt~~tzzWWWrGc@|jjd|_|j}|jj j }|jdkDrd|jd|d<|jjtd||tj|_ t|j|jj#|_y#t$r+}|jj dk(r d|_ Yd}~yd}~wwxYw) z%Open a connection to the remote file.Nrr r Ranger r i)r-r+r+rrrrrr* enter_contextr5rr r)r3r& status_code iter_bytes)rDrrrs rEr1z'HfFileSystemStreamFile._open_connections !!# $hhj'',,002 88a::S:$3$8 (5/ 3 PU D&S&!X;rGrLrScFtj|r|St|Sr)r,search safe_quoterQs rEr`r`s 299(C8]T\I]]rGsct|dS)Nr[r)r)rHs rErGrGs  rGr>r!c|}t|tr|d}n+t|tr|d}nt|tr|d}t ||)Nz} (repository not found). If the repo is private, make sure you are authenticated and your token has the required permissions.z (revision not found)z (invalid repository id))r{r'r(r1r,)r>r!msgs rErrse C#./f; ; C. /+, C* +./ C c)rGrrJrr$c,|j||||S)Nr')r)rr>rJrr$s rEr=r=s 774dzj7 QQrGc`||i|}|jD]\}}t||||Sr)rr)rrrrrrrs rErr s? d f B+113'kD+&' IrG)prrkrrrF collectionsrcollections.abcrrconcurrent.futuresr contextlibrrr dataclassesr r r r pathlibr rtypingrrr urllib.parserrfsspechttpx2fsspec.callbacksrrr fsspec.configr fsspec.utilsrr[r_bucket_live_followrrrr_bucketsrr _commit_apir!r" _local_folderr#errorsr$r%r&r'r( file_downloadr)r*r+hf_apir,r-r.r/r0utilsr1r2r3r4r5r6utils._runtimer7 utils._xetr8r9utils.insecure_hashlibr:r<rPrdrpAbstractFileSystemrhrMrLrr|rrrrspecAbstractBufferedFilerKrLr`rGrrrr=rhffsrNrGrErjs .1 11'''' JJ&#jj.C698\\ww,E' 66 6 &)A& &0 0%=0 0223 c3%*$z1L@%NQ% #s(^%ToIoId2l2j_ 6,,_ D%H&v{{77H&VN;V[[==N;b^C^C^## * *)d*: *x *R|R3RcRsRPSR~rG