kQj|ddlZddlZddlZddlmZddlZddlmZm Z ddl m Z m Z ddl mZddlmZmZmZddlmZmZmZmZmZmZmZer dd lmZmZm Z!Gd d Z"d e d e"fdZ#GddZ$GddZ%dZ&d e d e'e(effdZ)dejd ejfdZ*de'e(efd e fdZ+GddZ,y)N) deprecated)async_permutation_builderPermutationReader) LanceTableTable)LOOP)batch_to_tensorbatch_to_tensor_dictbatch_to_tensor_rows)AnyCallableIteratorLiteralOptional TYPE_CHECKINGUnion)pandasnumpypolarsceZdZdZdefdZddddddddeeedeee dee d ee d ee d eee d dfd Z ddddee dee dee d eee d df dZ ddddddeeedeee dee d eee d df dZ ddddddde deeedeee dee d ee d eee d dfdZ dde d eee d dfdZdddd ee d ee d dfdZde d dfdZd efdZy) PermutationBuildera A utility for creating a "permutation table" which is a table that defines an ordering on a base table. The permutation table does not store the actual data. It only stores row ids and split ids to define the ordering. The [Permutation] class can be used to read the data from the base table in the order defined by the permutation table. Permutations can split, shuffle, and filter the data in the base table. A filter limits the rows that are included in the permutation. Splits divide the data into subsets (for example, a test/train split, or K different splits for cross-validation). Shuffling randomizes the order of the rows in the permutation. Splits can optionally be named. If names are provided it will enable them to be referenced by name in the future. If names are not provided then they can only be referenced by their ordinal index. There is no requirement to name every split. The permutation is stored in memory and will be lost when the program exits. tablec$t||_y)a( Creates a new permutation builder for the given table. By default, the permutation builder will create a single split that contains all rows in the same order as the base table. Tables with an RTE write spec are rejected: unflushed rows have no row id. N)r_async)selfrs X/home/oem/CyberBrain26-unified/.venv/lib/python3.12/site-packages/lancedb/permutation.py__init__zPermutationBuilder.__init__,s06 Nratioscountsfixedseed clump_size split_namesr!r"r#r$r%r&returncH|jj|||||||S)a Configure random splits for the permutation. One of ratios, counts, or fixed must be provided. If ratios are provided, they will be used to determine the relative size of each split. For example, if ratios are [0.3, 0.7] then the first split will contain 30% of the rows and the second split will contain 70% of the rows. If counts are provided, they will be used to determine the absolute number of rows in each split. For example, if counts are [100, 200] then the first split will contain 100 rows and the second split will contain 200 rows. If fixed is provided, it will be used to determine the number of splits. For example, if fixed is 3 then the permutation will be split evenly into 3 splits. Rows will be randomly assigned to splits. The optional seed can be provided to make the assignment deterministic. If clump_size is provided, rows are shuffled as contiguous groups of that size, preserving I/O locality while still randomising the split assignment. The optional split_names can be provided to name the splits. If not provided, the splits can only be referenced by their index. r )r split_random)rr!r"r#r$r%r&s rr)zPermutationBuilder.split_random7s5H   !# !  rdiscard_weightr&columns split_weightsr+cD|jj|||||S)a Configure hash-based splits for the permutation. First, a hash will be calculated over the specified columns. The splits weights are then used to determine how many rows to assign to each split. For example, if split weights are [1, 2] then the first split will contain 1/3 of the rows and the second split will contain 2/3 of the rows. The optional discard weight can be provided to determine what percentage of rows should be discarded. For example, if split weights are [1, 2] and discard weight is 1 then 25% of the rows will be discarded. Hash-based splits are useful if you want the split to be more or less random but you don't want the split assignments to change if rows are added or removed from the table. The optional split_names can be provided to name the splits. If not provided, the splits can only be referenced by their index. r*)r split_hash)rr,r-r+r&s rr/zPermutationBuilder.split_hashes.6   )#   rr!r"r#r&cD|jj|||||S)aS Configure sequential splits for the permutation. One of ratios, counts, or fixed must be provided. If ratios are provided, they will be used to determine the relative size of each split. For example, if ratios are [0.3, 0.7] then the first split will contain 30% of the rows and the second split will contain 70% of the rows. If counts are provided, they will be used to determine the absolute number of rows in each split. For example, if counts are [100, 200] then the first split will contain 100 rows and the second split will contain 200 rows. If fixed is provided, it will be used to determine the number of splits. For example, if fixed is 3 then the permutation will be split evenly into 3 splits. Rows will be assigned to splits sequentially. The first N1 rows are assigned to split 1, the next N2 rows are assigned to split 2, etc. The optional split_names can be provided to name the splits. If not provided, the splits can only be referenced by their index. r0)rsplit_sequential)rr!r"r#r&s rr2z#PermutationBuilder.split_sequentials+> $$&; %  rr!r"r#r$r& num_samplescH|jj|||||||S)a# Sample rows uniformly **with replacement**, independently per split. Unlike the other split strategies, the number of output rows is ``num_samples``, not the number of (optionally filtered) input rows: the same row can appear zero, one, or many times, and ``num_samples`` may be larger or smaller than the input row count. Useful for bootstrap resampling, oversampling rare classes, or any training workload that wants a fixed-size stream of random draws rather than a single deterministic pass over every row. Combine with :meth:`filter` to restrict the eligible population. One of ratios, counts, or fixed must be provided, with the same meaning as in :meth:`split_random`, except they divide ``num_samples`` across splits rather than the input row count: if fixed is 4, each split gets ``num_samples // 4`` samples. Each split draws its own sample using a seed derived independently from ``(seed, split_index)``, so a split's content does not change if the samples are later redistributed across a different number of splits. The optional split_names can be provided to name the splits. If not provided, the splits can only be referenced by their index. r3)rbootstrap_sample)rr4r!r"r#r$r&s rr6z#PermutationBuilder.bootstrap_samples5D $$ # %  r calculationc@|jj|||S)aD Use pre-calculated splits for the permutation. The calculation should be an SQL statement that returns an integer value between 0 and the number of splits - 1. For example, if you have 3 splits then the calculation should return 0 for the first split, 1 for the second split, and 2 for the third split. This can be used to implement any kind of user-defined split strategy. The optional split_names can be provided to name the splits. If not provided, the splits can only be referenced by their index. )r&)rsplit_calculated)rr7r&s rr9z#PermutationBuilder.split_calculateds $$[k$J rr$r%c@|jj|||S)a| Randomly shuffle the rows in the permutation. An optional seed can be provided to make the shuffle deterministic. If a clump size is provided, then data will be shuffled as small "clumps" of contiguous rows. This allows for a balance between randomization and I/O performance. It can be useful when reading from cloud storage. r:)rshuffle)rr$r%s rr<zPermutationBuilder.shuffles  *= rfilterc<|jj||S)z Configure a filter for the permutation. The filter should be an SQL statement that returns a boolean value for each row. Only rows where the filter is true will be included in the permutation. )rr=)rr=s rr=zPermutationBuilder.filters 6" rcBfd}tj|S)zM Execute the configuration and create the permutation table. cKjjd{}tj|d{S7!7wN)rexecuter from_inner) inner_tblrs r do_executez.PermutationBuilder.execute..do_execute s9"kk1133I#..y99 949sAAAAAA)r run)rrEs` rrBzPermutationBuilder.executes  :xx %%rrA)__name__ __module__ __qualname____doc__rrrlistfloatintstrr)r/r2r6r9r<r=rrBrrrrsI, 7e 7)-&*#"$(+/,e%,c# , } , sm ,SM,d3i(, ,f)-+/ !c!Cy! ! ! d3i( ! !L)-&*#+/ "e%"c# " } " d3i( " "P)-&*#"+/**e% * c# * } *sm*d3i(* *ZDH-5d3i-@ ((, } 9A#   S%9 & &rrrr'ct|SrA)r)rs rpermutation_builderrQs e $$rcZeZdZdZdedefdZdeddfdZd e ddfd Z de ee fddfd Z y ) Permutationsa A collection of permutations indexed by name or ordinal index. Splits are defined when the permutation is created. Splits can always be referenced by their ordinal index. If names were provided when the permutation was created then they can also be referenced by name. Each permutation or "split" is a view of a portion of the base table. For more details see [Permutation]. Attributes ---------- base_table: Table The base table that the permutations are based on. permutation_table: LanceTable The permutation table that defines the splits. split_names: list[str] The names of the splits. split_dict: dict[str, int] A dictionary mapping split names to their ordinal index. Examples -------- >>> # Initial data >>> import lancedb >>> db = lancedb.connect("memory:///") >>> tbl = db.create_table("tbl", data=[{"x": x} for x in range(1000)]) >>> # Create a permutation >>> perm_tbl = ( ... permutation_builder(tbl) ... .split_random(ratios=[0.95, 0.05], split_names=["train", "test"]) ... .shuffle() ... .execute() ... ) >>> # Read the permutations >>> permutations = Permutations(tbl, perm_tbl) >>> permutations["train"] >>> permutations[0] >>> permutations.split_names ['train', 'test'] >>> permutations.split_dict {'train': 0, 'test': 1} base_tablepermutation_tablec||_||_|jj|jjj d}||j dnd}|It j||_t|jDcic]\}}|| c}}|_ yg|_i|_ yg|_i|_ ycc}}w)N split_namesutf-8) rTrUschemametadatagetdecodejsonloadsr& enumerate split_dict)rrTrUrawr&idxnames rrzPermutations.__init__Cs$!2  # # , , 8#**3377GC14#**W-dK&#'::k#: /89I9I/J#"+#tD#I# $& "$ "D  DO#s Crcr' Permutationc~|jj|d}|td|d|j|S)zu Get a permutation by name. If no split named `name` is found then an error will be raised. NzNo split named `z` found)r`r[ ValueError get_by_index)rrcrbs r get_by_namezPermutations.get_by_nameXsE oo!!$- ;/vW=> >  %%rindexcXtj|j|j|S)z- Get a permutation by index. )rd from_tablesrTrUrris rrgzPermutations.get_by_indexcs$&&t8N8NPUVVrct|tr|j|St|tr|j |St d|)NzInvalid split name or index: ) isinstancerNrhrMrg TypeError)rrcs r __getitem__zPermutations.__getitem__isK dC ##D) ) c "$$T* *;D6BC CrN) rGrHrIrJrrrrNrhrMrgrrprOrrrSrSsb,\!5!Z!* & & &W#W-W Dc3hDMDrrSc<eZdZdZedej deee e ffdZ edej dee ee ffdZ edej dej fdZ edej ddfdZedej dd fd Zed d Zy ) Transformsz7 Namespace for common transformation functions batchr'c"|jSrA) to_pylistrss r arrow2pythonzTransforms.arrow2pythonw  rc"|jSrA) to_pydictrvs rarrow2pythoncolzTransforms.arrow2pythoncol{rxrc|SrArOrvs r arrow2arrowzTransforms.arrow2arrows rz np.ndarrayc>|jjSrA) to_pandasto_numpyrvs r arrow2numpyzTransforms.arrow2numpys ))++rz pd.DataFramec"|jSrA)rrvs r arrow2pandaszTransforms.arrow2pandasrxrcTddldtjdjffd }|S)Nrrsr'c(j|SrA) from_arrow)rspls rimplz%Transforms.arrow2polars..impls 2==' 'r)rpa RecordBatch DataFrame)rrs @r arrow2polarszTransforms.arrow2polarss( ( (2<< ( rN)r'z pl.DataFrame)rGrHrIrJ staticmethodrrrKdictrNr rwr{r}rrrrOrrrrrrrs!BNN!tDcN/C!!!r~~!$sDI~2F!!2>>bnn,2>>,l,,!BNN!~!!rrrdchddlm}t||rd|dSt|tst dt ||j j}|jdrd|j|jdSd |j||j|j jd S) Nr) RemoteTableremote)kindrzCannot pickle table of type memory://memory)rrcdatalocal)rrcuri namespacestorage_options) remote.tablerrnrrftype_connr startswithrcto_arrow_namespace_pathr)rrbase_uris r_table_to_pickle_staters)%%  eZ (7U GHH{{H;'JJNN$   ** ;;66  rpermutation_datact|jjxsi}|jdd|S|jdd|j |S)zMStrip the recorded base version so the reader leaves the base table unpinned.s base_versionNs base_branch)rrYrZpopreplace_schema_metadata)rrZs r_drop_base_versionrsT$++44:;H||OT*2 LL&  3 3H ==rstatecddlm}|d}|dk(r|dS|dk(r|dj|d|d S|d k(r-||d |d  }|j|d|dxsdSt d|)Nrconnectrrrrrrcrrrrrrnamespace_pathz!Unknown table pickle state kind: )r create_table open_tablerf)rrrdbs r_table_from_pickle_staters =D xW~ x{#00vf NN w U5\59J3K L}}U6]5;M;UQU}VV 8? @@rceZdZdZ dEdedeededeeefdede e je fd eed eed ee egefd ee fd Zde fdZdeeefddfdZdeddfdZd e egefddfdZededdfdZe dFdedeedeeeefddfdZdeee ffdZdeee fddfdZdGdZede j6fdZedefdZedefdZedeefdZ ede!eeffdZ"defdZ#d edee fd!Z$dHd"Z%d#eeddfd$Z&d%ed&eddfd'Z'd(eeefddfd)Z(d#eeddfd*Z)de*eee ffd+Z+ dIded,e,de*eee ffd-Z-d.e.d/ddfd0Z/d1e e je fddfd2Z0d3eede fd4Z1d5ede fd6Z2d7eede fd8Z3e4d9:d;eddfd<Z5d;eddfd=Z6e4d>:d eddfd?Z7d eddfd@Z8e4dA:dBeddfdCZ9dBeddfdDZ:y)Jrda A Permutation is a view of a dataset that can be used as input to model training and evaluation. A Permutation fulfills the pytorch Dataset contract and is loosely modeled after the huggingface Dataset so it should be easy to use with existing code. A permutation is not a "materialized view" or copy of the underlying data. It is calculated on the fly from the base table. As a result, it is truly "lazy" and does not require materializing the entire dataset in memory. NrTrUsplit selection batch_size transform_fnoffsetlimitconnection_factory_readerc 0|Jd|Jd||_||_||_||_||_||_||_||_| |_| #tj|j} | |_ tj|_y)zx Internal constructor. Use [from_tables][lancedb.permutation.Permutation.from_tables] instead. Nbase_table is requiredzselection is required)rTrUrrrrrrrr rF _build_readerreaderosgetpid_pid) rrTrUrrrrrrrrs rrzPermutation.__init__s"%?'??%$=&==$$!2 "($  "4 ?hht1134G)0 IIK rr'cPKtj|j|j|jd{}|j #|j |j d{}|j#|j|jd{}|S7d777 wrA) rrkrTrUrr with_offsetr with_limit)rrs rrzPermutation._build_readers(44 OOT33TZZ   ;; "!--dkk::F :: !!,,TZZ88F  ;8s39B&B .B&*B"+.B&B$B&"B&$B&c>tj|}||_|S)z Creates a new permutation with the given selection Does not validation of the selection and it replaces it entirely. This is not intended for public use. )copyr)rrnews r_with_selectionzPermutation._with_selectionsiio!  rc>tj|}||_|S)zE Creates a new permutation with the given batch size )rr)rrrs rwith_batch_sizezPermutation.with_batch_sizesiio# rcP|Jdtj|}||_|S)a| Creates a new permutation that will use ``connection_factory`` to reopen the base table when this permutation is unpickled in a worker process. The factory is a callable that takes a single argument, the base table name, and returns a LanceDB table. It must be picklable; the worker will pickle it via standard ``pickle`` and call it to recover the base table. Picklable callables in practice means top-level (module-level) functions, ``functools.partial`` of such functions, or instances of picklable classes implementing ``__call__``. Lambdas and closures over local variables don't pickle with the default protocol. A factory is optional for normal local and remote LanceDB connections: if not set, ``__getstate__`` captures the table's own picklable reopen state. Use a factory when that default state is not enough, for example when credentials should be loaded from the worker environment instead of being embedded in the pickle. Examples -------- Basic native (file-system path), parameterized via ``functools.partial``:: import functools, lancedb from lancedb.permutation import Permutation def open_native_table(uri: str, table_name: str): return lancedb.connect(uri).open_table(table_name) factory = functools.partial(open_native_table, "/data/lance_db") permutation = Permutation.identity( factory("training") ).with_connection_factory(factory) Native via :func:`lancedb.connect_namespace` (e.g. a directory- or REST-backed namespace client). The factory takes the implementation name and properties dict as partial-bound args so the worker can rebuild the same namespace connection:: def open_via_namespace( impl: str, properties: dict[str, str], table_name: str, ): return lancedb.connect_namespace(impl, properties).open_table( table_name, ) factory = functools.partial( open_via_namespace, "dir", {"root": "/data/lance_db"}, ) LanceDB Cloud, reading credentials from env vars at worker startup so secrets aren't pickled into the dataset:: import os, lancedb def open_remote_table(table_name: str): db = lancedb.connect( "db://my-database", api_key=os.environ["LANCEDB_API_KEY"], region=os.environ.get("LANCEDB_REGION", "us-east-1"), ) return db.open_table(table_name) permutation = Permutation.identity( open_remote_table("training") ).with_connection_factory(open_remote_table) zconnection_factory is required)rr)rrrs rwith_connection_factoryz#Permutation.with_connection_factorys1N"-O/OO-iio!3 rrc0tj|ddS)zF Creates an identity permutation for the given table. N)rdrk)clsrs ridentityzPermutation.identityds &&udD99rcRJd։ tdttrjjtddjjj d}||j dnd}|tddtj|} |jn#ttrntd d fd }tj|S#t$rtddd wxYw) am Creates a permutation from the given base table and permutation table. A permutation table identifies which rows, and in what order, the data should be read from the base table. For more details see the [PermutationBuilder] class. If no permutation table is provided, then the identity permutation will be created. An identity permutation is a permutation that reads all rows in the base table in the order they are stored. The split parameter identifies which split to use. If no split is provided then the first split will be used. NrzWCannot create a permutation on split `{split}` because no permutation table is providedz&Cannot create a permutation on split `z=` because no split names are defined in the permutation tablerWrXz` because split `z)` is not defined in the permutation tablezInvalid split: rc Ktjd{}|jdd{}|jDcic]}||}}|tt j |S7X7Acc}ww)N)r)rrk output_schemanamesDEFAULT_BATCH_SIZErrrw)rrYrcinitial_selectionrTrrUrs rdo_from_tablesz/Permutation.from_tables..do_from_tabless,88-uF"//55F8> Et E  E!!"''  6 Es1BA7BA9B A;$B9B;B)rfrnrNrYrZr[r\r]r^rirMror rF)rrTrUrrar&rs```` rrkzPermutation.from_tablesksn*%?'??%   ( @%%$++44<$@HWW(..77;;NK58_cjj1$ &$@HWW#jj5 '--e4EE3'/% 9::E  xx());"$@H++0'2,,s 3D D&c pd}|j|jj}|jj||j|j |j |j|j|j|jd }|j|Si|dt|jiS)aBuild a picklable state dict for this permutation. The base table is captured either via a user-supplied ``connection_factory`` (see [with_connection_factory]) or, as a fallback, by the table's own picklable reopen state. The permutation table is captured as a pyarrow Table (which pickles via Arrow IPC natively). The reader is dropped from the wire format and rebuilt lazily on first use. N) base_table_namerrrrrrrrbase_table_state) rUrrTrcrrrrrrrr)rrcommons r __getstate__zPermutation.__getstate__s04  ! ! -#55>>@  $33 0ZZ// --kkZZ"&"9"9    " " .M   6t G  rrc:ddlm}|d}d}| ||d}nqd|vr|d}|ddk(}t|}nTd |vr#d }|d }|j|d|d }n-||d |d }|j |d|dxsd}d} |d} | '|r t | } |d }|jd| } ||_| |_|d|_|d|_ |d|_ |d|_ |d|_ |d|_ ||_d|_d|_y)NrrrFrrrrbase_table_dataTrbase_table_uribase_table_storage_optionsrbase_table_namespacerr permutationrrrrrr)rrrrrrrTrUrrrrrrrrr) rrrr rebuilt_baserT base_statemem_dbbase_dbrUrs r __setstate__zPermutation.__setstate__s"#78  )+E2C,DEJ 5 (12J%f-9L1*=J % 'L[)F,,'(%0A*BJ&' %&B CG!++'($%;<D,J .2 !34  '$66F#G [)F & 3 3MCS T $!27^ {+ -!.1Ho 7^ "4  rctj}|jt|dd|k(ryt |j dr|j j |j0t |jdr|jj tj|j|_||_ y)Nr _ensure_open) rrrgetattrhasattrrTrrUr rFrr)rpids rrzPermutation._ensure_opensiik ;; "wtVT'Bc'I  4??N 3 OO ( ( *  ! ! -'  " "N3   " " / / 1hht1134  rcbjfd}tj|S)NcjKjjjd{S7wrA)rrrrsrdo_output_schemaz,Permutation.schema..do_output_schema!s&224>>BB BBs )313)rr rF)rrs` rrYzPermutation.schemas*  Cxx(*++rc,t|jS)z: The number of columns in the permutation )lenrYrs r num_columnszPermutation.num_columns&s 4;;rcV|j|jjS)z7 The number of rows in the permutation )rr count_rowsrs rnum_rowszPermutation.num_rows-s# {{%%''rc.|jjS)z= The names of the columns in the permutation )rYrrs r column_nameszPermutation.column_names5s {{   rc2|j|jfS)zh The shape of the permutation This will return self.num_rows, self.num_columns )rrrs rshapezPermutation.shape<s}}d....rc|jS)z The number of rows in the permutation This is an alias for [num_rows][lancedb.permutation.Permutation.num_rows] )rrs r__len__zPermutation.__len__Es }}r_columnctd)z; Get the unique values in the given column zunique is not yet implemented Exception)rrs runiquezPermutation.uniqueMs788rctd)a/ Flatten the permutation Each column with a struct type will be flattened into multiple columns. This flattening operation happens at read time as a post-processing step so this call is cheap and no data is copied or modified in the underlying dataset. zflatten is not yet implementedrrs rflattenzPermutation.flattenSs899rr,c|Jd|jjDcic] \}}||vs ||}}}t|dk(r td|j |Scc}}w)aY Remove the given columns from the permutation Note: this does not actually modify the underlying dataset. It only changes which columns are visible from this permutation. Also, this does not introduce a post-processing step. Instead, we simply do not read those columns in the first place. If any of the provided columns does not exist in the current permutation then it will be ignored (no error is raised for missing columns) Returns a new permutation with the given columns removed. This does not modify self. columns is requiredrzCannot remove all columns)ritemsrrfr)rr,rcvalue new_selections rremove_columnszPermutation.remove_columns_s"9$99",0>>+?+?+A 'D%TQXEXD%K   }  "89 9##M22 s A&A&old_namenew_namec6|Jd|Jd||jvrtd|d||jvrtd|d|d|jjDcic]\}}||k(r|n||}}}|j|Scc}}w)a Rename a column in the permutation If there is no column named old_name then an error will be raised If there is already a column named new_name then an error will be raised Note: this does not actually modify the underlying dataset. It only changes the name of the column that is visible from this permutation. This is a post-processing step but done at the batch level and so it is very cheap. No data will be copied. zold_name is requiredznew_name is requiredzCannot rename column `` because it does not existz` to `z0` because a column with that name already exists)rrfrr)rr r rcrr s r rename_columnzPermutation.rename_columnys#;%;;##;%;;# 4>> )( 2MN  t~~ %( & C00  $~~335 e)Xte ;  ##M22  s-B column_mapcp|Jd|}|jD]\}}|j||}|S)a Rename the given columns in the permutation If any of the columns do not exist then an error will be raised If any of the new names already exist then an error will be raised Note: this does not actually modify the underlying dataset. It only changes the name of the column that is visible from this permutation. This is a post-processing step but done at the batch level and so it is very cheap. No data will be copied. zcolumn_map is required)rr)rrnew_permutationr r s rrename_columnszPermutation.rename_columnssR%?'??%","2"2"4 P Hh-;;HhOO Prc|Jdt|dk(r tdi}|D]<}|jj|d}||dk(rd}ntd|d|||<>|j |S)a Select the given columns from the permutation This method refines the current selection, potentially removing columns. It will not add back columns that were previously removed. If any of the columns do not exist then an error will be raised. This does not introduce a post-processing step. It simply reduces the amount of data we read. NrrzMust select at least one column_rowidzCannot select column `r)rrfrr[r)rr,r rcrs rselect_columnszPermutation.select_columnss"9$99" w<1 >? ?  (DNN&&tT2E}8#%E$06QR#(M$  (##M22rc<|j|jdS)z. Iterate over the permutation T)skip_last_batch)iterrrs r__iter__zPermutation.__iter__syy$y??rrc#Kjfd}tj|fd} tj|}|jk(s|sj |?#t $rYywxYww)z Iterate over the permutation in batches If skip_last_batch is True, the last batch will be skipped if it is not a multiple of batch_size. cnKjjjd{S7w)N)r)rreadr)rrsrget_iterz"Permutation.iter..get_iters+))$..Z)PP PP +535c@Kjd{S7wrA) __anext__) async_itersrget_nextz"Permutation.iter..get_nexts#--// //s N)rr rFrrStopAsyncIteration)rrrrr#rsr"s`` @rrzPermutation.iters  QXXhj)  0 ,>>Z/++E22"   s#6B AA<< BB BB format) rpython python_colrarrowtorch torch_row torch_colrc|Jd|dk(r|jtjS|dk(r|jtjS|dk(r|jtjS|dk(r|jtj S|dk(r|jtj S|dk(r|jtS|dk(r|jtS|d k(r|jtS|d k(r#|jtjStd |) a` Set the format for batches If this method is not called, the "python" format will be used. The format can be one of: - "numpy" - the batch will be a dict of numpy arrays (one per column) - "python" - the batch will be a list of dicts (one per row) - "python_col" - the batch will be a dict of lists (one entry per column) - "pandas" - the batch will be a pandas DataFrame - "arrow" - the batch will be a pyarrow RecordBatch - "torch" - the batch will be a list of per-row dicts mapping column name to a 0-D torch tensor. Works with the default ``torch.utils.data.DataLoader`` collate, which stacks the per-row dicts back into a dict of batched tensors. - "torch_row" - the batch will be a list of tensors, one per row - "torch_col" - the batch will be a 2D torch tensor (first dim indexes columns) - "polars" - the batch will be a polars DataFrame Conversion may or may not involve a data copy. Lance uses Arrow internally and so it is able to zero-copy to the arrow and polars formats. Conversion to torch and torch_col will be zero-copy but will only support a subset of data types (numeric types). Conversion to numpy and/or pandas will typically be zero-copy for numeric types. Conversion of strings, lists, and structs will require creating python objects and this is not zero-copy. For custom formatting, use [with_transform][lancedb.permutation.Permutation.with_transform] which overrides this method. zformat is requiredr&r'rrr(r)r*r+rzInvalid format: ) with_transformrrrwr{rrr}r r r rrf)rr%s r with_formatzPermutation.with_formats4^!7#77! X &&z'>'>? ? \ !&&z'A'AB B w &&z'='=> > x &&z'>'>? ? w &&z'='=> > w &&';< < { "&&';< < { "&&7 7 x &&z'>'>'@A A/x89 9r transformcP|Jdtj|}||_|S)ab Set a custom transform for the permutation The transform is a callable that will be invoked with each record batch. The return value will be used as the batch for iteration. Note: transforms are not invoked in parallel. This method is not a good place for expensive operations such as image decoding. ztransform is required)rr)rr/rs rr-zPermutation.with_transform/s0$=&==$iio$ roffsetscjfd}tj|}j|S)z Take rows from the permutation by offset The returned value is passed through the permutation's current transform, so `with_format` and `with_transform` affect this method in the same way they affect iteration. cnKjjjd{S7w)N)r)r take_offsetsr)r1rsrdo_take_offsetsz1Permutation.take_offsets..do_take_offsetsHs+11'T^^1TT TTr)rr rFr)rr1r5rss`` rr4zPermutation.take_offsets>s;  U*+  ''rric&|j|gS)zE Returns a single row from the permutation by offset r4rls rrpzPermutation.__getitem__Ns  %))rindicesc$|j|S)z= Returns rows from the permutation by offset r7)rr8s r __getitems__zPermutation.__getitems__Ts  ))rzUse with_skip instead)detailsskipc$|j|S)aB Skip the first `skip` rows of the permutation Note: this method returns a new permutation and does not modify `self` It is provided for compatibility with the huggingface Dataset API. Use [with_skip][lancedb.permutation.Permutation.with_skip] instead to avoid confusion. ) with_skip)rr<s rr<zPermutation.skipZs~~d##rc|jtj|}||_tj|j |_tj|_ |S)z? Skip the first `skip` rows of the permutation ) rrrr rFrrrrr)rr<rs rr>zPermutation.with_skipgsO iio XXc//12 99; rzUse with_take insteadc$|j|S)aQ Limit the permutation to `limit` rows (following any `skip`) Note: this method returns a new permutation and does not modify `self` It is provided for compatibility with the huggingface Dataset API. Use [with_take][lancedb.permutation.Permutation.with_take] instead to avoid confusion. ) with_take)rrs rtakezPermutation.takers~~e$$rc|jtj|}||_tj|j |_tj|_ |S)zN Limit the permutation to `limit` rows (following any `skip`) ) rrrr rFrrrrr)rrrs rrAzPermutation.with_takesO iio XXc//12 99; rzUse with_repeat insteadtimesc$|j|S)a= Repeat the permutation `times` times Note: this method returns a new permutation and does not modify `self` It is provided for compatibility with the huggingface Dataset API. Use [with_repeat][lancedb.permutation.Permutation.with_repeat] instead to avoid confusion. ) with_repeatrrDs rrepeatzPermutation.repeats&&rctd)z6 Repeat the permutation `times` times z"with_repeat is not yet implementedrrGs rrFzPermutation.with_repeats<==r)NNNN)NN)r'N)r'rd)F);rGrHrIrJrrrMrrNrrrr rrrrrr classmethodrrrkrrrpropertySchemarYrrrKrtuplerrrrr rrrrrboolrrr.r-r4rpr:rr<r>rBrArHrFrOrrrdrdsj (!%#?C/3  $E?   S>    r~~s23   } %XseUl%;< +, B%6 c3h M #-J"*C5%<"8J JX:U:}:: .2+/ H*H*$E?H*c3h( H*  H*H*T# d38n# J1$sCx.1T1f , ,, S  (#((!d3i!! /uS#X//9c9d3i9 :3d3i3M343c3S3]3<c3hM&3d3i3M3>@(4S>2@8=04 $sCx. !6C:   C: C:J 1D(E - (DI(#( *** *DI*#* /0 $ $ $1 $ c m /0 %# %- %1 % s } 12 'C 'M '3 '>>>rrd)-rr]r deprecationrpyarrowr_lancedbrrrrrbackground_loopr utilr r r typingr rrrrrrlancedb.dependenciesrpdrnprrrrQrSrrrrrNrrrrdrOrrrXs "B$!MMSSSLLx&x&v%u%);%[D[D|  H%DcN:>>bhh> ADcN Au AN >N >r