kQjҾ ddlmZddlZddlZddlZddlZddlZddlmZm Z ddl m Z ddl m Z m Z ddlmZddlmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZddl m!Z!dd l"m#Z#m$Z$m%Z%d d l&m'Z'd d l(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.d d l/m0Z0ddl1m2Z2ddl3m4Z4m5Z5ddl6m7Z7m8Z8m9Z9d dl:m;Z;mZ>m?Z@mAZBddlCZDddlEZCddlFmGZHddlImJZKddlLZMd dlNmOZOmPZPmQZQd dlRmSZSmTZTd dlUmVZVmWZWmXZXmYZYmZZZm[Z[m\Z\m]Z]m^Z^m_Z_m`Z`maZad dlbmcZcd dldmeZemfZgd dlhmiZid dljmkZkmlZld dlmmnZnmoZompZpmqZqmrZrmsZsmtZtmuZumvZvmwZwmxZxmyZymzZzm{Z{m|Z|d dl}m~Z~mZmZmZmZmZd dlUmZd dlmZmZmZmZ d~dZdZddZdd Zd!Zd"Z dd#Zdd$Zdd%Zdd& dd'ZerJd d(lmZd d)lmZmZmZmZmZmZmZmfZfmZmZmZmZmZmZmZmZmZd d*lUmZddl?Z?ddlZd d+l/mZmZmZmZmZmZeeWeXeYe[e`e^e_eVeZe\e]eaf ZGd,d-ed./Zhd0Zdd1Zdd2Z d dd3Z d dd4Z dd.d5 dd6Z d dd7Z dd8Zdd9Z dd:Zdd;Z dd<Zdd=Zdd>Zdd?Zdd@ZddAZddBZddCZe dDEGdFdGZddHZ ddIZddJZddKZ ddLZddMZddNZddOZ ddPZ d ddQZddRZdSZ dddT ddUZddVZddWZdXZGdYdZeZGd[d\eZ d dd]Z dd^Z dd_Zdd`Z d ddaZ ddbZddcZdddZddeZddfZ ddgZddhZddiZddjZddkZGdldmZe GdndoZe GdpdqZe GdrdsZe GdtduZGdvdwZGdxdyZGdzd{ZGd|d}Zy)) annotationsN)ABCabstractmethod) dataclass)datetime timedelta)cached_property) TYPE_CHECKINGAny AsyncIteratorCallableDictIterableIteratorListLiteralOptionalSequenceTuple TypedDictUnionoverload)urlparse) _hf_batches_register_optional_converters to_scannable) __version__)BlobFile_normalize_blob_descriptor_normalize_blob_row_ids_wrap_blob_filesstrip_auto_row_idsvalidate_blob_mode)BlobMode) peek_reader)LOOPembedding_executor)AsyncJobJob _typed_job)_check_for_hugging_face_check_for_lance_check_for_pandaslancepandaspolars)DATAVECVECTOR_COLUMN_NAME)EmbeddingFunctionConfigEmbeddingFunctionRegistry) BTreeIvfFlatIvfPqIvfSqBitmapIvfRq LabelListFmHnswPqHnswSqHnswFlatFTS)Expr)FunctionApplicationRefreshColumnResult)LanceMergeInsertBuilder) LanceModel model_to_dict)AnalyzePlanDistributedMetrics AsyncFTSQueryAsyncHybridQuery AsyncQueryAsyncTakeQueryAsyncVectorQueryDocumentGranularity FullTextQueryLanceEmptyQueryBuilderLanceFtsQueryBuilderLanceHybridQueryBuilderLanceQueryBuilderLanceVectorQueryBuilderLanceTakeQueryBuilderQuery)add_note fs_from_uriget_uri_schemeinfer_vector_column_namejoin_uri value_to_sql) lang_mapping)blob_v2_column_pathsis_blob_v2_fieldrow_addressable_blob_v2_pathsschema_has_blob_fieldc|duxrd|vS)N QueryTable)namespace_clientpushdown_operationss R/home/oem/CyberBrain26-unified/.venv/lib/python3.12/site-packages/lancedb/table.py_should_push_down_query_tablerixs 4 ' OL? . )?453>?s/A6B"6B BB"C1CCc|S)zBReturn a Polars frame unchanged while blocking predicate pushdown.re)frames rh"_polars_predicate_pushdown_barrierr~s Lrj)jiebalindera)zunknown base tokenizerzInvalid directory path:zFailed to load JiebazFailed to load tokenizer configz&Failed to initialize default tokenizercd}|=t|jdz}|dkrtjdtd|rtjdt d|S)Nrzoptimize(cleanup_older_than=0) removes every version except the latest. Any concurrent reader or writer still using an older version will fail. Use a longer cleanup_older_than unless no other process is working on this table. stacklevelzNThe 'retrain' parameter is deprecated and will be removed in a future version.)round total_secondswarningswarn UserWarningDeprecationWarning)cleanup_older_thanretraincleanup_since_mss rh_optimize_cleanup_since_msrsp '+% !3!A!A!Cd!JK q MMD       rjct|ddxsd}|jr,t|jdtr|jdnd}||vr||vr t ||yyy)N __notes__rer)getattrargsrnrorX) exceptionnoteexisting_notesmessages rh_add_unique_notersiY R8>BN >>j):C@ q   >!d'&9D!':!rjc4tfdtDS)Nc3TK|]}|k(xsj|d!yw)/N) startswith).0prefixbase_tokenizers rh z-_is_model_backed_tokenizer..s7  & KN$=$=l$KKs%()any _MODEL_BACKED_TOKENIZER_PREFIXES)rs`rh_is_model_backed_tokenizerrs 6 rj)languagect||7dvr3djtj}t |d|yt |syt fdtDsyt |dy)Nz"not support the requested languagez, zSupported languages: c3&K|]}|v ywNre)rmarkerrs rhrz,_maybe_add_fts_error_note..sNVv NsaKModel-backed tokenizers such as 'jieba/default' and 'lindera/ipadic' require tokenizer models in Lance's language model home. Set LANCE_LANGUAGE_MODEL_HOME to override the default platform data directory under 'lance/language_models'. Expected layouts include '/jieba/default/...' and '/lindera/ipadic/...'.)rojoinr^valuesrrr_MODEL_BACKED_TOKENIZER_ERRORS)rrrsupported_langsrs @rh_maybe_add_fts_error_notersv)nG D O))L$7$7$9:&;O;L$MN %n 5 N/MN N -rj)LanceDBConnection)Table OptimizeStats CleanupStatsCompactionStatsTagAddColumnsResultFunctionErrorsrE AddResultAlterColumnsResultUpdateFieldMetadataResult DeleteResultDropColumnsResultFtsToken RTEWriteSpec MergeResult UpdateResult) IndexConfig) QueryTypeOnBadVectorsTypeAddMode CreateModeScalarIndexTypeBaseTokenizerTypeceZdZUdZded< ded< ded< ded< d ed < ded < ded < ded < ded< ded< ded< ded< ded< ded< ded< ded< ded< ded<y)CompactionOptionsa!Options that control file compaction during table optimization. Unspecified options use Lance's defaults. Compaction planning is row based. Lowering ``target_rows_per_fragment`` based on the expected row size can bound later compaction passes once oversized fragments have been rewritten. It does not split an existing fragment, so the first pass over an oversized fragment is not subject to that bound. ``max_bytes_per_file`` limits output file size, not compaction memory. Source budgets keep whole planned tasks; if the first task exceeds a budget, that run performs no compaction work. Examples -------- Derive a steady-state row target from the expected row size: >>> desired_fragment_bytes = 750 * 1024 * 1024 >>> average_row_bytes = 1_500_000 >>> options: CompactionOptions = { ... "target_rows_per_fragment": max( ... 1, desired_fragment_bytes // average_row_bytes ... ), ... } >>> await table.optimize(compaction_options=options) # doctest: +SKIP inttarget_rows_per_fragmentmax_rows_per_group Optional[int]max_bytes_per_fileboolmaterialize_deletionsfloatmaterialize_deletions_threshold num_threads batch_sizeio_buffer_sizedefer_index_remapzLiteral['direct', 'compact']index_remap_modezEOptional[Literal['reencode', 'try_binary_copy', 'force_binary_copy']]compaction_modebinary_copy_read_batch_bytesmax_source_fragmentsmax_source_rowsmax_source_bytesz List[int]excluded_fragment_idsmax_overlays_per_fragmentrodata_storage_versionN__name__ __module__ __qualname____doc____annotations__rerjrhrr"s4"!O@%%OO%**U8.!!FN22>E"//G''B""K##G$$I,,@ErjrF)total>l2dotcosinehammingc*|yt|tttfr dt|iSt|tr|s t dd|iSt|t r0|jhdz }|rt dt||S ddl m }t||r/|j|j|j|jd St#d t%|j&#t$r}|j d vrYd}~>d}~wwxYw) zHKeep a Python blob's inline data or external URI before Arrow infers it.NdatazBlob uri cannot be emptyuri>rrsizepositionzUnknown blob fields: r)Blob)rrrr)r/z lance.blobzUnsupported blob value: )rnbytes bytearray memoryviewro ValueErrordictkeyssorted lance.blobrrrrrModuleNotFoundErrorname TypeErrortyper)valueunknownrrzs rh_blob_value_to_storagerzs  }%%J78e %%%78 8u~%**,!DD 4VG_4EFG G  # eT " yy!NN   .tE{/C/C.DE FF  882 2  3sC00 D9D  Dc |yt|tr|rt|dtr|Dchc]-tr t fd|Dr j /}}|sy|Dcic]"}||Dcgc]}|j |c}$}}}|Dcgc]}i||Dcic]}|dc}}}}tjj|}nt|rt|tjr|Dchc]1}t|r$|j |jvr |j 3}}|sy|D cic]O}|||D cgc]=} | tjus%t| trt!j"| rdn| ?c} Q}}} tjj%|j&d i|Dcic]}|dc}dj)d}ny|D]cj |vrtj*tj,rj*j.n j*} |j D cgc] } t1| } } ddht3| j4z r-t fd| Drt7d j d tj8| | } tj*tj,r*tj:j=j*| n| } |j?|j@jCj | }f|Scc}wcc}wcc}}wcc}wcc}}wcc}wcc} wcc} }wcc}wcc} w) zJConvert Python rows with blob fields before Arrow loses their value types.Nrc3^K|]$}t|txrj|v&ywr)rnrr)rrowfields rhrz'_blob_input_to_arrow..s)PCJsD)?ejjC.??Ps*-Fpreserve_indexrrc3TK|]duxrtfdD yw)Nc3DK|]}j|duywr)get)rrrs rhrz1_blob_input_to_arrow...sQDEIIdO4/Qs )r)rrmissing_range_fieldss @rhrz'_blob_input_to_arrow..s;(    RQZu%="((5/   $$ DKK 9%8$4:8 9%% ! !$ '   ::U " %**b&6&67 JJ # #  FLEJJEWXE07XX *F3c,:L:L6MM C( (( %  |,RR ((> =%**b&6&67    * *5::w ?    LL ( ( 4eV 5 : L 8J8J  9Ys`2M, M"6MM" M-' M(1M-6M3 M=AM8M= NNM"(M-8M=c~ ddlm}t|rt||jr?|j j }tjj|t|St||jjrzt|d}d|jvr7|jtj dtj"}tjj|t%|St|t&r t)dt|t*r t)dt|t,rf|s<| t)dtj.j1||j3St|dt&rt|dj4j7}|Dcgc] }t9|}}t;||xs|}tj.j1||j3St|dtj<r-tj.j|j3St;||}t?||}||j3Stj.j1|j3StA|rt|tBjDrt?||}|!tj.jG|d }|jHjJ|jHjJni}|jMDcic]\}}|d k7s ||}}}|jO|j3St|tj.r|j3St|tj<r+tjj|jH|gStQ|r8t|tRjTr|jWj3St|tjXjr|jWj3St|tjXjZr|j3St|tjr|St]|j^jad r7|j4jbd k(r|jej3St]|j^jad rE|j4jbd k(r,|jgjej3St|thr tk||Stmdt]|dcc}wcc}}w)Nr)datasetsrpz6Cannot add a single LanceModel to a table. Use a list.zWCannot create or add rows from a single dictionary. Use a list of dictionaries instead.z4Cannot create table from empty list without a schemarFrspandasr1r LazyFramezUnknown data type z. Supported types: list of dicts, pandas DataFrame, polars DataFrame, pyarrow Table/RecordBatch, or Pydantic models. See https://docs.lancedb.com/tables/ for examples.)7lancedb.dependenciesr'r,rnDatasetfeatures arrow_schemar RecordBatchReader from_batchesr dataset_dict DatasetDict_schema_from_hfrappendrstring_to_batches_with_splitrGrrr rr  to_reader __class__to_arrow_schemarH_serialize_json_values RecordBatchr%r.r rrrmetadataitemsrr-r/ LanceDatasetscannerdatasetScannerrrrrto_arrowcollectr_iterator_to_readerr) rrr' model_schemadr!metakvs rh_into_pyarrow_readerrIse.t$ dH,, -]]//F''44V[=NO O h33?? @$T40Ffll*rxx'EF''44.t4 $ #QRR$ 2  $~ !WXX88''V'<FFH H d1gz *7,,<<>L.23M!$3D3)$0F,GD88''\'BLLN N Q 088((.88: :)$7D(v6E*ik 020D0DT0Jik  4 Zbll%C$T62 =HH((e(DE(- (=(=(Iu||$$r!%@Ai1@@,,T2<<>> D"(( #~~ D".. )##00tfEE $ JtU5G5G$H||~'')) D"**,, -||~'')) D"**,, -~~ D".. /  T ((2 NN # #{ 2}}((** T ((2 NN # #{ 2||~&&(2244 D( #"400 d -A A  Y4"As:V4 V9V9c4t t}t|jfd}tj j |S#t$r0 tdtj j gcYSwxYw)Nz8Cannot create table from empty iterable without a schemac3TKEd{D]T}t|j}|jk7r |j}|jEd{Vy7^#tj j $rtdd|jwxYw7Ew)NzfInput iterator yielded a batch with schema that does not match the schema of other batches. Expected: z Got: ) rIread_allrcastr lib ArrowInvalidr to_batches)batchr!rfirstrs rhgenz _iterator_to_reader..genHs *E259BBDE||v%!JJv.E'') ) ) *  vv**$&&,XXell^E *s2B(A(1B(A* B( B&!B(*9B##B() iternext StopIterationrr r.r/rIr)rr first_batchrSrRs`` @rhrCrC9s :D=4j ! -E \\F *    , ,VSU ;;- = >WX X##00<<=s A6BB)allow_subschemact||}t|||}t|||||}|t|\}}|r%|j t |j |}t|t|||}|S)a Handle input data, applying all standard transformations. This includes: * Converting the data to a PyArrow Table * Adding vector columns defined in the metadata * Adding embedding metadata into the schema * Casting the table to the target schema * Handling bad vectors Parameters ---------- target_schema : Optional[pa.Schema], default None The schema to cast the table to. This is typically the schema of the table if it already exists. Otherwise it might be a user-requested schema. allow_subschema : bool, default False If True, the input table is allowed to omit columns from the target schema. The target schema will be filtered to only include columns that are present in the input table before casting. metadata : Optional[dict], default None The embedding metadata to add to the schema. on_bad_vectors : Literal["error", "drop", "fill", "null"], default "error" How to handle vectors with incorrect sizes, NaNs, or null elements. fill_value : float, default 0.0 The value to use when filling vectors. Only used if on_bad_vectors="fill". NaN or null elements and missing trailing elements are replaced with this value. Valid elements are preserved. r;)on_bad_vectors fill_value target_schemar;) rI_append_vector_columns_handle_bad_vectors_infer_target_schema with_metadata_merge_metadatar;_validate_schema_cast_to_target_schema)rr]r;r[r\rXreaders rh_sanitize_datarfZsT"$ 6F #FMH MF!%# F 4V < v%33 M22H = ]# #FM? KF Mrjcjj|drSttt jtt |}t j||j |s"tt|k7r td|rrtt|k7r[ttt jtt }t j||j fd}t jj|S)NTcheck_metadatarZz>Input table has different number of columns than target schemac3KD]~}t|}tjj|gj j }|sOtj j|djyw)Nrr() _coerce_blob_write_columnsr rr/rMrPr: from_arraysr)rQ cast_batchesrereordered_schemas rhrSz#_cast_to_target_schema..gens E.u6FGE%%ug.334DEPPR nn00 O++4D1 s AB3B) requals_align_field_typesr rTr r;lenr_infer_subschemar.r/)rer]rXfieldsrSrns` @rhrdrds}}M$? T&--%8 94]@S;T UFyy-2H2HI s#34M8JJ L  3/0C 4FF! fmm$ %tD1A,B'C 99Vm6L6LM     , ,-=su EErjc|Dcic]}|j|}}|Dcgc]+}|j|vrt|||jn|-}}tj||jScc}wcc}w)zKKeep inferred blob semantics without overriding explicit replacement types.rZ)r_preserve_blob_v2_fieldr rr;)inferred_schemaexisting_schemarexisting_fieldsrss rh_preserve_blob_v2_fieldsrys7FFUuzz5(FOF %   :: ( uzz'BC  F 99Vo&>&> ??Gs A20A7c2t|r|St|r|S|j|j}}tjj |rtjj |rt|Dcic]}|j |}}|Dcgc]+}|j |vrt |||j n|-}}|jtj|St|rt|rt |j|j}||jk(r|S|j|jj }tjj|r!tj||j}nJtjj|rtj |}ntj|}|j|St#tj$|gr|S|Scc}wcc}wr)r`rr types is_structrru with_typestruct _is_list_like value_field with_nameis_fixed_size_listlist_ list_size is_large_list large_listr_r) inferredexisting inferred_type existing_typerexisting_childrenchildrenr value_types rhrurus!!#+==(--=M xx-(RXX-?-? -N> % % &5<<3H3HI & rjc  t|r!t|jr t||S|j}tj j |rStj j |jr)g}g}d}|jD] |j j}t fd|Dd}|#|j||j Xt||}||urd}t|\} } |j| |jt j j| j|j|s|Stjj!|||j"r|j%SdSt'|r+t'|jrt)||j*S|S)NFc3VK|] }|jjk(s|"ywrr)rr source_fields rhrz%_coerce_blob_value..(s"S55::ARAR3RS))Trsmask)r`_can_coerce_to_blobr_coerce_value_to_blobr r{r|rrrUr3r_physical_array_and_typerr; StructArrayrl null_countis_nullr_coerce_blob_list_valuesr) r$r target_typerrsr source_column nested_targetr child_array child_typers @rhrrs %*=fkk*J$V\::##K xx+&288+=+=fkk+J"KK L"LL):):;M SKSM$ . l+( FGm+&>w&G #K OOK ( MM %% ))!**   .M~~)) %+%6%6!*  =A*  [!mFKK&@' 0G0GHH Mrjc|jr|jnd}tjj |j r|j j }|jj|j|zt||z}t||}||ur|St|\}}tjj|||S|j}|dj!} |jj| |dj!| z }t||}||ur|St|\}}| r4t#j$|tj&| |j }tjj)|j r"tj*j|||Stj,j|||S)zCoerce blob values inside a list column, preserving offsets and nulls. Works on the raw child values window instead of ``pc.list_flatten`` because flatten drops values spanned by null slots, which would misalign offsets. N)rr)rrr r{rrrrsliceoffsetrqrrFixedSizeListArrayrloffsetsas_pypcsubtractscalarrLargeListArray ListArray) r$target_value_fieldrrrrphysical_values_r first_offsets rhrrJs &006>> dD xx""6;;/KK)) $$V]]Y%>F i@WX$V-?@ f M5g>$$00)RV0WWnnG1:##%L ]]  l*F!);,> ?@ OOBHHV-2D2DE FGnn((!'!2!2V^^ )G ,##R%5%56  --l.?.?II Nrjct|jtjr"|j|jj fS||jfSr)rnrr rr#r)rs rhrrs>%**b../}}ejj5555 %** rjctt|xs,t|xstjj |Sr)_is_binary_likerr r{r data_types rhrrs3 " ' 9 % ' 88  I &rjctjj|xs@tjj|xstjj |Sr)r r{ is_stringis_large_stringis_string_viewrs rhrrG 9% . 88 # #I . . 88 " "9 -rjctjj|xs@tjj|xstjj |Sr)r r{ is_binaryis_large_binaryis_binary_viewrs rhrrrrjct|jdd}||S|jxsi}|jdxs|jd}t |t r|j S|S)Nextension_nameARROW:extension:namezARROW:extension:name)rrr;rrnrdecode)rrr;s rh_field_extension_namersrUZZ)94@N!~~#H\\"9:hll?N.%($$&& rjct|dvS)N) arrow.json lance.json)r)rs rh_is_json_fieldrs  '+G GGrjT)frozenc4eZdZUded<dZded<dZded<y)_JsonSerializationPlanpa.Field arrow_fieldNz-Optional[Dict[str, '_JsonSerializationPlan']]rz"Optional['_JsonSerializationPlan']item)rrrrrrrerjrhrrs>BH;B/3D ,3rjrct|r t|Stjj |j r?i}|j D]}t |}||||j<!|r t||St|j r.t |j j}| t||Sy)N)r)r) rrr r{r|r_json_serialization_planrrr)rr child_field child_plan item_plans rhrrse%e,, xx%**%68 :: 8K1+>J%-7))* 8 )%(C CUZZ ,UZZ-C-CD  )%i@ @ rjcRi}|D]}t|}||||j<!|Sr)rr)rplansrplans rh_json_serialization_plansrs=02E%'.   $E%** % Lrjcb|t|tr|St|jr-t|tt frt j|S|S|jht|trXd}|jjD]3\}}||vr t|||}|||us"| t |}|||<5||S|S|jXt|t rHd}t|D]2\}}t||j}||us!| t |}|||<4||S|S|Sr) rnrorrrr jsondumpsrr<_serialize_json_valuer enumerate) rr serialized child_namer child_valueindexrserialized_items rhrrsJ } 5#. d&&' edD\ *::e$ $  }} Zt%< &*mm&9&9&; 5 "J &/j0A:NK% "33%!%eJ)4 :& 5(3z>> yyE4!8 $U+ 4KE43D$))DOd*%!%eJ$3 5!  4 (3z>> Lrjc\|t|ts|St|}|s|Sg}|D]}t|ts|j |%d}|j D]3\}}||vr t |||}|||us"| t|}|||<5|j ||n||Sr)rnr rrr3r<r) rr]rserialized_rowsrserialized_row field_namerrs rhr9r9sJtT$: %m 4E  O V#t$  " "3 '  %  3 J$)#j/4@EC O+!)%)#YN-2z* 3 1K~QTU V rjcg}|D]Mtfd|Dd}|tdjd|jt |O|S)zD Apply the data types from the target_fields to the fields. c3VK|] }|jjk(s|"ywrr)rfrs rhrz%_align_field_types..!s N15::9MQNrNzField 'z' not found in target schema)rUrrr3 _align_field)rs target_fields new_fieldsrrs @rhrprpshJ=N NPTU  wuzzl2NOP P,ul;< = rjcLt||j|jSr)rrr)child target_childs rh_align_container_childr(s" | , 6 6|7H7H IIrjctjj|stjj|rtjStjj |rtj Sy)zsThe storage type arrow.json would use for ``input_type``. Returns None if the type cannot hold JSON text. N)r r{rrr4r large_string) input_types rh_arrow_json_storage_typer.sU  xx*%)@)@)Lyy{ xx +  rjc t|r/|j}tjj |r+t d|DsZt d|jdt|s6t|tjst d|jd|dtjj |s^t|tjr_t|jtjr t|t|jur|j|jSt|dk(rqt|dk(r|St|j}|Jt|jxsi}d |d <tj |j||j"|Stjj |jrtjj |jrItj$t'|jj(|jj(}n|j}ntjj+|jrlt-|jrItj.t1|jj2|jj2}nH|j}n:tjj5|jrlt-|jrItj6t1|jj2|jj2}n|j}ntjj9|jrt-|jr^tj.t1|jj2|jj2|jj:}n |j}ntjj=|jrtjj=|jrtj>t1|jj@|jj@t1|jjB|jjB|jjD }n|j}n |j}tj |j||j"|jS) Nc38K|]}|jdvyw))rrNr)rrs rhrz_align_field..>sNuzz_4Nszblob struct input for column 'z&' must contain a 'data' or 'uri' childzcannot coerce column 'z ' with type z into a blob v2 struct. expected binary bytes (Binary, LargeBinary, BinaryView), strings (Utf8, LargeUtf8, Utf8View), or a Struct with a 'data' or 'uri' childrrs arrow.jsonr) keys_sorted)#r`rr r{r|rrrrrnrrar;rrrrrr~rprsis_listrrrrrrrris_mapmap_ key_field item_fieldr)rr source_type json_storager;new_types rhrr:s %jj 88  k *N+NN 4UZZLA..%[1* )); ( L N((  88  k * {B$4$4 5|00"2B2BC $\->->(?? &&|'<'<= =\*l:  '< 7L0 ;  #L117R8H0=H, -88EJJ ennhO O xx,++, 88  ejj )yy"JJ%% %%,,H$((H   ,++ ,  $xx&JJ**L,=,=,I,IH $((H    1 1 2  $}}&JJ**L,=,=,I,IH $((H  $ $\%6%6 7  $xx&JJ**L,=,=,I,I!!++ H$((H ** + 88??5:: &ww&JJ((,*;*;*E*E'JJ))<+<+<+G+G)--99H$((H$$ 88EJJ%..,:O:O PPrjc"g}|Dcic]}|j|}}|D]}|j|j}|tdj|tj j |jrst jt|jj|jj}t j|j||j}n|}|j||Scc}w)z Transform the list of fields so the types match the reference_fields. The order of the fields is preserved. ``schema`` may have fewer fields than `reference_fields`, but it may not have more fields. zUnexpected field in schema: {})rrrformatr r{r|rr~rrrsrrr3) rreference_fieldsrsrlookupr referencer  new_fields rhrrrrsF!1 2Aaffai 2F 2!JJuzz*  =DDUKL L 88  inn -yy JJ%%NN))H  ""I "I i )!, M/3sD ctj|r t|tr|j }|-|| |j }t |||||}|j}n#|!tjjg|}|%| tdt|dr |j}|rt|j |}|j|}t|tjr|j!|}||fSt|tj"r tj"j||}||fS)N)r;r[r\r(z&Either data or schema must be providedr)inspectisclass issubclassrGr8r;rfrr rr/rhasattrrbrarnrr.)rrr;r[r\s rhsanitize_create_tablers0v:fj#A#224    2H  )!    88((F(;D ~ <EF F T8 $[[F"6??H=%%h/ dBHH %//9D <b22 3''44VTBD <rjc|jD]A}||jj}||jjk7s6d}t||S)z Extract pyarrow schema from HuggingFace DatasetDict and validate that they're all the same schema between splits zCAll datasets in a HuggingFace DatasetDict must have the same schema)rr,r-r)rrr?msgs rhr2r2sY ;;=! >%%22F w''44 4WCC. ! Mrjc #vK|jD]\}}t|D]}tjj |g}d|j vrF|j dtj|g|jztj}|jD]}|yw)zm Return a generator of RecordBatches from a HuggingFace DatasetDict with an extra `split` column rpN) r<rr rr/ column_names append_columnrnum_rowsr4rP)rkeyr?rQr!bs rhr5r5 s   W ) EHH))5'2Ee000++RXXseenn&.gen7sn E'0'8 # t}}#08J8J#J #rvvbjj}9M.N'O'U'U'W#HHd001 H&*]]%8%8%:N"* ]s1v/GTQ H ( % 3 3"LL7HHXFLL4O4T4TU! !& 0 0!..44]C"LL7HHXFLL4O4T4TU!/ 8K;  sA3F7(> ?F@    , ,VSU ;;rjct||}t|}|jdk(r|jd}|jdj S)z Get a table path that can be used in PyArrow FS. Removes any weird schemes (such as "s3+ddb") and drops any query params. zs3+ddbs3)schemeNquery) _table_urirr2_replacegeturl)base table_namerparseds rh _table_pathr;ZsO T: &C c]F }} - ??? & - - //rjc t||dS)Nz.lance)r\)r8r9s rhr5r5js DZL/ 00rjcB|durddlm}|ddfS|dus|y|dfS)zNormalize a ``progress`` parameter for :meth:`Table.add`. Returns ``(progress_obj, owns)`` where *owns* is True when we created a tqdm bar that the caller must close. Tr)tqdmz rows)unitF)NF) tqdm.autor>)progressr>s rh_normalize_progressrBns9 4"!4''5H, U?rjcneZdZdZeedodZedpdZedodZeedqdZ eedrdZ eedsdZ edtdZ dud Z dqd Zeedvd Zedwdxd ZdydzdZeddd{dZd|dZdwd}dZ d~dZ ddZeddZeddddd ddZe dddddd ddZdd d edd d ddd d!d"d#f d d d dd d$ dd%Zd d d dd dd&Zdd'Zed#(f dd)Zedd*Z edd+d d d, dd-Z!d d.d/d.d d.d0d1d2dd d d dd3d3d.d4e"jFd d d5 dd6Z$e d dd7Z%dd8Z&e d dd9Z'ed.d: dd;Z(dd<Z)ed.d: dd=Z*edd>Z+e dd?Z,e dd@Z-e ddAZ.e ddBZ/ddCZ0ed d dD ddEZ1edddFZ2edGdH ddIZ3eddJZ4e ddKZ5eddLZ6e dd dM ddNZ7e dwd.dO ddPZ8edQZ9ed d.d.d dR ddSZ:eddTZ;ed d dU ddVZeddZZ?edd[Z@e d dd\ZAedd]ZBe dd^ZCeDjd_eFd`addbZGeddcZHedddZIedeZJedwddfZKeddgZLeMdodhZNddiZOdjZPeddkZQedlZRddmZSddnZTy )ra& A Table is a collection of Records in a LanceDB Database. Examples -------- Create using [DBConnection.create_table][lancedb.DBConnection.create_table] (more examples in that method's documentation). >>> import lancedb >>> db = lancedb.connect("./.lancedb") >>> table = db.create_table("my_table", data=[{"vector": [1.1, 1.2], "b": 2}]) >>> table.head() pyarrow.Table vector: fixed_size_list[2] child 0, item: float b: int64 ---- vector: [[[1.1,1.2]]] b: [[2]] Can append new data with [Table.add()][lancedb.table.Table.add]. >>> table.add([{"vector": [0.5, 1.3], "b": 4}]) AddResult(version=2) Can query the table with [Table.search][lancedb.table.Table.search]. >>> table.search([0.4, 0.4]).select(["b", "vector"]).to_pandas() b vector _distance 0 4 [0.5, 1.3] 0.82 1 2 [1.1, 1.2] 1.13 Search queries are much faster when an index is created. See [Table.create_index][lancedb.table.Table.create_index]. ctd)zThe name of this Tablez)name is not supported for this table typeNotImplementedErrorselfs rhrz Table.names""MNNrjcgS)zFReturn the namespace path of the table (empty for the root namespace).rerGs rh namespacezTable.namespaces  rjcTdj|j|jgzS)z9Return the full identifier of the table (namespace$name).$)rrJrrGs rhidzTable.ids"xx$))455rjctd)zThe version of this Tablez,version is not supported for this table typerErGs rhversionz Table.versions""PQQrjctd)lThe [Arrow Schema](https://arrow.apache.org/docs/python/api/datatypes.html#) of this Table z+schema is not supported for this table typerErGs rhrz Table.schemas""OPPrjctd)aWTag management for the table. Similar to Git, tags are a way to add metadata to a specific version of the table. .. warning:: Tagged versions are exempted from the :py:meth:`cleanup_old_versions()` process. To remove a version that has been tagged, you must first :py:meth:`~Tags.delete` the associated tag. Examples -------- .. code-block:: python table = db.open_table("my_table") table.tags.create("v2-prod-20250203", 10) tags = table.tags.list() z)tags is not supported for this table typerErGs rhtagsz Table.tagss6""MNNrjctd)Branch management for the table. Branches are isolated, writable lines of history forked from another branch (or version). Writes on a branch do not affect ``main``. z-branches is not supported for this table typerErGs rhbrancheszTable.branchess""QRRrjctd)DThe branch this table handle is scoped to, or ``None`` for ``main``.z3current_branch is not supported for this table typerErGs rhcurrent_branchzTable.current_branchs!"WXXrjc$|jdS)z The number of rows in this TableN) count_rowsrGs rh__len__z Table.__len__st$$rjcy)z^ Get a mapping from vector column name to it's configured embedding function. NrerGs rhembedding_functionszTable.embedding_functionsrjNctd) Count the number of rows in the table. Parameters ---------- filter: str, optional A SQL where clause to filter the rows to count. z/count_rows is not supported for this table typerErHfilters rhr[zTable.count_rowss""STTrjlazyc B|jjdi|S)a~Return the table as a pandas DataFrame. Parameters ---------- blob_mode: str, default "lazy" Controls how blob columns are returned for backends that support Lance blob-aware pandas conversion. **kwargs Forwarded to PyArrow / Lance pandas conversion. Returns ------- pd.DataFrame re)rA to_pandas)rH blob_modekwargss rhrfzTable.to_pandass )t}}((2622rjrgctd)Return the table as a pyarrow Table. Parameters ---------- blob_mode: str, default "lazy" ``"lazy"`` returns blob v2 descriptors that carry the row id ``fetch_blobs`` reads. ``"descriptions"`` returns them without it. ``"bytes"`` returns payload bytes and raises for legacy blob columns and blobs inside lists. Returns ------- pa.Table z-to_arrow is not supported for this table typerErHrgs rhrAzTable.to_arrows ""QRRrjc td)aReturn a local table as a lance.LanceDataset. Remote tables cannot be opened as Lance datasets by the client. Use ``table.search().to_arrow()`` to read them through the server. Returns ------- lance.LanceDataset z-to_lance is not supported for this table typerE)rHrhs rhto_lancezTable.to_lance&s""QRRrjctd)amReturn the table as a Polars LazyFrame. For remote tables, use ``table.search().to_polars()`` to convert query results to a Polars DataFrame. Parameters ---------- batch_size: int, optional Passed to Polars as the maximum row count for scanned PyArrow record batches. Note ---- The Polars streaming engine is not supported because it does not currently implement Python PyArrow dataset scans. Use the default engine when collecting this LazyFrame. Returns ------- polars.LazyFrame z.to_polars is not supported for this table typerE)rHrs rh to_polarszTable.to_polars2s,""RSSrjcz|ytd||||fDry|tk7ry|jtvryy)z-Detect if this is a legacy create_index call.Fc3$K|]}|du ywrre)rxs rhrz5Table._is_legacy_create_index_call..Zs  TM sT)rr4lower KNOWN_METRICS)rH first_argconfignum_partitionsnum_sub_vectorsvector_column_namerxindex_cache_sizes rh_is_legacy_create_index_callz"Table._is_legacy_create_index_callJsZ    $o{DTU   !3 3 ??  -rjc f|dk(rt||||| | S|dk(rt||||| | S|dk(rt||||||| | S|dk(rt|||||| | S|dk(rt |||||||| | | S|d k(rt |||||| | | S|d k(rt |||||| | Std|)z4Build an index config object from legacy parameters.IVF_FLAT) distance_typerxmax_iterations sample_ratetarget_partition_sizerxIVF_SQIVF_PQ)rrxrynum_bitsrrrrxIVF_RQ)rrxrrrrrx IVF_HNSW_PQ) rrxryrrrmef_constructionrrx IVF_HNSW_SQ)rrxrrrrrrx IVF_HNSW_FLAT)rrxrrrrrUnknown index type )r8r:r9r<r?r@rAr) rHmetric index_typerxryrrrrrrrxs rh'_build_vector_config_from_legacy_paramsz-Table._build_vector_config_from_legacy_paramsksT  #$--'&;'  8 #$--'&;'  8 #$- /!-'&;'  8 #$-!-'&;' = ($- /!-' /&;'  = ($--' /&;'  ? *$--' /&; 2:,?@ @rjc 0|dk(r ddddddddddd S|d k(r d dddddddddd S|d k(r d dddddddddd St|d k7rtd ||dd }|dddk7rtd ||tvrtd|dt|ddddddddd S)NdefaultsimpleEnglish(TFr) rrmax_token_length lower_casestemremove_stop_words ascii_foldingngram_min_lengthngram_max_length prefix_onlyraw whitespacezInvalid tokenizer name _stemzInvalid language code )rqrr^)tokenizer_namelangs rhinfer_tokenizer_configszTable.infer_tokenizer_configss0 Y &"*%$&"%*!&$%$%$  u $"'%$(#%*!&$%$%$  | +".%$(#%*!&$%$%$   ~ ! #6~6FGH Hbq! "# ' )6~6FGH H | #5dV<= =&$T* "!&" ! !   rj.replace wait_timeoutrtraincyrrerHr$rwrrrrs rh create_indexzTable.create_indexrjrrrrcyrrerHrrxryrzrrxr{rrrrrrrrrrs rhrzTable.create_index.rjrTr2,rwrrrrctd)Create an index on a column. This method supports both the new unified API and the legacy API for backwards compatibility. The new API takes the column name as the first positional argument and an index configuration object via ``config``; the legacy API takes the distance metric as the first argument plus separate ``vector_column_name`` / ``num_partitions`` / etc. parameters, and emits a ``DeprecationWarning``. Parameters ---------- metric : str For new API: the column name to index. For legacy API: the distance metric ("l2", "cosine", "dot", "hamming"). config : IndexConfigType, optional The index configuration object. If provided, uses the new unified API. Can be one of: IvfFlat, IvfPq, IvfSq, IvfRq, HnswPq, HnswSq, BTree, Bitmap, LabelList, Fm, FTS. replace : bool, default True Whether to replace an existing index on this column. wait_timeout : timedelta, optional Timeout to wait for async indexing to complete. name : str, optional Custom name for the index. train : bool, default True Whether to train the index with existing data. Examples -------- New API (recommended): >>> table.create_index( # doctest: +SKIP ... "vector", config=IvfPq(distance_type="l2") ... ) >>> table.create_index("category", config=BTree()) # doctest: +SKIP >>> table.create_index("content", config=FTS()) # doctest: +SKIP Legacy API (deprecated): >>> table.create_index( # doctest: +SKIP ... "l2", vector_column_name="vector" ... ) z1create_index is not supported for this table typerE)rHrrxryrzrrxr{rrrrrrrwrrrrs rhrzTable.create_index-sR""UVVrjctd)aCreate an index, returning a handle to the indexing job. Takes the same arguments as :meth:`create_index`. The job may already be complete when returned; callers must not assume the index exists until :meth:`Job.wait` returns. z7create_index_async is not supported for this table typerErs rhcreate_index_asynczTable.create_index_asyncxs " E  rjctd)a Drop an index from the table. Parameters ---------- name: str The name of the index to drop. Notes ----- This does not delete the index from disk, it just removes it from the table. To delete the index, run [optimize][lancedb.table.Table.optimize] after dropping the index. Use [list_indices][lancedb.table.Table.list_indices] to find the names of the indices. z/drop_index is not supported for this table typerErHrs rh drop_indexzTable.drop_indexs$""STTrjsecondsctd) Wait for indexing to complete for the given index names. This will poll the table until all the indices are fully indexed, or raise a timeout exception if the timeout is reached. Parameters ---------- index_names: str The name of the indices to poll timeout: timedelta Timeout to wait for asynchronous indexing. The default is 5 minutes. z3wait_for_index is not supported for this table typerErH index_namestimeouts rhwait_for_indexzTable.wait_for_indexs""WXXrjctd)9 Retrieve table and fragment statistics. z*stats is not supported for this table typerErGs rhstatsz Table.statss ""NOOrjBTREErrrrctd)a Create a scalar index on a column. Parameters ---------- column : str The column to be indexed. Must be a boolean, integer, float, or string column. replace : bool, default True Replace the existing index if it exists. index_type: Literal["BTREE", "BITMAP", "LABEL_LIST"], default "BTREE" The type of index to create. wait_timeout: timedelta, optional The timeout to wait if indexing is asynchronous. name: str, optional The name of the index. If not provided, a default name will be generated. Examples -------- Scalar indices, like vector indices, can be used to speed up scans. A scalar index can speed up scans that contain filter expressions on the indexed column. For example, the following scan will be faster if the column ``my_col`` has a scalar index: >>> import lancedb # doctest: +SKIP >>> db = lancedb.connect("/data/lance") # doctest: +SKIP >>> img_table = db.open_table("images") # doctest: +SKIP >>> my_df = img_table.search().where("my_col = 7", # doctest: +SKIP ... prefilter=True).to_pandas() Scalar indices can also speed up scans containing a vector search and a prefilter: >>> import lancedb # doctest: +SKIP >>> db = lancedb.connect("/data/lance") # doctest: +SKIP >>> img_table = db.open_table("images") # doctest: +SKIP >>> img_table.search([1, 2, 3, 4], vector_column_name="vector") # doctest: +SKIP ... .where("my_col != 7", prefilter=True) ... .to_pandas() Scalar indices can only speed up scans for basic filters using equality, comparison, range (e.g. ``my_col BETWEEN 0 AND 100``), and set membership (e.g. `my_col IN (0, 1, 2)`) Scalar indices can be used if the filter contains multiple indexed columns and the filter criteria are AND'd or OR'd together (e.g. ``my_col < 0 AND other_col> 100``) Scalar indices may be used if the filter contains non-indexed columns but, depending on the structure of the filter, they may not be usable. For example, if the column ``not_indexed`` does not have a scalar index then the filter ``my_col = 0 OR not_indexed = 1`` will not be able to use any scalar index on ``my_col``. z8create_scalar_index is not supported for this table typerE)rHr$rrrrs rhcreate_scalar_indexzTable.create_scalar_indexs~" F  rjF@rrrrordering_field_namesrwriter_heap_size use_tantivyr with_positionrrrrrrcustom_stop_wordsrrrr block_sizedocument_granularityrrctd)uCreate a full-text search index on the table. Warning - this API is highly experimental and is highly likely to change in the future. Parameters ---------- field_names: str or list of str The name of the field to index. Native FTS indexes can only be created on a single field at a time. To search over multiple text fields, create a separate FTS index for each field. replace: bool, default False If True, replace the existing index if it exists. Note that this is not yet an atomic operation; the index will be temporarily unavailable while the new index is being created. writer_heap_size: int, default 1GB Deprecated legacy Tantivy parameter. Any value other than the default raises an error. ordering_field_names: Deprecated legacy Tantivy parameter. Setting this raises an error. tokenizer_name: str, default "default" A compatibility alias for native tokenizer configs. Can be "raw", "default" or the 2 letter language code followed by "_stem". So for english it would be "en_stem". For new native FTS indexes, use ``base_tokenizer`` directly; ``tokenizer_name`` is a legacy compatibility alias and does not expose model-backed tokenizer names such as ``jieba/default`` or ``lindera/ipadic``. use_tantivy: bool, default False Deprecated legacy Tantivy parameter. Setting this to True raises an error. with_position: bool, default False If False, do not store the positions of the terms in the text. This can reduce the size of the index and improve indexing speed. But it will raise an exception for phrase queries. base_tokenizer : str, default "simple" The base tokenizer to use for tokenization. Options are: - "simple": Splits text by whitespace and punctuation. - "whitespace": Split text by whitespace, but not punctuation. - "raw": No tokenization. The entire text is treated as a single token. - "code": Tokenizes source code and identifiers. - "ngram": N-Gram tokenizer. - "icu": ICU dictionary-based word segmentation. - "icu/split": ICU segmentation with simple-style delimiter splitting. - "jieba/*": Jieba tokenizer loaded from Lance's language model home. - "lindera/*": Lindera tokenizer loaded from Lance's language model home. language : str, default "English" The language to use for stemming and stop-word removal. This is not the primary way to enable CJK tokenization. max_token_length : int, default 40 The maximum token length to index. Tokens longer than this length will be ignored. lower_case : bool, default True Whether to convert the token to lower case. This makes queries case-insensitive. stem : bool, optional Whether to stem the token. Stemming reduces words to their root form. For example, in English "running" and "runs" would both be reduced to "run". ``None`` uses the base tokenizer's default: False for ``code`` and ``ngram``, True otherwise. remove_stop_words : bool, optional Whether to remove stop words. Stop words are common words that are often removed from text before indexing. For example, in English "the" and "and". ``None`` uses the base tokenizer's default: False for ``code`` and ``ngram``, True otherwise. custom_stop_words : list of str, optional Custom words that replace the built-in language stop words. ``None`` uses the built-in list; an empty list explicitly uses no stop words. ascii_folding : bool, default True Whether to fold ASCII characters. This converts accented characters to their ASCII equivalent. For example, "café" would be converted to "cafe". ngram_min_length: int, default 3 The minimum length of an n-gram. ngram_max_length: int, default 3 The maximum length of an n-gram. prefix_only: bool, default False Whether to only index the prefix of the token for ngram tokenizer. block_size: int, default 128 The number of documents per compressed posting block. Must be 128 or 256. A value of 256 uses the experimental FTS V3 format and may introduce breaking changes. document_granularity: DocumentGranularity, default ROW ``ROW`` treats the selected text in one table row as one document. ``LIST_ELEMENT`` treats each element of the deepest list on the field path as one document and returns its physical coordinates in ``_doc_index`` for matching queries. wait_timeout: timedelta, optional The timeout to wait if indexing is asynchronous. name: str, optional The name of the index. If not provided, a default name will be generated. Notes ----- Model-backed tokenizers such as ``jieba/default`` and ``lindera/ipadic`` require tokenizer models in Lance's language model home. Set ``LANCE_LANGUAGE_MODEL_HOME`` to override the default platform data directory under ``lance/language_models``. z5create_fts_index is not supported for this table typerE)rH field_namesrrrrrrrrrrrrrrrrrrrrrs rhcreate_fts_indexzTable.create_fts_indexsx" C  rjctd)ad Add more data to the [Table][lancedb.table.Table]. Parameters ---------- data: DATA The data to insert into the table. Acceptable types are: - list-of-dict - pandas.DataFrame - pyarrow.Table or pyarrow.RecordBatch mode: str The mode to use when writing the data. Valid values are "append" and "overwrite". Overwrite adopts explicitly declared blob layouts, while inferred bytes retain the existing blob v2 fields. on_bad_vectors: str, default "error" How to handle vectors with incorrect sizes, NaNs, or null elements. One of "error", "drop", "fill", "null". fill_value: float, default 0. The value to use when filling vectors. Only used if on_bad_vectors="fill". progress: bool, callable, or tqdm-like, optional Progress reporting during the add operation. Can be: - ``True`` to automatically create and display a tqdm progress bar (requires ``tqdm`` to be installed):: table.add(data, progress=True) - A **callable** that receives a dict with keys ``output_rows``, ``output_bytes``, ``total_rows``, ``elapsed_seconds``, ``active_tasks``, ``total_tasks``, and ``done``:: def on_progress(p): print(f"{p['output_rows']}/{p['total_rows']} rows, " f"{p['active_tasks']}/{p['total_tasks']} workers") table.add(data, progress=on_progress) - A **tqdm-compatible** progress bar whose ``total`` and ``update()`` will be called automatically. The postfix shows write throughput (MB/s) and active worker count:: with tqdm() as pbar: table.add(data, progress=pbar) write_parallelism: int, optional Number of partitions to write in parallel. Higher values increase throughput but also peak memory use, since each partition buffers data in flight. Defaults to an estimate based on the data size, capped at the number of CPU cores. Lower this if bulk ingestion is using too much memory. allow_external_blob_outside_bases: bool, default False Store blob URIs that sit outside registered blob bases. The row keeps a reference, so the object has to stay readable. Local tables only. Returns ------- AddResult An object containing the new version number of the table after adding data. z(add is not supported for this table typerE)rHrmoder[r\rAwrite_parallelism!allow_external_blob_outside_basess rhaddz Table.add{sP""LMMrjcht|tr|gntt|}t ||Sa Returns a [`LanceMergeInsertBuilder`][lancedb.merge.LanceMergeInsertBuilder] that can be used to create a "merge insert" operation This operation can add rows, update rows, and remove rows all in a single transaction. It is a very generic tool that can be used to create behaviors like "insert if not exists", "update or insert (i.e. upsert)", or even replace a portion of existing data with new data (e.g. replace all data where month="january") The merge insert operation works by combining new data from a **source table** with existing data in a **target table** by using a join. There are three categories of records. "Matched" records are records that exist in both the source table and the target table. "Not matched" records exist only in the source table (e.g. these are new data) "Not matched by source" records exist only in the target table (this is old data) The builder returned by this method can be used to customize what should happen for each category of data. Please note that the data may appear to be reordered as part of this operation. This is because updated rows will be deleted from the dataset and then reinserted at the end with the new values. Parameters ---------- on: Union[str, Iterable[str]] A column (or columns) to join on. This is how records from the source table and target table are matched. Typically this is some kind of key or id column. Passing several columns matches on the composite key: a source row updates a target row only when it agrees on every one of them. Examples -------- >>> import lancedb >>> data = pa.table({"a": [2, 1, 3], "b": ["a", "b", "c"]}) >>> db = lancedb.connect("./.lancedb") >>> table = db.create_table("my_table", data) >>> new_data = pa.table({"a": [2, 3, 4], "b": ["x", "y", "z"]}) >>> # Perform a "upsert" operation >>> res = table.merge_insert("a") \ ... .when_matched_update_all() \ ... .when_not_matched_insert_all() \ ... .execute(new_data) >>> res MergeResult(version=2, num_updated_rows=2, num_inserted_rows=1, num_deleted_rows=0, num_attempts=1, num_rows=3) >>> # The order of new rows is non-deterministic since we use >>> # a hash-join as part of this operation and so we sort here >>> table.to_arrow().sort_by("a").to_pandas() a b 0 1 b 1 2 x 2 3 y 3 4 z rnror rTrFrHons rh merge_insertzTable.merge_insert,x C(bTd48n&tR00rjctd)a Create a search query to find the nearest neighbors of the given query vector. We currently support [vector search](https://lancedb.com/docs/search/vector-search/) and [full-text search](https://lancedb.com/docs/search/full-text-search/). All query options are defined in [LanceQueryBuilder][lancedb.query.LanceQueryBuilder]. Examples -------- >>> import lancedb >>> db = lancedb.connect("./.lancedb") >>> data = [ ... {"original_width": 100, "caption": "bar", "vector": [0.1, 2.3, 4.5]}, ... {"original_width": 2000, "caption": "foo", "vector": [0.5, 3.4, 1.3]}, ... {"original_width": 3000, "caption": "test", "vector": [0.3, 6.2, 2.6]} ... ] >>> table = db.create_table("my_table", data) >>> query = [0.4, 1.4, 2.4] >>> (table.search(query) ... .where("original_width > 1000", prefilter=True) ... .select(["caption", "original_width", "vector"]) ... .limit(2) ... .to_pandas()) caption original_width vector _distance 0 foo 2000 [0.5, 3.4, 1.3] 5.220000 1 test 3000 [0.3, 6.2, 2.6] 23.089996 Parameters ---------- query: list/np.ndarray/str/PIL.Image.Image, default None The targeted vector to search for. - *default None*. Acceptable types are: list, np.ndarray, PIL.Image.Image - If None then the select/where/limit clauses are applied to filter the table vector_column_name: str, optional The name of the vector column to search. The vector column needs to be a pyarrow fixed size list type - If not specified then the vector column is inferred from the table schema - If the table has multiple vector columns then the *vector_column_name* needs to be specified. Otherwise, an error is raised. query_type: str *default "auto"*. Acceptable types are: "vector", "fts", "hybrid", or "auto" - If "auto" then the query type is inferred from the query; - If `query` is a list/np.ndarray then the query type is "vector"; - If `query` is a PIL.Image.Image then either do vector search, or raise an error if no corresponding embedding function is found. - If `query` is a string, then the query type is "vector" if the table has embedding functions else the query type is "fts" Returns ------- LanceQueryBuilder A query builder object representing the query. Once executed, the query returns - selected columns - the vector - and also the "_distance" column which is the distance between the query vector and the returned vector. z+search is not supported for this table typerErHr4rz query_typeordering_field_name fts_columns fast_searchs rhsearchz Table.search sn""OPPrj with_row_idcy)a Take a list of offsets from the table. Offsets are 0-indexed and relative to the current version of the table. Offsets are not stable. A row with an offset of N may have a different offset in a different version of the table (e.g. if an earlier row is deleted). Offsets are mostly useful for sampling as the set of all valid offsets is easily known in advance to be [0, len(table)). No guarantees are made regarding the order in which results are returned. Repeated offsets produce repeated rows, which makes this method suitable for sampling with replacement. Parameters ---------- offsets: list[int] The offsets to take. with_row_id: bool, default False If True, include the "_rowid" column in the results. Returns ------- LanceTakeQueryBuilder A query builder that can be executed to get the rows at the given offsets. Nre)rHrrs rh take_offsetszTable.take_offsets^ r_rjct}tt|}t|fd}dg|z}t|D] }||||< |jj }|j d|jj|jjdj|jjdg}|S)a Take a list of offsets from the table and return as a record batch. This method uses the `take_offsets` method to take the rows. However, it aligns the offsets to the passed in offsets. This means the return type is a record batch (and so users should take care not to pass in too many offsets) Note: this method is primarily intended to fulfill the Dataset contract for pytorch. Parameters ---------- offsets: list[int] The offsets to take. Returns ------- pa.RecordBatch A record batch containing the rows at the given offsets. c|Srre)idxrs rhz$Table.__getitems__.. s gclrj)rr _rowoffset)rqr rangerrrr3rselectrAsort_bytakecombine_chunks drop_columns) rHr num_offsetsindices permutationpermutation_invirtbls ` rh __getitems__zTable.__getitems__~ s:'l u[)*W*BC # +{# 0A./OKN + 0++##|$   g & VG_ XZ W\ " T/ " ^  \<. )  rjcy)a Take a list of row ids from the table. Row ids are not stable and are relative to the current version of the table. They can change due to compaction and updates. No guarantees are made regarding the order in which results are returned. If you desire an output order that matches the order of the given ids, you will need to add the row id column to the output and align it yourself. Unlike offsets, row ids are not 0-indexed and no assumptions should be made about the possible range of row ids. In order to use this method you must first obtain the row ids by scanning or searching the table. Even so, row ids are more stable than offsets and can be useful in some situations. There is an ongoing effort to make row ids stable which is tracked at https://github.com/lancedb/lancedb/issues/1120 Parameters ---------- row_ids: list[int] The row ids to take. with_row_id: bool, default False If True, include the "_rowid" column in the results. Returns ------- LanceTakeQueryBuilder A query builder that can be executed to get the rows. Nre)rHrow_idsrs rh take_row_idszTable.take_row_ids r_rjcy)zNames of blob v1 and v2 columns that can be fetched by row id. Struct-nested blobs use dotted paths. Blobs inside lists are omitted because a row id does not identify an individual list element. NrerGs rh blob_columnszTable.blob_columns r_rjcy)aMaterialize full blob bytes for ``column`` at the given rows. Local tables support both legacy blob v1 and blob v2 columns. The result has the same length and order as ``row_ids``. Null blobs produce null slots; valid empty blobs produce ``b""``. ``row_ids`` may be a list, an integer Arrow array or chunked array, or a query table with row ids. ``_rowid`` values stay valid after compaction when the table has stable row ids. One missing or deleted row ID rejects the entire batch; no partial results are returned. Blobs inside lists cannot be fetched by row id yet. Remote servers limit each request to 1024 row IDs and 64 MiB of blob bytes. The client splits requests automatically and reads an individual larger blob through the Range route. This method still materializes all bytes in memory; for large values use :meth:`fetch_blob_files`. NrerHr$rs rh fetch_blobszTable.fetch_blobs r_rjcy)aeMaterialize row-specific byte ranges from a blob column. Local tables support both legacy blob v1 and blob v2 columns. Each request is a three-item ``(row_id, offset, length)`` sequence. Requests may be repeated or reordered, including multiple ranges for the same blob. The result has the same length and order as ``requests``; null blobs produce null slots and empty ranges on non-null blobs produce ``b""``. ``_rowid`` values stay valid after compaction when the table has stable row ids. One missing or deleted row ID rejects the entire batch; no partial results are returned. Row IDs can be obtained from a query with ``with_row_id(True)``. Remote tables read the requested ranges through version-pinned blob handles using HTTP byte-range requests. NrerHr$requestss rhfetch_blob_rangeszTable.fetch_blob_ranges r_rjcy)aOpen lazy, seekable :class:`~lancedb._blob.BlobFile` handles. Local tables support both legacy blob v1 and blob v2 columns. Prefer this over :meth:`fetch_blobs` for large payloads. ``row_ids`` is a ``list[int]``, an integer Arrow array or chunked array, or a query ``pyarrow.Table`` carrying row identity via ``_rowid`` or a ``_lance_row_id`` field on the blob descriptor. Null blob values return ``None``. Remote tables require LanceDB Cloud server 0.5.0 or newer. ``_rowid`` values stay valid after compaction when the table has stable row ids. One missing or deleted row ID rejects the entire batch; no partial handles are returned. Nrer s rhfetch_blob_fileszTable.fetch_blob_files r_rjcy)aOpen a seekable file from a SQL ``blob_descriptor`` result. Accepts the descriptor dictionary or an Arrow struct scalar. Its source pins the table version, branch, column and row identity, so opening does not change this table's checkout. Null descriptors return ``None``. Payload bytes load only when the returned handle is read. An external blob with an unspecified size may require a metadata request on opening. ```python rows = db.execute_query( "SELECT blob_descriptor(video) AS video FROM videos" ).read_all() with videos.open_blob_file(rows["video"][0]) as video: header = video.read(4096) ``` NrerH descriptors rhopen_blob_filezTable.open_blob_file% r_rjc#@K|D]}|j|yw)aIterate over lazy files from descriptor dictionaries or Arrow scalars. Accepts an Arrow array/chunked array or any iterable. Opens each handle as iteration advances, preserving input order and nulls as ``None``. Close each non-null handle after use, normally with a ``with`` block. ```python for video in videos.open_blob_files(rows["video"]): if video is not None: with video: header = video.read(4096) ``` NrrH descriptorsrs rhopen_blob_fileszTable.open_blob_files: s(& 2J%%j1 1 2srrcyrre)rHr4rrs rh_execute_queryzTable._execute_queryK s #rjcyrrerHr4verboses rh _explain_planzTable._explain_planT sSVrj aggregatedistributed_metricscyrrerHr4r$s rh _analyze_planzTable._analyze_planW s rjcyrrerHr4s rh_output_schemazTable._output_schema_ s9>> import lancedb >>> data = [ ... {"x": 1, "vector": [1.0, 2]}, ... {"x": 2, "vector": [3.0, 4]}, ... {"x": 3, "vector": [5.0, 6]} ... ] >>> db = lancedb.connect("./.lancedb") >>> table = db.create_table("my_table", data) >>> table.to_pandas() x vector 0 1 [1.0, 2.0] 1 2 [3.0, 4.0] 2 3 [5.0, 6.0] >>> table.delete("x = 2") DeleteResult(num_deleted_rows=1, version=2) >>> table.to_pandas() x vector 0 1 [1.0, 2.0] 1 3 [5.0, 6.0] If you have a list of values to delete, you can combine them into a stringified list and use the `IN` operator: >>> to_remove = [1, 5] >>> to_remove = ", ".join([str(v) for v in to_remove]) >>> to_remove '1, 5' >>> table.delete(f"x IN ({to_remove})") DeleteResult(num_deleted_rows=1, version=3) >>> table.to_pandas() x vector 0 3 [5.0, 6.0] z+delete is not supported for this table typerE)rHwheres rhdeletez Table.deletek sp""OPPrj values_sqlctd)ad This can be used to update zero to all rows depending on how many rows match the where clause. If no where clause is provided, then all rows will be updated. Either `values` or `values_sql` must be provided. You cannot provide both. Parameters ---------- where: str or [Expr][lancedb.expr.Expr], optional The filter condition. Can be a SQL string or a type-safe [Expr][lancedb.expr.Expr] built with [col][lancedb.expr.col] and [lit][lancedb.expr.lit]. The filter must not be empty, or it will error. values: dict, optional The values to update. The keys are the column names and the values are the values to set. values_sql: dict, optional The values to update, expressed as SQL expression strings. These can reference existing columns. For example, {"x": "x + 1"} will increment the x column by 1. Returns ------- UpdateResult - rows_updated: The number of rows that were updated - version: The new version number of the table after the update Examples -------- >>> import lancedb >>> from lancedb.expr import col >>> import pandas as pd >>> data = pd.DataFrame({"x": [1, 2, 3], "vector": [[1.0, 2], [3, 4], [5, 6]]}) >>> db = lancedb.connect("./.lancedb") >>> table = db.create_table("my_table", data) >>> table.to_pandas() x vector 0 1 [1.0, 2.0] 1 2 [3.0, 4.0] 2 3 [5.0, 6.0] >>> table.update(where=col("x") == 2, values={"vector": [10.0, 10]}) UpdateResult(rows_updated=1, version=2) >>> table.to_pandas() x vector 0 1 [1.0, 2.0] 1 3 [5.0, 6.0] 2 2 [10.0, 10.0] >>> table.update(values_sql={"x": "x + 1"}) UpdateResult(rows_updated=3, version=3) >>> table.to_pandas() x vector 0 2 [1.0, 2.0] 1 4 [5.0, 6.0] 2 3 [10.0, 10.0] z+update is not supported for this table typerErHr1rr4s rhupdatez Table.update sB""OPPrjdelete_unverifiedcy)aT Clean up old versions of the table, freeing disk space. Parameters ---------- older_than: timedelta, default None The minimum age of the version to delete. If None, then this defaults to two weeks. delete_unverified: bool, default False Because they may be part of an in-progress transaction, files newer than 7 days old are not deleted by default. If you are sure that there are no in-progress transactions, then you can set this to True to delete all files older than `older_than`. Returns ------- CleanupStats The stats of the cleanup operation, including how many bytes were freed. See Also -------- [Table.optimize][lancedb.table.Table.optimize]: A more comprehensive optimization operation that includes cleanup as well as other operations. Notes ----- This function is not available in LanceDb Cloud (since LanceDB Cloud manages cleanup for you automatically) NrerH older_thanr9s rhcleanup_old_versionszTable.cleanup_old_versions r_rjcy)a Run the compaction process on the table. This can be run after making several small appends to optimize the table for faster reads. Arguments are passed onto Lance's `lance.dataset.DatasetOptimizer.compact_files`. For most cases, the default should be fine. See Also -------- [Table.optimize][lancedb.table.Table.optimize]: A more comprehensive optimization operation that includes cleanup as well as other operations. Notes ----- This function is not available in LanceDB Cloud (since LanceDB Cloud manages compaction for you automatically) Nre)rHrrhs rh compact_fileszTable.compact_files r_rjrr9rcompaction_optionscya Optimize the on-disk data and indices for better performance. Modeled after ``VACUUM`` in PostgreSQL. Optimization covers three operations: * Compaction: Merges small files into larger ones * Prune: Removes old versions of the dataset * Index: Optimizes the indices, adding new data to existing indices Parameters ---------- cleanup_older_than: timedelta, optional default 7 days All files belonging to versions older than this will be removed. Set to 0 days to remove all versions except the latest. The latest version is never removed. .. warning:: Setting this to 0 deletes the data files of every older version, so any other reader or writer still using an older version of the table will fail. Only set it to 0 if no other process is working on this dataset. delete_unverified: bool, default False Files leftover from a failed transaction may appear to be part of an in-progress operation (e.g. appending new data) and these files will not be deleted unless they are at least 7 days old. If delete_unverified is True then these files will be deleted regardless of their age. .. warning:: This should only be set to True if you can guarantee that no other process is currently working on this dataset. Otherwise the dataset could be put into a corrupted state. retrain: bool, default False This parameter is no longer used and is deprecated. compaction_options: CompactionOptions, optional Options that control file compaction. For large rows, derive a lower ``target_rows_per_fragment`` from the expected row size to bound later passes. This does not cap the first pass over an existing oversized fragment; see [CompactionOptions][lancedb.table.CompactionOptions]. Notes ----- The frequency an application should call optimize is based on the frequency of data modifications. If data is frequently added, deleted, or updated then optimize should be run frequently. A good rule of thumb is to run optimize if you have added or modified 100,000 or more records or run more than 20 data modification operations. NrerHrr9rrAs rhoptimizezTable.optimize$ r_rjcy)z} List all indices that have been created with [Table.create_index][lancedb.table.Table.create_index] NrerGs rh list_indiceszTable.list_indicesb r_rjr$ index_namecy) Tokenize a query using the tokenizer configured on an FTS index. Specify exactly one of ``column`` or ``index_name``. Model-backed tokenizers such as ``jieba/*`` and ``lindera/*`` are rebuilt in the client process from index metadata. For remote tables, this means the same tokenizer model files must also exist locally (under ``LANCE_LANGUAGE_MODEL_HOME``) and must match the files the server uses. A mismatch cannot be detected: the returned tokens may differ from what the server's index produces. If the model files are missing, an error naming the tokenizer is raised. NrerHr4r$rIs rhtokenizezTable.tokenizei r_rjcy)G Retrieve statistics about an index Parameters ---------- index_name: str The name of the index to retrieve statistics for Returns ------- IndexStatistics or None The statistics about the index. Returns None if the index does not exist. NrerHrIs rh index_statszTable.index_stats r_rjcomputedauto_refresh_enabledauto_refresh_interval_secondscy)a Add new columns with defined values. Parameters ---------- transforms: Dict[str, str | FunctionApplication], FunctionApplication, pa.Field, List[pa.Field], pa.Schema A map of column name to a SQL expression to use to calculate the value of the new column. These expressions will be evaluated for each row in the table, and can reference existing columns. Alternatively, a pyarrow Field or Schema can be provided to add new columns with the specified data types. The new columns will be initialized with null values. A mapping with one ``FunctionApplication`` value keeps its scalar or named-struct result in the named table column. A bare named-struct application expands its ordered result fields as one atomic binding; aliases come from ``rename(columns=...)``. Function columns are supported only on LanceDB Cloud and Enterprise. computed: Dict[str, str], optional Local declaration and refresh require a source build with the ``experimental-functions`` Cargo feature, which published packages do not enable. Remote Cloud/Enterprise operations remain available. A mapping from output column names to SQL expressions derives each output field from its expression. A direct projection of a Blob v2 field inherits Blob v2 semantics; other expressions derive their ordinary Arrow type. Mapping order is declaration and dependency order. Unlike ``transforms``, the expression is stored rather than evaluated now: the column is committed with no values, and rows get them from [`refresh_column`][lancedb.table.Table.refresh_column]. Declaring one therefore costs the same on a large table as on an empty one. A refresh also recomputes the rows whose inputs changed since they were computed, so a mutated input is reflected by the next refresh. While a declaration reads a column, that column cannot be renamed, retyped or dropped. On LanceDB Cloud and Enterprise the expression is planned by the server, and the refresh runs as a server job -- see [`refresh_column_async`][lancedb.table.Table.refresh_column_async]. Cannot be combined with ``transforms``. auto_refresh_enabled: bool, optional Enable automatic refresh for each declared computed output. Defaults to True. Local tables store this option; scheduling requires a server. auto_refresh_interval_seconds: int, optional Positive minimum interval after a successful refresh, in seconds. When omitted, the server determines when to refresh. Applies to every computed output in this call. Scheduling options require computed or Function columns. Returns ------- AddColumnsResult version: the new version number of the table after adding columns. Examples -------- >>> import lancedb >>> db = lancedb.connect("./.lancedb") >>> table = db.create_table("computed_demo", [{"x": 1}, {"x": 2}]) >>> table.add_columns(computed={"doubled": "x * 2"}) # doctest: +SKIP AddColumnsResult(version=2) >>> table.refresh_column("doubled") # doctest: +SKIP RefreshColumnResult(rows_filled=2, version=4) >>> table.to_arrow().sort_by("x").to_pandas() # doctest: +SKIP x doubled 0 1 2 1 2 4 NrerH transformsrSrTrUs rh add_columnszTable.add_columns r_rjcy)a Fill the rows of a computed column that hold no value yet. Local refresh requires the ``experimental-functions`` Cargo feature in a source build. Published packages leave it disabled. Declared with ``add_columns(computed=...)``, a column starts empty and gets its values here. Rows appended since the last refresh are filled by the next one, and rows whose inputs changed since they were computed are recomputed; everything else is left as it is. Local tables only: a remote refresh runs as a server job, through [`refresh_column_async`][lancedb.table.Table.refresh_column_async]. Parameters ---------- column: str The name of the computed column to fill. Returns ------- RefreshColumnResult rows_filled: the number of rows given a value. version: the new version number of the table. NrerHr$s rhrefresh_columnzTable.refresh_column r_rjcy)at Like :meth:`refresh_column`, but returns a handle to the refresh job instead of blocking until it completes. Local refresh requires a source build with ``experimental-functions``. Remote Cloud/Enterprise jobs do not require this feature. The job may already be complete when returned; callers must not assume the column is filled until :meth:`Job.wait` returns. Invalid input -- an unknown column, or one that is not computed -- raises here rather than failing the job. On local tables the job runs in-process; on LanceDB Cloud and Enterprise it is the server's backfill job. Returns ------- Job[RefreshColumnResult] A job whose successful ``wait`` returns row counts plus the source and published table versions. Examples -------- >>> import lancedb >>> db = lancedb.connect("./.lancedb") >>> table = db.create_table("computed_job_demo", [{"x": 1}, {"x": 2}]) >>> table.add_columns(computed={"doubled": "x * 2"}) # doctest: +SKIP AddColumnsResult(version=2) >>> job = table.refresh_column_async("doubled") # doctest: +SKIP >>> result = job.wait() # doctest: +SKIP >>> result.rows_assigned # doctest: +SKIP 2 >>> job.status() # doctest: +SKIP 'finished' Nrer[s rhrefresh_column_asynczTable.refresh_column_async r_rjcy)a The per-row errors Function refreshes recorded on this table. A refresh running under a skip policy records each row it skipped with the input that failed and the error. This lists those records, newest job first, plus a summary for any fragment whose per-row detail was capped. LanceDB Cloud and Enterprise only; reading errors needs read access to the table, since a message carries the value that failed. Parameters ---------- job_id: str, optional Only errors recorded by this job. column: str, optional Only errors on this column. limit: int, optional At most this many records (server default 10000, cap 100000). Returns ------- FunctionErrors ``records``, ``fragments`` and ``truncated``, the last saying whether the listing stopped at its limit. Examples -------- >>> errors = table.function_errors(column="embedding") # doctest: +SKIP >>> for record in errors.records: # doctest: +SKIP ... print(record.job_id, record.row_offset, record.error_message) >>> if errors.truncated: # doctest: +SKIP ... print("listing stopped at the limit") NrerHjob_idr$limits rhfunction_errorszTable.function_errors) r_rjcy)a Alter column names and nullability. Parameters ---------- alterations : Iterable[Dict[str, Any]] A sequence of dictionaries, each with the following keys: - "path": str The column path to alter. For a top-level column, this is the name. For a nested column, this is the dot-separated path, e.g. "a.b.c". - "rename": str, optional The new name of the column. If not specified, the column name is not changed. - "data_type": pyarrow.DataType, optional The new data type of the column. Existing values will be casted to this type. If not specified, the column data type is not changed. - "nullable": bool, optional Whether the column should be nullable. If not specified, the column nullability is not changed. Only non-nullable columns can be changed to nullable. Currently, you cannot change a nullable column to non-nullable. Each alteration must specify at least one of "rename", "data_type", or "nullable". The legacy key "name" is also accepted for renaming. Unknown keys raise ValueError before any alterations are applied. Returns ------- AlterColumnsResult version: the new version number of the table after the alteration. NrerH alterationss rh alter_columnszTable.alter_columnsR r_rjcy)u Update per-field (column) metadata. Parameters ---------- updates : dict One or more dicts, each with: - "path": str — dot-path to the field (e.g. "embedding" or "a.b.c"). - "metadata": dict[str, str | None] — keys to set; a value of ``None`` deletes that key. - "replace": bool, optional — replace the field's whole metadata map instead of merging (default False). The following keys are treated specially, by convention, and should be used when appropriate: - "lancedb:description": for a human-readable description of a field. - ``"lancedb:tag:"`` for a user-defined key-value tag, where the suffix names the tag category; e.g. "lancedb:tag:model": "clip". - "lancedb:status" for status options ("production", "candidate", "deprecated", "archived") to designate the current life cycle state of this column. Returns ------- UpdateFieldMetadataResult version: the new table version after the update. NrerHupdatess rhupdate_field_metadatazTable.update_field_metadatat r_rj0.33.1$Use update_field_metadata() instead. deprecated_incurrent_versiondetailsc.|j||ddy)aReplace all metadata of a field in the schema. .. deprecated:: 0.33.1 Use [update_field_metadata][lancedb.table.Table.update_field_metadata] with ``replace=True`` instead. Parameters ---------- field_name: str The name of the field whose metadata is replaced. new_metadata: dict The new metadata, replacing all existing keys. T)pathr;rN)rkrHr new_metadatas rhreplace_field_metadatazTable.replace_field_metadata s& ""\d K rjcy)a- Drop columns from the table. Parameters ---------- columns : Iterable[str] The names of the columns to drop. Returns ------- DropColumnsResult version: the new version number of the table dropping the columns. NrerHrs rhrzTable.drop_columns r_rjcy)h Checks out a specific version of the Table Any read operation on the table will now access the data at the checked out version. As a consequence, calling this method will disable any read consistency interval that was previously set. This is a read-only operation that turns the table into a sort of "view" or "detached head". Other table instances will not be affected. To make the change permanent you can use the `[Self::restore]` method. Any operation that modifies the table will fail while the table is in a checked out state. To return the table to a normal state use `[Self::checkout_latest]`. Parameters ---------- version: int | str, The version to check out. A version number (`int`) or a tag (`str`) can be provided. NrerHrOs rhcheckoutzTable.checkout r_rjcyz Ensures the table is pointing at the latest version This can be used to manually update a table when the read_consistency_interval is None It can also be used to undo a `[Self::checkout]` operation NrerGs rhcheckout_latestzTable.checkout_latest r_rjcy)aRestore a version of the table. This is an in-place operation. This creates a new version where the data is equivalent to the specified previous version. Data is not copied (as of python-v0.2.1). Parameters ---------- version : int or str, default None The version number or version tag to restore. If unspecified then restores the currently checked out version. If the currently checked out version is the latest version then this is a no-op. Nrer{s rhrestorez Table.restore r_rjcy)List all versions of the tableNrerGs rh list_versionszTable.list_versions r_rjcVt|jj|jSr)r5_connrrrGs rh _dataset_urizTable._dataset_uri s$**..$))44rjcddlm}t||st|jdk7ryt |jdd}t |\}}|j|jtjjk7}|||fS)Nr) RemoteTablefile)rNF_indicesfts) remote.tablerrnrZrr\rY get_file_inforpa_fsFileTypeNotFound)rHrrsfs index_existss rh_get_fts_index_pathzTable._get_fts_index_path sy- dK (N4;L;L,MQW,W$)):u=t$D''-22enn6M6MM b,''rjcN|j\}}}|rtd|dy)Nz%Legacy Tantivy FTS index detected at zo. Tantivy-based FTS has been removed. Delete the legacy index and recreate it with table.create_fts_index(...).)rr)rHrsrexistss rh_ensure_no_legacy_fts_indexz!Table._ensure_no_legacy_fts_index s=224a 7&//  rjcy Check if the table is using the new v2 manifest paths. Returns ------- bool True if the table is using the new v2 manifest paths, False otherwise. NrerGs rhuses_v2_manifest_pathszTable.uses_v2_manifest_paths r_rjcy)ap Migrate the manifest paths to the new format. This will update the manifest to use the new v2 format for paths. This function is idempotent, and can be run multiple times without changing the state of the object store. !!! danger This should not be run while other concurrent operations are happening. And it should also run until completion before resuming other operations. You can use [Table.uses_v2_manifest_paths][lancedb.table.Table.uses_v2_manifest_paths] to check if the table is already using the new path style. NrerGs rhmigrate_v2_manifest_pathszTable.migrate_v2_manifest_paths r_rjcy)zWhether a hybrid query here has already been refused ``_rowid``. Learned from a refusal, never probed, so asking is free. ``False`` for table types that never refuse. FrerGs rh_hybrid_pk_fusion_learnedzTable._hybrid_pk_fusion_learned+ s rjcy)z?Remember a ``_rowid`` refusal, so later hybrid queries skip it.NrerGs rh_note_hybrid_pk_fusionzTable._note_hybrid_pk_fusion3 r_rjreturnrorz List[str]rrr pa.SchemarTagsrz 'Branches'r Optional[str]rz"Dict[str, EmbeddingFunctionConfig]rrcrrrrd)rgr%rz'pandas.DataFrame'rgr%rpa.Tablerzlance.LanceDatasetrz'pl.LazyFrame')rvrorwOptional[IndexConfigType]rxrryrrzrorxrr{rrr)rrorrorxrryrrrrrrrrrrrrrrxrrIndexConfigType)rrorrr$rorwrrrrOptional[timedelta]rrrrrNone .............$rz)Literal['l2', 'cosine', 'dot', 'hamming']rxrryrrzrorrrxrr{rrrrzOLiteral['IVF_FLAT', 'IVF_SQ', 'IVF_PQ', 'IVF_RQ', 'IVF_HNSW_SQ', 'IVF_HNSW_PQ']rrrrrrrrrrrrrrrrrr$rrorxrryrrzrorrrxrr{rrrrz`Literal['IVF_FLAT', 'IVF_SQ', 'IVF_PQ', 'IVF_RQ', 'IVF_HNSW_SQ', 'IVF_HNSW_PQ', 'IVF_HNSW_FLAT']rrrrrrrrrwrrrrrrrrrr$rorwrrOptional[bool]rrrrrrrr*rrorrr Iterable[str]rrrrrTableStatistics r$rorrrrrrrr,rzUnion[str, List[str]]rOptional[Union[str, List[str]]]rrrrrrrrrrrrrrorrrrrrrrrOptional[List[str]]rrrrrrrrrrrrOrrrrr3errorNNFrr2rrr[rr\rrA$Optional[Union[bool, Callable, Any]]rrrrrrrUnion[str, Iterable[str]]rrFNNautoNNNr4BOptional[Union[VEC, str, 'PIL.Image.Image', Tuple, FullTextQuery]]rzrrrrrrrrrrrTr list[int]rrrrV)rrrpa.RecordBatchrrrrrrVrz list[str]r$ror5Union[list[int], pa.Array, pa.ChunkedArray, pa.Table]rpa.LargeBinaryArrayr$rorzSequence[Sequence[int]]rrr$rorrrz'list[Optional[BlobFile]]'rz"Union[dict, pa.StructScalar, None]rzOptional[BlobFile])rrrzIterator[Optional[BlobFile]]r4rWrrrrrpa.RecordBatchReaderFr4rWr rrror4rWr$rIrror4rWrr r-rFr.r2r[rr\rrrr1zUnion[str, Expr]rrNNr1Optional[Union[str, Expr]]rOptional[dict]r4Optional[Dict[str, str]]rrr<rr9rrz'CleanupStats'rrr9rrrrAOptional[CompactionOptions]rzIterable[IndexConfig]r4ror$rrIrrzIterable[FtsToken]rIrorzOptional[IndexStatistics])rXiDict[str, str | FunctionApplication] | FunctionApplication | pa.Field | List[pa.Field] | pa.Schema | NonerSDict[str, str] | NonerT bool | NonerU int | Noner$rorz'RefreshColumnResult'r$rorzJob[RefreshColumnJobResult]NNNrarr$rrbrrz'FunctionErrors')rfIterable[Dict[str, str]]rjzdict[str, Any]rrrroruzDict[str, str]rrrrrOzUnion[int, str]rOzOptional[Union[int, str]]rzList[Dict[str, Any]])rz"Tuple[str, pa_fs.FileSystem, bool]rrrr)UrrrrpropertyrrrJrMrOrrSrVrYr\r^r[rfrArnrpr|r staticmethodrrrr4rrrrrrrOROWrrrrrrrrr rrrrrr!r'r*r/r2r7r=r?rErGrMrQrYr\r^rcrgrk deprecation deprecatedrrvrr|rrrr rrrrrrrrerjrhrr}se #JOO66RRQQOO6SSY%   U U3"06SS" ST0*&  '   #( BYAYAYA& YA ' YA  YAYAYA YAYA -YA#YA YAv< < ~,/!      *      =@(+),"%%(*- !"!$-0!/2+9&'     #(   !$*%&'()* -+, -4(,)-"4%)*.  "-IW0-1,0"/39IWIW&IW' IW  IW  IW#IW(IWIW IW&'IW()IW* +IW,-IW0*1IW2*3IW45IW67IW8 -9IW`#',0"      *      (U*@IQT?UY(Y3<Y Y"PP  &-,0"@ @  @ $ @ * @ @ @ LAE*<!(,#,4!*,#,015" ! !!4G4K4K,0"3~ *~ > ~  ~ ( ~ ~ &~ ~ *~ ~ (~ ~ ~ *!~ "/#~ $%~ &'~ ()~ *+~ ,-~ .2/~ 0*1~ 23~ @!+29=+/27GNGNGN) GN  GN 7 GN)GN,0GN GNGNR>1@ ,0 &-17;&*VQ VQ * VQ  VQ+VQ5VQ$VQ VQVQp9>   26   >0d9>"  " 26" " " H    G     2  *     0  G  $   ( <   (2" %)'+ ##" # % #  ##VV >I ;  <<&)     7Q7Qr-1!%@Q 04 @Q)@Q@Q - @Q  @Q@QD+/# #( # '#  #  # # J  *37"':> ; 0;  ;  ; 8 ; ; z   !%$(     "     *    W +/,048W W (W *W (2W W r  6! ! F!% $# & & &  &  & & P  B & "  B[#6    $      .      --55(    (Nrjrc eZdZdZdddddddddd ddZeddZeddZeddZ edddddddd  dd Z e dd Z dd Z edd ZddZeddZdd ddZdd ddZddZ ddZ ddZ ddZ ddZeddZeddZddZ d ddZddZdZdddZdddZdd Z dd!Z!ddd"Z"ddd$Z#d#d%dd&Z$ddd'Z%e&d(d(d(d(d) dd*Z'e& dd(d(d(d(d+ dd,Z'd-dde(d.ddd/d0d1d2d3d4f dddd.dd5 dd6Z'dddd.d) dd7Z)dd8Z*dd9Z+ddd:Z,e-d4;f dd<Z.dd=Z/edd>Z0dd?Z1dd@Z2e3jhdAe5dBCd.dDdddE ddFZ6e3jhdAe5dGCdddHddddIdJdKd.dddd.dLdLddMe7jpdddN ddOZ9 d ddPZ: d ddQZ;e ddRZddZZ? ddd[ dd\Z@ddd] dd^ZAddd_ZBd`da ddbZCddcZD dddZEeddeZFe3jhdfe5dgC dddh ddiZGe3jhdfe5dgCddjZHdddddk ddlZIddmZJdddn ddoZKddpZL dddddq ddrZMddsZNddtZO d dduZP ddvZQ ddwZRddxZSddyZTddzZUdd{ZVdd|ZWdd}ZXdd~ZYddZZddddZ[ddZ\ddZ]dZ^e3jhde5dCddZ_y) LanceTablea A table in a LanceDB database. This can be opened in two modes: standard and time-travel. Standard mode is the default. In this mode, the table is mutable and tracks the latest version of the table. The level of read consistency is controlled by the `read_consistency_interval` parameter on the connection. Time-travel mode is activated by specifying a version number. In this mode, the table is immutable and fixed to a specific version. This is useful for querying historical versions of the table. NF) namespace_pathstorage_optionsr{locationrfmanaged_versioningrgroute_pushdown_to_rust_asyncc 6|g}||_||_||_||_| xs t |_| |_| | |_n:tj|jj||||||||_|j |j_y)N)rr r{r rfr ) r_namespace_path _location_namespace_clientr_pushdown_operations_route_pushdown_to_rust_tabler'run open_table) rH connectionrrr r{r rfr rgr r s rh__init__zLanceTable.__init__F s  !N -!!1$7$@35!(>$   DK((  ++#1$3%5%%5'9, DK,0+D+D (rjc.|jjSr)rrrGs rhrzLanceTable.nameu s{{rjc|jS)z'Return the namespace path of the table.)rrGs rhrJzLanceTable.namespacey s###rjcKddlm}m}t|}|j }||j d{}|j ||}|||j|S7,w)Nr)AsyncConnectionr)r )dbrr AsyncTabledatabaseget_read_consistency_interval from_innerr)clsrrr async_tbl inner_connread_consistency_intervalconns rhr!zLanceTable.from_inner~ sp:sO \\^ *9 + ' ' )%*!, ++J8QR  NN  %*s=A.A,-A.rr r{r rfr rgc |g}||||||||||  } | j| S#t$r#} dt| vrtd|d| d} ~ wwxYw)Nr'z Not found:zTable z does not exist)rOrroFileNotFoundError) r"rrrr r{r rfr rgres rhopenzLanceTable.open s~  !N  )+--1 3    KK   s1v%'&o(FGGG s % AA  Ac|j |jSt|jj|jSr)rr;rrrrGs rh _dataset_pathzLanceTable._dataset_path s1 >> %>> !4::>>49955rjc v ddl}|j}|dn |j}|j}|Rt |j dd}t|dk(s |jr!|j j}||_|F|j|jgz}|jd ||j j||d|}n4|j|jf||j jd|}||j||jf}|S#t$r tdwxYw) z+Return the LanceDataset backing this table.rN^The lance library is required to use this function. Please install with `pip install pylance`.rrrJ)rOr rftable_idrOr re)r/rsrYrOrrrrZrrfrr?r r-checkout_version) rHrhr/branchrOrfconn_urir0dss rhrnzLanceTable.to_lance sS  $$& ,$$,,11  #tzz5"5Hh';6$:N:N#'::#>#>#@ )9&  '++tyyk9H $ : :!1!   B"" $ : : B  $$fdll%;??rjc,t|jS)a0Tag management for the table. Similar to Git, tags are a way to add metadata to a specific version of the table. .. warning:: Tagged versions are exempted from the :py:meth:`cleanup_old_versions()` process. To remove a version that has been tagged, you must first :py:meth:`~Tags.delete` the associated tag. Returns ------- Tags The tag manager for managing tags for the table. Examples -------- >>> import lancedb >>> db = lancedb.connect("./.lancedb") >>> table = db.create_table("my_table", ... [{"vector": [1.1, 0.9], "type": "vector"}]) >>> table.tags.create("v1", table.version) >>> table.add([{"vector": [0.5, 0.2], "type": "vector"}]) AddResult(version=2) >>> tags = table.tags.list() >>> print(tags["v1"]["version"]) 1 >>> table.checkout("v1") >>> table.to_pandas() vector type 0 [1.1, 0.9] vector )rrrGs rhrSzLanceTable.tags$sJDKK  rjct|S)zBranch management for the table. ``create``/``checkout`` return a new table handle scoped to the branch; writes on it do not affect ``main``. )BranchesrGs rhrVzLanceTable.branchesKs~rjc6|jjSrX)rrYrGs rhrYzLanceTable.current_branchT{{))++rjc t|j|j|j|j|j |j |j|S)N)rrfrgr r r )rrrrrrrrrH async_tablerOs rh_wrap_branch_handlezLanceTable._wrap_branch_handleXsO  JJ   //!33 $ 9 9#'#?#?^^  rjc`tj|jj|y)aCheckout a version of the table. This is an in-place operation. This allows viewing previous versions of the table. If you wish to keep writing to the dataset starting from an old version, then use the `restore` function. Calling this method will set the table into time-travel mode. If you wish to return to standard mode, call `checkout_latest`. Parameters ---------- version: int | str, The version to check out. A version number (`int`) or a tag (`str`) can be provided. Examples -------- >>> import lancedb >>> db = lancedb.connect("./.lancedb") >>> table = db.create_table("my_table", ... [{"vector": [1.1, 0.9], "type": "vector"}]) >>> table.version 1 >>> table.to_pandas() vector type 0 [1.1, 0.9] vector >>> table.add([{"vector": [0.5, 0.2], "type": "vector"}]) AddResult(version=2) >>> table.version 2 >>> table.checkout(1) >>> table.to_pandas() vector type 0 [1.1, 0.9] vector N)r'rrr|r{s rhr|zLanceTable.checkouths H %%g./rjc^tj|jjy)zCheckout the latest version of the table. This is an in-place operation. The table will be set back into standard mode, and will track the latest version of the table. N)r'rrrrGs rhrzLanceTable.checkout_latests ,,./rjc|.tj|jj|tj|jj y)aRestore a version of the table. This is an in-place operation. This creates a new version where the data is equivalent to the specified previous version. Data is not copied (as of python-v0.2.1). Parameters ---------- version : int or str, default None The version number or version tag to restore. If unspecified then restores the currently checked out version. If the currently checked out version is the latest version then this is a no-op. Examples -------- >>> import lancedb >>> db = lancedb.connect("./.lancedb") >>> table = db.create_table("my_table", [ ... {"vector": [1.1, 0.9], "type": "vector"}]) >>> table.version 1 >>> table.to_pandas() vector type 0 [1.1, 0.9] vector >>> table.add([{"vector": [0.5, 0.2], "type": "vector"}]) AddResult(version=2) >>> table.version 2 >>> table.tags.create("v2", 2) >>> table.restore(1) >>> table.to_pandas() vector type 0 [1.1, 0.9] vector >>> len(table.list_versions()) 3 >>> table.restore("v2") >>> table.to_pandas() vector type 0 [1.1, 0.9] vector 1 [0.5, 0.2] vector >>> len(table.list_versions()) 4 N)r'rrr|rr{s rhrzLanceTable.restores?X   HHT[[))'2 3 $$&'rjc^tj|jj|Sr)r'rrr[rbs rhr[zLanceTable.count_rowss xx ..v677rjch|jjd|jd|jdS)Nz(name=z, _conn=))r7rrrrGs rh__repr__zLanceTable.__repr__s/..))*& Xdjj^STUUrjc"|jSr)rRrGs rh__str__zLanceTable.__str__s}}rjc^tj|jj|S)z%Return the first n rows of the table.)r'rrheadrHns rhrVzLanceTable.heads xx ((+,,rjrdc &t||dk(st|jsF|j}|dk(rt |t |j}|j di|S|dk(rD|j8t|jdk(r |jj di|S|dk(r7t|jr"|jj dd|i|S|jj dd|i|S)a1Return the table as a pandas DataFrame. Parameters ---------- blob_mode: str, default "lazy" Controls how Lance blob columns are returned. **kwargs Forwarded to Lance pandas conversion. Returns ------- pd.DataFrame descriptionsrdmemoryrrgre) r$rbrrAr#rarfrrZr-r_rrn)rHrgrh arrow_tbls rhrfzLanceTable.to_pandass 9%  &.CDKK.P IN*.,4==?,,6v6 6  $8$E*4;;=**IYI&I I(t}}((G9GGGrjrict||dk7r5t|jr |jj |St |j |jr3|js'|jtjStj|jj S)aReturn the table as a pyarrow Table. Parameters ---------- blob_mode: str, default "lazy" ``"lazy"`` returns blob v2 descriptors that carry the row id ``fetch_blobs`` reads. ``"descriptions"`` returns them without it. ``"bytes"`` returns payload bytes and raises for legacy blob columns and blobs inside lists. Returns ------- pa.Tablerdri)r$rbrrrArirrrrrWrLr'rrrls rhrAzLanceTable.to_arrows 9%  #8#E;;=))I)> > )&&(A(A 00&&uw/88: :xx ,,.//rjcvddlm}||}tj||j t dS)aReturn the table as a polars LazyFrame. Parameters ---------- batch_size: int, optional Passed to polars. This is the maximum row count for scanned pyarrow record batches Note ---- 1. This requires polars to be installed separately 2. Currently we've disabled push-down of the filters from polars because polars pushdown into pyarrow uses pyarrow compute expressions rather than SQl strings (which LanceDB supports) 3. The Polars streaming engine is not supported because it does not currently implement Python PyArrow dataset scans. Use the default engine when collecting this LazyFrame. Returns ------- pl.LazyFrame r)PyarrowDatasetAdapter)rF)predicate_pushdown)lancedb.integrations.pyarrowr_plscan_pyarrow_dataset map_batchesr~)rHrr_r?s rhrpzLanceTable.to_polarss@. G'-&&w:FRR .$S  rj.rcyrrers rhrzLanceTable.create_index4rrjrcyrrers rhrzLanceTable.create_indexBrrjrTrrrrrrrc|j|||||||}|rtjdtd|}|j || |||| | | | || }|9|j }t ||j|| |||||||| | | |jy|}|t|drt|dd}||j }t |dd d d d d d}t|j}|j|d }i}t|dd}|||d<|jd||t|ddt|ddt|dd||t|ddt|ddt|ddd ||jytj|j j||||||S)raFThe create_index() API with metric/num_partitions parameters is deprecated and will be removed in a future version. Please migrate to the new unified API: # Old (deprecated): table.create_index('l2', vector_column_name='my_vector') # New (recommended): table.create_index('my_vector', config=IvfPq(distance_type='l2'))rr) rrrxryrrrrrrrxN) r$rrrxryrrxr{rrrrrxr~rrrrr)r8r:r9r<r?r@rrrrrxryrrrrr) r$rrrxryrrxrrrrrwrrrre)r|rrrrrnr{rrrrrrrr'rr)rHrrxryrzrrxr{rrrrrrrwrrrr is_legacyr$r?accindex_type_mapcfg_typelance_index_typequantization_kwargss rhrzLanceTable.create_index[sQT55          MMV# (FAA%- /!-' /&;'B F&--/+K8$$!)!#1$3# +%5%$3*?% $$&F!gfm&DfmT:?"mmoG/4#-!)!)!)"/"/ &N $F|44H'5'9'9(H'M$*,'&vz1=H+:B+J7(G((%#3&vE'.v7G'N(/8I4(P '$'!&#r2(/8I3(O.5"$;T/.((*xx KK $ $) %   rjc |ttj|jj ||||||S)zCreate an index, returning a handle to the indexing job. The job may already be complete when returned; callers must not assume the index exists until :meth:`Job.wait` returns. rh)r*r'rrrrs rhrzLanceTable.create_index_asyncsF HH ..#!!- /   rjc^tj|jj|S)z Drops an index from the table Parameters ---------- name: str The name of the index to drop )r'rrrrs rhrzLanceTable.drop_index6s"xx ..t455rjc^tj|jj|S) Prewarm an index in the table. This is a hint to the database that the index will be accessed in the future and should be loaded into memory if possible. This can reduce cold-start latency for subsequent queries. This call initiates prewarming and returns once the request is accepted. It is idempotent and safe to call from multiple clients concurrently. It is generally wasteful to call this if the index does not fit into the available cache. Not all index types support prewarming; unsupported indices will silently ignore the request. Parameters ---------- name: str The name of the index to prewarm )r'rr prewarm_indexrs rhrszLanceTable.prewarm_indexAs"(xx 11$788rjc^tj|jj|S)n Prewarm data for the table. This is a hint to the database that the given columns will be accessed in the future and the database should prefetch the data if possible. Currently only supported on remote tables. This call initiates prewarming and returns once the request is accepted. It is idempotent and safe to call from multiple clients concurrently. This operation has a large upfront cost but can speed up future queries that need to fetch the given columns. Large columns such as embeddings or binary data may not be practical to prewarm. This feature is intended for workloads that issue many queries against the same columns. Parameters ---------- columns: list of str, optional The columns to prewarm. If None, all columns are prewarmed. )r'rr prewarm_datarxs rhrvzLanceTable.prewarm_dataWs"*xx 009::rjrc`tj|jj||Sr)r'rrrrs rhrzLanceTable.wait_for_indexns$xx 22;HIIrjc\tj|jjSr)r'rrrrGs rhrzLanceTable.statsssxx ))+,,rjc\tj|jjSr)r'rrrrGs rhrzLanceTable.urivsxx )**rjc\tj|jjS)Get the initial storage options that were passed in when opening this table. For dynamically refreshed options (e.g., credential vending), use :meth:`latest_storage_options`. Warning: This is an internal API and the return value is subject to change. Returns ------- Optional[Dict[str, str]] The storage options, or None if no storage options were configured. )r'rrinitial_storage_optionsrGs rhr|z"LanceTable.initial_storage_optionszs xx ;;=>>rjc\tj|jjS) Get the latest storage options, refreshing from provider if configured. This method is useful for credential vending scenarios where storage options may be refreshed dynamically. If no dynamic provider is configured, this returns the initial static options. Warning: This is an internal API and the return value is subject to change. Returns ------- Optional[Dict[str, str]] The storage options, or None if no storage options were configured. )r'rrlatest_storage_optionsrGs rhrz!LanceTable.latest_storage_optionss xx ::<==rjz0.25.0zDUse create_index() with config=BTree()/Bitmap()/LabelList() instead.rnrrc *|dk(rtjdtdd}|dk(r t}n.|dk(r t }n|dk(r t }nt d|tj|jj||||| S) zCreate a scalar index on a column. .. deprecated:: 0.25.0 Use :meth:`create_index` with a BTree, Bitmap, or LabelList config instead. Example: ``table.create_index("column", config=BTree())`` rz6index_type="scalar" is deprecated; use "BTREE" insteadrrrBITMAP LABEL_LISTr)rrwrr) rrrr7r;r=rr'rrr)rHr$rrrrrws rhrzLanceTable.create_scalar_indexs(  ! MMH"  !J  WF 8 #XF < '[F2:,?@ @xx KK $ $) %   rjz-Use create_index() with config=FTS() instead.rrrrrrrc |j|r td| td|dk7r tdt|ts td||| || | | | |||||d }n|j |}||d<t d ||d |} t j|jj||||| y#ttf$r)}t||j|j |d}~wwxYw) zCreate a full-text search index on a column. .. deprecated:: 0.25.0 Use :meth:`create_index` with an FTS config instead. Example: ``table.create_index("text_column", config=FTS())`` z^Tantivy-based FTS has been removed. Remove use_tantivy and recreate the index with native FTS.NzXordering_field_names was only supported by the removed Tantivy-based FTS implementation.rzTwriter_heap_size was only supported by the removed Tantivy-based FTS implementation.zNative FTS indexes can only be created on a single field at a time. To search over multiple text fields, create a separate FTS index for each field.) rrrrrrrrrrrrr)rr)rrwrrrrre)rrrnrorrBr'rrrrtrrr)rHrrrrrrrrrrrrrrrrrrrrrrtokenizer_configsrwr*s rhrzLanceTable.create_fts_indexs^L ((* M  +4  1 14 +s+5   !"0$!.$4(%6%6!.$4$4* ! !% < <^ L 5F 1 2 !!5      HH ((#!!- ) L)  %%44  G  s 3CC9$C44C9c t|\}} tj|jj ||||||||r|j SS#|r|j wwxYw)aAdd data to the table. If vector columns are missing and the table has embedding functions, then the vector columns are automatically computed and added. Parameters ---------- data: list-of-dict, pd.DataFrame The data to insert into the table. mode: str The mode to use when writing the data. Valid values are "append" and "overwrite". Overwrite adopts explicitly declared blob layouts, while inferred bytes retain the existing blob v2 fields. on_bad_vectors: str, default "error" How to handle vectors with incorrect sizes, NaNs, or null elements. One of "error", "drop", "fill", "null". fill_value: float, default 0. The value to use when filling vectors. Only used if on_bad_vectors="fill". progress: bool, callable, or tqdm-like, optional A callback or tqdm-compatible progress bar. See :meth:`Table.add` for details. write_parallelism: int, optional Number of partitions to write in parallel. Higher values increase throughput but also peak memory use, since each partition buffers data in flight. Defaults to an estimate based on the data size, capped at the number of CPU cores. Lower this if bulk ingestion is using too much memory. allow_external_blob_outside_bases: bool, default False Allow blob URIs outside registered bases. See :meth:`Table.add`. Local tables only. Returns ------- int The number of vectors in the table. rr[r\rArr)rBr'rrrclose) rHrrr[r\rArrownss rhrzLanceTable.add/su\-X6$ !88 #1)%&76W   t s 4AA-ct|tr|j}n t||}|jj |||||j y)aMerge another table into this table. Performs a left join, where the dataset is the left side and other_table is the right side. Rows existing in the dataset but not on the left will be filled with null values, unless Lance doesn't support null values for some types, in which case an error will be raised. The only overlapping column allowed is the join column. If other overlapping columns exist, an error will be raised. Parameters ---------- other_table: LanceTable or Reader-like The data to be merged. Acceptable types are: - Pandas DataFrame, Pyarrow Table, Dataset, Scanner, Iterator[RecordBatch], or RecordBatchReader - LanceTable left_on: str The name of the column in the dataset to join on. right_on: str or None The name of the column in other_table to join on. If None, defaults to left_on. schema: pa.Schema or LanceModel, optional The schema of the other_table. If not provided, the schema is inferred from the data. Examples -------- >>> import lancedb >>> import pyarrow as pa >>> df = pa.table({'x': [1, 2, 3], 'y': ['a', 'b', 'c']}) >>> db = lancedb.connect("./.lancedb") >>> table = db.create_table("dataset", df) >>> table.to_pandas() x y 0 1 a 1 2 b 2 3 c >>> new_df = pa.table({'x': [1, 2, 3], 'z': ['d', 'e', 'f']}) >>> table.merge(new_df, 'x') >>> table.to_pandas() x y z 0 1 a d 1 2 b e 2 3 c f )left_onright_onrN)rnrrnrfr-r)rH other_tablerrrs rhr-zLanceTable.mergens_h k: .%..0K(K  8F   rjcptjj|jjS) Get the embedding functions for the table Returns ------- funcs: Dict[str, EmbeddingFunctionConfig] A mapping of the vector column to the embedding function or empty dict if not configured. )r6r,r-rr;rGs rhr^zLanceTable.embedding_functionss-)557GG KK   rjcyrrers rhrzLanceTable.searchs#&rjcyrrers rhrzLanceTable.searchs #rjcyrrers rhrzLanceTable.searchs#&rjcyrrers rhrzLanceTable.searchs"%rjc t|trd}t|j|||}t j ||||||xsg|S)a0 Create a search query to find the nearest neighbors of the given query vector. We currently support [vector search](https://lancedb.com/docs/search/vector-search/) and [full-text search](https://lancedb.com/docs/search/full-text-search/). Examples -------- >>> import lancedb >>> db = lancedb.connect("./.lancedb") >>> data = [ ... {"original_width": 100, "caption": "bar", "vector": [0.1, 2.3, 4.5]}, ... {"original_width": 2000, "caption": "foo", "vector": [0.5, 3.4, 1.3]}, ... {"original_width": 3000, "caption": "test", "vector": [0.3, 6.2, 2.6]} ... ] >>> table = db.create_table("my_table", data) >>> query = [0.4, 1.4, 2.4] >>> (table.search(query) ... .where("original_width > 1000", prefilter=True) ... .select(["caption", "original_width", "vector"]) ... .limit(2) ... .to_pandas()) caption original_width vector _distance 0 foo 2000 [0.5, 3.4, 1.3] 5.220000 1 test 3000 [0.3, 6.2, 2.6] 23.089996 Parameters ---------- query: list/np.ndarray/str/PIL.Image.Image, default None The targeted vector to search for. - *default None*. Acceptable types are: list, np.ndarray, PIL.Image.Image - If None then the select/[where][lancedb.query.LanceQueryBuilder.where]/limit clauses are applied to filter the table vector_column_name: str, optional The name of the vector column to search. The vector column needs to be a pyarrow fixed size list type *default "vector"* - If not specified then the vector column is inferred from the table schema - If the table has multiple vector columns then the *vector_column_name* needs to be specified. Otherwise, an error is raised. query_type: str, default "auto" "vector", "fts", or "auto" If "auto" then the query type is inferred from the query; If `query` is a list/np.ndarray then the query type is "vector"; If `query` is a PIL.Image.Image then either do vector search or raise an error if no corresponding embedding function is found. If the `query` is a string, then the query type is "vector" if the table has embedding functions, else the query type is "fts" fts_columns: str or list of str, default None The column(s) to search in for full-text search. If None then the search is performed on all indexed columns. For now, only one column can be searched at a time. Returns ------- LanceQueryBuilder A query builder object representing the query. Once executed, the query returns selected columns, the vector, and also the "_distance" column which is the distance between the query vector and the returned vector. rrrr4rz)rzrrr)rnrPr[rrTcreaters rhrzLanceTable.searchs^\ e] +J5;;!1  !''   1 3#)r#  rj)rr renable_v2_manifest_pathsr rfrgr c | g} |j|}||_| |_||_||_|xs t |_||_| %tjdtd| i} | | d<| >> import lancedb >>> data = [ ... {"x": 1, "vector": [1.0, 2]}, ... {"x": 2, "vector": [3.0, 4]}, ... {"x": 3, "vector": [5.0, 6]} ... ] >>> db = lancedb.connect("./.lancedb") >>> table = db.create_table("my_table", data) >>> table.to_pandas() x vector 0 1 [1.0, 2.0] 1 2 [3.0, 4.0] 2 3 [5.0, 6.0] Parameters ---------- db: LanceDB The LanceDB instance to create the table in. name: str The name of the table to create. data: list-of-dict, dict, pd.DataFrame, default None The data to insert into the table. At least one of `data` or `schema` must be provided. schema: pa.Schema or LanceModel, optional The schema of the table. If not provided, the schema is inferred from the data. At least one of `data` or `schema` must be provided. mode: str, default "create" The mode to use when writing the data. Valid values are "create", "overwrite", and "append". exist_ok: bool, default False If the table already exists then raise an error if False, otherwise just open the table, it will not add the provided data but will validate against any schema that's specified. on_bad_vectors: str, default "error" How to handle vectors with incorrect sizes, NaNs, or null elements. One of "error", "drop", "fill", "null". fill_value: float, default 0. The value to use when filling vectors. Only used if on_bad_vectors="fill". embedding_functions: list of EmbeddingFunctionModel, default None The embedding functions to use when creating the table. data_storage_version: optional, str, default "stable" Deprecated. Set `storage_options` when connecting to the database and set `new_table_data_storage_version` in the options. enable_v2_manifest_paths: optional, bool, default False Deprecated. Set `storage_options` when connecting to the database and set `new_table_enable_v2_manifest_paths` in the options. zbsetting data_storage_version directly on create_table is deprecated. Use database_options instead.rrnew_table_data_storage_versionzfsetting enable_v2_manifest_paths directly on create_table is deprecated. Use database_options instead."new_table_enable_v2_manifest_paths) rrexist_okr[r\r^rr r rf)__new__rrrrrrrrrrrortr'r create_tabler)r"rrrrrrr[r\r^rr rrr rfrgr rHs rhrzLanceTable.createLs*V  !N{{3 -!!1$7$@35!'=$  + MM0"   &"$@TO< = # / MM<"   &"$DG(Eeg @ Ahh JJ   ) )!-%$7- /!!1 *    rjct|tr |jn|}tj|j j |Sr)rnrC_innerr'rrr2rHr1 predicates rhr2zLanceTable.deletes4$.ud$;ELL xx **9566rjr3cdtj|jj|||S)a This can be used to update zero to all rows depending on how many rows match the where clause. Parameters ---------- where: str or [Expr][lancedb.expr.Expr], optional The filter condition. Can be a SQL string or a type-safe [Expr][lancedb.expr.Expr] built with [col][lancedb.expr.col] and [lit][lancedb.expr.lit]. The filter must not be empty, or it will error. values: dict, optional The values to update. The keys are the column names and the values are the values to set. values_sql: dict, optional The values to update, expressed as SQL expression strings. These can reference existing columns. For example, {"x": "x + 1"} will increment the x column by 1. Returns ------- UpdateResult - rows_updated: The number of rows that were updated - version: The new version number of the table after the update Examples -------- >>> import lancedb >>> from lancedb.expr import col >>> import pandas as pd >>> data = pd.DataFrame({"x": [1, 2, 3], "vector": [[1.0, 2], [3, 4], [5, 6]]}) >>> db = lancedb.connect("./.lancedb") >>> table = db.create_table("my_table", data) >>> table.to_pandas() x vector 0 1 [1.0, 2.0] 1 2 [3.0, 4.0] 2 3 [5.0, 6.0] >>> table.update(where=col("x") == 2, values={"vector": [10.0, 10]}) UpdateResult(rows_updated=1, version=2) >>> table.to_pandas() x vector 0 1 [1.0, 2.0] 1 3 [5.0, 6.0] 2 2 [10.0, 10.0] r1 updates_sql)r'rrr7r6s rhr7zLanceTable.updates*lxx **6J*WXXrjrct|j|jrh|js\|j L|j @|j 4ddlm}|j|jgz}||j||Stj|jj|||fd}tj j#j$|S)Nr_execute_server_side_queryrc3zK tjj&#t$rYywxYwwr)r'r __anext__StopAsyncIteration) async_itersrh iter_syncz,LanceTable._execute_query..iter_sync"s= ((:#7#7#9::%  s;', 8;8;)rirrrrYr search_effortlancedb.namespacerrrr'rrrr r.r/r)rHr4rrrr0rrs @rhrzLanceTable._execute_querys *&&(A(A 00##%-""*##+ D++tyyk9H-d.D.DhPUV VXX KK & &uW & U  ##001B1BIKPPrjc`tj|jj||Sr)r'rrr!rs rhr!zLanceTable._explain_plan+s"xx 11%ABBrjr"r#cbtj|jj||S)Nr#)r'rrr'r&s rhr'zLanceTable._analyze_plan.s- xx KK % %eAT % U  rjc^tj|jj|Sr)r'rrr*r)s rhr*zLanceTable._output_schema8s xx 2259::rjcdtj|jj||||Sr)r'rrr/r,s rhr/zLanceTable._do_merge;s-xx KK ! !%>: N  rjc.|jjSr)rrrGs rhrzLanceTable._innerFs{{!!!rjz0.21.0zUse `Table.optimize` instead.r8cD|jj||S)a Clean up old versions of the table, freeing disk space. Parameters ---------- older_than: timedelta, default None The minimum age of the version to delete. If None, then this defaults to two weeks. delete_unverified: bool, default False Because they may be part of an in-progress transaction, files newer than 7 days old are not deleted by default. If you are sure that there are no in-progress transactions, then you can set this to True to delete all files older than `older_than`. Returns ------- CleanupStats The stats of the cleanup operation, including how many bytes were freed. r8)rnr=r;s rhr=zLanceTable.cleanup_old_versionsJs(>}}33 *;4  rjcz|jjj|i|}|j|S)aa Run the compaction process on the table. This can be run after making several small appends to optimize the table for faster reads. Arguments are passed onto `lance.dataset.DatasetOptimizer.compact_files`. (see Lance documentation for more details) For most cases, the default should be fine. )rnrEr?r)rHrrhrs rhr?zLanceTable.compact_filesms8 7 ((66GG  rjr@cxtj|jjt ||||yrC)r'rr _do_optimizerrDs rhrEzLanceTable.optimizes4x  KK $ $*+=wG!"  rjc\tj|jjS)Q List all indices that have been created with Self::create_index )r'rrrGrGs rhrGzLanceTable.list_indicess xx 00233rjrHcdtj|jj|||S)rKrH)r'rrrMrLs rhrMzLanceTable.tokenizes.(xx KK v* M  rjc^tj|jj|S)rO)r'rrrQrPs rhrQzLanceTable.index_statss"xx // ;<>wGHHrjc^tj|jj|S)z^Install an RTEWriteSpec. See [`AsyncTable.enable_rte`][lancedb.AsyncTable.enable_rte].)r'rr enable_rterHspecs rhrzLanceTable.enable_rte4s"xx ..t455rjc\tj|jjS)z`Remove the RTEWriteSpec. See [`AsyncTable.disable_rte`][lancedb.AsyncTable.disable_rte].)r'rr disable_rterGs rhrzLanceTable.disable_rte9 xx //122rjc\tj|jjS)zwRead the installed RTEWriteSpec, or ``None``. See [`AsyncTable.get_rte_spec`][lancedb.AsyncTable.get_rte_spec].)r'rr get_rte_specrGs rhrzLanceTable.get_rte_spec>s xx 00233rjc\tj|jjS)z`Synchronous version of [`AsyncTable.checkpoint_rte`][lancedb.AsyncTable.checkpoint_rte].)r'rrcheckpoint_rterGs rhrzLanceTable.checkpoint_rteCs xx 22455rjc\tj|jjS)zVSynchronous version of [`AsyncTable.flush_rte`][lancedb.AsyncTable.flush_rte].)r'rr flush_rterGs rhrzLanceTable.flush_rteHs xx --/00rjc\tj|jjS)zZSynchronous version of [`AsyncTable.compact_rte`][lancedb.AsyncTable.compact_rte].)r'rr compact_rterGs rhrzLanceTable.compact_rteMrrjinclude_sstable_rowsc`tj|jj|S)z^Synchronous version of [`AsyncTable.get_rte_stats`][lancedb.AsyncTable.get_rte_stats].r)r'rr get_rte_statsrHrs rhrzLanceTable.get_rte_statsRs+xx KK % %;O % P  rjc\tj|jjS)zsClose cached RTE shard writers. See [`AsyncTable.close_rte_writers`][lancedb.AsyncTable.close_rte_writers].)r'rrclose_rte_writersrGs rhrzLanceTable.close_rte_writersYs xx 55788rjc\tj|jjS)r)r'rrrrGs rhrz!LanceTable.uses_v2_manifest_paths^s xx ::<==rjc^tj|jjy)az Migrate the manifest paths to the new format. This will update the manifest to use the new v2 format for paths. This function is idempotent, and can be run multiple times without changing the state of the object store. !!! danger This should not be run while other concurrent operations are happening. And it should also run until completion before resuming other operations. You can use [LanceTable.uses_v2_manifest_paths][lancedb.table.LanceTable.uses_v2_manifest_paths] to check if the table is already using the new path style. N)r'rrmigrate_manifest_paths_v2rGs rhrz$LanceTable.migrate_v2_manifest_pathsis$ 6689rjrlrmcbtj|jj||yaR Replace the metadata of a field in the schema .. deprecated:: 0.33.1 Use :func:`update_field_metadata` instead. Parameters ---------- field_name: str The name of the field to replace the metadata for new_metadata: dict The new metadata to set N)r'rrrvrts rhrvz!LanceTable.replace_field_metadata}s!& 33J MNrj)rz'LanceDBConnection'rrorrr rr{rr rrf Optional[Any]r rrg Optional[set]r rr rrr)r LanceDBTable)rrr rr{rr rrfrr rrgrrrrrrrrrrrrrrrr)rJ 'AsyncTable'rOrr 'LanceTable'rrrrrrrgr%rz'pd.DataFrame'rrrrrrrrrrrrrrrrrrrrr)rzUnion[LanceTable, DATA]rrorrrz&Optional[Union[pa.Schema, LanceModel]]r)NNvectorNNN)r43Optional[Union[VEC, str, 'PIL.Image.Image', Tuple]]rzrrLiteral['vector']rrrrrrrrU)NNrNNN)r4rrzrrLiteral['fts']rrrrrrrrR)NNhybridNNN)r4rrzrrLiteral['hybrid']rrrrrrrrSr)r4rrzrrrrrrrrrrrQr)NNrFrrN)"rrrrorzOptional[DATA]rOptional[pa.Schema]rrrrr[rr\rr^z'Optional[List[EmbeddingFunctionConfig]]rrr zOptional[Dict[str, str | bool]]rrrrr rrfrrgrr rrrrrrrrr)rrr)rrrrrr) rXrrSrrTrrUrrrrrrr)rfrrrrrrrrrrz'RTEWriteSpec'rrrrzOptional['RTEWriteSpec']rrrrrr)`rrrrrrrrJ classmethodr!r+r r-rnrrrOrrrr rrrrSrVrYrKr|rrr[rRrTrVrfrArprrr4rrrsrvrrrrr|rrrrrrOrrrr-r^rrr2r7rr!r'r*r/rr=r?rErGrMrQrYr\r^rcrgrkrrrrrrrrrrrrrvrerjrhrr7 s &/348*."&*.-1-1',!-E'-E-E , -E 2 -E(-E -E(-E+-E+-E!%-E-E^  $$     /348*."&*.-1-1# , # 2 #(# #(#+#+##J66%N..5// :? 26 :? 26 4BBGB  BOO*O  OGGGG $ G@<@ @ $!$!L, CG ' 2?  $0L0.(`8V-!HF1706  F,/!      *      =@(+),"%%(*- !"!$-0!/2+9&'     #(   !$*%&'()* -+, -4(,)-"4%)*.  "-} 0-1,0"/39} } &} ' }  }  } #} (} }  } &'} ()} * +} ,-} 0*1} 2*3} 45} 67} 8 -9} H#',0"      *      8 69,;0@IQT?UJ(J3<J J -++ ?> [#V&-,0"& &  & $ & * & &  & P[#?AE*<!(,#,4!*,#,015" ! !!4G4K4K,0"3b*b> b  b ( bb&bb*bb(bbb *!b"/#b$%b&'b()b*+b,-b.2/b0*1b23b  bN!+29=+/27=!=!=!) =!  =! 7 =!)=!,0=! =!F#'9= >,>> > 7 >@    FJ,0(0-17;&*&B&*&& & + & 5 &$& !&&FJ,0%*-17;&*#B#*## # + # 5 #$# ## ,0(0-17;&* &  & * & & &+ &5 &$ & ! & &,0 &-17;&*%%*% % + % 5 %$% %% ,0 &-17;&*_  _ * _  _ +_ 5_ $_  _ B $&*#+2GK{/3;?.237"&*.-1','{ {{ { $ {  {{){{E{,{9{,{#1{  !{"(#{$+%{&!%'{{z7 -1!%6Y 04 6Y)6Y6Y - 6Y  6Yx%)'+ #Q#Q" #Q % #Q  #QJC>I   ;   ;  &    )      ""[#/+/ #(  '       <[#/   $37"':> B 0B  B  B 8 B H4!%$(     "    0=. +/,048  ( * (2   ,< G!% $#          A4A A E&E "E ;I 6 3 4 6 1 3 =B 9 >:([#6 O  Orjrctj|sStjfd}tjj |S)Nc 3KD]}g} D]D}|d} $|"t||d}|j|t||d||d}F|D]}|d t||d|d<|jj dr|t j j|gjj}|st jj|djyw) N expected_dimrexpected_value_type)rzr[r\rrTrhrr() _infer_vector_dimr3_handle_bad_vector_columnrror rr/rMrPr:rlr) rQ pending_dimsr%dimrmr\r[ output_schemarer]vector_columnss rhrSz _handle_bad_vectors..gens@ EL!/  #N3 ,+E-2G,HIC '' 61'4V'<#1)!$(56K(L  ".   084EmF345M.1  ||""="F %%ug.33MBMMO nn00 O++(19 sAD  A7D 4D )_find_vector_columnsr_vector_output_schemar r.r/)rer[r\r]r;rSrrs```` @@rhr_r_sW*&--QN  )&--HM  D    , ,]CE BBrjc|g}|D]}t|jxrHtjj |jj xr|j tk(}tjj|jxrNtjj |jj xr|jjdk\}|s|s|j|j ddd|St|j}t|j|}ttj j#|j%} g}|D]}|j |vrt|jr3tjj |jj s\|j'|j } |j | vxsO|j tk(xs:|j dk(xr)tjj|j}tjj| jxrNtjj | jj xr| jjdk\} |s| sU|j|j tjj|jr|jjnd|jj d|S)N )rrr embedding)rrr r{ is_floatingrrr4rrr3rrrbr;r6r,r-rr) reader_schemar]r;rrnamed_vector_collikely_vector_colreader_column_namesactive_metadataembedding_function_columns reader_fieldtyped_fixed_vector_cols rhrrs " Eejj)5HH(()>)>?5JJ"44  ++EJJ71HH(()>)>?1ZZ))R/   #4%% % (,/3 &m112%m&<&**K-eVxx &;<== n%1 HH ' ' 3 JJ M.$A Axx --.. ::rjc|jj|}||}t|js|S|tj j |jrx|jj|k7r_t j|jt j|jj}|j|||}|tj j|jjsftj j|jjs3tj j|jjrKt j|jt j|}|j|||}tj j!|jjr t#|}n"t jdgt%|z}||} nOtj j |jr|jj} nt'|} | |St)j|} t)j*| t)j,t)j.|| } t)j0|j3xs#t)j0| j3} | rt)j4|| } |dk(rAt)j0| j3rt7d|dt7d|d|dk(r+t)j8| t j:d|}n]|d k(r*|j=t)j>| }||}n.|d k(r| t7d tA|| |}nt7d ||j|||S) a Ensure that the vector column exists and has type fixed_size_list(float) Parameters ---------- data: pa.Table The table to sanitize. vector_column_name: str The name of the vector column. on_bad_vectors: str, default "error" How to handle vectors with incorrect sizes, NaNs, or null elements. One of "error", "drop", "fill", "null". fill_value: float, default 0.0 The value to use when filling vectors. Only used if on_bad_vectors="fill". Nr FrzVector column 'z' has variable length vectors. Set on_bad_vectors='drop' to remove them, set on_bad_vectors='fill' and fill_value= to replace them, or set on_bad_vectors='null' to replace them with null.z' has NaNs or null values. Set on_bad_vectors='drop' to remove them, set on_bad_vectors='fill' and fill_value= to replace them, or set on_bad_vectors='null' to replace them with null.nulldropfillz;`fill_value` must not be None if `on_bad_vectors` is 'fill'z"Invalid value for on_bad_vectors: )!rrrrr r{rrr to_pylistrrrrrrrhas_nan_valuesrqrr or_kleene not_equallist_value_lengthrror_rif_elserrcinvert_fill_bad_vector_values)rrzr[r\rrrvec_arrhas_nanrr has_wrong_dimhas_bad_vectorsis_bads rhrr0sC.  &&'9:H%&G  &   HH ' ' 5 LL " "l 2((7,,.RXXgll>U>U5VWx);WE& 001 88  w||66 7 88 ' ' (?(? @((7,,.RXX>Q5RSx);WE xxGLL334 )((E7S\12  $ $W\\ 2ll$$( ;Kjj!G LL R))'2C8M ffWo++-N 1F1L1L1NO/ W $vvm$**, %&8%9:NN!%&8%9:NN v %jj $G v %;;ryy01D-.G v %! Q.gsJGGA.AQRS S ??8%7 AArjc6t|tjstj|g}|j }t j |d|d}|j}t j|}tjj|jrwtjj|jr#|jtjn|}t j|t j |}tj"|j|j}t j$|||}tj&j)||} | j|jS)NrT)return_fixed_size_list)rnr  ChunkedArray chunked_arrayrr list_slicerrr{rr is_float16rMr.ris_nanrr!rrl) arrrr\slicedr needs_fillvalues_for_nan_check fill_scalar filled_valuesfilleds rhr#r#s c2?? +u%   C]]33t DF ]]FF#J xxFKK()+)<)[J%>>rjc|j}d}t|D]\}}tjj |j xs)tjj |j }t|jsq|st|t|\}}t|j|}|tjj|j jr*tjtj|}notjj!|j jr9|j|}t#|tj$r|j'}|j)} t+j,| dj/} | dk(r)tjtj0|}nt+j2| } | dj/} | dj/} | | dks| .| dkDr)tjtj|}n+tjtj0|}nmtj4|j||j6|j8}|j;||}||fS)N only_valid)rrminmax)rr;)rrr r{rrrrKrr&_modal_list_sizer$rrrr.rrnr+rr9rcountruint8min_maxrrr;r)rerpeekedrr is_list_typerrr flattened valid_countrT min_value max_valuers rhr`r`s]]F Ff%4.5xx'' 3 rxx7M7M JJ8  ( 3 ~!,V!4"6==#34C{xx##EJJ$9$9: hhrzz|S9 $$UZZ%:%:;q)fboo6#224F"NN,  hhy|DJJL !#"$((288:s";K jj3G ' 4 4 6I ' 4 4 6I!-)a-!-)c/&(hhrzz|S&A &(hhrxxz3&?   IZZ9-Fi4.l 6>rjctjtj|}t|dk(ry|dj dSNrr)rrrrqr)r0modess rhrQrQ,sA GGB((- .E 5zQ 8>> F ##rjct|jsytj|}tj|tj |d}t |dk(rytj|djdSr\) rrrrrcgreaterrqrr)r0lengthss rhrr6sm  """3'GiiGQ!78G 7|q 777 A  $ $ &v ..rjcH|jt|jyy)z. Make sure the metadata is valid utf8 N)r;_validate_metadatar(s rhrcrc@s"6??+#rjc|jD]D\}}t|tr |jd)t|t s:t|Fy#t$rt d|dwxYw)zo Make sure the metadata values are valid utf8 (can be nested) Raises ValueError if not valid utf8 utf8z Metadata key zG is not valid utf8. Consider base64 encode for generic binary metadata.N)r<rnrrUnicodeDecodeErrorrrrb)r;rGrHs rhrbrbHs~  "1 a   4 q ! "&  #A3'JJ s AA2cfeZdZdZddddd dodZddddd dpdZdZdZd Zdqd Z d Z drd Z dsd Z dtdZ dudZdqdZdtdZdtdZdtdZdtdZdd dvdZdtdZedwdZdxdZdydZdzd{dZd|d}dZd~dZddZddd Zdd!dd"Zddddd#d$ dd%Z ddddd#d$ dd&Z!dd'Z"dd(Z#dzdd)Z$e%d*+f dd,Z&dd-Z'dwd.Z(dd/Z)dd0Z*d1dddddd2 dd3Z+dd4Z,e- d dd5Z.e- d dd6Z.e- d dd7Z.e- d dd8Z.e- d dd9Z.e- d dd:Z. d dd;Z. dd<Z/ dd=Z0ddd> dd?Z1dd@Z2dAdB ddCZ3ddDZ4 ddEZ5ddFZ6 dzdddG ddHZ7 dzddddI ddJZ8ddKZ9 d ddLZ: ddMZ; ddNZ< ddOZ=ddPZ>ddQZ?dRZ@ddSZAdTZBdzddUZCddV ddWZDddV ddXZEddYZF ddZZG dd[ZH dd\ZI dd]ZJ dd^ZKedd_ZLedd`ZMddaZNdddddb ddcZO dz dddZPddeZQdddf ddgZRddhZSdqdiZTdjZUeVjdkeXdlm ddnZYy)ra8 An AsyncTable is a collection of Records in a LanceDB Database. An AsyncTable can be obtained from the [AsyncConnection.create_table][lancedb.AsyncConnection.create_table] and [AsyncConnection.open_table][lancedb.AsyncConnection.open_table] methods. An AsyncTable object is expected to be long lived and reused for multiple operations. AsyncTable objects will cache a certain amount of index data in memory. This cache will be freed when the Table is garbage collected. To eagerly free the cache you can call the [close][lancedb.AsyncTable.close] method. Once the AsyncTable is closed, it cannot be used for any further operations. An AsyncTable can also be used as a context manager, and will automatically close when the context is exited. Closing a table is optional. If you do not close the table, it will be closed when the AsyncTable object is garbage collected. Examples -------- Create using [AsyncConnection.create_table][lancedb.AsyncConnection.create_table] (more examples in that method's documentation). >>> import lancedb >>> async def create_a_table(): ... db = await lancedb.connect_async("./.lancedb") ... data = [{"vector": [1.1, 1.2], "b": 2}] ... table = await db.create_table("my_table", data=data) ... print(await table.query().limit(5).to_arrow()) >>> import asyncio >>> asyncio.run(create_a_table()) pyarrow.Table vector: fixed_size_list[2] child 0, item: float b: int64 ---- vector: [[[1.1,1.2]]] b: [[2]] Can append new data with [AsyncTable.add()][lancedb.table.AsyncTable.add]. >>> async def add_to_table(): ... db = await lancedb.connect_async("./.lancedb") ... table = await db.open_table("my_table") ... await table.add([{"vector": [0.5, 1.3], "b": 4}]) >>> asyncio.run(add_to_table()) Can query the table with [AsyncTable.vector_search][lancedb.table.AsyncTable.vector_search]. >>> async def search_table_for_vector(): ... db = await lancedb.connect_async("./.lancedb") ... table = await db.open_table("my_table") ... results = ( ... await table.vector_search([0.4, 0.4]).select(["b", "vector"]).to_pandas() ... ) ... print(results) >>> asyncio.run(search_table_for_vector()) b vector _distance 0 4 [0.5, 1.3] 0.82 1 2 [1.1, 1.2] 1.13 Search queries are much faster when an index is created. See [AsyncTable.create_index][lancedb.table.AsyncTable.create_index]. NF)rrfrgr cj||_|xsg|_||_|xs t|_||_y)aCreate a new AsyncTable object. You should not create AsyncTable objects directly. Use [AsyncConnection.create_table][lancedb.AsyncConnection.create_table] and [AsyncConnection.open_table][lancedb.AsyncConnection.open_table] to obtain Table objects.N)rrrrrr)rHr!rrfrgr s rhrzAsyncTable.__init__s9  -3!1$7$@35!(>$rjc^|xsg|_||_|xs t|_||_|Sr)rrrrr)rHrrfrgr s rh_set_namespace_contextz!AsyncTable._set_namespace_contexts5 .3!1$7$@35!'=$ rjc6|jjSr)rrRrGs rhrRzAsyncTable.__repr__s{{##%%rjc|SrrerGs rh __enter__zAsyncTable.__enter__s rjc$|jyr)r)rHrs rh__exit__zAsyncTable.__exit__s  rjc6|jjS)z!Return True if the table is open.)ris_openrGs rhrpzAsyncTable.is_opens{{""$$rjc6|jjS)zClose the table and free any resources associated with it. It is safe to call this method multiple times. Any attempt to use the table after it has been closed will raise an error.)rrrGs rhrzAsyncTable.closes {{  ""rjcKt|tr|g}n t|}|jj |d{y7w)aSet the unenforced primary key for this table to the given ordered list of columns. "Unenforced" means LanceDB does not check uniqueness on writes; the columns are recorded in the schema as the primary key so that features such as `merge_insert` can use them. Calling this again replaces any previously-set primary key. Parameters ---------- columns : str or Iterable[str] Either a single column name (single-column key) or an ordered iterable of column names (composite key). Each column dtype must be one of: int32, int64, utf8, large_utf8, binary, large_binary, fixed_size_binary. N)rnror rrrxs rhrz%AsyncTable.set_unenforced_primary_keys:& gs #iG7mGkk44W===s>AAAcVK|jj|d{y7w)uInstall an RTEWriteSpec on this table. The spec selects Lance's MemWAL LSM-style write path for future `merge_insert` calls. ``RTEWriteSpec`` chooses one of three sharding strategies: - ``RTEWriteSpec.bucket(column, num_buckets)`` — hash-bucket writes by the single-column unenforced primary key. - ``RTEWriteSpec.identity(column)`` — shard by the raw value of a scalar column. - ``RTEWriteSpec.unsharded()`` — route every write to a single shard. All variants require the table to have an unenforced primary key set via [`set_unenforced_primary_key`]; bucket sharding additionally requires it to be the single column being bucketed. By default the RTE maintains every index on the table, re-read as the table changes, so an index created later is maintained too; one of a kind the RTE cannot mirror is skipped. Name the set with ``with_maintained_indexes`` to pin it instead: each name must already exist and be maintainable (a still-building index is rejected, not omitted); ``[]`` maintains none. Parameters ---------- spec : RTEWriteSpec The sharding spec to install. Examples -------- >>> from lancedb import RTEWriteSpec >>> # table.set_unenforced_primary_key("id") >>> # table.enable_rte(RTEWriteSpec.bucket("id", 16)) N)rrrs rhrzAsyncTable.enable_rtes!Fkk$$T*** )')cTK|jjd{y7w)zRemove the RTEWriteSpec from this table. Reverts to the standard `merge_insert` write path. Errors if no spec is currently set. N)rrrGs rhrzAsyncTable.disable_rtes kk%%''' (&(cRK|jjd{S7w)uRead the RTEWriteSpec currently installed on this table. Returns ``None`` when the RTE write path is not enabled at all — no spec has been set, or one was removed with `disable_rte`. That is a different answer from a spec whose ``maintained_indexes`` is ``None``, which is an installed spec selecting indexes automatically. The spec read back is the one that was installed, selection included: ``None`` maintains every supported index the table has now or gains later, ``[]`` maintains none, and a non-empty list maintains exactly those. All three round-trip. N)rrrGs rhrzAsyncTable.get_rte_specs![[--//// '%'c6|jjS)z=See [`Table._hybrid_pk_fusion_learned`][lancedb.table.Table].)rhybrid_pk_fusion_learnedrGs rhrz$AsyncTable._hybrid_pk_fusion_learned0s{{3355rjc8|jjy)z:See [`Table._note_hybrid_pk_fusion`][lancedb.table.Table].N)rnote_hybrid_pk_fusionrGs rhrz!AsyncTable._note_hybrid_pk_fusion4s ))+rjcTK|jjd{y7w)aConverge this table's RTE write path into its base table. One flush, freezing every memtable into an SSTable, then compaction triggers until every SSTable that existed at that moment has reached base. The loop runs client-side, reading progress from ``get_rte_stats``. Best-effort: SSTables created *while* it runs are deliberately not waited on, which is what lets it terminate on a table taking writes. Idempotent and safe on a cadence. There is no deadline, and the caller owns that. It returns when the target SSTables are gone, raises on a terminal server fault, and otherwise waits however long the server takes. A slow table and a stuck one are the same picture from the client: the compactor pool is shared across every table on the node, so a checkpoint queued behind unrelated work looks exactly like one that is merging. Wrap this in ``asyncio.wait_for`` for a wall-clock bound; abandoning it partway costs nothing. N)rrrGs rhrzAsyncTable.checkpoint_rte8s(kk((***rvcTK|jjd{y7w)uFreeze every table shard's active memtable into an SSTable. Does not touch the base table — compacting SSTables into base is `compact_rte`. On a node that has not claimed this table, this claims it and replays its WAL log first. N)rrrGs rhrzAsyncTable.flush_rteNskk##%%%rvcTK|jjd{y7w)aTrigger a background SSTable compaction pass per table shard. Returns once the passes are dispatched, not once they finish: watch ``get_rte_stats`` for progress, or use ``checkpoint_rte`` to loop until the current SSTables have reached base. N)rrrGs rhrzAsyncTable.compact_rteWskk%%'''rvrcTK|jj|d{S7w)aRead live per-table-shard RTE state. Answers "how far behind is my fresh tier", "which table shard is hot", and "why is my fresh-tier vector search brute-force". Mutates no table state, though on a node that has not claimed this table it claims it, exactly as a read would. Returns ``None`` only when the RTE write path is not enabled. Parameters ---------- include_sstable_rows Report a row count per SSTable. Off by default: each count opens an uncached Lance dataset, and ``checkpoint_rte`` polls this needing only generation numbers. N)rrrs rhrzAsyncTable.get_rte_stats`s$&[[../CDDDD (&(cTK|jjd{y7w)a_Drain and close any cached RTE shard writers for this table. When an RTE write spec is installed, `merge_insert` opens RTE shard writers and caches them for reuse across calls. This closes them, flushing pending data; writers reopen lazily on the next `merge_insert`. It is a no-op when no writers are cached. N)rrrGs rhrzAsyncTable.close_rte_writersuskk++---rvc6|jjS)zThe name of the table.)rrrGs rhrzAsyncTable.names{{!!rjcRK|jjd{S7w)rQN)rrrGs rhrzAsyncTable.schemas! [[''))))rxcK|jd{}tjj|jS71w)rN)rr6r,r-r;)rHrs rhr^zAsyncTable.embedding_functionss9{{}$(557GGXX%sA A2A cTK|jj|d{S7w)raN)rr[rbs rhr[zAsyncTable.count_rowss#[[++F3333rcxK|jj|jd{S7w)z Return the first `n` rows of the table. Parameters ---------- n: int, default 5 The number of rows to return. N)r4rbrArWs rhrVzAsyncTable.heads.ZZ\''*335555s 1:8:cJt|jj|S)a] Returns an [AsyncQuery][lancedb.query.AsyncQuery] that can be used to search the table. Use methods on the returned query to control query behavior. The query can be executed with methods like [to_arrow][lancedb.query.AsyncQuery.to_arrow], [to_pandas][lancedb.query.AsyncQuery.to_pandas] and more. )rLrr4rGs rhr4zAsyncTable.querys$++++-t44rjcK|jjs td ddl}|j }|j d{}|dn|}|j|jd{f||jd{d|}||j||f}|S#t$r t dwxYw7{7S7<w)awReturn the Lance dataset backing a local table. Remote tables cannot be opened as Lance datasets by the client. Use [to_arrow][lancedb.table.AsyncTable.to_arrow] or [query][lancedb.table.AsyncTable.query] to read them through the server. Parameters ---------- **kwargs Forwarded to `lance.dataset`. Returns ------- lance.LanceDataset The Lance dataset at this table handle's version and branch. Examples -------- >>> async def get_lance_dataset(table): ... return await table.to_lance() zGto_lance() is not supported for remote tables; query the server insteadrNr/r1) r _is_nativerFr/rsrYrOr?rrr2)rHrhr/r3 table_versionrOr5s rhrnzAsyncTable.to_lances,{{%%'%+    $$&"lln,  ,$- U]]((*  "&"="="??     $$fm%<=B ' =  - ?sK&CB4#CC )C:C;CC  C4C  CCCrdc Kt||jd{}|dk(s t|sD|jd{}|dk(rt |t |}|j di|S|jjs*|jj dd|i|d{S|dk(rLt|jd{dk(r(|jd{j di|S|dk(r5t|r*|jj dd|i|d{S|jd{j dd|i|S7R7-7777:7#w)aReturn the table as a pandas DataFrame. Parameters ---------- blob_mode: str, default "lazy" Controls how Lance blob columns are returned. Remote tables support "descriptions"; "bytes" and "lazy" are not yet supported. **kwargs Forwarded to PyArrow / Lance pandas conversion. Returns ------- pd.DataFrame NrZrgrdr[rre)r$rrbrAr#rarfrrr4rZrr_rn)rHrgrhrr\s rhrfzAsyncTable.to_pandassY 9%{{}$  &.CF.K"mmo-IN*. 2B#Cx#O4$--/)44>v> >  $8$@///N)NvNN N0dmmo%00O9OOO!%-O2B)N%s}FE3'F E6 A/F9E9:!FE;F;E=rBrrorrrrrtrrr)rHr$rrwrrrr*s rhrzAsyncTable.create_index.sn   " F$v,'(  ++**) +  L) &#&)#)#8#8#__ G s<A+C%.$BBBC%BC")4CC""C%cvK|jj||||||d{}t|S7w)a Create an index, returning a handle to the indexing job. Takes the same arguments as :meth:`create_index`. The job may already be complete when returned; callers must not assume the index exists until :meth:`AsyncJob.wait` resolves. rN)rrr))rHr$rrwrrrjobs rhrzAsyncTable.create_index_asyncsI<KK22 % 3  } s %979cVK|jj|d{y7w)a Drop an index from the table. Parameters ---------- name: str The name of the index to drop. Notes ----- This does not delete the index from disk, it just removes it from the table. To delete the index, run [optimize][lancedb.table.AsyncTable.optimize] after dropping the index. Use [list_indices][lancedb.table.AsyncTable.list_indices] to find the names of the indices. N)rrrs rhrzAsyncTable.drop_indexs $kk$$T***rtcVK|jj|d{y7w)rrN)rrsrs rhrszAsyncTable.prewarm_indexs (kk''---rtcVK|jj|d{y7w)ruN)rrvrxs rhrvzAsyncTable.prewarm_datas *kk&&w///rtrrcXK|jj||d{y7w)rN)rrrs rhrzAsyncTable.wait_for_indexs"kk((g>>> *(*cRK|jjd{S7w)rN)rrrGs rhrzAsyncTable.statss![[&&((((rxcRK|jjd{S7w)a8 Get the table URI (storage location). For remote tables, this fetches the location from the server via describe. For local tables, this returns the dataset URI. Returns ------- str The full storage location of the table (e.g., S3/GCS path). N)rrrGs rhrzAsyncTable.uris[[__&&&&rxcRK|jjd{S7w)r{N)rr|rGs rhr|z"AsyncTable.initial_storage_optionss![[88::::rxcRK|jjd{S7w)r~N)rrrGs rhrz!AsyncTable.latest_storage_options+s![[779999rxr3rc K|jd{}t||}t||} | | }|d}|d}|dk(rWtjt |d||d{\}} t |rtt|j|} t|| }nQ|dk7s|j@d|jvr2tjt|||j||dd{}ttjt|d{}t|\}} |jj||xsd ||| d{| r|j!SS7P777Z7!#t"$r6} d t%| vr t'| d t%| vr t'| d} ~ wwxYw#| r|j!wwxYww) aAdd more data to the [AsyncTable][lancedb.table.AsyncTable]. Parameters ---------- data: DATA The data to insert into the table. Acceptable types are: - list-of-dict - pandas.DataFrame - pyarrow.Table or pyarrow.RecordBatch mode: str The mode to use when writing the data. Valid values are "append" and "overwrite". Overwrite adopts explicitly declared blob layouts, while inferred bytes retain the existing blob v2 fields. on_bad_vectors: str, default "error" How to handle vectors with incorrect sizes, NaNs, or null elements. One of "error", "drop", "fill", "null". fill_value: float, default 0. The value to use when filling vectors. Only used if on_bad_vectors="fill". progress: callable or tqdm-like, optional A callback or tqdm-compatible progress bar. See :meth:`Table.add` for details. write_parallelism: int, optional Number of partitions to write in parallel. Higher values increase throughput but also peak memory use, since each partition buffers data in flight. Defaults to an estimate based on the data size, capped at the number of CPU cores. Lower this if bulk ingestion is using too much memory. allow_external_blob_outside_bases: bool, default False Allow blob URIs outside registered bases. See :meth:`Table.add`. Local tables only. Nrr overwrite)r[r\sembedding_functionsTr;r[r\rXr3)rArrz Cast errorzVector column contains NaN)rr9r%asyncio to_threadrr_ryrdr;rfrrrBrrrrtror)rHrrr[r\rArrr blob_tablerr]rr*s rhrzAsyncTable.add;s\{{}$%dF3)$7  !D  !$N  J ; $--%-% GD!$F+!9f M -dMB w & OO ',Bfoo,U **-% $D &'&&|T::,X6$ ! !"32S ) {%; s1v% m#-Q7 m#   sG E&AG %E)&BG 'E,(+G E.G ''E2E0E2G )G ,G .G 0E22 F1;1F,,F11F44G  G cht|tr|gntt|}t ||Srrrs rhrzAsyncTable.merge_insertrrjc KywrrerHr4rzrrrs rhrzAsyncTable.searchs c Kywrrers rhrzAsyncTable.searchs DGrc Kywrrers rhrzAsyncTable.search rc Kywrrers rhrzAsyncTable.searchrrc Kywrrers rhrzAsyncTable.searchs rc Kywrrers rhrzAsyncTable.searchrrc K|jSd} dfd }d}|dk(r||r|} d}nt|trd}nt|trk t j j ||d|d{\} \} +| |d{} t fd| Drd }n~d}n{d}nxd}nu|dk(r2||r|} ne||||d{\} | |d{} n>|d k(r9||r td ||||d{\} | |d{} |dk(r4jj } |r| j|} | S|dk(r!jj|| S|d k(rEjj } |r| j|} | j|| Std|d7k7U#t$r)} d t| vrd t| vrd}n| Yd} ~ d} ~ wwxYw7K7:77w)aCreate a search query to find the nearest neighbors of the given query vector. We currently support [vector search](https://lancedb.com/docs/search/vector-search/) and [full-text search](https://lancedb.com/docs/search/full-text-search/). All query options are defined in [AsyncQuery][lancedb.query.AsyncQuery]. Parameters ---------- query: list/np.ndarray/str/PIL.Image.Image, default None The targeted vector to search for. - *default None*. Acceptable types are: list, np.ndarray, PIL.Image.Image - If None then a plain [AsyncQuery][lancedb.query.AsyncQuery] is returned, equivalent to calling [query][lancedb.table.AsyncTable.query]. The select/where/limit clauses are applied to filter the table. vector_column_name: str, optional The name of the vector column to search. The vector column needs to be a pyarrow fixed size list type - If not specified then the vector column is inferred from the table schema - If the table has multiple vector columns then the *vector_column_name* needs to be specified. Otherwise, an error is raised. query_type: str *default "auto"*. Acceptable types are: "vector", "fts", "hybrid", or "auto" - If "auto" then the query type is inferred from the query; - If `query` is a list/np.ndarray then the query type is "vector"; - If `query` is a PIL.Image.Image then either do vector search, or raise an error if no corresponding embedding function is found. - If `query` is a string, then the query type is "vector" if the table has embedding functions else the query type is "fts" Returns ------- AsyncQuery, AsyncHybridQuery, AsyncFTSQuery, or AsyncVectorQuery A query builder object representing the query. Nc~t|ttjtj tj fSr)rnr rndarrayr Arrayr+r3s rh is_embeddingz'AsyncTable.search..is_embeddingbs$edBJJ"//%RS SrjcKt|trd}jd{}t||||}t j j |j}|j|}|Std|d}t|dkDr(t|dt|j|t|d|||fS7w)NrrzColumn 'z'' has no registered embedding function.rz0Embedding functions are registered for columns: z6No embedding functions are registered for any columns.)rnrPrr[r6r,r-r;rrrqrXr r)rzrr4rfuncsr'rrHs rhget_embedding_funcz-AsyncTable.search..get_embedding_funces %/" ;;=(F!9%#5 "  .::<LLE99/0D|"123**u:>J -.0 W %t+ +7)s&CCB+CcK|Ntj}|jt|jj |d{dSy7w)Nr)rget_running_looprun_in_executorr(r!#compute_query_embeddings_with_retry)rr4loops rhmake_embeddingz)AsyncTable.search..make_embeddings`$//1..*,!**NN  sA A A Arrrc3xK|]1}|jdjk(xr|jdk(3yw)rrBN)rrr)rrembedding_confs rhrz$AsyncTable.search..sD!"IIaLN,H,HH6 !  56s7:rColumnz$has no registered embedding functionz#Hybrid search requires a text query)rzUnknown query type: '')rzrrrr4rrz#Tuple[str, EmbeddingFunctionConfig]) r4rnrProrgatherrGrr nearest_tor$nearest_to_text)rHr4rzrrrrrr vector_queryrr*r:rs` @rhrzAsyncTable.search%st =::<  T" , -" ,!" ,V" ,1 " ,H $  E"$ % E=1" E3'+&nn))+*+=vuM<+^&1-;NE-R'R &- *2J)1J%* & 8 #E"$ ;M& E<62"N&4NE%JJ 8 #E" !FGG;M& E<62"N&4NE%JJ  !jjl--l;G!!..);<N 5 ::<//{/K K 8 #jjl--l;G!!..);<**5+*F F4ZLBC Cw(S" 3$@CFJ&+ #  :6 K 6 KsAI"/HG< HI*G?+AI,H7-IH:*I+H=,I?IBJ}}'..u}}= }}('// >   '335  jjl ;; "%++EKK8K << #%,,U\\:K ==%,,U]];K << #%++ELL9K   %113K == $%--emm>%..u~~>K <<%00>MM!!5#4#4K"".)778K8KL }}()11%--@ "".)778K8KL $$0U5J5J5V)99))!/%"7"78&&2)99%:O:OP &&2)99%:O:OP "".)778K8KL "")001D1DE xx)nnUXX6 (()==?   %002K  %55%%++U-B-B-J-JKrjrcVKt|j|jrX|jsL|j@|j 4ddlm}|j|jgz}||j||S|j|}|j||d{S7w)Nrr)max_batch_lengthr) rirrrrrrrrrrrP)rHr4rrrr0rs rhrzAsyncTable._execute_query;s *&&(A(A 00""*##+ D++tyyk9H-d.D.DhPUV V //6  ++',    sB B)"B'#B)cbK|j|}|j|d{S7wr)r explain_plan)rHr4r rs rhr!zAsyncTable._explain_planYs-//6  --g6666 &/-/r"r#cbK|j|}|j|d{S7wr)r analyze_plan)rHr4r$rs rhr'zAsyncTable._analyze_plan^s0//6  --.ABBBBrc`K|j|}|jd{S7wr)rr)rHr4rs rhr*zAsyncTable._output_schemahs+//6  ..0000s %.,.czK|jd{}|d}|d}t|||j||d}t|tj r8tj j|j|j}|jj|t|j|j|j|j|j |j"|j$|j&|j(|j*|j, d{S7 7w)NrrTr) rwhen_matched_update_all!when_matched_update_all_conditionwhen_not_matched_insert_all!when_not_matched_by_source_delete$when_not_matched_by_source_condition)when_not_matched_by_source_condition_exprr use_indexrvalidate_single_shard)rrfr;rnr rr.r/rPrexecute_merge_insertr_on_when_matched_update_all"_when_matched_update_all_condition_when_not_matched_insert_all"_when_not_matched_by_source_delete%_when_not_matched_by_source_condition*_when_not_matched_by_source_condition_expr_timeout _use_index_use_rte_validate_single_shard)rHr-r.r[r\rrs rhr/zAsyncTable._do_mergels{{}$  !$N  J  __)!   dBHH %''44T[[$//BSTD[[55  99(-(F(F272Z2Z,1,N,N272Z2Z5:5`5`:?:j:j**&+&B&B    % s"D;D6DD;1D92D;9D;cKt|tr |jn|}|jj|d{S7w)aFDelete rows from the table. This can be used to delete a single row, many rows, all rows, or sometimes no rows (if your predicate matches nothing). Parameters ---------- where: str or :class:`~lancedb.expr.Expr` The filter condition. Can be a SQL string or a type-safe :class:`~lancedb.expr.Expr` built with :func:`~lancedb.expr.col` and :func:`~lancedb.expr.lit`. The filter must not be empty, or it will error. Examples -------- >>> import lancedb >>> data = [ ... {"x": 1, "vector": [1.0, 2]}, ... {"x": 2, "vector": [3.0, 4]}, ... {"x": 3, "vector": [5.0, 6]} ... ] >>> db = lancedb.connect("./.lancedb") >>> table = db.create_table("my_table", data) >>> table.to_pandas() x vector 0 1 [1.0, 2.0] 1 2 [3.0, 4.0] 2 3 [5.0, 6.0] >>> table.delete("x = 2") DeleteResult(num_deleted_rows=1, version=2) >>> table.to_pandas() x vector 0 1 [1.0, 2.0] 1 3 [5.0, 6.0] If you have a list of values to delete, you can combine them into a stringified list and use the `IN` operator: >>> to_remove = [1, 5] >>> to_remove = ", ".join([str(v) for v in to_remove]) >>> to_remove '1, 5' >>> table.delete(f"x IN ({to_remove})") DeleteResult(num_deleted_rows=1, version=3) >>> table.to_pandas() x vector 0 3 [5.0, 6.0] N)rnrCrr2rs rhr2zAsyncTable.deletes8d%/ud$;ELL [['' 2222s=AAArc>K| | td| | td|,|jDcic]\}}|t|}}}t|tr|j n|}|j j||d{Scc}}w7 w)a| This can be used to update zero to all rows in the table. If a filter is provided with `where` then only rows matching the filter will be updated. Otherwise all rows will be updated. Parameters ---------- updates: dict, optional The updates to apply. The keys should be the name of the column to update. The values should be the new values to assign. This is required unless updates_sql is supplied. where: str or [Expr][lancedb.expr.Expr], optional The filter condition. Can be a SQL string or a type-safe [Expr][lancedb.expr.Expr] built with [col][lancedb.expr.col] and [lit][lancedb.expr.lit]. Only rows that satisfy this filter will be updated. updates_sql: dict, optional The updates to apply, expressed as SQL expression strings. The keys should be column names. The values should be SQL expressions. These can be SQL literals (e.g. "7" or "'foo'") or they can be expressions based on the previous value of the row (e.g. "x + 1" to increment the x column by 1) Returns ------- UpdateResult An object containing: - rows_updated: The number of rows that were updated - version: The new version number of the table after the update Examples -------- >>> import asyncio >>> import lancedb >>> from lancedb.expr import col >>> import pandas as pd >>> async def demo_update(): ... data = pd.DataFrame({"x": [1, 2], "vector": [[1, 2], [3, 4]]}) ... db = await lancedb.connect_async("./.lancedb") ... table = await db.create_table("my_table", data) ... # x is [1, 2], vector is [[1, 2], [3, 4]] ... await table.update({"vector": [10, 10]}, where=col("x") == 2) ... # x is [1, 2], vector is [[1, 2], [10, 10]] ... await table.update(updates_sql={"x": "x + 1"}) ... # x is [2, 3], vector is [[1, 2], [10, 10]] >>> asyncio.run(demo_update()) Nz2Only one of updates or updates_sql can be providedz.Either updates or updates_sql must be provided)rr<r]rnrCto_sqlrr7)rHrjr1rrGrHrs rhr7zAsyncTable.updatesl  ;#:QR R ?{2MN N  :A--/J$!Q1l1o-JKJ&0&=ELLN5 [['' Y???K@s4BB ABB BrRc nK|t|tur td|3t|tusd|cxkrdkstdtd|duxs|du}d}d}t |t r|}nt |t r~td|jDr^t|dk7s td|jDs td tt|j\}}|Q|r td |j|jj!|j#|||d{St |t$j&r|g}t |t(rFt|Dchc]}t |t$j&c}rt%j*|}|rI|r td |jj-t)|j||d{S|r td | td t |t$j.r#|jj1|d{S|jj3t)|jd{S7Tcc}w77H7w)a Add new columns with defined values. Parameters ---------- transforms: Dict[str, str | FunctionApplication] or FunctionApplication A map of column name to a SQL expression to use to calculate the value of the new column. These expressions will be evaluated for each row in the table, and can reference existing columns. Alternatively, you can pass a pyarrow field or schema to add new columns with NULLs. A mapping with one ``FunctionApplication`` value keeps its scalar or named-struct result in the named table column. A bare named-struct application expands its ordered result fields as one atomic binding; aliases come from ``rename(columns=...)``. Function columns are supported only on LanceDB Cloud and Enterprise. computed: Dict[str, str], optional Local declaration and refresh require a source build with the ``experimental-functions`` Cargo feature, which published packages do not enable. Remote Cloud/Enterprise operations remain available. A mapping from output column names to SQL expressions derives each output field from its expression. A direct projection of a Blob v2 field inherits Blob v2 semantics; other expressions derive their ordinary Arrow type. Mapping order is declaration and dependency order. Unlike ``transforms``, the expression is stored rather than evaluated now: the column is committed with no values, and rows get them from [`refresh_column`][lancedb.table.AsyncTable.refresh_column]. A refresh also recomputes the rows whose inputs changed since they were computed, so a mutated input is reflected by the next refresh. While a declaration reads a column, that column cannot be renamed, retyped or dropped. On LanceDB Cloud and Enterprise the expression is planned by the server. Cannot be combined with ``transforms``. auto_refresh_enabled: bool, optional Enable automatic refresh for each computed output declared by this call. Defaults to True. Local tables store the setting; automatic scheduling requires Cloud or Enterprise. auto_refresh_interval_seconds: int, optional Positive minimum seconds after a successful refresh before another automatic refresh is eligible. When omitted, the server determines when to refresh. Unchanged inputs do not force recomputation. Applies to every declared computed output. These options cannot be used with one-time transforms or ordinary all-null columns. Examples -------- >>> await table.add_columns( # doctest: +SKIP ... computed={"total": "price * quantity"}, ... auto_refresh_enabled=True, ... auto_refresh_interval_seconds=300, ... ) Returns ------- AddColumnsResult version: the new version number of the table after adding columns. Nz&auto_refresh_enabled must be a booleanrl'$ zXauto_refresh_interval_seconds must be a positive integer no greater than u64::MAX / 1000c3<K|]}t|tywrrnrDrrs rhrz)AsyncTable.add_columns..ls2 7.os/;@ 5"56/rz:one add_columns call declares exactly one Function bindingzGadd_columns cannot mix a Function application with SQL computed columnszautomatic refresh options require computed or Function columnsz3add_columns requires transforms or computed columns)rrrrrnrDrrrrqr"rUrTr<_ensure_declarableradd_function_columnsto_canonical_jsonr Fieldr radd_computed_columnsSchemaadd_columns_with_schemarY) rHrXrSrTrUoptions_suppliedfunction_applicationfunction_output_namers rhrYzAsyncTable.add_columns sb +5I0JRV0VEF F ( 4 . /s :4K8KK2 L2  ! , 9,D8  $# j"5 6#-  D )c2 @J@Q@Q@S2 / :!#3/DNDUDUDW/,!P:>d:CSCSCU>V9W 6 "6  + ' ! 3 3 599$668$$-   j"(( +$J j$ 'C.8 9Z288 $ 9- :.J  R99X^^%&$-  P   RS S j")) ,<>> errors = await table.function_errors(column="embedding") # doctest: +SKIP >>> for record in errors.records: # doctest: +SKIP ... print(record.job_id, record.row_offset, record.error_message) >>> if errors.truncated: # doctest: +SKIP ... print("listing stopped at the limit") rN)rrcr`s rhrczAsyncTable.function_errorss6N[[00&1    "+)+cKt|jj|d{tjS7w)a Like :meth:`refresh_column`, but returns a handle to the refresh job instead of blocking until it completes. Local refresh requires a source build with ``experimental-functions``. Remote Cloud/Enterprise jobs do not require this feature. The job may already be complete when returned; callers must not assume the column is filled until :meth:`AsyncJob.wait` resolves. Invalid input -- an unknown column, or one that is not computed -- raises here rather than failing the job. On local tables the job runs in-process; on LanceDB Cloud and Enterprise it is the server's backfill job. Returns ------- AsyncJob[RefreshColumnResult] A job whose successful ``wait`` returns row counts plus the source and published table versions. Examples -------- >>> import asyncio >>> import lancedb >>> async def refresh_in_background(): ... db = await lancedb.connect_async("./.lancedb") ... table = await db.create_table("computed_job_async_demo", [{"x": 1}]) ... await table.add_columns(computed={"doubled": "x * 2"}) ... job = await table.refresh_column_async("doubled") ... result = await job.wait() ... assert result.rows_assigned == 1 ... return await job.status() >>> asyncio.run(refresh_in_background()) # doctest: +SKIP 'finished' N)r+rr^RefreshColumnJobResult from_jsonr[s rhr^zAsyncTable.refresh_column_asyncs7L++226: : " , ,  :s$A>AcTK|jj|d{S7w)a Alter column names and nullability. alterations : Iterable[Dict[str, Any]] A sequence of dictionaries, each with the following keys: - "path": str The column path to alter. For a top-level column, this is the name. For a nested column, this is the dot-separated path, e.g. "a.b.c". - "rename": str, optional The new name of the column. If not specified, the column name is not changed. - "data_type": pyarrow.DataType, optional The new data type of the column. Existing values will be casted to this type. If not specified, the column data type is not changed. - "nullable": bool, optional Whether the column should be nullable. If not specified, the column nullability is not changed. Only non-nullable columns can be changed to nullable. Currently, you cannot change a nullable column to non-nullable. Each alteration must specify at least one of "rename", "data_type", or "nullable". The legacy key "name" is also accepted for renaming. Unknown keys raise ValueError before any alterations are applied. Returns ------- AlterColumnsResult version: the new version number of the table after the alteration. N)rrgres rhrgzAsyncTable.alter_columnss$@[[..{;;;;rcTK|jj|d{S7w)zrUpdate per-field metadata. See [`Table.update_field_metadata`][lancedb.table.Table.update_field_metadata].N)rrkris rhrkz AsyncTable.update_field_metadata3s# [[66w????rcTK|jj|d{S7w)z Drop columns from the table. Parameters ---------- columns : Iterable[str] The names of the columns to drop. N)rrrxs rhrzAsyncTable.drop_columns:s#[[--g6666rcRK|jjd{S7w)a Retrieve the version of the table LanceDb supports versioning. Every operation that modifies the table increases version. As long as a version hasn't been deleted you can `[Self::checkout]` that version to view the data at that point. In addition, you can `[Self::restore]` the version to replace the current table with a previous version. N)rrOrGs rhrOzAsyncTable.versionEs![[((****rxcK|jjd{}|D]<}t|dd\}}tj|t |dzz|d<>|S7Gw)z0 List all versions of the table N timestampiʚ;r) microseconds)rrdivmodr fromtimestampr)rHversionsrHrnanoss rhrzAsyncTable.list_versionsQsv2244 A$AkNMBNGU%33Gt{{77@$G      rjcRK|jjd{S7wr)rrrGs rhrzAsyncTable.blob_columnss[[--////rxcjK|jj|t||d{S7wr)rr r!r s rhr zAsyncTable.fetch_blobss6 [[,, +GV<    *313chK|jj|t|d{S7wr)rrr r s rhrzAsyncTable.fetch_blob_rangess) [[2264>JJJJ )202cK|jj|t||d{}t|S7wr)rrr!r")rHr$rhandless rhrzAsyncTable.fetch_blob_filess@  44 +GV<   (( s *><>cKt|}|yt|jjt j |d{S7w)zOpen a snapshot-bound descriptor as a seekable file. See [Table.open_blob_file][lancedb.table.Table.open_blob_file]. N)r rrrrrrs rhrzAsyncTable.open_blob_filesD0 ;  dkk88J9OPPQQPsAAA AcVK|D]}|j|d{ y7 w)aAsynchronously iterate over files in descriptor order, preserving nulls. Use ``async for file in table.open_blob_files(descriptors)``. Each non-null file must be closed after use. ```python async for video in videos.open_blob_files(rows["video"]): if video is not None: with video: payload = await video.aread() ``` Nrrs rhrzAsyncTable.open_blob_filess.& 8J++J77 7 87s )' )c,t|jS)aTag management for the dataset. Similar to Git, tags are a way to add metadata to a specific version of the dataset. .. warning:: Tagged versions are exempted from the :py:meth:`optimize(cleanup_older_than)` process. To remove a version that has been tagged, you must first :py:meth:`~Tags.delete` the associated tag. ) AsyncTagsrrGs rhrSzAsyncTable.tagss %%rjc,t|jS)rU) AsyncBranchesrrGs rhrVzAsyncTable.branches,sT[[))rjc6|jjSrF)rrYrGs rhrYzAsyncTable.current_branch5rGrjr@cXK|jt||||d{S7wrC)rrrDs rhrEzAsyncTable.optimize9s7x&& &'97 C      s !*(*cZK|jj|||d{S7w)N)rr9rA)rrE)rHrr9rAs rhrzAsyncTable._do_optimize{s7 [[))-/1*    rcRK|jjd{S7w)rN)rrGrGs rhrGzAsyncTable.list_indicess![[--////rxrHcZK|jj|||d{S7w)rKrHN)rrMrLs rhrMzAsyncTable.tokenizes*([[))%:)VVVVrcpK|jj|d{}|ytdi|S7w)rONre)rrQIndexStatistics)rHrIrs rhrQzAsyncTable.index_statss:kk--j99 ="+U+ + :s 646cRK|jjd{S7wr)rrrGs rhrz!AsyncTable.uses_v2_manifest_pathss![[779999rxcTK|jjd{y7w)az Migrate the manifest paths to the new format. This will update the manifest to use the new v2 format for paths. This function is idempotent, and can be run multiple times without changing the state of the object store. !!! danger This should not be run while other concurrent operations are happening. And it should also run until completion before resuming other operations. You can use [AsyncTable.uses_v2_manifest_paths][lancedb.table.AsyncTable.uses_v2_manifest_paths] to check if the table is already using the new path style. N)rrrGs rhrz$AsyncTable.migrate_manifest_paths_v2s$kk33555rvrlrmrncXK|jj||d{y7wr)rrvrts rhrvz!AsyncTable.replace_field_metadatas"*kk00\JJJr) r!rrrrfrrgrr r) rrrfrrgrr rrrrrrrrrrrrrrrr)rrLrrrr) r$rorrrwcOptional[Union[IvfFlat, IvfPq, IvfRq, HnswPq, HnswSq, HnswFlat, BTree, Bitmap, LabelList, Fm, FTS]]rrrrrr)r$rorrrwr3rrrrrrrr)rrrrr)rr2rz(Optional[Literal['append', 'overwrite']]r[zOptional[OnBadVectorsType]r\zOptional[float]rArrrrrrrr)NNrNN) r4rrzrrrrrrrrrL)N.NN) r4rorzrrLiteral['auto']rrrrrz8Union[AsyncHybridQuery, AsyncFTSQuery, AsyncVectorQuery]) r4rorzrrrrrrrrrK) r4z$Union[VEC, 'PIL.Image.Image', Tuple]rzrrr4rrrrrrN) r4rorzrrrrrrrrrJ) r4z8Union[VEC, str, 'PIL.Image.Image', Tuple, FullTextQuery]rzrrrrrrrrrN) r4rrzrrrrrrrrzDUnion[AsyncQuery, AsyncHybridQuery, AsyncFTSQuery, AsyncVectorQuery])rzUnion[VEC, Tuple]rrN)r4rWrzQAsyncHybridQuery | AsyncFTSQuery | AsyncVectorQuery | AsyncQuery | AsyncTakeQueryrrrrrr)rjzOptional[Dict[str, Any]]r1rrrrr) rXzidict[str, str | FunctionApplication] | FunctionApplication | pa.Field | List[pa.Field] | pa.Schema | NonerSzdict[str, str] | NonerTrrUrrr)r$rorrErr)r$rorz AsyncJob[RefreshColumnJobResult])rfzIterable[dict[str, Any]]rrr)rrr)rOz int | str)rOzOptional[int | str])rrrrrrM)rrrrrrMrrrrr)rrrz!AsyncIterator[Optional[BlobFile]])rr&)rr(r)rrr9rrArrr)rrr9rrArrrrrr)rroruzdict[str, str])ZrrrrrrirRrlrnrprrrrrrrrrrrrrrrr^r[rVr4rnrfrArrrrsrvrrrrr|rrrrrrrrr!r'r*r/r2r7rYr\rcr^rgrkrrOrr|rrrrrr rrrrrSrVrYrErrGrMrQrrrrrrvrerjrhrr[s @L/3*.-1',>>, > ( > + >!%>6/3*.-1', , (  +  !%   &%#>0> >2#+J(0"6,+,&(/4E'+E E*.""* Y 4 6 51f PD7=@H#' ,0"-^^ ^  ^(*)^*+^,-^H#' ,0"-&& &  &(*)&*+&,-&. /&P+(.,00@IQT?U?(?3<? ?") ' ;:(:B59&*9=+/27l!l!7 l! 3 l! $ l!7l!)l!,0l! l!\>1@,0 &-17; *  +  5  -1&)-17; GG*G$ G + G 5 G BGG-1(+-17; *&  +  5  -1&)-17; 3*$  +  5  -1%(-17; *#  +  5  -1(+-17; G*&  +  5   ,0 &-17;}D }D * }D  }D+}D5}D N}D~ 5' 5  5HH H\%)'+   "  %    <7>I CC; C C1% &% % ) %  %  % N33n-1?@-104 ?@)?@* ?@ . ?@  ?@PUK+/,048UKUK(UK*UK(2UK UKn8:!% $# ) ) )  )  ) V) ) )) V <4 <   @ 0@  @ 8 @  @ L;?  '     8    0!%$( WW W " W  W,,( :6([#6 KK-;K  KrjrcReZdZUdZded<ded<ded<dZded <dZd ed <d Zy) r/a Statistics about an index. Attributes ---------- num_indexed_rows: int The number of rows that are covered by this index. num_unindexed_rows: int The number of rows that are not covered by this index. index_type: str The type of index that was created. distance_type: Optional[str] The distance type used by the index. num_indices: Optional[int] The number of parts the index is split into. rnum_indexed_rowsnum_unindexed_rowszLiteral['IVF_FLAT', 'IVF_SQ', 'IVF_PQ', 'IVF_RQ', 'IVF_HNSW_SQ', 'IVF_HNSW_PQ', 'IVF_HNSW_FLAT', 'FTS', 'BTREE', 'BITMAP', 'LABEL_LIST']rNz(Optional[Literal['l2', 'cosine', 'dot']]rr num_indicesct||Sr)r)rHrs rh __getitem__zIndexStatistics.__getitem__stS!!rj)rrrrrrr8r:rerjrhr/r/s<"  ?CM;B!%K%"rjr/cDeZdZUdZded<ded<ded<ded<ded <y ) ra Statistics about a table and fragments. Attributes ---------- total_bytes: int The total size, in bytes, of the table's data files, index files, and overlay files. Read from the manifest, so this excludes deletion files and manifests. num_rows: int The total number of rows in the table. num_deleted_rows: Optional[int] The total number of rows marked as deleted across all fragments of the table. These rows are not counted in ``num_rows``, but still occupy space on disk until the table is compacted, so a large value here indicates that the table should be optimized. Fragments in which every row was deleted are dropped outright, so their rows are not counted here. ``None`` when the backend does not report deletion counts. num_indices: int The total number of indices in the table. fragment_stats: FragmentStatistics Statistics about fragments in the table. r total_bytesrrnum_deleted_rowsr8FragmentStatisticsfragment_statsNrrerjrhrrs&0M##&&rjrc0eZdZUdZded<ded<ded<y)r>a Statistics about fragments. Attributes ---------- num_fragments: int The total number of fragments in the table. num_small_fragments: int The total number of small fragments in the table. Small fragments have low row counts and may need to be compacted. lengths: FragmentSummaryStats Statistics about the number of rows in the table fragments. r num_fragmentsnum_small_fragmentsFragmentSummaryStatsr`Nrrerjrhr>r>:s  !!rjr>cXeZdZUdZded<ded<ded<ded<ded<ded<ded <y ) rCaW Statistics about fragments sizes Attributes ---------- min: int The number of rows in the fragment with the fewest rows. max: int The number of rows in the fragment with the most rows. mean: int The mean number of rows in the fragments. p25: int The 25th percentile of number of rows in the fragments. p50: int The 50th percentile of number of rows in the fragments. p75: int The 75th percentile of number of rows in the fragments. p99: int The 99th percentile of number of rows in the fragments. rrNrOmeanp25p50p75p99NrrerjrhrCrCOs,* H H I H H H HrjrCc>eZdZdZdZd dZd dZd dZd dZd dZ y) rz Table tag manager. c||_yrrrHr!s rhrz Tags.__init__t  rjcptj|jjj S) List all table tags. Returns ------- dict[str, Tag] A dictionary mapping tag names to version numbers. )r'rrrSr rGs rhr z Tags.listws&xx ((--/00rjcrtj|jjj |S) Get the version of a tag. Parameters ---------- tag: str, The name of the tag to get the version for. )r'rrrS get_versionrHtags rhrSzTags.get_versions(xx ((44S9::rjcvtj|jjj ||ya! Create a tag for a given table version. Parameters ---------- tag: str, The name of the tag to create. This name must be unique among all tag names for the table. version: int, The table version to tag. N)r'rrrSrrHrUrOs rhrz Tags.creates' !!((g67rjcttj|jjj |yz Delete tag from the table. Parameters ---------- tag: str, The name of the tag to delete. N)r'rrrSr2rTs rhr2z Tags.deletes% !!((-.rjcvtj|jjj ||yz Update tag to a new version. Parameters ---------- tag: str, The name of the tag to update. version: int, The new table version to tag. N)r'rrrSr7rXs rhr7z Tags.updates' !!((g67rjNrzDict[str, Tag]rUrorrrUrorOrrrrUrorr rrrrrr rSrr2r7rerjrhrros% 1 ; 8 / 8rjrcveZdZdZd dZd dZ d ddZdddZddZddZ ddd Z d dd Z y)rDz Table branch manager. c4||_|j|_yr)_parentr)rHparents rhrzBranches.__init__s mm rjcptj|jjj S)3List all branches, mapping name to branch metadata.)r'rrrVr rGs rhr z Branches.lists$xx ,,11344rjNctj|jjj |||}|j |S)jCreate a branch and return a handle scoped to it. Parameters ---------- name: str Name of the new branch. from_ref: str, optional Source branch to fork from. Defaults to ``main``. from_version: int, optional A specific version on ``from_ref`` to fork from. Defaults to latest. )r'rrrVr_wrap)rHrfrom_ref from_versionrJs rhrzBranches.creates?"hh KK ' 'h E zz+&&rjctj|jjj ||}|j ||S)Check out an existing branch and return a handle scoped to it. Parameters ---------- name: str Name of the branch to check out. version: int, optional A specific version on the branch to pin. When set, the returned handle is a read-only view of that version; when omitted it tracks the branch's latest and stays writable. )r'rrrVr|rj)rHrrOrJs rhr|zBranches.checkouts:hht{{33<??rjcttj|jjj ||S)auCherry-pick a branch onto main, or dry-run. Parameters ---------- from_branch: str Branch to cherry-pick from. dry_run: bool, default False When True, only preview. When False, attempt the cherry-pick. Notes ----- A failed cherry-pick returns ``status="failed"`` instead of raising. )r'rrrV cherry_pickrHrudry_runs rhrwzBranches.cherry_picks*xx ,,88gNOOrjc:|jj||Sr)rdrKrIs rhrjzBranches._wraps ||// WEErj)rerrDict[str, Any]r)rrorkrrlrr'Table'r)rrorOrrr}rrurorr|rruroryrrr|)rJrrOrrr}) rrrrrr rr|r2rsrwrjrerjrhrDrDs$5#'&* '' '$ '  ', 04@P"CGF'F2?F FrjrDc>eZdZdZdZd dZd dZd dZd dZd dZ y) r&z" Async table tag manager. c||_yrrLrMs rhrzAsyncTags.__init__ rNrjcfK|jjjd{S7w)rPN)rrSr rGs rhr zAsyncTags.list s'[[%%**,,,, (1/1chK|jjj|d{S7w)rRN)rrSrSrTs rhrSzAsyncTags.get_version s)[[%%11#6666r clK|jjj||d{y7wrW)rrSrrXs rhrzAsyncTags.create% s(kk%%c7333 *424cjK|jjj|d{y7wrZ)rrSr2rTs rhr2zAsyncTags.delete3 s&kk%%c*** )313clK|jjj||d{y7wr\)rrSr7rXs rhr7zAsyncTags.update> s(kk%%c7333rNr]r^r_r`rarerjrhr&r& s% - 7 4 + 4rjr&cjeZdZdZdZd dZ d d dZd ddZddZddZ d dd Z y)r(zAsync table branch manager.c||_yrrLrMs rhrzAsyncBranches.__init__O rNrjcfK|jjjd{S7w)rgN)rrVr rGs rhr zAsyncBranches.listR s%[[))..0000rNcK|dk(rd}|jjj|||d{}t|S7w)rimainN)rrVrr)rHrrkrlinners rhrzAsyncBranches.createV sG& v Hkk**11$,OO%  Ps2AAAc|Kt|jjj||d{S7w)rnN)rrrVr|)rHrrOs rhr|zAsyncBranches.checkoutn s0 4 4 = =dG LLMMLs /<: <cjK|jjj|d{y7wrp)rrVr2rs rhr2zAsyncBranches.delete| s$kk""))$///rchK|jjj|d{S7w)rrN)rrVrsrts rhrszAsyncBranches.diff s'[[))..{;;;;r cjK|jjj||d{S7w)zCherry-pick a branch onto main, or dry-run. A failed cherry-pick returns ``status="failed"`` instead of raising. N)rrVrwrxs rhrwzAsyncBranches.cherry_pick s+[[))55k7KKKKrr{r)rrorkrrlrrrr)rrorOrrrrr~rr) rrrrrr rr|r2rsrwrerjrhr(r(L sx%1#'&* !! !$ !  !0 N0< 16LL)-L Lrjr()rfrrgrrr)rxr rr)r}r rr )rrrrrr)r BaseExceptionrrorr)rrorr)rrrrorrrr)rr rr)rr rrrzOptional[pa.Table]r)rrrr)rrrrrr)NNrr)rz'DATA'r]rr;rr[rr\rrXrrrr)rerr]rrXrrr)rvrrwrrr)rrrrrr)rQrr]rrr)r$pa.Arrayrrrr)r$rrrrr)rrrrrr)rrrztuple[pa.Array, pa.DataType])r pa.DataTyperr)rrrr)rrrr)rrrz Optional[_JsonSerializationPlan])rrrz!Dict[str, _JsonSerializationPlan])rr rrrr )rr r]rrr )rsList[pa.Field]rrrr)rrrrrr)rrrOptional[pa.DataType])rrrrrr)rrr rrr)Nrr)rzUnion[pa.Schema, LanceModel]r[rr\rr)rerrrr;rrr)r8ror9rorro)rrNN) rerr[z(Literal['error', 'drop', 'fill', 'null']r\rr]rr;rrr)rrr]rr;rr List[dict])rrrrrr)rrr%rrr)rrrzror[ror\rrrrrrr)r0 Union[pa.Array, pa.ChunkedArray]rrr\rrr)r0$Union[pa.ListArray, pa.ChunkedArray]rzpa.BooleanArray)rGrrr)rrorr)rerrz&Tuple[pa.Schema, pa.RecordBatchReader])r0rrr)r0rrr)rr)r;r) __future__rrrrrrabcrr dataclassesrrr functoolsr typingr r r r rrrrrrrrrrr urllib.parserlancedb.scannablerrrrr_blobrr r!r"r#r$r{r% lancedb.arrowr&lancedb.background_loopr'r( lancedb.jobr)r*r+ dependenciesr,r-r.r/r0r r1rbpyarrowr pyarrow.datasetpyarrow.computecomputer pyarrow.fsrrnumpyrcommonr2r3r4 embeddingsr5r6rr7r8r9r:r;r<r=r>r?r@rArBexprrCr+rDrEr r-rFpydanticrGrHr4rIrJrKrLrMrNrOrPrQrRrSrTrUrVrWutilrXrYrZr[r\r]r^rr_r`rarbrirvr{r~rrrrrrrr_lancedbrrrrrrrrrrrrrrrrrrPILrrrrrrrrrurr%rIrCrfrdryrurkrrrrrrrrrrrrrr9rprrrrrrr2r5r^r;r5rBrr_rrrrr#rrrbrKr`rQrrcrbrr/rr>rCrrDr&r(rerjrhrs# #!(%""VV%<1111J    +/" P#P:=P PQ?( $8 "+6:8"QU14@M 4%&#              QF QFj3 G@ER)-T %T T p37< </<< $ >< >< >< ><  > > > > 0""&15 iB iBiBiB iB  iB / iBiBX! )! !! !6.4 ? < <+<~$/,"&P"KP"KfD '"'" '"T '' '@ "" "(     >B8B8JPFPFfB4B4J?L?Lrj