kQj*K ddlZddlZddlZddlZddlZddlZddlZddlmZmZddl m Z ddlm Z ddl m Z mZmZmZmZddlmZddlZddlZddlmZddlmZd ZeZd ee ejBfd e fd Z"d ee ejBfd e fd Z#d e d e ejHe ffdZ%dee ejBfde d e fdZ&dFde fdZ'dFdejPdeee)e*ffdZ+dee d e fdZ, dFdejZdee dee)d ee fdZ.dFdej^dee)d e fdZ0dejbd e*fdZ2dejbd ee)fd Z3d!ed dfd"Z4d!eed ee)fd#Z5dej^d$e d!eed%ee fd&Z6e d'Z7e7jqe d(e fd)Z9e7jqe:d(e:fd*Z9e7jqe)d(e)fd+Z9e7jqe;d(e;fd,Z9e7jqe d(e fd-Z9e7jqe*d(e*fd.Z9e7jqed(ee=e>ffd2Z9e7jqe?d(e?fd3Z9e7jqejd(ejfd4Z9e7jqejd(ejfd5Z9e7jqejd(ejfd6Z9e7jqejd(ejfd7Z9e7jqejd(ejfd8Z9e7jqejd(ejfd9Z9e7jqejd(ejfd:Z9d;ZGdeHd?e fd@ZIdejPfdAZJdBejfdCZLdBejfdDZMdBejfdEZNy)GN)datedatetime)Decimal)singledispatch)TupleUnionOptionalAnyList)urlparse)validate_table_namec. ddl}|S#t$rYywxYw)Nr)adlfs ImportError)rs Q/home/oem/CyberBrain26-unified/.venv/lib/python3.12/site-packages/lancedb/util.pysafe_import_adlfsrs! s  urireturnctt|}|j}|sd}|S|dvrd}|St|dk(rd}|S)z Get the scheme of a URI. If the URI does not have a scheme, assume it is a file URI. Parameters ---------- uri : str or pathlib.Path The URI to parse. Returns ------- str: The scheme of the URI. file)s3as3ns3r )r strschemelen)rparsedrs rget_uri_schemer%sZc#h F ]]F  M > ! M V  Mct|}t|}t|jdk(r|S|js |j S|j|j zS)z Get the location of a URI. If the parameter is not a url, assumes it is just a path Parameters ---------- uri : str or pathlib.Path The URI to parse. Returns ------- str: Location part of the URL, without scheme r )rr rrnetlocpath)ruri_strrs rget_uri_locationr%@sS#hG g F 6==Q =={{}}v{{**r ct|dk(rDtjtjj ddd}t |}||fSt|dk(rttjtjj dtjj d}tjtj|}t |}||fStjj|S) zT Get a PyArrow FileSystem from a URI, handling extra environment variables. r AWS_ENDPOINT)endpoint_overriderequest_timeoutconnect_timeoutazAZURE_STORAGE_ACCOUNT_NAMEAZURE_STORAGE_ACCOUNT_KEY) account_name account_key) rpa_fs S3FileSystemosenvirongetr%rAzureBlobFileSystem PyFileSystem FSSpecHandler FileSystemfrom_uri)rfsr# az_blob_fss r fs_from_urir=[scd"    jjnn^<   $4x   $):..(DE 'BC/    3 3J ? @$4x    $ $S ))r basepartsct|tjr|j|St |}t |dk(rt tj|g|St |}dj|jg|Dcgc]}|jdc}}|j|jScc}w)a Join a URI with multiple parts, handles both local and remote paths Parameters ---------- base : str The base URI parts : str The parts to join to the base URI, each separated by the appropriate path separator for the URI scheme and OS r/)r#) isinstancepathlibPathjoinpathrrr joinr#rstrip_replacegeturl)r>r?urlpnew_paths rjoin_urirMvs$ %t}}e$$ t9Ddv%7<<-u-..tn88SXX4F4FGqQXXc]GH|||*1133Hs Cmodulecl tj|S#t$rtd|xs|wxYw)ac Import the specified module. If the module is not installed, raise an ImportError with a helpful message. Parameters ---------- module : str The name of the module to import mitigation : Optional[str] The package(s) to install to mitigate the error. If not provided then the module name will be used. zPlease install ) importlib import_moduler)rN mitigations rattempt_import_or_raiserSsBD&&v.. DOJ,@&+ABCCDs3tblflattenc|dur1 |j}td|jDr. |St|tr@t|t s0|dkr t d|dkDr|j}|dz}|dkDr|S)aJ Flatten all struct columns in a table. Parameters ---------- flatten: Optional[Union[int, bool]] If flatten is True, flatten all nested columns. If flatten is an integer, flatten the nested columns up to the specified depth. If unspecified, do not flatten the nested columns. Tc3nK|]-}tjj|j/ywN)patypes is_structtype).0cols r z"flatten_columns..s#FC288%%chh/Fs35rzIPlease specify a positive integer for flatten or the boolean value `True`r )rUanyschemarBintbool ValueError)rTrUs rflatten_columnsres$++-CF3::FF J GS !*Wd*C a< k++-C qLGk Jr r#cXdtdtfddjfd|DS)NsegmentrcVtd|Dr|Sd|jdddS)Nc3JK|]}|jxs|dk(yw)_N)isalnum)r]chars rr_z=_format_field_path..format_segment..s"At||~,,As!#`z``)allreplace)rgs rformat_segmentz*_format_field_path..format_segments0 AA AN7??3-.a00r .c3.K|] }|ywrX)r]rgrps rr_z%_format_field_path..s@N7+@s)rrF)r#rps @r_format_field_pathrts-111 88@4@ @@r fielddimcg||j}t|jr*t|j}|||k(r t |gSgSt j j|jr]g}t|jjD]7}|jt|jj|||9|SgSrX) nameis_vector_columnr\infer_vector_column_dimrtrYrZr[range num_fieldsextend_iter_vector_columnsru)rur#rv field_path vector_dimcolumnsidxs rr~r~s%4$$J #,UZZ8 ;*+&z23 3  xx%**%../ YC NN/ 0@0@0EzSVW X Y Ir racRg}|D]}|jt|g| t|dk(r;|9g}|D]}|jt|gt|dk(r|dS|}t|dkDrtd|t|dk(r td|dS)z Get the vector column name Parameters ---------- schema : pa.Schema The schema of the vector column. Returns ------- str: the vector column name. rr zmSchema has more than one vector column. Please specify the vector column name for vector search. Candidates: z^There is no vector column in the data. Please specify the vector column name for vector search)r}r~rrd)rarvvector_col_namesru unfiltereds rinf_vector_column_queryrsF 4UB DEF !co   ?E   25"= > ? z?a a= % q  ..>-? A   ! F   A r data_typecPtjj|rStjj|js)tjj |jrytjj |rt|jSy)z Check if the column is a vector column. Parameters ---------- data_type : pa.DataType The data type of the column. Returns ------- bool: True if the column is a vector column. TF)rYrZis_fixed_size_list is_floating value_typeis_uint8is_listryrs rryry so xx""9- Y112 88  Y11 2   ) $ 4 455 r ctjj|r |jStjj |rt |j SyrX)rYrZr list_sizerrzrrs rrzrz"sF xx""9-""" xx "&y';';<< r queryct|tr|r)t|tjr|jdk(r t dyy)zGReject empty vector inputs before vector-column inference or execution.rzQuery vector must not be emptyN)rBlistnpndarraysizerd)rs r_validate_query_vectorr*s:5$5"**%%**/9::+:%r c|yt|tjr|jdk(ry|jdSt|t j t jfrt|dk(ryt jj|jr|jjSt jj|js)t jj|jr"|dj}| t|SdSt|St|t t"frB|r@|d}t|t t"tjfr t|St|Sy)Nr)rBrrndimshaperYArray ChunkedArrayrrZrr\rr is_large_listas_pyrtuple)rfirsts r_query_vector_dimr2s }%$ ::?{{2%"((BOO45 u:? 88 & &uzz 2::'' ' 88  EJJ '288+A+A%**+M!HNN$E!&!23u: < <5z%$'Ea edE2::6 7u: 5z r query_typevector_column_namec|dk7r t|||S|dk(ry||dk7ryt|t|}| td|S)Nftshybridrvz~No vector column found in the schema. Please specify the vector column name explicitly via the `vector_column_name` parameter.)rrrrd)rarrrs rinfer_vector_column_namerJso Uu%%!!U }x/0=Nu=UV!   r ctd)Nz/SQL conversion is not implemented for this type)NotImplementedErrorvalues r value_to_sqlrks O PPr rc2|jdd}d|dS)N'z'')rors rrjrjps MM#t $E ugQ<r cPdtj|jdS)z~Convert bytes to a hex string literal. See https://datafusion.apache.org/user-guide/sql/data_types.html#binary-types zX'r)binasciihexlifydecoders rrjrjvs(   '..01 33r ct|SrX)rrs rrjrjs u:r ctj|rytj|r |dkDrdSdSt|S)NzCAST('NaN' AS DOUBLE)rzCAST('Infinity' AS DOUBLE)zCAST('-Infinity' AS DOUBLE))mathisnanisinfrrs rrjrjsB zz%& zz%,1AI ( ;X  u:r cb|js tdtt|dS)Nz4Non-finite Decimal values cannot be converted to SQLf) is_finiterdrformatrs rrjrjs* ?? OPP uc* ++r c4t|jSrX)rupperrs rrjrjs u:   r cy)NNULLrsrs rrjrjs r c*d|jdSNr isoformatrs rrjrj u ! ##r c*d|jdSrrrs rrjrjrr cLddjtt|zdzS)N[, ])rFmaprrs rrjrjs$ 3|U34 4s ::r cZddjd|jDzdzS)Nz named_struct(rc3fK|])\}}tt|dt|+yw)rN)rr)r]kvs rr_z_..s3 =AQ|CF#$B|A&7 8 s/1))rFitemsrs rrjrjs:  )) EJ[[]      r c4t|jSrX)rrrs rrjrjs  &&r c|js tdStt|jj d}tt|jj d}d|d|dS)Nrr zmap(rr)is_validrrvaluesru)rkeysrs rrjrjse >>D!! U\\//23 4D $u||11!45 6F $r& ##r c4t|jSrX)rtolistrs rrjrjs   ''r c*tt|SrX)rrcrs rrjrjs U $$r c*tt|SrX)rrbrs rrjrjs E ##r c*tt|SrX)rfloatrs rrjrjs e %%r c tj|r tdStj|jddvr t dtt tj|ddS)Nr)psr;asz7SQL timestamps cannot preserve sub-nanosecond precisionnsUTC)unittimezone)risnatr datetime_datadtyperdrdatetime_as_stringrs rrjrjs` xxD!!  $Q'+==RSS B11%dUST UUr cBtjfd}|S)zThis is a decorator which can be used to mark functions as deprecated. It will result in a warning being emitted when the function is used.ctjdttjdjdtdtjdt|i|S)Nalwaysz Function z6 is deprecated and will be removed in a future version)category stackleveldefault)warnings simplefilterDeprecationWarningwarn__name__)argskwargsfuncs rnew_funczdeprecated..new_funcs`h(:; DMM?+..(   i);<T$V$$r ) functoolswraps)rrs` r deprecatedrs' __T % % Or rxct|y)zVerify the table name is valid.N)native_validate_table_name)rxs rrrs t$r base_exceptionnotect|dr|j|yt|jdtr,|jddz|zg|jdd|_yt d)Nadd_noter r zCannot add note to exception)hasattrrrBrrrd)rrs rrrst~z*% N''*C 0    "T )D 0   $  788r c tdddtjffd }jt |j Dcgc]}||j |c}Scc}w)a& Convert a PyArrow Table to a PyTorch Tensor. Each column is converted to a tensor (using zero-copy via DLPack) and the columns are then stacked into a single tensor. Fails if torch is not installed. Fails if any column is more than one chunk. Fails if a column's data type is not supported by PyTorch. Parameters ---------- tbl : pa.Table or pa.RecordBatch The table or record batch to convert to a tensor. Returns ------- torch.Tensor: The tensor containing the columns of the table. torchr^cx|jdkDr tdj|jdS)Nr z8Single batch was too large to fit into a one-chunk tabler) num_chunks Exception from_dlpackchunk)r^rs r to_tensorz tbl_to_tensor..to_tensor*s4 >>A VW W  1..r )rSrYrstackr{ num_columnscolumn)rTr irs @r tbl_to_tensorrsV( $GW 5E/r/ ;;%:PQQ #**Q-0Q RRQsA+batchctdd}|j|jDcgc]}|j|c}Scc}w)a Convert a PyArrow RecordBatch to a PyTorch Tensor. Each column is converted to a tensor (using zero-copy via DLPack) and the columns are then stacked into a single tensor. Fails if torch is not installed. Fails if a column's data type is not supported by PyTorch. Parameters ---------- batch : pa.RecordBatch The record batch to convert to a tensor. Returns ------- torch.Tensor: The tensor containing the columns of the record batch. r)rSr rr )rrr^s rbatch_to_tensorr2s<& $GW 5E ;;%--H3))#.H IIHsActdd}tdd}|jDcgc]}|jd}}|j|j |}t |j d}|Scc}w)a Convert a PyArrow RecordBatch to a list of PyTorch Tensor, one per row Each column is converted to a tensor (using zero-copy via DLPack) and the columns are then stacked into a single tensor. The 2D tensor is then converted to a list of tensors, one per row Fails if torch or numpy is not installed. Fails if a column's data type is not supported by PyTorch. rnumpyF)zero_copy_onlyrr)rSrto_numpytensor column_stackrunbind)rrrr^rstackedrowss rbatch_to_tensor_rowsrIsw $GW 5E #GW 5E=B]]Kcs||5|1KGKll5--g67G 1% &D KLsA?c ptdd}t|jj|jDcic]\}}||j |}}}t |jDcgc])}|jDcic] \}}||| c}}+c}}}Scc}}wcc}}wcc}}}w)a6 Convert a PyArrow RecordBatch to a list of per-row dicts of PyTorch Tensors. Each column is first converted to a 1-D tensor (zero-copy via DLPack), then sliced per row. The result is a list whose length is ``batch.num_rows`` and whose items are dicts keyed by column name. This shape composes directly with PyTorch's default ``DataLoader`` collate, which stacks the per-row dicts back into a dict of batched tensors. Fails if torch is not installed. Fails if a column's data type is not supported by PyTorch. Parameters ---------- batch : pa.RecordBatch The record batch to convert. Returns ------- list[dict[str, torch.Tensor]] One per-row dict per row in the batch. Each dict maps column name to a 0-D tensor view into the column. r) rSzipranamesrr r{num_rowsr)rrrxr^tensorsrrs rbatch_to_tensor_dictr#\s0 $GW 5EU\\//? D# e$$G u~~&   .5]]_=\T6vay=   > sB%2B1 B+B1+B1rX)OrrrPrr3rCrrrdecimalrrtypingrrr r r urllib.parser rrpyarrowrY pyarrow.fsr;r1_lancedbrrrrrrDrr%r9r=rMrSTablerbrcrertFieldr~SchemarDataTyperyrzrrrrregisterrjbytesrr\rrdictScalar MapScalarrbool_integerfloating datetime64r BaseExceptionrr RecordBatchrrr#rsr rr9s  #$44!G c7<</0S6+%W\\ 12+s+6*S*U5#3#3S#89*645gll*+4S4S44DCD&  HU395E,F FAT#YA3A<@ 889+3C= #Y"(BII(HSM(S(V .r{{x};#;$;Xc]x}0 II C=! BQQsS u4U44sSuUw,W, ,tTtDz"T$Z#x $X$!$t$T$$t u;U4; ;;t T  ryy!'RYY'"'r||$$R\\$%$rzz"(RZZ(#(rxx %RXX%!%rzz"$RZZ$#$r{{#&R[[&$&r}}%VR]]V&V,%c% 9] 9# 9SrxxS<J2>>J.&  r