kQj0ddlmZddlmZmZmZmZddlZddl Z ddl Z ddl m Z mZeeede j"ee j$fZeeej*e j,e j.fZeeefZdZGdd eZd ed efd Zd ee j$de j<d ee j$fdZ ddedee j<d e j@fdZ!de j<fdZ"defdZ#y))Path)IterableListOptionalUnionN)_check_for_pandaspandasz pd.DataFramevectorc(eZdZdZdefdZdefdZy) CredentialzCredential fieldreturncyNz********selfs S/home/oem/CyberBrain26-unified/.venv/lib/python3.12/site-packages/lancedb/common.py__repr__zCredential.__repr__cyrrrs r__str__zCredential.__str__rrN)__name__ __module__ __qualname____doc__strrrrrrr r s#rr urirct|SN)r)rs r sanitize_urir"s s8Or input_iterschemac#K|D]}t|tjstdt ||j |k7rA tj j|gj|jd}|y#tjj$rtd|d|j wxYww)a Wrapper around an iterator of record batches. If the batches don't match the schema, try to cast them to the schema. If that fails, raise an error. This is helpful for users who might have written the iterator with default data types in PyArrow, but specified more specific types in the schema. For example, PyArrow defaults to float64 for floating point types, but Lance uses float32 for vectors. zExpected RecordBatch, got rzjInput RecordBatch iterator yielded a batch with schema that does not match the expected schema. Expected: z Got: N) isinstancepa RecordBatch TypeErrortyper$Table from_batchescast to_batcheslib ArrowInvalid ValueError)r#r$batchs r_casting_recordbatch_iterr3"s %08e FG G <<6 ! --ug6;;FCNNPQRS  66&&  GGMhO"\\N, sAC  AB C 9C  C datac\t|rIt|tjr/tj j ||jSt|tj r|jSt|tjr.tj j|gjSt|tjjr7tjjj|jSt|tjjr|jSt|tjr|St|j j#dr7|j$j&dk(r|j)jSt|t*r9|,t-||}tjj||St/dt1dt|d)z7Convert various types of input into a RecordBatchReaderr$polars DataFramez>Must provide schema to write dataset from RecordBatch iterablezUnknown data type z. Supported types: list of dicts, pandas DataFrame, polars DataFrame, pyarrow Table/RecordBatch, or Pydantic models. See https://docs.lancedb.com/tables/ for examples.)r r&pdr8r'r+ from_pandas to_readerr(r,datasetDatasetScanner from_datasetRecordBatchReaderr*r startswith __class__rto_arrowrr3r1r))r4r$s rdata_to_readerrD>s:dBLL#Axx##D#8BBDD D"(( #~~ D".. )xx$$dV,6688 D"**,, -zz!!..t4>>@@ D"**,, -~~ D".. /  T ((2 NN # #{ 2}}((** D( #  ,T6:D''44VTB BP  d -A A  rcH|jt|jyy)z. Make sure the metadata is valid utf8 N)metadata_validate_metadatar6s rvalidate_schemarHgs"6??+#rrFc|jD]D\}}t|tr |jd)t|t s:t|Fy#t$rt d|dwxYw)zo Make sure the metadata values are valid utf8 (can be nested) Raises ValueError if not valid utf8 utf8z Metadata key zG is not valid utf8. Consider base64 encode for generic binary metadata.N)itemsr&bytesdecodeUnicodeDecodeErrorr1dictrG)rFkvs rrGrGos~  "1 a   4 q ! "&  #A3'JJ s AA2r!)$pathlibrtypingrrrrnumpynppyarrowr'pyarrow.dataset dependenciesr r r9rOr+r(DATAlistndarrayArray ChunkedArrayVECrURIVECTOR_COLUMN_NAMEr r"Schemar3r@rDrHrGrrrrbs 229 T$Z8BNN3K KL D"**bhh 78 CIcc(24)) bnn:/3& &  +& & R,BII,""r