kQj/ddlmZmZddlZddlmZmZmZmZddl m Z ddl Z ddl ZddlmZmZmZddlmZmZGdd eZGd d eeZGd d eZGddeZGddeZy))ABCabstractmethodN)ListOptionalTupleUnion)add_note) BaseModelField PrivateAttr)TEXTretry_with_exponential_backoffceZdZdZy)UntrustedEmbeddingConfigErrora< Raised when using an embedding function restored from table metadata whose settings this process has not approved. See [allow_remote_code][lancedb.embeddings.EmbeddingFunctionRegistry.allow_remote_code] and [allow_endpoint][lancedb.embeddings.EmbeddingFunctionRegistry.allow_endpoint]. N)__name__ __module__ __qualname____doc__\/home/oem/CyberBrain26-unified/.venv/lib/python3.12/site-packages/lancedb/embeddings/base.pyrrsrrcFeZdZUdZdZeed<eZeed<eZ e ed<e dZ e de de fd Z e de de fd Zedeefd Zdeeeeeffd Ze defd ZdeefdZedeeej:dffdZedeeej:dffdZdeeej:dffdZ deeej:dffdZ!de"deeeejFffdZ$dZ%edefdZ&dZ'dZ(de)defdZ*defdZ+y)EmbeddingFunctiona An ABC for embedding functions. All concrete embedding functions must implement the following methods: 1. compute_query_embeddings() which takes a query and returns a list of embeddings 2. compute_source_embeddings() which returns a list of embeddings for the source column For text data, the two will be the same. For multi-modal data, the source column might be images and the vector column might be text. 3. ndims() which returns the number of dimensions of the vector column  max_retries_ndims_original_argsc *ddlm}|j|}|di|}t||_|j 4d|j vr&|jjd|j|jj||S)z> Create an instance of the embedding function r EmbeddingFunctionRegistrytrust_remote_coder) registryr!$_EmbeddingFunction__resolveVariablesdictrremote_code_model model_fields setdefaultr" get_instance_trust)clskwargsr!resolved_kwargsinstances rcreatezEmbeddingFunction.create1s 8008))"&v,  & & ( 4#s'7'77  # # . .#X%?%?  "..077Arargsreturnc$|j|SN)r$)r+r0s r_resolve_variablesz$EmbeddingFunction._resolve_variablesIs%%d++rcddlm}tj|}|j }|j }|j D]\}}t|tr2|jds!||vrtd|d}t|d|t|tsY|jdsk|ddjd d } t| dk(r |j| d ||<| \} } |j| ||<|S#t$r,td j!| d }t|d |wxYw#t$r | ||<YwxYw)z/ Resolve variables in the args r r z$var:zSensitive key 'z$' cannot be set to a hardcoded valuez2Help: Use $var: to set sensitive keys to variablesN:)maxsplitrz#Variable '{}' not found in registryzZHelp: Variables are reset in new Python sessions. Use `registry.set_var` to set variables.)r#r!copydeepcopyr)sensitive_keysitems isinstancestr startswith ValueErrorr splitlenget_varKeyErrorformat) r+r0r!new_argsr#r;kvexcpartsnamedefaults r__resolveVariablesz$EmbeddingFunction.__resolveVariablesMsp 8==&,99;++-NN$ .DAq!S!!,,w*?ADW %aS(LMRS !S!all7&;!" C! 4u:? "&.&6&6uQx&@ %*MD'.&.&6&6t&< 5 .:!$ "(AHHqR!G "  "$.&- .sD:E 5E EEcgS)z Return a list of keys that are sensitive and should not be allowed to be set to hardcoded values in the config. For example, API keys. rrrrr;z EmbeddingFunction.sensitive_keysws  rcy)a Return the model whose repository code this function runs when its `trust_remote_code` field is enabled, as a `(name, revision)` pair. Return None if the function never runs code from a model repository. When the function is restored from table metadata, `trust_remote_code` is only enabled for models allowed with [allow_remote_code][lancedb.embeddings.EmbeddingFunctionRegistry.allow_remote_code]. Nrselfs rr&z#EmbeddingFunction.remote_code_modelsrcj|jjd}|t|jSdS)z The `trust_remote_code` to use for table metadata that omits it. Only tables written before the flag was always recorded omit it, so this is the default from that time. r"F)r'getboolrL)r+fields r!_legacy_trust_remote_code_defaultz3EmbeddingFunction._legacy_trust_remote_code_defaults4  $$%89&+&7tEMM"BUBrcgS)a Return the non-default URLs this function sends data or credentials to. When the function is restored from table metadata, each of these must be allowed with [allow_endpoint][lancedb.embeddings.EmbeddingFunctionRegistry.allow_endpoint]. rrPs r endpointszEmbeddingFunction.endpointss  rNcy)z Compute the embeddings for a given user query Returns ------- A list of embeddings for each input. The embedding of each input can be None when the embedding is not valid. NrrQr0r,s rcompute_query_embeddingsz*EmbeddingFunction.compute_query_embeddingss rcy)zCompute the embeddings for the source column in the database Returns ------- A list of embeddings for each input. The embedding of each input can be None when the embedding is not valid. NrrZs rcompute_source_embeddingsz+EmbeddingFunction.compute_source_embeddingss rcPt|j|j|i|S)zCompute the embeddings for a given user query with retries Returns ------- A list of embeddings for each input. The embedding of each input can be None when the embedding is not valid. r)rr[rrZs r#compute_query_embeddings_with_retryz5EmbeddingFunction.compute_query_embeddings_with_retrys9 -  ) )t7G7G     rcPt|j|j|i|S)zCompute the embeddings for the source column in the database with retries. Returns ------- A list of embeddings for each input. The embedding of each input can be None when the embedding is not valid. r_)rr]rrZs r$compute_source_embeddings_with_retryz6EmbeddingFunction.compute_source_embeddings_with_retrys5 -  * *8H8H  rtextsct|tr|g}|St|tjr|j }|St|tj r|j j }|S)z? Sanitize the input to the embedding function. )r=r>paArray to_pylist ChunkedArraycombine_chunks)rQrcs rsanitize_inputz EmbeddingFunction.sanitize_inputsk eS !GE  rxx (OO%E r /((*446E rcHt|ds td|jS)NrzAEmbeddingFunction was not created with EmbeddingFunction.create())hasattrr@rrPs rsafe_model_dumpz!EmbeddingFunction.safe_model_dumps+t-.S """rcy)z< Return the dimensions of the vector column NrrPs rndimszEmbeddingFunction.ndims rc dt|jddxsi}||d<tdd|i|S)z Creates a pydantic Field that can automatically annotate the source column for this embedding function json_schema_extraNsource_column_forrr%popr rQr,rrs r SourceFieldzEmbeddingFunction.SourceFields@ !,?!F!L"M15-.C'8CFCCrc dt|jddxsi}||d<tdd|i|S)aH Creates a pydantic Field that can automatically annotate the target vector column for this embedding function. The field can be annotated as ``list[float]`` for compatibility with static type checkers. LanceDB will infer the fixed vector dimension from this embedding function. rrNvector_column_forrrtrvs r VectorFieldzEmbeddingFunction.VectorFields@!,?!F!L"M15-.C'8CFCCr_EmbeddingFunction__valuecJt|dsyt|t|k(S)N__dict__F)rlvars)rQr{s r__eq__zEmbeddingFunction.__eq__s"w +DzT']**rcXttt|jSr3)hash frozensetr~r<rPs r__hash__zEmbeddingFunction.__hash__ sId4j..0122r),rrrrrint__annotations__r rrr% classmethodr/r4r$ staticmethodrr>r;rrr&rTrVrXrlistrnparrayr[r]r`rbrndarrayrjrmrorwrzobjectrrrrrrrs  K-FC&=ND(.,d,t,,'d't''RDI 8E#x}2D,E#F C$CC49  4bhhPTn@U;V     DrxxQU~AVrrrrrrrs rrceZdZdZdedeeejdffdZ de deeejdffdZ e dee eejfdeeejdffdZy) TextEmbeddingFunctionzH A callable ABC for embedding functions that take text as input queryr1Nc.|j|g|i|Sr3)r])rQrr0r,s rr[z.TextEmbeddingFunction.compute_query_embeddingss!.t--eEdEfEErrccF|j|}|j|Sr3)rjgenerate_embeddingsrQrcr0r,s rr]z/TextEmbeddingFunction.compute_source_embeddings$s%##E*''..rcy)z+Generate the embeddings for the given textsNrrs rrz)TextEmbeddingFunction.generate_embeddings*rpr)rrrrr>rrrrr[rr]rrrrrrrrrsFF eBHHdN# $F // eBHHdN# $/  49bjj01 eBHHdN# $  rrcTeZdZUdZeed<eedfed<defdZdZ dZ de fd Z y ) UntrustedEmbeddingFunctiona Stands in for an embedding function restored from table metadata whose settings this process has not approved. The table stays usable for anything that doesn't compute embeddings, while computing embeddings raises [UntrustedEmbeddingConfigError][lancedb.embeddings.UntrustedEmbeddingConfigError]. The real function is never constructed, since some load their model in `__init__`. function_name.reasonsr1c~djd|jD}td|jd|zS)N c3&K|] }d| yw)z - Nr).0reasons r z4UntrustedEmbeddingFunction._error..BsGd6(OGszEmbedding function 'zP' was restored from table metadata with settings this process has not approved: )joinrrr)rQdetailss r_errorz!UntrustedEmbeddingFunction._errorAsJ))G$,,GG,"4#5#5"67F FHO P  rc"|jr3rrZs rr[z3UntrustedEmbeddingFunction.compute_query_embeddingsHkkmrc"|jr3rrZs rr]z4UntrustedEmbeddingFunction.compute_source_embeddingsKrrc"|jr3rrPs rroz UntrustedEmbeddingFunction.ndimsNrrN) rrrrr>rrrrr[r]rrorrrrr2s@  38_ 5 srr)abcrrr9typingrrrr lancedb.utilr numpyrpyarrowrepydanticr r r utilsrrPermissionErrorrrrrrrrrrsh$ //!227 O o3 3o3d  i   - 0!2r