kQjxddlmZddlZddlmZddlmZddlm Z ddl m Z m Z e d Gd d eZ y) )ListN)attempt_import_or_raise)TextEmbeddingFunction)register)TEXTweak_lru instructorc"eZdZUdZdZeed<dZeed<dZ eed<dZ e ed <dZ e ed <d Z e ed <d Zeed<dZeed<eddZdedeej(fdZdedeej(fdZdedefdZeddZy)InstructorEmbeddingFunctiona< An embedding function that uses the InstructorEmbedding library. Instructor models support multi-task learning, and can be used for a variety of tasks, including text classification, sentence similarity, and document retrieval. If you want to calculate customized embeddings for specific sentences, you may follow the unified template to write instructions: "Represent the `domain` `text_type` for `task_objective`": * domain is optional, and it specifies the domain of the text, e.g., science, finance, medicine, etc. * text_type is required, and it specifies the encoding unit, e.g., sentence, document, paragraph, etc. * task_objective is optional, and it specifies the objective of embedding, e.g., retrieve a document, classify the sentence, etc. For example, if you want to calculate embeddings for a document, you may write the instruction as follows: "Represent the document for retrieval" Parameters ---------- name : str The name of the model to use. Available models are listed at https://github.com/xlang-ai/instructor-embedding#model-list; The default model is hkunlp/instructor-base batch_size : int, default 32 The batch size to use when generating embeddings device : str, default "cpu" The device to use when generating embeddings show_progress_bar : bool, default True Whether to show a progress bar when generating embeddings normalize_embeddings : bool, default True Whether to normalize the embeddings quantize : bool, default False Whether to quantize the model source_instruction : str, default "represent the document for retrieval" The instruction for the source column query_instruction : str, default "represent the document for retrieving the most similar documents" The instruction for the query Examples -------- import lancedb from lancedb.pydantic import LanceModel, Vector from lancedb.embeddings import get_registry, InstructorEmbeddingFunction instructor = get_registry().get("instructor").create( source_instruction="represent the document for retrieval", query_instruction="represent the document for retrieving the most " "similar documents" ) class Schema(LanceModel): vector: Vector(instructor.ndims()) = instructor.VectorField() text: str = instructor.SourceField() db = lancedb.connect("~/.lancedb") tbl = db.create_table("test", schema=Schema, mode="overwrite") texts = [{"text": "Capitalism has been dominant in the Western world since the " "end of feudalism, but most feel[who?] that..."}, {"text": "The disparate impact theory is especially controversial under " "the Fair Housing Act because the Act..."}, {"text": "Disparate impact in United States labor law refers to practices " "in employment, housing, and other areas that.."}] tbl.add(texts) zhkunlp/instructor-basename batch_sizecpudeviceTshow_progress_barnormalize_embeddingsFquantizez$represent the document for retrievalsource_instructionz@represent the document for retrieving the most similar documentsquery_instructionr)maxsizecVt|j|jdggdS)Nfoor)lengenerate_embeddingsr)selfs b/home/oem/CyberBrain26-unified/.venv/lib/python3.12/site-packages/lancedb/embeddings/instructor.pyndimsz!InstructorEmbeddingFunction.ndimsfs,4++d.E.Eu-M,NOPQRSSqueryreturnc>|j|j|ggSN)rr)rr!argskwargss rcompute_query_embeddingsz4InstructorEmbeddingFunction.compute_query_embeddingsjs!''$*@*@%)H(IJJr textsc|j|}|Dcgc]}|j|g}}|j|Scc}wr$)sanitize_inputrr)rr(r%r&texttexts_formatteds rcompute_source_embeddingsz5InstructorEmbeddingFunction.compute_source_embeddingsmsG##E*GLMtD33T:MM''88Ns>c|j}|j||j|j|j|j j }|S)N)rrrr) get_modelencoderrrrtolist)rr(modelress rrz/InstructorEmbeddingFunction.generate_embeddingsrsW ll "44!%!:!:;;  &(  r cxtdd}tdd}|j|j}|jrzd|jj j vrd|jj _|jj||jjh|j}|S)NInstructorEmbeddingtorchqnnpack)dtype) r INSTRUCTORrrbackends quantizedsupported_enginesengine quantizationquantize_dynamicnnLinearqint8)rinstructor_embeddingr6r2s rr/z%InstructorEmbeddingFunction.get_model}s6 !#8 (9$// : ==U^^55GGG2;((/&&77( 8E r N)__name__ __module__ __qualname____doc__rstr__annotations__rintrrboolrrrrr rrnparrayr'r r-rr/r rr r sFP)D#(JFC"t"!%$%HdEDJsaTTKcKtBHH~K9t9bhh9  $ ar r )typingrnumpyrLutilrbaserregistryrutilsr r r rNr rrUs= *'! ,}"7}}r