uPj2dZddlmZddlZddlmZmZddlmZm Z m Z m Z ddl Z ddl mZddlmZddlmZmZdd lmZdd lmZmZdd lmZdd lmZdd lmZmZerddl m Z GddeZ!y)z Base RAG Ingestion class. Provides abstract methods for: - OCR - Chunking - Embedding - Vector Store operations Providers can inherit and override methods as needed. ) annotationsN)ABCabstractmethod) TYPE_CHECKINGAnyFinalcast)verbose_logger)uuid4)DEFAULT_CHUNK_OVERLAPDEFAULT_CHUNK_SIZE)async_safe_get)get_async_httpx_clienthttpxSpecialProvider)extract_text_from_pdf)RecursiveCharacterTextSplitter)RAGIngestOptionsRAGIngestResponse)RouterceZdZUdZdZded< d ddZddZeddZ d dd Z dd Z dd Z dd Z e d dd Z d ddZy)BaseRAGIngestiona Base class for RAG ingestion. Providers should inherit from this class and override methods as needed. For example, OpenAI handles embedding internally when attaching files to vector stores, so it overrides the embedding step to be a no-op. Fboolsupports_existing_file_idNc||_||_dt|_|j d|_t tttf|j dxsddi|_ |j d|_ t tttf|j dxsi|_ |j d|_ |jy) Ningest_ocrchunking_strategytypeauto embedding vector_storename)ingest_optionsrouterr ingest_idget ocr_configr dictstrrrembedding_configvector_store_config ingest_name_load_credentials_from_config)selfr#r$s i/home/oem/CyberBrain26-unified/.venv/lib/python3.12/site-packages/litellm/rag/ingestion/base_ingestion.py__init__zBaseRAGIngestion.__init__/s - "57),),,U315 cN   2 3 G7G2 !/ 2 2; ?37S#XHZHZ[iHjHpnp3q )--f5 **,c4ddlm}|jjd}|rutj rd|j |}|sy|jD]\}}||j|<dD]#}||jvs||vs|j|=%yyy)a Load credentials from litellm_credential_name if provided in vector_store config. This allows users to specify a credential name in the vector_store config which will be resolved from litellm.credential_list. When a stored credential is used, its values take precedence over caller-supplied equivalents so endpoint and identity fields stay consistent with the credential definition. r)CredentialAccessorlitellm_credential_nameN)api_baseaws_sts_endpointaws_web_identity_token).litellm.litellm_core_utils.credential_accessorr3r+r&litellmcredential_listget_credential_valuesitems)r.r3credential_namecredential_valueskeyvalues r/r-z.BaseRAGIngestion._load_credentials_from_configEs V!%!9!9!=!=>W!X w66'9'O'OP_'` $/557 6 U05((- 6 6 $222sBS7S005  6 7?r1c:|jjddS)zGet the vector store provider.custom_llm_provideropenai)r+r&)r.s r/rBz$BaseRAGIngestion.custom_llm_provider`s''++,A8LLr1c\K|r |\}}}|||dfS|rttj}t||d{}|j |j }|j ddxsd}|jjdd}|||dfS|rddd|fStd7mw) aG Upload / prepare file for ingestion. Args: file_data: Tuple of (filename, content_bytes, content_type) file_url: URL to fetch file from file_id: Existing file ID to use Returns: Tuple of (filename, file_content, content_type, existing_file_id) N) llm_provider/documentz content-typezapplication/octet-streamz,Must provide file_data, file_url, or file_id) rrRAGrraise_for_statuscontentsplitheadersr& ValueError) r. file_datafile_urlfile_idfilename file_content content_type http_clientresponses r/uploadzBaseRAGIngestion.uploades" 3< 0HlL\<= = !7EYE]E]!^K$2;$IIH  % % '#++L~~c*2.<*H#++//@Z\<= = tW, ,GHHJs:B,B*A.B,c K|jr|sy|jjdd}|r d|vrd\}}nd\}}tj|j d}d|d |}|j *|j j |d |||i d{}n#tj |d |||i d{}t|d r.|jr"d jd|jDSy7c7Aw)z Perform OCR on file content to extract text. Args: file_content: Raw file bytes content_type: MIME type of the file Returns: Extracted text or None if OCR not configured/needed Nmodelzmistral/mistral-ocr-latestimage) image_urlr[) document_urlr\utf-8zdata:z;base64,r)rYrHpagesz c3NK|]}t|ds|jyw)markdownN)hasattrr`).0pages r/ z'BaseRAGIngestion.ocr..siwW[]gOht}}is%%) r'r&base64 b64encodedecoder$aocrr9rar^join) r.rSrT ocr_modeldoc_typeurl_key b64_contentdata_url ocr_responses r/rzBaseRAGIngestion.ocrsl??..w8TU  G|3 8 Hg > Hg$--l;BB7K !,x }E ;; "!%!1!1 (GX>"2"L ") (GX>"L < )l.@.@;;i9K9Kii i s%BDC?#D?DADDcd}|r|}n|r|s |jd}|sgS|j xsi}|jdt}|jdt}|jd d}||d } |r|| d <td i| } | j|S#t$rh|jdr;tjdt |}|s2tjdgcYStjdgcYSYwxYw) a Split text into chunks using RecursiveCharacterTextSplitter. Args: text: Text from OCR (if used) file_content: Raw file content bytes ocr_was_used: Whether OCR was performed Returns: List of text chunks Nr]s%PDFz(PDF detected, attempting text extractionzkPDF text extraction failed. Install 'pypdf' or 'PyPDF2' for PDF support, or enable OCR with a vision model.z,Binary file detected, skipping text chunking chunk_size chunk_overlap separators)rqrr) rgUnicodeDecodeError startswithr debugrrr&r r r split_text) r.textrS ocr_was_used text_to_chunk splitter_argsrqrrrssplitter_kwargs text_splitters r/chunkzBaseRAGIngestion.chunks+$%)  M ,  , 3 3G < I $55; )--l, .> 0 (F(F ~~../0DD +/(())+3+%N !JJ#)!__D8'F'+jjj&? ?J59JJ)!)%!1 5?5/ +O^%>>" / 52&:*:  OB %!@/    $ $%> B$>> "!f   suEC- EAC5+C/,9C5%C1&C5C3'C5,E/C51C53C55 E>8D<6E7E<EE)N)r#rr$z Router | None)returnNone)rr))NNN)rOtuple[str, bytes, str] | NonerP str | NonerQrrz7tuple[str | None, bytes | None, str | None, str | None])rS bytes | NonerTrrr)ryrrSrrzrr list[str])rrrlist[list[float]] | None)rSrrRrrTrrrrrrrrztuple[str | None, str | None])rOrrPrrQrrr)r __module__ __qualname____doc__r__annotations__r0r-propertyrBrWrrrrrrrtr1r/rr$s',t+ !%-(--,66MM 48#" !I0!I!I !I A !IF.".!.  .`:7:7#:7 :7  :7x== "=2(, "  !    -  %  '  848#" K0KK K  Kr1r)"r __future__rreabcrrtypingrrrr r9litellm._loggingr litellm._uuidr litellm.constantsr r $litellm.litellm_core_utils.url_utilsr&litellm.llms.custom_httpx.http_handlerrr"litellm.rag.ingestion.file_parsersrlitellm.rag.text_splittersrlitellm.types.ragrrrrrtr1r/rsO # #22+G?EEAOsOr1