uPj*dZddlmZddlZddlmZmZmZddlm Z ddl m Z ddl m Z ddlmZer dd lmZdd lmZdd Zdd ZGd deZy)a Vertex AI RAG Engine Ingestion implementation. Uses: - litellm.files.acreate_file for uploading files to GCS - Vertex AI RAG Engine REST API for importing files into corpus (via httpx) Key differences from OpenAI: - Files must be uploaded to GCS first (via litellm.files.acreate_file) - Embedding is handled internally using text-embedding-005 by default - Chunking configured via unified chunking_strategy in ingest_options ) annotationsN) TYPE_CHECKINGAnyFinal)get_secret_str)verbose_logger)VertexAIRAGTransformation)BaseRAGIngestion)Router)RAGIngestOptionsc | t|SdS)z#Cast config value to Optional[str].N)str)values p/home/oem/CyberBrain26-unified/.venv/lib/python3.12/site-packages/litellm/llms/vertex_ai/rag_engine/ingestion.py_get_str_or_noners*3u:44c ||St|S)z&Cast config value to int with default.)int)rdefaults r_get_intr"s } u:rceZdZdZ d d fd Zd dZ d dZ d dZddZ ddZ ddZ xZ S)VertexAIRAGIngestionaq Vertex AI RAG Engine ingestion. Uses litellm.files.acreate_file for GCS upload, then imports into RAG corpus. Required config in vector_store: - vector_store_id: RAG corpus ID (required) Optional config in vector_store: - vertex_project: GCP project ID (uses env VERTEXAI_PROJECT if not set) - vertex_location: GCP region (default: us-central1) - vertex_credentials: Path to credentials JSON (uses ADC if not set) - wait_for_import: Wait for import to complete (default: True) - import_timeout: Timeout in seconds (default: 600) Chunking is configured via ingest_options["chunking_strategy"]: - chunk_size: Maximum size of chunks (default: 1000) - chunk_overlap: Overlap between chunks (default: 200) Authentication: - Uses Application Default Credentials (ADC) - Run: gcloud auth application-default login ct||||jjd|_|js t d|jjdxs t d|_|jjdxst dxsd|_|jjd |_ |jjd xstjjd |_ |js t d |jjd d|_ t|jjdd|_|js t dy)N)ingest_optionsroutervector_store_idznvector_store_id (corpus ID) is required for Vertex AI RAG ingestion. Please provide an existing RAG corpus ID.vertex_projectVERTEXAI_PROJECTvertex_locationVERTEXAI_LOCATIONz us-central1vertex_credentials gcs_bucketGCS_BUCKET_NAMEzkgcs_bucket is required for Vertex AI RAG ingestion. Set via vector_store config or GCS_BUCKET_NAME env var.wait_for_importTimport_timeoutiXzpvertex_project is required for Vertex AI RAG ingestion. Set via vector_store config or VERTEXAI_PROJECT env var.)super__init__vector_store_configget corpus_id ValueErrorrrrr!osenvironr"r$rr%)selfrr __class__s rr'zVertexAIRAGIngestion.__init__Bsg vF11556GH~~<  #66::;KLrP^_qPr  $ $ ( (): ; s~Na?b sfs #'":":">">?S"T2266|Di WhHiJ  $77;;=1.60.0. Install with: pip install 'google-cloud-aiplatform>=1.60.0')projectlocationzImporting %s into corpus %s) corpus_namepathstransformation_configtimeoutz"Import complete: %s files imported)rLrMrNzImport started asynchronouslyN)vertexairFrH ImportErrorrr_build_transformation_configr1rr9r*r$ import_filesr%r<imported_rag_files_countimport_files_async)r.rC vertexai_initrHrNrLrB_s r_import_file_to_corpus_via_sdkz3VertexAIRAGIngestion._import_file_to_corpus_via_sdks  6 $ d11D|L ./@A/R ''..%+/(  ) @  s A66B c Kyw)z Vertex AI handles embedding internally - skip this step. Returns: None (Vertex AI embeds when files are imported) N)r.chunkss rembedzVertexAIRAGIngestion.embeds scK|r|s,tjdt|jdfS|j |||xsdd{} |j |d{t|j|fS767#t $r*}tjd|td||d}~wwxYww)a) Store content in Vertex AI RAG corpus. Vertex AI workflow: 1. Upload file to GCS via litellm.files.acreate_file 2. Import file into RAG corpus via SDK 3. (Optional) Wait for import to complete Args: file_content: Raw file bytes filename: Name of the file content_type: MIME type chunks: Ignored - Vertex AI handles chunking embeddings: Ignored - Vertex AI handles embedding Returns: Tuple of (corpus_id, gcs_uri) zr?)rCz)Failed to import file into RAG corpus: %sz'Failed to import file into RAG corpus: ) rwarningrr*rDrX Exceptionerror RuntimeErrorr)r.r=r>r?rh embeddingsrCes rstorezVertexAIRAGIngestion.stores48  " "#a b#DNN3T9 9 $77%%C)C 8   U55g5F F F 4>>"G++  G U  !La P!HLMST T UsHA B?BB?B )B*B .B?B B<%B77B<<B?)N)rr rz Router | None)returnr)r=bytesr>rr?rrrr)rCrrrNone)rrr)rh list[str]rrlist[list[float]] | None) r=z bytes | Noner> str | Noner?rwrhrurorvrrztuple[str | None, str | None]) __name__ __module__ __qualname____doc__r'r1rDrXrRrirq __classcell__)r/s@rrr)s6!%'(''Rn020202 02 02d-A-A -A^! F   " ,,",,,,! ,,  ,, - ,, ',,rr)rrrrrw)rrrrrrr)r{ __future__rr,r_rrrr8rlitellm._loggingr0litellm.llms.vertex_ai.rag_engine.transformationr $litellm.rag.ingestion.base_ingestionr r r`r rrrrgrrrsF # ,,"+VA25 B,+B,r