Mj|^dZddlmZddlZddlZddlZddlmZddlm Z m Z ddl m Z ddl mZmZmZddlmZmZmZdd lmZmZmZmZmZmZmZdd lmZmZdd l m!Z!dd l"m#Z#dd l$m%Z%ddl&m'Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1m2Z2ejfe4Z5GddZ6dgZ7y)uUnified memory service for the knowledge plane. Tied together all knowledge plane components into a single callable API: file registration → chunking → embedding → vector store → navigation → semantic search → retrieval ) annotationsN)Path)Anycast)SQLiteDatabase)DocumentFingerprintEmbeddingCacheSearchResultCache)ClassificationResult LLMClassifierdetect_sensitivity_from_bytes)DocumentRegistrationDocumentStatus DocumentTrustKG1NodeKnowledgeDocumentKnowledgeSource Sensitivity)DeduplicationEngineDeduplicationResult)EmbeddingConfig)EmbeddingService)LocalDirectorySource) MarkdownStore)KnowledgeMetrics)KnowledgeNavigator)SQLiteKnowledgeRepository)KnowledgeVectorStore)AuditEventTypeAuditLogc TeZdZdZdddddddddd d%dZ d&dZdddddd d'dZd dddeje jd  d(d Z d ddd  d)dZ e d* d+dZd d d,dZd ddd  d)dZd ddd d-dZd.dZd/dZd0dZd1dZd1dZ d2dZd3dZdd d4dZd5dZdddd  d6d!Zddd" d7d#Zd1d$Zy)8 MemoryServiceuJednotný vstupní bod pro paměťovou vrstvu knowledge plane. Spolupracuje: - SQLiteKnowledgeRepository pro metadata dokumentů - LocalDirectorySource pro čtení lokálních souborů - TextChunker pro dělení textu na chunky - EmbeddingService pro generování vektorů přes Ollama - KnowledgeVectorStore (LanceDB) pro vektorové uložiště a sémantické vyhledávání - KnowledgeNavigator pro topic/tag navigaci - DeduplicationEngine pro detekci duplicit NT) audit_logconfig repository vector_storeembedding_service navigatordeduplication_engine markdown_root auto_classifyc ||_||_|xs t|_|xs t |||_|xst |j|_|xst|j|_ |xs t|_ |xs!t|j|j|_ tdtdj!t"j$t&j(|_| |_t/|j|_t3| xs,t|jj4j6dz | |_t|jj4j6d z } t;t=|  |_tAd d |_!tE|_#y)N)r#)r$)r'r&local. source_idroottrust sensitivityknowledge_docs)r1r+embedding_cache)db_pathdgr@)max_size ttl_seconds)$databaser#rr$rr%rr&rr'rr(rr)rrresolverREVIEWEDrPERSONAL local_sourcer+r classifierrlancedb_directoryparentmarkdown_storer strr5r search_cachermetrics) selfr:r#r$r%r&r'r(r)r*r+ cache_dirs Q/home/oem/CyberBrain26-unified/src/cyberbrain26/knowledge_plane/memory_service.py__init__zMemoryService.__init__Fsq! "1 1 $`(A(V_(`(T,@ ,T!2!Z6Fdkk6Z":&8&:  ""&"8"8!.. !1c""$((#,,  +'t{{;,_$t{{'D'D"E"L"LO_"_' 667>>ARR -c)nE.sN() c|jsy |jdd}|jj|S#t$r!}t j d||Yd}~yd}~wwxYw)uRun LLM classification on document content. Returns ClassificationResult or None if classification is disabled or fails. Fail-closed: never returns a partial result — caller must handle the None case by storing document as unclassified. Nutf-8replaceerrorsz Classification failed for %s: %s)r+decoder?classify_document Exceptionloggerwarning)rFcontent source_urir0textexcs rH_classify_documentz MemoryService._classify_document~s`!! >>')>|j6d<| j@|j6d<| jB|j6d<| jD|j6d<ddl#}|jF}ddl$m%}|||}|r|jL|jLjOtPjRd|jT|jT|jV|jX|jZj|j\j|j^j| dud|ja|||fS#t $rt j}YtwxYw#t $rYwxYw#t$$r"} tjd|| Yd} ~ Cd} ~ wwxYw) uZaregistruje lokální soubor do paměťového systému. Postup: 1. Čte soubor z disku 2. (NEW) Klasifikuje obsah přes LLM (pokud auto_classify=True) 3. Ukládá klasifikaci do Markdown store 4. Rozdělí na chunky 5. Vygeneruje embeddingy 6. Uloží do vektorového úložiště 7. Proindexuje do navigátoru (topics, tags) 8. Zaznamená do auditu T)strictr)publicpersonal sensitive restrictedNz>Auto-upgrading sensitivity for %s from %s to %s (PII detected)r-rLrMrN)classificationrUrVr0filenamez Markdown store failed for %s: %szlocal-r/r&r'auto_classifiedclassification_topicclassification_subtopicclassification_tagsclassification_summaryclassification_confidenceclassification_intentclassification_entitiesrM) valid_frommemory_file_registered)r0rV trust_levelr3statusrgactionresource memory_writemetadata)1r expanduserr; read_bytesr r ValueErrorr=getvaluerSrTnamerYrCrBstore_classifiedrPrRrrr< register_filer%r&r'rArxtopicsubtopictagssummary confidenceintententitiestime dataclassesrMr#recordrMEMORY_UPDATED document_idr0rVr2r3rs_index_in_navigator)rFpathr0r1r2r3rZ file_pathrUdetected_sensitivitysensitivity_rankrdrX root_pathsourcedocumentcreatedr current_timerMs rHrzMemoryService.register_filesD,J))+3343@ &&( =WE     3)*>?  ##$8!>#7#7#+#7#7#=#=&oo33'5T'A  "    *  A 3)22  3" SA9cRR Ss= P7 P:%AQ P76P7: QQ Q5Q00Q5z local-text)r0rrrr2r3c t||||d} t| |d|xsddj|xsgdj|xsgd} |jj | |j |j \} } | rf|jZ|jjtjd | j| j| j|||xsg|xsgd  |j| |||| | fS) u@Zaregistruje čistý text (v paměti) do paměťového systému.T)r0rVr2r3approvedz text/plain,rrr)rrU media_typerxrfmemory_text_registered)r0rVrrrrt)rrjoinr%registerr&r'r#rrrrr0r) rFrUrVr0rrrr2r3r registrationrrs rH register_textzMemoryService.register_texts!!#  ,#${CHHTZR4HVYV^V^_g_mkmVno  !OO44 **"445 ' t~~1 NN ! !--/!--%11!)!3!3"," JB (B  "    TH U  rJ limitrtagcddl}|j}|jj||||}|du}|jj ||G|jj |j|z |r|djdnd|Stj|j} |jj| |jj|jj} | |j} |jj|} |jj!|j| z d|jj#| |jj| n|jj!dd |j} |j$j'| | } |j| z }|s|r8|j)||}|r#| Dcgc]}|jd |vs|} }|jj#||| ||| r| djdnd}|jj ||| Scc}w) uVSémantické vyhledávání v paměťovém systému. Postup: 1. Cache lookup přes SearchResultCache (TTL-based) 2. Vytvoří embedding pro dotaz (z content-addressable cache pokud možné) 3. Hledá v LanceDB vector store 4. Volitelně filtruje podle topiku/štítku přes navigátor rNrrscore)latency_seconds best_distanceF)rcachedTrid)r monotonicrDr|rErecord_search_cache_hitrecord_vector_searchr from_contentcontent_sha256r5r$model vector_dimr' embed_textrecord_embedding_latencysetr&search_get_candidate_ids)rF query_textrrrr start_timer cache_hit query_hash query_vector embed_start search_startresultssearch_latency candidate_idsrrs rHsemantic_searchzMemoryService.semantic_searchNs_ #T^^% ""&&z53&O$&  ,,Y7   LL - - . 0: =8>fQimmG4D . M)55jAPP ++// DKK 2   $ $Z1B1BL Q LL 1 1 # 2  &t~~' ##**z)MemoryService._rrf_fuse..sl6F6FsA6NrJTkeyreverseNgrr rrf_score) enumerater|rkeysfloatsortedappend) results_a results_bkrank_airdoc_idrank_ball_idsrank fused_results best_result best_scoreridrs @rH _rrf_fusezMemoryService._rrf_fuses"$iq1 #DAqUU4[IAEE-$8IAEE&MF&.!"v # "$iq1 #DAqUU4[IAEE-$8IAEE&MF&.!"v #   .V[[]1CC*,  MF::feEl3fjju6VVD/3uU|/C1t8L  M /1 W*NX\] 2F15KJ* 3eeDkJQUU=%9JQUU6]&=QUU7A%6%C"#K!"w!2J  3 &+7+? K($$[1 2rJrc $|j|jj5}|jdddddfj }dddg}D]}|drt j |dni}|jdxsdj}|jdxsgD cgc]} | j} } |jdxsdj} d } d } |vrd } | d z } tfd | Drd } | dz } | vrd } | dz } | s|j|d|d|d|jdd| d|jdd |d|S#1swY+xYwcc} w)uKeyword search over document metadata (topic, tags, summary). Searches across topic, tags, and summary fields in the SQLite knowledge_documents table metadata_json using LIKE patterns. This provides keyword-based retrieval complementary to semantic (vector) search. Research reference: RAGraph — hybrid BM25+semantic retrieval with RRF fusion. z SELECT document_id, source_uri, media_type, metadata_json FROM knowledge_documents WHERE LOWER(source_uri) LIKE ? OR LOWER(metadata_json) LIKE ? %N metadata_jsonrrrrF?Trc3&K|]}|v yw)N).0t query_lowers rH z/MemoryService.keyword_search..s2;!#2sr]rrVr)rrVrrWrc&|jddS)Nrrr)rs rHrz.MemoryService.keyword_search..s155!#4rJr) lowerr: connectionexecutefetchalljsonloadsr|anyrsort)rFqueryrrrowsrrowrxrrrrmatchedrrs @rHkeyword_searchzMemoryService.keyword_searchskkm ]] % % ' :%% [M#q Q%78hj  )+ C;>;Otzz#o"67UWH\\'*0b779E(0 V(<(BD!AGGIDDD||I.4";;=GGEe# 2T22 g% !-0&),&7&),&7 ( Y ;!& ) >  4d CvY  Es+FF F c|j||dz}|j||dz||}|j||d}|d|S)uHybrid search: keyword (metadata) + semantic (vector) + RRF fusion. Combines keyword search over document metadata (topic, tags, summary) with semantic (vector) search via Ollama embeddings, then fuses the two result lists using Reciprocal Rank Fusion (RRF) per RAGraph research. Postup: 1. Keyword search over topic/tags/summary → results_a 2. Semantic search via Ollama embeddings → results_b 3. RRF fusion → ranked fused results with rrf_score Research reference: RAGraph (arXiv:2408.03245) — hybrid BM25+vector retrieval with reciprocal rank fusion for improved precision. r^rr<)rN)rrr)rFrrrrkeyword_resultssemantic_resultsfuseds rH hybrid_searchzMemoryService.hybrid_search se.--j -J // eaiu#0  0@BGVe}rJr^)rdepthrelationship_typec|j}tjjjj }g}|D]&}||jvs|j |(|sgSt} i|D]} | | vr| j| d| <jj| ||} | jD]J\} } | D]@}|| vs| j|ttj|d| |<BLi}jD]G\}} jj|}|D]"dd| zz }|jd|z|<$Ig}t|jddD]\}j}||j |j |j"|j$jd d |tfd jjj&D| Dcgc]!}jj|vr|#c}d |d |Scc}w)uGraph-based retrieval via entity relationship traversal (MemGraphRAG). Performs breadth-first traversal over the entity relationship graph starting from the query entity, collecting all documents reachable within `depth` hops. Returns documents sorted by graph proximity. Research reference: MemGraphRAG — memory-aware hierarchical retrieval through entity co-occurrence graphs. Postup: 1. Najde entity v navigátoru podle entity_query 2. Provede BFS do hloubky `depth` přes KnowledgeNavigator.get_related_entities 3. Sbírá všechny dokumenty spojené s nalezenými entity 4. Řadí dokumenty podle vzdálenosti v grafu (blíž = relevantnější) r)rrrrc |dS)Nr]r)items rHrz,MemoryService.graph_search..qs d1grJTrNrrc3K|]P}jjjj|t vrj|dRyw)rN)r( _entity_graphentity_documentsr|r)rerentity_distancesrFs rHrz-MemoryService.graph_search..|sP. !%)E)E)V)V)Z)Z[\^a^c)dd-00C8.sAA)rrVrrWrgraph_distancematched_entities)rrr(r rrrraddget_related_entitiesitemsintminr|get_entity_documents get_documentrVrrxr )rF entity_queryrrrr all_entitiesstarting_entitiesentityvisited_entities start_entityrelateddistance entity_listdocument_scoresdoc_ids contributionrrrrrrs` @@rH graph_searchzMemoryService.graph_search+s0#((* dnn::CCHHJK (*" 1Fflln,!((0 1!I&)U+-- L//   .-. \ *nn99E=N:G*1 %+)F%55(,,V436s,00=x84(0  $-/ 0 6 6 8 Z FHnn99&AG! Z #cHn5 *9*=*=fc*J\*Y'  Z Z)+#  ! ! #)=t  MFE((0H#$&.&9&9&.&9&9 ( 1 1 5 5i D!&*-.%)^^%A%A%R%R.+&6- !%)L)LQ)OO-  2v-s/&I%c8|jj|S)uVrátí dokument podle ID.)r%r|)rFrs rHrzMemoryService.get_documents"";//rJc8|jj|S)u%Vrátí dokumentová ID podle topiku.)r(get_documents_by_topic)rFrs rHsearch_by_topiczMemoryService.search_by_topics~~44U;;rJc8|jj|S)u'Vrátí dokumentová ID podle štítku.)r(get_documents_by_tag)rFrs rH search_by_tagzMemoryService.search_by_tags~~22377rJc6|jjS)u Vrátí všechny známé topiky.)r( get_topicsrFs rHr.zMemoryService.get_topicss~~((**rJc6|jjS)u"Vrátí všechny známé štítky.)r( get_all_tagsr/s rHget_tagszMemoryService.get_tagss~~**,,rJc|jDcgc]}|j|}}|Dcgc]}|| }}ttt|}|j j |||Scc}wcc}w)u&Zkontroluje, zda je obsah duplicitní.)_all_document_idsrrlistrr)check_duplicate)rFrUrVrexisting_documentsdcast_documentss rHr6zMemoryService.check_duplicates002    f %  *<MAq}aMM26 " #%73 ((88 Z   NsA3A8A8c|jj}||jjt |j j t |j j|jdddS)u?Rychlá kontrola stavu všech komponent paměťového systému.healthyF) embeddingvector_store_countnavigator_topicsnavigator_tagsr;) r' health_checkr&countlenr(r.r1r|)rFembedding_healths rHr@zMemoryService.health_checksp11>>@)"&"3"3"9"9"; #DNN$=$=$? @!$.."="="?@'++Iu=   rJ)auditcj|js td|jjt |j |j j|j jdd|j j|j jtj|j|j|j|jd|j t#j"t#j" |r|j$|j$j't(j*d|j |j |j|j jt-|jj/t1|jd|S) zJurassic-pattern: register an Instance Resource Graph node. Stores the node in the knowledge graph layer (navigation + audit). The node must have business_semantics (IPOMSQ) and domain_constraints. z6KG1Node requires business_semantics (IPOMSQ framework)rzknowledge/graph)kg1_typebusiness_semanticsdomain_constraintsevidence) rr0rVrrr2r3rsrx created_at updated_atmemory_kg1_registered)rFr0business_semantics_keysevidence_countrt)rGr{r( add_documentrrrr0rVr2r3rINDEXEDtyperHrIrxrr#rrrr5rrB)rFnoderDs rHregister_kg1_nodezMemoryService.register_kg1_nodesH&&UV V ##  GG++//;;11!,kk'' KK33%-- $ *.*A*A*.*A*A $  mm  99;99;#  , T^^/ NN ! !--.!WW $ !%!6!6/3D4K4K4P4P4R/S&)$--&8  "  rJcl|jj|jjy)uUzavře všechna připojení.N)r&closer'r/s rHrUzMemoryService.closes& ! $$&rJrc|s|s|r_|j}|xs|jdd|d<|r||d<|r||d<ddlm}|||}|jj |y|jj |y) u'Přidá dokument do KnowledgeNavigator.r uncategorizedrrrro)rxN)rxr|rrMr(rO)rFrrrrrxrMupdated_documents rHrz!MemoryService._index_in_navigators DH((H % Og)OHW #' '/$ +&x(C  NN ' '(8 9 NN ' ' 1rJrct}|r*|j|jj||r*|j|jj ||S)u8Získá sadu dokumentových ID pro daný topik/štítek.)rupdater(r(r+)rFrridss rHrz MemoryService._get_candidate_idssL  JJt~~<>@@G H ICL!!rJ)r:rr#zAuditLog | Noner$zEmbeddingConfig | Noner%z SQLiteKnowledgeRepository | Noner&zKnowledgeVectorStore | Noner'zEmbeddingService | Noner(zKnowledgeNavigator | Noner)zDeduplicationEngine | Noner*str | Path | Noner+boolreturnNone)rUbytesrVrCr0rCr`zClassificationResult | None)rz str | Pathr0 str | Noner1r^r2zDocumentTrust | Noner3zSensitivity | NonerZr_r`tuple[KnowledgeDocument, bool])rUrbrVrCr0rCrrcrlist[str] | Nonerrer2rr3rr`rd) rrCrrrrcrrcr`list[dict[str, Any]])r)rrfrrfrrr`rf)rrCrrr`rf) rrCrrrrrrcr`rf)rrCr`zKnowledgeDocument | None)rrCr` list[str])rrCr`rg)r`rg)rUrbrVrCr`r)r`zdict[str, Any])rRrrDr_r`r)r`ra) rrrrcrrerrer`ra)rrcrrcr`zset[str])__name__ __module__ __qualname____doc__rIrYrrr<rr=rr staticmethodrrrr%rr)r,r.r2r6r@rSrUrrr4rrJrHr"r"9s  &*)-7;4859/3;?+/"6*!6*# 6* ' 6* 5 6*26*36*-6*96*)6*6* 6*p*-:= $,!%"&&**.F!F! F!  F! $ F!(F!F! (F!Z& !%%),55#.#7#74!4! 4!  4!  4!4!#4!4!!4! (4!t  FF F  F  F FP2'2'2 2  22p << <  <D          L(, ^^ ^  ^ & ^ ^@0<8+-      $   33 3  3j'!!%%) 2#2 2  2 # 2 28!     "rJr")8rk __future__rrloggingrpathlibrtypingrr$cyberbrain26.data_plane.sqlite_storer$cyberbrain26.knowledge_plane.cachingrr r 'cyberbrain26.knowledge_plane.classifierr r r &cyberbrain26.knowledge_plane.contractsrrrrrrr*cyberbrain26.knowledge_plane.deduplicationrr-cyberbrain26.knowledge_plane.embedding_configr.cyberbrain26.knowledge_plane.embedding_servicer)cyberbrain26.knowledge_plane.local_sourcer+cyberbrain26.knowledge_plane.markdown_storer'cyberbrain26.knowledge_plane.monitoringr'cyberbrain26.knowledge_plane.navigationr'cyberbrain26.knowledge_plane.repositoryr)cyberbrain26.knowledge_plane.vector_storer!cyberbrain26.security_plane.auditrr getLoggerrhrSr"__all__rrJrHrs#  ?  JKJEDNG   8 $n "n "b  rJ