&j.dZddlmZddlZddlmcmZddl Z ddl m Z ddl Z ddl mZmZddlmZmZmZddlmZddlmZdd lmZdd lmZdd lmZe j<dd Ze j<dd Z e j<ddZ!e j<ddZ"GddZ#GddZ$GddZ%GddZ&GddZ'e(dk(re)e jTe+gy)uBenchmarky knowledge plane (Fáze C4). Připravuje benchmark suite postavenou na pytest-benchmark pro: 1. Performance microbenchmarks (embedding latency, search latency, ingestion throughput) 2. Retrieval quality metrics (precision@k, recall@k, MRR, NDCG@k) z BEIR/ragcheck formátu 3. Knowledge plane specifické metriky (entity extraction, temporal validity, graph traversal) Vzory z výzkumu: - pytest-benchmark (BSD, wdm0006/python-skills) — `--benchmark-save=main`, `--benchmark-compare`, `--benchmark-compare-fail=median:10%` - BEIR (MIT) — retrieval metriky: nDCG@k, MAP@k, Recall@k, Precision@k, MRR@k - ragcheck (MIT) — offline deterministic retrieval harness s regresními diffy - LongMemEval (MIT) / LoCoMo (CC BY-NC) — pro temporal/multi-session memory metriky - PrecisionMemBench (tenurehq) — izolovaná retrieval precision metrika bez LLM-as-judge CI integrace: - Baseline save na main: `pytest --benchmark-save=main` - CI regression guard: `pytest --benchmark-compare=main --benchmark-compare-fail=median:10%` - Retrieval quality: regresní diff s `--threshold recall@5=0.01` Poznámka: LLM-as-a-judge metriky (RAGAS/DeepEval) jsou vhodné jen jako volitelná nadřazená vrstva, ne jako součást CI suite (ne-deterministické, vyžaduje API klíč). ) annotationsN)Any)BatchProcessorEmbeddingRetryPolicy)DocumentFingerprintEmbeddingCacheSearchResultCache)KnowledgeDocumentEmbeddingConfigEmbeddingServiceKnowledgeNavigatorKnowledgeVectorStorec"td|dz S)Nlancedb) vector_dimlancedb_directoryr )tmp_paths @/home/oem/CyberBrain26-unified/tests/test_knowledge_benchmark.pyconfigr4s dh>R SSct|SNrr rs rembedding_servicer9s 6 **rct|Srrrs r vector_storer!>s v ..rctSNrrr navigatorr%Cs  rc eZdZdZddZddZy)TestEmbeddingPerformancez&Benchmark: embedding generace latency.c|jd}d}||u}|stjd|fd||fdtjvstj |rtj |ndtj |dz}dd|iz}ttj|dx}}t|}d }||k(}|stjd |fd ||fd tjvstj trtj tnd dtjvstj |rtj |ndtj |tj |d z} dd| iz} ttj| dx}x}}y)z(Benchmark single-text embedding latency.zTest document for benchmarkingNis notz%(py0)s is not %(py3)sresultpy0py3assert %(py5)spy5r==z0%(py3)s {%(py3)s = %(py0)s(%(py1)s) } == %(py6)slenr.py1r/py6assert %(py8)spy8) embed_text @pytest_ar_call_reprcompare @py_builtinslocals_should_repr_global_name _safereprAssertionError_format_explanationr5) selfrr, @py_assert2 @py_assert1 @py_format4 @py_format6 @py_assert5 @py_assert4 @py_format7 @py_format9s rtest_embedding_latency_singlez6TestEmbeddingPerformance.test_embedding_latency_singlePs"--.NO!!vT!!!!vT!!!!!!v!!!v!!!T!!!!!!!6{"d"{d""""{d""""""s"""s""""""6"""6"""{"""d"""""""rctdDcgc]}d| }}tj}|D]}|j|tj|z }tj}|j |tj|z }|dkDr||z nd} d} | | kD} | st j d| fd| | fdtjvst j| rt j| ndt j| d z} d d | iz} tt j| d x} } y cc}w) uBenchmark batch vs single embedding. Research: batch size 32 je sweet spot (ishwarj.com). Single embedding je ~32× slower than batch for same tokens. zBenchmark document r?g?>)z%(py0)s > %(py3)sspeedupr-r0r1N) rangetime monotonicr; embed_textsr<r=r>r?r@rArBrC)rDritexts single_startt single_time batch_start batch_timerSrErFrGrHs rtest_embedding_batch_vs_singlez7TestEmbeddingPerformance.test_embedding_batch_vs_singleVs 5:"I>q&qc*>>~~'  ,A  ( ( + ,nn&5 nn& %%e,^^% 3 .81n+ *#w}www?s ENreturnNone)__name__ __module__ __qualname____doc__rMr_r$rrr'r'Ms0# rr'ceZdZdZddZy)TestSearchPerformancez,Benchmark: search latency (vector + hybrid).cddlm}tdDcgc]#}|d|dgdzd|dd |d d |di %}}|j|dgdz}|j |d}t |}d}||kD} | st jd | fd||fdtjvst jt rt jt nddtjvst j|rt j|ndt j|t j|dz} dd| iz} tt j| dx}x} }ycc}w)z Benchmark vector search latency.r)VectorDocument zdoc-皙?rzDoc testz /test/doc-zhash-) idvectortext source_idpathmtimesha256 chunk_indexmetadatarQ)z/%(py3)s {%(py3)s = %(py0)s(%(py1)s) } > %(py6)sr5resultsr6r9r:N))cyberbrain26.knowledge_plane.vector_storerjrT add_documentssearchr5r<r=r>r?r@rArBrC) rDr!rjrXdocs query_vectorrxrErIrJrKrLs rtest_search_latencyz)TestSearchPerformance.test_search_latencyos'L2Y   !:ut|A3Z !!%qc{      ""4(ut| %%lB77|a|a|ass77|a%  s(E8Nr`)rcrdrerfr~r$rrrhrhls 6 rrhc,eZdZdZ ddZddZy)TestRetrievalQualityz@Benchmark: retrieval quality metriky (BEIR/ragcheck compatible).c2|rsddddddS|Dcgc]:}|jdxs%|jdxs|jdd<}}j|}rt|tz nd}|rt|t|z nd}d}t|dD]\} } | vs d | z }nt fd t|Dt|dz} | d kDrd | z nd} t fd |D} |||| | rd dSddScc}w) zrCompute retrieval quality metrics. Returns: precision@k, recall@k, f1@k, hit_rate@k, mrr, ndcg@k rn) precisionrecallmrrndcghit_ratero document_idrs)startrPc38K|]\}}|vs |dzyw)rNr$).0rXdoc_id relevant_idss r z8TestRetrievalQuality._compute_metrics..s Qyq&&L:PQU Qs  rc3&K|]}|v ywr#r$)rrrs rrz8TestRetrievalQuality._compute_metrics..s?V&L(?s)get intersectionr5 enumeratenextany)rDrxrrdoc_idsrrrrrankrfirst_relevant_rankrhits ` r_compute_metricsz%TestRetrievalQuality._compute_metricssWl!$CZ]^ ^OV IJAEE$K D155/ D1553D D  $009 :F\"S%66C8?C %G 4S %gQ7 LD&%Dj  # QIg$6 Q L1  -@!,Cs((?w??#"   ),   / s?DcJdddddddddg}|j|ddh}|d }d }||kD}|sltjd |fd ||ftj|tj|d z}dd|iz}t tj |dx}x}}|d}d}||k(}|sltjd|fd||ftj|tj|d z}dd|iz}t tj |dx}x}}|d}d}||k(}|sltjd|fd||ftj|tj|d z}dd|iz}t tj |dx}x}}y)z;Test retrieval quality metrics computation with known data.doc-1?roscoredoc-2g?doc-3gffffff?)rrrnrQ)z%(py1)s > %(py4)sr7py4assert %(py6)sr8NrrPr2z%(py1)s == %(py4)sr)rr<r=rArBrC)rDrxmetrics @py_assert0 @py_assert3rE @py_format5rKs rtest_retrieval_quality_basicz1TestRetrievalQuality.test_retrieval_quality_basics1S )S )S )  ''w>P'Qx &3& 3&&&& 3&&& &&&3&&&&&&&z")c)"c))))"c)))")))c)))))))u~$$~$$$$~$$$~$$$$$$$$$$rN)rxzlist[dict[str, Any]]rzset[str]razdict[str, float]r`)rcrdrerfrrr$rrrrs(J' +' ;C' ' R %rrc(eZdZdZddZddZddZy)TestKnowledgePlaneMetricsu/Benchmark: knowledge plane specifické metriky.ctdddddddditjd z tjd z tjd z tjd zd }|j}||}d }||u}|stjd |fd||fdt j vstj|rtj|ndtj|dt j vstj|rtj|ndtj|tj|dz}dd|iz}ttj|dx}x}x}}tdddddddditjd z tjd z d } |j}|| }d}||u}|stjd |fd||fdt j vstj|rtj|ndtj|dt j vstj| rtj| ndtj|tj|dz}dd|iz}ttj|dx}x}x}}y)zATest temporal validity: valid docs are valid, superseded are not.z doc-validrmztest://doc-validabc123z text/plainreviewedpersonalindexedg @r)rrr source_uricontent_sha256 media_typetrust sensitivitystatusrw created_at updated_at valid_fromvalid_to superseded_byTis)z]%(py5)s {%(py5)s = %(py2)s {%(py2)s = %(py0)s.check_temporal_validity }(%(py3)s) } is %(py8)sr% doc_valid)r.py2r/r1r:zassert %(py10)spy10Nz doc-staleztest://doc-staledef456z doc-newer) rrrrrrrrrrwrrrF doc_stale) r rUcheck_temporal_validityr<r=r>r?r@rArBrC) rDr%rrFrJ @py_assert7 @py_assert6rL @py_format11rs rtest_temporal_validity_accuracyz9TestKnowledgePlaneMetrics.test_temporal_validity_accuracys&#)##"yy{V+yy{V+yy{V+YY[6) 00C0;CtC;tCCCC;tCCCCCCyCCCyCCC0CCCCCCCCCCCC;CCCtCCCCCCC&#)##"yy{V+yy{V+%  00D0;DuD;uDDDD;uDDDDDDyDDDyDDD0DDDDDDDDDDDD;DDDuDDDDDDDrc|jdddd|jdddd|jdddd |jd}d}||v}|stjd |fd ||ftj|d t j vstj|rtj|nd d z}dd|iz}ttj|dx}}d}||v}|stjd |fd ||ftj|d t j vstj|rtj|nd d z}dd|iz}ttj|dx}}t|}d}||k(}|stjd|fd||fdt j vstjtrtjtndd t j vstj|rtj|nd tj|tj|dz} dd| iz} ttj| dx}x}}|jdd} d}|| v}|stjd |fd || ftj|dt j vstj| rtj| ndd z}dd|iz}ttj|dx}}| d} t| } ddh}| |k(}|stjd|fd| |fdt j vstjtrtjtndtj| tj| tj|dz}dd|iz}ttj|dx} x} x}}y) z.Test entity graph: add edges, query neighbors.AliceBobknowsrCharlierDave works_withr)in)z%(py1)s in %(py3)s neighbors)r7r/r0r1Nr2r4r5r6r9r:r)depthrelated)z0%(py4)s {%(py4)s = %(py0)s(%(py2)s) } == %(py7)sset)r.rrpy7zassert %(py9)spy9) add_entity_edgeget_entity_neighborsr<r=rAr>r?r@rBrCr5get_related_entitiesr)rDr%rrrErGrHrIrJrKrLrrFrr @py_format8 @py_format10s rtest_entity_graph_traversalz5TestKnowledgePlaneMetrics.test_entity_graph_traversals!!'5'7C!!'9gwG!!%wG227; !u !!!!u !!!u!!!!!! !!! !!!!!!!%yI%%%%yI%%%y%%%%%%I%%%I%%%%%%%9~""~""""~""""""s"""s""""""9"""9"""~""""""""""000BqG|qGqGG1:4s:45)"44"44444"4444444s444s444:444444"44444444rc(tdD]}|jd|d|dzdd!tj}|j d}tj|z dz}d}||k}|st j d |fd ||fd tjvst j|rt j|nd t j|d z}d d|iz} tt j| dx}}y)zTest graph traversal latency.2zentity-rrdocz entity-25g@@g$@)<)z%(py0)s < %(py3)s elapsed_msr-r0r1N) rTrrUrVrr<r=r>r?r@rArBrC) rDr%rXrrrrErFrGrHs rtest_graph_traversal_speedz4TestKnowledgePlaneMetrics.test_graph_traversal_speedsr VA  % %smwqse_gu U V 22;? nn&.&8   zD    zD      z   z   D       rNr`)rcrdrerfrrrr$rrrrs9$EL 5 !rrc(eZdZdZddZddZddZy)TestCachingEffectivenessz/Benchmark: caching hit rates and effectiveness.c,t|dz }d}d}dgdz}|j||||j||d}d}||u}|stjd|fd ||fd t j vstj|rtj|nd tj|d z} d d | iz} ttj| dx}}t|}d} || k(} | stjd| fd|| fdt j vstjtrtjtndd t j vstj|rtj|nd tj|tj| dz} dd| iz}ttj|dx}x} } |j}|d}d}||k(}|sltjd|fd||ftj|tj|dz}dd|iz} ttj| dx}x}}|d}d}||k\}|sltjd|fd||ftj|tj|dz}dd|iz} ttj| dx}x}}|jy)z5Test embedding cache: second lookup should hit cache.zembedding_cache.sqlite)db_path0abc123abc123abc123abc123abc123abc123abc123abc123z test-modelrlrNr)r+r,r-r0r1r2r4r5r6r9r: total_entriesrrrrr8 total_hits)>=)z%(py1)s >= %(py4)s)rrrr<r=r>r?r@rArBrCr5statsclose)rDrcache content_hashmodelrpr,rErFrGrHrIrJrKrLrrrrs rtest_embedding_cache_hit_ratez6TestCachingEffectiveness.test_embedding_cache_hit_ratesx2J'JK#  ,v.<5!!vT!!!!vT!!!!!!v!!!v!!!T!!!!!!!6{"d"{d""""{d""""""s"""s""""""6"""6"""{"""d""""""" _%**%****%***%**********\"'a'"a''''"a'''"'''a''''''' rctdd}dddg}|jdd|d |jddd }d }||u}|stjd |fd ||fd t j vstj|rtj|nd tj|dz}dd|iz}ttj|d x}}|dd}d} || k(}|sltjd|fd|| ftj|tj| dz} dd| iz} ttj| d x}x}} |jddd } d }| |u}|stjd|fd| |fdt j vstj| rtj| ndtj|dz}dd|iz}ttj|d x}}|j} d}| |k(}|stjd|fd| |fdt j vstj| rtj| ndtj|dz}dd|iz}ttj|d x}}y )z-Test search result cache: get/set/invalidate.rkgN@)max_size ttl_secondsrrrqueryrm)topicNr)r+cachedr-r0r1rror2rrrr8zdifferent-queryr)z%(py0)s is %(py3)smissedr)z%(py0)s == %(py3)s invalidated) r rrr<r=r>r?r@rArBrC invalidate)rDrrxrrErFrGrHrrrrKrrs rtest_search_result_cachez1TestCachingEffectiveness.test_search_result_cache*s!24@!C01 '2wf 57Bf5!!vT!!!!vT!!!!!!v!!!v!!!T!!!!!!!ay)')'))))'))))))'))))))),b?v~vvv&&( {a{a{{arc d}tj|}tj|}|j}|j}||k(}|stjd|fd||fdt j vstj|rtj|ndtj|dt j vstj|rtj|ndtj|dz}dd|iz}ttj|d x}x}}|j}d }||}|j} d } | | } || k7} | s>tjd | fd || fdt j vstj|rtj|ndtj|tj|tj|dt j vstj|rtj|ndtj| tj| tj| dz} dd| iz}ttj|d x}x}x}x} x} x} } y )z1Test document fingerprint: stable hash for dedup.s Hello, World!r2)zZ%(py2)s {%(py2)s = %(py0)s.content_sha256 } == %(py6)s {%(py6)s = %(py4)s.content_sha256 }fp1fp2)r.rrr8r9r:Nzmodel-v1zmodel-v2)!=)z%(py6)s {%(py6)s = %(py2)s {%(py2)s = %(py0)s.cache_key }(%(py4)s) } != %(py14)s {%(py14)s = %(py10)s {%(py10)s = %(py8)s.cache_key }(%(py12)s) })r.rrr8r:rpy12py14zassert %(py16)spy16) r from_contentrr<r=r>r?r@rArBrC cache_key)rDcontentrrrFrIrrKrL @py_assert9 @py_assert11 @py_assert13r @py_format15 @py_format17s rtest_document_fingerprintz2TestCachingEffectiveness.test_document_fingerprint;s"!..w7!..w7!!7S%7%77!%77777!%7777777s777s777!777777S777S777%77777777}}EZE}Z(ECMME*EM*,EE(,EEEEE(,EEEEEEEsEEEsEEE}EEEZEEE(EEEEEECEEECEEEMEEE*EEE,EEEEEEEEErNr`)rcrdrerfrrr r$rrrrs9$ "Frr__main__)rar )rar)rar)rar),rf __future__rbuiltinsr>_pytest.assertion.rewrite assertionrewriter<rUtypingrpytest,cyberbrain26.knowledge_plane.batch_processorrr$cyberbrain26.knowledge_plane.cachingrrr &cyberbrain26.knowledge_plane.contractsr -cyberbrain26.knowledge_plane.embedding_configr .cyberbrain26.knowledge_plane.embedding_servicer'cyberbrain26.knowledge_plane.navigationrryrfixturerrr!r%r'rhrrrrc SystemExitmain__file__r$rrrs.#   EIKFJTT++//  >  B6%6%|B!B!T,F,F^ z [V[[(, --r