,jr-dZddlmZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z ejeZdZed Gd d ZGd dZej(dZej(dZej(dZej(dZej(dej2Zej(dej2ZddZddZy)uLLM-driven document classifier for auto-categorization. Provides structured JSON classification of documents via Ollama /api/chat. All prompts enforce strict JSON output with temperature=0 for determinism. ClassificationResult has all fields mandatory — never empty/None. Research references: - ByteRover (arXiv:2604.01599) — hierarchical Context Tree, knowledge in markdown - Memex (jagdeepvirdi/memex) — Ollama auto-classify, confidence scores, category review - secondmem — AI classification into topic markdown files with JSON schema - Grimore-MD — two-tier change detection, RRF fusion - RAGraph — entity/topic resolution, graph-based organization ) annotationsN) dataclass)Any)HTTPStatusError)EmbeddingConfiguMYou are classifying a technical document into a knowledge hierarchy. Return ONLY a valid JSON object matching this exact schema — no prose, no markdown, no extra keys. { "topic": "ONE of: ai, architecture, security, research, notes, reports, general", "subtopic": "short free-text domain within the topic (e.g. 'audit', 'design', 'networking')", "tags": ["TAG1", "TAG2", ...], // 2-5 tags max, lowercase, relevant to content "summary": "2-3 sentence summary of the document's core content", "confidence": "FLOAT string 0.0-1.0 (e.g. "0.87") — your confidence in this classification", "entities": ["ENTITY1", "ENTITY2", ...], // 0-4 named entities extracted from text "intent": "ONE of: actionable, reference, idea" } Temperature must be 0 for deterministic output. Do NOT invent tags or entities not supported by the text. T)frozencXeZdZUdZded<ded<ded<ded<ded <ded <ded <y ) ClassificationResultuResult of LLM-driven document classification. All fields are mandatory — never None/empty. If the LLM output is malformed, the caller must handle the exception rather than producing a partial result. strtopicsubtopicz list[str]tagssummaryfloat confidenceentitiesintentN)__name__ __module__ __qualname____doc____annotations__M/home/oem/CyberBrain26-unified/src/cyberbrain26/knowledge_plane/classifier.pyr r 6s. JM O L Krr cheZdZdZd d dZedZddd ddZddZdd Z dd Z dd Z y) LLMClassifieraClassify documents using a local Ollama model. Uses Ollama /api/chat endpoint with strict JSON prompt and temperature=0 for deterministic output. All fields in ClassificationResult are mandatory. Research-backed design: - ByteRover: hierarchical memory via agent-native classification - Memex: confidence-scored auto-classification, categorical review queue - secondmem: AI classification into topic markdown files Nc8|xs t|_d|_yN)rconfig_client)selfr s r__init__zLLMClassifier.__init__Zs1 1  rc|j5tdj|jjd|_|jS)z4Lazily create httpx client for Ollama chat endpoint.httpxgN@)base_urltimeout)r! __import__Clientr ollama_base_urlr"s rclientzLLMClassifier.client^sF << %g.55446DL||r temperaturenum_ctxc|jjdtd|dg||xsdddd} |jj d| }|j |j }|jd ijd d }t j|}t|tstd t||S#tt jtf$r} tj!d| d} ~ wwxYw)z6Call Ollama /api/chat and return parsed JSON response.userz Document text: )rolecontenti r.json)modelmessagesoptionsformatz /api/chat)r5messager4z{}zExpected dict from LLM, got zLLM classification failed: %sN)r chat_modelCLASSIFY_PROMPTr,postraise_for_statusr5getloads isinstancedict ValueErrortyperJSONDecodeErrorloggererror) r"textr/r0payloadresponsedatar4parsedexcs r _call_ollamazLLMClassifier._call_ollamahs [[++#"1!22FtfM (3w$O #  {{'' ''BH  % % '==?Dhhy"-11)TBGZZ(Ffd+ #?V ~!NOOM!5!5zB  LL8# >  sBCD,DDc d|j|}hd}|t|jz }|rtd|t |dt rt d|dDstdt|dt |dt rt d|dDstdt|d|d td Dcgc]}t|d z c}d d gzvr+ t|d }d|cxkrdksntd| dddd}|dj}||vr,tdt |jd|dd||} hd} |dj} | | vrtjd| d} n| } |dr|dnd} |ddd }t|d!kr|d"d#gdd!t|z z}|ddd$}t|d }t!| | ||d%||| &Scc}w#ttf$rtd|d ddwxYw)'uVRun LLM classification on document text. Returns ClassificationResult with all fields mandatory. If the LLM fails (network error, malformed output), raises RuntimeError — caller must decide whether to retry, stage for review, or assign to fallback category. Design rationale (per Memex and ByteRover research): - temperature=0 ensures deterministic output across runs - format=json forces Ollama to return structured data - all fields mandatory — no empty/None values in result - 2-5 tags max prevents tag spray >rr rrrr rzLLM output missing keys: rc3<K|]}t|tywrrAr ).0ts r z2LLMClassifier.classify_document..s7`q 1c8J7`ztags must be list of str, got rc3<K|]}t|tywrrQ)rRes rrTz2LLMClassifier.classify_document..s;hSTJq#ainotesgeneralreportsresearchsecurity architecturer z.Topic '%s' not in allowed set, using 'general'r`r  knowledger^r)r r rrrrr)rNsetkeysrCrAlistallrDranger r TypeErrorlowerrFwarninglenr )r"rHraw required_keysmissingiconfidence_val intent_map intent_rawrallowed_topics topic_lowerr r rrs rclassify_documentzLLMClassifier.classify_documents%e #chhj/1 8 BC C#f+t,C7`TWX^T_7`4`=d3v;>O=PQR R#j/40;hX[\fXg;h8hA$s:BWAXYZ Z | s$DAS3Z$Du~$U U !&s<'8!9n33$'GGW%XYY4%1{TZ[ ]((* Z '5d:??;L6M5NgVYZbVcUddefg gJ'g'l((* n , NNK[ YEE'**o3z?96{2A t9q=;-oCI >>Dz?2A&s<01# N%  S%E  *  FsK`K`akKl$$Z0rs #  !!I   8 $ * $,QQr BJJH I BJJF G "**|} "**+ ,bjjLbmm\ dfhfsfstJ$r