j" U d Z ddlmZ ddlZddlZddlZddlZddlZddlZddl m
Z
ddl m Z ej e ZdZded< d Zded
< d Zded < d
Zded< dZded< dZded< dZded< dZded< dZded< g dZd&d Zd'd(d Z d)d Z d*d Z d)d Z d+d!Z d'd,d"Z! d' d-d#Z" d' d,d$Z#def d.d%Z$y)/ac Resolution of the unified ingest shared root and batch claiming.
Discovers and initializes a single shared folder used across devices
(Debian Mentem, virtual machine with CyberBrain26, Android phone, tablet
with local model container). The orchestrator claims files from inbox/
into processing/ batches and records everything in an append-only manifest.
) annotationsN)Path)uuid4UNIFIED_INGEST_ROOTstrENV_SHARED_ROOTz~/UnifiedIngestDEFAULT_SHARED_ROOTz ~/.config/syncthing/config.xmlSYNCTHING_CONFIGz ~/.config/rclone/rclone.conf
RCLONE_CONFIG)inbox
processingsorted
duplicatesindextasksmanifestztuple[str, ...]INGEST_SUBDIRSzingest-manifest.jsonlMANIFEST_FILENAME
intBATCH_LIMIT_MAX) z.ssh/z.gnupg/z.password-store/z auth.jsonzcredentials.jsonsecrettokencredpasswordSENSITIVE_PATTERNS)z.keyz.pemz.kdbxSENSITIVE_EXTENSIONS)r r r r r r r r r
claim_batchdetect_syncthing_foldersensure_shared_rootis_rclone_configuredis_sensitive_pathlist_mountpoints
manifest_pathmanifest_writeresolve_shared_rootc t t j j t j j t | S N)r ospath
expandvars
expanduserr )r* s F/home/oem/CyberBrain26-unified/src/cyberbrain26/product/shared_root.py_expandr. G s/ ""277#5#5c$i#@A BB c | t | j S t j j t
}|r7|j
r't |j
j S t D ] }|c S dD ]/ }t | j }|j s-|c S t t j S )zResolve the shared ingest root.
Precedence: explicit argument > env UNIFIED_INGEST_ROOT >
existing Syncthing folder > ~/Sync/ > ~/Documents/ > ~/Downloads/ >
fallback ~/UnifiedIngest/. Does NOT create the directory.
)z~/Sync/z ~/Documents/z ~/Downloads/)
r. resolver) environgetr stripr existsr )explicit env_valuefolder candidateps r- r& r& K s x ((**
/IY__&y()1133*,
@ I & & ( 88:H & ' / / 11r/ c | t t } | j sg S | j dd }g }t j d| D ]Z }t |j d j j }|j sJ|j | \ |S # t $ r g cY S w xY w)zEReturn paths of configured Syncthing folders (config existence only).utf-8replaceencodingerrorsz([^<]+) ) r. r
r5 read_textOSErrorrefinditerr groupr, r1 append)cfgtextfoldersmatchr: s r- r r ` s
"
#C ::< }}gi}@ G 4d; Q + + - 5 5 7 88: NN1 N s B- - B;:B;c < t t j S )z>Check whether rclone config exists (never runs rclone config).)r. r r5 r/ r- r! r! q s = ! ( ( **r/ c g } t ddd 5 }|D ]Q }|j }t | dk\ s"t |d }|j sA| j | S ddd dD ]U }t | }|j s |j D ]$ }|j s| j | & W | S # 1 sw Y exY w# t $ r Y sw xY w) z9List candidate mount points (/proc/mounts, /mnt, /media).z /proc/mountsr< r= r> rA N)z/mntz/media)
opensplitlenr is_absoluterG rC r5 iterdiris_dir)mountsflinepartsr: basebpchilds r- r# r# v s F
.79
E )
)
u:? U1XA}}
a(
) ) # )
$Z
99;
)<<>MM%(
)) M ) )
s3 C $C CC*C CC C&%C&c t | j }t D ] }||v s y t D ] }|j | s y y)z@Return True if path contains sensitive credentials/keys/secrets.TF)r lowerr r endswith)r* rI patexts r- r" r" sP t9?? D! $; $ == r/ c | r't | j j n t }|dz t z S )z)Return the path to ingest-manifest.jsonl.r )r r, r1 r& r ) shared_rootroots r- r$ r$ s7 7B4 ' ' ) 1 1 3H[H]D * 0 00r/ c 8 t | }|j j ddd t j | ddd }t |dd 5 }|j
|d
z d d d |j d y # 1 sw Y xY w# t $ r Y y w xY w)
zFAppend one JSON line to the manifest (mode 0o600). All paths relative.T parentsexist_okmode),:F)
separators ensure_ascii allow_nanar< )r?
N ) r$ parentmkdirjsondumpsrP writechmodrC )entryrc mpathrX rW s r- r% r% s
+ &E LLtd? ::e
RW XD
eS7 + q t
E
s B/B
B
BBc | r't | j j n t }|j }|j ddd |s |j
d t D ] }||z j ddd t | }|j s|j d |S # t $ r Y \w xY w)zBCreate the shared root and all ingest subdirectories (mode 0o700).Trf rg rr )rj ) r r, r1 r& r5 rt rx rC r r$ touch)rc rd existedsubrz s r- r r s 8C4 ' ' ) 1 1 3H[H]DkkmGJJtdJ7 JJu D
4$UCD
$ E <<>
K s B= = C C c | r't | j j n t }|dz }dt j
dd }|dz |z }|j
ddd g }g }d }|j rt |j D ] } ||k\ r n| j s t | r|j t | j | t t j
|t! j t | j | dd| j# j$ | j& j) d
d
d d dd d
| t+ j, | j/ j1 }
|| j2 z } | j5 | |j t | j | t t j
|t! j t | j | t | j | |
| j# j$ | j& j) ddd ddd d
| |dz
} ||||dS # t6 $ r"} t8 j; d| | Y d} ~ *d} ~ ww xY w)zClaim up to `limit` files from inbox/ into processing//.
Returns a dict with batch_id, claimed (list of relative paths),
skipped (list of sensitive paths), and manifest entries written.
r zb-N r
Trf rg r skipped orchestratorz
[REDACTED])idbatch_idts src_path_rel dst_path_relsha256size mime_or_exttypestatusworkerreason
dup_of_sha256 task_hits)rc otherclaimedrA zCould not claim %s: %s)r r r count) r r, r1 r& r hexrt r5 r rT is_filer" rG r relative_tor% timestatst_sizesuffixr^ hashlibr
read_bytes hexdigestnamer= rC loggerwarning)
rc limitrd r r r
r r r ry r dstexcs
r- r r sy 8C4 ' ' ) 1 1 3H[H]D 7NEEGKKO$%H $x/JTDu=GG
E ||~EMMO, 5 EE~==? 's5#4#4T#: ;< #gkk$, "iik(+E,=,=d,C(D(,"& %
4 4',||'9'9'; )"+"0".)-%& !%#& ^^E$4$4$67AACF uzz)C
E
c"s3??4#8 9: #gkk$, "iik(+E,=,=d,C(D(+COOD,A(B"( #
2 2'*zz'7'7'9 '"+"0"&)-%& !%#&
g5 Ep
E 7DD
Es :C J## K,K K)r*
str | Pathreturnr r( )r6 str | Path | Noner r )r z
list[Path])r bool)r* r r r )rc r r r )ry dictrc r r None)rc r r r r r )%__doc__
__future__r r ru loggingr) rD r pathlibr uuidr getLogger__name__r r __annotations__r r
r r r r r r __all__r. r& r r! r# r" r$ r% r r rM r/ r- r s9 # 8 $, ,, S ,8 # 83
s 3# 1 3 0 ' O )B o A ,C2*"+
, 1 &*
"
" &*" , &* P"P P
Pr/