kQj+dZddlmZddlmZmZmZmZmZm Z m Z ddl Z ddl mZerddlmZe dZd Zd Zhd Zdd Zd ZdZddZddZ ddZddZGddZddZ ddZ ddZy) aWAL-PK-FUSION: hybrid search's primary-key fallback for RTE tables. Hybrid search joins its vector and full-text legs on `_rowid`, which an RTE table cannot supply: the fresh tier has no stable row id, so the server rejects `with_row_id` before it plans. When that happens the legs are joined on a surrogate `_rowid` built from the table's primary key instead. This is a stopgap until RTE supports `_rowid`, and everything it needs lives here so it can be removed in one pass: 1. Delete this file and `tests/test_wal_hybrid.py`. 2. Run `grep -rn WAL-PK-FUSION python/ rust/` and follow the note at each marker; the Rust side has its own copy in `rust/lancedb/src/query/wal_fusion.rs`. ) annotations) TYPE_CHECKING AwaitableCallableListOptionalTupleTypeVarN)blob_v2_projection_sources)QueryProjectionTs#lance-schema:unenforced-primary-keys,lance-schema:unenforced-primary-key:position>1yonyestruecg}t|D]\}}|jxsi}|jt}| t |}|>|jt }|$|jjtvrwd}|j|dk(|||jft|Dcgc]^}}| c}}S#t $rd}YwxYwcc}}w)zThe unenforced primary key columns of `schema`, in key order. Mirrors Lance's ordering: fields with an explicit 1-based position first, in position order, then fields carrying only the legacy boolean marker, in declaration order. Nr) enumeratemetadataget_PK_POSITION_MARKERint ValueError _PK_MARKERdecodelower_TRUTHYappendnamesorted) schemakeyedindexfieldrpositionlegacy_r s X/home/oem/CyberBrain26-unified/.venv/lib/python3.12/site-packages/lancedb/_wal_hybrid.py _pk_columnsr**s E!&) C u>>'R<< 34   x=  \\*-F~!6!6!8!GH h!mXuejjAB C"( /XaD //   0s C 9 C CCzdoes not support with_row_idzhybrid search on an RTE table cannot return _rowid: the fresh tier has no stable row id, and the ids the fusion joins on are synthesized from the primary key. Set use_rte(False) to read the base table only (results will exclude un-compacted RTE data)c$tt|vS)zWhether `error` is the server declining `_rowid` because the table is RTE-backed, as opposed to any other failure the query might hit.)_WAL_ROW_ID_REFUSALstr)errors r)_is_wal_row_id_refusalr/Ts #e* ,,c,|s|gfSt|ttfr$t|dttfr t|}|Dcgc] }||vs| }}t|tri||Dcic]}||c}|fSt||z|fScc}wcc}w)zAdd the key columns a projection is missing. Returns the projection to use and the columns added, which the caller drops from the output again so an injected key never reaches a caller who did not select it. r) isinstancelisttupledict)columns pk_columnscmissings r)_with_pk_columnsr:Zs {'D%=)jdE].Sw-$9Q(8q9G9'4 5'5G4qad45w>> =7 "G ++:4s B B , Bc : |s tdi}g}|D] jdk(r*tjgtj}n|Dcgc]}| j vs|}}|rtd|d jdt fd|D}tj|Dcgc]}|j|t|c}tj}|j jd||Scc}wcc}w) uGive each table a `_rowid` column derived from its primary key. The fusion only ever compares that column for equality — to deduplicate, to group by in a reranker, and to restore the original scores — so on an RTE table, where the server has no stable row id to hand out, any injective mapping of the primary key does the same job. Ids are assigned in first-seen order across `tables`, so passing them in fusion order (vector, then FTS) preserves "first occurrence wins" and its tie break. Writing it under the `_rowid` name is what lets every reranker, third-party implementations included, run unmodified. The column never reaches the caller. zmhybrid search on an RTE table needs an unenforced primary key to deduplicate on, and this table declares noner)typez;hybrid search could not deduplicate: primary key column(s) z missing from a result set with z rowsc3\K|]#}j|j%ywN)column to_pylist).0r8tables r) z+_stamp_surrogate_row_ids..s"Ia224Is),_rowid) rnum_rowspaarrayuint64 column_nameszip setdefaultlenr append_column) tablesr7idsstampedrow_idsr8r9keyskeyrBs @r)_stamp_surrogate_row_idsrTms  ;  CG? >>Q hhr 4G",LQ9K9K0KqLGL !!( )I~~&e- IjIJDhh:>?3SX.?biikG u**8W=>#?$ NM@sD)D1"D c4||dury|jduS)zWhether a previous hybrid query on `table` was refused `_rowid`, so this one should go straight to the primary key. Learned from a refusal, never probed, so asking is free. FT)_hybrid_pk_fusion_learned)rBuse_rtes r)pk_fusion_learnedrXs)  }5(  * * , 44r0c\eZdZdZddZe d dZdZ d dZd dZ y) PkFusionzrHow the legs of a hybrid query on an RTE table are joined: on a surrogate `_rowid` built from the primary key.c ||_g|_yr>)r7injected)selfr7s r)__init__zPkFusion.__init__s$$& r0ct|rttt||r td|t|S)zZResolve the key for a hybrid query, refusing the shapes that need a real `_rowid`.zhybrid search cannot project a blob column on an RTE table: fetching blobs needs a real _rowid, and the fresh tier has no stable one. Set use_rte(False) to read the base table only (results will exclude un-compacted RTE data))NotImplementedErrorWAL_ROW_ID_UNSUPPORTEDr r*)clsr" projectioncaller_requested_row_ids r) for_queryzPkFusion.for_querysC #%&<= = &fj 9%?  ;v&''r0cBt||j\}|_|S)zt`columns` with the key columns it is missing, to project in place of the `_rowid` an RTE table would refuse.)r:r7r\)r]r6s r)injectzPkFusion.injects "2'4??!Kr0c@t||g|j\}}||fS)z7Give both legs a surrogate `_rowid` built from the key.)rTr7)r]vector_results fts_resultss r)stampzPkFusion.stamps. '? [ )4??' # {**r0cdg|jDcgc]}||jvs|}}|r|j|S|Scc}w)zDrop the surrogate and any injected key columns from the reranked results. The surrogate is an internal join key, never an answer, and a caller who asked for `_rowid` was refused in `for_query`. rD)r\rIdrop)r]resultsr8rms r)stripzPkFusion.stripsL%5t}}5Sag>R>R9RSS%)w||D!6w6Ts AANr7 List[str])r" pa.Schemarcr rdboolreturnrZ)ripa.TablerjrurtzTuple[pa.Table, pa.Table])rnrurtru) __name__ __module__ __qualname____doc__r^ classmethodrergrkror0r)rZrZsm6& (($( "& (  ((, +& +5= + " +7r0rZcF|s t|s||rtt|y)z\Re-raise `error` unless it is an RTE `_rowid` refusal worth retrying on the primary key.N)r/r`ra)r.fusedrds r)_check_refusalr~s, *51 !"89uDr0c |S#t$r4}t||||j|cYd}~Sd}~wwxYw)aRun a hybrid query, retrying on the primary key if the table refuses `_rowid`. `_rowid` is asked for first rather than paying a round trip up front to find out: only an RTE table refuses, and the refusal has to be handled anyway -- a write spec installed elsewhere can arrive between any two queries. It costs nothing server-side: it is raised before the query is planned. `fused` reports whether the attempt already used the key. r}rdN Exceptionr~_note_hybrid_pk_fusionrunrBr}rdes r)with_pk_fallbackrsE u  UW6M  $$&u s A)AAAcK |d{S7#t$r=}t||||j|d{7cYd}~Sd}~wwxYww)z%The async form of `with_pk_fallback`.Nrrrs r)with_pk_fallback_asyncr sRU{{  UW6M  $$&U{{ sHA A A-A A AAAAA)r"rrrtrq)r. BaseExceptionrtrsrp)rNList[pa.Table]r7rqrtr)rWzOptional[bool]rtrs)r.rr}rsrdrs)rzCallable[[], T]r}Callable[[], bool]rdrsrtr)rzCallable[[], Awaitable[T]]r}rrdrsrtr) ry __future__rtypingrrrrrr r pyarrowrF_blobr typesr rrrrr*r,rar/r:rTrXrZr~rrr{r0r)rs #UUU-& CL 4 E )0@5*- ,&* *(1**Z 5:7:7z E    "   6 #   "   r0