kQj!\ddlmZddlZddlmZddlmZmZddlm Z d dZ Gd dZ y) ) annotationsN) __version__)MissingColumnErrorMissingValueError)pandasContextualizerct|S)a"Create a Contextualizer object for the given DataFrame. Used to create context windows. Context windows are rolling subsets of text data. The input text column should already be separated into rows that will be the unit of the window. So to create a context window over tokens, start with a DataFrame with one token per row. To create a context window over sentences, start with a DataFrame with one sentence per row. Examples -------- >>> from lancedb.context import contextualize >>> import pandas as pd >>> data = pd.DataFrame({ ... 'token': ['The', 'quick', 'brown', 'fox', 'jumped', 'over', ... 'the', 'lazy', 'dog', 'I', 'love', 'sandwiches'], ... 'document_id': [1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2] ... }) ``window`` determines how many rows to include in each window. In our case this how many tokens, but depending on the input data, it could be sentences, paragraphs, messages, etc. >>> contextualize(data).window(3).stride(1).text_col('token').to_pandas() token document_id 0 The quick brown 1 1 quick brown fox 1 2 brown fox jumped 1 3 fox jumped over 1 4 jumped over the 1 5 over the lazy 1 6 the lazy dog 1 7 lazy dog I 1 8 dog I love 1 9 I love sandwiches 2 10 love sandwiches 2 >>> (contextualize(data).window(7).stride(1).min_window_size(7) ... .text_col('token').to_pandas()) token document_id 0 The quick brown fox jumped over the 1 1 quick brown fox jumped over the lazy 1 2 brown fox jumped over the lazy dog 1 3 fox jumped over the lazy dog I 1 4 jumped over the lazy dog I love 1 5 over the lazy dog I love sandwiches 1 ``stride`` determines how many rows to skip between each window start. This can be used to reduce the total number of windows generated. >>> contextualize(data).window(4).stride(2).text_col('token').to_pandas() token document_id 0 The quick brown fox 1 2 brown fox jumped over 1 4 jumped over the lazy 1 6 the lazy dog I 1 8 dog I love sandwiches 1 10 love sandwiches 2 ``groupby`` determines how to group the rows. For example, we would like to have context windows that don't cross document boundaries. In this case, we can pass ``document_id`` as the group by. >>> (contextualize(data) ... .window(4).stride(2).text_col('token').groupby('document_id') ... .to_pandas()) token document_id 0 The quick brown fox 1 2 brown fox jumped over 1 4 jumped over the lazy 1 6 the lazy dog 1 9 I love sandwiches 2 ``min_window_size`` determines the minimum size of the context windows that are generated.This can be used to trim the last few context windows which have size less than ``min_window_size``. By default context windows of size 1 are skipped. >>> (contextualize(data) ... .window(6).stride(3).text_col('token').groupby('document_id') ... .to_pandas()) token document_id 0 The quick brown fox jumped over 1 3 fox jumped over the lazy dog 1 6 the lazy dog 1 9 I love sandwiches 2 >>> (contextualize(data) ... .window(6).stride(3).min_window_size(4).text_col('token') ... .groupby('document_id') ... .to_pandas()) token document_id 0 The quick brown fox jumped over 1 3 fox jumped over the lazy dog 1 )r )raw_dfs T/home/oem/CyberBrain26-unified/.venv/lib/python3.12/site-packages/lancedb/context.py contextualizer sB & !!ceZdZdZdZddZddZddZddZddZ e jdd e d dd Z dd Zy)r zXCreate context windows from a DataFrame. See [lancedb.context.contextualize][]. cXd|_d|_d|_d|_d|_||_y)N) _text_col_groupby_stride_window_min_window_size_raw_df)selfr s r __init__zContextualizer.__init__ws-   ! rc||_|S)zSet the window size. i.e., how many rows to include in each window. Parameters ---------- window: int The window size. )r)rwindows r rzContextualizer.window  rc||_|S)zSet the stride. i.e., how many rows to skip between each window. Parameters ---------- stride: int The stride. )r)rstrides r rzContextualizer.striderrc||_|S)zSet the groupby column. i.e., how to group the rows. Windows don't cross groups Parameters ---------- groupby: str The groupby column. )r)rgroupbys r r zContextualizer.groupbys   rc||_|S)zSet the text column used to make the context window. Parameters ---------- text_col: str The text column. )r)rtext_cols r r"zContextualizer.text_cols" rc||_|S)zSet the (optional) min_window_size size for the context window. Parameters ---------- min_window_size: int The min_window_size. )r)rmin_window_sizes r r$zContextualizer.min_window_sizes!0 rz0.3.1z0.4.0zUse to_pandas() instead) deprecated_in removed_incurrent_versiondetailsc"|jS)N) to_pandas)rs r to_dfzContextualizer.to_dfs~~rc <t tdjjjj vrt jjjdkr tdjjdkr tdfd}j|jStjjjjDcgc] \}}||c}}Scc}}w)z2Create the context windows and return a DataFrame.z:pandas is required to create context windows using lancedbrznThe value of window is None or less than 1. Specify the window size (number of rows to include in each window)zkThe value of stride is None or less than 1. Specify the stride (number of rows to skip between each window)c F|jj}|jddjddfj }t dt |jDcgc]l}|jzt |kst ||z jk\r4dj||t|jzt |n}}t |t |kr|jdt |}||j<|Scc}w)Nr ) rvaluesilocrcopyrangelenrrjoinmin)grptextcontextsstart_iwindowsrs r process_groupz/Contextualizer.to_pandas..process_groupst~~&--Dxx4<< 2388:H %QC$,,?T\\)SX5s8g%)>)>>gGdll,BCH(MNOG7|c(m+#==3w<8'.HT^^ $Os+A1D) pd ImportErrorrrcolumnstolistrrrrrconcatr )rr;_r6s` r r*zContextualizer.to_pandass :L  >>!5!5! >$T^^4 4 << 4<D N)rintreturnr )rrBrCr )r strrCr )r"rDrCr )r$rBrCr )rC'pd.DataFrame')__name__ __module__ __qualname____doc__rrrr r"r$ deprecation deprecatedrr+r*rr r r rsW     [#)     , r)r rErCr ) __future__rrJr exceptionsrr dependenciesrr<r r rLrr rQs) #=&a"Hz z r