hKjQdZddlmZddlmZddlZddlZdZejeZ dZ er?dZ ddl Z eje je jejeZGd d eZGd d eZd ZGddeZej,dej.ej0zej2zZdZy)u, Aho-Corasick string search algorithm in pure Python Original Author: Wojciech Muła, wojciech_mula@poczta.onet.pl WWW : http://0x80.pl License : public domain This is the pure Python Aho-Corasick automaton from pyahocorasick modified for use in the license_expression library for advanced tokenization: - add support for unicode strings. - case insensitive search using sequence of words and not characters - improve returned results with the actual start,end and matched string. - support returning non-matched parts of a string )deque) OrderedDictNFcyNargss f/home/oem/CyberBrain26-unified/.venv/lib/python3.12/site-packages/license_expression/_pyahocorasick.py logger_debugr !scXtjdjd|DS)N c3^K|]%}t|txr|xs t|'ywr) isinstancestrrepr).0as r zlogger_debug..(s($WQZ3%7%=A%Ha%H$Ws+-)loggerdebugjoinrs r r r 's!||CHH$WRV$WWXXr )streamc(eZdZdZgdZefdZdZy)TrieNodez2 Node of the Trie/Aho-Corasick automaton. )tokenoutputfailchildrencD||_||_t|_i|_yr)rrnilrr)selfrrs r __init__zTrieNode.__init__:s$    r c~|jturd|jd|jdSd|jzS)Nz TrieNode(z, )z TrieNode(%r))rr!rr"s r __repr__zTrieNode.__repr__Js0 ;;c !)-T[[A A!DJJ. .r N)__name__ __module__ __qualname____doc__ __slots__r!r#r'rr r rr3s8I%( /r rcbeZdZdZdZddZdZefdZdZ dZ d Z d Z d Z d Zdd ZddZy)Triez A Trie and Aho-Corasick automaton. This behaves more or less like a mapping of key->value. This is the main entry point. cPtd|_t|_d|_y)z( Initialize a new Trie. FN)rrootset _known_tokens _convertedr&s r r#z Trie.__init__Ws$RL !U r Nc|jr td|rt|tsyt |Dcgc]}|j s|}}|j j||j}|D]} |j|}||xs|f|_ ycc}w#t$rt|}||j|<|}YOwxYw)a~ Add a new tokens_string and its associated value to the trie. If the tokens_string already exists in the Trie, its value is replaced with the provided value, typically a Token object. If a value is not provided, the tokens_string is used as value. A tokens_string is any string. It will be tokenized when added to the Trie. zQThis Trie has been converted to an Aho-Corasick automaton and cannot be modified.N) r4 Exceptionrr get_tokensstripr3updater1rKeyErrorrr)r" tokens_stringvaluettokensnoderchilds r addzTrie.addes ??c J}c$B ' 6D!'')!DD !!&)yy E }}U+ %e&<}= #E  ', e$ sB B B%%%C  C c|rt|tsyt|Dcgc]}|js|}}|j}|D]} |j |}|Scc}w#t $rYywxYw)z Return a node for this tokens_string or None if the trie does not contain the tokens_string. Private function retrieving a final node of the Trie for a given tokens_string. N)rrr7r8r1rr:)r"r;r=r>r?rs r __get_nodezTrie.__get_nodes J}c$B ' 6D!'')!DDyy E }}U+   E   sA"A"A'' A43A4c|j|}t}|r |j}|tur|tur t||S|S)a Return the output value found associated with a `tokens_string`. If there is no such tokens_string in the Trie, return the default value (other than nil). If `default` is not provided or is `nil`, raise a KeyError. )_Trie__get_noder!rr:)r"r;defaultr?rs r getzTrie.getsF}- [[F S=#~}--Mr c0d|jDS)z5 Yield all keys stored in this trie. c3&K|] \}}| ywrr)rkey_s r rzTrie.keys..s/Q/itemsr&s r keysz Trie.keyss0$**,//r c0d|jDS)zL Yield all values associated with keys stored in this trie. c3&K|] \}}| ywrr)rrKr<s r rzTrie.values..s3(!U3rLrMr&s r valuesz Trie.valuess4djjl33r cTgfd|jgtS)zF Yield tuple of all (key, value) stored in this trie. c(||jgzDcgc]}|s| }}|jtur-j|jd|jdf|jj D]}||us||ycc}w)z5 Walk the trie, depth first. rN)rrr!appendrrR)r?r>r=r@rNwalks r rWzTrie.items..walks"(4::,!6)r1iter)r"rNrWs @@r rNz Trie.itemss) (" TYYr"E{r cb|j|}|rt|jtk7Sy)zA Return True if the key is present in this trie. F)rEboolrr!)r"r;r?s r existsz Trie.existss,}-  s*+ +r c:t|j|duS)ze Return True if tokens_string is a prefix of any existing tokens_string in the trie. N)rZrE)r"r;s r is_prefixzTrie.is_prefixsDOOM2$>??r ct}|jD]y}||jjvr<|jj|}|j|_|j |W|j|jj|<{|r|j }|jjD]}|j ||j}|j|jvr%|j}|j|jvr%|jj|j|j|_|rd|_ y)z Convert this trie to an Aho-Corasick automaton. Note that this is an error to add new keys to a Trie once it has been converted to an Automaton. TN) rr3r1rrrVpopleftrRrrGr4)r"queuerr? current_nodestates r make_automatonzTrie.make_automatons  '' 6E ***yy))%0 II  T",0II ""5) 6 ==?L$--446 F T"$))jj6!JJEjj6!NN..tzz499E  Fr c #K|syt|}|j}tr!tdt |td|d}|D]}|t |z }tr+ttdt |td||s"|j str tdg||jvr^|j}tr td|r>t |}||z d z} t| ||| |d zd } tr td | | d } ||jvr|j}||jvr|jj||j}|} | tur| jturq| j\} }trtd t |t| t | }||z d z} tr td| t| ||| |d z|d} | j} | tur| r|str tdt |}||z d z} t| ||| |d zd} tr td| | tyw)aS Yield Token objects for matched strings by performing the Aho-Corasick search procedure. The Token start and end positions in the searched string are such that the matched string is "tokens_string[start:end+1]". And the start is computed from the end_index collected by the Aho-Corasick search procedure such that "start=end_index - n + 1" where n is the length of a matched string. The Token.value is an object associated with a matched string. For example: >>> a = Trie() >>> a.add('BCDEF') >>> a.add('CDE') >>> a.add('DEFGH') >>> a.add('EFGH') >>> a.add('KL') >>> a.make_automaton() >>> tokens_string = 'a bcdef ghij kl m' >>> strings = Token.sort(a.iter(tokens_string)) >>> expected = [ ... Token(2, 6, u'bcdef', u'BCDEF'), ... Token(13, 14, u'kl', u'KL') ... ] >>> strings == expected True >>> list(a.iter('')) == [] True >>> list(a.iter(' ')) == [] True NzTrie.iter() with:z tokens: token_stringz end_posz include_space skippedz unmatchedrUstartendstringr<z unmatched tok:Fz type outputz start_posTz unmatched but known tokenz unmatched tok 2:)r7r1TRACEr rlenr8r3TokenrrrGr!rtype)r"r;include_unmatched include_spacer>rbend_posrfn start_postokyieldedmatchmatched_string output_values r rXz Trie.iters`J M*   ,d=.A B V ,"= L s<( (G^T,-?@Z1 ););)= !:;4#5#55  /$L)A '! aI'#,Y1E" C $%7=IGenn4 enn4NN&&|TYY?EEs"<? %#aK!O Iw i'TU+0VX\] !5s; {= ~ s EI2CI2I2I2AI2c|j|||}t|}tr td||s*|Dcgc]}|jj s|!}}t |}|Scc}w)a` Tokenize a string for matched and unmatched sub-sequences and yield non- overlapping Token objects performing a modified Aho-Corasick search procedure: - return both matched and unmatched sub-sequences. - do not return matches with positions that are contained or overlap with another match: - discard smaller matches contained in a larger match. - when there is overlap (but not containment), the matches are sorted by start and biggest length and then: - we return the largest match of two overlaping matches - if they have the same length, keep the match starting the earliest and return the non-overlapping portion of the other discarded match as a non-match. Each Token contains the start and end position, the corresponding string and an associated value object. For example: >>> a = Trie() >>> a.add('BCDEF') >>> a.add('CDE') >>> a.add('DEFGH') >>> a.add('EFGH') >>> a.add('KL') >>> a.make_automaton() >>> string = 'a bcdef ghij kl' >>> tokens = list(a.tokenize(string, include_space=True)) >>> expected = [ ... Token(0, 0, u'a', None), ... Token(1, 1, u' ', None), ... Token(2, 6, u'bcdef', u'BCDEF'), ... Token(7, 7, u' ', None), ... Token(8, 11, u'ghij', None), ... Token(12, 12, u' ', None), ... Token(13, 14, u'kl', u'KL') ... ] >>> tokens == expected True )rorpztokenize.tokens:)rXlistrkr rjr8filter_overlapping)r"rjrorpr>r=s r tokenizez Trie.tokenizeysiV65FVcdf  +V 4!'>+;aJ$*-(0 4 6@ Dob2r r.ctj|}d}|t|dz kr|dz}|t|kr||}||}td|||||j |r td|np||vrtd|||=Y|j |r7t|t|k\rtd|||=td|||=n|dz }|t|kr|dz }|t|dz kr|S)a{ Return a new list from an iterable of `tokens` discarding contained and overlaping Tokens using these rules: - skip a token fully contained in another token. - keep the biggest, left-most token of two overlapping tokens and skip the other For example: >>> tokens = [ ... Token(0, 0, 'a'), ... Token(1, 5, 'bcdef'), ... Token(2, 4, 'cde'), ... Token(3, 7, 'defgh'), ... Token(4, 7, 'efgh'), ... Token(8, 9, 'ij'), ... Token(10, 13, 'klmn'), ... Token(11, 15, 'lmnop'), ... Token(16, 16, 'q'), ... ] >>> expected = [ ... Token(0, 0, 'a'), ... Token(1, 5, 'bcdef'), ... Token(8, 9, 'ij'), ... Token(11, 15, 'lmnop'), ... Token(16, 16, 'q'), ... ] >>> filtered = list(filter_overlapping(tokens)) >>> filtered == expected True rrUzcurr_tok, i, next_tok, j:z break to nextz del next_tok contained:z del next_tok smaller overlap:z del curr_tok smaller overlap:)rmsortrlr is_afteroverlap)r>ijcurr_toknext_toks r r{r{s BZZ F A c&kAo  E#f+oayHayH 4h8Q O  *.98#8(C1I)x=CM1 !BHMq  !BHMq  FA7#f+o8 Q= c&kAo > Mr cbeZdZdZdZddZdZdZdZdZ d Z e d Z d Z d Zd ZdZy)rma0 A Token is used to track the tokenization an expression with its start and end as index position in the original string and other attributes: - `start` and `end` are zero-based index in the original string S such that S[start:end+1] will yield `string`. - `string` is the matched substring from the original string for this Token. - `value` is the corresponding object for this token as one of: - a LicenseSymbol object - a "Keyword" object (and, or, with, left and right parens) - None if this is a space. rgNc<||_||_||_||_yrrg)r"rhrirjr<s r r#zToken.__init__ s   r cV|jjd|jzzS)Nz+(%(start)r, %(end)r, %(string)r, %(value)r)) __class__r(as_dictr&s r r'zToken.__repr__s) NN # #&SVZVbVbVd&d d r c jt|jDcgc]}|t||fc}Scc}wr)rr,getattr)r"ss r rz Token.as_dicts+4>>JaQa 01JKKJs0c:|j|jz dzS)NrUrirhr&s r __len__z Token.__len__sxx$**$q((r ct|txrj|j|jk(xrO|j|jk(xr4|j|jk(xr|j |j k(Sr)rrmrhrirjr<r"others r __eq__z Token.__eq__sg%' JJ%++ % *EII% * u||+ * ekk)  r ct|j|j|j|jf}t |Sr)rhrirjr<hash)r"tups r __hash__zToken.__hash__#s*jj$((DKK;Cyr c"d}t||S)a Return a new sorted sequence of tokens given a sequence of tokens. The primary sort is on start and the secondary sort is on longer lengths. Therefore if two tokens have the same start, the longer token will sort first. For example: >>> tokens = [Token(0, 0), Token(5, 5), Token(1, 1), Token(2, 4), Token(2, 5)] >>> expected = [Token(0, 0), Token(1, 1), Token(2, 5), Token(2, 4), Token(5, 5)] >>> expected == Token.sort(tokens) True c2|jt| fSr)rhrl)rs r rJzToken.sort..key6sQ r )rJ)sorted)clsr>rJs r r~z Token.sort's  f#&&r c4|j|jkDS)aJ Return True if this token is after the other token. For example: >>> Token(1, 2).is_after(Token(5, 6)) False >>> Token(5, 6).is_after(Token(5, 6)) False >>> Token(2, 3).is_after(Token(1, 2)) False >>> Token(5, 6).is_after(Token(3, 4)) True rhrirs r rzToken.is_after>szzEII%%r c4|j|jkSrrrs r is_beforezToken.is_beforeNsxx%++%%r cj|j|jkxr|j|jkS)aa Return True if this token contains the other token. For example: >>> Token(5, 7) in Token(5, 7) True >>> Token(6, 8) in Token(5, 7) False >>> Token(6, 6) in Token(4, 8) True >>> Token(3, 9) in Token(4, 8) False >>> Token(4, 8) in Token(3, 9) True rrs r __contains__zToken.__contains__Qs) zzU[[(BUYY$((-BBr c|j}|j}||jcxkxr|kncxs||jcxkxr|kScS)a Return True if this token and the other token overlap. For example: >>> Token(1, 2).overlap(Token(5, 6)) False >>> Token(5, 6).overlap(Token(5, 6)) True >>> Token(4, 5).overlap(Token(5, 6)) True >>> Token(4, 5).overlap(Token(5, 7)) True >>> Token(4, 5).overlap(Token(6, 7)) False r)r"rrhris r rz Token.overlapcsC  hh++K%))1Js1JK1JKr )r0N)r(r)r*r+r,r#r'rrrr classmethodr~rrrrrr r rmrmsW I  L) '',& &C$Lr rmzV (?P[^\s\(\)]+) | (?P\s+) | (?P[\(\)]) cttj|jDcgc]}|s| c}Scc}w)zG Return an iterable of strings splitting on spaces and parens. ) _tokenizersplitlower)r;rvs r r7r7s. *// 0C0C0EF Pe%E PP Ps55)r+ collectionsrrloggingrerk getLoggerr(rr sys basicConfigstdoutsetLevelDEBUGobjectr!rr.r{rmcompileVERBOSE MULTILINEUNICODErr7rr r rs #     8 $  YGszz* OOGMM" h/v/