3 g@sPdZddlZddlmZddlmZmZddlmZGdddZ d dd Z dS) z SQL LexerN) TextIOBase)tokenskeywords)consumec@sReZdZdZdZeddZddZddZd d Z d d Z d dZ dddZ dS)LexerzrThe Lexer supports configurable syntax. To add support for additional keywords, use the `add_keywords` method.NcCs"|jdkr||_|jj|jS)zRReturns the lexer instance used internally by the sqlparse core functions.N)_default_intancedefault_initialization)clsr L/var/www/tester-filtro-web/env/lib/python3.6/site-packages/sqlparse/lexer.pyget_default_instance.s  zLexer.get_default_instancecCs`|j|jtj|jtj|jtj|jtj|jtj|jtj |jtj dS)zlInitialize the lexer with default dictionaries. Useful if you need to revert custom syntax settings.N) clear set_SQL_REGEXr SQL_REGEX add_keywordsZKEYWORDS_COMMONZKEYWORDS_ORACLEZKEYWORDS_PLPGSQLZ KEYWORDS_HQLZKEYWORDS_MSACCESSZKEYWORDS)selfr r r r7s      zLexer.default_initializationcCsg|_g|_dS)zClear all syntax configurations. Useful if you want to load a reduced set of syntax configurations. After this call, regexps and keyword dictionaries need to be loaded to make the lexer functional again.N) _SQL_REGEX _keywords)rr r r r Csz Lexer.clearcs$tjtjBfdd|D|_dS)z.Set the list of regex that will parse the SQL.cs"g|]\}}tj|j|fqSr )recompilematch).0rxtt)FLAGSr r Osz'Lexer.set_SQL_REGEX..N)r IGNORECASEUNICODEr)rrr )rr rKs  zLexer.set_SQL_REGEXcCs|jj|dS)zhAdd keyword dictionaries. Keywords are looked up in the same order that dictionaries were added.N)rappend)rrr r r rSszLexer.add_keywordscCs:|j}x,|jD]}||kr|||fSqWtj|fSdS)zChecks for a keyword. If the given value is in one of the KEYWORDS_* dictionary it's considered a keyword. Otherwise, tokens.Name is returned. N)upperrrName)rvaluevalZkwdictr r r is_keywordXs  zLexer.is_keywordc cst|tr|j}t|trn\t|trh|r8|j|}qzy|jd}Wqztk rd|jd}YqzXntdjt |t |}x|D]\}}x~|j D]h\}}|||}|sqn6t|t j r||jfVn|tjkr|j|jVt||j|dPqWt j|fVqWdS)a Return an iterable of (tokentype, value) pairs generated from `text`. If `unfiltered` is set to `True`, the filtering mechanism is bypassed even if filters are defined. Also preprocess the text, i.e. expand tabs and strip it if wanted and applies registered filters. Split ``text`` into (tokentype, text) pairs. ``stack`` is the initial stack (default: ``['root']``) zutf-8zunicode-escapez+Expected text or file-like object, got {!r}N) isinstancerreadstrbytesdecodeUnicodeDecodeError TypeErrorformattype enumeraterrZ _TokenTypegrouprZPROCESS_AS_KEYWORDr#rendError) rtextencodingiterableposcharZrexmatchactionmr r r get_tokenses4        zLexer.get_tokens)N) __name__ __module__ __qualname____doc__r classmethodr rr rrr#r9r r r r rs   rcCstjj||S)zTokenize sql. Tokenize *sql* using the :class:`Lexer` and return a 2-tuple stream of ``(token type, value)`` items. )rr r9)Zsqlr3r r r tokenizesr?)N) r=riorZsqlparserrZsqlparse.utilsrrr?r r r r s