֦iFm d Z dZddlZddlmZ dZi ad Z G d de Z G d d
e Z
G d d e Z d
Z G d de Z
G d de
Z G d de
Z G d de
Z G d de
Zd Zy)z
Python parse tree definitions.
This is a very concrete parse tree; we need to keep every token and
even the comments and whitespace between tokens.
There's also a pattern matching implementation here.
z#Guido van Rossum N)StringIOic t sDddlm} |j j D ]! \ }}t | t k( s|t |< # t j | | S )N )python_symbols) _type_reprspygramr __dict__itemstypeint
setdefault)type_numr namevals '/usr/lib64/python3.12/lib2to3/pytree.py type_reprr sO * (00668ID#CyC DS!1 9 ! !(H 55 c e Zd ZdZdZdZdZdZdZd Z d Z
dZ d Z d Z
d Zd
Zd Zd Zd
Zd Zed Zed Zd Zd Zd Zej6 dk rd Z yy)Basez
Abstract base class for Node and Leaf.
This provides some default functionality and boilerplate using the
template pattern.
A node may be a subnode of at most one parent.
N Fc J | t usJ d t j | S )z7Constructor that prevents Base from being instantiated.zCannot instantiate Base)r object__new__clsargskwdss r r z Base.__new__1 s# $9 99~~c""r c ` | j |j urt S | j | S )zW
Compare two nodes for equality.
This calls the method _eq().
) __class__NotImplemented_eqselfothers r __eq__z Base.__eq__6 s( >> 0! !xx r c t )a_
Compare two nodes for equality.
This is called by __eq__ and __ne__. It is only called if the two nodes
have the same type. This must be implemented by the concrete subclass.
Nodes should be considered equal if they have the same structure,
ignoring the prefix string and other context information.
NotImplementedErrorr" s r r! zBase._eqB s
"!r c t )zr
Return a cloned (deep) copy of self.
This must be implemented by the concrete subclass.
r' r# s r clonez
Base.cloneM
"!r c t )zx
Return a post-order iterator for the tree.
This must be implemented by the concrete subclass.
r' r* s r
post_orderzBase.post_orderU r, r c t )zw
Return a pre-order iterator for the tree.
This must be implemented by the concrete subclass.
r' r* s r pre_orderzBase.pre_order] r, r c | j J t | |J t |t s|g}g }d}| j j D ]M }|| u r6|r J | j j | |f ||j | d}=|j
| O |sJ | j | |f | j j || j _ |D ] }| j |_ d| _ y)z/Replace this node with a new one in the parent.NFT)parentstr
isinstancelistchildrenextendappendchanged)r# new
l_childrenfoundchxs r replacez Base.replacee s {{&1D 1& #t$%C
++&&BTz C4;;#7#7s"CCy? %%c* !!"% ' 0t}}dC00u ) A{{AH r c | }t |t s-|j sy|j d }t |t s-|j S )z9Return the line number which generated the invocant node.Nr )r4 Leafr6 linenor# nodes r
get_linenozBase.get_lineno| sA T4(====#D T4( {{r c ^ | j r| j j d| _ y )NT)r2 r9 was_changedr* s r r9 z Base.changed s! ;; KK ! r c | j rht | j j D ]E \ }}|| u s | j j | j j |= d| _ |c S yy)z
Remove the node from the tree. Returns the position of the node in its
parent's children before it was removed.
N)r2 enumerater6 r9 )r# irD s r removez Base.remove sb
;;$T[[%9%9:44<KK'') ,,Q/"&DK H ; r c | j yt | j j D ]( \ }}|| u s | j j |dz c S y# t $ r Y yw xY w)z
The node immediately following the invocant in their parent's children
list. If the invocant does not have a next sibling, it is None
Nr )r2 rI r6
IndexErrorr# rJ childs r next_siblingzBase.next_sibling si ;; "$++"6"67HAu} ;;//!44 8 " s A A&%A&c | j yt | j j D ]. \ }}|| u s |dk( r y| j j |dz
c S y)z
The node immediately preceding the invocant in their parent's children
list. If the invocant does not have a previous sibling, it is None.
Nr r )r2 rI r6 rN s r prev_siblingzBase.prev_sibling sZ ;; "$++"6"67HAu}6 {{++AaC00 8r c # b K | j D ] }|j E d { y 7 wN)r6 leavesr# rO s r rU z Base.leaves s& ]]E ||~ % % # %s #/-/c V | j yd| j j z S )Nr r )r2 depthr* s r rX z
Base.depth s' ;; 4;;$$&&&r c 8 | j }|y|j S )z
Return the string immediately following the invocant node. This is
effectively equivalent to node.next_sibling.prefix
)rP prefix)r# next_sibs r
get_suffixzBase.get_suffix s"
$$ r r c 6 t | j d S )Nascii)r3 encoder* s r __str__z Base.__str__ s t9##G, ,r ) __name__
__module__ __qualname____doc__r r2 r6 rG was_checkedr r% __hash__r! r+ r. r0 r? rE r9 rK propertyrP rR rU rX r] sys version_inforc r r r r r s D
FHKK#
H """".
1
1 &'
& - !r r c e Zd ZdZ ddZd Zd Zej dkD reZ d Z
d Z d Z d
Z
ed Zej d Zd
Zd Zd Zy)Nodez+Concrete implementation for interior nodes.Nc |dk\ sJ | || _ t | | _ | j D ]% }|j J t | | |_ ' ||| _ |r |dd | _ yd| _ y)z
Initializer.
Takes a type constant (a symbol number >= 256), a sequence of
child nodes, and an optional context keyword argument.
As a side effect, the parent pointers of the children are updated.
N)r r5 r6 r2 reprr[ fixers_applied)r# r r6 contextr[ rr r= s r __init__z
Node.__init__ s| s{ D { X
--B99$ .d2h .$ BI DK "0"3D "&D r c z | j j dt | j d| j dS z)Return a canonical string representation.(, ))r rd r r r6 r* s r __repr__z
Node.__repr__ s, #~~ 6 6 ( 3 #}}. .r c T dj t t | j S )k
Return a pretty string representation.
This reproduces the input source exactly.
rZ )joinmapr3 r6 r* s r __unicode__zNode.__unicode__ s wws3
.//r r^ c d | j | j f|j |j fk( S z Compare two nodes for equality.)r r6 r" s r r! zNode._eq s' 4==)ejj%..-IIIr c t | j | j D cg c] }|j c}| j S c c}w z$Return a cloned (deep) copy of self.)rr )rn r r6 r+ rr )r# r= s r r+ z
Node.clone s< DIIT]] C]r
] C#'#6#68 8 Cs A
c # j K | j D ] }|j E d{ | y7 wz*Return a post-order iterator for the tree.N)r6 r. rV s r r. zNode.post_order s0 ]]E '') ) ) #
*s #31 3c # j K | | j D ] }|j E d{ y7 wz)Return a pre-order iterator for the tree.N)r6 r0 rV s r r0 zNode.pre_order s,
]]E ( ( ( # (s '313c N | j sy| j d j S )zO
The whitespace and comments preceding this node in the input.
rZ r r6 r[ r* s r r[ z Node.prefix s#
}}}}Q &&&r c F | j r|| j d _ y y Nr r r# r[ s r r[ z Node.prefix s ==&,DMM! # r c x | |_ d| j | _ || j |< | j y)z
Equivalent to 'node.children[i] = child'. This method also sets the
child's parent attribute appropriately.
N)r2 r6 r9 rN s r set_childzNode.set_child s3
"&
a
a r c j | |_ | j j || | j y)z
Equivalent to 'node.children.insert(i, child)'. This method also sets
the child's parent attribute appropriately.
N)r2 r6 insertr9 rN s r insert_childzNode.insert_child* s(
Q& r c h | |_ | j j | | j y)z
Equivalent to 'node.children.append(child)'. This method also sets the
child's parent attribute appropriately.
N)r2 r6 r8 r9 rV s r append_childzNode.append_child3 s&
U# r NNN)rd re rf rg rt rz r rk rl rc r! r+ r. r0 rj r[ setterr r r r r r rn rn s 5 $'2. 0 & J8
) ' ' ]]- -r rn c e Zd ZdZdZdZdZddg fdZd Zd Z e
j dkD re Z d Z
d
Zd Zd Zd
Zed Zej( d Zy)rA z'Concrete implementation for leaf nodes.rZ r Nc d|cxk r dk sJ | J | ||\ | _ \ | _ | _ || _ || _ ||| _ |dd | _ y)z
Initializer.
Takes a type constant (a token number < 256), a string value, and an
optional context keyword argument.
r rp N)_prefixrB columnr valuerr )r# r r rs r[ rr s r rt z
Leaf.__init__F sf D3$$$$ 7> 4DL44;
!DL ,Q /r c h | j j d| j d| j dS rv )r rd r r r* s r rz z
Leaf.__repr__Y s' #~~ 6 6 #yy #zz+ +r c F | j t | j z S )r| )r[ r3 r r* s r r zLeaf.__unicode___ s {{S_,,r r^ c d | j | j f|j |j fk( S r )r r r" s r r! zLeaf._eqj s' 4::&5::u{{*CCCr c t | j | j | j | j | j
ff| j S r )rA r r r[ rB r rr r* s r r+ z
Leaf.clonen s= DIItzz[[4;; "<=#'#6#68 8r c # K | y wrT r r* s r rU z Leaf.leavest s
c # K | ywr r r* s r r. zLeaf.post_orderw
r c # K | ywr r r* s r r0 zLeaf.pre_order{ r r c | j S )zP
The whitespace and comments preceding this token in the input.
)r r* s r r[ z Leaf.prefix s
||r c 2 | j || _ y rT )r9 r r s r r[ z Leaf.prefix s r )rd re rf rg r rB r rt rz r rk rl rc r! r+ rU r. r0 rj r[ r r r r rA rA = s 1 G
F
F "0&+ - & D8 ]] r rA c |\ }}}}|s|| j v r!t | dk( r|d S t ||| S t ||| S )z
Convert raw node information to a Node or Leaf instance.
This is passed to the parser driver which calls it whenever a reduction of a
grammar rule produces a new complete node, so that the tree is build
strictly bottom-up.
r r )rs )
number2symbollenrn rA )grraw_noder r rs r6 s r convertr sX &."D%(42+++ x=A A; D(G44D%11r c D e Zd ZdZdZdZdZd Zd Zd Z d dZ
d dZ d Z y)
BasePatterna
A pattern is a tree matching pattern.
It looks for a specific node type (token or symbol), and
optionally for a specific content.
This is an abstract base class. There are three concrete
subclasses:
- LeafPattern matches a single leaf node;
- NodePattern matches a single node (usually non-leaf);
- WildcardPattern matches a sequence of nodes of variable length.
Nc J | t usJ d t j | S )z>Constructor that prevents BasePattern from being instantiated.z Cannot instantiate BasePattern)r r r r s r r zBasePattern.__new__ s% +%G'GG%~~c""r c t | j | j | j g}|r|d |d= |r|d | j j
ddj
t t | dS )Nrw rx ry ) r r contentr r rd r} r~ rq )r# r s r rz zBasePattern.__repr__ sd $))$dllDII>tBx'R tBx' >>22DIIc$o4NOOr c | S )z
A subclass can define this as a hook for optimizations.
Returns either self or another node with the same effect.
r r* s r optimizezBasePattern.optimize s r c | j |j | j k7 ry| j ,d}|i }| j || sy|r|j | || j r||| j < y)a#
Does this pattern exactly match a node?
Returns True if it matches, False if not.
If results is not None, it must be a dict which will be
updated with the nodes matching named subpatterns.
Default implementation for non-wildcard patterns.
NFT)r r _submatchupdater )r# rD resultsrs r matchzBasePattern.match sw 99 TYY$))%; << #A">>$* q! 499!%GDII r c J t | dk7 ry| j |d | S )z
Does this pattern exactly match a sequence of nodes?
Default implementation for non-wildcard patterns.
r Fr )r r )r# nodesr s r match_seqzBasePattern.match_seq s' u:?zz%(G,,r c # N K i }|r | j |d | rd|f yyyw)z}
Generator yielding all matches for this pattern.
Default implementation for non-wildcard patterns.
r r N)r )r# r r s r generate_matchesz BasePattern.generate_matches s1
TZZa!,Q$J -5s #%rT )
rd re rf rg r r r r rz r r r r r r r r r s7 DG D#
P 2-r r c $ e Zd ZddZddZddZy) LeafPatternNc |d|cxk r dk sJ | J | | t |t sJ t | || _ || _ || _ y)ap
Initializer. Takes optional type, content, and name.
The type, if given must be a token type (< 256). If not given,
this matches any *leaf* node; the content may still be required.
The content, if given, must be a string.
If a name is given, the matching node is stored in the results
dict under that key.
Nr rp )r4 r3 rq r r r )r# r r r s r rt zLeafPattern.__init__ s] ?s? (D (? (D (? gs+ :T'] :+ r c R t |t syt j | || S )z*Override match() to insist on a leaf node.F)r4 rA r r r# rD r s r r zLeafPattern.match
s$ $% tW55r c 4 | j |j k( S )
Match the pattern's content to the node's children.
This assumes the node type matches and self.content is not None.
Returns True if it matches, False if not.
If results is not None, it must be a dict which will be
updated with the nodes matching named subpatterns.
When returning False, the results dict may still be updated.
)r r r s r r zLeafPattern._submatch s ||tzz))r r rT )rd re rf rt r r r r r r r s (6
*r r c e Zd ZdZddZddZy) NodePatternFNc , | |dk\ sJ | |ot |t rJ t | t | }t | D ]6 \ }}t |t
s J ||f t |t s0d| _ 8 || _ || _ || _
y)ad
Initializer. Takes optional type, content, and name.
The type, if given, must be a symbol type (>= 256). If the
type is None this matches *any* single node (leaf or not),
except if content is not None, in which it only matches
non-leaf nodes that also match the content pattern.
The content, if not None, must be a sequence of Patterns that
must match the node's children exactly. If the content is
given, the type must not be None.
If a name is given, the matching node is stored in the results
dict under that key.
Nrp T) r4 r3 rq r5 rI r WildcardPattern wildcardsr r r )r# r r r rJ items r rt zNodePattern.__init__$ s 3; $ $; !'3/ >g >/7mG$W-4!$ 4?q$i?4 dO4%)DN . r c | j rVt | j |j D ]2 \ }}|t |j k( s ||j | y yt | j t |j k7 ryt
| j |j D ] \ }}|j || r y y)r TF)r r r r6 r r zipr )r# rD r cr
subpatternrO s r r zNodePattern._submatchA s >>( t}}E1DMM** * q) F
t|| DMM 2 2!$T\\4==!A J ##E73 "B r r rT )rd re rf r rt r r r r r r s I:r r c N e Zd ZdZddedfdZd Zd dZd dZd Z d Z
d
Z d Z y)
r a
A wildcard pattern can match zero or more nodes.
This has all the flexibility needed to implement patterns like:
.* .+ .? .{m,n}
(a b c | d e | f)
(...)* (...)+ (...)? (...){m,n}
except it always uses non-greedy matching.
Nr c , d|cxk r|cxk r t k s n J ||f |Vt t t | }t | sJ t | |D ] }t | rJ t | || _ || _ || _ || _ y)a
Initializer.
Args:
content: optional sequence of subsequences of patterns;
if absent, matches one node;
if present, each subsequence is an alternative [*]
min: optional minimum number of times to match, default 0
max: optional maximum number of times to match, default HUGE
name: optional name assigned to this match
[*] Thus, if content is [[a, b, c], [d, e], [f, g, h]] this is
equivalent to (a b c | d e | f g h); if content is None,
this is equivalent to '.' in regular expression terms.
The min and max parameters work as follows:
min=0, max=maxint: .*
min=1, max=maxint: .+
min=0, max=1: .?
min=1, max=1: .
If content is not None, replace the dot with the parenthesized
list of alternatives, e.g. (a b c | d e | f g h)*
r N) HUGEtupler~ r rq r minmaxr )r# r r r r alts r rt zWildcardPattern.__init__k s . C&3&$&2c
2& Cw /0Gw< .g .< 3x*c*x r c d}| j Et | j dk( r-t | j d dk( r| j d d }| j dk( r\| j dk( rM| j t | j
S |)| j
|j
k( r|j
S | j dk rt |t rx|j dk ri| j
|j
k( rPt |j | j |j z | j |j z |j
S | S )z+Optimize certain stacked wildcard patterns.Nr r )r ) r r r r r r r r4 r )r# r s r r zWildcardPattern.optimize s
LL $ "s4<<?';q'@ a +J 88q=TXX]||#" 22%499
+G!**,, HHMj_ E NNa DII $@":#5#5#'88JNN#:#'88JNN#:#-??4
4 r c ( | j |g| S )z'Does this pattern exactly match a node?)r r s r r zWildcardPattern.match s ~~tfg..r c | j | D ]L \ }}|t | k( s|5|j | | j rt | || j < y y)z4Does this pattern exactly match a sequence of nodes?TF)r r r r r5 )r# r r r r s r r zWildcardPattern.match_seq sY ))%0DAqCJ&NN1%yy-1%[ *
1 r c # & K | j bt | j dt t | | j z D ]( }i }| j
r|d| || j
< ||f * y| j
dk( r| j
| yt t d r#t j }t t _ | j |d D ]) \ }}| j
r|d| || j
< ||f + t t d r t _ yy# t $ r@ | j | D ]) \ }}| j
r|d| || j
< ||f + Y ew xY w# t t d r t _ w w xY ww)a"
Generator yielding matches for a sequence of nodes.
Args:
nodes: sequence of nodes
Yields:
(count, results) tuples where:
count: the match comprises nodes[:count];
results: dict containing named submatches.
Nr bare_name getrefcountr )r ranger r r r _bare_name_matcheshasattrrk stderrr _recursive_matches RuntimeError_iterative_matches)r# r countr save_stderrs r r z WildcardPattern.generate_matches sd << txxSUTXX-F)FG99#(%=AdiiLQh H
YY+
%))%0 0 sM* !jj %Z
- $ 7 7q AHE1yy',Ve}$)) (N !B 3
.!,CJ /
# !% 7 7 >HE1yy',Ve}$)) (N !?
# 3
.!,CJ /s= CF>D$ E0 F$AE-*E0 ,E--E0 0 FFc # K t | }d| j k\ rdi f g }| j D ]/ }t || D ] \ }}||f |j ||f 1 |rg }|D ] \ }} ||k s || j
k s | j D ]b }t |||d D ]N \ }
} |
dkD s i }|j
| |j
| ||
z |f |j ||
z |f P d |}|ryyw)z(Helper to iteratively yield the matches.r N)r r r r r8 r r ) r# r nodelenr r r r new_resultsc0r0c1r1s r r z"WildcardPattern._iterative_matches s e* =R%K<r s 3
6n-6 n-`k4 k\L 4 L \2&S& Sl)*+ )*X:+ :zy)k y)x [ F%r