cjsk d Z dZddlZddlZddlZddlZddlZddlZddlZddl m
Z
ddl m Z m
Z
mZ ddlmZ ddlmZmZ dd lmZ dd
Z G d d e Zd
Zd Zd Zd Z d Z G d de Z G d de Z G d de Z! G d de Z"y)zRefactoring framework.
Used as a main program, this can refactor any number of files and/or
recursively descend down directories. Imported as a module, this
provides infrastructure to write your own refactoring tool.
z#Guido van Rossum N)chain )drivertokenizetoken) find_root)pytreepygram) btm_matcherc t | g g dg }g }t j |j D ]0 \ }}}|j d s|r|dd }|j | 2 |S )zEReturn a sorted list of all available fix names in the given package.*fix_ N)
__import__pkgutil iter_modules__path__
startswithappend) fixer_pkg
remove_prefixpkg fix_namesfindernameispkgs )/usr/lib64/python3.12/lib2to3/refactor.pyget_all_fix_namesr sj
YB
.CI & 3 3CLL Ae ??6 " ABx T " B
c e Zd Zy)
_EveryNodeN__name__
__module__ __qualname__ r r r! r! + r r! c t | t j t j f r | j t
| j hS t | t j r'| j rt | j S t
t | t j r>t }| j D ]# }|D ] }|j t | % |S t d| z )zf Accepts a pytree Pattern Node and returns a set
of the pattern types which will match first. z$Oh no! I don't understand pattern %s)
isinstancer NodePattern LeafPatterntyper! NegatedPatterncontent_get_head_typesWildcardPatternsetupdate Exception)patrpxs r r/ r/ / s #**F,>,>?@ 88 z#v,,- ;;"3;;/ /#v--. EA +,
:SA
BBr c X t j t }g }| D ]| }|j r2 t |j }|D ] }|| j | A|j ||j j | l|j | ~ t t j j j t j j D ] }|| j | t | S # t $ r |j | Y
w xY w)z^ Accepts a list of fixers and returns a dictionary
of head node type --> fixer list. ) collections defaultdictlistpatternr/ r r! _accept_typer r
python_grammar
symbol2numbervaluestokensextenddict)
fixer_list
head_nodeseveryfixerheads node_types r _get_headnode_dictrJ K s ((.J E ==
8'
6 "'I y)007 "' !!-5--.55e< U# 600>>EEG!00779 9 $$U+9
$ U#
$s D D)(D)c L t | d D cg c]
}| dz |z c}S c c}w )zN
Return the fully qualified names for fixers in the package pkg_name.
F.)r )pkg_namefix_names r get_fixers_from_packagerO d s=
.h > @ >
sNX % > @ @ @s !c | S Nr& )objs r _identityrS k s Jr c | d}t j t j | j fd}t t j t j t j h }t } | \ }}||v r|t j k( r|rnd}n|t j k( r|dk( r | \ }}|t j k7 s|dk7 rn | \ }}|t j k7 s|dk7 rn | \ }}|t j k( r|dk( r
| \ }}|t j k( rT|j | | \ }}|t j k7 s|dk7 rn | \ }}|t j k( rRnnt | S # t $ r
Y t | S w xY w) NFc . t } | d | d fS )Nr r )next)tokgens r advancez(_detect_future_features..advancer s 3i1vs1v~ r Tfrom
__future__import(,)r generate_tokensioStringIOreadline frozensetr NEWLINENLCOMMENTr1 STRINGNAMEOPadd
StopIteration)sourcehave_docstringrY ignorefeaturestpvaluerX s @r _detect_future_featuresrr o s N
"
"2;;v#6#?#?
@C
x{{EMMB
CFuH
IBV|u||#!!%uzz!evo #I E#u '< #I E#u'8 #I E>esl ' IBEJJ& LL' ' IBUXX~# ' IB EJJ& 3 8 X
X
s >DF% F% % F;:F;c e Zd ZdZy)
FixerErrorz A fixer could not be loaded.N)r# r$ r% __doc__r& r r rt rt s &r rt c e Zd ZddddZdZdZd dZd Zd Zd Z d
Z
d Z d d Z d d
Z
d Zd dZd Zd dZd Zd Z d!dZd"dZdZdZd Zd Zd Zd Zd Z d Z y)#RefactoringToolF)print_function
exec_functionwrite_unchanged_filesFixr Nc || _ |xs g | _ | j j | _ || j j | t j j | _ | j d r| j j d= n&| j d r| j j d= | j j d | _ g | _ t j d | _ g | _ d| _ t% j& | j t( j* | j | _ | j- \ | _ | _ g | _ t5 j6 | _ g | _ g | _ t? | j0 | j. D ]~ }|j@ r | j8 jC | +|| j. v r | j: jE | U|| j0 v sd| j< jE | tG | j: | _$ tG | j< | _% y)
zInitializer.
Args:
fixer_names: a list of fixers to import
options: a dict with configuration.
explicit: a list of fixers to run even if they are explicit.
Nrx printry execrz rw F)convertlogger)&fixersexplicit_default_optionscopyoptionsr2 r
r> grammarkeywordsgetrz errorslogging getLoggerr fixer_logwroter Driverr r
get_fixers pre_order
post_orderfilesbm
BottomMatcherBM
bmi_pre_orderbmi_post_orderr
BM_compatible add_fixerr rJ bmi_pre_order_headsbmi_post_order_heads)self fixer_namesr r rG s r __init__zRefactoringTool.__init__ s " B
,,113 LL ( ,,113 <<( ) %%g.
\\/
* %%f-
&*\\%5%56M%N" ''(9:
mmDLL,2NN+/;;8 +///*;'
""$ 4??DNN;E""!!%( $..(""))%0$//)##**51 < $6d6H6H#I $6t7J7J$K!r c g }g }| j D ]w }t |i i dg }|j dd d }|j | j r|t | j d }|j
d }| j dj |D cg c] }|j c} z } t || } | | j | j }
|
j r0| j d ur"|| j vr| j! d
| | j# d | |
j$ d k( r|j' |
@|
j$ d
k( r|j' |
bt d|
j$ z t) j* d } |j- | |j- | ||fS c c}w # t $ r t d|d| dw xY w)a Inspects the options to load the requested patterns and handlers.
Returns:
(pre_order, post_order), where pre_order is the list of fixers that
want a pre-order AST traversal, and post_order is the list that want
post-order traversal.
r
rL r N_ z Can't find TzSkipping optional fixer: %szAdding transformation: %sprepostzIllegal fixer order: %r run_orderkey)r r rsplitr FILE_PREFIXlensplit CLASS_PREFIXjointitlegetattrAttributeErrorrt r r r log_message log_debugorderr operator
attrgettersort) r pre_order_fixerspost_order_fixers fix_mod_pathmodrN partsr6
class_name fix_classrG key_funcs r r zRefactoringTool.get_fixers s KKL \2rC59C#**3226H""4#3#34#C(8(8$9$:; NN3'E **RWW5OAaggi5O-PPJ
X #C 4 dllDNN;E~~$--t";
5 !>I NN6 A{{e# ''.&!((/ !:U[[!HII/ (2 &&{3 ( + 8 , "344- 6P "
X x!LMSWW
Xs G 8 G G*c )z Called when an error occurs.r& )r msgargskwdss r log_errorzRefactoringTool.log_error s
r c H |r||z }| j j | y)zHook to log a message.N)r infor r r s r r zRefactoringTool.log_message s *C r c H |r||z }| j j | y rQ )r debugr s r r zRefactoringTool.log_debug
s *C # r c y)zTCalled with the old version, new version, and filename of a
refactored file.Nr& )r old_textnew_textfilenameequals r print_outputz RefactoringTool.print_output s
r c |D ]H }t j j | r| j ||| 6| j ||| J y)z)Refactor a list of files and directories.N)ospathisdir refactor_dir
refactor_file)r itemswrite
doctests_only dir_or_files r refactorzRefactoringTool.refactor sB !Kww}}[)!!+umD"";}E !r c t j dz }t j | D ] \ }}}| j d| |j |j |D ]m }|j d rt j j | d |k( s;t j j || } | j | || o |D
cg c] }
|
j d r|
c}
|dd yc c}
w )zDescends down a directory and refactor every Python file found.
Python files are assumed to have a .py extension.
Files and subdirectories starting with '.' are skipped.
pyzDescending into %srL r N)
r extsepwalkr r r r splitextr r ) r dir_namer r py_extdirpathdirnames filenamesr fullnamedns r r z RefactoringTool.refactor_dir s T!,.GGH,= (GXy NN/ 9 MMO NN ! ,GG$$T*1-7 !ww||GT :H&&x
F " )1K"
c8J2KHQK -> Ls C</C<c t |d } t j |j
d }|j
t j |d|d 5 }|j |fcddd S # t $ r }| j d|| Y d}~yd}~ww xY w# |j
w xY w# 1 sw Y yxY w) zG
Do our best to decode a Python source file correctly.
rbzCan't open %s: %sNNNr r5 r encodingnewline) openOSErrorr r detect_encodingrb closer` read)r r ferrr s r _read_python_sourcez#RefactoringTool._read_python_source4 s Xt$A //
;A>H
GGI
WWXsXr
Ba668X% C
B NN.# >
GGI
B
Bs. A6 "B B46 B ?BB B14B=c | j | \ }}|y|dz
}|r^| j d| | j || }| j s||k7 r| j ||||| y| j d| y| j || }| j s|r.|j r"| j t | dd ||| y| j d| y)zRefactors a file.N
zRefactoring doctests in %szNo doctest changes in %sr )r r zNo changes in %s)r r refactor_docstringrz processed_filerefactor_string was_changedstr)r r r r inputr outputtrees r r z RefactoringTool.refactor_fileD s 228<x =
NN7 B,,UH=F))Vu_##FHeUHM 98D''x8D))dt7G7G##CIcrNH*/( $ D 18 "&,,DKK
#',,DKK s) B C* (C%C- %C**C- - D
c t j j }|rZ| j d | j |d }| j
s||k7 r| j
|d| y | j d y | j |d }| j
s|r)|j r | j
t | d| y | j d y )Nz Refactoring doctests in stdinzzNo doctest changes in stdinzNo changes in stdin)
sysstdinr r r rz r r r r )r r r r r s r refactor_stdinz RefactoringTool.refactor_stdinv s NN: ;,,UI>F))Vu_##FIu= <=''y9D))dt7G7G##CIy%@ 45r c t | j | j D ] }|j || | j | j
|j | j | j |j | j j |j }t |j r| j j D ] }||v s || s|| j t j j d |j" r-|| j t j j$ t' || D ] }||| v r|| j) | t+ | |j. r||j. v rF|j1 | }|sZ|j3 || }|o|j5 | |j D ]0 }|j. sg |_ |j. j7 | 2 | j j |j }|D ]" } | |vrg || < || j9 || $ t |j rt | j | j D ] }|j; || |j< S # t, $ r Y w xY w)a Refactors a parse tree (modifying the tree in place).
For compatible patterns the bottom matcher module is
used. Otherwise the tree is traversed node-to-node for
matches.
Args:
tree: a pytree.Node instance representing the root of the tree
to be refactored.
name: a human-readable name for this tree.
Returns:
True if the tree was modified, False otherwise.
T)r reverser ) r r r
start_tree traverse_byr r r runleavesanyr@ r r r Basedepthkeep_line_order
get_linenor; remover
ValueErrorfixers_appliedmatch transformreplacer rB finish_treer )
r r r rG match_setnoderesultsnew new_matchesfxrs
r r z RefactoringTool.refactor_tree sv 4>>4??;E T4 ( <
114>>3CD 22DOO4EF GGKK
. )""$% I%)E*: e$))fkk.?.?)N ,, "%(--&++2H2H-I $Yu%5 6 9U#33 %e , 3 3D 9% %dO ..5D+>>@(;$($7$7$>$>u$E -= /3ggkk#**,.G +6C+.)+;79 #$-cN$9$9+c:J$K ,7A !7 ( )""$%b 4>>4??;E dD ) < E * % %%s K K-,K-c |sy|D ]R }||j D ]> }|j | }|s|j || }|,|j | |}@ T y)a Traverse an AST, applying a set of fixers to each node.
This is a helper method for refactor_tree().
Args:
fixers: a list of fixer instances.
traversal: a generator that yields AST nodes.
Returns:
None
N)r, r r r )r r traversalr rG r r s r r zRefactoringTool.traverse_by s^ D * ++d+ //$8C S) "
+ r c 2 | j j | || j | d }|y||k( }| j |||| |r | j d| | j
sy|r| j
|||| y| j d| y)zR
Called when a file has been refactored and there may be changes.
Nr zNo changes to %szNot writing changes to %s)r r r r r rz
write_file)r r r r r r r s r r z RefactoringTool.processed_file s
(# //9! NN-x 8-- OOHh( C NN6 Ar c ` t j |d|d }|5 |j | ddd | j d| d| _ y# t $ r }| j d|| Y d}~yd}~ww xY w# t $ r }| j d|| Y d}~ld}~ww xY w# 1 sw Y uxY w) zWrites a string to a file.
It first shows a unified diff between the old text and the new text, and
then rewrites the file; the latter is only done if the write option is
set.
wr r zCan't create %s: %sNzCan't write %s: %szWrote changes to %sT)r` r r r r r r )r r r r r fpr s r r$ zRefactoringTool.write_file s 32FB
D"
,h7
NN0(C @
D 3XsCC
D RsE A B$A; A8A33A8; B!B B$ B!!B$$B-z>>> z... c g }d}d}d}d}|j d D ] }|dz
}|j j | j rK|#|j | j |||| |}|g}|j
| j } |d| }}|S|j || j z s#||| j j z dz k( r|j | |#|j | j |||| d}d}|j | |#|j | j |||| dj | S )a Refactors a docstring, looking for doctests.
This returns a modified version of the input string. It looks
for doctests, which start with a ">>>" prompt, and may be
continued with "..." prompts, as long as the "..." is indented
the same as the ">>>".
(Unfortunately we can't use the doctest module's parser,
since, like most parsers, it is not geared towards preserving
the original source.)
Nr Tkeependsr r r )
splitlineslstripr PS1rB refactor_doctestfindPS2rstripr r )
r r r resultblock block_linenoindentlinenolineis
r r z"RefactoringTool.refactor_docstring sg $$d$3D aKF{{}''1$MM$"7"7|8>#J K % IIdhh' bq$??6DHH#456DHHOO$55<< T"$MM$"7"7|8>#J K
d#) 4* MM$//|06B
Cwwv r c | j ||| }| j || rt | j d }|d|dz
||dz
d }} |d j d s
|dxx dz
cc< || j z |j! d z g}|r#||D cg c] }|| j" z |z c}z
}|S # t $ r}| j j t j
r(|D ]# }| j
d|j d % | j d|||j j | |cY d}~S d}~ww xY wc c}w )
zRefactors one doctest.
A doctest is given as a block of lines, the first of which starts
with ">>>" (possibly indented), while the remaining lines start
with "..." (identically indented).
z
Source: %sr z+Can't parse docstring in %s line %s: %s: %sNTr) r r r ) parse_blockr3 r isEnabledForr DEBUGr r1 r r r# r r r+ endswithr- popr0 )
r r3 r6 r5 r r r r7 r clippeds
r r. z RefactoringTool.refactor_doctestD sC ##E66:D dH -d)&&&5C yq>3vaxy>SGr7##D)B4dhh&34EsCst&488+d2sCC # {{''
6 !DNN<T1BC " NNH#VS]]-C-CS
JL
Ds$ B< E< E A;E E E c X | j rd}nd}| j s| j d| n4| j d| | j D ] }| j | | j r3| j d | j D ] }| j | | j r{t | j dk( r| j d n%| j dt | j | j D ] \ }}} | j |g|i | y y ) Nwerez
need to bezNo files %s modified.zFiles that %s modified:z$Warnings/messages while refactoring:r zThere was 1 error:zThere were %d errors:)r r r r r r )r rA filemessager r r s r summarizezRefactoringTool.summarizea s ::D Dzz 4d ; 6 =
& # >> C D >> ) * ;;4;; 1$ !56 !8#dkk:JK#';;T4 4t4t4 $/ r c | | j j | j ||| }t |_ |S )zParses a block into a tree.
This is necessary to get correct line number / offset information
in the parser diagnostics and embedded into the parse tree.
)r parse_tokens wrap_toksrc r )r r3 r6 r5 r s r r: zRefactoringTool.parse_blockx s4 {{''uff(MN ({ r c # K t j | j || j }|D ]+ \ }}\ }}\ } }
} ||dz
z
}| |dz
z
} ||||f| |
f| f - yw)z;Wraps a tokenize stream to systematically modify start/end.r N)r r_ gen_lines__next__) r r3 r6 r5 rA r, rq line0col0line1col1 line_texts r rG zRefactoringTool.wrap_toks s} ))$..*G*P*PQDJ @D% %
y VaZ E VaZ E t}udmYF F EKs A!A#c # K || j z }|| j z }|}|D ]R }|j | r|t | d n,||j dz k( rd nt d|d| |}T d w)zGenerates lines as expected by tokenize from a list of lines.
This strips the first len(indent + self.PS1) characters off each line.
Nr zline=z , prefix=r )r- r0 r r r1 AssertionError)r r3 r5 prefix1prefix2prefixr7 s r rI zRefactoringTool.gen_lines s
488#488#Dv&3v;<(( 4//
$T6%JKK F H s A>B r )FF)F)NFNrQ ) r# r$ r% r r r r r r r r r r r r r r r r r r r$ r- r0 r r. rD r: rG rI r& r r rw rw s +0).279 LK3Ln&5P
FL(& =.66 M ^#. GL $B** C
C) V:5. G r rw c e Zd Zy)MultiprocessingUnsupportedNr" r& r r rV rV r' r rV c B e Zd Z fdZ d fd Z fdZ fdZ xZS )MultiprocessRefactoringToolc H t t |
|i | d | _ d | _ y rQ )superrX r queue output_lockr r kwargsr s r r z$MultiprocessRefactoringTool.__init__ s'
)494J6J
r c |dk( rt t | ||| S dd l}| j t d |j | _ |j | _
t | D cg c] }|j | j }} |D ] }|j t t | ||| | j j t | D ] }| j j! d |D ]# }|j# s|j % d | _ y # t $ r t
w xY wc c}w # | j j t | D ] }| j j! d |D ]# }|j# s|j % d | _ w xY w)Nr r z already doing multiple processes)target)rZ rX r multiprocessing ImportErrorrV r[ RuntimeError
JoinableQueueLockr\ rangeProcess_childstartr putis_alive)
r r r r
num_processesra r8 processesr6 r s
r r z$MultiprocessRefactoringTool.refactor s A 4dDum-
- - " :: ! AB B$224
*//1 #M 24 2 %,,DKK,@ 2 4 -t =eU>K
M
JJOO =)
t$ * ::<FFH DJ) -, , - 4
JJOO =)
t$ * ::<FFH DJs$ D6 /#E ,E 6EAG*Gc | j j }|Q|\ }} t t | |i | | j j | j j }|Py y # | j j w xY wrQ )r[ r rZ rX r task_done)r taskr r^ r s r rh z"MultiprocessRefactoringTool._child s zz~~ LD&
'14F% #%
$$&::>>#D
$$&s A0 0 B c ~ | j | j j ||f y t t | |i |S rQ )r[ rj rZ rX r r] s r r z)MultiprocessRefactoringTool.refactor_file sA :: ! JJNND&> *4dI! !
!r )FFr )r# r$ r% r r rh r
__classcell__)r s @r rX rX s$
:? : $! !r rX )T)#ru
__author__r` r r r r r r9 itertoolsr pgen2r r r
fixer_utilr r r r
r r r r3 r! r/ rJ rO rS rr rt objectrw rV rX r& r r rx s 3
+ * ! C8 2@% P' 'Ff FR 4!/ 4!r