Coverage for C:\leo.repo\leo-editor\leo\core\leoAst.py: 100%

Shortcuts on this page

r m x   toggle line displays

j k   next/prev highlighted chunk

0   (zero) top of page

1   (one) first highlighted chunk

1987 statements  

1# -*- coding: utf-8 -*- 

2#@+leo-ver=5-thin 

3#@+node:ekr.20141012064706.18389: * @file leoAst.py 

4#@@first 

5# This file is part of Leo: https://leoeditor.com 

6# Leo's copyright notice is based on the MIT license: http://leoeditor.com/license.html 

7#@+<< docstring >> 

8#@+node:ekr.20200113081838.1: ** << docstring >> (leoAst.py) 

9""" 

10leoAst.py: This file does not depend on Leo in any way. 

11 

12The classes in this file unify python's token-based and ast-based worlds by 

13creating two-way links between tokens in the token list and ast nodes in 

14the parse tree. For more details, see the "Overview" section below. 

15 

16 

17**Stand-alone operation** 

18 

19usage: 

20 leoAst.py --help 

21 leoAst.py [--fstringify | --fstringify-diff | --orange | --orange-diff] PATHS 

22 leoAst.py --py-cov [ARGS] 

23 leoAst.py --pytest [ARGS] 

24 leoAst.py --unittest [ARGS] 

25 

26examples: 

27 --py-cov "-f TestOrange" 

28 --pytest "-f TestOrange" 

29 --unittest TestOrange 

30 

31positional arguments: 

32 PATHS directory or list of files 

33 

34optional arguments: 

35 -h, --help show this help message and exit 

36 --fstringify leonine fstringify 

37 --fstringify-diff show fstringify diff 

38 --orange leonine Black 

39 --orange-diff show orange diff 

40 --py-cov run pytest --cov on leoAst.py 

41 --pytest run pytest on leoAst.py 

42 --unittest run unittest on leoAst.py 

43 

44 

45**Overview** 

46 

47leoAst.py unifies python's token-oriented and ast-oriented worlds. 

48 

49leoAst.py defines classes that create two-way links between tokens 

50created by python's tokenize module and parse tree nodes created by 

51python's ast module: 

52 

53The Token Order Generator (TOG) class quickly creates the following 

54links: 

55 

56- An *ordered* children array from each ast node to its children. 

57 

58- A parent link from each ast.node to its parent. 

59 

60- Two-way links between tokens in the token list, a list of Token 

61 objects, and the ast nodes in the parse tree: 

62 

63 - For each token, token.node contains the ast.node "responsible" for 

64 the token. 

65 

66 - For each ast node, node.first_i and node.last_i are indices into 

67 the token list. These indices give the range of tokens that can be 

68 said to be "generated" by the ast node. 

69 

70Once the TOG class has inserted parent/child links, the Token Order 

71Traverser (TOT) class traverses trees annotated with parent/child 

72links extremely quickly. 

73 

74 

75**Applicability and importance** 

76 

77Many python developers will find asttokens meets all their needs. 

78asttokens is well documented and easy to use. Nevertheless, two-way 

79links are significant additions to python's tokenize and ast modules: 

80 

81- Links from tokens to nodes are assigned to the nearest possible ast 

82 node, not the nearest statement, as in asttokens. Links can easily 

83 be reassigned, if desired. 

84 

85- The TOG and TOT classes are intended to be the foundation of tools 

86 such as fstringify and black. 

87 

88- The TOG class solves real problems, such as: 

89 https://stackoverflow.com/questions/16748029/ 

90 

91**Known bug** 

92 

93This file has no known bugs *except* for Python version 3.8. 

94 

95For Python 3.8, syncing tokens will fail for function call such as: 

96 

97 f(1, x=2, *[3, 4], y=5) 

98 

99that is, for calls where keywords appear before non-keyword args. 

100 

101There are no plans to fix this bug. The workaround is to use Python version 

1023.9 or above. 

103 

104 

105**Figures of merit** 

106 

107Simplicity: The code consists primarily of a set of generators, one 

108for every kind of ast node. 

109 

110Speed: The TOG creates two-way links between tokens and ast nodes in 

111roughly the time taken by python's tokenize.tokenize and ast.parse 

112library methods. This is substantially faster than the asttokens, 

113black or fstringify tools. The TOT class traverses trees annotated 

114with parent/child links even more quickly. 

115 

116Memory: The TOG class makes no significant demands on python's 

117resources. Generators add nothing to python's call stack. 

118TOG.node_stack is the only variable-length data. This stack resides in 

119python's heap, so its length is unimportant. In the worst case, it 

120might contain a few thousand entries. The TOT class uses no 

121variable-length data at all. 

122 

123**Links** 

124 

125Leo... 

126Ask for help: https://groups.google.com/forum/#!forum/leo-editor 

127Report a bug: https://github.com/leo-editor/leo-editor/issues 

128leoAst.py docs: http://leoeditor.com/appendices.html#leoast-py 

129 

130Other tools... 

131asttokens: https://pypi.org/project/asttokens 

132black: https://pypi.org/project/black/ 

133fstringify: https://pypi.org/project/fstringify/ 

134 

135Python modules... 

136tokenize.py: https://docs.python.org/3/library/tokenize.html 

137ast.py https://docs.python.org/3/library/ast.html 

138 

139**Studying this file** 

140 

141I strongly recommend that you use Leo when studying this code so that you 

142will see the file's intended outline structure. 

143 

144Without Leo, you will see only special **sentinel comments** that create 

145Leo's outline structure. These comments have the form:: 

146 

147 `#@<comment-kind>:<user-id>.<timestamp>.<number>: <outline-level> <headline>` 

148""" 

149#@-<< docstring >> 

150#@+<< imports >> 

151#@+node:ekr.20200105054219.1: ** << imports >> (leoAst.py) 

152import argparse 

153import ast 

154import codecs 

155import difflib 

156import glob 

157import io 

158import os 

159import re 

160import sys 

161import textwrap 

162import tokenize 

163import traceback 

164from typing import Any, Callable, Dict, Generator, List, Optional, Tuple, Union 

165#@-<< imports >> 

166Node = ast.AST 

167ActionList = List[Tuple[Callable, Any]] 

168v1, v2, junk1, junk2, junk3 = sys.version_info 

169py_version = (v1, v2) 

170 

171# Async tokens exist only in Python 3.5 and 3.6. 

172# https://docs.python.org/3/library/token.html 

173has_async_tokens = (3, 5) <= py_version <= (3, 6) 

174 

175# has_position_only_params = (v1, v2) >= (3, 8) 

176#@+others 

177#@+node:ekr.20191226175251.1: ** class LeoGlobals 

178#@@nosearch 

179 

180 

181class LeoGlobals: # pragma: no cover 

182 """ 

183 Simplified version of functions in leoGlobals.py. 

184 """ 

185 

186 total_time = 0.0 # For unit testing. 

187 

188 #@+others 

189 #@+node:ekr.20191226175903.1: *3* LeoGlobals.callerName 

190 def callerName(self, n: int) -> str: 

191 """Get the function name from the call stack.""" 

192 try: 

193 f1 = sys._getframe(n) 

194 code1 = f1.f_code 

195 return code1.co_name 

196 except Exception: 

197 return '' 

198 #@+node:ekr.20191226175426.1: *3* LeoGlobals.callers 

199 def callers(self, n: int=4) -> str: 

200 """ 

201 Return a string containing a comma-separated list of the callers 

202 of the function that called g.callerList. 

203 """ 

204 i, result = 2, [] 

205 while True: 

206 s = self.callerName(n=i) 

207 if s: 

208 result.append(s) 

209 if not s or len(result) >= n: 

210 break 

211 i += 1 

212 return ','.join(reversed(result)) 

213 #@+node:ekr.20191226190709.1: *3* leoGlobals.es_exception & helper 

214 def es_exception(self, full: bool=True) -> Tuple[str, int]: 

215 typ, val, tb = sys.exc_info() 

216 for line in traceback.format_exception(typ, val, tb): 

217 print(line) 

218 fileName, n = self.getLastTracebackFileAndLineNumber() 

219 return fileName, n 

220 #@+node:ekr.20191226192030.1: *4* LeoGlobals.getLastTracebackFileAndLineNumber 

221 def getLastTracebackFileAndLineNumber(self) -> Tuple[str, int]: 

222 typ, val, tb = sys.exc_info() 

223 if typ == SyntaxError: 

224 # IndentationError is a subclass of SyntaxError. 

225 # SyntaxError *does* have 'filename' and 'lineno' attributes. 

226 return val.filename, val.lineno 

227 # 

228 # Data is a list of tuples, one per stack entry. 

229 # The tuples have the form (filename, lineNumber, functionName, text). 

230 data = traceback.extract_tb(tb) 

231 item = data[-1] # Get the item at the top of the stack. 

232 filename, n, functionName, text = item 

233 return filename, n 

234 #@+node:ekr.20200220065737.1: *3* LeoGlobals.objToString 

235 def objToString(self, obj: Any, tag: str=None) -> str: 

236 """Simplified version of g.printObj.""" 

237 result = [] 

238 if tag: 

239 result.append(f"{tag}...") 

240 if isinstance(obj, str): 

241 obj = g.splitLines(obj) 

242 if isinstance(obj, list): 

243 result.append('[') 

244 for z in obj: 

245 result.append(f" {z!r}") 

246 result.append(']') 

247 elif isinstance(obj, tuple): 

248 result.append('(') 

249 for z in obj: 

250 result.append(f" {z!r}") 

251 result.append(')') 

252 else: 

253 result.append(repr(obj)) 

254 result.append('') 

255 return '\n'.join(result) 

256 #@+node:ekr.20220327132500.1: *3* LeoGlobals.pdb 

257 def pdb(self) -> None: 

258 import pdb as _pdb 

259 # pylint: disable=forgotten-debug-statement 

260 _pdb.set_trace() 

261 #@+node:ekr.20191226190425.1: *3* LeoGlobals.plural 

262 def plural(self, obj: Any) -> str: 

263 """Return "s" or "" depending on n.""" 

264 if isinstance(obj, (list, tuple, str)): 

265 n = len(obj) 

266 else: 

267 n = obj 

268 return '' if n == 1 else 's' 

269 #@+node:ekr.20191226175441.1: *3* LeoGlobals.printObj 

270 def printObj(self, obj: Any, tag: str=None) -> None: 

271 """Simplified version of g.printObj.""" 

272 print(self.objToString(obj, tag)) 

273 #@+node:ekr.20220327120618.1: *3* LeoGlobals.shortFileName 

274 def shortFileName(self, fileName: str) -> str: 

275 """Return the base name of a path.""" 

276 return os.path.basename(fileName) if fileName else '' 

277 #@+node:ekr.20191226190131.1: *3* LeoGlobals.splitLines 

278 def splitLines(self, s: str) -> List[str]: 

279 """Split s into lines, preserving the number of lines and 

280 the endings of all lines, including the last line.""" 

281 # g.stat() 

282 if s: 

283 return s.splitlines(True) 

284 # This is a Python string function! 

285 return [] 

286 #@+node:ekr.20191226190844.1: *3* LeoGlobals.toEncodedString 

287 def toEncodedString(self, s: Any, encoding: str='utf-8') -> bytes: 

288 """Convert unicode string to an encoded string.""" 

289 if not isinstance(s, str): 

290 return s 

291 try: 

292 s = s.encode(encoding, "strict") 

293 except UnicodeError: 

294 s = s.encode(encoding, "replace") 

295 print(f"toEncodedString: Error converting {s!r} to {encoding}") 

296 return s 

297 #@+node:ekr.20191226190006.1: *3* LeoGlobals.toUnicode 

298 def toUnicode(self, s: Any, encoding: str='utf-8') -> str: 

299 """Convert bytes to unicode if necessary.""" 

300 tag = 'g.toUnicode' 

301 if isinstance(s, str): 

302 return s 

303 if not isinstance(s, bytes): 

304 print(f"{tag}: bad s: {s!r}") 

305 return '' 

306 b: bytes = s 

307 try: 

308 s2 = b.decode(encoding, 'strict') 

309 except(UnicodeDecodeError, UnicodeError): 

310 s2 = b.decode(encoding, 'replace') 

311 print(f"{tag}: unicode error. encoding: {encoding!r}, s2:\n{s2!r}") 

312 g.trace(g.callers()) 

313 except Exception: 

314 g.es_exception() 

315 print(f"{tag}: unexpected error! encoding: {encoding!r}, s2:\n{s2!r}") 

316 g.trace(g.callers()) 

317 return s2 

318 #@+node:ekr.20191226175436.1: *3* LeoGlobals.trace 

319 def trace(self, *args: Any) -> None: 

320 """Print a tracing message.""" 

321 # Compute the caller name. 

322 try: 

323 f1 = sys._getframe(1) 

324 code1 = f1.f_code 

325 name = code1.co_name 

326 except Exception: 

327 name = '' 

328 print(f"{name}: {' '.join(str(z) for z in args)}") 

329 #@+node:ekr.20191226190241.1: *3* LeoGlobals.truncate 

330 def truncate(self, s: str, n: int) -> str: 

331 """Return s truncated to n characters.""" 

332 if len(s) <= n: 

333 return s 

334 s2 = s[: n - 3] + f"...({len(s)})" 

335 return s2 + '\n' if s.endswith('\n') else s2 

336 #@-others 

337#@+node:ekr.20200702114522.1: ** leoAst.py: top-level commands 

338#@+node:ekr.20200702114557.1: *3* command: fstringify_command 

339def fstringify_command(files: List[str]) -> None: 

340 """ 

341 Entry point for --fstringify. 

342 

343 Fstringify the given file, overwriting the file. 

344 """ 

345 for filename in files: # pragma: no cover 

346 if os.path.exists(filename): 

347 print(f"fstringify {filename}") 

348 Fstringify().fstringify_file_silent(filename) 

349 else: 

350 print(f"file not found: {filename}") 

351#@+node:ekr.20200702121222.1: *3* command: fstringify_diff_command 

352def fstringify_diff_command(files: List[str]) -> None: 

353 """ 

354 Entry point for --fstringify-diff. 

355 

356 Print the diff that would be produced by fstringify. 

357 """ 

358 for filename in files: # pragma: no cover 

359 if os.path.exists(filename): 

360 print(f"fstringify-diff {filename}") 

361 Fstringify().fstringify_file_diff(filename) 

362 else: 

363 print(f"file not found: {filename}") 

364#@+node:ekr.20200702115002.1: *3* command: orange_command 

365def orange_command(files: List[str]) -> None: 

366 

367 for filename in files: # pragma: no cover 

368 if os.path.exists(filename): 

369 print(f"orange {filename}") 

370 Orange().beautify_file(filename) 

371 else: 

372 print(f"file not found: {filename}") 

373#@+node:ekr.20200702121315.1: *3* command: orange_diff_command 

374def orange_diff_command(files: List[str]) -> None: 

375 

376 for filename in files: # pragma: no cover 

377 if os.path.exists(filename): 

378 print(f"orange-diff {filename}") 

379 Orange().beautify_file_diff(filename) 

380 else: 

381 print(f"file not found: {filename}") 

382#@+node:ekr.20160521104628.1: ** leoAst.py: top-level utils 

383if 1: # pragma: no cover 

384 #@+others 

385 #@+node:ekr.20200702102239.1: *3* function: main (leoAst.py) 

386 def main() -> None: 

387 """Run commands specified by sys.argv.""" 

388 description = textwrap.dedent("""\ 

389 leo-editor/leo/unittests/core/test_leoAst.py contains unit tests (100% coverage). 

390 """) 

391 parser = argparse.ArgumentParser(description=description, formatter_class=argparse.RawTextHelpFormatter) 

392 parser.add_argument('PATHS', nargs='*', help='directory or list of files') 

393 group = parser.add_mutually_exclusive_group(required=False) # Don't require any args. 

394 add = group.add_argument 

395 add('--fstringify', dest='f', action='store_true', help='leonine fstringify') 

396 add('--fstringify-diff', dest='fd', action='store_true', help='show fstringify diff') 

397 add('--orange', dest='o', action='store_true', help='leonine Black') 

398 add('--orange-diff', dest='od', action='store_true', help='show orange diff') 

399 args = parser.parse_args() 

400 files = args.PATHS 

401 if len(files) == 1 and os.path.isdir(files[0]): 

402 files = glob.glob(f"{files[0]}{os.sep}*.py") 

403 if args.f: 

404 fstringify_command(files) 

405 if args.fd: 

406 fstringify_diff_command(files) 

407 if args.o: 

408 orange_command(files) 

409 if args.od: 

410 orange_diff_command(files) 

411 #@+node:ekr.20200107114409.1: *3* functions: reading & writing files 

412 #@+node:ekr.20200218071822.1: *4* function: regularize_nls 

413 def regularize_nls(s: str) -> str: 

414 """Regularize newlines within s.""" 

415 return s.replace('\r\n', '\n').replace('\r', '\n') 

416 #@+node:ekr.20200106171502.1: *4* function: get_encoding_directive 

417 # This is the pattern in PEP 263. 

418 encoding_pattern = re.compile(r'^[ \t\f]*#.*?coding[:=][ \t]*([-_.a-zA-Z0-9]+)') 

419 

420 def get_encoding_directive(bb: bytes) -> str: 

421 """ 

422 Get the encoding from the encoding directive at the start of a file. 

423 

424 bb: The bytes of the file. 

425 

426 Returns the codec name, or 'UTF-8'. 

427 

428 Adapted from pyzo. Copyright 2008 to 2020 by Almar Klein. 

429 """ 

430 for line in bb.split(b'\n', 2)[:2]: 

431 # Try to make line a string 

432 try: 

433 line2 = line.decode('ASCII').strip() 

434 except Exception: 

435 continue 

436 # Does the line match the PEP 263 pattern? 

437 m = encoding_pattern.match(line2) 

438 if not m: 

439 continue 

440 # Is it a known encoding? Correct the name if it is. 

441 try: 

442 c = codecs.lookup(m.group(1)) 

443 return c.name 

444 except Exception: 

445 pass 

446 return 'UTF-8' 

447 #@+node:ekr.20200103113417.1: *4* function: read_file 

448 def read_file(filename: str, encoding: str='utf-8') -> Optional[str]: 

449 """ 

450 Return the contents of the file with the given name. 

451 Print an error message and return None on error. 

452 """ 

453 tag = 'read_file' 

454 try: 

455 # Translate all newlines to '\n'. 

456 with open(filename, 'r', encoding=encoding) as f: 

457 s = f.read() 

458 return regularize_nls(s) 

459 except Exception: 

460 print(f"{tag}: can not read {filename}") 

461 return None 

462 #@+node:ekr.20200106173430.1: *4* function: read_file_with_encoding 

463 def read_file_with_encoding(filename: str) -> Tuple[str, str]: 

464 """ 

465 Read the file with the given name, returning (e, s), where: 

466 

467 s is the string, converted to unicode, or '' if there was an error. 

468 

469 e is the encoding of s, computed in the following order: 

470 

471 - The BOM encoding if the file starts with a BOM mark. 

472 - The encoding given in the # -*- coding: utf-8 -*- line. 

473 - The encoding given by the 'encoding' keyword arg. 

474 - 'utf-8'. 

475 """ 

476 # First, read the file. 

477 tag = 'read_with_encoding' 

478 try: 

479 with open(filename, 'rb') as f: 

480 bb = f.read() 

481 except Exception: 

482 print(f"{tag}: can not read {filename}") 

483 if not bb: 

484 return 'UTF-8', '' 

485 # Look for the BOM. 

486 e, bb = strip_BOM(bb) 

487 if not e: 

488 # Python's encoding comments override everything else. 

489 e = get_encoding_directive(bb) 

490 s = g.toUnicode(bb, encoding=e) 

491 s = regularize_nls(s) 

492 return e, s 

493 #@+node:ekr.20200106174158.1: *4* function: strip_BOM 

494 def strip_BOM(bb: bytes) -> Tuple[Optional[str], bytes]: 

495 """ 

496 bb must be the bytes contents of a file. 

497 

498 If bb starts with a BOM (Byte Order Mark), return (e, bb2), where: 

499 

500 - e is the encoding implied by the BOM. 

501 - bb2 is bb, stripped of the BOM. 

502 

503 If there is no BOM, return (None, bb) 

504 """ 

505 assert isinstance(bb, bytes), bb.__class__.__name__ 

506 table = ( 

507 # Test longer bom's first. 

508 (4, 'utf-32', codecs.BOM_UTF32_BE), 

509 (4, 'utf-32', codecs.BOM_UTF32_LE), 

510 (3, 'utf-8', codecs.BOM_UTF8), 

511 (2, 'utf-16', codecs.BOM_UTF16_BE), 

512 (2, 'utf-16', codecs.BOM_UTF16_LE), 

513 ) 

514 for n, e, bom in table: 

515 assert len(bom) == n 

516 if bom == bb[: len(bom)]: 

517 return e, bb[len(bom) :] 

518 return None, bb 

519 #@+node:ekr.20200103163100.1: *4* function: write_file 

520 def write_file(filename: str, s: str, encoding: str='utf-8') -> None: 

521 """ 

522 Write the string s to the file whose name is given. 

523 

524 Handle all exeptions. 

525 

526 Before calling this function, the caller should ensure 

527 that the file actually has been changed. 

528 """ 

529 try: 

530 # Write the file with platform-dependent newlines. 

531 with open(filename, 'w', encoding=encoding) as f: 

532 f.write(s) 

533 except Exception as e: 

534 g.trace(f"Error writing {filename}\n{e}") 

535 #@+node:ekr.20200113154120.1: *3* functions: tokens 

536 #@+node:ekr.20191223093539.1: *4* function: find_anchor_token 

537 def find_anchor_token(node: Node, global_token_list: List["Token"]) -> Optional["Token"]: 

538 """ 

539 Return the anchor_token for node, a token such that token.node == node. 

540 

541 The search starts at node, and then all the usual child nodes. 

542 """ 

543 

544 node1 = node 

545 

546 def anchor_token(node: Node) -> Optional["Token"]: 

547 """Return the anchor token in node.token_list""" 

548 # Careful: some tokens in the token list may have been killed. 

549 for token in get_node_token_list(node, global_token_list): 

550 if is_ancestor(node1, token): 

551 return token 

552 return None 

553 

554 # This table only has to cover fields for ast.Nodes that 

555 # won't have any associated token. 

556 

557 fields = ( 

558 # Common... 

559 'elt', 'elts', 'body', 'value', 

560 # Less common... 

561 'dims', 'ifs', 'names', 's', 

562 'test', 'values', 'targets', 

563 ) 

564 while node: 

565 # First, try the node itself. 

566 token = anchor_token(node) 

567 if token: 

568 return token 

569 # Second, try the most common nodes w/o token_lists: 

570 if isinstance(node, ast.Call): 

571 node = node.func 

572 elif isinstance(node, ast.Tuple): 

573 node = node.elts # type:ignore 

574 # Finally, try all other nodes. 

575 else: 

576 # This will be used rarely. 

577 for field in fields: 

578 node = getattr(node, field, None) 

579 if node: 

580 token = anchor_token(node) 

581 if token: 

582 return token 

583 else: 

584 break 

585 return None 

586 #@+node:ekr.20191231160225.1: *4* function: find_paren_token (changed signature) 

587 def find_paren_token(i: int, global_token_list: List["Token"]) -> int: 

588 """Return i of the next paren token, starting at tokens[i].""" 

589 while i < len(global_token_list): 

590 token = global_token_list[i] 

591 if token.kind == 'op' and token.value in '()': 

592 return i 

593 if is_significant_token(token): 

594 break 

595 i += 1 

596 return None 

597 #@+node:ekr.20200113110505.4: *4* function: get_node_tokens_list 

598 def get_node_token_list(node: Node, global_tokens_list: List["Token"]) -> List["Token"]: 

599 """ 

600 tokens_list must be the global tokens list. 

601 Return the tokens assigned to the node, or []. 

602 """ 

603 i = getattr(node, 'first_i', None) 

604 j = getattr(node, 'last_i', None) 

605 return [] if i is None else global_tokens_list[i : j + 1] 

606 #@+node:ekr.20191124123830.1: *4* function: is_significant & is_significant_token 

607 def is_significant(kind: str, value: str) -> bool: 

608 """ 

609 Return True if (kind, value) represent a token that can be used for 

610 syncing generated tokens with the token list. 

611 """ 

612 # Making 'endmarker' significant ensures that all tokens are synced. 

613 return ( 

614 kind in ('async', 'await', 'endmarker', 'name', 'number', 'string') or 

615 kind == 'op' and value not in ',;()') 

616 

617 def is_significant_token(token: "Token") -> bool: 

618 """Return True if the given token is a syncronizing token""" 

619 return is_significant(token.kind, token.value) 

620 #@+node:ekr.20191224093336.1: *4* function: match_parens 

621 def match_parens(filename: str, i: int, j: int, tokens: List["Token"]) -> int: 

622 """Match parens in tokens[i:j]. Return the new j.""" 

623 if j >= len(tokens): 

624 return len(tokens) 

625 # Calculate paren level... 

626 level = 0 

627 for n in range(i, j + 1): 

628 token = tokens[n] 

629 if token.kind == 'op' and token.value == '(': 

630 level += 1 

631 if token.kind == 'op' and token.value == ')': 

632 if level == 0: 

633 break 

634 level -= 1 

635 # Find matching ')' tokens *after* j. 

636 if level > 0: 

637 while level > 0 and j + 1 < len(tokens): 

638 token = tokens[j + 1] 

639 if token.kind == 'op' and token.value == ')': 

640 level -= 1 

641 elif token.kind == 'op' and token.value == '(': 

642 level += 1 

643 elif is_significant_token(token): 

644 break 

645 j += 1 

646 if level != 0: # pragma: no cover. 

647 line_n = tokens[i].line_number 

648 raise AssignLinksError( 

649 f"\n" 

650 f"Unmatched parens: level={level}\n" 

651 f" file: {filename}\n" 

652 f" line: {line_n}\n") 

653 return j 

654 #@+node:ekr.20191223053324.1: *4* function: tokens_for_node 

655 def tokens_for_node(filename: str, node: Node, global_token_list: List["Token"]) -> List["Token"]: 

656 """Return the list of all tokens descending from node.""" 

657 # Find any token descending from node. 

658 token = find_anchor_token(node, global_token_list) 

659 if not token: 

660 if 0: # A good trace for debugging. 

661 print('') 

662 g.trace('===== no tokens', node.__class__.__name__) 

663 return [] 

664 assert is_ancestor(node, token) 

665 # Scan backward. 

666 i = first_i = token.index 

667 while i >= 0: 

668 token2 = global_token_list[i - 1] 

669 if getattr(token2, 'node', None): 

670 if is_ancestor(node, token2): 

671 first_i = i - 1 

672 else: 

673 break 

674 i -= 1 

675 # Scan forward. 

676 j = last_j = token.index 

677 while j + 1 < len(global_token_list): 

678 token2 = global_token_list[j + 1] 

679 if getattr(token2, 'node', None): 

680 if is_ancestor(node, token2): 

681 last_j = j + 1 

682 else: 

683 break 

684 j += 1 

685 last_j = match_parens(filename, first_i, last_j, global_token_list) 

686 results = global_token_list[first_i : last_j + 1] 

687 return results 

688 #@+node:ekr.20200101030236.1: *4* function: tokens_to_string 

689 def tokens_to_string(tokens: List[Any]) -> str: 

690 """Return the string represented by the list of tokens.""" 

691 if tokens is None: 

692 # This indicates an internal error. 

693 print('') 

694 g.trace('===== token list is None ===== ') 

695 print('') 

696 return '' 

697 return ''.join([z.to_string() for z in tokens]) 

698 #@+node:ekr.20191223095408.1: *3* node/token nodes... 

699 # Functions that associate tokens with nodes. 

700 #@+node:ekr.20200120082031.1: *4* function: find_statement_node 

701 def find_statement_node(node: Node) -> Optional[Node]: 

702 """ 

703 Return the nearest statement node. 

704 Return None if node has only Module for a parent. 

705 """ 

706 if isinstance(node, ast.Module): 

707 return None 

708 parent = node 

709 while parent: 

710 if is_statement_node(parent): 

711 return parent 

712 parent = parent.parent 

713 return None 

714 #@+node:ekr.20191223054300.1: *4* function: is_ancestor 

715 def is_ancestor(node: Node, token: "Token") -> bool: 

716 """Return True if node is an ancestor of token.""" 

717 t_node = token.node 

718 if not t_node: 

719 assert token.kind == 'killed', repr(token) 

720 return False 

721 while t_node: 

722 if t_node == node: 

723 return True 

724 t_node = t_node.parent 

725 return False 

726 #@+node:ekr.20200120082300.1: *4* function: is_long_statement 

727 def is_long_statement(node: Node) -> bool: 

728 """ 

729 Return True if node is an instance of a node that might be split into 

730 shorter lines. 

731 """ 

732 return isinstance(node, ( 

733 ast.Assign, ast.AnnAssign, ast.AsyncFor, ast.AsyncWith, ast.AugAssign, 

734 ast.Call, ast.Delete, ast.ExceptHandler, ast.For, ast.Global, 

735 ast.If, ast.Import, ast.ImportFrom, 

736 ast.Nonlocal, ast.Return, ast.While, ast.With, ast.Yield, ast.YieldFrom)) 

737 #@+node:ekr.20200120110005.1: *4* function: is_statement_node 

738 def is_statement_node(node: Node) -> bool: 

739 """Return True if node is a top-level statement.""" 

740 return is_long_statement(node) or isinstance(node, ( 

741 ast.Break, ast.Continue, ast.Pass, ast.Try)) 

742 #@+node:ekr.20191231082137.1: *4* function: nearest_common_ancestor 

743 def nearest_common_ancestor(node1: Node, node2: Node) -> Optional[Node]: 

744 """ 

745 Return the nearest common ancestor node for the given nodes. 

746 

747 The nodes must have parent links. 

748 """ 

749 

750 def parents(node: Node) -> List[Node]: 

751 aList = [] 

752 while node: 

753 aList.append(node) 

754 node = node.parent 

755 return list(reversed(aList)) 

756 

757 result = None 

758 parents1 = parents(node1) 

759 parents2 = parents(node2) 

760 while parents1 and parents2: 

761 parent1 = parents1.pop(0) 

762 parent2 = parents2.pop(0) 

763 if parent1 == parent2: 

764 result = parent1 

765 else: 

766 break 

767 return result 

768 #@+node:ekr.20191231072039.1: *3* functions: utils... 

769 # General utility functions on tokens and nodes. 

770 #@+node:ekr.20191119085222.1: *4* function: obj_id 

771 def obj_id(obj: Any) -> str: 

772 """Return the last four digits of id(obj), for dumps & traces.""" 

773 return str(id(obj))[-4:] 

774 #@+node:ekr.20191231060700.1: *4* function: op_name 

775 #@@nobeautify 

776 

777 # https://docs.python.org/3/library/ast.html 

778 

779 _op_names = { 

780 # Binary operators. 

781 'Add': '+', 

782 'BitAnd': '&', 

783 'BitOr': '|', 

784 'BitXor': '^', 

785 'Div': '/', 

786 'FloorDiv': '//', 

787 'LShift': '<<', 

788 'MatMult': '@', # Python 3.5. 

789 'Mod': '%', 

790 'Mult': '*', 

791 'Pow': '**', 

792 'RShift': '>>', 

793 'Sub': '-', 

794 # Boolean operators. 

795 'And': ' and ', 

796 'Or': ' or ', 

797 # Comparison operators 

798 'Eq': '==', 

799 'Gt': '>', 

800 'GtE': '>=', 

801 'In': ' in ', 

802 'Is': ' is ', 

803 'IsNot': ' is not ', 

804 'Lt': '<', 

805 'LtE': '<=', 

806 'NotEq': '!=', 

807 'NotIn': ' not in ', 

808 # Context operators. 

809 'AugLoad': '<AugLoad>', 

810 'AugStore': '<AugStore>', 

811 'Del': '<Del>', 

812 'Load': '<Load>', 

813 'Param': '<Param>', 

814 'Store': '<Store>', 

815 # Unary operators. 

816 'Invert': '~', 

817 'Not': ' not ', 

818 'UAdd': '+', 

819 'USub': '-', 

820 } 

821 

822 def op_name(node: Node) -> str: 

823 """Return the print name of an operator node.""" 

824 class_name = node.__class__.__name__ 

825 assert class_name in _op_names, repr(class_name) 

826 return _op_names[class_name].strip() 

827 #@+node:ekr.20200107114452.1: *3* node/token creators... 

828 #@+node:ekr.20200103082049.1: *4* function: make_tokens 

829 def make_tokens(contents: str) -> List["Token"]: 

830 """ 

831 Return a list (not a generator) of Token objects corresponding to the 

832 list of 5-tuples generated by tokenize.tokenize. 

833 

834 Perform consistency checks and handle all exeptions. 

835 """ 

836 

837 def check(contents: str, tokens: List["Token"]) -> bool: 

838 result = tokens_to_string(tokens) 

839 ok = result == contents 

840 if not ok: 

841 print('\nRound-trip check FAILS') 

842 print('Contents...\n') 

843 g.printObj(contents) 

844 print('\nResult...\n') 

845 g.printObj(result) 

846 return ok 

847 

848 try: 

849 five_tuples = tokenize.tokenize( 

850 io.BytesIO(contents.encode('utf-8')).readline) 

851 except Exception: 

852 print('make_tokens: exception in tokenize.tokenize') 

853 g.es_exception() 

854 return None 

855 tokens = Tokenizer().create_input_tokens(contents, five_tuples) 

856 assert check(contents, tokens) 

857 return tokens 

858 #@+node:ekr.20191027075648.1: *4* function: parse_ast 

859 def parse_ast(s: str) -> Optional[Node]: 

860 """ 

861 Parse string s, catching & reporting all exceptions. 

862 Return the ast node, or None. 

863 """ 

864 

865 def oops(message: str) -> None: 

866 print('') 

867 print(f"parse_ast: {message}") 

868 g.printObj(s) 

869 print('') 

870 

871 try: 

872 s1 = g.toEncodedString(s) 

873 tree = ast.parse(s1, filename='before', mode='exec') 

874 return tree 

875 except IndentationError: 

876 oops('Indentation Error') 

877 except SyntaxError: 

878 oops('Syntax Error') 

879 except Exception: 

880 oops('Unexpected Exception') 

881 g.es_exception() 

882 return None 

883 #@+node:ekr.20191231110051.1: *3* node/token dumpers... 

884 #@+node:ekr.20191027074436.1: *4* function: dump_ast 

885 def dump_ast(ast: Node, tag: str='dump_ast') -> None: 

886 """Utility to dump an ast tree.""" 

887 g.printObj(AstDumper().dump_ast(ast), tag=tag) 

888 #@+node:ekr.20191228095945.4: *4* function: dump_contents 

889 def dump_contents(contents: str, tag: str='Contents') -> None: 

890 print('') 

891 print(f"{tag}...\n") 

892 for i, z in enumerate(g.splitLines(contents)): 

893 print(f"{i+1:<3} ", z.rstrip()) 

894 print('') 

895 #@+node:ekr.20191228095945.5: *4* function: dump_lines 

896 def dump_lines(tokens: List["Token"], tag: str='Token lines') -> None: 

897 print('') 

898 print(f"{tag}...\n") 

899 for z in tokens: 

900 if z.line.strip(): 

901 print(z.line.rstrip()) 

902 else: 

903 print(repr(z.line)) 

904 print('') 

905 #@+node:ekr.20191228095945.7: *4* function: dump_results 

906 def dump_results(tokens: List["Token"], tag: str='Results') -> None: 

907 print('') 

908 print(f"{tag}...\n") 

909 print(tokens_to_string(tokens)) 

910 print('') 

911 #@+node:ekr.20191228095945.8: *4* function: dump_tokens 

912 def dump_tokens(tokens: List["Token"], tag: str='Tokens') -> None: 

913 print('') 

914 print(f"{tag}...\n") 

915 if not tokens: 

916 return 

917 print("Note: values shown are repr(value) *except* for 'string' tokens.") 

918 tokens[0].dump_header() 

919 for i, z in enumerate(tokens): 

920 # Confusing. 

921 # if (i % 20) == 0: z.dump_header() 

922 print(z.dump()) 

923 print('') 

924 #@+node:ekr.20191228095945.9: *4* function: dump_tree 

925 def dump_tree(tokens: List["Token"], tree: Node, tag: str='Tree') -> None: 

926 print('') 

927 print(f"{tag}...\n") 

928 print(AstDumper().dump_tree(tokens, tree)) 

929 #@+node:ekr.20200107040729.1: *4* function: show_diffs 

930 def show_diffs(s1: str, s2: str, filename: str='') -> None: 

931 """Print diffs between strings s1 and s2.""" 

932 lines = list(difflib.unified_diff( 

933 g.splitLines(s1), 

934 g.splitLines(s2), 

935 fromfile=f"Old {filename}", 

936 tofile=f"New {filename}", 

937 )) 

938 print('') 

939 tag = f"Diffs for {filename}" if filename else 'Diffs' 

940 g.printObj(lines, tag=tag) 

941 #@+node:ekr.20191225061516.1: *3* node/token replacers... 

942 # Functions that replace tokens or nodes. 

943 #@+node:ekr.20191231162249.1: *4* function: add_token_to_token_list 

944 def add_token_to_token_list(token: "Token", node: Node) -> None: 

945 """Insert token in the proper location of node.token_list.""" 

946 if getattr(node, 'first_i', None) is None: 

947 node.first_i = node.last_i = token.index 

948 else: 

949 node.first_i = min(node.first_i, token.index) 

950 node.last_i = max(node.last_i, token.index) 

951 #@+node:ekr.20191225055616.1: *4* function: replace_node 

952 def replace_node(new_node: Node, old_node: Node) -> None: 

953 """Replace new_node by old_node in the parse tree.""" 

954 parent = old_node.parent 

955 new_node.parent = parent 

956 new_node.node_index = old_node.node_index 

957 children = parent.children 

958 i = children.index(old_node) 

959 children[i] = new_node 

960 fields = getattr(old_node, '_fields', None) 

961 if fields: 

962 for field in fields: 

963 field = getattr(old_node, field) 

964 if field == old_node: 

965 setattr(old_node, field, new_node) 

966 break 

967 #@+node:ekr.20191225055626.1: *4* function: replace_token 

968 def replace_token(token: "Token", kind: str, value: str) -> None: 

969 """Replace kind and value of the given token.""" 

970 if token.kind in ('endmarker', 'killed'): 

971 return 

972 token.kind = kind 

973 token.value = value 

974 token.node = None # Should be filled later. 

975 #@-others 

976#@+node:ekr.20191027072910.1: ** Exception classes 

977class AssignLinksError(Exception): 

978 """Assigning links to ast nodes failed.""" 

979 

980 

981class AstNotEqual(Exception): 

982 """The two given AST's are not equivalent.""" 

983 

984 

985class FailFast(Exception): 

986 """Abort tests in TestRunner class.""" 

987#@+node:ekr.20220402062255.1: ** Classes 

988#@+node:ekr.20141012064706.18390: *3* class AstDumper 

989class AstDumper: # pragma: no cover 

990 """A class supporting various kinds of dumps of ast nodes.""" 

991 #@+others 

992 #@+node:ekr.20191112033445.1: *4* dumper.dump_tree & helper 

993 def dump_tree(self, tokens: List["Token"], tree: Node) -> str: 

994 """Briefly show a tree, properly indented.""" 

995 self.tokens = tokens 

996 result = [self.show_header()] 

997 self.dump_tree_and_links_helper(tree, 0, result) 

998 return ''.join(result) 

999 #@+node:ekr.20191125035321.1: *5* dumper.dump_tree_and_links_helper 

1000 def dump_tree_and_links_helper(self, node: Node, level: int, result: List[str]) -> None: 

1001 """Return the list of lines in result.""" 

1002 if node is None: 

1003 return 

1004 # Let block. 

1005 indent = ' ' * 2 * level 

1006 children: List[ast.AST] = getattr(node, 'children', []) 

1007 node_s = self.compute_node_string(node, level) 

1008 # Dump... 

1009 if isinstance(node, (list, tuple)): 

1010 for z in node: 

1011 self.dump_tree_and_links_helper(z, level, result) 

1012 elif isinstance(node, str): 

1013 result.append(f"{indent}{node.__class__.__name__:>8}:{node}\n") 

1014 elif isinstance(node, ast.AST): 

1015 # Node and parent. 

1016 result.append(node_s) 

1017 # Children. 

1018 for z in children: 

1019 self.dump_tree_and_links_helper(z, level + 1, result) 

1020 else: 

1021 result.append(node_s) 

1022 #@+node:ekr.20191125035600.1: *4* dumper.compute_node_string & helpers 

1023 def compute_node_string(self, node: Node, level: int) -> str: 

1024 """Return a string summarizing the node.""" 

1025 indent = ' ' * 2 * level 

1026 parent = getattr(node, 'parent', None) 

1027 node_id = getattr(node, 'node_index', '??') 

1028 parent_id = getattr(parent, 'node_index', '??') 

1029 parent_s = f"{parent_id:>3}.{parent.__class__.__name__} " if parent else '' 

1030 class_name = node.__class__.__name__ 

1031 descriptor_s = f"{node_id}.{class_name}: " + self.show_fields( 

1032 class_name, node, 30) 

1033 tokens_s = self.show_tokens(node, 70, 100) 

1034 lines = self.show_line_range(node) 

1035 full_s1 = f"{parent_s:<16} {lines:<10} {indent}{descriptor_s} " 

1036 node_s = f"{full_s1:<62} {tokens_s}\n" 

1037 return node_s 

1038 #@+node:ekr.20191113223424.1: *5* dumper.show_fields 

1039 def show_fields(self, class_name: str, node: Node, truncate_n: int) -> str: 

1040 """Return a string showing interesting fields of the node.""" 

1041 val = '' 

1042 if class_name == 'JoinedStr': 

1043 values = node.values 

1044 assert isinstance(values, list) 

1045 # Str tokens may represent *concatenated* strings. 

1046 results = [] 

1047 fstrings, strings = 0, 0 

1048 for z in values: 

1049 assert isinstance(z, (ast.FormattedValue, ast.Str)) 

1050 if isinstance(z, ast.Str): 

1051 results.append(z.s) 

1052 strings += 1 

1053 else: 

1054 results.append(z.__class__.__name__) 

1055 fstrings += 1 

1056 val = f"{strings} str, {fstrings} f-str" 

1057 elif class_name == 'keyword': 

1058 if isinstance(node.value, ast.Str): 

1059 val = f"arg={node.arg}..Str.value.s={node.value.s}" 

1060 elif isinstance(node.value, ast.Name): 

1061 val = f"arg={node.arg}..Name.value.id={node.value.id}" 

1062 else: 

1063 val = f"arg={node.arg}..value={node.value.__class__.__name__}" 

1064 elif class_name == 'Name': 

1065 val = f"id={node.id!r}" 

1066 elif class_name == 'NameConstant': 

1067 val = f"value={node.value!r}" 

1068 elif class_name == 'Num': 

1069 val = f"n={node.n}" 

1070 elif class_name == 'Starred': 

1071 if isinstance(node.value, ast.Str): 

1072 val = f"s={node.value.s}" 

1073 elif isinstance(node.value, ast.Name): 

1074 val = f"id={node.value.id}" 

1075 else: 

1076 val = f"s={node.value.__class__.__name__}" 

1077 elif class_name == 'Str': 

1078 val = f"s={node.s!r}" 

1079 elif class_name in ('AugAssign', 'BinOp', 'BoolOp', 'UnaryOp'): # IfExp 

1080 name = node.op.__class__.__name__ 

1081 val = f"op={_op_names.get(name, name)}" 

1082 elif class_name == 'Compare': 

1083 ops = ','.join([op_name(z) for z in node.ops]) 

1084 val = f"ops='{ops}'" 

1085 else: 

1086 val = '' 

1087 return g.truncate(val, truncate_n) 

1088 #@+node:ekr.20191114054726.1: *5* dumper.show_line_range 

1089 def show_line_range(self, node: Node) -> str: 

1090 

1091 token_list = get_node_token_list(node, self.tokens) 

1092 if not token_list: 

1093 return '' 

1094 min_ = min([z.line_number for z in token_list]) 

1095 max_ = max([z.line_number for z in token_list]) 

1096 return f"{min_}" if min_ == max_ else f"{min_}..{max_}" 

1097 #@+node:ekr.20191113223425.1: *5* dumper.show_tokens 

1098 def show_tokens(self, node: Node, n: int, m: int, show_cruft: bool=False) -> str: 

1099 """ 

1100 Return a string showing node.token_list. 

1101 

1102 Split the result if n + len(result) > m 

1103 """ 

1104 token_list = get_node_token_list(node, self.tokens) 

1105 result = [] 

1106 for z in token_list: 

1107 val = None 

1108 if z.kind == 'comment': 

1109 if show_cruft: 

1110 val = g.truncate(z.value, 10) # Short is good. 

1111 result.append(f"{z.kind}.{z.index}({val})") 

1112 elif z.kind == 'name': 

1113 val = g.truncate(z.value, 20) 

1114 result.append(f"{z.kind}.{z.index}({val})") 

1115 elif z.kind == 'newline': 

1116 # result.append(f"{z.kind}.{z.index}({z.line_number}:{len(z.line)})") 

1117 result.append(f"{z.kind}.{z.index}") 

1118 elif z.kind == 'number': 

1119 result.append(f"{z.kind}.{z.index}({z.value})") 

1120 elif z.kind == 'op': 

1121 if z.value not in ',()' or show_cruft: 

1122 result.append(f"{z.kind}.{z.index}({z.value})") 

1123 elif z.kind == 'string': 

1124 val = g.truncate(z.value, 30) 

1125 result.append(f"{z.kind}.{z.index}({val})") 

1126 elif z.kind == 'ws': 

1127 if show_cruft: 

1128 result.append(f"{z.kind}.{z.index}({len(z.value)})") 

1129 else: 

1130 # Indent, dedent, encoding, etc. 

1131 # Don't put a blank. 

1132 continue 

1133 if result and result[-1] != ' ': 

1134 result.append(' ') 

1135 # 

1136 # split the line if it is too long. 

1137 # g.printObj(result, tag='show_tokens') 

1138 if 1: 

1139 return ''.join(result) 

1140 line, lines = [], [] 

1141 for r in result: 

1142 line.append(r) 

1143 if n + len(''.join(line)) >= m: 

1144 lines.append(''.join(line)) 

1145 line = [] 

1146 lines.append(''.join(line)) 

1147 pad = '\n' + ' ' * n 

1148 return pad.join(lines) 

1149 #@+node:ekr.20191110165235.5: *4* dumper.show_header 

1150 def show_header(self) -> str: 

1151 """Return a header string, but only the fist time.""" 

1152 return ( 

1153 f"{'parent':<16} {'lines':<10} {'node':<34} {'tokens'}\n" 

1154 f"{'======':<16} {'=====':<10} {'====':<34} {'======'}\n") 

1155 #@+node:ekr.20141012064706.18392: *4* dumper.dump_ast & helper 

1156 annotate_fields = False 

1157 include_attributes = False 

1158 indent_ws = ' ' 

1159 

1160 def dump_ast(self, node: Node, level: int=0) -> str: 

1161 """ 

1162 Dump an ast tree. Adapted from ast.dump. 

1163 """ 

1164 sep1 = '\n%s' % (self.indent_ws * (level + 1)) 

1165 if isinstance(node, ast.AST): 

1166 fields = [(a, self.dump_ast(b, level + 1)) for a, b in self.get_fields(node)] 

1167 if self.include_attributes and node._attributes: 

1168 fields.extend([(a, self.dump_ast(getattr(node, a), level + 1)) 

1169 for a in node._attributes]) 

1170 if self.annotate_fields: 

1171 aList = ['%s=%s' % (a, b) for a, b in fields] 

1172 else: 

1173 aList = [b for a, b in fields] 

1174 name = node.__class__.__name__ 

1175 sep = '' if len(aList) <= 1 else sep1 

1176 return '%s(%s%s)' % (name, sep, sep1.join(aList)) 

1177 if isinstance(node, list): 

1178 sep = sep1 

1179 return 'LIST[%s]' % ''.join( 

1180 ['%s%s' % (sep, self.dump_ast(z, level + 1)) for z in node]) 

1181 return repr(node) 

1182 #@+node:ekr.20141012064706.18393: *5* dumper.get_fields 

1183 def get_fields(self, node: Node) -> Generator: 

1184 

1185 return ( 

1186 (a, b) for a, b in ast.iter_fields(node) 

1187 if a not in ['ctx',] and b not in (None, []) 

1188 ) 

1189 #@-others 

1190#@+node:ekr.20191222083453.1: *3* class Fstringify 

1191class Fstringify: 

1192 """A class to fstringify files.""" 

1193 

1194 silent = True # for pytest. Defined in all entries. 

1195 line_number = 0 

1196 line = '' 

1197 

1198 #@+others 

1199 #@+node:ekr.20191222083947.1: *4* fs.fstringify 

1200 def fstringify(self, contents: str, filename: str, tokens: List["Token"], tree: Node) -> str: 

1201 """ 

1202 Fstringify.fstringify: 

1203 

1204 f-stringify the sources given by (tokens, tree). 

1205 

1206 Return the resulting string. 

1207 """ 

1208 self.filename = filename 

1209 self.tokens = tokens 

1210 self.tree = tree 

1211 # Prepass: reassign tokens. 

1212 ReassignTokens().reassign(filename, tokens, tree) 

1213 # Main pass. 

1214 for node in ast.walk(tree): 

1215 if ( 

1216 isinstance(node, ast.BinOp) 

1217 and op_name(node.op) == '%' 

1218 and isinstance(node.left, ast.Str) 

1219 ): 

1220 self.make_fstring(node) 

1221 results = tokens_to_string(self.tokens) 

1222 return results 

1223 #@+node:ekr.20200103054101.1: *4* fs.fstringify_file (entry) 

1224 def fstringify_file(self, filename: str) -> bool: # pragma: no cover 

1225 """ 

1226 Fstringify.fstringify_file. 

1227 

1228 The entry point for the fstringify-file command. 

1229 

1230 f-stringify the given external file with the Fstrinfify class. 

1231 

1232 Return True if the file was changed. 

1233 """ 

1234 tag = 'fstringify-file' 

1235 self.filename = filename 

1236 self.silent = False 

1237 tog = TokenOrderGenerator() 

1238 try: 

1239 contents, encoding, tokens, tree = tog.init_from_file(filename) 

1240 if not contents or not tokens or not tree: 

1241 print(f"{tag}: Can not fstringify: {filename}") 

1242 return False 

1243 results = self.fstringify(contents, filename, tokens, tree) 

1244 except Exception as e: 

1245 print(e) 

1246 return False 

1247 # Something besides newlines must change. 

1248 changed = regularize_nls(contents) != regularize_nls(results) 

1249 status = 'Wrote' if changed else 'Unchanged' 

1250 print(f"{tag}: {status:>9}: {filename}") 

1251 if changed: 

1252 write_file(filename, results, encoding=encoding) 

1253 return changed 

1254 #@+node:ekr.20200103065728.1: *4* fs.fstringify_file_diff (entry) 

1255 def fstringify_file_diff(self, filename: str) -> bool: # pragma: no cover 

1256 """ 

1257 Fstringify.fstringify_file_diff. 

1258 

1259 The entry point for the diff-fstringify-file command. 

1260 

1261 Print the diffs that would resulf from the fstringify-file command. 

1262 

1263 Return True if the file would be changed. 

1264 """ 

1265 tag = 'diff-fstringify-file' 

1266 self.filename = filename 

1267 self.silent = False 

1268 tog = TokenOrderGenerator() 

1269 try: 

1270 contents, encoding, tokens, tree = tog.init_from_file(filename) 

1271 if not contents or not tokens or not tree: 

1272 return False 

1273 results = self.fstringify(contents, filename, tokens, tree) 

1274 except Exception as e: 

1275 print(e) 

1276 return False 

1277 # Something besides newlines must change. 

1278 changed = regularize_nls(contents) != regularize_nls(results) 

1279 if changed: 

1280 show_diffs(contents, results, filename=filename) 

1281 else: 

1282 print(f"{tag}: Unchanged: {filename}") 

1283 return changed 

1284 #@+node:ekr.20200112060218.1: *4* fs.fstringify_file_silent (entry) 

1285 def fstringify_file_silent(self, filename: str) -> bool: # pragma: no cover 

1286 """ 

1287 Fstringify.fstringify_file_silent. 

1288 

1289 The entry point for the silent-fstringify-file command. 

1290 

1291 fstringify the given file, suppressing all but serious error messages. 

1292 

1293 Return True if the file would be changed. 

1294 """ 

1295 self.filename = filename 

1296 self.silent = True 

1297 tog = TokenOrderGenerator() 

1298 try: 

1299 contents, encoding, tokens, tree = tog.init_from_file(filename) 

1300 if not contents or not tokens or not tree: 

1301 return False 

1302 results = self.fstringify(contents, filename, tokens, tree) 

1303 except Exception as e: 

1304 print(e) 

1305 return False 

1306 # Something besides newlines must change. 

1307 changed = regularize_nls(contents) != regularize_nls(results) 

1308 status = 'Wrote' if changed else 'Unchanged' 

1309 # Write the results. 

1310 print(f"{status:>9}: {filename}") 

1311 if changed: 

1312 write_file(filename, results, encoding=encoding) 

1313 return changed 

1314 #@+node:ekr.20191222095754.1: *4* fs.make_fstring & helpers 

1315 def make_fstring(self, node: Node) -> None: 

1316 """ 

1317 node is BinOp node representing an '%' operator. 

1318 node.left is an ast.Str node. 

1319 node.right reprsents the RHS of the '%' operator. 

1320 

1321 Convert this tree to an f-string, if possible. 

1322 Replace the node's entire tree with a new ast.Str node. 

1323 Replace all the relevant tokens with a single new 'string' token. 

1324 """ 

1325 trace = False 

1326 assert isinstance(node.left, ast.Str), (repr(node.left), g.callers()) 

1327 # Careful: use the tokens, not Str.s. This preserves spelling. 

1328 lt_token_list = get_node_token_list(node.left, self.tokens) 

1329 if not lt_token_list: # pragma: no cover 

1330 print('') 

1331 g.trace('Error: no token list in Str') 

1332 dump_tree(self.tokens, node) 

1333 print('') 

1334 return 

1335 lt_s = tokens_to_string(lt_token_list) 

1336 if trace: 

1337 g.trace('lt_s:', lt_s) # pragma: no cover 

1338 # Get the RHS values, a list of token lists. 

1339 values = self.scan_rhs(node.right) 

1340 if trace: # pragma: no cover 

1341 for i, z in enumerate(values): 

1342 dump_tokens(z, tag=f"RHS value {i}") 

1343 # Compute rt_s, self.line and self.line_number for later messages. 

1344 token0 = lt_token_list[0] 

1345 self.line_number = token0.line_number 

1346 self.line = token0.line.strip() 

1347 rt_s = ''.join(tokens_to_string(z) for z in values) 

1348 # Get the % specs in the LHS string. 

1349 specs = self.scan_format_string(lt_s) 

1350 if len(values) != len(specs): # pragma: no cover 

1351 self.message( 

1352 f"can't create f-fstring: {lt_s!r}\n" 

1353 f":f-string mismatch: " 

1354 f"{len(values)} value{g.plural(len(values))}, " 

1355 f"{len(specs)} spec{g.plural(len(specs))}") 

1356 return 

1357 # Replace specs with values. 

1358 results = self.substitute_values(lt_s, specs, values) 

1359 result = self.compute_result(lt_s, results) 

1360 if not result: 

1361 return 

1362 # Remove whitespace before ! and :. 

1363 result = self.clean_ws(result) 

1364 # Show the results 

1365 if trace: # pragma: no cover 

1366 before = (lt_s + ' % ' + rt_s).replace('\n', '<NL>') 

1367 after = result.replace('\n', '<NL>') 

1368 self.message( 

1369 f"trace:\n" 

1370 f":from: {before!s}\n" 

1371 f": to: {after!s}") 

1372 # Adjust the tree and the token list. 

1373 self.replace(node, result, values) 

1374 #@+node:ekr.20191222102831.3: *5* fs.clean_ws 

1375 ws_pat = re.compile(r'(\s+)([:!][0-9]\})') 

1376 

1377 def clean_ws(self, s: str) -> str: 

1378 """Carefully remove whitespace before ! and : specifiers.""" 

1379 s = re.sub(self.ws_pat, r'\2', s) 

1380 return s 

1381 #@+node:ekr.20191222102831.4: *5* fs.compute_result & helpers 

1382 def compute_result(self, lt_s: str, tokens: List["Token"]) -> str: 

1383 """ 

1384 Create the final result, with various kinds of munges. 

1385 

1386 Return the result string, or None if there are errors. 

1387 """ 

1388 # Fail if there is a backslash within { and }. 

1389 if not self.check_back_slashes(lt_s, tokens): 

1390 return None # pragma: no cover 

1391 # Ensure consistent quotes. 

1392 if not self.change_quotes(lt_s, tokens): 

1393 return None # pragma: no cover 

1394 return tokens_to_string(tokens) 

1395 #@+node:ekr.20200215074309.1: *6* fs.check_back_slashes 

1396 def check_back_slashes(self, lt_s: str, tokens: List["Token"]) -> bool: 

1397 """ 

1398 Return False if any backslash appears with an {} expression. 

1399 

1400 Tokens is a list of lokens on the RHS. 

1401 """ 

1402 count = 0 

1403 for z in tokens: 

1404 if z.kind == 'op': 

1405 if z.value == '{': 

1406 count += 1 

1407 elif z.value == '}': 

1408 count -= 1 

1409 if (count % 2) == 1 and '\\' in z.value: 

1410 if not self.silent: 

1411 self.message( # pragma: no cover (silent during unit tests) 

1412 f"can't create f-fstring: {lt_s!r}\n" 

1413 f":backslash in {{expr}}:") 

1414 return False 

1415 return True 

1416 #@+node:ekr.20191222102831.7: *6* fs.change_quotes 

1417 def change_quotes(self, lt_s: str, aList: List[Any]) -> bool: 

1418 """ 

1419 Carefully check quotes in all "inner" tokens as necessary. 

1420 

1421 Return False if the f-string would contain backslashes. 

1422 

1423 We expect the following "outer" tokens. 

1424 

1425 aList[0]: ('string', 'f') 

1426 aList[1]: ('string', a single or double quote. 

1427 aList[-1]: ('string', a single or double quote matching aList[1]) 

1428 """ 

1429 # Sanity checks. 

1430 if len(aList) < 4: 

1431 return True # pragma: no cover (defensive) 

1432 if not lt_s: # pragma: no cover (defensive) 

1433 self.message("can't create f-fstring: no lt_s!") 

1434 return False 

1435 delim = lt_s[0] 

1436 # Check tokens 0, 1 and -1. 

1437 token0 = aList[0] 

1438 token1 = aList[1] 

1439 token_last = aList[-1] 

1440 for token in token0, token1, token_last: 

1441 # These are the only kinds of tokens we expect to generate. 

1442 ok = ( 

1443 token.kind == 'string' or 

1444 token.kind == 'op' and token.value in '{}') 

1445 if not ok: # pragma: no cover (defensive) 

1446 self.message( 

1447 f"unexpected token: {token.kind} {token.value}\n" 

1448 f": lt_s: {lt_s!r}") 

1449 return False 

1450 # These checks are important... 

1451 if token0.value != 'f': 

1452 return False # pragma: no cover (defensive) 

1453 val1 = token1.value 

1454 if delim != val1: 

1455 return False # pragma: no cover (defensive) 

1456 val_last = token_last.value 

1457 if delim != val_last: 

1458 return False # pragma: no cover (defensive) 

1459 # 

1460 # Check for conflicting delims, preferring f"..." to f'...'. 

1461 for delim in ('"', "'"): 

1462 aList[1] = aList[-1] = Token('string', delim) 

1463 for z in aList[2:-1]: 

1464 if delim in z.value: 

1465 break 

1466 else: 

1467 return True 

1468 if not self.silent: # pragma: no cover (silent unit test) 

1469 self.message( 

1470 f"can't create f-fstring: {lt_s!r}\n" 

1471 f": conflicting delims:") 

1472 return False 

1473 #@+node:ekr.20191222102831.6: *5* fs.munge_spec 

1474 def munge_spec(self, spec: str) -> Tuple[str, str]: 

1475 """ 

1476 Return (head, tail). 

1477 

1478 The format is spec !head:tail or :tail 

1479 

1480 Example specs: s2, r3 

1481 """ 

1482 # To do: handle more specs. 

1483 head, tail = [], [] 

1484 if spec.startswith('+'): 

1485 pass # Leave it alone! 

1486 elif spec.startswith('-'): 

1487 tail.append('>') 

1488 spec = spec[1:] 

1489 if spec.endswith('s'): 

1490 spec = spec[:-1] 

1491 if spec.endswith('r'): 

1492 head.append('r') 

1493 spec = spec[:-1] 

1494 tail_s = ''.join(tail) + spec 

1495 head_s = ''.join(head) 

1496 return head_s, tail_s 

1497 #@+node:ekr.20191222102831.9: *5* fs.scan_format_string 

1498 # format_spec ::= [[fill]align][sign][#][0][width][,][.precision][type] 

1499 # fill ::= <any character> 

1500 # align ::= "<" | ">" | "=" | "^" 

1501 # sign ::= "+" | "-" | " " 

1502 # width ::= integer 

1503 # precision ::= integer 

1504 # type ::= "b" | "c" | "d" | "e" | "E" | "f" | "F" | "g" | "G" | "n" | "o" | "s" | "x" | "X" | "%" 

1505 

1506 format_pat = re.compile(r'%(([+-]?[0-9]*(\.)?[0.9]*)*[bcdeEfFgGnoxrsX]?)') 

1507 

1508 def scan_format_string(self, s: str) -> List[re.Match]: 

1509 """Scan the format string s, returning a list match objects.""" 

1510 result = list(re.finditer(self.format_pat, s)) 

1511 return result 

1512 #@+node:ekr.20191222104224.1: *5* fs.scan_rhs 

1513 def scan_rhs(self, node: Node) -> List[Any]: 

1514 """ 

1515 Scan the right-hand side of a potential f-string. 

1516 

1517 Return a list of the token lists for each element. 

1518 """ 

1519 trace = False 

1520 # First, Try the most common cases. 

1521 if isinstance(node, ast.Str): 

1522 token_list = get_node_token_list(node, self.tokens) 

1523 return [token_list] 

1524 if isinstance(node, (list, tuple, ast.Tuple)): 

1525 result = [] 

1526 elts = node.elts if isinstance(node, ast.Tuple) else node 

1527 for i, elt in enumerate(elts): 

1528 tokens = tokens_for_node(self.filename, elt, self.tokens) 

1529 result.append(tokens) 

1530 if trace: # pragma: no cover 

1531 g.trace(f"item: {i}: {elt.__class__.__name__}") 

1532 g.printObj(tokens, tag=f"Tokens for item {i}") 

1533 return result 

1534 # Now we expect only one result. 

1535 tokens = tokens_for_node(self.filename, node, self.tokens) 

1536 return [tokens] 

1537 #@+node:ekr.20191226155316.1: *5* fs.substitute_values 

1538 def substitute_values(self, lt_s: str, specs: List[re.Match], values: List) -> List["Token"]: 

1539 """ 

1540 Replace specifiers with values in lt_s string. 

1541 

1542 Double { and } as needed. 

1543 """ 

1544 i, results = 0, [Token('string', 'f')] 

1545 for spec_i, m in enumerate(specs): 

1546 value = tokens_to_string(values[spec_i]) 

1547 start, end, spec = m.start(0), m.end(0), m.group(1) 

1548 if start > i: 

1549 val = lt_s[i:start].replace('{', '{{').replace('}', '}}') 

1550 results.append(Token('string', val[0])) 

1551 results.append(Token('string', val[1:])) 

1552 head, tail = self.munge_spec(spec) 

1553 results.append(Token('op', '{')) 

1554 results.append(Token('string', value)) 

1555 if head: 

1556 results.append(Token('string', '!')) 

1557 results.append(Token('string', head)) 

1558 if tail: 

1559 results.append(Token('string', ':')) 

1560 results.append(Token('string', tail)) 

1561 results.append(Token('op', '}')) 

1562 i = end 

1563 # Add the tail. 

1564 tail = lt_s[i:] 

1565 if tail: 

1566 tail = tail.replace('{', '{{').replace('}', '}}') 

1567 results.append(Token('string', tail[:-1])) 

1568 results.append(Token('string', tail[-1])) 

1569 return results 

1570 #@+node:ekr.20200214142019.1: *4* fs.message 

1571 def message(self, message: str) -> None: # pragma: no cover. 

1572 """ 

1573 Print one or more message lines aligned on the first colon of the message. 

1574 """ 

1575 # Print a leading blank line. 

1576 print('') 

1577 # Calculate the padding. 

1578 lines = g.splitLines(message) 

1579 pad = max(lines[0].find(':'), 30) 

1580 # Print the first line. 

1581 z = lines[0] 

1582 i = z.find(':') 

1583 if i == -1: 

1584 print(z.rstrip()) 

1585 else: 

1586 print(f"{z[:i+2].strip():>{pad+1}} {z[i+2:].strip()}") 

1587 # Print the remaining message lines. 

1588 for z in lines[1:]: 

1589 if z.startswith('<'): 

1590 # Print left aligned. 

1591 print(z[1:].strip()) 

1592 elif z.startswith(':') and -1 < z[1:].find(':') <= pad: 

1593 # Align with the first line. 

1594 i = z[1:].find(':') 

1595 print(f"{z[1:i+2].strip():>{pad+1}} {z[i+2:].strip()}") 

1596 elif z.startswith('>'): 

1597 # Align after the aligning colon. 

1598 print(f"{' ':>{pad+2}}{z[1:].strip()}") 

1599 else: 

1600 # Default: Put the entire line after the aligning colon. 

1601 print(f"{' ':>{pad+2}}{z.strip()}") 

1602 # Print the standard message lines. 

1603 file_s = f"{'file':>{pad}}" 

1604 ln_n_s = f"{'line number':>{pad}}" 

1605 line_s = f"{'line':>{pad}}" 

1606 print( 

1607 f"{file_s}: {self.filename}\n" 

1608 f"{ln_n_s}: {self.line_number}\n" 

1609 f"{line_s}: {self.line!r}") 

1610 #@+node:ekr.20191225054848.1: *4* fs.replace 

1611 def replace(self, node: Node, s: str, values: List["Token"]) -> None: 

1612 """ 

1613 Replace node with an ast.Str node for s. 

1614 Replace all tokens in the range of values with a single 'string' node. 

1615 """ 

1616 # Replace the tokens... 

1617 tokens = tokens_for_node(self.filename, node, self.tokens) 

1618 i1 = i = tokens[0].index 

1619 replace_token(self.tokens[i], 'string', s) 

1620 j = 1 

1621 while j < len(tokens): 

1622 replace_token(self.tokens[i1 + j], 'killed', '') 

1623 j += 1 

1624 # Replace the node. 

1625 new_node = ast.Str() 

1626 new_node.s = s 

1627 replace_node(new_node, node) 

1628 # Update the token. 

1629 token = self.tokens[i1] 

1630 token.node = new_node 

1631 # Update the token list. 

1632 add_token_to_token_list(token, new_node) 

1633 #@-others 

1634#@+node:ekr.20220330191947.1: *3* class IterativeTokenGenerator 

1635class IterativeTokenGenerator: 

1636 """ 

1637 Self-contained iterative token syncing class. 

1638 """ 

1639 

1640 begin_end_stack: List[str] = [] # A stack of node names. 

1641 n_nodes = 0 # The number of nodes that have been visited. 

1642 node = None # The current node. 

1643 node_index = 0 # The index into the node_stack. 

1644 node_stack: List[ast.AST] = [] # The stack of parent nodes. 

1645 

1646 #@+others 

1647 #@+node:ekr.20220402095550.1: *4* iterative: Init... 

1648 # Same as in the TokenOrderGenerator class. 

1649 #@+node:ekr.20220402095550.2: *5* iterative.balance_tokens 

1650 def balance_tokens(self, tokens: List["Token"]) -> int: 

1651 """ 

1652 TOG.balance_tokens. 

1653 

1654 Insert two-way links between matching paren tokens. 

1655 """ 

1656 count, stack = 0, [] 

1657 for token in tokens: 

1658 if token.kind == 'op': 

1659 if token.value == '(': 

1660 count += 1 

1661 stack.append(token.index) 

1662 if token.value == ')': 

1663 if stack: 

1664 index = stack.pop() 

1665 tokens[index].matching_paren = token.index 

1666 tokens[token.index].matching_paren = index 

1667 else: # pragma: no cover 

1668 g.trace(f"unmatched ')' at index {token.index}") 

1669 if stack: # pragma: no cover 

1670 g.trace("unmatched '(' at {','.join(stack)}") 

1671 return count 

1672 #@+node:ekr.20220402095550.3: *5* iterative.create_links (changed) 

1673 def create_links(self, tokens: List["Token"], tree: Node, file_name: str='') -> List: 

1674 """ 

1675 A generator creates two-way links between the given tokens and ast-tree. 

1676 

1677 Callers should call this generator with list(tog.create_links(...)) 

1678 

1679 The sync_tokens method creates the links and verifies that the resulting 

1680 tree traversal generates exactly the given tokens in exact order. 

1681 

1682 tokens: the list of Token instances for the input. 

1683 Created by make_tokens(). 

1684 tree: the ast tree for the input. 

1685 Created by parse_ast(). 

1686 """ 

1687 # Init all ivars. 

1688 self.file_name = file_name # For tests. 

1689 self.node = None # The node being visited. 

1690 self.tokens = tokens # The immutable list of input tokens. 

1691 self.tree = tree # The tree of ast.AST nodes. 

1692 # Traverse the tree. 

1693 self.main_loop(tree) 

1694 # Ensure that all tokens are patched. 

1695 self.node = tree 

1696 self.token(('endmarker', '')) 

1697 # Return [] for compatibility with legacy code: list(tog.create_links). 

1698 return [] 

1699 #@+node:ekr.20220402095550.4: *5* iterative.init_from_file 

1700 def init_from_file(self, filename: str) -> Tuple[str, str, List["Token"], Node]: # pragma: no cover 

1701 """ 

1702 Create the tokens and ast tree for the given file. 

1703 Create links between tokens and the parse tree. 

1704 Return (contents, encoding, tokens, tree). 

1705 """ 

1706 self.filename = filename 

1707 encoding, contents = read_file_with_encoding(filename) 

1708 if not contents: 

1709 return None, None, None, None 

1710 self.tokens = tokens = make_tokens(contents) 

1711 self.tree = tree = parse_ast(contents) 

1712 self.create_links(tokens, tree) 

1713 return contents, encoding, tokens, tree 

1714 #@+node:ekr.20220402095550.5: *5* iterative.init_from_string 

1715 def init_from_string(self, contents: str, filename: str) -> Tuple[List["Token"], Node]: # pragma: no cover 

1716 """ 

1717 Tokenize, parse and create links in the contents string. 

1718 

1719 Return (tokens, tree). 

1720 """ 

1721 self.filename = filename 

1722 self.tokens = tokens = make_tokens(contents) 

1723 self.tree = tree = parse_ast(contents) 

1724 self.create_links(tokens, tree) 

1725 return tokens, tree 

1726 #@+node:ekr.20220402094825.1: *4* iterative: Syncronizers... 

1727 # Same as in the TokenOrderGenerator class. 

1728 

1729 # The synchronizer sync tokens to nodes. 

1730 #@+node:ekr.20220402094825.2: *5* iterative.find_next_significant_token 

1731 def find_next_significant_token(self) -> Optional["Token"]: 

1732 """ 

1733 Scan from *after* self.tokens[px] looking for the next significant 

1734 token. 

1735 

1736 Return the token, or None. Never change self.px. 

1737 """ 

1738 px = self.px + 1 

1739 while px < len(self.tokens): 

1740 token = self.tokens[px] 

1741 px += 1 

1742 if is_significant_token(token): 

1743 return token 

1744 # This will never happen, because endtoken is significant. 

1745 return None # pragma: no cover 

1746 #@+node:ekr.20220402094825.3: *5* iterative.set_links 

1747 last_statement_node = None 

1748 

1749 def set_links(self, node: Node, token: "Token") -> None: 

1750 """Make two-way links between token and the given node.""" 

1751 # Don't bother assigning comment, comma, parens, ws and endtoken tokens. 

1752 if token.kind == 'comment': 

1753 # Append the comment to node.comment_list. 

1754 comment_list: List["Token"] = getattr(node, 'comment_list', []) 

1755 node.comment_list = comment_list + [token] 

1756 return 

1757 if token.kind in ('endmarker', 'ws'): 

1758 return 

1759 if token.kind == 'op' and token.value in ',()': 

1760 return 

1761 # *Always* remember the last statement. 

1762 statement = find_statement_node(node) 

1763 if statement: 

1764 self.last_statement_node = statement 

1765 assert not isinstance(self.last_statement_node, ast.Module) 

1766 if token.node is not None: # pragma: no cover 

1767 line_s = f"line {token.line_number}:" 

1768 raise AssignLinksError( 

1769 f" file: {self.filename}\n" 

1770 f"{line_s:>12} {token.line.strip()}\n" 

1771 f"token index: {self.px}\n" 

1772 f"token.node is not None\n" 

1773 f" token.node: {token.node.__class__.__name__}\n" 

1774 f" callers: {g.callers()}") 

1775 # Assign newlines to the previous statement node, if any. 

1776 if token.kind in ('newline', 'nl'): 

1777 # Set an *auxilliary* link for the split/join logic. 

1778 # Do *not* set token.node! 

1779 token.statement_node = self.last_statement_node 

1780 return 

1781 if is_significant_token(token): 

1782 # Link the token to the ast node. 

1783 token.node = node 

1784 # Add the token to node's token_list. 

1785 add_token_to_token_list(token, node) 

1786 #@+node:ekr.20220402094825.4: *5* iterative.sync_name (aka name) 

1787 def sync_name(self, val: str) -> None: 

1788 aList = val.split('.') 

1789 if len(aList) == 1: 

1790 self.sync_token(('name', val)) 

1791 else: 

1792 for i, part in enumerate(aList): 

1793 self.sync_token(('name', part)) 

1794 if i < len(aList) - 1: 

1795 self.sync_op('.') 

1796 

1797 name = sync_name # for readability. 

1798 #@+node:ekr.20220402094825.5: *5* iterative.sync_op (aka op) 

1799 def sync_op(self, val: str) -> None: 

1800 """ 

1801 Sync to the given operator. 

1802 

1803 val may be '(' or ')' *only* if the parens *will* actually exist in the 

1804 token list. 

1805 """ 

1806 self.sync_token(('op', val)) 

1807 

1808 op = sync_op # For readability. 

1809 #@+node:ekr.20220402094825.6: *5* iterative.sync_token (aka token) 

1810 px = -1 # Index of the previously synced token. 

1811 

1812 ### def sync_token(self, kind: str, val: str) -> None: 

1813 def sync_token(self, data: Tuple[Any, Any]) -> None: 

1814 """ 

1815 Sync to a token whose kind & value are given. The token need not be 

1816 significant, but it must be guaranteed to exist in the token list. 

1817 

1818 The checks in this method constitute a strong, ever-present, unit test. 

1819 

1820 Scan the tokens *after* px, looking for a token T matching (kind, val). 

1821 raise AssignLinksError if a significant token is found that doesn't match T. 

1822 Otherwise: 

1823 - Create two-way links between all assignable tokens between px and T. 

1824 - Create two-way links between T and self.node. 

1825 - Advance by updating self.px to point to T. 

1826 """ 

1827 kind, val = data ### New 

1828 node, tokens = self.node, self.tokens 

1829 assert isinstance(node, ast.AST), repr(node) 

1830 # g.trace( 

1831 # f"px: {self.px:2} " 

1832 # f"node: {node.__class__.__name__:<10} " 

1833 # f"kind: {kind:>10}: val: {val!r}") 

1834 # 

1835 # Step one: Look for token T. 

1836 old_px = px = self.px + 1 

1837 while px < len(self.tokens): 

1838 token = tokens[px] 

1839 if (kind, val) == (token.kind, token.value): 

1840 break # Success. 

1841 if kind == token.kind == 'number': 

1842 val = token.value 

1843 break # Benign: use the token's value, a string, instead of a number. 

1844 if is_significant_token(token): # pragma: no cover 

1845 line_s = f"line {token.line_number}:" 

1846 val = str(val) # for g.truncate. 

1847 raise AssignLinksError( 

1848 f" file: {self.filename}\n" 

1849 f"{line_s:>12} {token.line.strip()}\n" 

1850 f"Looking for: {kind}.{g.truncate(val, 40)!r}\n" 

1851 f" found: {token.kind}.{token.value!r}\n" 

1852 f"token.index: {token.index}\n") 

1853 # Skip the insignificant token. 

1854 px += 1 

1855 else: # pragma: no cover 

1856 val = str(val) # for g.truncate. 

1857 raise AssignLinksError( 

1858 f" file: {self.filename}\n" 

1859 f"Looking for: {kind}.{g.truncate(val, 40)}\n" 

1860 f" found: end of token list") 

1861 # 

1862 # Step two: Assign *secondary* links only for newline tokens. 

1863 # Ignore all other non-significant tokens. 

1864 while old_px < px: 

1865 token = tokens[old_px] 

1866 old_px += 1 

1867 if token.kind in ('comment', 'newline', 'nl'): 

1868 self.set_links(node, token) 

1869 # 

1870 # Step three: Set links in the found token. 

1871 token = tokens[px] 

1872 self.set_links(node, token) 

1873 # 

1874 # Step four: Advance. 

1875 self.px = px 

1876 

1877 token = sync_token # For readability. 

1878 #@+node:ekr.20220330164313.1: *4* iterative: Traversal... 

1879 #@+node:ekr.20220402094946.2: *5* iterative.enter_node 

1880 def enter_node(self, node: Node) -> None: 

1881 """Enter a node.""" 

1882 # Update the stats. 

1883 self.n_nodes += 1 

1884 # Create parent/child links first, *before* updating self.node. 

1885 # 

1886 # Don't even *think* about removing the parent/child links. 

1887 # The nearest_common_ancestor function depends upon them. 

1888 node.parent = self.node 

1889 if self.node: 

1890 children: List[Node] = getattr(self.node, 'children', []) 

1891 children.append(node) 

1892 self.node.children = children 

1893 # Inject the node_index field. 

1894 assert not hasattr(node, 'node_index'), g.callers() 

1895 node.node_index = self.node_index 

1896 self.node_index += 1 

1897 # begin_visitor and end_visitor must be paired. 

1898 self.begin_end_stack.append(node.__class__.__name__) 

1899 # Push the previous node. 

1900 self.node_stack.append(self.node) 

1901 # Update self.node *last*. 

1902 self.node = node 

1903 #@+node:ekr.20220402094946.3: *5* iterative.leave_node 

1904 def leave_node(self, node: Node) -> None: 

1905 """Leave a visitor.""" 

1906 # Make *sure* that begin_visitor and end_visitor are paired. 

1907 entry_name = self.begin_end_stack.pop() 

1908 assert entry_name == node.__class__.__name__, f"{entry_name!r} {node.__class__.__name__}" 

1909 assert self.node == node, (repr(self.node), repr(node)) 

1910 # Restore self.node. 

1911 self.node = self.node_stack.pop() 

1912 #@+node:ekr.20220330120220.1: *5* iterative.main_loop 

1913 def main_loop(self, node: Node) -> None: 

1914 

1915 func = getattr(self, 'do_' + node.__class__.__name__, None) 

1916 if not func: # pragma: no cover (defensive code) 

1917 print('main_loop: invalid ast node:', repr(node)) 

1918 return 

1919 exec_list: ActionList = [(func, node)] 

1920 while exec_list: 

1921 func, arg = exec_list.pop(0) 

1922 result = func(arg) 

1923 if result: 

1924 # Prepend the result, a list of tuples. 

1925 assert isinstance(result, list), repr(result) 

1926 exec_list[:0] = result 

1927 

1928 # For debugging... 

1929 # try: 

1930 # func, arg = data 

1931 # if 0: 

1932 # func_name = g.truncate(func.__name__, 15) 

1933 # print( 

1934 # f"{self.node.__class__.__name__:>10}:" 

1935 # f"{func_name:>20} " 

1936 # f"{arg.__class__.__name__}") 

1937 # except ValueError: 

1938 # g.trace('BAD DATA', self.node.__class__.__name__) 

1939 # if isinstance(data, (list, tuple)): 

1940 # for z in data: 

1941 # print(data) 

1942 # else: 

1943 # print(repr(data)) 

1944 # raise 

1945 #@+node:ekr.20220330155314.1: *5* iterative.visit 

1946 def visit(self, node: Node) -> ActionList: 

1947 """'Visit' an ast node by return a new list of tuples.""" 

1948 # Keep this trace. 

1949 if False: # pragma: no cover 

1950 cn = node.__class__.__name__ if node else ' ' 

1951 caller1, caller2 = g.callers(2).split(',') 

1952 g.trace(f"{caller1:>15} {caller2:<14} {cn}") 

1953 if node is None: 

1954 return [] 

1955 # More general, more convenient. 

1956 if isinstance(node, (list, tuple)): 

1957 result = [] 

1958 for z in node: 

1959 if isinstance(z, ast.AST): 

1960 result.append((self.visit, z)) 

1961 else: # pragma: no cover (This might never happen). 

1962 # All other fields should contain ints or strings. 

1963 assert isinstance(z, (int, str)), z.__class__.__name__ 

1964 return result 

1965 # We *do* want to crash if the visitor doesn't exist. 

1966 assert isinstance(node, ast.AST), repr(node) 

1967 method = getattr(self, 'do_' + node.__class__.__name__) 

1968 # Don't call *anything* here. Just return a new list of tuples. 

1969 return [ 

1970 (self.enter_node, node), 

1971 (method, node), 

1972 (self.leave_node, node), 

1973 ] 

1974 #@+node:ekr.20220330133336.1: *4* iterative: Visitors 

1975 #@+node:ekr.20220330133336.2: *5* iterative.keyword: not called! 

1976 # keyword arguments supplied to call (NULL identifier for **kwargs) 

1977 

1978 # keyword = (identifier? arg, expr value) 

1979 

1980 def do_keyword(self, node: Node) -> List: # pragma: no cover 

1981 """A keyword arg in an ast.Call.""" 

1982 # This should never be called. 

1983 # iterative.hande_call_arguments calls self.visit(kwarg_arg.value) instead. 

1984 filename = getattr(self, 'filename', '<no file>') 

1985 raise AssignLinksError( 

1986 f"file: {filename}\n" 

1987 f"do_keyword should never be called\n" 

1988 f"{g.callers(8)}") 

1989 #@+node:ekr.20220330133336.3: *5* iterative: Contexts 

1990 #@+node:ekr.20220330133336.4: *6* iterative.arg 

1991 # arg = (identifier arg, expr? annotation) 

1992 

1993 def do_arg(self, node: Node) -> ActionList: 

1994 """This is one argument of a list of ast.Function or ast.Lambda arguments.""" 

1995 

1996 annotation = getattr(node, 'annotation', None) 

1997 result: List = [ 

1998 (self.name, node.arg), 

1999 ] 

2000 if annotation: 

2001 result.extend([ 

2002 (self.op, ':'), 

2003 (self.visit, annotation), 

2004 ]) 

2005 return result 

2006 

2007 #@+node:ekr.20220330133336.5: *6* iterative.arguments 

2008 # arguments = ( 

2009 # arg* posonlyargs, arg* args, arg? vararg, arg* kwonlyargs, 

2010 # expr* kw_defaults, arg? kwarg, expr* defaults 

2011 # ) 

2012 

2013 def do_arguments(self, node: Node) -> ActionList: 

2014 """Arguments to ast.Function or ast.Lambda, **not** ast.Call.""" 

2015 # 

2016 # No need to generate commas anywhere below. 

2017 # 

2018 # Let block. Some fields may not exist pre Python 3.8. 

2019 n_plain = len(node.args) - len(node.defaults) 

2020 posonlyargs = getattr(node, 'posonlyargs', []) 

2021 vararg = getattr(node, 'vararg', None) 

2022 kwonlyargs = getattr(node, 'kwonlyargs', []) 

2023 kw_defaults = getattr(node, 'kw_defaults', []) 

2024 kwarg = getattr(node, 'kwarg', None) 

2025 result: ActionList = [] 

2026 # 1. Sync the position-only args. 

2027 if posonlyargs: 

2028 for n, z in enumerate(posonlyargs): 

2029 # self.visit(z) 

2030 result.append((self.visit, z)) 

2031 # self.op('/') 

2032 result.append((self.op, '/')) 

2033 # 2. Sync all args. 

2034 for i, z in enumerate(node.args): 

2035 # self.visit(z) 

2036 result.append((self.visit, z)) 

2037 if i >= n_plain: 

2038 # self.op('=') 

2039 # self.visit(node.defaults[i - n_plain]) 

2040 result.extend([ 

2041 (self.op, '='), 

2042 (self.visit, node.defaults[i - n_plain]), 

2043 ]) 

2044 # 3. Sync the vararg. 

2045 if vararg: 

2046 # self.op('*') 

2047 # self.visit(vararg) 

2048 result.extend([ 

2049 (self.op, '*'), 

2050 (self.visit, vararg), 

2051 ]) 

2052 # 4. Sync the keyword-only args. 

2053 if kwonlyargs: 

2054 if not vararg: 

2055 # self.op('*') 

2056 result.append((self.op, '*')) 

2057 for n, z in enumerate(kwonlyargs): 

2058 # self.visit(z) 

2059 result.append((self.visit, z)) 

2060 val = kw_defaults[n] 

2061 if val is not None: 

2062 # self.op('=') 

2063 # self.visit(val) 

2064 result.extend([ 

2065 (self.op, '='), 

2066 (self.visit, val), 

2067 ]) 

2068 # 5. Sync the kwarg. 

2069 if kwarg: 

2070 # self.op('**') 

2071 # self.visit(kwarg) 

2072 result.extend([ 

2073 (self.op, '**'), 

2074 (self.visit, kwarg), 

2075 ]) 

2076 return result 

2077 

2078 

2079 

2080 #@+node:ekr.20220330133336.6: *6* iterative.AsyncFunctionDef 

2081 # AsyncFunctionDef(identifier name, arguments args, stmt* body, expr* decorator_list, 

2082 # expr? returns) 

2083 

2084 def do_AsyncFunctionDef(self, node: Node) -> ActionList: 

2085 

2086 returns = getattr(node, 'returns', None) 

2087 result: ActionList = [] 

2088 # Decorators... 

2089 # @{z}\n 

2090 for z in node.decorator_list or []: 

2091 result.extend([ 

2092 (self.op, '@'), 

2093 (self.visit, z) 

2094 ]) 

2095 # Signature... 

2096 # def name(args): -> returns\n 

2097 # def name(args):\n 

2098 result.extend([ 

2099 (self.name, 'async'), 

2100 (self.name, 'def'), 

2101 (self.name, node.name), # A string. 

2102 (self.op, '('), 

2103 (self.visit, node.args), 

2104 (self.op, ')'), 

2105 ]) 

2106 if returns is not None: 

2107 result.extend([ 

2108 (self.op, '->'), 

2109 (self.visit, node.returns), 

2110 ]) 

2111 # Body... 

2112 result.extend([ 

2113 (self.op, ':'), 

2114 (self.visit, node.body), 

2115 ]) 

2116 return result 

2117 #@+node:ekr.20220330133336.7: *6* iterative.ClassDef 

2118 def do_ClassDef(self, node: Node) -> ActionList: 

2119 

2120 result: ActionList = [] 

2121 for z in node.decorator_list or []: 

2122 # @{z}\n 

2123 result.extend([ 

2124 (self.op, '@'), 

2125 (self.visit, z), 

2126 ]) 

2127 # class name(bases):\n 

2128 result.extend([ 

2129 (self.name, 'class'), 

2130 (self.name, node.name), # A string. 

2131 ]) 

2132 if node.bases: 

2133 result.extend([ 

2134 (self.op, '('), 

2135 (self.visit, node.bases), 

2136 (self.op, ')'), 

2137 ]) 

2138 result.extend([ 

2139 (self.op, ':'), 

2140 (self.visit, node.body), 

2141 ]) 

2142 return result 

2143 #@+node:ekr.20220330133336.8: *6* iterative.FunctionDef 

2144 # FunctionDef( 

2145 # identifier name, arguments args, 

2146 # stmt* body, 

2147 # expr* decorator_list, 

2148 # expr? returns, 

2149 # string? type_comment) 

2150 

2151 def do_FunctionDef(self, node: Node) -> ActionList: 

2152 

2153 returns = getattr(node, 'returns', None) 

2154 result: ActionList = [] 

2155 # Decorators... 

2156 # @{z}\n 

2157 for z in node.decorator_list or []: 

2158 result.extend([ 

2159 (self.op, '@'), 

2160 (self.visit, z) 

2161 ]) 

2162 # Signature... 

2163 # def name(args): -> returns\n 

2164 # def name(args):\n 

2165 result.extend([ 

2166 (self.name, 'def'), 

2167 (self.name, node.name), # A string. 

2168 (self.op, '('), 

2169 (self.visit, node.args), 

2170 (self.op, ')'), 

2171 ]) 

2172 if returns is not None: 

2173 result.extend([ 

2174 (self.op, '->'), 

2175 (self.visit, node.returns), 

2176 ]) 

2177 # Body... 

2178 result.extend([ 

2179 (self.op, ':'), 

2180 (self.visit, node.body), 

2181 ]) 

2182 return result 

2183 #@+node:ekr.20220330133336.9: *6* iterative.Interactive 

2184 def do_Interactive(self, node: Node) -> ActionList: # pragma: no cover 

2185 

2186 return [ 

2187 (self.visit, node.body), 

2188 ] 

2189 #@+node:ekr.20220330133336.10: *6* iterative.Lambda 

2190 def do_Lambda(self, node: Node) -> ActionList: 

2191 

2192 return [ 

2193 (self.name, 'lambda'), 

2194 (self.visit, node.args), 

2195 (self.op, ':'), 

2196 (self.visit, node.body), 

2197 ] 

2198 

2199 #@+node:ekr.20220330133336.11: *6* iterative.Module 

2200 def do_Module(self, node: Node) -> ActionList: 

2201 

2202 # Encoding is a non-syncing statement. 

2203 return [ 

2204 (self.visit, node.body), 

2205 ] 

2206 #@+node:ekr.20220330133336.12: *5* iterative: Expressions 

2207 #@+node:ekr.20220330133336.13: *6* iterative.Expr 

2208 def do_Expr(self, node: Node) -> ActionList: 

2209 """An outer expression.""" 

2210 # No need to put parentheses. 

2211 return [ 

2212 (self.visit, node.value), 

2213 ] 

2214 #@+node:ekr.20220330133336.14: *6* iterative.Expression 

2215 def do_Expression(self, node: Node) -> ActionList: # pragma: no cover 

2216 """An inner expression.""" 

2217 # No need to put parentheses. 

2218 return [ 

2219 (self.visit, node.body), 

2220 ] 

2221 #@+node:ekr.20220330133336.15: *6* iterative.GeneratorExp 

2222 def do_GeneratorExp(self, node: Node) -> ActionList: 

2223 # '<gen %s for %s>' % (elt, ','.join(gens)) 

2224 # No need to put parentheses or commas. 

2225 return [ 

2226 (self.visit, node.elt), 

2227 (self.visit, node.generators), 

2228 ] 

2229 #@+node:ekr.20220330133336.16: *6* iterative.NamedExpr 

2230 # NamedExpr(expr target, expr value) 

2231 

2232 def do_NamedExpr(self, node: Node) -> ActionList: # Python 3.8+ 

2233 

2234 return [ 

2235 (self.visit, node.target), 

2236 (self.op, ':='), 

2237 (self.visit, node.value), 

2238 ] 

2239 #@+node:ekr.20220402160128.1: *5* iterative: Operands 

2240 #@+node:ekr.20220402160128.2: *6* iterative.Attribute 

2241 # Attribute(expr value, identifier attr, expr_context ctx) 

2242 

2243 def do_Attribute(self, node: Node) -> ActionList: 

2244 

2245 return [ 

2246 (self.visit, node.value), 

2247 (self.op, '.'), 

2248 (self.name, node.attr), # A string. 

2249 ] 

2250 #@+node:ekr.20220402160128.3: *6* iterative.Bytes 

2251 def do_Bytes(self, node: Node) -> ActionList: 

2252 

2253 """ 

2254 It's invalid to mix bytes and non-bytes literals, so just 

2255 advancing to the next 'string' token suffices. 

2256 """ 

2257 token = self.find_next_significant_token() 

2258 return [ 

2259 (self.token, ('string', token.value)), 

2260 ] 

2261 #@+node:ekr.20220402160128.4: *6* iterative.comprehension 

2262 # comprehension = (expr target, expr iter, expr* ifs, int is_async) 

2263 

2264 def do_comprehension(self, node: Node) -> ActionList: 

2265 

2266 # No need to put parentheses. 

2267 result: ActionList = [ 

2268 (self.name, 'for'), 

2269 (self.visit, node.target), # A name 

2270 (self.name, 'in'), 

2271 (self.visit, node.iter), 

2272 ] 

2273 for z in node.ifs or []: 

2274 result.extend([ 

2275 (self.name, 'if'), 

2276 (self.visit, z), 

2277 ]) 

2278 return result 

2279 #@+node:ekr.20220402160128.5: *6* iterative.Constant 

2280 def do_Constant(self, node: Node) -> ActionList: # pragma: no cover 

2281 """ 

2282 

2283 https://greentreesnakes.readthedocs.io/en/latest/nodes.html 

2284 

2285 A constant. The value attribute holds the Python object it represents. 

2286 This can be simple types such as a number, string or None, but also 

2287 immutable container types (tuples and frozensets) if all of their 

2288 elements are constant. 

2289 """ 

2290 # Support Python 3.8. 

2291 if node.value is None or isinstance(node.value, bool): 

2292 # Weird: return a name! 

2293 return [ 

2294 (self.token, ('name', repr(node.value))), 

2295 ] 

2296 if node.value == Ellipsis: 

2297 return [ 

2298 (self.op, '...'), 

2299 ] 

2300 if isinstance(node.value, str): 

2301 return self.do_Str(node) 

2302 if isinstance(node.value, (int, float)): 

2303 return [ 

2304 (self.token, ('number', repr(node.value))), 

2305 ] 

2306 if isinstance(node.value, bytes): 

2307 return self.do_Bytes(node) 

2308 if isinstance(node.value, tuple): 

2309 return self.do_Tuple(node) 

2310 if isinstance(node.value, frozenset): 

2311 return self.do_Set(node) 

2312 g.trace('----- Oops -----', repr(node.value), g.callers()) 

2313 return [] 

2314 

2315 #@+node:ekr.20220402160128.6: *6* iterative.Dict 

2316 # Dict(expr* keys, expr* values) 

2317 

2318 def do_Dict(self, node: Node) -> ActionList: 

2319 

2320 assert len(node.keys) == len(node.values) 

2321 result: List = [ 

2322 (self.op, '{'), 

2323 ] 

2324 # No need to put commas. 

2325 for i, key in enumerate(node.keys): 

2326 key, value = node.keys[i], node.values[i] 

2327 result.extend([ 

2328 (self.visit, key), # a Str node. 

2329 (self.op, ':'), 

2330 ]) 

2331 if value is not None: 

2332 result.append((self.visit, value)) 

2333 result.append((self.op, '}')) 

2334 return result 

2335 #@+node:ekr.20220402160128.7: *6* iterative.DictComp 

2336 # DictComp(expr key, expr value, comprehension* generators) 

2337 

2338 # d2 = {val: key for key, val in d} 

2339 

2340 def do_DictComp(self, node: Node) -> ActionList: 

2341 

2342 result: ActionList = [ 

2343 (self.token, ('op', '{')), 

2344 (self.visit, node.key), 

2345 (self.op, ':'), 

2346 (self.visit, node.value), 

2347 ] 

2348 for z in node.generators or []: 

2349 result.extend([ 

2350 (self.visit, z), 

2351 (self.token, ('op', '}')), 

2352 ]) 

2353 return result 

2354 

2355 #@+node:ekr.20220402160128.8: *6* iterative.Ellipsis 

2356 def do_Ellipsis(self, node: Node) -> ActionList: # pragma: no cover (Does not exist for python 3.8+) 

2357 

2358 return [ 

2359 (self.op, '...'), 

2360 ] 

2361 #@+node:ekr.20220402160128.9: *6* iterative.ExtSlice 

2362 # https://docs.python.org/3/reference/expressions.html#slicings 

2363 

2364 # ExtSlice(slice* dims) 

2365 

2366 def do_ExtSlice(self, node: Node) -> ActionList: # pragma: no cover (deprecated) 

2367 

2368 result: ActionList = [] 

2369 for i, z in enumerate(node.dims): 

2370 # self.visit(z) 

2371 result.append((self.visit, z)) 

2372 if i < len(node.dims) - 1: 

2373 # self.op(',') 

2374 result.append((self.op, ',')) 

2375 return result 

2376 #@+node:ekr.20220402160128.10: *6* iterative.Index 

2377 def do_Index(self, node: Node) -> ActionList: # pragma: no cover (deprecated) 

2378 

2379 return [ 

2380 (self.visit, node.value), 

2381 ] 

2382 #@+node:ekr.20220402160128.11: *6* iterative.FormattedValue: not called! 

2383 # FormattedValue(expr value, int? conversion, expr? format_spec) 

2384 

2385 def do_FormattedValue(self, node: Node) -> ActionList: # pragma: no cover 

2386 """ 

2387 This node represents the *components* of a *single* f-string. 

2388 

2389 Happily, JoinedStr nodes *also* represent *all* f-strings, 

2390 so the TOG should *never visit this node! 

2391 """ 

2392 filename = getattr(self, 'filename', '<no file>') 

2393 raise AssignLinksError( 

2394 f"file: {filename}\n" 

2395 f"do_FormattedValue should never be called") 

2396 

2397 # This code has no chance of being useful... 

2398 # conv = node.conversion 

2399 # spec = node.format_spec 

2400 # self.visit(node.value) 

2401 # if conv is not None: 

2402 # self.token('number', conv) 

2403 # if spec is not None: 

2404 # self.visit(node.format_spec) 

2405 #@+node:ekr.20220402160128.12: *6* iterative.JoinedStr & helpers 

2406 # JoinedStr(expr* values) 

2407 

2408 def do_JoinedStr(self, node: Node) -> ActionList: 

2409 """ 

2410 JoinedStr nodes represent at least one f-string and all other strings 

2411 concatentated to it. 

2412 

2413 Analyzing JoinedStr.values would be extremely tricky, for reasons that 

2414 need not be explained here. 

2415 

2416 Instead, we get the tokens *from the token list itself*! 

2417 """ 

2418 # for z in self.get_concatenated_string_tokens(): 

2419 # self.gen_token(z.kind, z.value) 

2420 return [ 

2421 (self.token, (z.kind, z.value)) 

2422 for z in self.get_concatenated_string_tokens() 

2423 ] 

2424 #@+node:ekr.20220402160128.13: *6* iterative.List 

2425 def do_List(self, node: Node) -> ActionList: 

2426 

2427 # No need to put commas. 

2428 return [ 

2429 (self.op, '['), 

2430 (self.visit, node.elts), 

2431 (self.op, ']'), 

2432 ] 

2433 #@+node:ekr.20220402160128.14: *6* iterative.ListComp 

2434 # ListComp(expr elt, comprehension* generators) 

2435 

2436 def do_ListComp(self, node: Node) -> ActionList: 

2437 

2438 result: List = [ 

2439 (self.op, '['), 

2440 (self.visit, node.elt), 

2441 ] 

2442 for z in node.generators: 

2443 result.append((self.visit, z)) 

2444 result.append((self.op, ']')) 

2445 return result 

2446 #@+node:ekr.20220402160128.15: *6* iterative.Name & NameConstant 

2447 def do_Name(self, node: Node) -> ActionList: 

2448 

2449 return [ 

2450 (self.name, node.id), 

2451 ] 

2452 

2453 def do_NameConstant(self, node: Node) -> ActionList: # pragma: no cover (Does not exist in Python 3.8+) 

2454 

2455 return [ 

2456 (self.name, repr(node.value)), 

2457 ] 

2458 #@+node:ekr.20220402160128.16: *6* iterative.Num 

2459 def do_Num(self, node: Node) -> ActionList: # pragma: no cover (Does not exist in Python 3.8+) 

2460 

2461 return [ 

2462 (self.token, ('number', node.n)), 

2463 ] 

2464 #@+node:ekr.20220402160128.17: *6* iterative.Set 

2465 # Set(expr* elts) 

2466 

2467 def do_Set(self, node: Node) -> ActionList: 

2468 

2469 return [ 

2470 (self.op, '{'), 

2471 (self.visit, node.elts), 

2472 (self.op, '}'), 

2473 ] 

2474 #@+node:ekr.20220402160128.18: *6* iterative.SetComp 

2475 # SetComp(expr elt, comprehension* generators) 

2476 

2477 def do_SetComp(self, node: Node) -> ActionList: 

2478 

2479 result: List = [ 

2480 (self.op, '{'), 

2481 (self.visit, node.elt), 

2482 ] 

2483 for z in node.generators or []: 

2484 result.append((self.visit, z)) 

2485 result.append((self.op, '}')) 

2486 return result 

2487 #@+node:ekr.20220402160128.19: *6* iterative.Slice 

2488 # slice = Slice(expr? lower, expr? upper, expr? step) 

2489 

2490 def do_Slice(self, node: Node) -> ActionList: 

2491 

2492 lower = getattr(node, 'lower', None) 

2493 upper = getattr(node, 'upper', None) 

2494 step = getattr(node, 'step', None) 

2495 result: ActionList = [] 

2496 if lower is not None: 

2497 result.append((self.visit, lower)) 

2498 # Always put the colon between upper and lower. 

2499 result.append((self.op, ':')) 

2500 if upper is not None: 

2501 result.append((self.visit, upper)) 

2502 # Put the second colon if it exists in the token list. 

2503 if step is None: 

2504 result.append((self.slice_helper, node)) 

2505 else: 

2506 result.extend([ 

2507 (self.op, ':'), 

2508 (self.visit, step), 

2509 ]) 

2510 return result 

2511 

2512 def slice_helper(self, node: Node) -> ActionList: 

2513 """Delayed evaluation!""" 

2514 token = self.find_next_significant_token() 

2515 if token and token.value == ':': 

2516 return [ 

2517 (self.op, ':'), 

2518 ] 

2519 return [] 

2520 #@+node:ekr.20220402160128.20: *6* iterative.Str & helper 

2521 def do_Str(self, node: Node) -> ActionList: 

2522 """This node represents a string constant.""" 

2523 # This loop is necessary to handle string concatenation. 

2524 

2525 # for z in self.get_concatenated_string_tokens(): 

2526 # self.gen_token(z.kind, z.value) 

2527 

2528 return [ 

2529 (self.token, (z.kind, z.value)) 

2530 for z in self.get_concatenated_string_tokens() 

2531 ] 

2532 

2533 #@+node:ekr.20220402160128.21: *7* iterative.get_concatenated_tokens 

2534 def get_concatenated_string_tokens(self) -> List: 

2535 """ 

2536 Return the next 'string' token and all 'string' tokens concatenated to 

2537 it. *Never* update self.px here. 

2538 """ 

2539 trace = False 

2540 tag = 'iterative.get_concatenated_string_tokens' 

2541 i = self.px 

2542 # First, find the next significant token. It should be a string. 

2543 i, token = i + 1, None 

2544 while i < len(self.tokens): 

2545 token = self.tokens[i] 

2546 i += 1 

2547 if token.kind == 'string': 

2548 # Rescan the string. 

2549 i -= 1 

2550 break 

2551 # An error. 

2552 if is_significant_token(token): # pragma: no cover 

2553 break 

2554 # Raise an error if we didn't find the expected 'string' token. 

2555 if not token or token.kind != 'string': # pragma: no cover 

2556 if not token: 

2557 token = self.tokens[-1] 

2558 filename = getattr(self, 'filename', '<no filename>') 

2559 raise AssignLinksError( 

2560 f"\n" 

2561 f"{tag}...\n" 

2562 f"file: {filename}\n" 

2563 f"line: {token.line_number}\n" 

2564 f" i: {i}\n" 

2565 f"expected 'string' token, got {token!s}") 

2566 # Accumulate string tokens. 

2567 assert self.tokens[i].kind == 'string' 

2568 results = [] 

2569 while i < len(self.tokens): 

2570 token = self.tokens[i] 

2571 i += 1 

2572 if token.kind == 'string': 

2573 results.append(token) 

2574 elif token.kind == 'op' or is_significant_token(token): 

2575 # Any significant token *or* any op will halt string concatenation. 

2576 break 

2577 # 'ws', 'nl', 'newline', 'comment', 'indent', 'dedent', etc. 

2578 # The (significant) 'endmarker' token ensures we will have result. 

2579 assert results 

2580 if trace: # pragma: no cover 

2581 g.printObj(results, tag=f"{tag}: Results") 

2582 return results 

2583 #@+node:ekr.20220402160128.22: *6* iterative.Subscript 

2584 # Subscript(expr value, slice slice, expr_context ctx) 

2585 

2586 def do_Subscript(self, node: Node) -> ActionList: 

2587 

2588 return [ 

2589 (self.visit, node.value), 

2590 (self.op, '['), 

2591 (self.visit, node.slice), 

2592 (self.op, ']'), 

2593 ] 

2594 #@+node:ekr.20220402160128.23: *6* iterative.Tuple 

2595 # Tuple(expr* elts, expr_context ctx) 

2596 

2597 def do_Tuple(self, node: Node) -> ActionList: 

2598 

2599 # Do not call gen_op for parens or commas here. 

2600 # They do not necessarily exist in the token list! 

2601 

2602 return [ 

2603 (self.visit, node.elts), 

2604 ] 

2605 #@+node:ekr.20220330133336.40: *5* iterative: Operators 

2606 #@+node:ekr.20220330133336.41: *6* iterative.BinOp 

2607 def do_BinOp(self, node: Node) -> ActionList: 

2608 

2609 return [ 

2610 (self.visit, node.left), 

2611 (self.op, op_name(node.op)), 

2612 (self.visit, node.right), 

2613 ] 

2614 

2615 #@+node:ekr.20220330133336.42: *6* iterative.BoolOp 

2616 # BoolOp(boolop op, expr* values) 

2617 

2618 def do_BoolOp(self, node: Node) -> ActionList: 

2619 

2620 result: ActionList = [] 

2621 op_name_ = op_name(node.op) 

2622 for i, z in enumerate(node.values): 

2623 result.append((self.visit, z)) 

2624 if i < len(node.values) - 1: 

2625 result.append((self.name, op_name_)) 

2626 return result 

2627 #@+node:ekr.20220330133336.43: *6* iterative.Compare 

2628 # Compare(expr left, cmpop* ops, expr* comparators) 

2629 

2630 def do_Compare(self, node: Node) -> ActionList: 

2631 

2632 assert len(node.ops) == len(node.comparators) 

2633 result: List = [(self.visit, node.left)] 

2634 for i, z in enumerate(node.ops): 

2635 op_name_ = op_name(node.ops[i]) 

2636 if op_name_ in ('not in', 'is not'): 

2637 for z in op_name_.split(' '): 

2638 # self.name(z) 

2639 result.append((self.name, z)) 

2640 elif op_name_.isalpha(): 

2641 # self.name(op_name_) 

2642 result.append((self.name, op_name_)) 

2643 else: 

2644 # self.op(op_name_) 

2645 result.append((self.op, op_name_)) 

2646 # self.visit(node.comparators[i]) 

2647 result.append((self.visit, node.comparators[i])) 

2648 return result 

2649 #@+node:ekr.20220330133336.44: *6* iterative.UnaryOp 

2650 def do_UnaryOp(self, node: Node) -> ActionList: 

2651 

2652 op_name_ = op_name(node.op) 

2653 result: List = [] 

2654 if op_name_.isalpha(): 

2655 # self.name(op_name_) 

2656 result.append((self.name, op_name_)) 

2657 else: 

2658 # self.op(op_name_) 

2659 result.append((self.op, op_name_)) 

2660 # self.visit(node.operand) 

2661 result.append((self.visit, node.operand)) 

2662 return result 

2663 #@+node:ekr.20220330133336.45: *6* iterative.IfExp (ternary operator) 

2664 # IfExp(expr test, expr body, expr orelse) 

2665 

2666 def do_IfExp(self, node: Node) -> ActionList: 

2667 

2668 #'%s if %s else %s' 

2669 return [ 

2670 (self.visit, node.body), 

2671 (self.name, 'if'), 

2672 (self.visit, node.test), 

2673 (self.name, 'else'), 

2674 (self.visit, node.orelse), 

2675 ] 

2676 

2677 #@+node:ekr.20220330133336.46: *5* iterative: Statements 

2678 #@+node:ekr.20220330133336.47: *6* iterative.Starred 

2679 # Starred(expr value, expr_context ctx) 

2680 

2681 def do_Starred(self, node: Node) -> ActionList: 

2682 """A starred argument to an ast.Call""" 

2683 return [ 

2684 (self.op, '*'), 

2685 (self.visit, node.value), 

2686 ] 

2687 #@+node:ekr.20220330133336.48: *6* iterative.AnnAssign 

2688 # AnnAssign(expr target, expr annotation, expr? value, int simple) 

2689 

2690 def do_AnnAssign(self, node: Node) -> ActionList: 

2691 

2692 # {node.target}:{node.annotation}={node.value}\n' 

2693 result: ActionList = [ 

2694 (self.visit, node.target), 

2695 (self.op, ':'), 

2696 (self.visit, node.annotation), 

2697 ] 

2698 if node.value is not None: # #1851 

2699 result.extend([ 

2700 (self.op, '='), 

2701 (self.visit, node.value), 

2702 ]) 

2703 return result 

2704 #@+node:ekr.20220330133336.49: *6* iterative.Assert 

2705 # Assert(expr test, expr? msg) 

2706 

2707 def do_Assert(self, node: Node) -> ActionList: 

2708 

2709 # No need to put parentheses or commas. 

2710 msg = getattr(node, 'msg', None) 

2711 result: List = [ 

2712 (self.name, 'assert'), 

2713 (self.visit, node.test), 

2714 ] 

2715 if msg is not None: 

2716 result.append((self.visit, node.msg)) 

2717 return result 

2718 #@+node:ekr.20220330133336.50: *6* iterative.Assign 

2719 def do_Assign(self, node: Node) -> ActionList: 

2720 

2721 result: ActionList = [] 

2722 for z in node.targets: 

2723 result.extend([ 

2724 (self.visit, z), 

2725 (self.op, '=') 

2726 ]) 

2727 result.append((self.visit, node.value)) 

2728 return result 

2729 #@+node:ekr.20220330133336.51: *6* iterative.AsyncFor 

2730 def do_AsyncFor(self, node: Node) -> ActionList: 

2731 

2732 # The def line... 

2733 # Py 3.8 changes the kind of token. 

2734 async_token_type = 'async' if has_async_tokens else 'name' 

2735 result: List = [ 

2736 (self.token, (async_token_type, 'async')), 

2737 (self.name, 'for'), 

2738 (self.visit, node.target), 

2739 (self.name, 'in'), 

2740 (self.visit, node.iter), 

2741 (self.op, ':'), 

2742 # Body... 

2743 (self.visit, node.body), 

2744 ] 

2745 # Else clause... 

2746 if node.orelse: 

2747 result.extend([ 

2748 (self.name, 'else'), 

2749 (self.op, ':'), 

2750 (self.visit, node.orelse), 

2751 ]) 

2752 return result 

2753 #@+node:ekr.20220330133336.52: *6* iterative.AsyncWith 

2754 def do_AsyncWith(self, node: Node) -> ActionList: 

2755 

2756 async_token_type = 'async' if has_async_tokens else 'name' 

2757 return [ 

2758 (self.token, (async_token_type, 'async')), 

2759 (self.do_With, node), 

2760 ] 

2761 #@+node:ekr.20220330133336.53: *6* iterative.AugAssign 

2762 # AugAssign(expr target, operator op, expr value) 

2763 

2764 def do_AugAssign(self, node: Node) -> ActionList: 

2765 

2766 # %s%s=%s\n' 

2767 return [ 

2768 (self.visit, node.target), 

2769 (self.op, op_name(node.op) + '='), 

2770 (self.visit, node.value), 

2771 ] 

2772 #@+node:ekr.20220330133336.54: *6* iterative.Await 

2773 # Await(expr value) 

2774 

2775 def do_Await(self, node: Node) -> ActionList: 

2776 

2777 #'await %s\n' 

2778 async_token_type = 'await' if has_async_tokens else 'name' 

2779 return [ 

2780 (self.token, (async_token_type, 'await')), 

2781 (self.visit, node.value), 

2782 ] 

2783 #@+node:ekr.20220330133336.55: *6* iterative.Break 

2784 def do_Break(self, node: Node) -> ActionList: 

2785 

2786 return [ 

2787 (self.name, 'break'), 

2788 ] 

2789 #@+node:ekr.20220330133336.56: *6* iterative.Call & helpers 

2790 # Call(expr func, expr* args, keyword* keywords) 

2791 

2792 # Python 3 ast.Call nodes do not have 'starargs' or 'kwargs' fields. 

2793 

2794 def do_Call(self, node: Node) -> ActionList: 

2795 

2796 # The calls to op(')') and op('(') do nothing by default. 

2797 # No need to generate any commas. 

2798 # Subclasses might handle them in an overridden iterative.set_links. 

2799 return [ 

2800 (self.visit, node.func), 

2801 (self.op, '('), 

2802 (self.handle_call_arguments, node), 

2803 (self.op, ')'), 

2804 ] 

2805 #@+node:ekr.20220330133336.57: *7* iterative.arg_helper 

2806 def arg_helper(self, node: Node) -> ActionList: 

2807 """ 

2808 Yield the node, with a special case for strings. 

2809 """ 

2810 result: List = [] 

2811 if isinstance(node, str): 

2812 result.append((self.token, ('name', node))) 

2813 else: 

2814 result.append((self.visit, node)) 

2815 return result 

2816 #@+node:ekr.20220330133336.58: *7* iterative.handle_call_arguments 

2817 def handle_call_arguments(self, node: Node) -> ActionList: 

2818 """ 

2819 Generate arguments in the correct order. 

2820 

2821 Call(expr func, expr* args, keyword* keywords) 

2822 

2823 https://docs.python.org/3/reference/expressions.html#calls 

2824 

2825 Warning: This code will fail on Python 3.8 only for calls 

2826 containing kwargs in unexpected places. 

2827 """ 

2828 # *args: in node.args[]: Starred(value=Name(id='args')) 

2829 # *[a, 3]: in node.args[]: Starred(value=List(elts=[Name(id='a'), Num(n=3)]) 

2830 # **kwargs: in node.keywords[]: keyword(arg=None, value=Name(id='kwargs')) 

2831 # 

2832 # Scan args for *name or *List 

2833 args = node.args or [] 

2834 keywords = node.keywords or [] 

2835 

2836 def get_pos(obj: Any) -> Tuple: 

2837 line1 = getattr(obj, 'lineno', None) 

2838 col1 = getattr(obj, 'col_offset', None) 

2839 return line1, col1, obj 

2840 

2841 def sort_key(aTuple: Tuple) -> int: 

2842 line, col, obj = aTuple 

2843 return line * 1000 + col 

2844 

2845 assert py_version >= (3, 9) 

2846 

2847 places = [get_pos(z) for z in args + keywords] 

2848 places.sort(key=sort_key) 

2849 ordered_args = [z[2] for z in places] 

2850 result: ActionList = [] 

2851 for z in ordered_args: 

2852 if isinstance(z, ast.Starred): 

2853 result.extend([ 

2854 (self.op, '*'), 

2855 (self.visit, z.value), 

2856 ]) 

2857 elif isinstance(z, ast.keyword): 

2858 if getattr(z, 'arg', None) is None: 

2859 result.extend([ 

2860 (self.op, '**'), 

2861 (self.arg_helper, z.value), 

2862 ]) 

2863 else: 

2864 result.extend([ 

2865 (self.arg_helper, z.arg), 

2866 (self.op, '='), 

2867 (self.arg_helper, z.value), 

2868 ]) 

2869 else: 

2870 result.append((self.arg_helper, z)) 

2871 return result 

2872 #@+node:ekr.20220330133336.59: *6* iterative.Continue 

2873 def do_Continue(self, node: Node) -> ActionList: 

2874 

2875 return [ 

2876 (self.name, 'continue'), 

2877 ] 

2878 #@+node:ekr.20220330133336.60: *6* iterative.Delete 

2879 def do_Delete(self, node: Node) -> ActionList: 

2880 

2881 # No need to put commas. 

2882 return [ 

2883 (self.name, 'del'), 

2884 (self.visit, node.targets), 

2885 ] 

2886 #@+node:ekr.20220330133336.61: *6* iterative.ExceptHandler 

2887 def do_ExceptHandler(self, node: Node) -> ActionList: 

2888 

2889 # Except line... 

2890 result: List = [ 

2891 (self.name, 'except'), 

2892 ] 

2893 if getattr(node, 'type', None): 

2894 result.append((self.visit, node.type)) 

2895 if getattr(node, 'name', None): 

2896 result.extend([ 

2897 (self.name, 'as'), 

2898 (self.name, node.name), 

2899 ]) 

2900 result.extend([ 

2901 (self.op, ':'), 

2902 # Body... 

2903 (self.visit, node.body), 

2904 ]) 

2905 return result 

2906 #@+node:ekr.20220330133336.62: *6* iterative.For 

2907 def do_For(self, node: Node) -> ActionList: 

2908 

2909 result: ActionList = [ 

2910 # The def line... 

2911 (self.name, 'for'), 

2912 (self.visit, node.target), 

2913 (self.name, 'in'), 

2914 (self.visit, node.iter), 

2915 (self.op, ':'), 

2916 # Body... 

2917 (self.visit, node.body), 

2918 ] 

2919 # Else clause... 

2920 if node.orelse: 

2921 result.extend([ 

2922 (self.name, 'else'), 

2923 (self.op, ':'), 

2924 (self.visit, node.orelse), 

2925 ]) 

2926 return result 

2927 #@+node:ekr.20220330133336.63: *6* iterative.Global 

2928 # Global(identifier* names) 

2929 

2930 def do_Global(self, node: Node) -> ActionList: 

2931 

2932 result = [ 

2933 (self.name, 'global'), 

2934 ] 

2935 for z in node.names: 

2936 result.append((self.name, z)) 

2937 return result 

2938 #@+node:ekr.20220330133336.64: *6* iterative.If & helpers 

2939 # If(expr test, stmt* body, stmt* orelse) 

2940 

2941 def do_If(self, node: Node) -> ActionList: 

2942 #@+<< do_If docstring >> 

2943 #@+node:ekr.20220330133336.65: *7* << do_If docstring >> 

2944 """ 

2945 The parse trees for the following are identical! 

2946 

2947 if 1: if 1: 

2948 pass pass 

2949 else: elif 2: 

2950 if 2: pass 

2951 pass 

2952 

2953 So there is *no* way for the 'if' visitor to disambiguate the above two 

2954 cases from the parse tree alone. 

2955 

2956 Instead, we scan the tokens list for the next 'if', 'else' or 'elif' token. 

2957 """ 

2958 #@-<< do_If docstring >> 

2959 # Use the next significant token to distinguish between 'if' and 'elif'. 

2960 token = self.find_next_significant_token() 

2961 result: List = [ 

2962 (self.name, token.value), 

2963 (self.visit, node.test), 

2964 (self.op, ':'), 

2965 # Body... 

2966 (self.visit, node.body), 

2967 ] 

2968 # Else and elif clauses... 

2969 if node.orelse: 

2970 # We *must* delay the evaluation of the else clause. 

2971 result.append((self.if_else_helper, node)) 

2972 return result 

2973 

2974 def if_else_helper(self, node: Node) -> ActionList: 

2975 """Delayed evaluation!""" 

2976 token = self.find_next_significant_token() 

2977 if token.value == 'else': 

2978 return [ 

2979 (self.name, 'else'), 

2980 (self.op, ':'), 

2981 (self.visit, node.orelse), 

2982 ] 

2983 return [ 

2984 (self.visit, node.orelse), 

2985 ] 

2986 #@+node:ekr.20220330133336.66: *6* iterative.Import & helper 

2987 def do_Import(self, node: Node) -> ActionList: 

2988 

2989 result: List = [ 

2990 (self.name, 'import'), 

2991 ] 

2992 for alias in node.names: 

2993 result.append((self.name, alias.name)) 

2994 if alias.asname: 

2995 result.extend([ 

2996 (self.name, 'as'), 

2997 (self.name, alias.asname), 

2998 ]) 

2999 return result 

3000 #@+node:ekr.20220330133336.67: *6* iterative.ImportFrom 

3001 # ImportFrom(identifier? module, alias* names, int? level) 

3002 

3003 def do_ImportFrom(self, node: Node) -> ActionList: 

3004 

3005 result: List = [ 

3006 (self.name, 'from'), 

3007 ] 

3008 for i in range(node.level): 

3009 result.append((self.op, '.')) 

3010 if node.module: 

3011 result.append((self.name, node.module)) 

3012 result.append((self.name, 'import')) 

3013 # No need to put commas. 

3014 for alias in node.names: 

3015 if alias.name == '*': # #1851. 

3016 result.append((self.op, '*')) 

3017 else: 

3018 result.append((self.name, alias.name)) 

3019 if alias.asname: 

3020 result.extend([ 

3021 (self.name, 'as'), 

3022 (self.name, alias.asname), 

3023 ]) 

3024 return result 

3025 #@+node:ekr.20220402124844.1: *6* iterative.Match* (Python 3.10+) 

3026 # Match(expr subject, match_case* cases) 

3027 

3028 # match_case = (pattern pattern, expr? guard, stmt* body) 

3029 

3030 # Full syntax diagram: # https://peps.python.org/pep-0634/#appendix-a 

3031 

3032 def do_Match(self, node: Node) -> ActionList: 

3033 

3034 cases = getattr(node, 'cases', []) 

3035 result: List = [ 

3036 (self.name, 'match'), 

3037 (self.visit, node.subject), 

3038 (self.op, ':'), 

3039 ] 

3040 for case in cases: 

3041 result.append((self.visit, case)) 

3042 return result 

3043 #@+node:ekr.20220402124844.2: *7* iterative.match_case 

3044 # match_case = (pattern pattern, expr? guard, stmt* body) 

3045 

3046 def do_match_case(self, node: Node) -> ActionList: 

3047 

3048 guard = getattr(node, 'guard', None) 

3049 body = getattr(node, 'body', []) 

3050 result: List = [ 

3051 (self.name, 'case'), 

3052 (self.visit, node.pattern), 

3053 ] 

3054 if guard: 

3055 result.extend([ 

3056 (self.name, 'if'), 

3057 (self.visit, guard), 

3058 ]) 

3059 result.append((self.op, ':')) 

3060 for statement in body: 

3061 result.append((self.visit, statement)) 

3062 return result 

3063 #@+node:ekr.20220402124844.3: *7* iterative.MatchAs 

3064 # MatchAs(pattern? pattern, identifier? name) 

3065 

3066 def do_MatchAs(self, node: Node) -> ActionList: 

3067 pattern = getattr(node, 'pattern', None) 

3068 name = getattr(node, 'name', None) 

3069 result: ActionList = [] 

3070 if pattern and name: 

3071 result.extend([ 

3072 (self.visit, pattern), 

3073 (self.name, 'as'), 

3074 (self.name, name), 

3075 ]) 

3076 elif pattern: 

3077 result.append((self.visit, pattern)) # pragma: no cover 

3078 else: 

3079 result.append((self.name, name or '_')) 

3080 return result 

3081 #@+node:ekr.20220402124844.4: *7* iterative.MatchClass 

3082 # MatchClass(expr cls, pattern* patterns, identifier* kwd_attrs, pattern* kwd_patterns) 

3083 

3084 def do_MatchClass(self, node: Node) -> ActionList: 

3085 

3086 cls = node.cls 

3087 patterns = getattr(node, 'patterns', []) 

3088 kwd_attrs = getattr(node, 'kwd_attrs', []) 

3089 kwd_patterns = getattr(node, 'kwd_patterns', []) 

3090 result: List = [ 

3091 (self.visit, node.cls), 

3092 (self.op, '('), 

3093 ] 

3094 for pattern in patterns: 

3095 result.append((self.visit, pattern)) 

3096 for i, kwd_attr in enumerate(kwd_attrs): 

3097 result.extend([ 

3098 (self.name, kwd_attr), # a String. 

3099 (self.op, '='), 

3100 (self.visit, kwd_patterns[i]), 

3101 ]) 

3102 result.append((self.op, ')')) 

3103 return result 

3104 #@+node:ekr.20220402124844.5: *7* iterative.MatchMapping 

3105 # MatchMapping(expr* keys, pattern* patterns, identifier? rest) 

3106 

3107 def do_MatchMapping(self, node: Node) -> ActionList: 

3108 keys = getattr(node, 'keys', []) 

3109 patterns = getattr(node, 'patterns', []) 

3110 rest = getattr(node, 'rest', None) 

3111 result: List = [ 

3112 (self.op, '{'), 

3113 ] 

3114 for i, key in enumerate(keys): 

3115 result.extend([ 

3116 (self.visit, key), 

3117 (self.op, ':'), 

3118 (self.visit, patterns[i]), 

3119 ]) 

3120 if rest: 

3121 result.extend([ 

3122 (self.op, '**'), 

3123 (self.name, rest), # A string. 

3124 ]) 

3125 result.append((self.op, '}')) 

3126 return result 

3127 #@+node:ekr.20220402124844.6: *7* iterative.MatchOr 

3128 # MatchOr(pattern* patterns) 

3129 

3130 def do_MatchOr(self, node: Node) -> ActionList: 

3131 

3132 patterns = getattr(node, 'patterns', []) 

3133 result: List = [] 

3134 for i, pattern in enumerate(patterns): 

3135 if i > 0: 

3136 result.append((self.op, '|')) 

3137 result.append((self.visit, pattern)) 

3138 return result 

3139 #@+node:ekr.20220402124844.7: *7* iterative.MatchSequence 

3140 # MatchSequence(pattern* patterns) 

3141 

3142 def do_MatchSequence(self, node: Node) -> ActionList: 

3143 patterns = getattr(node, 'patterns', []) 

3144 result: List = [] 

3145 # Scan for the next '(' or '[' token, skipping the 'case' token. 

3146 token = None 

3147 for token in self.tokens[self.px + 1 :]: 

3148 if token.kind == 'op' and token.value in '([': 

3149 break 

3150 if is_significant_token(token): 

3151 # An implicit tuple: there is no '(' or '[' token. 

3152 token = None 

3153 break 

3154 else: 

3155 raise AssignLinksError('Ill-formed tuple') # pragma: no cover 

3156 if token: 

3157 result.append((self.op, token.value)) 

3158 for i, pattern in enumerate(patterns): 

3159 result.append((self.visit, pattern)) 

3160 if token: 

3161 val = ']' if token.value == '[' else ')' 

3162 result.append((self.op, val)) 

3163 return result 

3164 #@+node:ekr.20220402124844.8: *7* iterative.MatchSingleton 

3165 # MatchSingleton(constant value) 

3166 

3167 def do_MatchSingleton(self, node: Node) -> ActionList: 

3168 """Match True, False or None.""" 

3169 return [ 

3170 (self.token, ('name', repr(node.value))), 

3171 ] 

3172 #@+node:ekr.20220402124844.9: *7* iterative.MatchStar 

3173 # MatchStar(identifier? name) 

3174 

3175 def do_MatchStar(self, node: Node) -> ActionList: 

3176 

3177 name = getattr(node, 'name', None) 

3178 result: List = [ 

3179 (self.op, '*'), 

3180 ] 

3181 if name: 

3182 result.append((self.name, name)) 

3183 return result 

3184 #@+node:ekr.20220402124844.10: *7* iterative.MatchValue 

3185 # MatchValue(expr value) 

3186 

3187 def do_MatchValue(self, node: Node) -> ActionList: 

3188 

3189 return [ 

3190 (self.visit, node.value), 

3191 ] 

3192 #@+node:ekr.20220330133336.78: *6* iterative.Nonlocal 

3193 # Nonlocal(identifier* names) 

3194 

3195 def do_Nonlocal(self, node: Node) -> ActionList: 

3196 

3197 # nonlocal %s\n' % ','.join(node.names)) 

3198 # No need to put commas. 

3199 result: List = [ 

3200 (self.name, 'nonlocal'), 

3201 ] 

3202 for z in node.names: 

3203 result.append((self.name, z)) 

3204 return result 

3205 #@+node:ekr.20220330133336.79: *6* iterative.Pass 

3206 def do_Pass(self, node: Node) -> ActionList: 

3207 

3208 return ([ 

3209 (self.name, 'pass'), 

3210 ]) 

3211 #@+node:ekr.20220330133336.80: *6* iterative.Raise 

3212 # Raise(expr? exc, expr? cause) 

3213 

3214 def do_Raise(self, node: Node) -> ActionList: 

3215 

3216 # No need to put commas. 

3217 exc = getattr(node, 'exc', None) 

3218 cause = getattr(node, 'cause', None) 

3219 tback = getattr(node, 'tback', None) 

3220 result: List = [ 

3221 (self.name, 'raise'), 

3222 (self.visit, exc), 

3223 ] 

3224 if cause: 

3225 result.extend([ 

3226 (self.name, 'from'), # #2446. 

3227 (self.visit, cause), 

3228 ]) 

3229 result.append((self.visit, tback)) 

3230 return result 

3231 

3232 #@+node:ekr.20220330133336.81: *6* iterative.Return 

3233 def do_Return(self, node: Node) -> ActionList: 

3234 

3235 return [ 

3236 (self.name, 'return'), 

3237 (self.visit, node.value), 

3238 ] 

3239 #@+node:ekr.20220330133336.82: *6* iterative.Try 

3240 # Try(stmt* body, excepthandler* handlers, stmt* orelse, stmt* finalbody) 

3241 

3242 def do_Try(self, node: Node) -> ActionList: 

3243 

3244 result: List = [ 

3245 # Try line... 

3246 (self.name, 'try'), 

3247 (self.op, ':'), 

3248 # Body... 

3249 (self.visit, node.body), 

3250 (self.visit, node.handlers), 

3251 ] 

3252 # Else... 

3253 if node.orelse: 

3254 result.extend([ 

3255 (self.name, 'else'), 

3256 (self.op, ':'), 

3257 (self.visit, node.orelse), 

3258 ]) 

3259 # Finally... 

3260 if node.finalbody: 

3261 result.extend([ 

3262 (self.name, 'finally'), 

3263 (self.op, ':'), 

3264 (self.visit, node.finalbody), 

3265 ]) 

3266 return result 

3267 #@+node:ekr.20220330133336.83: *6* iterative.While 

3268 def do_While(self, node: Node) -> ActionList: 

3269 

3270 # While line... 

3271 # while %s:\n' 

3272 result: List = [ 

3273 (self.name, 'while'), 

3274 (self.visit, node.test), 

3275 (self.op, ':'), 

3276 # Body... 

3277 (self.visit, node.body), 

3278 ] 

3279 # Else clause... 

3280 if node.orelse: 

3281 result.extend([ 

3282 (self.name, 'else'), 

3283 (self.op, ':'), 

3284 (self.visit, node.orelse), 

3285 ]) 

3286 return result 

3287 #@+node:ekr.20220330133336.84: *6* iterative.With 

3288 # With(withitem* items, stmt* body) 

3289 

3290 # withitem = (expr context_expr, expr? optional_vars) 

3291 

3292 def do_With(self, node: Node) -> ActionList: 

3293 

3294 expr: Optional[ast.AST] = getattr(node, 'context_expression', None) 

3295 items: List[ast.AST] = getattr(node, 'items', []) 

3296 result: List = [ 

3297 (self.name, 'with'), 

3298 (self.visit, expr), 

3299 ] 

3300 # No need to put commas. 

3301 for item in items: 

3302 result.append((self.visit, item.context_expr)) 

3303 optional_vars = getattr(item, 'optional_vars', None) 

3304 if optional_vars is not None: 

3305 result.extend([ 

3306 (self.name, 'as'), 

3307 (self.visit, item.optional_vars), 

3308 ]) 

3309 result.extend([ 

3310 # End the line. 

3311 (self.op, ':'), 

3312 # Body... 

3313 (self.visit, node.body), 

3314 ]) 

3315 return result 

3316 #@+node:ekr.20220330133336.85: *6* iterative.Yield 

3317 def do_Yield(self, node: Node) -> ActionList: 

3318 

3319 result: List = [ 

3320 (self.name, 'yield'), 

3321 ] 

3322 if hasattr(node, 'value'): 

3323 result.extend([ 

3324 (self.visit, node.value), 

3325 ]) 

3326 return result 

3327 #@+node:ekr.20220330133336.86: *6* iterative.YieldFrom 

3328 # YieldFrom(expr value) 

3329 

3330 def do_YieldFrom(self, node: Node) -> ActionList: 

3331 

3332 return ([ 

3333 (self.name, 'yield'), 

3334 (self.name, 'from'), 

3335 (self.visit, node.value), 

3336 ]) 

3337 #@-others 

3338#@+node:ekr.20200107165250.1: *3* class Orange 

3339class Orange: 

3340 """ 

3341 A flexible and powerful beautifier for Python. 

3342 Orange is the new black. 

3343 

3344 *Important*: This is a predominantly a *token*-based beautifier. 

3345 However, orange.colon and orange.possible_unary_op use the parse 

3346 tree to provide context that would otherwise be difficult to 

3347 deduce. 

3348 """ 

3349 # This switch is really a comment. It will always be false. 

3350 # It marks the code that simulates the operation of the black tool. 

3351 black_mode = False 

3352 

3353 # Patterns... 

3354 nobeautify_pat = re.compile(r'\s*#\s*pragma:\s*no\s*beautify\b|#\s*@@nobeautify') 

3355 

3356 # Patterns from FastAtRead class, specialized for python delims. 

3357 node_pat = re.compile(r'^(\s*)#@\+node:([^:]+): \*(\d+)?(\*?) (.*)$') # @node 

3358 start_doc_pat = re.compile(r'^\s*#@\+(at|doc)?(\s.*?)?$') # @doc or @ 

3359 at_others_pat = re.compile(r'^(\s*)#@(\+|-)others\b(.*)$') # @others 

3360 

3361 # Doc parts end with @c or a node sentinel. Specialized for python. 

3362 end_doc_pat = re.compile(r"^\s*#@(@(c(ode)?)|([+]node\b.*))$") 

3363 #@+others 

3364 #@+node:ekr.20200107165250.2: *4* orange.ctor 

3365 def __init__(self, settings: Optional[Dict]=None): 

3366 """Ctor for Orange class.""" 

3367 if settings is None: 

3368 settings = {} 

3369 valid_keys = ( 

3370 'allow_joined_strings', 

3371 'max_join_line_length', 

3372 'max_split_line_length', 

3373 'orange', 

3374 'tab_width', 

3375 ) 

3376 # For mypy... 

3377 self.kind: str = '' 

3378 # Default settings... 

3379 self.allow_joined_strings = False # EKR's preference. 

3380 self.max_join_line_length = 88 

3381 self.max_split_line_length = 88 

3382 self.tab_width = 4 

3383 # Override from settings dict... 

3384 for key in settings: # pragma: no cover 

3385 value = settings.get(key) 

3386 if key in valid_keys and value is not None: 

3387 setattr(self, key, value) 

3388 else: 

3389 g.trace(f"Unexpected setting: {key} = {value!r}") 

3390 #@+node:ekr.20200107165250.51: *4* orange.push_state 

3391 def push_state(self, kind: str, value: Any=None) -> None: 

3392 """Append a state to the state stack.""" 

3393 state = ParseState(kind, value) 

3394 self.state_stack.append(state) 

3395 #@+node:ekr.20200107165250.8: *4* orange: Entries 

3396 #@+node:ekr.20200107173542.1: *5* orange.beautify (main token loop) 

3397 def oops(self) -> None: # pragma: no cover 

3398 g.trace(f"Unknown kind: {self.kind}") 

3399 

3400 def beautify(self, contents: str, filename: str, tokens: List["Token"], tree: Node, 

3401 

3402 max_join_line_length: Optional[int]=None, max_split_line_length: Optional[int]=None, 

3403 ) -> str: 

3404 """ 

3405 The main line. Create output tokens and return the result as a string. 

3406 """ 

3407 # Config overrides 

3408 if max_join_line_length is not None: 

3409 self.max_join_line_length = max_join_line_length 

3410 if max_split_line_length is not None: 

3411 self.max_split_line_length = max_split_line_length 

3412 # State vars... 

3413 self.curly_brackets_level = 0 # Number of unmatched '{' tokens. 

3414 self.decorator_seen = False # Set by do_name for do_op. 

3415 self.in_arg_list = 0 # > 0 if in an arg list of a def. 

3416 self.level = 0 # Set only by do_indent and do_dedent. 

3417 self.lws = '' # Leading whitespace. 

3418 self.paren_level = 0 # Number of unmatched '(' tokens. 

3419 self.square_brackets_stack: List[bool] = [] # A stack of bools, for self.word(). 

3420 self.state_stack: List["ParseState"] = [] # Stack of ParseState objects. 

3421 self.val = None # The input token's value (a string). 

3422 self.verbatim = False # True: don't beautify. 

3423 # 

3424 # Init output list and state... 

3425 self.code_list: List[Token] = [] # The list of output tokens. 

3426 self.code_list_index = 0 # The token's index. 

3427 self.tokens = tokens # The list of input tokens. 

3428 self.tree = tree 

3429 self.add_token('file-start', '') 

3430 self.push_state('file-start') 

3431 for i, token in enumerate(tokens): 

3432 self.token = token 

3433 self.kind, self.val, self.line = token.kind, token.value, token.line 

3434 if self.verbatim: 

3435 self.do_verbatim() 

3436 else: 

3437 func = getattr(self, f"do_{token.kind}", self.oops) 

3438 func() 

3439 # Any post pass would go here. 

3440 return tokens_to_string(self.code_list) 

3441 #@+node:ekr.20200107172450.1: *5* orange.beautify_file (entry) 

3442 def beautify_file(self, filename: str) -> bool: # pragma: no cover 

3443 """ 

3444 Orange: Beautify the the given external file. 

3445 

3446 Return True if the file was changed. 

3447 """ 

3448 self.filename = filename 

3449 tog = TokenOrderGenerator() 

3450 contents, encoding, tokens, tree = tog.init_from_file(filename) 

3451 if not contents or not tokens or not tree: 

3452 return False # #2529: Not an error. 

3453 # Beautify. 

3454 results = self.beautify(contents, filename, tokens, tree) 

3455 # Something besides newlines must change. 

3456 if regularize_nls(contents) == regularize_nls(results): 

3457 return False 

3458 if 0: # This obscures more import error messages. 

3459 show_diffs(contents, results, filename=filename) 

3460 # Write the results 

3461 print(f"Beautified: {g.shortFileName(filename)}") 

3462 write_file(filename, results, encoding=encoding) 

3463 return True 

3464 #@+node:ekr.20200107172512.1: *5* orange.beautify_file_diff (entry) 

3465 def beautify_file_diff(self, filename: str) -> bool: # pragma: no cover 

3466 """ 

3467 Orange: Print the diffs that would resulf from the orange-file command. 

3468 

3469 Return True if the file would be changed. 

3470 """ 

3471 tag = 'diff-beautify-file' 

3472 self.filename = filename 

3473 tog = TokenOrderGenerator() 

3474 contents, encoding, tokens, tree = tog.init_from_file(filename) 

3475 if not contents or not tokens or not tree: 

3476 print(f"{tag}: Can not beautify: {filename}") 

3477 return False 

3478 # fstringify. 

3479 results = self.beautify(contents, filename, tokens, tree) 

3480 # Something besides newlines must change. 

3481 if regularize_nls(contents) == regularize_nls(results): 

3482 print(f"{tag}: Unchanged: {filename}") 

3483 return False 

3484 # Show the diffs. 

3485 show_diffs(contents, results, filename=filename) 

3486 return True 

3487 #@+node:ekr.20200107165250.13: *4* orange: Input token handlers 

3488 #@+node:ekr.20200107165250.14: *5* orange.do_comment 

3489 in_doc_part = False 

3490 

3491 def do_comment(self) -> None: 

3492 """Handle a comment token.""" 

3493 val = self.val 

3494 # 

3495 # Leo-specific code... 

3496 if self.node_pat.match(val): 

3497 # Clear per-node state. 

3498 self.in_doc_part = False 

3499 self.verbatim = False 

3500 self.decorator_seen = False 

3501 # Do *not clear other state, which may persist across @others. 

3502 # self.curly_brackets_level = 0 

3503 # self.in_arg_list = 0 

3504 # self.level = 0 

3505 # self.lws = '' 

3506 # self.paren_level = 0 

3507 # self.square_brackets_stack = [] 

3508 # self.state_stack = [] 

3509 else: 

3510 # Keep track of verbatim mode. 

3511 if self.beautify_pat.match(val): 

3512 self.verbatim = False 

3513 elif self.nobeautify_pat.match(val): 

3514 self.verbatim = True 

3515 # Keep trace of @doc parts, to honor the convention for splitting lines. 

3516 if self.start_doc_pat.match(val): 

3517 self.in_doc_part = True 

3518 if self.end_doc_pat.match(val): 

3519 self.in_doc_part = False 

3520 # 

3521 # General code: Generate the comment. 

3522 self.clean('blank') 

3523 entire_line = self.line.lstrip().startswith('#') 

3524 if entire_line: 

3525 self.clean('hard-blank') 

3526 self.clean('line-indent') 

3527 # #1496: No further munging needed. 

3528 val = self.line.rstrip() 

3529 else: 

3530 # Exactly two spaces before trailing comments. 

3531 val = ' ' + self.val.rstrip() 

3532 self.add_token('comment', val) 

3533 #@+node:ekr.20200107165250.15: *5* orange.do_encoding 

3534 def do_encoding(self) -> None: 

3535 """ 

3536 Handle the encoding token. 

3537 """ 

3538 pass 

3539 #@+node:ekr.20200107165250.16: *5* orange.do_endmarker 

3540 def do_endmarker(self) -> None: 

3541 """Handle an endmarker token.""" 

3542 # Ensure exactly one blank at the end of the file. 

3543 self.clean_blank_lines() 

3544 self.add_token('line-end', '\n') 

3545 #@+node:ekr.20200107165250.18: *5* orange.do_indent & do_dedent & helper 

3546 def do_dedent(self) -> None: 

3547 """Handle dedent token.""" 

3548 self.level -= 1 

3549 self.lws = self.level * self.tab_width * ' ' 

3550 self.line_indent() 

3551 if self.black_mode: # pragma: no cover (black) 

3552 state = self.state_stack[-1] 

3553 if state.kind == 'indent' and state.value == self.level: 

3554 self.state_stack.pop() 

3555 state = self.state_stack[-1] 

3556 if state.kind in ('class', 'def'): 

3557 self.state_stack.pop() 

3558 self.handle_dedent_after_class_or_def(state.kind) 

3559 

3560 def do_indent(self) -> None: 

3561 """Handle indent token.""" 

3562 new_indent = self.val 

3563 old_indent = self.level * self.tab_width * ' ' 

3564 if new_indent > old_indent: 

3565 self.level += 1 

3566 elif new_indent < old_indent: # pragma: no cover (defensive) 

3567 g.trace('\n===== can not happen', repr(new_indent), repr(old_indent)) 

3568 self.lws = new_indent 

3569 self.line_indent() 

3570 #@+node:ekr.20200220054928.1: *6* orange.handle_dedent_after_class_or_def 

3571 def handle_dedent_after_class_or_def(self, kind: str) -> None: # pragma: no cover (black) 

3572 """ 

3573 Insert blank lines after a class or def as the result of a 'dedent' token. 

3574 

3575 Normal comment lines may precede the 'dedent'. 

3576 Insert the blank lines *before* such comment lines. 

3577 """ 

3578 # 

3579 # Compute the tail. 

3580 i = len(self.code_list) - 1 

3581 tail: List[Token] = [] 

3582 while i > 0: 

3583 t = self.code_list.pop() 

3584 i -= 1 

3585 if t.kind == 'line-indent': 

3586 pass 

3587 elif t.kind == 'line-end': 

3588 tail.insert(0, t) 

3589 elif t.kind == 'comment': 

3590 # Only underindented single-line comments belong in the tail. 

3591 # @+node comments must never be in the tail. 

3592 single_line = self.code_list[i].kind in ('line-end', 'line-indent') 

3593 lws = len(t.value) - len(t.value.lstrip()) 

3594 underindent = lws <= len(self.lws) 

3595 if underindent and single_line and not self.node_pat.match(t.value): 

3596 # A single-line comment. 

3597 tail.insert(0, t) 

3598 else: 

3599 self.code_list.append(t) 

3600 break 

3601 else: 

3602 self.code_list.append(t) 

3603 break 

3604 # 

3605 # Remove leading 'line-end' tokens from the tail. 

3606 while tail and tail[0].kind == 'line-end': 

3607 tail = tail[1:] 

3608 # 

3609 # Put the newlines *before* the tail. 

3610 # For Leo, always use 1 blank lines. 

3611 n = 1 # n = 2 if kind == 'class' else 1 

3612 # Retain the token (intention) for debugging. 

3613 self.add_token('blank-lines', n) 

3614 for i in range(0, n + 1): 

3615 self.add_token('line-end', '\n') 

3616 if tail: 

3617 self.code_list.extend(tail) 

3618 self.line_indent() 

3619 #@+node:ekr.20200107165250.20: *5* orange.do_name 

3620 def do_name(self) -> None: 

3621 """Handle a name token.""" 

3622 name = self.val 

3623 if self.black_mode and name in ('class', 'def'): # pragma: no cover (black) 

3624 # Handle newlines before and after 'class' or 'def' 

3625 self.decorator_seen = False 

3626 state = self.state_stack[-1] 

3627 if state.kind == 'decorator': 

3628 # Always do this, regardless of @bool clean-blank-lines. 

3629 self.clean_blank_lines() 

3630 # Suppress split/join. 

3631 self.add_token('hard-newline', '\n') 

3632 self.add_token('line-indent', self.lws) 

3633 self.state_stack.pop() 

3634 else: 

3635 # Always do this, regardless of @bool clean-blank-lines. 

3636 self.blank_lines(2 if name == 'class' else 1) 

3637 self.push_state(name) 

3638 self.push_state('indent', self.level) 

3639 # For trailing lines after inner classes/defs. 

3640 self.word(name) 

3641 return 

3642 # 

3643 # Leo mode... 

3644 if name in ('class', 'def'): 

3645 self.word(name) 

3646 elif name in ( 

3647 'and', 'elif', 'else', 'for', 'if', 'in', 'not', 'not in', 'or', 'while' 

3648 ): 

3649 self.word_op(name) 

3650 else: 

3651 self.word(name) 

3652 #@+node:ekr.20200107165250.21: *5* orange.do_newline & do_nl 

3653 def do_newline(self) -> None: 

3654 """Handle a regular newline.""" 

3655 self.line_end() 

3656 

3657 def do_nl(self) -> None: 

3658 """Handle a continuation line.""" 

3659 self.line_end() 

3660 #@+node:ekr.20200107165250.22: *5* orange.do_number 

3661 def do_number(self) -> None: 

3662 """Handle a number token.""" 

3663 self.blank() 

3664 self.add_token('number', self.val) 

3665 #@+node:ekr.20200107165250.23: *5* orange.do_op 

3666 def do_op(self) -> None: 

3667 """Handle an op token.""" 

3668 val = self.val 

3669 if val == '.': 

3670 self.clean('blank') 

3671 prev = self.code_list[-1] 

3672 # #2495 & #2533: Special case for 'from .' 

3673 if prev.kind == 'word' and prev.value == 'from': 

3674 self.blank() 

3675 self.add_token('op-no-blanks', val) 

3676 elif val == '@': 

3677 if self.black_mode: # pragma: no cover (black) 

3678 if not self.decorator_seen: 

3679 self.blank_lines(1) 

3680 self.decorator_seen = True 

3681 self.clean('blank') 

3682 self.add_token('op-no-blanks', val) 

3683 self.push_state('decorator') 

3684 elif val == ':': 

3685 # Treat slices differently. 

3686 self.colon(val) 

3687 elif val in ',;': 

3688 # Pep 8: Avoid extraneous whitespace immediately before 

3689 # comma, semicolon, or colon. 

3690 self.clean('blank') 

3691 self.add_token('op', val) 

3692 self.blank() 

3693 elif val in '([{': 

3694 # Pep 8: Avoid extraneous whitespace immediately inside 

3695 # parentheses, brackets or braces. 

3696 self.lt(val) 

3697 elif val in ')]}': 

3698 # Ditto. 

3699 self.rt(val) 

3700 elif val == '=': 

3701 # Pep 8: Don't use spaces around the = sign when used to indicate 

3702 # a keyword argument or a default parameter value. 

3703 if self.paren_level: 

3704 self.clean('blank') 

3705 self.add_token('op-no-blanks', val) 

3706 else: 

3707 self.blank() 

3708 self.add_token('op', val) 

3709 self.blank() 

3710 elif val in '~+-': 

3711 self.possible_unary_op(val) 

3712 elif val == '*': 

3713 self.star_op() 

3714 elif val == '**': 

3715 self.star_star_op() 

3716 else: 

3717 # Pep 8: always surround binary operators with a single space. 

3718 # '==','+=','-=','*=','**=','/=','//=','%=','!=','<=','>=','<','>', 

3719 # '^','~','*','**','&','|','/','//', 

3720 # Pep 8: If operators with different priorities are used, 

3721 # consider adding whitespace around the operators with the lowest priority(ies). 

3722 self.blank() 

3723 self.add_token('op', val) 

3724 self.blank() 

3725 #@+node:ekr.20200107165250.24: *5* orange.do_string 

3726 def do_string(self) -> None: 

3727 """Handle a 'string' token.""" 

3728 # Careful: continued strings may contain '\r' 

3729 val = regularize_nls(self.val) 

3730 self.add_token('string', val) 

3731 self.blank() 

3732 #@+node:ekr.20200210175117.1: *5* orange.do_verbatim 

3733 beautify_pat = re.compile( 

3734 r'#\s*pragma:\s*beautify\b|#\s*@@beautify|#\s*@\+node|#\s*@[+-]others|#\s*@[+-]<<') 

3735 

3736 def do_verbatim(self) -> None: 

3737 """ 

3738 Handle one token in verbatim mode. 

3739 End verbatim mode when the appropriate comment is seen. 

3740 """ 

3741 kind = self.kind 

3742 # 

3743 # Careful: tokens may contain '\r' 

3744 val = regularize_nls(self.val) 

3745 if kind == 'comment': 

3746 if self.beautify_pat.match(val): 

3747 self.verbatim = False 

3748 val = val.rstrip() 

3749 self.add_token('comment', val) 

3750 return 

3751 if kind == 'indent': 

3752 self.level += 1 

3753 self.lws = self.level * self.tab_width * ' ' 

3754 if kind == 'dedent': 

3755 self.level -= 1 

3756 self.lws = self.level * self.tab_width * ' ' 

3757 self.add_token('verbatim', val) 

3758 #@+node:ekr.20200107165250.25: *5* orange.do_ws 

3759 def do_ws(self) -> None: 

3760 """ 

3761 Handle the "ws" pseudo-token. 

3762 

3763 Put the whitespace only if if ends with backslash-newline. 

3764 """ 

3765 val = self.val 

3766 # Handle backslash-newline. 

3767 if '\\\n' in val: 

3768 self.clean('blank') 

3769 self.add_token('op-no-blanks', val) 

3770 return 

3771 # Handle start-of-line whitespace. 

3772 prev = self.code_list[-1] 

3773 inner = self.paren_level or self.square_brackets_stack or self.curly_brackets_level 

3774 if prev.kind == 'line-indent' and inner: 

3775 # Retain the indent that won't be cleaned away. 

3776 self.clean('line-indent') 

3777 self.add_token('hard-blank', val) 

3778 #@+node:ekr.20200107165250.26: *4* orange: Output token generators 

3779 #@+node:ekr.20200118145044.1: *5* orange.add_line_end 

3780 def add_line_end(self) -> "Token": 

3781 """Add a line-end request to the code list.""" 

3782 # This may be called from do_name as well as do_newline and do_nl. 

3783 assert self.token.kind in ('newline', 'nl'), self.token.kind 

3784 self.clean('blank') # Important! 

3785 self.clean('line-indent') 

3786 t = self.add_token('line-end', '\n') 

3787 # Distinguish between kinds of 'line-end' tokens. 

3788 t.newline_kind = self.token.kind 

3789 return t 

3790 #@+node:ekr.20200107170523.1: *5* orange.add_token 

3791 def add_token(self, kind: str, value: Any) -> "Token": 

3792 """Add an output token to the code list.""" 

3793 tok = Token(kind, value) 

3794 tok.index = self.code_list_index # For debugging only. 

3795 self.code_list_index += 1 

3796 self.code_list.append(tok) 

3797 return tok 

3798 #@+node:ekr.20200107165250.27: *5* orange.blank 

3799 def blank(self) -> None: 

3800 """Add a blank request to the code list.""" 

3801 prev = self.code_list[-1] 

3802 if prev.kind not in ( 

3803 'blank', 

3804 'blank-lines', 

3805 'file-start', 

3806 'hard-blank', # Unique to orange. 

3807 'line-end', 

3808 'line-indent', 

3809 'lt', 

3810 'op-no-blanks', 

3811 'unary-op', 

3812 ): 

3813 self.add_token('blank', ' ') 

3814 #@+node:ekr.20200107165250.29: *5* orange.blank_lines (black only) 

3815 def blank_lines(self, n: int) -> None: # pragma: no cover (black) 

3816 """ 

3817 Add a request for n blank lines to the code list. 

3818 Multiple blank-lines request yield at least the maximum of all requests. 

3819 """ 

3820 self.clean_blank_lines() 

3821 prev = self.code_list[-1] 

3822 if prev.kind == 'file-start': 

3823 self.add_token('blank-lines', n) 

3824 return 

3825 for i in range(0, n + 1): 

3826 self.add_token('line-end', '\n') 

3827 # Retain the token (intention) for debugging. 

3828 self.add_token('blank-lines', n) 

3829 self.line_indent() 

3830 #@+node:ekr.20200107165250.30: *5* orange.clean 

3831 def clean(self, kind: str) -> None: 

3832 """Remove the last item of token list if it has the given kind.""" 

3833 prev = self.code_list[-1] 

3834 if prev.kind == kind: 

3835 self.code_list.pop() 

3836 #@+node:ekr.20200107165250.31: *5* orange.clean_blank_lines 

3837 def clean_blank_lines(self) -> bool: 

3838 """ 

3839 Remove all vestiges of previous blank lines. 

3840 

3841 Return True if any of the cleaned 'line-end' tokens represented "hard" newlines. 

3842 """ 

3843 cleaned_newline = False 

3844 table = ('blank-lines', 'line-end', 'line-indent') 

3845 while self.code_list[-1].kind in table: 

3846 t = self.code_list.pop() 

3847 if t.kind == 'line-end' and getattr(t, 'newline_kind', None) != 'nl': 

3848 cleaned_newline = True 

3849 return cleaned_newline 

3850 #@+node:ekr.20200107165250.32: *5* orange.colon 

3851 def colon(self, val: str) -> None: 

3852 """Handle a colon.""" 

3853 

3854 def is_expr(node: Node) -> bool: 

3855 """True if node is any expression other than += number.""" 

3856 if isinstance(node, (ast.BinOp, ast.Call, ast.IfExp)): 

3857 return True 

3858 return ( 

3859 isinstance(node, ast.UnaryOp) 

3860 and not isinstance(node.operand, ast.Num) 

3861 ) 

3862 

3863 node = self.token.node 

3864 self.clean('blank') 

3865 if not isinstance(node, ast.Slice): 

3866 self.add_token('op', val) 

3867 self.blank() 

3868 return 

3869 # A slice. 

3870 lower = getattr(node, 'lower', None) 

3871 upper = getattr(node, 'upper', None) 

3872 step = getattr(node, 'step', None) 

3873 if any(is_expr(z) for z in (lower, upper, step)): 

3874 prev = self.code_list[-1] 

3875 if prev.value not in '[:': 

3876 self.blank() 

3877 self.add_token('op', val) 

3878 self.blank() 

3879 else: 

3880 self.add_token('op-no-blanks', val) 

3881 #@+node:ekr.20200107165250.33: *5* orange.line_end 

3882 def line_end(self) -> None: 

3883 """Add a line-end request to the code list.""" 

3884 # This should be called only be do_newline and do_nl. 

3885 node, token = self.token.statement_node, self.token 

3886 assert token.kind in ('newline', 'nl'), (token.kind, g.callers()) 

3887 # Create the 'line-end' output token. 

3888 self.add_line_end() 

3889 # Attempt to split the line. 

3890 was_split = self.split_line(node, token) 

3891 # Attempt to join the line only if it has not just been split. 

3892 if not was_split and self.max_join_line_length > 0: 

3893 self.join_lines(node, token) 

3894 self.line_indent() 

3895 # Add the indentation for all lines 

3896 # until the next indent or unindent token. 

3897 #@+node:ekr.20200107165250.40: *5* orange.line_indent 

3898 def line_indent(self) -> None: 

3899 """Add a line-indent token.""" 

3900 self.clean('line-indent') 

3901 # Defensive. Should never happen. 

3902 self.add_token('line-indent', self.lws) 

3903 #@+node:ekr.20200107165250.41: *5* orange.lt & rt 

3904 #@+node:ekr.20200107165250.42: *6* orange.lt 

3905 def lt(self, val: str) -> None: 

3906 """Generate code for a left paren or curly/square bracket.""" 

3907 assert val in '([{', repr(val) 

3908 if val == '(': 

3909 self.paren_level += 1 

3910 elif val == '[': 

3911 self.square_brackets_stack.append(False) 

3912 else: 

3913 self.curly_brackets_level += 1 

3914 self.clean('blank') 

3915 prev = self.code_list[-1] 

3916 if prev.kind in ('op', 'word-op'): 

3917 self.blank() 

3918 self.add_token('lt', val) 

3919 elif prev.kind == 'word': 

3920 # Only suppress blanks before '(' or '[' for non-keyworks. 

3921 if val == '{' or prev.value in ('if', 'else', 'return', 'for'): 

3922 self.blank() 

3923 elif val == '(': 

3924 self.in_arg_list += 1 

3925 self.add_token('lt', val) 

3926 else: 

3927 self.clean('blank') 

3928 self.add_token('op-no-blanks', val) 

3929 #@+node:ekr.20200107165250.43: *6* orange.rt 

3930 def rt(self, val: str) -> None: 

3931 """Generate code for a right paren or curly/square bracket.""" 

3932 assert val in ')]}', repr(val) 

3933 if val == ')': 

3934 self.paren_level -= 1 

3935 self.in_arg_list = max(0, self.in_arg_list - 1) 

3936 elif val == ']': 

3937 self.square_brackets_stack.pop() 

3938 else: 

3939 self.curly_brackets_level -= 1 

3940 self.clean('blank') 

3941 self.add_token('rt', val) 

3942 #@+node:ekr.20200107165250.45: *5* orange.possible_unary_op & unary_op 

3943 def possible_unary_op(self, s: str) -> None: 

3944 """Add a unary or binary op to the token list.""" 

3945 node = self.token.node 

3946 self.clean('blank') 

3947 if isinstance(node, ast.UnaryOp): 

3948 self.unary_op(s) 

3949 else: 

3950 self.blank() 

3951 self.add_token('op', s) 

3952 self.blank() 

3953 

3954 def unary_op(self, s: str) -> None: 

3955 """Add an operator request to the code list.""" 

3956 assert s and isinstance(s, str), repr(s) 

3957 self.clean('blank') 

3958 prev = self.code_list[-1] 

3959 if prev.kind == 'lt': 

3960 self.add_token('unary-op', s) 

3961 else: 

3962 self.blank() 

3963 self.add_token('unary-op', s) 

3964 #@+node:ekr.20200107165250.46: *5* orange.star_op 

3965 def star_op(self) -> None: 

3966 """Put a '*' op, with special cases for *args.""" 

3967 val = '*' 

3968 node = self.token.node 

3969 self.clean('blank') 

3970 if isinstance(node, ast.arguments): 

3971 self.blank() 

3972 self.add_token('op', val) 

3973 return # #2533 

3974 if self.paren_level > 0: 

3975 prev = self.code_list[-1] 

3976 if prev.kind == 'lt' or (prev.kind, prev.value) == ('op', ','): 

3977 self.blank() 

3978 self.add_token('op', val) 

3979 return 

3980 self.blank() 

3981 self.add_token('op', val) 

3982 self.blank() 

3983 #@+node:ekr.20200107165250.47: *5* orange.star_star_op 

3984 def star_star_op(self) -> None: 

3985 """Put a ** operator, with a special case for **kwargs.""" 

3986 val = '**' 

3987 node = self.token.node 

3988 self.clean('blank') 

3989 if isinstance(node, ast.arguments): 

3990 self.blank() 

3991 self.add_token('op', val) 

3992 return # #2533 

3993 if self.paren_level > 0: 

3994 prev = self.code_list[-1] 

3995 if prev.kind == 'lt' or (prev.kind, prev.value) == ('op', ','): 

3996 self.blank() 

3997 self.add_token('op', val) 

3998 return 

3999 self.blank() 

4000 self.add_token('op', val) 

4001 self.blank() 

4002 #@+node:ekr.20200107165250.48: *5* orange.word & word_op 

4003 def word(self, s: str) -> None: 

4004 """Add a word request to the code list.""" 

4005 assert s and isinstance(s, str), repr(s) 

4006 node = self.token.node 

4007 if isinstance(node, ast.ImportFrom) and s == 'import': # #2533 

4008 self.clean('blank') 

4009 self.add_token('blank', ' ') 

4010 self.add_token('word', s) 

4011 elif self.square_brackets_stack: 

4012 # A previous 'op-no-blanks' token may cancel this blank. 

4013 self.blank() 

4014 self.add_token('word', s) 

4015 elif self.in_arg_list > 0: 

4016 self.add_token('word', s) 

4017 self.blank() 

4018 else: 

4019 self.blank() 

4020 self.add_token('word', s) 

4021 self.blank() 

4022 

4023 def word_op(self, s: str) -> None: 

4024 """Add a word-op request to the code list.""" 

4025 assert s and isinstance(s, str), repr(s) 

4026 self.blank() 

4027 self.add_token('word-op', s) 

4028 self.blank() 

4029 #@+node:ekr.20200118120049.1: *4* orange: Split/join 

4030 #@+node:ekr.20200107165250.34: *5* orange.split_line & helpers 

4031 def split_line(self, node: Node, token: "Token") -> bool: 

4032 """ 

4033 Split token's line, if possible and enabled. 

4034 

4035 Return True if the line was broken into two or more lines. 

4036 """ 

4037 assert token.kind in ('newline', 'nl'), repr(token) 

4038 # Return if splitting is disabled: 

4039 if self.max_split_line_length <= 0: # pragma: no cover (user option) 

4040 return False 

4041 # Return if the node can't be split. 

4042 if not is_long_statement(node): 

4043 return False 

4044 # Find the *output* tokens of the previous lines. 

4045 line_tokens = self.find_prev_line() 

4046 line_s = ''.join([z.to_string() for z in line_tokens]) 

4047 # Do nothing for short lines. 

4048 if len(line_s) < self.max_split_line_length: 

4049 return False 

4050 # Return if the previous line has no opening delim: (, [ or {. 

4051 if not any(z.kind == 'lt' for z in line_tokens): # pragma: no cover (defensive) 

4052 return False 

4053 prefix = self.find_line_prefix(line_tokens) 

4054 # Calculate the tail before cleaning the prefix. 

4055 tail = line_tokens[len(prefix) :] 

4056 # Cut back the token list: subtract 1 for the trailing line-end. 

4057 self.code_list = self.code_list[: len(self.code_list) - len(line_tokens) - 1] 

4058 # Append the tail, splitting it further, as needed. 

4059 self.append_tail(prefix, tail) 

4060 # Add the line-end token deleted by find_line_prefix. 

4061 self.add_token('line-end', '\n') 

4062 return True 

4063 #@+node:ekr.20200107165250.35: *6* orange.append_tail 

4064 def append_tail(self, prefix: List["Token"], tail: List["Token"]) -> None: 

4065 """Append the tail tokens, splitting the line further as necessary.""" 

4066 tail_s = ''.join([z.to_string() for z in tail]) 

4067 if len(tail_s) < self.max_split_line_length: 

4068 # Add the prefix. 

4069 self.code_list.extend(prefix) 

4070 # Start a new line and increase the indentation. 

4071 self.add_token('line-end', '\n') 

4072 self.add_token('line-indent', self.lws + ' ' * 4) 

4073 self.code_list.extend(tail) 

4074 return 

4075 # Still too long. Split the line at commas. 

4076 self.code_list.extend(prefix) 

4077 # Start a new line and increase the indentation. 

4078 self.add_token('line-end', '\n') 

4079 self.add_token('line-indent', self.lws + ' ' * 4) 

4080 open_delim = Token(kind='lt', value=prefix[-1].value) 

4081 value = open_delim.value.replace('(', ')').replace('[', ']').replace('{', '}') 

4082 close_delim = Token(kind='rt', value=value) 

4083 delim_count = 1 

4084 lws = self.lws + ' ' * 4 

4085 for i, t in enumerate(tail): 

4086 if t.kind == 'op' and t.value == ',': 

4087 if delim_count == 1: 

4088 # Start a new line. 

4089 self.add_token('op-no-blanks', ',') 

4090 self.add_token('line-end', '\n') 

4091 self.add_token('line-indent', lws) 

4092 # Kill a following blank. 

4093 if i + 1 < len(tail): 

4094 next_t = tail[i + 1] 

4095 if next_t.kind == 'blank': 

4096 next_t.kind = 'no-op' 

4097 next_t.value = '' 

4098 else: 

4099 self.code_list.append(t) 

4100 elif t.kind == close_delim.kind and t.value == close_delim.value: 

4101 # Done if the delims match. 

4102 delim_count -= 1 

4103 if delim_count == 0: 

4104 # Start a new line 

4105 self.add_token('op-no-blanks', ',') 

4106 self.add_token('line-end', '\n') 

4107 self.add_token('line-indent', self.lws) 

4108 self.code_list.extend(tail[i:]) 

4109 return 

4110 lws = lws[:-4] 

4111 self.code_list.append(t) 

4112 elif t.kind == open_delim.kind and t.value == open_delim.value: 

4113 delim_count += 1 

4114 lws = lws + ' ' * 4 

4115 self.code_list.append(t) 

4116 else: 

4117 self.code_list.append(t) 

4118 g.trace('BAD DELIMS', delim_count) # pragma: no cover 

4119 #@+node:ekr.20200107165250.36: *6* orange.find_prev_line 

4120 def find_prev_line(self) -> List["Token"]: 

4121 """Return the previous line, as a list of tokens.""" 

4122 line = [] 

4123 for t in reversed(self.code_list[:-1]): 

4124 if t.kind in ('hard-newline', 'line-end'): 

4125 break 

4126 line.append(t) 

4127 return list(reversed(line)) 

4128 #@+node:ekr.20200107165250.37: *6* orange.find_line_prefix 

4129 def find_line_prefix(self, token_list: List["Token"]) -> List["Token"]: 

4130 """ 

4131 Return all tokens up to and including the first lt token. 

4132 Also add all lt tokens directly following the first lt token. 

4133 """ 

4134 result = [] 

4135 for i, t in enumerate(token_list): 

4136 result.append(t) 

4137 if t.kind == 'lt': 

4138 break 

4139 return result 

4140 #@+node:ekr.20200107165250.39: *5* orange.join_lines 

4141 def join_lines(self, node: Node, token: "Token") -> None: 

4142 """ 

4143 Join preceding lines, if possible and enabled. 

4144 token is a line_end token. node is the corresponding ast node. 

4145 """ 

4146 if self.max_join_line_length <= 0: # pragma: no cover (user option) 

4147 return 

4148 assert token.kind in ('newline', 'nl'), repr(token) 

4149 if token.kind == 'nl': 

4150 return 

4151 # Scan backward in the *code* list, 

4152 # looking for 'line-end' tokens with tok.newline_kind == 'nl' 

4153 nls = 0 

4154 i = len(self.code_list) - 1 

4155 t = self.code_list[i] 

4156 assert t.kind == 'line-end', repr(t) 

4157 # Not all tokens have a newline_kind ivar. 

4158 assert t.newline_kind == 'newline' 

4159 i -= 1 

4160 while i >= 0: 

4161 t = self.code_list[i] 

4162 if t.kind == 'comment': 

4163 # Can't join. 

4164 return 

4165 if t.kind == 'string' and not self.allow_joined_strings: 

4166 # An EKR preference: don't join strings, no matter what black does. 

4167 # This allows "short" f-strings to be aligned. 

4168 return 

4169 if t.kind == 'line-end': 

4170 if getattr(t, 'newline_kind', None) == 'nl': 

4171 nls += 1 

4172 else: 

4173 break # pragma: no cover 

4174 i -= 1 

4175 # Retain at the file-start token. 

4176 if i <= 0: 

4177 i = 1 

4178 if nls <= 0: # pragma: no cover (rare) 

4179 return 

4180 # Retain line-end and and any following line-indent. 

4181 # Required, so that the regex below won't eat too much. 

4182 while True: 

4183 t = self.code_list[i] 

4184 if t.kind == 'line-end': 

4185 if getattr(t, 'newline_kind', None) == 'nl': # pragma: no cover (rare) 

4186 nls -= 1 

4187 i += 1 

4188 elif self.code_list[i].kind == 'line-indent': 

4189 i += 1 

4190 else: 

4191 break # pragma: no cover (defensive) 

4192 if nls <= 0: # pragma: no cover (defensive) 

4193 return 

4194 # Calculate the joined line. 

4195 tail = self.code_list[i:] 

4196 tail_s = tokens_to_string(tail) 

4197 tail_s = re.sub(r'\n\s*', ' ', tail_s) 

4198 tail_s = tail_s.replace('( ', '(').replace(' )', ')') 

4199 tail_s = tail_s.rstrip() 

4200 # Don't join the lines if they would be too long. 

4201 if len(tail_s) > self.max_join_line_length: # pragma: no cover (defensive) 

4202 return 

4203 # Cut back the code list. 

4204 self.code_list = self.code_list[:i] 

4205 # Add the new output tokens. 

4206 self.add_token('string', tail_s) 

4207 self.add_token('line-end', '\n') 

4208 #@-others 

4209#@+node:ekr.20200107170847.1: *3* class OrangeSettings 

4210class OrangeSettings: 

4211 

4212 pass 

4213#@+node:ekr.20200107170126.1: *3* class ParseState 

4214class ParseState: 

4215 """ 

4216 A class representing items in the parse state stack. 

4217 

4218 The present states: 

4219 

4220 'file-start': Ensures the stack stack is never empty. 

4221 

4222 'decorator': The last '@' was a decorator. 

4223 

4224 do_op(): push_state('decorator') 

4225 do_name(): pops the stack if state.kind == 'decorator'. 

4226 

4227 'indent': The indentation level for 'class' and 'def' names. 

4228 

4229 do_name(): push_state('indent', self.level) 

4230 do_dendent(): pops the stack once or twice if state.value == self.level. 

4231 

4232 """ 

4233 

4234 def __init__(self, kind: str, value: str) -> None: 

4235 self.kind = kind 

4236 self.value = value 

4237 

4238 def __repr__(self) -> str: 

4239 return f"State: {self.kind} {self.value!r}" # pragma: no cover 

4240 

4241 __str__ = __repr__ 

4242#@+node:ekr.20191231084514.1: *3* class ReassignTokens 

4243class ReassignTokens: 

4244 """A class that reassigns tokens to more appropriate ast nodes.""" 

4245 #@+others 

4246 #@+node:ekr.20191231084640.1: *4* reassign.reassign 

4247 def reassign(self, filename: str, tokens: List["Token"], tree: Node) -> None: 

4248 """The main entry point.""" 

4249 self.filename = filename 

4250 self.tokens = tokens 

4251 # For now, just handle Call nodes. 

4252 for node in ast.walk(tree): 

4253 if isinstance(node, ast.Call): 

4254 self.visit_call(node) 

4255 #@+node:ekr.20191231084853.1: *4* reassign.visit_call 

4256 def visit_call(self, node: Node) -> None: 

4257 """ReassignTokens.visit_call""" 

4258 tokens = tokens_for_node(self.filename, node, self.tokens) 

4259 node0, node9 = tokens[0].node, tokens[-1].node 

4260 nca = nearest_common_ancestor(node0, node9) 

4261 if not nca: 

4262 return 

4263 # Associate () with the call node. 

4264 i = tokens[-1].index 

4265 j = find_paren_token(i + 1, self.tokens) 

4266 if j is None: 

4267 return # pragma: no cover 

4268 k = find_paren_token(j + 1, self.tokens) 

4269 if k is None: 

4270 return # pragma: no cover 

4271 self.tokens[j].node = nca 

4272 self.tokens[k].node = nca 

4273 add_token_to_token_list(self.tokens[j], nca) 

4274 add_token_to_token_list(self.tokens[k], nca) 

4275 #@-others 

4276#@+node:ekr.20191110080535.1: *3* class Token 

4277class Token: 

4278 """ 

4279 A class representing a 5-tuple, plus additional data. 

4280 """ 

4281 

4282 def __init__(self, kind: str, value: str): 

4283 

4284 self.kind = kind 

4285 self.value = value 

4286 # 

4287 # Injected by Tokenizer.add_token. 

4288 self.five_tuple = None 

4289 self.index = 0 

4290 # The entire line containing the token. 

4291 # Same as five_tuple.line. 

4292 self.line = '' 

4293 # The line number, for errors and dumps. 

4294 # Same as five_tuple.start[0] 

4295 self.line_number = 0 

4296 # 

4297 # Injected by Tokenizer.add_token. 

4298 self.level = 0 

4299 self.node: Optional[Node] = None 

4300 

4301 def __repr__(self) -> str: # pragma: no cover 

4302 nl_kind = getattr(self, 'newline_kind', '') 

4303 s = f"{self.kind:}.{self.index:<3}" 

4304 return f"{s:>18}:{nl_kind:7} {self.show_val(80)}" 

4305 

4306 def __str__(self) -> str: # pragma: no cover 

4307 nl_kind = getattr(self, 'newline_kind', '') 

4308 return f"{self.kind}.{self.index:<3}{nl_kind:8} {self.show_val(80)}" 

4309 

4310 def to_string(self) -> str: 

4311 """Return the contribution of the token to the source file.""" 

4312 return self.value if isinstance(self.value, str) else '' 

4313 #@+others 

4314 #@+node:ekr.20191231114927.1: *4* token.brief_dump 

4315 def brief_dump(self) -> str: # pragma: no cover 

4316 """Dump a token.""" 

4317 return ( 

4318 f"{self.index:>3} line: {self.line_number:<2} " 

4319 f"{self.kind:>11} {self.show_val(100)}") 

4320 #@+node:ekr.20200223022950.11: *4* token.dump 

4321 def dump(self) -> str: # pragma: no cover 

4322 """Dump a token and related links.""" 

4323 # Let block. 

4324 node_id = self.node.node_index if self.node else '' 

4325 node_cn = self.node.__class__.__name__ if self.node else '' 

4326 return ( 

4327 f"{self.line_number:4} " 

4328 f"{node_id:5} {node_cn:16} " 

4329 f"{self.index:>5} {self.kind:>11} " 

4330 f"{self.show_val(100)}") 

4331 #@+node:ekr.20200121081151.1: *4* token.dump_header 

4332 def dump_header(self) -> None: # pragma: no cover 

4333 """Print the header for token.dump""" 

4334 print( 

4335 f"\n" 

4336 f" node {'':10} token token\n" 

4337 f"line index class {'':10} index kind value\n" 

4338 f"==== ===== ===== {'':10} ===== ==== =====\n") 

4339 #@+node:ekr.20191116154328.1: *4* token.error_dump 

4340 def error_dump(self) -> str: # pragma: no cover 

4341 """Dump a token or result node for error message.""" 

4342 if self.node: 

4343 node_id = obj_id(self.node) 

4344 node_s = f"{node_id} {self.node.__class__.__name__}" 

4345 else: 

4346 node_s = "None" 

4347 return ( 

4348 f"index: {self.index:<3} {self.kind:>12} {self.show_val(20):<20} " 

4349 f"{node_s}") 

4350 #@+node:ekr.20191113095507.1: *4* token.show_val 

4351 def show_val(self, truncate_n: int) -> str: # pragma: no cover 

4352 """Return the token.value field.""" 

4353 if self.kind in ('ws', 'indent'): 

4354 val = str(len(self.value)) 

4355 elif self.kind == 'string': 

4356 # Important: don't add a repr for 'string' tokens. 

4357 # repr just adds another layer of confusion. 

4358 val = g.truncate(self.value, truncate_n) 

4359 else: 

4360 val = g.truncate(repr(self.value), truncate_n) 

4361 return val 

4362 #@-others 

4363#@+node:ekr.20191110165235.1: *3* class Tokenizer 

4364class Tokenizer: 

4365 

4366 """Create a list of Tokens from contents.""" 

4367 

4368 results: List[Token] = [] 

4369 

4370 #@+others 

4371 #@+node:ekr.20191110165235.2: *4* tokenizer.add_token 

4372 token_index = 0 

4373 prev_line_token = None 

4374 

4375 def add_token(self, kind: str, five_tuple: Any, line: str, s_row: int, value: str) -> None: 

4376 """ 

4377 Add a token to the results list. 

4378 

4379 Subclasses could override this method to filter out specific tokens. 

4380 """ 

4381 tok = Token(kind, value) 

4382 tok.five_tuple = five_tuple 

4383 tok.index = self.token_index 

4384 # Bump the token index. 

4385 self.token_index += 1 

4386 tok.line = line 

4387 tok.line_number = s_row 

4388 self.results.append(tok) 

4389 #@+node:ekr.20191110170551.1: *4* tokenizer.check_results 

4390 def check_results(self, contents: str) -> None: 

4391 

4392 # Split the results into lines. 

4393 result = ''.join([z.to_string() for z in self.results]) 

4394 result_lines = g.splitLines(result) 

4395 # Check. 

4396 ok = result == contents and result_lines == self.lines 

4397 assert ok, ( 

4398 f"\n" 

4399 f" result: {result!r}\n" 

4400 f" contents: {contents!r}\n" 

4401 f"result_lines: {result_lines}\n" 

4402 f" lines: {self.lines}" 

4403 ) 

4404 #@+node:ekr.20191110165235.3: *4* tokenizer.create_input_tokens 

4405 def create_input_tokens(self, contents: str, tokens: Generator) -> List["Token"]: 

4406 """ 

4407 Generate a list of Token's from tokens, a list of 5-tuples. 

4408 """ 

4409 # Create the physical lines. 

4410 self.lines = contents.splitlines(True) 

4411 # Create the list of character offsets of the start of each physical line. 

4412 last_offset, self.offsets = 0, [0] 

4413 for line in self.lines: 

4414 last_offset += len(line) 

4415 self.offsets.append(last_offset) 

4416 # Handle each token, appending tokens and between-token whitespace to results. 

4417 self.prev_offset, self.results = -1, [] 

4418 for token in tokens: 

4419 self.do_token(contents, token) 

4420 # Print results when tracing. 

4421 self.check_results(contents) 

4422 # Return results, as a list. 

4423 return self.results 

4424 #@+node:ekr.20191110165235.4: *4* tokenizer.do_token (the gem) 

4425 header_has_been_shown = False 

4426 

4427 def do_token(self, contents: str, five_tuple: Any) -> None: 

4428 """ 

4429 Handle the given token, optionally including between-token whitespace. 

4430 

4431 This is part of the "gem". 

4432 

4433 Links: 

4434 

4435 - 11/13/19: ENB: A much better untokenizer 

4436 https://groups.google.com/forum/#!msg/leo-editor/DpZ2cMS03WE/VPqtB9lTEAAJ 

4437 

4438 - Untokenize does not round-trip ws before bs-nl 

4439 https://bugs.python.org/issue38663 

4440 """ 

4441 import token as token_module 

4442 # Unpack.. 

4443 tok_type, val, start, end, line = five_tuple 

4444 s_row, s_col = start # row/col offsets of start of token. 

4445 e_row, e_col = end # row/col offsets of end of token. 

4446 kind = token_module.tok_name[tok_type].lower() 

4447 # Calculate the token's start/end offsets: character offsets into contents. 

4448 s_offset = self.offsets[max(0, s_row - 1)] + s_col 

4449 e_offset = self.offsets[max(0, e_row - 1)] + e_col 

4450 # tok_s is corresponding string in the line. 

4451 tok_s = contents[s_offset:e_offset] 

4452 # Add any preceding between-token whitespace. 

4453 ws = contents[self.prev_offset:s_offset] 

4454 if ws: 

4455 # No need for a hook. 

4456 self.add_token('ws', five_tuple, line, s_row, ws) 

4457 # Always add token, even if it contributes no text! 

4458 self.add_token(kind, five_tuple, line, s_row, tok_s) 

4459 # Update the ending offset. 

4460 self.prev_offset = e_offset 

4461 #@-others 

4462#@+node:ekr.20191113063144.1: *3* class TokenOrderGenerator 

4463class TokenOrderGenerator: 

4464 """ 

4465 A class that traverses ast (parse) trees in token order. 

4466 

4467 Overview: https://github.com/leo-editor/leo-editor/issues/1440#issue-522090981 

4468 

4469 Theory of operation: 

4470 - https://github.com/leo-editor/leo-editor/issues/1440#issuecomment-573661883 

4471 - http://leoeditor.com/appendices.html#tokenorder-classes-theory-of-operation 

4472 

4473 How to: http://leoeditor.com/appendices.html#tokenorder-class-how-to 

4474 

4475 Project history: https://github.com/leo-editor/leo-editor/issues/1440#issuecomment-574145510 

4476 """ 

4477 

4478 begin_end_stack: List[str] = [] 

4479 n_nodes = 0 # The number of nodes that have been visited. 

4480 node_index = 0 # The index into the node_stack. 

4481 node_stack: List[ast.AST] = [] # The stack of parent nodes. 

4482 

4483 #@+others 

4484 #@+node:ekr.20200103174914.1: *4* tog: Init... 

4485 #@+node:ekr.20191228184647.1: *5* tog.balance_tokens 

4486 def balance_tokens(self, tokens: List["Token"]) -> int: 

4487 """ 

4488 TOG.balance_tokens. 

4489 

4490 Insert two-way links between matching paren tokens. 

4491 """ 

4492 count, stack = 0, [] 

4493 for token in tokens: 

4494 if token.kind == 'op': 

4495 if token.value == '(': 

4496 count += 1 

4497 stack.append(token.index) 

4498 if token.value == ')': 

4499 if stack: 

4500 index = stack.pop() 

4501 tokens[index].matching_paren = token.index 

4502 tokens[token.index].matching_paren = index 

4503 else: # pragma: no cover 

4504 g.trace(f"unmatched ')' at index {token.index}") 

4505 if stack: # pragma: no cover 

4506 g.trace("unmatched '(' at {','.join(stack)}") 

4507 return count 

4508 #@+node:ekr.20191113063144.4: *5* tog.create_links 

4509 def create_links(self, tokens: List["Token"], tree: Node, file_name: str='') -> List: 

4510 """ 

4511 A generator creates two-way links between the given tokens and ast-tree. 

4512 

4513 Callers should call this generator with list(tog.create_links(...)) 

4514 

4515 The sync_tokens method creates the links and verifies that the resulting 

4516 tree traversal generates exactly the given tokens in exact order. 

4517 

4518 tokens: the list of Token instances for the input. 

4519 Created by make_tokens(). 

4520 tree: the ast tree for the input. 

4521 Created by parse_ast(). 

4522 """ 

4523 # Init all ivars. 

4524 self.file_name = file_name # For tests. 

4525 self.level = 0 # Python indentation level. 

4526 self.node = None # The node being visited. 

4527 self.tokens = tokens # The immutable list of input tokens. 

4528 self.tree = tree # The tree of ast.AST nodes. 

4529 # Traverse the tree. 

4530 self.visit(tree) 

4531 # Ensure that all tokens are patched. 

4532 self.node = tree 

4533 self.token('endmarker', '') 

4534 # Return [] for compatibility with legacy code: list(tog.create_links). 

4535 return [] 

4536 #@+node:ekr.20191229071733.1: *5* tog.init_from_file 

4537 def init_from_file(self, filename: str) -> Tuple[str, str, List["Token"], Node]: # pragma: no cover 

4538 """ 

4539 Create the tokens and ast tree for the given file. 

4540 Create links between tokens and the parse tree. 

4541 Return (contents, encoding, tokens, tree). 

4542 """ 

4543 self.level = 0 

4544 self.filename = filename 

4545 encoding, contents = read_file_with_encoding(filename) 

4546 if not contents: 

4547 return None, None, None, None 

4548 self.tokens = tokens = make_tokens(contents) 

4549 self.tree = tree = parse_ast(contents) 

4550 self.create_links(tokens, tree) 

4551 return contents, encoding, tokens, tree 

4552 #@+node:ekr.20191229071746.1: *5* tog.init_from_string 

4553 def init_from_string(self, contents: str, filename: str) -> Tuple[List["Token"], Node]: # pragma: no cover 

4554 """ 

4555 Tokenize, parse and create links in the contents string. 

4556 

4557 Return (tokens, tree). 

4558 """ 

4559 self.filename = filename 

4560 self.level = 0 

4561 self.tokens = tokens = make_tokens(contents) 

4562 self.tree = tree = parse_ast(contents) 

4563 self.create_links(tokens, tree) 

4564 return tokens, tree 

4565 #@+node:ekr.20220402052020.1: *4* tog: Syncronizers... 

4566 # The synchronizer sync tokens to nodes. 

4567 #@+node:ekr.20200110162044.1: *5* tog.find_next_significant_token 

4568 def find_next_significant_token(self) -> Optional["Token"]: 

4569 """ 

4570 Scan from *after* self.tokens[px] looking for the next significant 

4571 token. 

4572 

4573 Return the token, or None. Never change self.px. 

4574 """ 

4575 px = self.px + 1 

4576 while px < len(self.tokens): 

4577 token = self.tokens[px] 

4578 px += 1 

4579 if is_significant_token(token): 

4580 return token 

4581 # This will never happen, because endtoken is significant. 

4582 return None # pragma: no cover 

4583 #@+node:ekr.20191125120814.1: *5* tog.set_links 

4584 last_statement_node = None 

4585 

4586 def set_links(self, node: Node, token: "Token") -> None: 

4587 """Make two-way links between token and the given node.""" 

4588 # Don't bother assigning comment, comma, parens, ws and endtoken tokens. 

4589 if token.kind == 'comment': 

4590 # Append the comment to node.comment_list. 

4591 comment_list: List["Token"] = getattr(node, 'comment_list', []) 

4592 node.comment_list = comment_list + [token] 

4593 return 

4594 if token.kind in ('endmarker', 'ws'): 

4595 return 

4596 if token.kind == 'op' and token.value in ',()': 

4597 return 

4598 # *Always* remember the last statement. 

4599 statement = find_statement_node(node) 

4600 if statement: 

4601 self.last_statement_node = statement 

4602 assert not isinstance(self.last_statement_node, ast.Module) 

4603 if token.node is not None: # pragma: no cover 

4604 line_s = f"line {token.line_number}:" 

4605 raise AssignLinksError( 

4606 f" file: {self.filename}\n" 

4607 f"{line_s:>12} {token.line.strip()}\n" 

4608 f"token index: {self.px}\n" 

4609 f"token.node is not None\n" 

4610 f" token.node: {token.node.__class__.__name__}\n" 

4611 f" callers: {g.callers()}") 

4612 # Assign newlines to the previous statement node, if any. 

4613 if token.kind in ('newline', 'nl'): 

4614 # Set an *auxilliary* link for the split/join logic. 

4615 # Do *not* set token.node! 

4616 token.statement_node = self.last_statement_node 

4617 return 

4618 if is_significant_token(token): 

4619 # Link the token to the ast node. 

4620 token.node = node 

4621 # Add the token to node's token_list. 

4622 add_token_to_token_list(token, node) 

4623 #@+node:ekr.20191124083124.1: *5* tog.sync_name (aka name) 

4624 def sync_name(self, val: str) -> None: 

4625 aList = val.split('.') 

4626 if len(aList) == 1: 

4627 self.sync_token('name', val) 

4628 else: 

4629 for i, part in enumerate(aList): 

4630 self.sync_token('name', part) 

4631 if i < len(aList) - 1: 

4632 self.sync_op('.') 

4633 

4634 name = sync_name # for readability. 

4635 #@+node:ekr.20220402052102.1: *5* tog.sync_op (aka op) 

4636 def sync_op(self, val: str) -> None: 

4637 """ 

4638 Sync to the given operator. 

4639 

4640 val may be '(' or ')' *only* if the parens *will* actually exist in the 

4641 token list. 

4642 """ 

4643 self.sync_token('op', val) 

4644 

4645 op = sync_op # For readability. 

4646 #@+node:ekr.20191113063144.7: *5* tog.sync_token (aka token) 

4647 px = -1 # Index of the previously synced token. 

4648 

4649 def sync_token(self, kind: str, val: str) -> None: 

4650 """ 

4651 Sync to a token whose kind & value are given. The token need not be 

4652 significant, but it must be guaranteed to exist in the token list. 

4653 

4654 The checks in this method constitute a strong, ever-present, unit test. 

4655 

4656 Scan the tokens *after* px, looking for a token T matching (kind, val). 

4657 raise AssignLinksError if a significant token is found that doesn't match T. 

4658 Otherwise: 

4659 - Create two-way links between all assignable tokens between px and T. 

4660 - Create two-way links between T and self.node. 

4661 - Advance by updating self.px to point to T. 

4662 """ 

4663 node, tokens = self.node, self.tokens 

4664 assert isinstance(node, ast.AST), repr(node) 

4665 # g.trace( 

4666 # f"px: {self.px:2} " 

4667 # f"node: {node.__class__.__name__:<10} " 

4668 # f"kind: {kind:>10}: val: {val!r}") 

4669 # 

4670 # Step one: Look for token T. 

4671 old_px = px = self.px + 1 

4672 while px < len(self.tokens): 

4673 token = tokens[px] 

4674 if (kind, val) == (token.kind, token.value): 

4675 break # Success. 

4676 if kind == token.kind == 'number': 

4677 val = token.value 

4678 break # Benign: use the token's value, a string, instead of a number. 

4679 if is_significant_token(token): # pragma: no cover 

4680 line_s = f"line {token.line_number}:" 

4681 val = str(val) # for g.truncate. 

4682 raise AssignLinksError( 

4683 f" file: {self.filename}\n" 

4684 f"{line_s:>12} {token.line.strip()}\n" 

4685 f"Looking for: {kind}.{g.truncate(val, 40)!r}\n" 

4686 f" found: {token.kind}.{token.value!r}\n" 

4687 f"token.index: {token.index}\n") 

4688 # Skip the insignificant token. 

4689 px += 1 

4690 else: # pragma: no cover 

4691 val = str(val) # for g.truncate. 

4692 raise AssignLinksError( 

4693 f" file: {self.filename}\n" 

4694 f"Looking for: {kind}.{g.truncate(val, 40)}\n" 

4695 f" found: end of token list") 

4696 # 

4697 # Step two: Assign *secondary* links only for newline tokens. 

4698 # Ignore all other non-significant tokens. 

4699 while old_px < px: 

4700 token = tokens[old_px] 

4701 old_px += 1 

4702 if token.kind in ('comment', 'newline', 'nl'): 

4703 self.set_links(node, token) 

4704 # 

4705 # Step three: Set links in the found token. 

4706 token = tokens[px] 

4707 self.set_links(node, token) 

4708 # 

4709 # Step four: Advance. 

4710 self.px = px 

4711 

4712 token = sync_token # For readability. 

4713 #@+node:ekr.20191223052749.1: *4* tog: Traversal... 

4714 #@+node:ekr.20191113063144.3: *5* tog.enter_node 

4715 def enter_node(self, node: Node) -> None: 

4716 """Enter a node.""" 

4717 # Update the stats. 

4718 self.n_nodes += 1 

4719 # Do this first, *before* updating self.node. 

4720 node.parent = self.node 

4721 if self.node: 

4722 children: List[Node] = getattr(self.node, 'children', []) 

4723 children.append(node) 

4724 self.node.children = children 

4725 # Inject the node_index field. 

4726 assert not hasattr(node, 'node_index'), g.callers() 

4727 node.node_index = self.node_index 

4728 self.node_index += 1 

4729 # begin_visitor and end_visitor must be paired. 

4730 self.begin_end_stack.append(node.__class__.__name__) 

4731 # Push the previous node. 

4732 self.node_stack.append(self.node) 

4733 # Update self.node *last*. 

4734 self.node = node 

4735 #@+node:ekr.20200104032811.1: *5* tog.leave_node 

4736 def leave_node(self, node: Node) -> None: 

4737 """Leave a visitor.""" 

4738 # begin_visitor and end_visitor must be paired. 

4739 entry_name = self.begin_end_stack.pop() 

4740 assert entry_name == node.__class__.__name__, f"{entry_name!r} {node.__class__.__name__}" 

4741 assert self.node == node, (repr(self.node), repr(node)) 

4742 # Restore self.node. 

4743 self.node = self.node_stack.pop() 

4744 #@+node:ekr.20191113081443.1: *5* tog.visit 

4745 def visit(self, node: Node) -> None: 

4746 """Given an ast node, return a *generator* from its visitor.""" 

4747 # This saves a lot of tests. 

4748 if node is None: 

4749 return 

4750 if 0: # pragma: no cover 

4751 # Keep this trace! 

4752 cn = node.__class__.__name__ if node else ' ' 

4753 caller1, caller2 = g.callers(2).split(',') 

4754 g.trace(f"{caller1:>15} {caller2:<14} {cn}") 

4755 # More general, more convenient. 

4756 if isinstance(node, (list, tuple)): 

4757 for z in node or []: 

4758 if isinstance(z, ast.AST): 

4759 self.visit(z) 

4760 else: # pragma: no cover 

4761 # Some fields may contain ints or strings. 

4762 assert isinstance(z, (int, str)), z.__class__.__name__ 

4763 return 

4764 # We *do* want to crash if the visitor doesn't exist. 

4765 method = getattr(self, 'do_' + node.__class__.__name__) 

4766 # Don't even *think* about removing the parent/child links. 

4767 # The nearest_common_ancestor function depends upon them. 

4768 self.enter_node(node) 

4769 method(node) 

4770 self.leave_node(node) 

4771 #@+node:ekr.20191113063144.13: *4* tog: Visitors... 

4772 #@+node:ekr.20191113063144.32: *5* tog.keyword: not called! 

4773 # keyword arguments supplied to call (NULL identifier for **kwargs) 

4774 

4775 # keyword = (identifier? arg, expr value) 

4776 

4777 def do_keyword(self, node: Node) -> None: # pragma: no cover 

4778 """A keyword arg in an ast.Call.""" 

4779 # This should never be called. 

4780 # tog.hande_call_arguments calls self.visit(kwarg_arg.value) instead. 

4781 filename = getattr(self, 'filename', '<no file>') 

4782 raise AssignLinksError( 

4783 f"file: {filename}\n" 

4784 f"do_keyword should never be called\n" 

4785 f"{g.callers(8)}") 

4786 #@+node:ekr.20191113063144.14: *5* tog: Contexts 

4787 #@+node:ekr.20191113063144.28: *6* tog.arg 

4788 # arg = (identifier arg, expr? annotation) 

4789 

4790 def do_arg(self, node: Node) -> None: 

4791 """This is one argument of a list of ast.Function or ast.Lambda arguments.""" 

4792 self.name(node.arg) 

4793 annotation = getattr(node, 'annotation', None) 

4794 if annotation is not None: 

4795 self.op(':') 

4796 self.visit(node.annotation) 

4797 #@+node:ekr.20191113063144.27: *6* tog.arguments 

4798 # arguments = ( 

4799 # arg* posonlyargs, arg* args, arg? vararg, arg* kwonlyargs, 

4800 # expr* kw_defaults, arg? kwarg, expr* defaults 

4801 # ) 

4802 

4803 def do_arguments(self, node: Node) -> None: 

4804 """Arguments to ast.Function or ast.Lambda, **not** ast.Call.""" 

4805 # 

4806 # No need to generate commas anywhere below. 

4807 # 

4808 # Let block. Some fields may not exist pre Python 3.8. 

4809 n_plain = len(node.args) - len(node.defaults) 

4810 posonlyargs = getattr(node, 'posonlyargs', []) 

4811 vararg = getattr(node, 'vararg', None) 

4812 kwonlyargs = getattr(node, 'kwonlyargs', []) 

4813 kw_defaults = getattr(node, 'kw_defaults', []) 

4814 kwarg = getattr(node, 'kwarg', None) 

4815 # 1. Sync the position-only args. 

4816 if posonlyargs: 

4817 for n, z in enumerate(posonlyargs): 

4818 # g.trace('pos-only', ast.dump(z)) 

4819 self.visit(z) 

4820 self.op('/') 

4821 # 2. Sync all args. 

4822 for i, z in enumerate(node.args): 

4823 self.visit(z) 

4824 if i >= n_plain: 

4825 self.op('=') 

4826 self.visit(node.defaults[i - n_plain]) 

4827 # 3. Sync the vararg. 

4828 if vararg: 

4829 self.op('*') 

4830 self.visit(vararg) 

4831 # 4. Sync the keyword-only args. 

4832 if kwonlyargs: 

4833 if not vararg: 

4834 self.op('*') 

4835 for n, z in enumerate(kwonlyargs): 

4836 self.visit(z) 

4837 val = kw_defaults[n] 

4838 if val is not None: 

4839 self.op('=') 

4840 self.visit(val) 

4841 # 5. Sync the kwarg. 

4842 if kwarg: 

4843 self.op('**') 

4844 self.visit(kwarg) 

4845 #@+node:ekr.20191113063144.15: *6* tog.AsyncFunctionDef 

4846 # AsyncFunctionDef(identifier name, arguments args, stmt* body, expr* decorator_list, 

4847 # expr? returns) 

4848 

4849 def do_AsyncFunctionDef(self, node: Node) -> None: 

4850 

4851 if node.decorator_list: 

4852 for z in node.decorator_list: 

4853 # '@%s\n' 

4854 self.op('@') 

4855 self.visit(z) 

4856 # 'asynch def (%s): -> %s\n' 

4857 # 'asynch def %s(%s):\n' 

4858 async_token_type = 'async' if has_async_tokens else 'name' 

4859 self.token(async_token_type, 'async') 

4860 self.name('def') 

4861 self.name(node.name) # A string 

4862 self.op('(') 

4863 self.visit(node.args) 

4864 self.op(')') 

4865 returns = getattr(node, 'returns', None) 

4866 if returns is not None: 

4867 self.op('->') 

4868 self.visit(node.returns) 

4869 self.op(':') 

4870 self.level += 1 

4871 self.visit(node.body) 

4872 self.level -= 1 

4873 #@+node:ekr.20191113063144.16: *6* tog.ClassDef 

4874 def do_ClassDef(self, node: Node) -> None: 

4875 

4876 for z in node.decorator_list or []: 

4877 # @{z}\n 

4878 self.op('@') 

4879 self.visit(z) 

4880 # class name(bases):\n 

4881 self.name('class') 

4882 self.name(node.name) # A string. 

4883 if node.bases: 

4884 self.op('(') 

4885 self.visit(node.bases) 

4886 self.op(')') 

4887 self.op(':') 

4888 # Body... 

4889 self.level += 1 

4890 self.visit(node.body) 

4891 self.level -= 1 

4892 #@+node:ekr.20191113063144.17: *6* tog.FunctionDef 

4893 # FunctionDef( 

4894 # identifier name, arguments args, 

4895 # stmt* body, 

4896 # expr* decorator_list, 

4897 # expr? returns, 

4898 # string? type_comment) 

4899 

4900 def do_FunctionDef(self, node: Node) -> None: 

4901 

4902 # Guards... 

4903 returns = getattr(node, 'returns', None) 

4904 # Decorators... 

4905 # @{z}\n 

4906 for z in node.decorator_list or []: 

4907 self.op('@') 

4908 self.visit(z) 

4909 # Signature... 

4910 # def name(args): -> returns\n 

4911 # def name(args):\n 

4912 self.name('def') 

4913 self.name(node.name) # A string. 

4914 self.op('(') 

4915 self.visit(node.args) 

4916 self.op(')') 

4917 if returns is not None: 

4918 self.op('->') 

4919 self.visit(node.returns) 

4920 self.op(':') 

4921 # Body... 

4922 self.level += 1 

4923 self.visit(node.body) 

4924 self.level -= 1 

4925 #@+node:ekr.20191113063144.18: *6* tog.Interactive 

4926 def do_Interactive(self, node: Node) -> None: # pragma: no cover 

4927 

4928 self.visit(node.body) 

4929 #@+node:ekr.20191113063144.20: *6* tog.Lambda 

4930 def do_Lambda(self, node: Node) -> None: 

4931 

4932 self.name('lambda') 

4933 self.visit(node.args) 

4934 self.op(':') 

4935 self.visit(node.body) 

4936 #@+node:ekr.20191113063144.19: *6* tog.Module 

4937 def do_Module(self, node: Node) -> None: 

4938 

4939 # Encoding is a non-syncing statement. 

4940 self.visit(node.body) 

4941 #@+node:ekr.20191113063144.21: *5* tog: Expressions 

4942 #@+node:ekr.20191113063144.22: *6* tog.Expr 

4943 def do_Expr(self, node: Node) -> None: 

4944 """An outer expression.""" 

4945 # No need to put parentheses. 

4946 self.visit(node.value) 

4947 #@+node:ekr.20191113063144.23: *6* tog.Expression 

4948 def do_Expression(self, node: Node) -> None: # pragma: no cover 

4949 """An inner expression.""" 

4950 # No need to put parentheses. 

4951 self.visit(node.body) 

4952 #@+node:ekr.20191113063144.24: *6* tog.GeneratorExp 

4953 def do_GeneratorExp(self, node: Node) -> None: 

4954 

4955 # '<gen %s for %s>' % (elt, ','.join(gens)) 

4956 # No need to put parentheses or commas. 

4957 self.visit(node.elt) 

4958 self.visit(node.generators) 

4959 #@+node:ekr.20210321171703.1: *6* tog.NamedExpr 

4960 # NamedExpr(expr target, expr value) 

4961 

4962 def do_NamedExpr(self, node: Node) -> None: # Python 3.8+ 

4963 

4964 self.visit(node.target) 

4965 self.op(':=') 

4966 self.visit(node.value) 

4967 #@+node:ekr.20191113063144.26: *5* tog: Operands 

4968 #@+node:ekr.20191113063144.29: *6* tog.Attribute 

4969 # Attribute(expr value, identifier attr, expr_context ctx) 

4970 

4971 def do_Attribute(self, node: Node) -> None: 

4972 

4973 self.visit(node.value) 

4974 self.op('.') 

4975 self.name(node.attr) # A string. 

4976 #@+node:ekr.20191113063144.30: *6* tog.Bytes 

4977 def do_Bytes(self, node: Node) -> None: 

4978 

4979 """ 

4980 It's invalid to mix bytes and non-bytes literals, so just 

4981 advancing to the next 'string' token suffices. 

4982 """ 

4983 token = self.find_next_significant_token() 

4984 self.token('string', token.value) 

4985 #@+node:ekr.20191113063144.33: *6* tog.comprehension 

4986 # comprehension = (expr target, expr iter, expr* ifs, int is_async) 

4987 

4988 def do_comprehension(self, node: Node) -> None: 

4989 

4990 # No need to put parentheses. 

4991 self.name('for') # #1858. 

4992 self.visit(node.target) # A name 

4993 self.name('in') 

4994 self.visit(node.iter) 

4995 for z in node.ifs or []: 

4996 self.name('if') 

4997 self.visit(z) 

4998 #@+node:ekr.20191113063144.34: *6* tog.Constant 

4999 def do_Constant(self, node: Node) -> None: # pragma: no cover 

5000 """ 

5001 https://greentreesnakes.readthedocs.io/en/latest/nodes.html 

5002 

5003 A constant. The value attribute holds the Python object it represents. 

5004 This can be simple types such as a number, string or None, but also 

5005 immutable container types (tuples and frozensets) if all of their 

5006 elements are constant. 

5007 """ 

5008 # Support Python 3.8. 

5009 if node.value is None or isinstance(node.value, bool): 

5010 # Weird: return a name! 

5011 self.token('name', repr(node.value)) 

5012 elif node.value == Ellipsis: 

5013 self.op('...') 

5014 elif isinstance(node.value, str): 

5015 self.do_Str(node) 

5016 elif isinstance(node.value, (int, float)): 

5017 self.token('number', repr(node.value)) 

5018 elif isinstance(node.value, bytes): 

5019 self.do_Bytes(node) 

5020 elif isinstance(node.value, tuple): 

5021 self.do_Tuple(node) 

5022 elif isinstance(node.value, frozenset): 

5023 self.do_Set(node) 

5024 else: 

5025 # Unknown type. 

5026 g.trace('----- Oops -----', repr(node.value), g.callers()) 

5027 #@+node:ekr.20191113063144.35: *6* tog.Dict 

5028 # Dict(expr* keys, expr* values) 

5029 

5030 def do_Dict(self, node: Node) -> None: 

5031 

5032 assert len(node.keys) == len(node.values) 

5033 self.op('{') 

5034 # No need to put commas. 

5035 for i, key in enumerate(node.keys): 

5036 key, value = node.keys[i], node.values[i] 

5037 self.visit(key) # a Str node. 

5038 self.op(':') 

5039 if value is not None: 

5040 self.visit(value) 

5041 self.op('}') 

5042 #@+node:ekr.20191113063144.36: *6* tog.DictComp 

5043 # DictComp(expr key, expr value, comprehension* generators) 

5044 

5045 # d2 = {val: key for key, val in d} 

5046 

5047 def do_DictComp(self, node: Node) -> None: 

5048 

5049 self.token('op', '{') 

5050 self.visit(node.key) 

5051 self.op(':') 

5052 self.visit(node.value) 

5053 for z in node.generators or []: 

5054 self.visit(z) 

5055 self.token('op', '}') 

5056 #@+node:ekr.20191113063144.37: *6* tog.Ellipsis 

5057 def do_Ellipsis(self, node: Node) -> None: # pragma: no cover (Does not exist for python 3.8+) 

5058 

5059 self.op('...') 

5060 #@+node:ekr.20191113063144.38: *6* tog.ExtSlice 

5061 # https://docs.python.org/3/reference/expressions.html#slicings 

5062 

5063 # ExtSlice(slice* dims) 

5064 

5065 def do_ExtSlice(self, node: Node) -> None: # pragma: no cover (deprecated) 

5066 

5067 # ','.join(node.dims) 

5068 for i, z in enumerate(node.dims): 

5069 self.visit(z) 

5070 if i < len(node.dims) - 1: 

5071 self.op(',') 

5072 #@+node:ekr.20191113063144.40: *6* tog.Index 

5073 def do_Index(self, node: Node) -> None: # pragma: no cover (deprecated) 

5074 

5075 self.visit(node.value) 

5076 #@+node:ekr.20191113063144.39: *6* tog.FormattedValue: not called! 

5077 # FormattedValue(expr value, int? conversion, expr? format_spec) 

5078 

5079 def do_FormattedValue(self, node: Node) -> None: # pragma: no cover 

5080 """ 

5081 This node represents the *components* of a *single* f-string. 

5082 

5083 Happily, JoinedStr nodes *also* represent *all* f-strings, 

5084 so the TOG should *never visit this node! 

5085 """ 

5086 filename = getattr(self, 'filename', '<no file>') 

5087 raise AssignLinksError( 

5088 f"file: {filename}\n" 

5089 f"do_FormattedValue should never be called") 

5090 

5091 # This code has no chance of being useful... 

5092 

5093 # conv = node.conversion 

5094 # spec = node.format_spec 

5095 # self.visit(node.value) 

5096 # if conv is not None: 

5097 # self.token('number', conv) 

5098 # if spec is not None: 

5099 # self.visit(node.format_spec) 

5100 #@+node:ekr.20191113063144.41: *6* tog.JoinedStr & helpers 

5101 # JoinedStr(expr* values) 

5102 

5103 def do_JoinedStr(self, node: Node) -> None: 

5104 """ 

5105 JoinedStr nodes represent at least one f-string and all other strings 

5106 concatentated to it. 

5107 

5108 Analyzing JoinedStr.values would be extremely tricky, for reasons that 

5109 need not be explained here. 

5110 

5111 Instead, we get the tokens *from the token list itself*! 

5112 """ 

5113 for z in self.get_concatenated_string_tokens(): 

5114 self.token(z.kind, z.value) 

5115 #@+node:ekr.20191113063144.42: *6* tog.List 

5116 def do_List(self, node: Node) -> None: 

5117 

5118 # No need to put commas. 

5119 self.op('[') 

5120 self.visit(node.elts) 

5121 self.op(']') 

5122 #@+node:ekr.20191113063144.43: *6* tog.ListComp 

5123 # ListComp(expr elt, comprehension* generators) 

5124 

5125 def do_ListComp(self, node: Node) -> None: 

5126 

5127 self.op('[') 

5128 self.visit(node.elt) 

5129 for z in node.generators: 

5130 self.visit(z) 

5131 self.op(']') 

5132 #@+node:ekr.20191113063144.44: *6* tog.Name & NameConstant 

5133 def do_Name(self, node: Node) -> None: 

5134 

5135 self.name(node.id) 

5136 

5137 def do_NameConstant(self, node: Node) -> None: # pragma: no cover (Does not exist in Python 3.8+) 

5138 

5139 self.name(repr(node.value)) 

5140 

5141 #@+node:ekr.20191113063144.45: *6* tog.Num 

5142 def do_Num(self, node: Node) -> None: # pragma: no cover (Does not exist in Python 3.8+) 

5143 

5144 self.token('number', node.n) 

5145 #@+node:ekr.20191113063144.47: *6* tog.Set 

5146 # Set(expr* elts) 

5147 

5148 def do_Set(self, node: Node) -> None: 

5149 

5150 self.op('{') 

5151 self.visit(node.elts) 

5152 self.op('}') 

5153 #@+node:ekr.20191113063144.48: *6* tog.SetComp 

5154 # SetComp(expr elt, comprehension* generators) 

5155 

5156 def do_SetComp(self, node: Node) -> None: 

5157 

5158 self.op('{') 

5159 self.visit(node.elt) 

5160 for z in node.generators or []: 

5161 self.visit(z) 

5162 self.op('}') 

5163 #@+node:ekr.20191113063144.49: *6* tog.Slice 

5164 # slice = Slice(expr? lower, expr? upper, expr? step) 

5165 

5166 def do_Slice(self, node: Node) -> None: 

5167 

5168 lower = getattr(node, 'lower', None) 

5169 upper = getattr(node, 'upper', None) 

5170 step = getattr(node, 'step', None) 

5171 if lower is not None: 

5172 self.visit(lower) 

5173 # Always put the colon between upper and lower. 

5174 self.op(':') 

5175 if upper is not None: 

5176 self.visit(upper) 

5177 # Put the second colon if it exists in the token list. 

5178 if step is None: 

5179 token = self.find_next_significant_token() 

5180 if token and token.value == ':': 

5181 self.op(':') 

5182 else: 

5183 self.op(':') 

5184 self.visit(step) 

5185 #@+node:ekr.20191113063144.50: *6* tog.Str & helper 

5186 def do_Str(self, node: Node) -> None: 

5187 """This node represents a string constant.""" 

5188 # This loop is necessary to handle string concatenation. 

5189 for z in self.get_concatenated_string_tokens(): 

5190 self.token(z.kind, z.value) 

5191 #@+node:ekr.20200111083914.1: *7* tog.get_concatenated_tokens 

5192 def get_concatenated_string_tokens(self) -> List["Token"]: 

5193 """ 

5194 Return the next 'string' token and all 'string' tokens concatenated to 

5195 it. *Never* update self.px here. 

5196 """ 

5197 trace = False 

5198 tag = 'tog.get_concatenated_string_tokens' 

5199 i = self.px 

5200 # First, find the next significant token. It should be a string. 

5201 i, token = i + 1, None 

5202 while i < len(self.tokens): 

5203 token = self.tokens[i] 

5204 i += 1 

5205 if token.kind == 'string': 

5206 # Rescan the string. 

5207 i -= 1 

5208 break 

5209 # An error. 

5210 if is_significant_token(token): # pragma: no cover 

5211 break 

5212 # Raise an error if we didn't find the expected 'string' token. 

5213 if not token or token.kind != 'string': # pragma: no cover 

5214 if not token: 

5215 token = self.tokens[-1] 

5216 filename = getattr(self, 'filename', '<no filename>') 

5217 raise AssignLinksError( 

5218 f"\n" 

5219 f"{tag}...\n" 

5220 f"file: {filename}\n" 

5221 f"line: {token.line_number}\n" 

5222 f" i: {i}\n" 

5223 f"expected 'string' token, got {token!s}") 

5224 # Accumulate string tokens. 

5225 assert self.tokens[i].kind == 'string' 

5226 results = [] 

5227 while i < len(self.tokens): 

5228 token = self.tokens[i] 

5229 i += 1 

5230 if token.kind == 'string': 

5231 results.append(token) 

5232 elif token.kind == 'op' or is_significant_token(token): 

5233 # Any significant token *or* any op will halt string concatenation. 

5234 break 

5235 # 'ws', 'nl', 'newline', 'comment', 'indent', 'dedent', etc. 

5236 # The (significant) 'endmarker' token ensures we will have result. 

5237 assert results 

5238 if trace: # pragma: no cover 

5239 g.printObj(results, tag=f"{tag}: Results") 

5240 return results 

5241 #@+node:ekr.20191113063144.51: *6* tog.Subscript 

5242 # Subscript(expr value, slice slice, expr_context ctx) 

5243 

5244 def do_Subscript(self, node: Node) -> None: 

5245 

5246 self.visit(node.value) 

5247 self.op('[') 

5248 self.visit(node.slice) 

5249 self.op(']') 

5250 #@+node:ekr.20191113063144.52: *6* tog.Tuple 

5251 # Tuple(expr* elts, expr_context ctx) 

5252 

5253 def do_Tuple(self, node: Node) -> None: 

5254 

5255 # Do not call op for parens or commas here. 

5256 # They do not necessarily exist in the token list! 

5257 self.visit(node.elts) 

5258 #@+node:ekr.20191113063144.53: *5* tog: Operators 

5259 #@+node:ekr.20191113063144.55: *6* tog.BinOp 

5260 def do_BinOp(self, node: Node) -> None: 

5261 

5262 op_name_ = op_name(node.op) 

5263 self.visit(node.left) 

5264 self.op(op_name_) 

5265 self.visit(node.right) 

5266 #@+node:ekr.20191113063144.56: *6* tog.BoolOp 

5267 # BoolOp(boolop op, expr* values) 

5268 

5269 def do_BoolOp(self, node: Node) -> None: 

5270 

5271 # op.join(node.values) 

5272 op_name_ = op_name(node.op) 

5273 for i, z in enumerate(node.values): 

5274 self.visit(z) 

5275 if i < len(node.values) - 1: 

5276 self.name(op_name_) 

5277 #@+node:ekr.20191113063144.57: *6* tog.Compare 

5278 # Compare(expr left, cmpop* ops, expr* comparators) 

5279 

5280 def do_Compare(self, node: Node) -> None: 

5281 

5282 assert len(node.ops) == len(node.comparators) 

5283 self.visit(node.left) 

5284 for i, z in enumerate(node.ops): 

5285 op_name_ = op_name(node.ops[i]) 

5286 if op_name_ in ('not in', 'is not'): 

5287 for z in op_name_.split(' '): 

5288 self.name(z) 

5289 elif op_name_.isalpha(): 

5290 self.name(op_name_) 

5291 else: 

5292 self.op(op_name_) 

5293 self.visit(node.comparators[i]) 

5294 #@+node:ekr.20191113063144.58: *6* tog.UnaryOp 

5295 def do_UnaryOp(self, node: Node) -> None: 

5296 

5297 op_name_ = op_name(node.op) 

5298 if op_name_.isalpha(): 

5299 self.name(op_name_) 

5300 else: 

5301 self.op(op_name_) 

5302 self.visit(node.operand) 

5303 #@+node:ekr.20191113063144.59: *6* tog.IfExp (ternary operator) 

5304 # IfExp(expr test, expr body, expr orelse) 

5305 

5306 def do_IfExp(self, node: Node) -> None: 

5307 

5308 #'%s if %s else %s' 

5309 self.visit(node.body) 

5310 self.name('if') 

5311 self.visit(node.test) 

5312 self.name('else') 

5313 self.visit(node.orelse) 

5314 #@+node:ekr.20191113063144.60: *5* tog: Statements 

5315 #@+node:ekr.20191113063144.83: *6* tog.Starred 

5316 # Starred(expr value, expr_context ctx) 

5317 

5318 def do_Starred(self, node: Node) -> None: 

5319 """A starred argument to an ast.Call""" 

5320 self.op('*') 

5321 self.visit(node.value) 

5322 #@+node:ekr.20191113063144.61: *6* tog.AnnAssign 

5323 # AnnAssign(expr target, expr annotation, expr? value, int simple) 

5324 

5325 def do_AnnAssign(self, node: Node) -> None: 

5326 

5327 # {node.target}:{node.annotation}={node.value}\n' 

5328 self.visit(node.target) 

5329 self.op(':') 

5330 self.visit(node.annotation) 

5331 if node.value is not None: # #1851 

5332 self.op('=') 

5333 self.visit(node.value) 

5334 #@+node:ekr.20191113063144.62: *6* tog.Assert 

5335 # Assert(expr test, expr? msg) 

5336 

5337 def do_Assert(self, node: Node) -> None: 

5338 

5339 # Guards... 

5340 msg = getattr(node, 'msg', None) 

5341 # No need to put parentheses or commas. 

5342 self.name('assert') 

5343 self.visit(node.test) 

5344 if msg is not None: 

5345 self.visit(node.msg) 

5346 #@+node:ekr.20191113063144.63: *6* tog.Assign 

5347 def do_Assign(self, node: Node) -> None: 

5348 

5349 for z in node.targets: 

5350 self.visit(z) 

5351 self.op('=') 

5352 self.visit(node.value) 

5353 #@+node:ekr.20191113063144.64: *6* tog.AsyncFor 

5354 def do_AsyncFor(self, node: Node) -> None: 

5355 

5356 # The def line... 

5357 # Py 3.8 changes the kind of token. 

5358 async_token_type = 'async' if has_async_tokens else 'name' 

5359 self.token(async_token_type, 'async') 

5360 self.name('for') 

5361 self.visit(node.target) 

5362 self.name('in') 

5363 self.visit(node.iter) 

5364 self.op(':') 

5365 # Body... 

5366 self.level += 1 

5367 self.visit(node.body) 

5368 # Else clause... 

5369 if node.orelse: 

5370 self.name('else') 

5371 self.op(':') 

5372 self.visit(node.orelse) 

5373 self.level -= 1 

5374 #@+node:ekr.20191113063144.65: *6* tog.AsyncWith 

5375 def do_AsyncWith(self, node: Node) -> None: 

5376 

5377 async_token_type = 'async' if has_async_tokens else 'name' 

5378 self.token(async_token_type, 'async') 

5379 self.do_With(node) 

5380 #@+node:ekr.20191113063144.66: *6* tog.AugAssign 

5381 # AugAssign(expr target, operator op, expr value) 

5382 

5383 def do_AugAssign(self, node: Node) -> None: 

5384 

5385 # %s%s=%s\n' 

5386 op_name_ = op_name(node.op) 

5387 self.visit(node.target) 

5388 self.op(op_name_ + '=') 

5389 self.visit(node.value) 

5390 #@+node:ekr.20191113063144.67: *6* tog.Await 

5391 # Await(expr value) 

5392 

5393 def do_Await(self, node: Node) -> None: 

5394 

5395 #'await %s\n' 

5396 async_token_type = 'await' if has_async_tokens else 'name' 

5397 self.token(async_token_type, 'await') 

5398 self.visit(node.value) 

5399 #@+node:ekr.20191113063144.68: *6* tog.Break 

5400 def do_Break(self, node: Node) -> None: 

5401 

5402 self.name('break') 

5403 #@+node:ekr.20191113063144.31: *6* tog.Call & helpers 

5404 # Call(expr func, expr* args, keyword* keywords) 

5405 

5406 # Python 3 ast.Call nodes do not have 'starargs' or 'kwargs' fields. 

5407 

5408 def do_Call(self, node: Node) -> None: 

5409 

5410 # The calls to op(')') and op('(') do nothing by default. 

5411 # Subclasses might handle them in an overridden tog.set_links. 

5412 self.visit(node.func) 

5413 self.op('(') 

5414 # No need to generate any commas. 

5415 self.handle_call_arguments(node) 

5416 self.op(')') 

5417 #@+node:ekr.20191204114930.1: *7* tog.arg_helper 

5418 def arg_helper(self, node: Union[Node, str]) -> None: 

5419 """ 

5420 Yield the node, with a special case for strings. 

5421 """ 

5422 if isinstance(node, str): 

5423 self.token('name', node) 

5424 else: 

5425 self.visit(node) 

5426 #@+node:ekr.20191204105506.1: *7* tog.handle_call_arguments 

5427 def handle_call_arguments(self, node: Node) -> None: 

5428 """ 

5429 Generate arguments in the correct order. 

5430 

5431 Call(expr func, expr* args, keyword* keywords) 

5432 

5433 https://docs.python.org/3/reference/expressions.html#calls 

5434 

5435 Warning: This code will fail on Python 3.8 only for calls 

5436 containing kwargs in unexpected places. 

5437 """ 

5438 # *args: in node.args[]: Starred(value=Name(id='args')) 

5439 # *[a, 3]: in node.args[]: Starred(value=List(elts=[Name(id='a'), Num(n=3)]) 

5440 # **kwargs: in node.keywords[]: keyword(arg=None, value=Name(id='kwargs')) 

5441 # 

5442 # Scan args for *name or *List 

5443 args = node.args or [] 

5444 keywords = node.keywords or [] 

5445 

5446 def get_pos(obj: Any) -> Tuple[int, int, Any]: 

5447 line1 = getattr(obj, 'lineno', None) 

5448 col1 = getattr(obj, 'col_offset', None) 

5449 return line1, col1, obj 

5450 

5451 def sort_key(aTuple: Tuple) -> int: 

5452 line, col, obj = aTuple 

5453 return line * 1000 + col 

5454 

5455 if 0: # pragma: no cover 

5456 g.printObj([ast.dump(z) for z in args], tag='args') 

5457 g.printObj([ast.dump(z) for z in keywords], tag='keywords') 

5458 

5459 if py_version >= (3, 9): 

5460 places = [get_pos(z) for z in args + keywords] 

5461 places.sort(key=sort_key) 

5462 ordered_args = [z[2] for z in places] 

5463 for z in ordered_args: 

5464 if isinstance(z, ast.Starred): 

5465 self.op('*') 

5466 self.visit(z.value) 

5467 elif isinstance(z, ast.keyword): 

5468 if getattr(z, 'arg', None) is None: 

5469 self.op('**') 

5470 self.arg_helper(z.value) 

5471 else: 

5472 self.arg_helper(z.arg) 

5473 self.op('=') 

5474 self.arg_helper(z.value) 

5475 else: 

5476 self.arg_helper(z) 

5477 else: # pragma: no cover 

5478 # 

5479 # Legacy code: May fail for Python 3.8 

5480 # 

5481 # Scan args for *arg and *[...] 

5482 kwarg_arg = star_arg = None 

5483 for z in args: 

5484 if isinstance(z, ast.Starred): 

5485 if isinstance(z.value, ast.Name): # *Name. 

5486 star_arg = z 

5487 args.remove(z) 

5488 break 

5489 elif isinstance(z.value, (ast.List, ast.Tuple)): # *[...] 

5490 # star_list = z 

5491 break 

5492 raise AttributeError(f"Invalid * expression: {ast.dump(z)}") # pragma: no cover 

5493 # Scan keywords for **name. 

5494 for z in keywords: 

5495 if hasattr(z, 'arg') and z.arg is None: 

5496 kwarg_arg = z 

5497 keywords.remove(z) 

5498 break 

5499 # Sync the plain arguments. 

5500 for z in args: 

5501 self.arg_helper(z) 

5502 # Sync the keyword args. 

5503 for z in keywords: 

5504 self.arg_helper(z.arg) 

5505 self.op('=') 

5506 self.arg_helper(z.value) 

5507 # Sync the * arg. 

5508 if star_arg: 

5509 self.arg_helper(star_arg) 

5510 # Sync the ** kwarg. 

5511 if kwarg_arg: 

5512 self.op('**') 

5513 self.visit(kwarg_arg.value) 

5514 #@+node:ekr.20191113063144.69: *6* tog.Continue 

5515 def do_Continue(self, node: Node) -> None: 

5516 

5517 self.name('continue') 

5518 #@+node:ekr.20191113063144.70: *6* tog.Delete 

5519 def do_Delete(self, node: Node) -> None: 

5520 

5521 # No need to put commas. 

5522 self.name('del') 

5523 self.visit(node.targets) 

5524 #@+node:ekr.20191113063144.71: *6* tog.ExceptHandler 

5525 def do_ExceptHandler(self, node: Node) -> None: 

5526 

5527 # Except line... 

5528 self.name('except') 

5529 if getattr(node, 'type', None): 

5530 self.visit(node.type) 

5531 if getattr(node, 'name', None): 

5532 self.name('as') 

5533 self.name(node.name) 

5534 self.op(':') 

5535 # Body... 

5536 self.level += 1 

5537 self.visit(node.body) 

5538 self.level -= 1 

5539 #@+node:ekr.20191113063144.73: *6* tog.For 

5540 def do_For(self, node: Node) -> None: 

5541 

5542 # The def line... 

5543 self.name('for') 

5544 self.visit(node.target) 

5545 self.name('in') 

5546 self.visit(node.iter) 

5547 self.op(':') 

5548 # Body... 

5549 self.level += 1 

5550 self.visit(node.body) 

5551 # Else clause... 

5552 if node.orelse: 

5553 self.name('else') 

5554 self.op(':') 

5555 self.visit(node.orelse) 

5556 self.level -= 1 

5557 #@+node:ekr.20191113063144.74: *6* tog.Global 

5558 # Global(identifier* names) 

5559 

5560 def do_Global(self, node: Node) -> None: 

5561 

5562 self.name('global') 

5563 for z in node.names: 

5564 self.name(z) 

5565 #@+node:ekr.20191113063144.75: *6* tog.If & helpers 

5566 # If(expr test, stmt* body, stmt* orelse) 

5567 

5568 def do_If(self, node: Node) -> None: 

5569 #@+<< do_If docstring >> 

5570 #@+node:ekr.20191122222412.1: *7* << do_If docstring >> 

5571 """ 

5572 The parse trees for the following are identical! 

5573 

5574 if 1: if 1: 

5575 pass pass 

5576 else: elif 2: 

5577 if 2: pass 

5578 pass 

5579 

5580 So there is *no* way for the 'if' visitor to disambiguate the above two 

5581 cases from the parse tree alone. 

5582 

5583 Instead, we scan the tokens list for the next 'if', 'else' or 'elif' token. 

5584 """ 

5585 #@-<< do_If docstring >> 

5586 # Use the next significant token to distinguish between 'if' and 'elif'. 

5587 token = self.find_next_significant_token() 

5588 self.name(token.value) 

5589 self.visit(node.test) 

5590 self.op(':') 

5591 # 

5592 # Body... 

5593 self.level += 1 

5594 self.visit(node.body) 

5595 self.level -= 1 

5596 # 

5597 # Else and elif clauses... 

5598 if node.orelse: 

5599 self.level += 1 

5600 token = self.find_next_significant_token() 

5601 if token.value == 'else': 

5602 self.name('else') 

5603 self.op(':') 

5604 self.visit(node.orelse) 

5605 else: 

5606 self.visit(node.orelse) 

5607 self.level -= 1 

5608 #@+node:ekr.20191113063144.76: *6* tog.Import & helper 

5609 def do_Import(self, node: Node) -> None: 

5610 

5611 self.name('import') 

5612 for alias in node.names: 

5613 self.name(alias.name) 

5614 if alias.asname: 

5615 self.name('as') 

5616 self.name(alias.asname) 

5617 #@+node:ekr.20191113063144.77: *6* tog.ImportFrom 

5618 # ImportFrom(identifier? module, alias* names, int? level) 

5619 

5620 def do_ImportFrom(self, node: Node) -> None: 

5621 

5622 self.name('from') 

5623 for i in range(node.level): 

5624 self.op('.') 

5625 if node.module: 

5626 self.name(node.module) 

5627 self.name('import') 

5628 # No need to put commas. 

5629 for alias in node.names: 

5630 if alias.name == '*': # #1851. 

5631 self.op('*') 

5632 else: 

5633 self.name(alias.name) 

5634 if alias.asname: 

5635 self.name('as') 

5636 self.name(alias.asname) 

5637 #@+node:ekr.20220401034726.1: *6* tog.Match* (Python 3.10+) 

5638 # Match(expr subject, match_case* cases) 

5639 

5640 # match_case = (pattern pattern, expr? guard, stmt* body) 

5641 

5642 # Full syntax diagram: # https://peps.python.org/pep-0634/#appendix-a 

5643 

5644 def do_Match(self, node: Node) -> None: 

5645 

5646 cases = getattr(node, 'cases', []) 

5647 self.name('match') 

5648 self.visit(node.subject) 

5649 self.op(':') 

5650 for case in cases: 

5651 self.visit(case) 

5652 #@+node:ekr.20220401034726.2: *7* tog.match_case 

5653 # match_case = (pattern pattern, expr? guard, stmt* body) 

5654 

5655 def do_match_case(self, node: Node) -> None: 

5656 

5657 guard = getattr(node, 'guard', None) 

5658 body = getattr(node, 'body', []) 

5659 self.name('case') 

5660 self.visit(node.pattern) 

5661 if guard: 

5662 self.name('if') 

5663 self.visit(guard) 

5664 self.op(':') 

5665 for statement in body: 

5666 self.visit(statement) 

5667 #@+node:ekr.20220401034726.3: *7* tog.MatchAs 

5668 # MatchAs(pattern? pattern, identifier? name) 

5669 

5670 def do_MatchAs(self, node: Node) -> None: 

5671 pattern = getattr(node, 'pattern', None) 

5672 name = getattr(node, 'name', None) 

5673 if pattern and name: 

5674 self.visit(pattern) 

5675 self.name('as') 

5676 self.name(name) 

5677 elif pattern: 

5678 self.visit(pattern) # pragma: no cover 

5679 else: 

5680 self.name(name or '_') 

5681 #@+node:ekr.20220401034726.4: *7* tog.MatchClass 

5682 # MatchClass(expr cls, pattern* patterns, identifier* kwd_attrs, pattern* kwd_patterns) 

5683 

5684 def do_MatchClass(self, node: Node) -> None: 

5685 

5686 cls = node.cls 

5687 patterns = getattr(node, 'patterns', []) 

5688 kwd_attrs = getattr(node, 'kwd_attrs', []) 

5689 kwd_patterns = getattr(node, 'kwd_patterns', []) 

5690 self.visit(node.cls) 

5691 self.op('(') 

5692 for pattern in patterns: 

5693 self.visit(pattern) 

5694 for i, kwd_attr in enumerate(kwd_attrs): 

5695 self.name(kwd_attr) # a String. 

5696 self.op('=') 

5697 self.visit(kwd_patterns[i]) 

5698 self.op(')') 

5699 #@+node:ekr.20220401034726.5: *7* tog.MatchMapping 

5700 # MatchMapping(expr* keys, pattern* patterns, identifier? rest) 

5701 

5702 def do_MatchMapping(self, node: Node) -> None: 

5703 keys = getattr(node, 'keys', []) 

5704 patterns = getattr(node, 'patterns', []) 

5705 rest = getattr(node, 'rest', None) 

5706 self.op('{') 

5707 for i, key in enumerate(keys): 

5708 self.visit(key) 

5709 self.op(':') 

5710 self.visit(patterns[i]) 

5711 if rest: 

5712 self.op('**') 

5713 self.name(rest) # A string. 

5714 self.op('}') 

5715 #@+node:ekr.20220401034726.6: *7* tog.MatchOr 

5716 # MatchOr(pattern* patterns) 

5717 

5718 def do_MatchOr(self, node: Node) -> None: 

5719 patterns = getattr(node, 'patterns', []) 

5720 for i, pattern in enumerate(patterns): 

5721 if i > 0: 

5722 self.op('|') 

5723 self.visit(pattern) 

5724 #@+node:ekr.20220401034726.7: *7* tog.MatchSequence 

5725 # MatchSequence(pattern* patterns) 

5726 

5727 def do_MatchSequence(self, node: Node) -> None: 

5728 patterns = getattr(node, 'patterns', []) 

5729 # Scan for the next '(' or '[' token, skipping the 'case' token. 

5730 token = None 

5731 for token in self.tokens[self.px + 1 :]: 

5732 if token.kind == 'op' and token.value in '([': 

5733 break 

5734 if is_significant_token(token): 

5735 # An implicit tuple: there is no '(' or '[' token. 

5736 token = None 

5737 break 

5738 else: 

5739 raise AssignLinksError('Ill-formed tuple') # pragma: no cover 

5740 if token: 

5741 self.op(token.value) 

5742 for i, pattern in enumerate(patterns): 

5743 self.visit(pattern) 

5744 if token: 

5745 self.op(']' if token.value == '[' else ')') 

5746 #@+node:ekr.20220401034726.8: *7* tog.MatchSingleton 

5747 # MatchSingleton(constant value) 

5748 

5749 def do_MatchSingleton(self, node: Node) -> None: 

5750 """Match True, False or None.""" 

5751 # g.trace(repr(node.value)) 

5752 self.token('name', repr(node.value)) 

5753 #@+node:ekr.20220401034726.9: *7* tog.MatchStar 

5754 # MatchStar(identifier? name) 

5755 

5756 def do_MatchStar(self, node: Node) -> None: 

5757 name = getattr(node, 'name', None) 

5758 self.op('*') 

5759 if name: 

5760 self.name(name) 

5761 #@+node:ekr.20220401034726.10: *7* tog.MatchValue 

5762 # MatchValue(expr value) 

5763 

5764 def do_MatchValue(self, node: Node) -> None: 

5765 

5766 self.visit(node.value) 

5767 #@+node:ekr.20191113063144.78: *6* tog.Nonlocal 

5768 # Nonlocal(identifier* names) 

5769 

5770 def do_Nonlocal(self, node: Node) -> None: 

5771 

5772 # nonlocal %s\n' % ','.join(node.names)) 

5773 # No need to put commas. 

5774 self.name('nonlocal') 

5775 for z in node.names: 

5776 self.name(z) 

5777 #@+node:ekr.20191113063144.79: *6* tog.Pass 

5778 def do_Pass(self, node: Node) -> None: 

5779 

5780 self.name('pass') 

5781 #@+node:ekr.20191113063144.81: *6* tog.Raise 

5782 # Raise(expr? exc, expr? cause) 

5783 

5784 def do_Raise(self, node: Node) -> None: 

5785 

5786 # No need to put commas. 

5787 self.name('raise') 

5788 exc = getattr(node, 'exc', None) 

5789 cause = getattr(node, 'cause', None) 

5790 tback = getattr(node, 'tback', None) 

5791 self.visit(exc) 

5792 if cause: 

5793 self.name('from') # #2446. 

5794 self.visit(cause) 

5795 self.visit(tback) 

5796 #@+node:ekr.20191113063144.82: *6* tog.Return 

5797 def do_Return(self, node: Node) -> None: 

5798 

5799 self.name('return') 

5800 self.visit(node.value) 

5801 #@+node:ekr.20191113063144.85: *6* tog.Try 

5802 # Try(stmt* body, excepthandler* handlers, stmt* orelse, stmt* finalbody) 

5803 

5804 def do_Try(self, node: Node) -> None: 

5805 

5806 # Try line... 

5807 self.name('try') 

5808 self.op(':') 

5809 # Body... 

5810 self.level += 1 

5811 self.visit(node.body) 

5812 self.visit(node.handlers) 

5813 # Else... 

5814 if node.orelse: 

5815 self.name('else') 

5816 self.op(':') 

5817 self.visit(node.orelse) 

5818 # Finally... 

5819 if node.finalbody: 

5820 self.name('finally') 

5821 self.op(':') 

5822 self.visit(node.finalbody) 

5823 self.level -= 1 

5824 #@+node:ekr.20191113063144.88: *6* tog.While 

5825 def do_While(self, node: Node) -> None: 

5826 

5827 # While line... 

5828 # while %s:\n' 

5829 self.name('while') 

5830 self.visit(node.test) 

5831 self.op(':') 

5832 # Body... 

5833 self.level += 1 

5834 self.visit(node.body) 

5835 # Else clause... 

5836 if node.orelse: 

5837 self.name('else') 

5838 self.op(':') 

5839 self.visit(node.orelse) 

5840 self.level -= 1 

5841 #@+node:ekr.20191113063144.89: *6* tog.With 

5842 # With(withitem* items, stmt* body) 

5843 

5844 # withitem = (expr context_expr, expr? optional_vars) 

5845 

5846 def do_With(self, node: Node) -> None: 

5847 

5848 expr: Optional[ast.AST] = getattr(node, 'context_expression', None) 

5849 items: List[ast.AST] = getattr(node, 'items', []) 

5850 self.name('with') 

5851 self.visit(expr) 

5852 # No need to put commas. 

5853 for item in items: 

5854 self.visit(item.context_expr) 

5855 optional_vars = getattr(item, 'optional_vars', None) 

5856 if optional_vars is not None: 

5857 self.name('as') 

5858 self.visit(item.optional_vars) 

5859 # End the line. 

5860 self.op(':') 

5861 # Body... 

5862 self.level += 1 

5863 self.visit(node.body) 

5864 self.level -= 1 

5865 #@+node:ekr.20191113063144.90: *6* tog.Yield 

5866 def do_Yield(self, node: Node) -> None: 

5867 

5868 self.name('yield') 

5869 if hasattr(node, 'value'): 

5870 self.visit(node.value) 

5871 #@+node:ekr.20191113063144.91: *6* tog.YieldFrom 

5872 # YieldFrom(expr value) 

5873 

5874 def do_YieldFrom(self, node: Node) -> None: 

5875 

5876 self.name('yield') 

5877 self.name('from') 

5878 self.visit(node.value) 

5879 #@-others 

5880#@+node:ekr.20191226195813.1: *3* class TokenOrderTraverser 

5881class TokenOrderTraverser: 

5882 """ 

5883 Traverse an ast tree using the parent/child links created by the 

5884 TokenOrderGenerator class. 

5885  

5886 **Important**: 

5887  

5888 This class is a curio. It is no longer used in this file! 

5889 The Fstringify and ReassignTokens classes now use ast.walk. 

5890 """ 

5891 #@+others 

5892 #@+node:ekr.20191226200154.1: *4* TOT.traverse 

5893 def traverse(self, tree: Node) -> int: 

5894 """ 

5895 Call visit, in token order, for all nodes in tree. 

5896 

5897 Recursion is not allowed. 

5898 

5899 The code follows p.moveToThreadNext exactly. 

5900 """ 

5901 

5902 def has_next(i: int, node: Node, stack: List[int]) -> bool: 

5903 """Return True if stack[i] is a valid child of node.parent.""" 

5904 # g.trace(node.__class__.__name__, stack) 

5905 parent = node.parent 

5906 return bool(parent and parent.children and i < len(parent.children)) 

5907 

5908 # Update stats 

5909 

5910 self.last_node_index = -1 # For visit 

5911 # The stack contains child indices. 

5912 node, stack = tree, [0] 

5913 seen = set() 

5914 while node and stack: 

5915 # g.trace( 

5916 # f"{node.node_index:>3} " 

5917 # f"{node.__class__.__name__:<12} {stack}") 

5918 # Visit the node. 

5919 assert node.node_index not in seen, node.node_index 

5920 seen.add(node.node_index) 

5921 self.visit(node) 

5922 # if p.v.children: p.moveToFirstChild() 

5923 children: List[ast.AST] = getattr(node, 'children', []) 

5924 if children: 

5925 # Move to the first child. 

5926 stack.append(0) 

5927 node = children[0] 

5928 # g.trace(' child:', node.__class__.__name__, stack) 

5929 continue 

5930 # elif p.hasNext(): p.moveToNext() 

5931 stack[-1] += 1 

5932 i = stack[-1] 

5933 if has_next(i, node, stack): 

5934 node = node.parent.children[i] 

5935 continue 

5936 # else... 

5937 # p.moveToParent() 

5938 node = node.parent 

5939 stack.pop() 

5940 # while p: 

5941 while node and stack: 

5942 # if p.hasNext(): 

5943 stack[-1] += 1 

5944 i = stack[-1] 

5945 if has_next(i, node, stack): 

5946 # Move to the next sibling. 

5947 node = node.parent.children[i] 

5948 break # Found. 

5949 # p.moveToParent() 

5950 node = node.parent 

5951 stack.pop() 

5952 # not found. 

5953 else: 

5954 break # pragma: no cover 

5955 return self.last_node_index 

5956 #@+node:ekr.20191227160547.1: *4* TOT.visit 

5957 def visit(self, node: Node) -> None: 

5958 

5959 self.last_node_index += 1 

5960 assert self.last_node_index == node.node_index, ( 

5961 self.last_node_index, node.node_index) 

5962 #@-others 

5963#@-others 

5964g = LeoGlobals() 

5965if __name__ == '__main__': 

5966 main() # pragma: no cover 

5967#@@language python 

5968#@@tabwidth -4 

5969#@@pagewidth 70 

5970#@-leo