0 Fixing Issue #293 (Python 3.8 grammar problem)

R. Bernstein edited this page 2019-10-29 14:03:39 -04:00

See Issue #293.

The reporter kindly gave some small source-code file to show the problem. Here it is with some comments added

for i in range(2):
    pass

First step then is to byte-compile the source code.

After that, we us "assembly" -a, and grammar -g switches to see what's going on. I have annotated the relevant parts below:

./bin/uncompyle6 -ag /tmp/bug-3.8.pyc
# uncompyle6 version 3.5.0
# Python bytecode 3.8 (3413)
# Decompiled from: Python 3.8.0 (default, Oct 28 2019, 12:12:11)
# [GCC 9.2.1 20191008]
# Embedded file name: exec

L.   1       0  LOAD_CONST               0
             2  LOAD_CONST               None
             4  IMPORT_NAME              sys
             6  STORE_NAME               sys

L.   2       8  LOAD_CONST               0
            10  LOAD_CONST               None
            12  IMPORT_NAME              uncompyle6
            14  STORE_NAME               uncompyle6

L.   4      16  LOAD_NAME                range
            18  LOAD_CONST               2
            20  CALL_FUNCTION         1  ''
            22  GET_ITER
            24  FOR_ITER             30  'to 30'
            26  STORE_NAME               i

L.   5      28  CONTINUE             24  'to 24'
                ^^^^^^^^  (note ingester turned this from JUMP_BACK into CONTINUE)
            30  LOAD_CONST               None
            32  RETURN_VALUE


# Size of source mod 2**32: 58 bytes
               expr ::= LOAD_CONST (1)
               ret_expr ::= expr (1)
               assert_expr ::= expr (1)
...
L.  4:  16-22  get_iter ::= expr GET_ITER (12)
L.  4:  16     expr ::= get_iter (12)
L.  4:  16     ret_expr ::= expr (12)
L.  4:  16     assert_expr ::= expr (12)
        22-24  for_iter ::= GET_ITER FOR_ITER (13)
        26     store ::= STORE_NAME (14)
               continue ::= CONTINUE (15)
	                    ^^^^^^^^^^^^
               continues ::= continue (15)
               stmt ::= continue (15)
               l_stmts ::= continues (15)
               _stmts ::= stmt (15)
               for_block ::= l_stmts (15)

# Note: Once "continue" has been merged into stmt/for_block it is lost for matching

               l_stmts_opt ::= l_stmts (15)
               l_stmts ::= _stmts (15)

-- Stacks of completed symbols:
START ::= |- stmts .
_come_froms ::= \e__come_froms . COME_FROM
...
for38 ::= expr . for_iter store for_block JUMP_BACK
for38 ::= expr . for_iter store for_block JUMP_BACK POP_BLOCK
for38 ::= expr . get_iter store for_block JUMP_BACK
for38 ::= expr for_iter . store for_block JUMP_BACK
for38 ::= expr for_iter . store for_block JUMP_BACK POP_BLOCK
for38 ::= expr for_iter store . for_block JUMP_BACK
for38 ::= expr for_iter store . for_block JUMP_BACK POP_BLOCK
for38 ::= expr for_iter store for_block . JUMP_BACK
                                          ^^^^^^^^^ Instruction is "CONTINUE"
for38 ::= expr for_iter store for_block . JUMP_BACK POP_BLOCK
...
Instruction context:

   4      16  LOAD_NAME                range
             18  LOAD_CONST               2
             20  CALL_FUNCTION_1       1  ''
             22  GET_ITER
             24  FOR_ITER             30  'to 30'
->           26  STORE_NAME               i


# file /tmp/bug-3.8.pyc
# --- This code section failed: ---

   1       0  LOAD_CONST               0
           2  LOAD_CONST               None
           4  IMPORT_NAME              sys
           6  STORE_NAME               sys

   2       8  LOAD_CONST               0
          10  LOAD_CONST               None
          12  IMPORT_NAME              uncompyle6
          14  STORE_NAME               uncompyle6

   4      16  LOAD_NAME                range
          18  LOAD_CONST               2
          20  CALL_FUNCTION_1       1  ''
          22  GET_ITER
          24  FOR_ITER             30  'to 30'
          26  STORE_NAME               i

   5      28  CONTINUE             24  'to 24'
              ^^^^^^^^

Parse error at or near `STORE_NAME' instruction at offset 26

The essential fix applied in commit dba6d24361 was to add a grammar rule that doesn't need a JUMP_BACK since that is subsumed into the for_block.

In particular:

 for38 ::= expr for_iter store for_block

Another possibility is to making the JUMP_BACK optional.

In sum the bug is turning a JUMP_BACK into a CONTINUE so for loop has no JUMP_BACK.

Also there is no POP_BLOCK since there isn't anything in the loop.

In the future when we have better control flow, we might redo all of this.