Get ready for release 2.14.0

This commit is contained in:
rocky 2017-11-26 09:35:50 -05:00
parent bfac9a6260
commit 5f6f78531f
4 changed files with 412 additions and 18 deletions

371
ChangeLog
View File

@ -1,7 +1,374 @@
2017-11-26 rocky <rb@dustyfeet.com>
* uncompyle6/version.py: Get ready for release 2.14.0
2017-11-26 rocky <rb@dustyfeet.com>
* uncompyle6/parsers/parse2.py: Back of build_set/build_list
separation
2017-11-26 rocky <rb@dustyfeet.com>
* uncompyle6/parsers/parse2.py, uncompyle6/parsers/parse3.py,
uncompyle6/semantics/pysource.py: Isolate build_set from build_list
in Python 2... More work is needed. Not done in Python 3 yet
2017-11-26 rocky <rb@dustyfeet.com>
* test/stdlib/runtests.sh: stdlib/runtess.sh python version
detection
2017-11-26 rocky <rb@dustyfeet.com>
* pytest/test_grammar.py, test/simple_source/bug22/01_kv.py,
test/simple_source/bug32/05_while_true_break.py: Work around grammar
remove rule bug... And reinstate source to a current 3.3 bug (which we don't detect).
But at least it is noted for future work.
2017-11-26 rocky <rb@dustyfeet.com>
* uncompyle6/parser.py, uncompyle6/parsers/parse22.py,
uncompyle6/parsers/parse24.py, uncompyle6/parsers/parse25.py,
uncompyle6/parsers/parse26.py, uncompyle6/parsers/parse27.py,
uncompyle6/parsers/parse32.py: Isolaate kv, kv2 and kv3 better
2017-11-26 rocky <rb@dustyfeet.com>
* uncompyle6/parser.py, uncompyle6/parsers/parse22.py: Localize kv
2017-11-26 rocky <rb@dustyfeet.com>
* uncompyle6/parser.py, uncompyle6/parsers/parse26.py,
uncompyle6/parsers/parse32.py: remove comp_ifnot
2017-11-25 rocky <rb@dustyfeet.com>
* : Another test
2017-11-25 rocky <rb@dustyfeet.com>
* test/Makefile, test/simple_source/bug27+/05_for_try_except.py,
uncompyle6/parsers/parse27.py: 2.7 control-flow bug: except/pass in
loop
2017-11-25 rocky <rb@dustyfeet.com>
* uncompyle6/parsers/parse2.py, uncompyle6/parsers/parse3.py,
uncompyle6/semantics/pysource.py: back off of build_slice{2,3}
->slice{2,3} There is another slice rule that interferes with this.
2017-11-25 rocky <rb@dustyfeet.com>
* test/simple_source/slice/02_slice.py, uncompyle6/parser.py,
uncompyle6/parsers/parse2.py, uncompyle6/parsers/parse3.py,
uncompyle6/scanners/scanner2.py, uncompyle6/scanners/scanner26.py,
uncompyle6/semantics/pysource.py: localize 2 and 3 argument
BUILD_SLICE... Nontermninal name matches AST anme now. Add test.
2017-11-25 rocky <rb@dustyfeet.com>
* test/Makefile: Improve grammar-cover targets
2017-11-25 rocky <rb@dustyfeet.com>
* pytest/test_grammar.py, test/Makefile, uncompyle6/parser.py,
uncompyle6/parsers/parse2.py, uncompyle6/semantics/fragments.py,
uncompyle6/semantics/make_function.py,
uncompyle6/semantics/pysource.py: remove unpack_list rule and ... tidy things a bit more
2017-11-24 rocky <rb@dustyfeet.com>
* uncompyle6/semantics/pysource.py: Fix bug in 2.x with decorator of
old-style class
2017-11-24 rocky <rb@dustyfeet.com>
* test/simple_source/expression/01_float.py,
uncompyle6/semantics/pysource.py: Handle inf, +inf, -nan, and nan
constants
2017-11-24 rocky <rb@dustyfeet.com>
* test/simple_source/expression/03_tuple_assign.py,
test/simple_source/stmts/04_withas.py,
uncompyle6/semantics/consts.py, uncompyle6/semantics/pysource.py: A
couple more bugs found running 2.7 stdlib tests
2017-11-24 rocky <rb@dustyfeet.com>
* : One more (lambda) test
2017-11-24 rocky <rb@dustyfeet.com>
* test/simple_source/expression/10_lambda.py,
uncompyle6/semantics/make_function.py: Fix bug where lambda has a
yield in it
2017-11-24 rocky <rb@dustyfeet.com>
* uncompyle6/parser.py, uncompyle6/semantics/consts.py: cmp ->
compare ... to better match the Python AST name Compare. This forces us to
change compare -> compare_single Relates to Issue #29
2017-11-24 rocky <rb@dustyfeet.com>
* test/simple_source/bug33/01_triple_compare.py,
uncompyle6/parser.py, uncompyle6/parsers/parse2.py,
uncompyle6/parsers/parse26.py, uncompyle6/parsers/parse27.py,
uncompyle6/parsers/parse3.py, uncompyle6/parsers/parse32.py,
uncompyle6/semantics/consts.py: cmp_list -> compare_chained ... to better match the Python AST name Compare. Relates to Issue #29
2017-11-23 rocky <rb@dustyfeet.com>
* test/stdlib/README.md, test/stdlib/compile-file.py,
test/stdlib/runtests.sh, uncompyle6/parsers/parse2.py: Start another
kind of testing... And fix a bug found in that with 3-way equal
2017-11-23 rocky <rb@dustyfeet.com>
* uncompyle6/parsers/parse36.py: Forgot to update a call in last
change
2017-11-23 rocky <rb@dustyfeet.com>
* uncompyle6/parsers/parse3.py: Reduce 3.5 call_aysnc grammar rules
2017-11-23 rocky <rb@dustyfeet.com>
* test/simple_source/bug32/03_if_try_raise.py,
uncompyle6/parsers/parse32.py: Trystmt grammar from 3.3 needed in
3.2 Add test to cover this
2017-11-23 rocky <rb@dustyfeet.com>
* uncompyle6/parsers/parse32.py, uncompyle6/parsers/parse33.py,
uncompyle6/parsers/parse35.py: Add triple compare from 3.3 into 3.2
2017-11-23 rocky <rb@dustyfeet.com>
* uncompyle6/parsers/parse32.py, uncompyle6/parsers/parse33.py:
Improve try else in 3.2... Grammar from 3.3 is relevant here
2017-11-23 rocky <rb@dustyfeet.com>
* uncompyle6/parser.py, uncompyle6/parsers/parse27.py,
uncompyle6/parsers/parse3.py, uncompyle6/parsers/parse34.py: Remove
whileelselastsmt ... Right now I don't know what it does and we don't have something that
needs it
2017-11-23 rocky <rb@dustyfeet.com>
* uncompyle6/parsers/parse27.py: provisional 2.7 whileelselast rule
2017-11-23 rocky <rb@dustyfeet.com>
* uncompyle6/parser.py, uncompyle6/parsers/parse26.py,
uncompyle6/parsers/parse27.py, uncompyle6/parsers/parse34.py:
grammar reduction of while loops
2017-11-23 rocky <rb@dustyfeet.com>
* test/simple_source/bug30/01_ops.py, uncompyle6/parser.py,
uncompyle6/parsers/parse2.py, uncompyle6/parsers/parse3.py,
uncompyle6/parsers/parse32.py: 3.x Grammar reduction of custom
rules... by looking for token patterns GET_ITER CALL_FUNCTION_1
2017-11-23 rocky <rb@dustyfeet.com>
* test/Makefile, test/simple_source/bug33/01_triple_compare.py,
uncompyle6/parser.py, uncompyle6/parsers/parse26.py,
uncompyle6/parsers/parse3.py, uncompyle6/parsers/parse32.py,
uncompyle6/parsers/parse33.py: Grammar coverage and pruning
2017-11-22 rocky <rb@dustyfeet.com>
* uncompyle6/parser.py, uncompyle6/parsers/parse3.py: Isotate
conditionalnot
2017-11-22 rocky <rb@dustyfeet.com>
* test/simple_source/expression/02_true_divide.py,
uncompyle6/parsers/parse2.py, uncompyle6/parsers/parse25.py,
uncompyle6/parsers/parse26.py, uncompyle6/parsers/parse27.py: 2.x
Grammar reduction/isolation
2017-11-22 rocky <rb@dustyfeet.com>
* uncompyle6/parser.py, uncompyle6/parsers/parse2.py,
uncompyle6/parsers/parse24.py: Grammar reduction for load_attrs
2017-11-22 rocky <rb@dustyfeet.com>
* uncompyle6/parsers/parse2.py, uncompyle6/parsers/parse3.py: Reduce
unecessary grammar rules in 2.x
2017-11-22 rocky <rb@dustyfeet.com>
* test/simple_source/stmts/01_augmented_assign.py,
uncompyle6/parsers/parse2.py, uncompyle6/parsers/parse27.py:
Increase grammar coverage
2017-11-22 rocky <rb@dustyfeet.com>
* admin-tools/setup-master.sh, admin-tools/setup-python-2.4.sh:
Administrivia: add "git pull"s
2017-11-18 rocky <rb@dustyfeet.com>
* uncompyle6/parsers/parse3.py, uncompyle6/parsers/parse33.py,
uncompyle6/parsers/parse34.py: 3.3. grammar cleanup ... need build_list in for BUILD_TUPLE_0
2017-11-18 rocky <rb@dustyfeet.com>
* pytest/test_grammar.py, uncompyle6/parser.py,
uncompyle6/parsers/parse24.py, uncompyle6/semantics/pysource.py:
Grammar cleanup: import_as_cont -> import_as
2017-11-18 rocky <rb@dustyfeet.com>
* __pkginfo__.py, pytest/test_grammar.py: Track spark_parser changes
2017-11-17 rocky <rb@dustyfeet.com>
* test/Makefile, uncompyle6/parsers/parse3.py: custom rule hacking
... Reduce extraneous 3.x "load_list" and "load_closure" rules
2017-11-17 rocky <rb@dustyfeet.com>
* uncompyle6/parsers/parse3.py, uncompyle6/parsers/parse33.py:
Python 3 grammar clean up and reorganization
2017-11-17 rocky <rb@dustyfeet.com>
* test/Makefile, uncompyle6/parser.py,
uncompyle6/parsers/parse2.py, uncompyle6/parsers/parse3.py,
uncompyle6/parsers/parse36.py: More grammar isolation. narrow custom
rules... Specifically: _mklambda with load_closure is Python2 only Add
classdefdeco2 rule only when we have seen a LOAD_BUILD_CLASS
2017-11-16 rocky <rb@dustyfeet.com>
* uncompyle6/parsers/parse34.py: Python 3.4 while grammar cleanup
2017-11-16 rocky <rb@dustyfeet.com>
* uncompyle6/parser.py, uncompyle6/parsers/parse3.py: More grammar
cleanups
2017-11-16 rocky <rb@dustyfeet.com>
* uncompyle6/semantics/consts.py: Tag %c nonterminals in more
template rules
2017-11-16 rocky <rb@dustyfeet.com>
* uncompyle6/parsers/parse2.py: Bump copyright
2017-11-16 rocky <rb@dustyfeet.com>
* uncompyle6/parser.py, uncompyle6/parsers/parse2.py,
uncompyle6/parsers/parse3.py: More grammar isolation: assert2 in
Python3
2017-11-16 rocky <rb@dustyfeet.com>
* uncompyle6/parsers/parse25.py, uncompyle6/parsers/parse3.py:
Isolate "assert2" rule
2017-11-16 rocky <rb@dustyfeet.com>
* uncompyle6/parser.py, uncompyle6/parsers/parse23.py: Isolate
"and2" rule to 2.3
2017-11-16 rocky <rb@dustyfeet.com>
* test/simple_source/stmts/05_with.py,
uncompyle6/parsers/parse25.py, uncompyle6/parsers/parse26.py: Python
2.5 "with"; Isolate 2.5-2.6 grammar better
2017-11-16 rocky <rb@dustyfeet.com>
* test/simple_source/expression/02_true_divide.py: Add
BINARY_TRUE_DIVIDE tests for 2.{6,7}
2017-11-16 rocky <rb@dustyfeet.com>
* uncompyle6/parsers/parse26.py, uncompyle6/parsers/parse27.py: More
grammar cleanup
2017-11-16 rocky <rb@dustyfeet.com>
* uncompyle6/scanners/scanner2.py: 2.7 continue-detection bug
2017-11-15 rocky <rb@dustyfeet.com>
* uncompyle6/parsers/parse2.py, uncompyle6/parsers/parse27.py: More
2.7/2.7- grammer separation & cleanup
2017-11-15 rocky <rb@dustyfeet.com>
* uncompyle6/parsers/parse2.py, uncompyle6/parsers/parse27.py:
Grammar cleanup: separate some 2.7 from 2.7- rules
2017-11-15 rocky <rb@dustyfeet.com>
* uncompyle6/parser.py: I said, remove dict/setcomp from <= 2.6!
2017-11-15 rocky <rb@dustyfeet.com>
* uncompyle6/parsers/parse2.py, uncompyle6/parsers/parse27.py,
uncompyle6/parsers/parse3.py: grammar cleanup: dict/set
comprehensions 2.7+
2017-11-15 rocky <rb@dustyfeet.com>
* HOW-TO-REPORT-A-BUG.md: Grammar typo
2017-11-15 rocky <rb@dustyfeet.com>
* HOW-TO-REPORT-A-BUG.md: Grammar typo
2017-11-15 rocky <rb@dustyfeet.com>
* : One more pypy test
2017-11-15 rocky <rb@dustyfeet.com>
* : Keep bytecode_2.7 pypy free... We need to add _another_ pypy directory
2017-11-15 rocky <rb@dustyfeet.com>
* : del stmt testing on 2.7
2017-11-15 rocky <rb@dustyfeet.com>
* test/Makefile, uncompyle6/parsers/parse3.py: Stronger 3.4 testing
... parse3.py: remove unused grammar rule
2017-11-15 rocky <rb@dustyfeet.com>
* test/Makefile, test/simple_source/stmts/10_del.py,
test/test_pyenvlib.py, uncompyle6/parsers/parse26.py,
uncompyle6/semantics/check_ast.py, uncompyle6/semantics/pysource.py:
Profiling workarounds, more coverage ... test/Makefile: more grammar checking. Update python versions
10_del.pyc add test of DEL_GLOBAL check_ast.py, pysource.py: Profileing workarounds
2017-11-14 rocky <rb@dustyfeet.com>
* uncompyle6/parsers/parse2.py: Guard around null ast This can happen in profiling
2017-11-13 rocky <rb@dustyfeet.com>
* uncompyle6/parser.py, uncompyle6/semantics/pysource.py: detected
old-style Python 2.4 class better
* HOW-TO-REPORT-A-BUG.md: bug report doc tweak
2017-11-13 rocky <rb@dustyfeet.com>
* HOW-TO-REPORT-A-BUG.md: add note on illegal use
2017-11-13 rocky <rb@dustyfeet.com>
* ChangeLog, uncompyle6/parser.py, uncompyle6/semantics/pysource.py:
detected old-style Python 2.4 class better
2017-11-13 rocky <rb@dustyfeet.com>

8
NEWS
View File

@ -1,3 +1,11 @@
uncompyle6 2.14.0 2017-11-26 johnnybamazing
- Start to isolate grammar rules between versions
and remove used grammar rules
- Fix a number of bytecode decompile problems
(many more remain)
- Add stdlib/runtests.sh for even more rigourous testing
uncompyle6 2.13.3 2017-11-13
Overall: better 3.6 decompiling and some much needed code refactoring and cleanup

View File

@ -17,17 +17,23 @@ source code. It accepts bytecodes from Python version 1.5, and 2.1 to
Why this?
---------
There were a number of decompyle, uncompile, uncompyle2, uncompyle3
There were a number of decompyle, uncompyle, uncompyle2, uncompyle3
forks around. All of them came basically from the same code base, and
almost all of them no were no longer actively maintained. Only one
handled Python 3, and even there, only 3.2 or 3.3 depending on which
code is used. This code pulls these together and moves forward. This
project has the most complete support for Python 3.3 and above. It
also addresses a number of open issues in the previous forks.
code is used. This code pulls these together and moves forward.
What makes this different from other CPython bytecode decompilers?: its
ability to deparse just fragments and give source-code information
around a given bytecode offset.
This project has the most complete support for Python 3.3 and above
and the best all-around Pythoin support.
We are serious about testing, and use automated processes to find
bugs. In the issue trackers for other decompilers, you will find a
number of bugs we've found along the way. Very few to none of them are
not fixed in the other decompilers.
Another thing that makes this different from other CPython bytecode
decompilers is the ability to deparse just fragments and give
source-code information around a given bytecode offset.
I use this to deparse fragments of code inside my trepan_
debuggers_. For that, I need to record text fragments for all
@ -109,14 +115,14 @@ Known Bugs/Restrictions
-----------------------
The biggest known and possibly fixable (but hard) problem has to do
with handling control flow. All of the Python decompilers I have looked
at have the same problem. In some cases we can detect an erroneous
decompilation and report that.
with handling control flow. (Python has probably the most diverse and
screwy set of compound statements I've ever seen; a number of the
usual ones like else clauses on loops and try blocks I suspect most
programmers don't know aobut.)
Over 98% of the decompilation of Python standard library packages in
Python 2.7.12 verifies correctly. Over 99% of Python 2.7 and 3.3-3.5
"weakly" verify. Python 2.6 drops down to 96% weakly verifying.
Other versions drop off in quality too.
All of the Python decompilers I have looked at have the same
problem. In some cases we can detect an erroneous decompilation and
report that.
*Verification* is the process of decompiling bytecode, compiling with
a Python for that bytecode version, and then comparing the bytecode
@ -134,6 +140,17 @@ program by running the Python interpreter. Because the Python language
has changed so much, for best results you should use the same Python
Version in checking as used in the bytecode.
Finally, we have automated running the standard Python tests after
first compiling and decompiling the test program. Results here are a
bit weak (if not better than most other Python decompilers). But over
time this will probably get better.
Python support is strongest in Python 2 for 2.7 and drops off as you
get further away from that. Support is also probably pretty good for
python 2.3-2.4 since a lot of the goodness of early the version of the
decompiler from that era has been preserved (and Python compilation in
that era was minimal)
Later distributions average about 200 files. There is some work to do
on the lower end Python versions which is more difficult for us to
handle since we don't have a Python interpreter for versions 1.5, 1.6,
@ -144,7 +161,9 @@ In the Python 3 series, Python support is is strongest around 3.4 or
3.6 changes things drastically by using word codes rather than byte
codes. That has been addressed, but then it also changes function call
opcodes and its semantics and has more problems with control flow than
3.5 has.
3.5 has. Between Python 3.5, 3.6 and 3.7 there have been major changes
to the `MAKE_FUNCTION` and `CALL_FUNCTION` instructions. Those are
not handled yet.
Currently not all Python magic numbers are supported. Specifically in
some versions of Python, notably Python 3.6, the magic number has

View File

@ -1,3 +1,3 @@
# This file is suitable for sourcing inside bash as
# well as importing into Python
VERSION='2.13.3'
VERSION='2.14.0'