* Add Type Annotations for parts of pwnlib/util
* fix: use Callable in pwnlib/util/lists.py
* fix: Import Errors
* fix: use getLogger in pwnlib/util/net
* feat: use collections.abc.Buffer where appropriate
* fix: remove collections.abc.Buffer to use Sequence
* add CHANGELOG.md entry
* Improve packing and fiddling types
* Introduce own `BytesLike` type instead of 3.12 Buffer
* Improve web.wget types
* Use TypedDict in util.net
Specify generics properly.
* Improve pwnlib.proc types
* Improve pwnlib.util.lists types
* Fix util.splash types
* Make TypeVars private
* Improve util.hexdump types
* Remove partial types from packing.fit/flat
Those require more work to be useful.
* Fix implicit wget return
* Fix term.text type export
* Fix js_escape return type
* Fix sh_string doctest types
* Fix usage of ssh_channel as ssh_process
We access the pid, so a plain ssh_channel is not enough.
* Improve util.lists types
* Improve packing.dd types
* Fix relying on socket.AddressFamily type at runtime
The type is only in typeshed and not the real socket module. Don't access it at runtime, only at type checking.
* Fix lists.findall with string needle
* Fix dd on two files
* Update mypy-baseline.txt
* Remove testing artifacts from mypy-baseline.txt
---------
Co-authored-by: peace-maker <[email protected]>
* Fix some global type hint errors
Errors in global scope should be fixed before we're enabling strict mode in mypy and tackle the local problems.
* Remove pwnlib.util.pairwise since it's in itertools now
pairwise was added to itertools itself in 3.10.
https://docs.python.org/3/library/itertools.html#itertools.pairwise
* Start to gradually add type hints
Use mypy-baseline to ignore all existing errors for now while avoiding adding new ones in new code.
* Remove Python 2 fallback imports
* Remove py2compat layer
* Avoid redefining a tube helper function
* Remove explicit new-style class (object) syntax
old-style classes were removed in Python 3 so there is no distinction anymore.
* Ignore type error in shellcraft helper script
* Fix repeatedly changing `linux_dirent.d_name`
* Avoid typechecking windows_termcap on Linux
We run mypy on Linux in CI, ignore Windows related import errors
* Run mypy-baseline in CI
* Add type hint plan to CONTRIBUTING
* Sort the mypy baseline
Run with `--sort-baseline` for better diffs.
`mypy | mypy-baseline sync --sort-baseline`
* Switch dev dependencies to dependency-groups
The dependency groups aren't baked into build metadata and won't appear on Pypi.
* [android] Split Android AVD testing into its own Workflow
* [android] Modify to trigger tests
* [android] Make paths more succinct and explicit
* Remove environment variable
* Rename to test
* Always run Android jobs for debugging
* Enhance Android workflow
* Disable gdbserver download
* Make the workflow actually work
* Travis missing gdbserver
* Try to fix travis
* Add good ol' SIGALRM-based debugs for tests
* Revert changes made to Travis
* But fix GHA
* Actually enable the android tests
* Actually enable
* Not necessary anymore
* Actually run the tests with AVD installation
Also fix incorrect text-vs-bytes usages in protocols.adb.
Co-authored-by: Arusekk <[email protected]>
* Fix several sphinx warnings
* Several code style fixes
Mostly grep -Er '[=!]= (None|True|False)'
* Compile-time str concat
also reduced redundant characters at line ends
* Fix python2 toplevel regression
* No equality comparison with singletons again
* Clarification: lambda: [] is just list
* Reintroduce reverted fragment
* Try GitHub actions instead of Travis CI.
This is just a proof-of-concept until we can
have it the clean way it should be done.
More commits will likely follow.
* Travis CI: Use flake8 to detect syntax errors (#1385)
* Travis CI: Use flake8 to detect syntax errors
Repeat #1305 on the Python 3 branch. Also, remove the hardcoding of default values for dist and Py2.
https://flake8.pycqa.org/en/latest/user/error-codes.html
On the flake8 test selection, this PR does _not_ focus on "_style violations_" (the majority of flake8 error codes that [__psf/black__](https://github.com/psf/black) can autocorrect). Instead these tests are focus on runtime safety and correctness:
* E9 tests are about Python syntax errors usually raised because flake8 can not build an Abstract Syntax Tree (AST). Often these issues are a sign of unused code or code that has not been ported to Python 3. These would be compile-time errors in a compiled language but in a dynamic language like Python they result in the script halting/crashing on the user.
* F63 tests are usually about the confusion between identity and equality in Python. Use ==/!= to compare str, bytes, and int literals is the classic case. These are areas where __a == b__ is True but __a is b__ is False (or vice versa). Python >= 3.8 will raise SyntaxWarnings on these instances.
* F7 tests logic errors and syntax errors in type hints
* F82 tests are almost always _undefined names_ which are usually a sign of a typo, missing imports, or code that has not been ported to Python 3. These also would be compile-time errors in a compiled language but in Python a __NameError__ is raised which will halt/crash the script on the user.
* Fail fast, exclude generated source files
* --select=E9,F63,F7
* 2to3 -f print -wn examples
* No more <> in python 3.
* Use ==/!= to compare str, bytes, and int literals (#1386)
Identity is not the same thing as equality in Python so use ==/!= to compare str, bytes, and int literals. In Python >= 3.8, these instances will raise _SyntaxWarnings_ so it is best to fix them now. https://docs.python.org/3.8/whatsnew/3.8.html#porting-to-python-3-8
$ python
```python
>>> raw = 'ra'
>>> raw += 'w'
>>> raw == 'raw'
True
>>> raw is 'raw'
False
```
* Use print() function in both Python 2 and Python 3 (#1387)
$ __pip install black future isort__
$ __futurize -f libfuturize.fixes.fix_print_with_import -w android.py__
$ __isort android.py__
$ __black android.py__
* Add cache and building artifacts
* Enable caching
* Actually enable caching for all of it
* Mistake
* Try different ABI on CI
* Try different ABI + version
* Try it harder
* Update SDK version
* Try building using newer SDK
* Accept licenses
* Ugh android emulator documentation is unfriendly
* Environment variables for AVD
* Add QEMU-system for android virtual devices
* Use the correct emulator
* New android image
* Was actually too new image
* Update ssh credentials to reflect GHA setup
* Fix spelling
* Try finding endless loops
* Actually use the timeout
* Reduce possible hang surface in doctests
* Try SSH immediately when set up
* SSH strict host key checking disable for localhost
* Try stackoverflow.com research
* Try to fix failing doctests
* Add required gcc for cross-arch tests
* Ensure gcc not failing in fmtstr
* Fix issues with shellcraft
* Try different libs
* Try again, with verbosity
* Why do these conflict???
* Fix issues related to PIE binaries
* Add bash-static
* Verbose 'could not set transport: any'
* Add AT_SECURE in auxv to prevent SEGV on libc bug
* Killing ADB server actually kills it
* Android image has /data/user/0 -> /data/data
* Set coverage mode from dev
* Try new actions
* Fix coveralls complaining
* A little time.sleep would not hurt or would it
Co-authored-by: Christian Clauss <[email protected]>
* Updates to requirements.txt.
Removed the < 1.8.0 requirement for sphinx. It appeared no longer
necessary.
Alphabetized the listing for easier searching.
Moved the requirements file under the docker folder. Docker does not
like relative paths to parent directories due to security restrictions.
* Updated Dockerfile to use Python3.
* Fixes to srop test errors.
Moving the imports to the beginning of the file resolved the issue.
The following errors were popping up during testing within Docker:
Document: rop/srop
------------------
**********************************************************************
File "rop/srop.rst", line ?, in default
Failed example:
p.recvline()
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.5/doctest.py", line 1321, in __run
compileflags, 1), test.globs)
File "<doctest default[18]>", line 1, in <module>
p.recvline()
File "/home/travis/pwntools/pwnlib/tubes/tube.py", line 467, in recvline
return self.recvuntil(self.newline, drop = not keepends, timeout = timeout)
File "/home/travis/pwntools/pwnlib/tubes/tube.py", line 310, in recvuntil
res = self.recv(timeout=self.timeout)
File "/home/travis/pwntools/pwnlib/tubes/tube.py", line 82, in recv
return self._recv(numb, timeout) or b''
File "/home/travis/pwntools/pwnlib/tubes/tube.py", line 160, in _recv
if not self.buffer and not self._fillbuffer(timeout):
File "/home/travis/pwntools/pwnlib/tubes/tube.py", line 131, in _fillbuffer
data = self.recv_raw(self.buffer.get_fill_size())
File "/home/travis/pwntools/pwnlib/tubes/process.py", line 707, in recv_raw
raise EOFError
EOFError
**********************************************************************
File "rop/srop.rst", line ?, in default
Failed example:
p.poll(block=True)
Expected:
0
Got:
-11
**********************************************************************
File "rop/srop.rst", line ?, in default
Failed example:
p.recvline()
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.5/doctest.py", line 1321, in __run
compileflags, 1), test.globs)
File "<doctest default[18]>", line 1, in <module>
p.recvline()
File "/home/travis/pwntools/pwnlib/tubes/tube.py", line 467, in recvline
return self.recvuntil(self.newline, drop = not keepends, timeout = timeout)
File "/home/travis/pwntools/pwnlib/tubes/tube.py", line 310, in recvuntil
res = self.recv(timeout=self.timeout)
File "/home/travis/pwntools/pwnlib/tubes/tube.py", line 82, in recv
return self._recv(numb, timeout) or b''
File "/home/travis/pwntools/pwnlib/tubes/tube.py", line 160, in _recv
if not self.buffer and not self._fillbuffer(timeout):
File "/home/travis/pwntools/pwnlib/tubes/tube.py", line 131, in _fillbuffer
data = self.recv_raw(self.buffer.get_fill_size())
File "/home/travis/pwntools/pwnlib/tubes/process.py", line 707, in recv_raw
raise EOFError
EOFError
**********************************************************************
File "rop/srop.rst", line ?, in default
Failed example:
p.poll(block=True)
Expected:
0
Got:
-11
**********************************************************************
File "rop/srop.rst", line ?, in default
Failed example:
p.recvline()
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.5/doctest.py", line 1321, in __run
compileflags, 1), test.globs)
File "<doctest default[18]>", line 1, in <module>
p.recvline()
File "/home/travis/pwntools/pwnlib/tubes/tube.py", line 467, in recvline
return self.recvuntil(self.newline, drop = not keepends, timeout = timeout)
File "/home/travis/pwntools/pwnlib/tubes/tube.py", line 310, in recvuntil
res = self.recv(timeout=self.timeout)
File "/home/travis/pwntools/pwnlib/tubes/tube.py", line 82, in recv
return self._recv(numb, timeout) or b''
File "/home/travis/pwntools/pwnlib/tubes/tube.py", line 160, in _recv
if not self.buffer and not self._fillbuffer(timeout):
File "/home/travis/pwntools/pwnlib/tubes/tube.py", line 131, in _fillbuffer
data = self.recv_raw(self.buffer.get_fill_size())
File "/home/travis/pwntools/pwnlib/tubes/process.py", line 707, in recv_raw
raise EOFError
EOFError
**********************************************************************
File "rop/srop.rst", line ?, in default
Failed example:
p.poll(block=True)
Expected:
0
Got:
-11
**********************************************************************
1 items had failures:
6 of 159 in default
159 tests in 1 items.
153 passed and 6 failed.
***Test Failed*** 6 failures.
* Fix for docstring warning in dynelf.py.
The following warning is resolved with this fix:
/home/travis/pwntools/pwnlib/dynelf.py:docstring of pwnlib.dynelf.DynELF._resolve_symbol_sysv:11: WARNING: Definition list ends without a blank line; unexpected unindent.
* Fix for docstring warnings in gdb.py.
The following warnings are resolved with this fix:
/home/travis/pwntools/pwnlib/gdb.py:docstring of pwnlib.gdb.debug_assembly:8: WARNING: Inline strong start-string without end-string.
/home/travis/pwntools/pwnlib/gdb.py:docstring of pwnlib.gdb.debug_assembly:21: WARNING: Block quote ends without a blank line; unexpected unindent.
/home/travis/pwntools/pwnlib/gdb.py:docstring of pwnlib.gdb.debug_shellcode:4: WARNING: Inline strong start-string without end-string.
/home/travis/pwntools/pwnlib/gdb.py:docstring of pwnlib.gdb.debug_shellcode:17: WARNING: Block quote ends without a blank line; unexpected unindent.
* Fix for adb/__init__.py.
The following error is resolved with this fix:
Document: protocols/adb
-----------------------
**********************************************************************
File "../../pwnlib/protocols/adb/__init__.py", line ?, in default
Failed example:
pprint(AdbClient().list('/data/user'))
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.5/doctest.py", line 1321, in __run
compileflags, 1), test.globs)
File "<doctest default[0]>", line 1, in <module>
pprint(AdbClient().list('/data/user'))
TypeError: 'module' object is not callable
**********************************************************************
1 items had failures:
1 of 10 in default
10 tests in 1 items.
9 passed and 1 failed.
***Test Failed*** 1 failures.
The following warning is resolved with this fix:
/home/travis/pwntools/pwnlib/protocols/adb/__init__.py:docstring of pwnlib.protocols.adb.AdbClient.stat:6: WARNING: Inline literal start-string without end-string.
* Fix for docstring warning in rop.py.
The following warning is resolved with this fix:
/home/travis/pwntools/pwnlib/rop/rop.py:docstring of pwnlib.rop.rop.DescriptiveStack.descriptions:3: WARNING: Inline literal start-string without end-string.
* Fix for docstring warning in ssh.py.
The following warning is resolved with this fix:
/home/travis/pwntools/pwnlib/tubes/ssh.py:docstring of pwnlib.tubes.ssh.ssh._init_remote_platform_info:6: WARNING: Block quote ends without a blank line; unexpected unindent.
* Doctest dictionary ordering problem in misc.py.
There was an ordering problem while running the doctest in misc.py.
Sorting the dictionary with pprint and updating the results fixed the
issue.
Document: util/misc
-------------------
**********************************************************************
File "../../pwnlib/util/misc.py", line ?, in default
Failed example:
sizes
Expected:
{'eax': 32, 'ax': 16, 'al': 8, 'ah': 8, 'ebx': 32, 'bx': 16, 'bl': 8, 'bh': 8, 'ecx': 32, 'cx': 16, 'cl': 8, 'ch': 8, 'edx': 32, 'dx': 16, 'dl': 8, 'dh': 8, 'edi': 32, 'di': 16, 'esi': 32, 'si': 16, 'ebp': 32, 'bp': 16, 'esp': 32, 'sp': 16}
Got:
{'si': 16, 'cx': 16, 'cl': 8, 'dl': 8, 'dh': 8, 'eax': 32, 'bx': 16, 'sp': 16, 'bl': 8, 'bh': 8, 'di': 16, 'esi': 32, 'edx': 32, 'dx': 16, 'ebx': 32, 'ebp': 32, 'ax': 16, 'ch': 8, 'al': 8, 'ah': 8, 'edi': 32, 'esp': 32, 'bp': 16, 'ecx': 32}
**********************************************************************
File "../../pwnlib/util/misc.py", line ?, in default
Failed example:
bigger
Expected:
{'eax': ['eax'], 'ax': ['eax', 'ax'], 'al': ['eax', 'ax', 'al'], 'ah': ['eax', 'ax', 'ah'], 'ebx': ['ebx'], 'bx': ['ebx', 'bx'], 'bl': ['ebx', 'bx', 'bl'], 'bh': ['ebx', 'bx', 'bh'], 'ecx': ['ecx'], 'cx': ['ecx', 'cx'], 'cl': ['ecx', 'cx', 'cl'], 'ch': ['ecx', 'cx', 'ch'], 'edx': ['edx'], 'dx': ['edx', 'dx'], 'dl': ['edx', 'dx', 'dl'], 'dh': ['edx', 'dx', 'dh'], 'edi': ['edi'], 'di': ['edi', 'di'], 'esi': ['esi'], 'si': ['esi', 'si'], 'ebp': ['ebp'], 'bp': ['ebp', 'bp'], 'esp': ['esp'], 'sp': ['esp', 'sp']}
Got:
{'si': ['esi', 'si'], 'cx': ['ecx', 'cx'], 'cl': ['ecx', 'cx', 'cl'], 'dl': ['edx', 'dx', 'dl'], 'dh': ['edx', 'dx', 'dh'], 'eax': ['eax'], 'bx': ['ebx', 'bx'], 'sp': ['esp', 'sp'], 'bl': ['ebx', 'bx', 'bl'], 'bh': ['ebx', 'bx', 'bh'], 'di': ['edi', 'di'], 'esi': ['esi'], 'edx': ['edx'], 'dx': ['edx', 'dx'], 'ebx': ['ebx'], 'ebp': ['ebp'], 'ax': ['eax', 'ax'], 'ch': ['ecx', 'cx', 'ch'], 'al': ['eax', 'ax', 'al'], 'ah': ['eax', 'ax', 'ah'], 'edi': ['edi'], 'esp': ['esp'], 'bp': ['ebp', 'bp'], 'ecx': ['ecx']}
**********************************************************************
File "../../pwnlib/util/misc.py", line ?, in default
Failed example:
smaller
Expected:
{'eax': ['ax', 'al', 'ah'], 'ax': ['al', 'ah'], 'al': [], 'ah': [], 'ebx': ['bx', 'bl', 'bh'], 'bx': ['bl', 'bh'], 'bl': [], 'bh': [], 'ecx': ['cx', 'cl', 'ch'], 'cx': ['cl', 'ch'], 'cl': [], 'ch': [], 'edx': ['dx', 'dl', 'dh'], 'dx': ['dl', 'dh'], 'dl': [], 'dh': [], 'edi': ['di'], 'di': [], 'esi': ['si'], 'si': [], 'ebp': ['bp'], 'bp': [], 'esp': ['sp'], 'sp': []}
Got:
{'si': [], 'cx': ['cl', 'ch'], 'cl': [], 'dl': [], 'dh': [], 'eax': ['ax', 'al', 'ah'], 'bx': ['bl', 'bh'], 'sp': [], 'bl': [], 'bh': [], 'di': [], 'esi': ['si'], 'edx': ['dx', 'dl', 'dh'], 'dx': ['dl', 'dh'], 'ebx': ['bx', 'bl', 'bh'], 'ebp': ['bp'], 'ax': ['al', 'ah'], 'ch': [], 'al': [], 'ah': [], 'edi': ['di'], 'esp': ['sp'], 'bp': [], 'ecx': ['cx', 'cl', 'ch']}
**********************************************************************
1 items had failures:
3 of 20 in default
20 tests in 1 items.
17 passed and 3 failed.
***Test Failed*** 3 failures.
* Fix for docstring warning in packing.py.
The following warning is resolved with this fix:
/home/travis/pwntools/pwnlib/util/packing.py:docstring of pwnlib.util.packing.flat:2: WARNING: Inline emphasis start-string without end-string.
* Removed commented out test in Dockerfile.
* Forgot to update location of requirements.txt.
Forgot to update the location of requirements.txt in .travis.yml.
* Made recommended changes from Arusekk.
* Replace division with floor division as of PEP-238
This will also get us closer to python3.
See https://www.python.org/dev/peps/pep-0238/
* Add from __future__ import division to source files
* Add pwnlib.protocols module
Migrate pwnlib.adb.protocol into that directory, and add compatibility shims
* Add documentation links, and run doctests
* Make doctests run
* pwnlib.protocols.adb.Client.transport should work with no device
