Release 3.4.1

* Do not fail with gdb.debug() over ssh when given an absolute path

Fixes Gallopsled/pwntools#885

* Prevent failing on stable since this directory doesn't exist

.github/ISSUE_TEMPLATE.md

@@ -6,4 +6,4 @@ When reporting an issue, be sure that you are running the latest released versio
 
 Please verify that your issue occurs on 64-bit Ubuntu 14.04.  You can use the Dockerfile in `extra/docker/Dockerfile` to save time.
 
-If possible, provide a proof-of-concept which demonstrates the problem.  Include any binaries or scripts necessary to reproduce the issue.
+If possible, provide a proof-of-concept which demonstrates the problem.  Include any binaries or scripts necessary to reproduce the issue, and please include the full debug output via setting the environment variable `PWNLIB_DEBUG=1`.

.travis.yml

@@ -13,6 +13,7 @@ addons:
     - mksh
     - zsh
     - pandoc
+    - gdb
 cache:
     - pip
     - directories:
@@ -22,11 +23,14 @@ cache:
 python:
   - "2.7"
 before_install:
+  - echo $-
   - source travis/install.sh
   - source travis/ssh_setup.sh
+  - echo $-
 install:
+  - pip install --upgrade pip
   - pip install --upgrade --editable .
-  - pip install --requirement docs/requirements.txt
+  - pip install --upgrade --requirement docs/requirements.txt
 before_script:
   - PWNLIB_NOTERM=1 python -c 'from pwn import *; print pwnlib.term.term_mode'
   - PWNLIB_NOTERM=1 python -c 'from pwn import *; print pwnlib.term.term_mode'

CHANGELOG.md

@@ -9,9 +9,17 @@ The table below shows which release corresponds to each branch, and what date th
 
 | Version          | Branch   | Release Date           |
 | ---------------- | -------- | ---------------------- |
-| [3.4.0](#340)    | `dev`    | Feb 4, 2017 (planned)
-| [3.3.0](#330)    | `beta`   | Dec 24, 2016 (planned)
-| [3.2.0](#320)    | `stable` | Nov 12, 2016
+| [3.6.0](#350)    | `dev`    | May 6, 2017 (planned)
+| [3.5.0](#350)    | `beta`   | Mar 25, 2017 (planned)
+| [3.4.1](#341)    | `stable` | Feb 17, 2017
+| [3.4.0](#340)    |          | Feb 13, 2017
+| [3.3.4](#334)    |          | Jan 12, 2016
+| [3.3.3](#333)    |          | Jan 10, 2016
+| [3.3.2](#332)    |          | Jan 10, 2016
+| [3.3.1](#331)    |          | Jan 10, 2016
+| [3.3.0](#330)    |          | Dec 24, 2016
+| [3.2.1](#321)    |          | Dec 24, 2016
+| [3.2.0](#320)    |          | Nov 12, 2016
 | [3.1.1](#311)    |          | Oct 23, 2016
 | [3.1.0](#310)    |          | Oct 2, 2016
 | [3.0.4](#304)    |          | Sept 19, 2016
@@ -21,19 +29,136 @@ The table below shows which release corresponds to each branch, and what date th
 | [3.0.0](#300)    |          | Aug 20, 2016
 | [2.2.0](#220)    |          | Jan 5, 2015
 
+
+## 3.6.0
+
+To be released on May 6, 2017.
+
+## 3.5.0
+
+To be released on Mar 25, 2017.
+
+## 3.4.1
+
+- [#894][894] Fix a bug when using `gdb.debug()` over ssh.
+- [e021f57][e021f57] Fix a bug ([#891][891]) in `rop` when needing to insert padding to fix alignment
+
+[e021f57]: https://github.com/Gallopsled/pwntools/commit/e021f57
+[894]: https://github.com/Gallopsled/pwntools/pull/894
+[891]: https://github.com/Gallopsled/pwntools/issues/891
+
+
 ## 3.4.0
 
-To be released on Feb 4, 2017.
+- [#800][800] Add `shell=` option to `ssh.process()`
+- [#806][806] Add `context.buffer_size` for fine-tuning `tube` performance
+    + Also adds `buffer_fill_size=` argument for all tubes
+- [b83a6c7][b83a6c7] Fix undocumented `process.leak` function
+- [546061e][546061e] Modify `coredump_filter` of all spawned processes, so that core dumps are more complete
+- [#809][809] Add several functions to `adb` (`unlink`, `mkdir`, `makedirs`, `isdir`, `exists`)
+- [#817][817] Make disconnection detection more robust
+
+[800]: https://github.com/Gallopsled/pwntools/pull/800
+[806]: https://github.com/Gallopsled/pwntools/pull/806
+[809]: https://github.com/Gallopsled/pwntools/pull/809
+[817]: https://github.com/Gallopsled/pwntools/pull/817
+[5d9792f]: https://github.com/Gallopsled/pwntools/commit/5d9792f
+[b83a6c7]: https://github.com/Gallopsled/pwntools/commit/b83a6c7
+[546061e]: https://github.com/Gallopsled/pwntools/commit/546061e
+
+## 3.3.4
+
+- [#850][850] and [#846][846] fix issues with `hexdump` and the `phd` command-line utility, when using pipes (e.g. `echo foo | phd`)
+- [#852][852] Fixes register ordering in `regsort`
+- [#853][853] Fixes the registers restored in `shellcraft.amd64.popad`
+
+[846]: https://github.com/gallopsled/pwntools/pull/846
+[850]: https://github.com/gallopsled/pwntools/pull/850
+[852]: https://github.com/gallopsled/pwntools/pull/852
+[853]: https://github.com/gallopsled/pwntools/pull/853
+
+## 3.3.3
+
+- [#843][843] fixed a bug in `amd64.mov`.
+
+[843]: https://github.com/gallopsled/pwntools/pull/843
+
+## 3.3.2
+
+- [#840][840] fixed a regression introduced by [#837][837].
+
+[840]: https://github.com/gallopsled/pwntools/pull/840
+
+## 3.3.1
+
+- [#833][833] Fixed a performance-impacting bug in the adb module.
+- [#837][837] Fixed a bug([#836][836]) causing `hexdump(cyclic=True)` to throw an exception.
+
+[833]: https://github.com/Gallopsled/pwntools/pull/833
+[837]: https://github.com/Gallopsled/pwntools/pull/837
+[836]: https://github.com/Gallopsled/pwntools/issues/836
 
 ## 3.3.0
 
-To be released on Dec 24, 2016.
+- [b198ec8][b198ec8] Added `tube.stream()` function, which is like `tube.interact()` without a prompt or keyboard input.
+    + Effectively, this is similar to `cat file` and just prints data as fast as it is received.
+- [aec3fa6][aec3fa6] Disable update checks against GitHub
+    + These checks frequently broke due to GitHub query limits
+- [#757][757] Fixed `adb.wait_for_device()` re-use of the same connection
+- [f9133b1][f9133b1] Add a `STDERR` magic argument to make logging go to `stderr` instead of `stdout`
+    + Usage is e.g. `python foo.py STDERR` or `PWNLIB_STDERR=1 python foo.py`
+    + Also adds `context.log_console` to log to any file or terminal
+- [67e11a9][67e11a9] Add faster error checking to `cyclic()` when provided very large values
+- [5fda658][5fda658] Expose BitPolynom in `globals()`
+- [#765][765] Added `-d` option for hex-escaped output for `shellcraft` command-line tool
+- [#772][772] Fixed bash completion regressions
+- [30c34b7][30c34b7] Fix `ROP.call()` with `Function` objects from `ELF.functions`
+- [fa402ce][fa402ce] Add `adb.uptime` and `adb.boot_time`
+- [82312ba][82312ba] Add `cyclic_metasploit` and `cyclic_metasploit_find`
+
+[757]: https://github.com/Gallopsled/pwntools/pull/757
+[765]: https://github.com/Gallopsled/pwntools/pull/765
+[772]: https://github.com/Gallopsled/pwntools/pull/772
+[b198ec8]: https://github.com/Gallopsled/pwntools/commit/b198ec8
+[aec3fa6]: https://github.com/Gallopsled/pwntools/commit/aec3fa6
+[f9133b1]: https://github.com/Gallopsled/pwntools/commit/f9133b1
+[67e11a9]: https://github.com/Gallopsled/pwntools/commit/67e11a9
+[5fda658]: https://github.com/Gallopsled/pwntools/commit/5fda658
+[30c34b7]: https://github.com/Gallopsled/pwntools/commit/30c34b7
+[fa402ce]: https://github.com/Gallopsled/pwntools/commit/fa402ce
+[82312ba]: https://github.com/Gallopsled/pwntools/commit/82312ba
+
+## 3.2.1
+
+Multiple bug fixes.
+
+- [#783][783] Fix `adb.uninstall` typo
+- [#787][787] Added error handling for `ssh.process` argument `preexec_fn`
+- [#793][793] Fixed progress message in `remote()` when connections failed
+- [#802][802] Fixed partition listing in `adb.partitions`, which accidentally shelled out to the `adb` binary
+- [#804][804] Fix error message for 32-bit distributions
+- [#805][805] Fix exception in `Core.segments` when a segment has no name
+- [#811][811] Fixes and performance improvements for `adb.wait_for_device()`
+- [#813][813] Fixed a release script
+- [#814][814] Fixed exceptions thrown if the `$HOME` directory is not writable
+- [#815][815] Properly handle `None` in `MemLeak`
+
+[783]: https://github.com/Gallopsled/pwntools/pull/783
+[787]: https://github.com/Gallopsled/pwntools/pull/787
+[793]: https://github.com/Gallopsled/pwntools/pull/793
+[802]: https://github.com/Gallopsled/pwntools/pull/802
+[804]: https://github.com/Gallopsled/pwntools/pull/804
+[805]: https://github.com/Gallopsled/pwntools/pull/805
+[811]: https://github.com/Gallopsled/pwntools/pull/811
+[813]: https://github.com/Gallopsled/pwntools/pull/813
+[814]: https://github.com/Gallopsled/pwntools/pull/814
+[815]: https://github.com/Gallopsled/pwntools/pull/815
 
 ## 3.2.0
 
 - [#695][695] Fixed a performance regression in `phd`.
-- [452605e][452605e] Fixed [#629][#629] related to correct removal of temporary files.
-- [ea94ee4][ea94ee4] Disallows semi-colons in for the `run_in_terminal` function, since it did not work properly in all cases. 
+- [452605e][452605e] Fixed [#629][629] related to correct removal of temporary files.
+- [ea94ee4][ea94ee4] Disallows semi-colons in for the `run_in_terminal` function, since it did not work properly in all cases.
 - [6376d07][6376d07] Added the mips shellcode `pushstr_array`.
 - [#700][700] Added missing MIPS shellcode documentation to readthedocs, and enabled unit tests
 - [#701][701] Command line tools refactored to have a common `pwn` entry point.
@@ -60,11 +185,12 @@ To be released on Dec 24, 2016.
 - [1e414af][1e414af] Added `connect()` alias for `remote()`
     + For example, `io=connect('google.com', 80)`
     + This also works with `tcp(...)` and `udp(...)` aliases
-- [869ec42][869ec42] Added `ssh.read()` and `ssh.write()` aiases
+- [869ec42][869ec42] Added `ssh.read()` and `ssh.write()` aliases
 - [2af55c9][2af55c9] `AdbDevice` objects exposed via e.g. `adb.devices()` now offer scoped access to all `adb` module properties
     + It is now possible to e.g. `map(lambda d: d.process(['id']).recvall(), adb.devices())`
 
 
+[629]: https://github.com/Gallopsled/pwntools/issues/629
 [695]: https://github.com/Gallopsled/pwntools/pull/695
 [700]: https://github.com/Gallopsled/pwntools/pull/700
 [701]: https://github.com/Gallopsled/pwntools/pull/701
@@ -146,7 +272,7 @@ Android support via a new `adb` module, `context.device`, `context.adb_host`, an
 - Added `context.log_file` setting for sending logs to a file.  This can be set with the `LOG_FILE` magic command-line option.
 - Added `context.noptrace` setting for disabling actions which require `ptrace` support.  This is useful for turning all `gdb.debug` and `gdb.attach` options into no-ops, and can be set via the `NOPTRACE` magic command-line option.
 - Added `context.proxy` which hooks all connections and sends them to a SOCKS4/SOCKS5.  This can be set via the `PROXY` magic command-line option.
-- Added `context.randomize` to control randommization of settings like XOR keys and register ordering (default off).
+- Added `context.randomize` to control randomization of settings like XOR keys and register ordering (default off).
 - Added `context.terminal` for setting how to launch commands in a new terminal.
 
 #### DynELF and MemLeak Module
@@ -159,7 +285,7 @@ Android support via a new `adb` module, `context.device`, `context.adb_host`, an
 #### Encoders Module
 
 - Added a `pwnlib.encoders` module for assembled-shellcode encoders/decoders
-- Includes position-indepentent basic XOR encoders
+- Includes position-independent basic XOR encoders
 - Includes position-independent delta encoders
 - Includes non-position-independent alphanumeric encoders for Intel
 - Includes position-independent alphanumeric encoders for ARM/Thumb

README.md

@@ -2,7 +2,7 @@
 ![pwntools logo](https://github.com/Gallopsled/pwntools/blob/stable/docs/source/logo.png?raw=true)
 
 [![Docs](https://readthedocs.org/projects/pwntools/badge/?version=stable)](https://docs.pwntools.com/)
-[![PyPI](https://img.shields.io/badge/pypi-v3.2.0-green.svg?style=flat)](https://pypi.python.org/pypi/pwntools/)
+[![PyPI](https://img.shields.io/badge/pypi-v3.4.1-green.svg?style=flat)](https://pypi.python.org/pypi/pwntools/)
 [![Travis](https://travis-ci.org/Gallopsled/pwntools.svg)](https://travis-ci.org/Gallopsled/pwntools)
 [![Coveralls](https://img.shields.io/coveralls/Gallopsled/pwntools/dev.svg)](https://coveralls.io/github/Gallopsled/pwntools?branch=dev)
 [![Twitter](https://img.shields.io/badge/twitter-pwntools-4099FF.svg?style=flat)](https://twitter.com/pwntools)
@@ -38,7 +38,7 @@ Most of the functionality of pwntools is self-contained and Python-only.  You sh
 
 ```sh
 apt-get update
-apt-get install python2.7 python-pip python-dev git libssl-dev
+apt-get install python2.7 python-pip python-dev git libssl-dev libffi-dev
 pip install --upgrade pwntools
 ```
 

TESTING.md

@@ -5,7 +5,7 @@ Pwntools makes extensive use of unit tests and integration tests to ensure every
 ## Test Suite
 
 
-To run the test suite, it is beste to use Ubuntu 12.04 or 14.04, and run the following commands.  **Be aware** that this will add a user to the machine, and create a public key for SSH login!
+To run the test suite, it is best to use Ubuntu 12.04 or 14.04, and run the following commands.  **Be aware** that this will add a user to the machine, and create a public key for SSH login!
 
 ```sh
 bash travis/install.sh

docs/source/args.rst

@@ -0,0 +1,9 @@
+.. testsetup:: *
+
+   from pwn import *
+
+:mod:`pwnlib.args` --- Magic Command-Line Arguments
+=====================================================
+
+.. automodule:: pwnlib.args
+   :members:

docs/source/conf.py

@@ -55,6 +55,16 @@ pwnlib.context.ContextType.defaults['randomize'] = False
 pwnlib.term.text.when = 'never'
 pwnlib.log.install_default_handler()
 pwnlib.log.rootlogger.setLevel(1)
+
+# Sphinx modifies sys.stdout, and context.log_terminal has
+# a reference to the original instance.  We need to update
+# it for logging to be captured.
+class stdout(object):
+    def __getattr__(self, name):
+        return getattr(sys.stdout, name)
+    def __setattr__(self, name, value):
+        return setattr(sys.stdout, name, value)
+pwnlib.context.ContextType.defaults['log_console'] = stdout()
 '''
 
 autodoc_member_order = 'alphabetical'

docs/source/index.rst

@@ -43,6 +43,7 @@ Each of the ``pwntools`` modules is documented here.
    :glob:
 
    adb
+   args
    asm
    atexception
    atexit

docs/source/install.rst

@@ -24,7 +24,8 @@ pwntools is available as a ``pip`` package.
 
 .. code-block:: bash
 
-    $ apt-get install python2.7 python2.7-dev python-pip
+    $ apt-get update
+    $ apt-get install python2.7 python-pip python-dev git libssl-dev libffi-dev
     $ pip install --upgrade pwntools
 
 Development

pwn/__init__.py

@@ -13,4 +13,5 @@ if not platform.architecture()[0].startswith('64'):
     See Gallopsled/pwntools#518 for more information."""
     log.warn_once('Pwntools does not support 32-bit Python.  Use a 64-bit release.')
 
-pwnlib.update.check_automatically()
+with context.local(log_console=sys.stderr):
+    pwnlib.update.check_automatically()

pwn/toplevel.py

@@ -51,6 +51,7 @@ from pwnlib.util import iters
 from pwnlib.util import net
 from pwnlib.util import proc
 from pwnlib.util import safeeval
+from pwnlib.util.crc import BitPolynom
 from pwnlib.util.cyclic import *
 from pwnlib.util.fiddling import *
 from pwnlib.util.getdents import *

pwnlib/adb/adb.py

@@ -10,12 +10,38 @@ If you have multiple devices, you have a handful of options to select one, or it
 over the devices.
 
 First and most important is the ``context.device`` property, which declares the "currently"
-seelcted device in any scope.  It can be set manually to a serial number, or to a ``Device``
+selected device in any scope.  It can be set manually to a serial number, or to a ``Device``
 instance.
 
 .. code-block:: python
 
-    >>> context.device = ''
+    # Take the first available device
+    context.device = adb.wait_for_device()
+
+    # Set a device by serial number
+    context.device = 'ZX1G22LH8S'
+
+    # Set a device by its product name
+    for device in adb.devices():
+        if device.product == 'shamu':
+            break
+    else:
+        error("Could not find any shamus!")
+
+Once a device is selected, you can operate on it with any of the functions in
+the ``pwnlib.adb`` module.
+
+.. code-block:: python
+
+    # Get a process listing
+    print adb.process(['ps']).recvall()
+
+    # Fetch properties
+    print adb.properties.ro.build.fingerprint
+
+    # Read and write files
+    print adb.read('/proc/version')
+    adb.write('/data/local/tmp/foo', 'my data')
 
 """
 import functools
@@ -162,8 +188,46 @@ def reboot_bootloader():
     with Client() as c:
         c.reboot_bootloader()
 
+@with_device
+def uptime():
+    """uptime() -> float
+
+    Returns:
+        Uptime of the device, in seconds
+    """
+    up, idle = map(float, read('/proc/uptime').split())
+    return up
+
+@with_device
+def boot_time():
+    """boot_time() -> int
+
+    Returns:
+        Boot time of the device, in Unix time, rounded to the
+        nearest second.
+    """
+    for line in read('/proc/stat').splitlines():
+        name, value = line.split(None, 1)
+        if name == 'btime':
+            return int(value)
+
 class AdbDevice(Device):
-    """Encapsulates information about a connected device."""
+    """Encapsulates information about a connected device.
+
+    Example:
+
+        >>> device = adb.wait_for_device()
+        >>> device.arch
+        'arm'
+        >>> device.bits
+        32
+        >>> device.os
+        'android'
+        >>> device.product
+        'sdk_phone_armv7'
+        >>> device.serial
+        'emulator-5554'
+    """
     def __init__(self, serial, type, port=None, product='unknown', model='unknown', device='unknown', features=None, **kw):
         self.serial  = serial
         self.type    = type
@@ -176,13 +240,45 @@ class AdbDevice(Device):
         if product == 'unknown':
             return
 
-        with context.local(device=serial):
+        # Deferred fields
+        self._initialized = False
+        self._arch = None
+        self._bits = None
+        self._endian = None
+        self._avd = None
+
+    @property
+    def arch(self):
+        self.__do_deferred_initialization()
+        return self._arch
+
+    @property
+    def avd(self):
+        self.__do_deferred_initialization()
+        return self._avd
+
+    @property
+    def bits(self):
+        self.__do_deferred_initialization()
+        return self._bits
+
+    @property
+    def endian(self):
+        self.__do_deferred_initialization()
+        return self._endian
+
+
+    def __do_deferred_initialization(self):
+        if self._initialized:
+            return
+
+        with context.local(device=self.serial):
             abi = str(properties.ro.product.cpu.abi)
             context.clear()
             context.arch = str(abi)
-            self.arch = context.arch
-            self.bits = context.bits
-            self.endian = context.endian
+            self._arch = context.arch
+            self._bits = context.bits
+            self._endian = context.endian
 
         if self.port == 'emulator':
             emulator, port = self.serial.split('-')
@@ -195,7 +291,8 @@ class AdbDevice(Device):
                     self.avd = r.recvline().strip()
             except:
                 pass
-            # r = remote('localhost')
+
+        self._initialized = True
 
     def __str__(self):
         return self.serial
@@ -296,7 +393,9 @@ def wait_for_device(kick=False):
         for device in devices():
             if context.device == device:
                 return device
-            break
+
+            if not serial:
+                break
         else:
             log.error("Could not find any devices")
 
@@ -500,6 +599,150 @@ def write(path, data=''):
         misc.write(temp.name, data)
         push(temp.name, path)
 
+@context.quiet
+@with_device
+def mkdir(path):
+    """Create a directory on the target device.
+
+    Note:
+        Silently succeeds if the directory already exists.
+
+    Arguments:
+        path(str): Directory to create.
+
+    Examples:
+
+        >>> adb.mkdir('/')
+
+        >>> path = '/data/local/tmp/mkdir_test'
+        >>> adb.exists(path)
+        False
+        >>> adb.mkdir(path)
+        >>> adb.exists(path)
+        True
+
+        >>> adb.mkdir('/init')
+        Traceback (most recent call last):
+        ...
+        PwnlibException: mkdir failed for /init, File exists
+    """
+    if not path.startswith('/'):
+        log.error("Must provide an absolute path: %r" % path)
+
+    with Client() as c:
+        st = c.stat(path)
+
+        # Don't re-create existing directories
+        if st and stat.S_ISDIR(st['mode']):
+            return
+
+        result = process(['mkdir', path]).recvall()
+
+        # Any output at all is an error
+        if result:
+            log.error(result)
+
+@context.quiet
+@with_device
+def makedirs(path):
+    """Create a directory and all parent directories on the target device.
+
+    Note:
+        Silently succeeds if the directory already exists.
+
+    Examples:
+
+        >>> adb.makedirs('/data/local/tmp/this/is/a/directory/heirarchy')
+        >>> adb.listdir('/data/local/tmp/this/is/a/directory')
+        ['heirarchy']
+    """
+    if path != '/':
+        makedirs(os.path.dirname(path))
+
+    mkdir(path)
+
+@context.quiet
+@with_device
+def exists(path):
+    """Return ``True`` if ``path`` exists on the target device.
+
+    Examples:
+
+        >>> adb.exists('/')
+        True
+        >>> adb.exists('/init')
+        True
+        >>> adb.exists('/does/not/exist')
+        False
+    """
+    with Client() as c:
+        return bool(c.stat(path))
+
+@context.quiet
+@with_device
+def isdir(path):
+    """Return ``True`` if ``path`` is a on the target device.
+
+    Examples:
+
+        >>> adb.isdir('/')
+        True
+        >>> adb.isdir('/init')
+        False
+        >>> adb.isdir('/does/not/exist')
+        False
+    """
+    with Client() as c:
+        st = c.stat(path)
+        return bool(st and stat.S_ISDIR(st['mode']))
+
+@context.quiet
+@with_device
+def unlink(path, recursive=False):
+    """Unlinks a file or directory on the target device.
+
+    Examples:
+
+        >>> adb.unlink("/does/not/exist")
+        Traceback (most recent call last):
+        ...
+        PwnlibException: Could not unlink '/does/not/exist': Does not exist
+
+        >>> filename = '/data/local/tmp/unlink-test'
+        >>> adb.write(filename, 'hello')
+        >>> adb.exists(filename)
+        True
+        >>> adb.unlink(filename)
+        >>> adb.exists(filename)
+        False
+
+        >>> adb.mkdir(filename)
+        >>> adb.write(filename + '/contents', 'hello')
+        >>> adb.unlink(filename)
+        Traceback (most recent call last):
+        ...
+        PwnlibException: Cannot delete non-empty directory '/data/local/tmp/unlink-test' without recursive=True
+
+        >>> adb.unlink(filename, recursive=True)
+        >>> adb.exists(filename)
+        False
+    """
+    with Client() as c:
+        st = c.stat(path)
+        if not st:
+            log.error("Could not unlink %r: Does not exist" % path)
+
+        # If the directory is not empty, do not delete it
+        if isdir(path) and c.list(path) and not recursive:
+            log.error("Cannot delete non-empty directory %r without recursive=True" % path)
+
+        flags = '-rf' if recursive else '-r'
+
+        output = c.execute(['rm', flags, path]).recvall()
+
+        if output:
+            log.error(output)
+
 @with_device
 def process(argv, *a, **kw):
     """Execute a process on the device.
@@ -661,16 +904,15 @@ def listdir(directory='/'):
     """
     return list(sorted(Client().list(directory)))
 
+@with_device
 def fastboot(args, *a, **kw):
     """Executes a fastboot command.
 
     Returns:
         The command output.
     """
-    serial = context.device
-    if not serial:
-        log.error("Unknown device")
-    return tubes.process.process(['fastboot', '-s', serial] + list(args), **kw).recvall()
+    argv = ['fastboot', '-s', str(context.device)] + list(args)
+    return tubes.process.process(argv, *a, **kw).recvall()
 
 @with_device
 def fingerprint():
@@ -850,7 +1092,7 @@ _android_mk_template = '''
 LOCAL_PATH := $(call my-dir)
 
 include $(CLEAR_VARS)
-LOCAL_MODULE := poc
+LOCAL_MODULE := %(local_module)s
 LOCAL_SRC_FILES := %(local_src_files)s
 
 include $(BUILD_EXECUTABLE)
@@ -880,6 +1122,8 @@ def _generate_ndk_project(file_list, abi='arm-v7a', platform_version=21):
     # Create the directories
 
     # Populate Android.mk
+    local_module = os.path.basename(file_list[0])
+    local_module, _ = os.path.splitext(local_module)
     local_src_files = ' '.join(list(map(os.path.basename, file_list)))
     Android_mk = os.path.join(jni_directory, 'Android.mk')
     with open(Android_mk, 'w+') as f:
@@ -914,7 +1158,7 @@ def compile(source):
         abi = 'armeabi-v7a'
         sdk = '21'
 
-        # If we have an atatched device, use its settings.
+        # If we have an attached device, use its settings.
         if context.device:
             abi = str(properties.ro.product.cpu.abi)
             sdk = str(properties.ro.build.version.sdk)
@@ -951,12 +1195,51 @@ class Partition(object):
         with log.waitfor('Fetching %r partition (%s)' % (self.name, self.path)):
             return read(self.path)
 
+@with_device
+def walk(top, topdown=True):
+    join = os.path.join
+    isdir = lambda x: stat.S_ISDIR(x['mode'])
+    client = Client()
+    names = client.list(top)
+
+    dirs, nondirs = [], []
+    for name, metadata in names.items():
+        if isdir(metadata):
+            dirs.append(name)
+        else:
+            nondirs.append(name)
+
+    if topdown:
+        yield top, dirs, nondirs
+    for name in dirs:
+        new_path = join(top, name)
+        for x in walk(new_path, topdown):
+            yield x
+    if not topdown:
+        yield top, dirs, nondirs
+
+@with_device
+def find(top, name):
+    for root, dirs, files in walk(top):
+        if name in files or name in dirs:
+            yield os.path.join(root, name)
+
+@with_device
+def readlink(path):
+    path = process(['readlink', path]).recvall()
+
+    # Readlink will emit a single newline
+    # We can't use the '-n' flag since old versions don't support it
+    if path.endswith('\n'):
+        path = path[:-1]
+
+    return path
+
 class Partitions(object):
     @property
     @context.quiet
     def by_name_dir(self):
-        cmd = ['shell','find /dev/block/platform -type d -name by-name']
-        return adb(cmd).strip()
+        return next(find('/dev/block/platform','by-name'))
 
     @context.quiet
     def __dir__(self):
@@ -972,6 +1255,7 @@ class Partitions(object):
             yield name
 
     @context.quiet
+    @with_device
     def __getattr__(self, attr):
         for name in self:
             if name == attr:
@@ -982,7 +1266,7 @@ class Partitions(object):
         path = os.path.join(self.by_name_dir, name)
 
         # Find the actual path of the device
-        devpath = process(['readlink', '-n', path]).recvall()
+        devpath = readlink(path)
         devname = os.path.basename(devpath)
 
         # Get the size of the partition
@@ -1036,13 +1320,12 @@ def uninstall(package, *arguments):
         package(str): Name of the package to uninstall (e.g. ``'com.foo.MyPackage'``)
         arguments: Supplementary arguments to ``'pm install'``, e.g. ``'-k'``.
     """
-    with log.progress("Installing package {}".format(package)):
+    with log.progress("Uninstalling package {}".format(package)):
         with context.quiet:
-            return process(['pm','uninstall',apk] + list(arguments)).recvall()
+            return process(['pm','uninstall',package] + list(arguments)).recvall()
 
 @context.quiet
 def packages():
     """Returns a list of packages installed on the system"""
     packages = process(['pm', 'list', 'packages']).recvall()
     return [line.split('package:', 1)[-1] for line in packages.splitlines()]
-

pwnlib/adb/protocol.py

@@ -5,6 +5,7 @@ Documentation is available here:
 https://android.googlesource.com/platform/system/core/+/master/adb/protocol.txt
 """
 import functools
+import stat
 import time
 
 from ..context import context
@@ -42,14 +43,7 @@ class Message(object):
 class Connection(remote):
     """Connection to the ADB server"""
     def __init__(self, host, port, level=None, *a, **kw):
-
-        # Try to make sure ADB is running if it's on the default host and port.
-        if host == context.defaults['adb_host'] \
-        and port == context.defaults['adb_port']:
-            process(context.adb + ['start-server'], level='error').wait_for_close()
-
-        with context.quiet:
-            super(Connection, self).__init__(host, port, level=level, *a, **kw)
+        super(Connection, self).__init__(host, port, level=level, *a, **kw)
 
         self._executable = None
         self._argv       = None
@@ -90,6 +84,20 @@ class Client(Logger):
     @property
     def c(self):
         """Client's connection to the ADB server"""
+        if not self._c:
+            try:
+                self._c = Connection(self.host, self.port, level=self.level)
+            except Exception:
+                # If the connection fails, try starting a server on that port
+                # as long as it's the *default* port.
+                if self.host == context.defaults['adb_host'] \
+                and self.port == context.defaults['adb_port']:
+                    log.warn("Could not connect to ADB server, trying to start it")
+                    process(context.adb + ['start-server']).recvall()
+                else:
+                    log.exception('Could not connect to ADB server')
+
+        # Final attempt...
         if not self._c:
             self._c = Connection(self.host, self.port, level=self.level)
         return self._c
@@ -134,12 +142,23 @@ class Client(Logger):
 
     @_autoclose
     def kill(self):
-        """Kills the remote ADB server"""
+        """Kills the remote ADB server"
+
+        >>> c=adb.protocol.Client()
+        >>> c.kill()
+
+        The server is automatically re-started on the next request,
+        if the default host/port are used.
+
+        >>> c.version() > (4,0)
+        True
+        """
         try:
             self.send('host:kill')
         except EOFError:
             pass
 
+    @_autoclose
     def version(self):
         """
         Returns:
@@ -296,8 +315,6 @@ class Client(Logger):
             return rv
         return wrapper
 
-    @_with_transport
-    @_sync
     def list(self, path):
         """Execute the ``LIST`` command of the ``SYNC`` API.
 
@@ -328,8 +345,23 @@ class Client(Logger):
             >>> pprint(adb.Client().list('/data/user'))
             {'0': {'mode': 41471, 'size': 11, 'time': ...}}
             >>> adb.Client().list('/does/not/exist')
-            {}
+            Traceback (most recent call last):
+            ...
+            PwnlibException: Cannot list directory '/does/not/exist': Does not exist
         """
+        st = self.stat(path)
+
+        if not st:
+            log.error("Cannot list directory %r: Does not exist" % path)
+
+        if not stat.S_ISDIR(st['mode']):
+            log.error("Cannot list directory %r: Path is not a directory" % path)
+
+        return self._list(path)
+
+    @_with_transport
+    @_sync
+    def _list(self, path):
         self.c.flat('LIST', len(path), path)
         files = {}
         while True:
@@ -389,9 +421,35 @@ class Client(Logger):
 
         return {'mode': mode, 'size': size, 'time': time}
 
+    def write(self, path, data, mode=0o755, timestamp=None, callback=None):
+        """Execute the ``WRITE`` command of the ``SYNC`` API.
+
+        Arguments:
+            path(str): Path to the file to write
+            data(str): Data to write to the file
+            mode(int): File mode to set (e.g. ``0o755``)
+            timestamp(int): Unix timestamp to set the file date to
+            callback(callable): Callback function invoked as data
+                is written.  Arguments provided are:
+
+                - File path
+                - All data
+                - Expected size of all data
+                - Current chunk
+                - Expected size of chunk
+        """
+        # We must ensure that 'path' is not a directory
+        # Writing to a directory is supported, but creates a temporary file
+        st = self.stat(path)
+
+        if st and stat.S_ISDIR(st['mode']):
+            log.error("Cannot write to %r: Path is a directory" % path)
+
+        return self._write(path, data, mode=0o755, timestamp=None, callback=None)
+
     @_with_transport
     @_sync
-    def write(self, path, data, mode=0o755, timestamp=None, callback=None):
+    def _write(self, path, data, mode=0o755, timestamp=None, callback=None):
         path += ',' + str(mode)
         self.c.flat('SEND', len(path), path)
 

pwnlib/args.py

@@ -1,5 +1,43 @@
 #!/usr/bin/env python2
 """
+Pwntools exposes several magic command-line arguments and environment
+variables when operating in `from pwn import *` mode.
+
+The arguments extracted from the command-line and removed from ``sys.argv``.
+
+Arguments can be set by appending them to the command-line, or setting
+them in the environment prefixed by ``PWNLIB_``.
+
+The easiest example is to enable more verbose debugging.  Just set ``DEBUG``.
+
+.. code-block:: bash
+
+    $ PWNLIB_DEBUG=1 python exploit.py
+    $ python exploit.py DEBUG
+
+These arguments are automatically extracted, regardless of their name, and
+exposed via ``pwnlib.args.args``, which is exposed as the global variable
+``args``.  Arguments which ``pwntools`` reserves internally are not exposed
+this way.
+
+.. code-block:: bash
+
+    $ python -c 'from pwn import *; print args' A=1 B=Hello HOST=1.2.3.4 DEBUG
+    defaultdict(<type 'str'>, {'A': '1', 'HOST': '1.2.3.4', 'B': 'Hello'})
+
+This is very useful for conditional code, for example determining whether to
+run an exploit locally or to connect to a remote server.  Arguments which are
+not specified evaluate to an empty string.
+
+.. code-block:: python
+
+    if args['REMOTE']:
+        io = remote('exploitme.com', 4141)
+    else:
+        io = process('./pwnable')
+
+The full list of supported "magic arguments" and their effects are listed
+below.
 """
 import collections
 import logging
@@ -42,46 +80,67 @@ def asbool(s):
     else:
         raise ValueError('must be integer or boolean: %r' % s)
 
-def set_log_level(x):
+def LOG_LEVEL(x):
+    """Sets the logging verbosity used via ``context.log_level``,
+    e.g. ``LOG_LEVEL=debug``.
+    """
     with context.local(log_level=x):
         context.defaults['log_level']=context.log_level
 
-def set_log_file(x):
+def LOG_FILE(x):
+    """Sets a log file to be used via ``context.log_file``, e.g.
+    ``LOG_FILE=./log.txt``"""
     context.log_file=x
 
-def set_log_level_error(x):
-    set_log_level('error')
+def SILENT(x):
+    """Sets the logging verbosity to ``error`` which silences most
+    output."""
+    LOG_LEVEL('error')
 
-def set_log_level_debug(x):
-    set_log_level('debug')
+def DEBUG(x):
+    """Sets the logging verbosity to ``debug`` which displays much
+    more information, including logging each byte sent by tubes."""
+    LOG_LEVEL('debug')
 
-def set_noterm(v):
+def NOTERM(v):
+    """Disables pretty terminal settings and animations."""
     if asbool(v):
         global term_mode
         term_mode = False
 
-def set_timeout(v):
+def TIMEOUT(v):
+    """Sets a timeout for tube operations (in seconds) via
+    ``context.timeout``, e.g. ``TIMEOUT=30``"""
     context.defaults['timeout'] = int(v)
 
-def set_randomize(v):
+def RANDOMIZE(v):
+    """Enables randomization of various pieces via ``context.randomize``"""
     context.defaults['randomize'] = asbool(v)
 
-def set_aslr(v):
+def NOASLR(v):
+    """Disables ASLR via ``context.aslr``"""
     context.defaults['aslr'] = not asbool(v)
 
-def set_noptrace(v):
+def NOPTRACE(v):
+    """Disables facilities which require ``ptrace`` such as ``gdb.attach()``
+    statements, via ``context.noptrace``."""
     context.defaults['noptrace'] = asbool(v)
 
+def STDERR(v):
+    """Sends logging to ``stderr`` by default, instead of ``stdout``"""
+    context.log_console = sys.stderr
+
 hooks = {
-    'LOG_LEVEL': set_log_level,
-    'LOG_FILE': set_log_file,
-    'DEBUG': set_log_level_debug,
-    'NOTERM': set_noterm,
-    'SILENT': set_log_level_error,
-    'RANDOMIZE': set_randomize,
-    'TIMEOUT': set_timeout,
-    'NOASLR': set_aslr,
-    'NOPTRACE': set_noptrace,
+    'LOG_LEVEL': LOG_LEVEL,
+    'LOG_FILE': LOG_FILE,
+    'DEBUG': DEBUG,
+    'NOTERM': NOTERM,
+    'SILENT': SILENT,
+    'RANDOMIZE': RANDOMIZE,
+    'TIMEOUT': TIMEOUT,
+    'NOASLR': NOASLR,
+    'NOPTRACE': NOPTRACE,
+    'STDERR': STDERR,
 }
 
 def initialize():

pwnlib/commandline/common.py

@@ -5,8 +5,6 @@ import sys
 import pwnlib
 from pwnlib.context import context
 
-pwnlib.log.console.stream = sys.stderr
-
 choices = map(str, [16,32,64])
 choices += list(context.oses)
 choices += list(context.architectures)

pwnlib/commandline/main.py

@@ -1,3 +1,5 @@
+import sys
+
 from . import asm
 from . import checksec
 from . import common
@@ -14,6 +16,7 @@ from . import scramble
 from . import shellcraft
 from . import unhex
 from . import update
+from ..context import context
 from .common import parser
 
 commands = {
@@ -36,7 +39,8 @@ commands = {
 
 def main():
     args = parser.parse_args()
-    commands[args.command](args)
+    with context.local(log_console = sys.stderr):
+        commands[args.command](args)
 
 if __name__ == '__main__':
     main()

pwnlib/commandline/shellcraft.py

@@ -57,9 +57,10 @@ p.add_argument(
                'p',
                'i', 'hexii',
                'e', 'elf',
+               'd', 'escaped',
                'default'],
     default = 'default',
-    help = 'Output format (default: hex), choose from {r}aw, {s}tring, {c}-style array, {h}ex string, hex{i}i, {a}ssembly code, {p}reprocssed code',
+    help = 'Output format (default: hex), choose from {e}lf, {r}aw, {s}tring, {c}-style array, {h}ex string, hex{i}i, {a}ssembly code, {p}reprocssed code, escape{d} hex string',
 )
 
 p.add_argument(
@@ -329,7 +330,8 @@ def main(args):
         code = pwnlib.util.fiddling.enhex(code) + '\n'
     elif args.format in ['i', 'hexii']:
         code = hexii(code) + '\n'
-
+    elif args.format in ['d', 'escaped']:
+        code = ''.join('\\x%02x' % ord(c) for c in code) + '\n'
     if not sys.stdin.isatty():
         args.out.write(sys.stdin.read())
 

pwnlib/commandline/update.py

@@ -21,7 +21,7 @@ p.add_argument('--pre', action='store_true', help='''
 def main(a):
     result = pwnlib.update.perform_check(prerelease=a.pre)
     if a.install:
-        subprocess.check_call(result, shell=True)
+        subprocess.check_call(result, shell=False)
 
 if __name__ == '__main__':
     pwnlib.common.main(__file__)

pwnlib/context/__init__.py

@@ -10,6 +10,7 @@ import logging
 import os
 import platform
 import socket
+import stat
 import string
 import subprocess
 import sys
@@ -329,11 +330,13 @@ class ContextType(object):
         'aslr': True,
         'binary': None,
         'bits': 32,
+        'buffer_size': 4096,
         'device': os.getenv('ANDROID_SERIAL', None) or None,
         'endian': 'little',
         'kernel': None,
         'log_level': logging.INFO,
         'log_file': _devnull(),
+        'log_console': sys.stdout,
         'randomize': False,
         'newline': '\n',
         'noptrace': False,
@@ -922,6 +925,24 @@ class ContextType(object):
         value.flush()
         return value
 
+    @_validator
+    def log_console(self, stream):
+        """
+        Sets the default logging console target.
+
+        Examples:
+
+            >>> context.log_level = 'warn'
+            >>> log.warn("Hello")
+            [!] Hello
+            >>> context.log_console=open('/dev/null', 'w')
+            >>> log.warn("Hello")
+            >>> context.clear()
+        """
+        if isinstance(stream, str):
+            stream = open(stream, 'wt')
+        return stream
+
     @property
     def mask(self):
         return (1 << self.bits) - 1
@@ -1107,8 +1128,13 @@ class ContextType(object):
 
     @property
     def adb(self):
-        """Returns an argument array for connecting to adb."""
-        command = ['adb']
+        """Returns an argument array for connecting to adb.
+
+        Unless ``$ADB_PATH`` is set, uses the default ``adb`` binary in ``$PATH``.
+        """
+        ADB_PATH = os.environ.get('ADB_PATH', 'adb')
+
+        command = [ADB_PATH]
 
         if self.adb_host != self.defaults['adb_host']:
             command += ['-H', self.adb_host]
@@ -1121,6 +1147,53 @@ class ContextType(object):
 
         return command
 
+    @_validator
+    def buffer_size(self, size):
+        """Internal buffer size to use for ``tube`` objects.
+
+        This is not the maximum size of the buffer, but this is the amount of data
+        which is passed to each raw ``read`` syscall (or equivalent).
+        """
+        return int(size)
+
+    @property
+    def cache_dir(self):
+        """Directory used for caching data.
+
+        Note:
+            May be either a path string, or ``None``.
+
+        Example:
+
+            >>> cache_dir = context.cache_dir
+            >>> cache_dir is not None
+            True
+            >>> os.chmod(cache_dir, 0o000)
+            >>> context.cache_dir is None
+            True
+            >>> os.chmod(cache_dir, 0o755)
+            >>> cache_dir == context.cache_dir
+            True
+        """
+        home = os.path.expanduser('~')
+
+        if not os.access(home, os.W_OK):
+            return None
+
+        cache = os.path.join(home, '.pwntools-cache')
+
+        if not os.path.exists(cache):
+            try:
+                os.mkdir(cache)
+            except OSError:
+                return None
+
+        # Some wargames e.g. pwnable.kr have created dummy directories
+        # which cannot be modified by the user account (owned by root).
+        if not os.access(cache, os.W_OK):
+            return None
+
+        return cache
 
     #*************************************************************************
     #                               ALIASES

pwnlib/elf/corefile.py

@@ -6,6 +6,7 @@ from elftools.common.py3compat import bytes2str
 from elftools.common.utils import roundup
 from elftools.common.utils import struct_parse
 from elftools.construct import CString
+
 from ..context import context
 from ..log import getLogger
 from ..tubes.tube import tube
@@ -235,7 +236,7 @@ class Core(ELF):
     def libc(self):
         """Return the first mapping in libc"""
         for m in self.mappings:
-            if m.name.startswith('libc') and m.name.endswith('.so'):
+            if m.name and m.name.startswith('libc') and m.name.endswith('.so'):
                 return m
 
     @property

pwnlib/elf/datatypes.py

@@ -539,7 +539,8 @@ class elf_prstatus_i386(ctypes.Structure):
 assert ctypes.sizeof(elf_prstatus_i386) == 0x90
 
 class elf_prstatus_amd64(ctypes.Structure):
-    _fields_ = generate_prstatus_common(64, user_regs_struct_amd64)
+    _fields_ = generate_prstatus_common(64, user_regs_struct_amd64) \
+             + [('padding', ctypes.c_uint32)]
 
 assert ctypes.sizeof(elf_prstatus_amd64) == 0x150
 

pwnlib/elf/elf.py

@@ -1,5 +1,6 @@
 """Exposes functionality for manipulating ELF files
 """
+import codecs
 import mmap
 import os
 import subprocess
@@ -30,7 +31,18 @@ log = getLogger(__name__)
 
 __all__ = ['load', 'ELF']
 
-Function = namedtuple('Function', 'address size')
+class Function(object):
+    def __init__(self, name, address, size):
+        self.name    = name
+        self.address = address
+        self.size    = size
+    def __repr__(self):
+        return '%s(name=%r, address=%#x, size=%#x)' % (
+            self.__class__.__name__,
+            self.name,
+            self.address,
+            self.size
+            )
 
 def load(*args, **kwargs):
     """Compatibility wrapper for pwntools v1"""
@@ -348,11 +360,15 @@ class ELF(ELFFile):
                     continue
                 if sym.entry.st_info['type'] == 'STT_FUNC' and sym.entry.st_size != 0:
                     name = sym.name
+                    try:
+                        name = codecs.encode(name, 'latin-1')
+                    except Exception:
+                        pass
                     if name not in self.symbols:
                         continue
                     addr = self.symbols[name]
                     size = sym.entry.st_size
-                    self.functions[name] = Function(addr, size)
+                    self.functions[name] = Function(name, addr, size)
 
     def _populate_symbols(self):
         """

pwnlib/flag/flag.py

@@ -3,8 +3,8 @@
 import os
 
 from ..args import args
-from ..tubes.remote import remote
 from ..log import getLogger
+from ..tubes.remote import remote
 
 env_server  = args.get('FLAG_HOST', 'flag-submission-server').strip()
 env_port    = args.get('FLAG_PORT', '31337').strip()

pwnlib/gdb.py

@@ -226,7 +226,7 @@ def debug(args, execute=None, exe=None, ssh=None, env=None, **kwargs):
     if not ssh and context.os == 'android':
         host = context.adb_host
 
-    attach((host, port), exe=exe, execute=execute, need_ptrace_scope = False)
+    attach((host, port), exe=exe, execute=execute, need_ptrace_scope = False, ssh=ssh)
 
     # gdbserver outputs a message when a client connects
     garbage = gdbserver.recvline(timeout=1)
@@ -247,8 +247,8 @@ def get_gdb_arch():
 
 
 @LocalContext
-def attach(target, execute = None, exe = None, need_ptrace_scope = True):
-    """attach(target, execute = None, exe = None, arch = None) -> None
+def attach(target, execute = None, exe = None, need_ptrace_scope = True, ssh = None):
+    """attach(target, execute = None, exe = None, arch = None, ssh = None) -> None
 
     Start GDB in a new terminal and attach to `target`.
     :func:`pwnlib.util.proc.pidof` is used to find the PID of `target` except
@@ -401,8 +401,11 @@ def attach(target, execute = None, exe = None, need_ptrace_scope = True):
     cmd += ' -q '
 
     if exe and context.native:
+        if ssh:
+            ssh.download_file(exe)
+            exe = os.path.basename(exe)
         if not os.path.isfile(exe):
-            log.error('no such file: %s' % exe)
+            log.error('No such file: %s' % exe)
         cmd += ' "%s"' % exe
 
     if pid and not context.os == 'android':
@@ -460,7 +463,7 @@ def ssh_gdb(ssh, process, execute = None, arch = None, **kwargs):
     l = tubes.listen.listen(0)
     forwardport = l.lport
 
-    attach(('127.0.0.1', forwardport), execute, local_exe, arch)
+    attach(('127.0.0.1', forwardport), execute, local_exe, arch, ssh=ssh)
     l.wait_for_connection() <> ssh.connect_remote('127.0.0.1', gdbport)
     return c
 

pwnlib/log.py

@@ -113,7 +113,7 @@ from .term import spinners
 from .term import text
 
 # list of prefixes to use for the different message types.  note that the `text`
-# module won't add any escape codes if `sys.stderr.isatty()` is `False`
+# module won't add any escape codes if `pwnlib.context.log_console.isatty()` is `False`
 _msgtype_prefixes = {
     'status'       : [text.magenta, 'x'],
     'success'      : [text.bold_green, '+'],
@@ -166,7 +166,6 @@ class Progress(object):
     using :meth:`Logger.progress`.
     """
     def __init__(self, logger, msg, status, level, args, kwargs):
-        global _progressid
         self._logger = logger
         self._msg = msg
         self._status = status
@@ -264,7 +263,7 @@ class Logger(object):
             # This is a minor hack to permit user-defined classes which inherit
             # from a tube (which do not actually reside in the pwnlib library)
             # to receive logging abilities that behave as they would expect from
-            # the resto f the library
+            # the rest of the library
             module = self.__module__
             if not module.startswith('pwnlib'):
                 module = 'pwnlib.' + module
@@ -490,10 +489,14 @@ class Handler(logging.StreamHandler):
     logger will not be emitted but rather an animated progress line will be
     created.
 
-    This handler outputs to ``sys.stderr``.
-
     An instance of this handler is added to the ``'pwnlib'`` logger.
     """
+    @property
+    def stream(self):
+        return context.log_console
+    @stream.setter
+    def stream(self, value):
+        pass
     def emit(self, record):
         """
         Emit a log record or create/update an animated progress logger
@@ -652,7 +655,7 @@ log_file.setFormatter(logging.Formatter(fmt, iso_8601))
 #     logger.addHandler(myCoolPitchingHandler)
 #
 rootlogger = getLogger('pwnlib')
-console   = Handler(sys.stdout)
+console   = Handler()
 formatter = Formatter()
 console.setFormatter(formatter)
 
@@ -663,8 +666,6 @@ def install_default_handler():
     the ``pwnlib`` root logger.  This function is automatically called from when
     importing :mod:`pwn`.
     '''
-    console.stream = sys.stdout
-
     logger         = logging.getLogger('pwnlib')
 
     if console not in logger.handlers:

pwnlib/memleak.py

@@ -129,6 +129,8 @@ class MemLeak(object):
         size   = obj.size
         offset = obj.offset
         data   = self.n(address + offset, size)
+        if not data:
+            return None
         return unpack(data, size*8)
 
     def field_compare(self, address, obj, expected):
@@ -604,3 +606,14 @@ class MemLeak(object):
             return result
 
         return MemLeak(string_wrapper)
+
+    # Aliases for convenience
+    u64 = q
+    u32 = d
+    u16 = w
+    u8 = b
+
+    p64 = setq
+    p32 = setd
+    p16 = setw
+    p8 = setb

pwnlib/regsort.py

@@ -224,6 +224,8 @@ def regsort(in_out, all_regs, tmp = None, xchg = True, randomize = None):
          ('mov', 'b', 'c'),
          ('mov', 'c', 'x'),
          ('mov', 'x', '1')]
+         >>> regsort({'a': 'b', 'b': 'c'}, ['a','b','c'], xchg=0)
+         [('mov', 'a', 'b'), ('mov', 'b', 'c')]
     """
     if randomize is None:
         randomize = context.randomize
@@ -333,7 +335,9 @@ def regsort(in_out, all_regs, tmp = None, xchg = True, randomize = None):
     # If XCHG is expressly disabled, and there is no temporary register,
     # try to see if there is any register which can be used as a temp
     # register instead.
-    if not (xchg or tmp):
+    #
+    # If there are no cycles, there's no need for a temporary register.
+    if in_cycle and not (xchg or tmp):
         for reg in in_out:
             if not depends_on_cycle(reg, in_out, in_cycle):
                 tmp = reg

pwnlib/rop/rop.py

@@ -478,7 +478,9 @@ class ROP(object):
         """
         Generates padding to be inserted into the ROP stack.
         """
-        return cyclic.cyclic(offset + count)[-count:]
+        if count:
+            return cyclic.cyclic(offset + count)[-count:]
+        return ''
 
     def describe(self, object):
         """
@@ -700,6 +702,9 @@ class ROP(object):
         # If we can find a function with that name, just call it
         if isinstance(resolvable, str):
             addr = self.resolve(resolvable)
+        elif hasattr(resolvable, 'name') and hasattr(resolvable, 'address'):
+            addr = resolvable.address
+            resolvable = str(resolvable.name)
         else:
             addr = resolvable
             resolvable = ''

pwnlib/shellcraft/internal.py

@@ -1,6 +1,8 @@
 import os
 from collections import defaultdict
 
+from ..context import context
+
 __all__ = ['make_function']
 
 loaded = {}
@@ -33,10 +35,14 @@ def init_mako():
 
     render_global = IsInside()
 
+    cache  = context.cache_dir
+    if cache:
+        cache = os.path.join(cache, 'mako')
+
     curdir = os.path.dirname(os.path.abspath(__file__))
     lookup = TemplateLookup(
         directories      = [os.path.join(curdir, 'templates')],
-        module_directory = os.path.expanduser('~/.pwntools-cache/mako')
+        module_directory = cache
     )
 
     # The purpose of this definition is to create a new Tag.

pwnlib/shellcraft/templates/amd64/mov.asm

@@ -61,17 +61,26 @@ Example:
         mov rax, 0x1010110dfac01fe
         xor [rsp], rax
         pop rax
-   >>> with context.local(os = 'linux'):
-   ...     print shellcraft.amd64.mov('eax', 'SYS_read').rstrip()
-       xor eax, eax /* (SYS_read) */
-   >>> with context.local(os = 'freebsd'):
-   ...     print shellcraft.amd64.mov('eax', 'SYS_read').rstrip()
-       push (SYS_read) /* 3 */
-       pop rax
-   >>> with context.local(os = 'linux'):
-   ...     print shellcraft.amd64.mov('eax', 'PROT_READ | PROT_WRITE | PROT_EXEC').rstrip()
-       push (PROT_READ | PROT_WRITE | PROT_EXEC) /* 7 */
-       pop rax
+    >>> print shellcraft.amd64.mov('rax', 0xffffffff).rstrip()
+        mov eax, 0xffffffff
+    >>> print shellcraft.amd64.mov('rax', 0x7fffffff).rstrip()
+        mov eax, 0x7fffffff
+    >>> print shellcraft.amd64.mov('rax', 0x80010101).rstrip()
+        mov eax, 0x80010101
+    >>> print shellcraft.amd64.mov('rax', 0x80000000).rstrip()
+        mov eax, 0x1010101 /* 2147483648 == 0x80000000 */
+        xor eax, 0x81010101
+    >>> with context.local(os = 'linux'):
+    ...     print shellcraft.amd64.mov('eax', 'SYS_read').rstrip()
+        xor eax, eax /* (SYS_read) */
+    >>> with context.local(os = 'freebsd'):
+    ...     print shellcraft.amd64.mov('eax', 'SYS_read').rstrip()
+        push (SYS_read) /* 3 */
+        pop rax
+    >>> with context.local(os = 'linux'):
+    ...     print shellcraft.amd64.mov('eax', 'PROT_READ | PROT_WRITE | PROT_EXEC').rstrip()
+        push (PROT_READ | PROT_WRITE | PROT_EXEC) /* 7 */
+        pop rax
 
 Args:
   dest (str): The destination register.
@@ -99,7 +108,6 @@ if get_register(src):
     if dest.size == 64 and src.size <= 32:
         dest = get_register(dest.native32)
 
-    src_size = src.size
 else:
     with ctx.local(arch = 'amd64'):
         src = eval(src)
@@ -107,9 +115,9 @@ else:
     if not dest.fits(src):
         log.error("cannot mov %s, %r: dest is smaller than src" % (dest, src))
 
-    src_size = bits_required(src)
+    orig_dest = dest
 
-    if dest.size == 64 and src_size <= 32:
+    if dest.size == 64 and bits_required(src) <= 32:
         dest = get_register(dest.native32)
 
     # Calculate the packed version
@@ -117,7 +125,15 @@ else:
 
     # Calculate the unsigned and signed versions
     srcu = packing.unpack(srcp, dest.size, sign=False)
-    srcs = packing.unpack(srcp, dest.size, sign=True)
+
+    # N.B.: We may have downsized the register for e.g. mov('rax', 0xffffffff)
+    #       In this case, srcp is now a 4-byte packed value, which will expand
+    #       to "-1", which isn't correct.
+    if orig_dest.size != dest.size:
+        srcs = src
+    else:
+        srcs = packing.unpack(srcp, dest.size, sign=True)
+
 %>\
 % if is_register(src):
     % if src == dest:

pwnlib/shellcraft/templates/amd64/popad.asm

@@ -6,8 +6,8 @@ in the same order.
     pop rdi
     pop rsi
     pop rbp
-    pop rsp
-    pop rbp
+    pop rbx /* add rsp, 8 */
+    pop rbx
     pop rdx
     pop rcx
     pop rax

pwnlib/term/term.py

@@ -22,6 +22,7 @@ import threading
 import traceback
 
 from . import termcap
+from ..context import ContextType
 
 settings = None
 _graphics_mode = False
@@ -127,6 +128,11 @@ def init():
         sys.stdout = Wrapper(sys.stdout)
     if sys.stderr.isatty():
         sys.stderr = Wrapper(sys.stderr)
+
+    console = ContextType.defaults['log_console']
+    if console.isatty():
+        ContextType.defaults['log_console'] = Wrapper(console)
+
     # freeze all cells if an exception is thrown
     orig_hook = sys.excepthook
     def hook(*args):

pwnlib/tubes/buffer.py

@@ -1,5 +1,8 @@
 #!/usr/bin/env python2
 
+from ..context import context
+
+
 class Buffer(Exception):
     """
     List of strings with some helper routines.
@@ -28,10 +31,10 @@ class Buffer(Exception):
         The ``0th`` item in the buffer is the oldest item, and
         will be received first.
     """
-    def __init__(self):
+    def __init__(self, buffer_fill_size = None):
         self.data = [] # Buffer
         self.size = 0  # Length
-
+        self.buffer_fill_size = buffer_fill_size
 
     def __len__(self):
         """
@@ -169,3 +172,19 @@ class Buffer(Exception):
         self.size -= len(data)
 
         return data
+
+    def get_fill_size(self, size=None):
+        """
+        Retrieves the default fill size for this buffer class.
+
+        Arguments:
+            size (int): (Optional) If set and not None, returns the size variable back.
+
+        Returns:
+            Fill size as integer if size == None, else size.
+        """
+        if size is None:
+            size = self.buffer_fill_size
+
+        with context.local(buffer_size=size):
+            return context.buffer_size

pwnlib/tubes/listen.py

@@ -20,13 +20,11 @@ class listen(sock):
         bindaddr(str): The address to bind to.
         fam: The string "any", "ipv4" or "ipv6" or an integer to pass to :func:`socket.getaddrinfo`.
         typ: The string "tcp" or "udp" or an integer to pass to :func:`socket.getaddrinfo`.
-        timeout: A positive number, None
     """
 
     def __init__(self, port=0, bindaddr = "0.0.0.0",
-                 fam = "any", typ = "tcp",
-                 timeout = Timeout.default, level = None):
-        super(listen, self).__init__(timeout, level = level)
+                 fam = "any", typ = "tcp", *args, **kwargs):
+        super(listen, self).__init__(*args, **kwargs)
 
         port = int(port)
         fam  = {socket.AF_INET: 'ipv4',

pwnlib/tubes/process.py

@@ -3,6 +3,7 @@ import errno
 import fcntl
 import logging
 import os
+import platform
 import pty
 import resource
 import select
@@ -43,8 +44,6 @@ class process(tube):
             Working directory.  Uses the current working directory by default.
         env(dict):
             Environment variables.  By default, inherits from Python's environment.
-        timeout(int):
-            Timeout to use on ``tube`` ``recv`` operations.
         stdin(int):
             File object or file descriptor number to use for ``stdin``.
             By default, a pipe is used.  A pty can be used instead by setting
@@ -190,6 +189,9 @@ class process(tube):
         >>> io = process(['sh','-c','sleep 10; exit 7'], alarm=2)
         >>> io.poll(block=True) == -signal.SIGALRM
         True
+
+        >>> binary = ELF.from_assembly('nop', arch='mips')
+        >>> p = process(binary.path)
     """
 
     PTY = PTY
@@ -197,16 +199,14 @@ class process(tube):
     #: Have we seen the process stop?
     _stop_noticed = False
 
-    def __init__(self, argv,
+    def __init__(self, argv = None,
                  shell = False,
                  executable = None,
                  cwd = None,
                  env = None,
-                 timeout = Timeout.default,
                  stdin  = PIPE,
                  stdout = PTY,
                  stderr = STDOUT,
-                 level = None,
                  close_fds = True,
                  preexec_fn = lambda: None,
                  raw = True,
@@ -214,8 +214,19 @@ class process(tube):
                  setuid = None,
                  where = 'local',
                  display = None,
-                 alarm = None):
-        super(process, self).__init__(timeout, level = level)
+                 alarm = None,
+                 *args,
+                 **kwargs
+                 ):
+        super(process, self).__init__(*args,**kwargs)
+
+        # Permit using context.binary
+        if argv is None:
+            if context.binary:
+                argv = [context.binary.path]
+            else:
+                raise TypeError('Must provide argv or set context.binary')
+
 
         #: `subprocess.Popen` object
         self.proc = None
@@ -276,7 +287,7 @@ class process(tube):
         with self.progress(message) as p:
 
             if not self.aslr:
-                log.warn_once("ASLR is disabled!")
+                self.warn_once("ASLR is disabled!")
 
             # In the event the binary is a foreign architecture,
             # and binfmt is not installed (e.g. when running on
@@ -286,12 +297,6 @@ class process(tube):
             executables = [executable]
             exception = None
 
-            try:
-                if not context.native:
-                    qemu = get_qemu_user()
-                    prefixes.append(([qemu], qemu))
-            except: pass
-
             for prefix, executable in prefixes:
                 try:
                     self.proc = subprocess.Popen(args = prefix + argv,
@@ -308,11 +313,7 @@ class process(tube):
                 except OSError as exception:
                     if exception.errno != errno.ENOEXEC:
                         raise
-            else:
-                try:
-                    raise exception
-                except:
-                    log.exception(str(prefixes))
+                    prefixes.append(self.__on_enoexec(exception))
 
         if self.pty is not None:
             if stdin is slave:
@@ -349,7 +350,7 @@ class process(tube):
 
                 resource.setrlimit(resource.RLIMIT_STACK, (-1, -1))
             except:
-                log.exception("Could not disable ASLR")
+                self.exception("Could not disable ASLR")
 
         # Assume that the user would prefer to have core dumps.
         resource.setrlimit(resource.RLIMIT_CORE, (-1, -1))
@@ -373,13 +374,43 @@ class process(tube):
 
         self.preexec_fn()
 
+    def __on_enoexec(self, exception):
+        """We received an 'exec format' error (ENOEXEC)
+
+        This implies that the user tried to execute e.g.
+        an ARM binary on a non-ARM system, and does not have
+        binfmt helpers installed for QEMU.
+        """
+        # Get the ELF binary for the target executable
+        with context.quiet:
+            # XXX: Cyclic imports :(
+            from ..elf import ELF
+            binary = ELF(self.executable)
+
+        # If we're on macOS, this will never work.  Bail now.
+        # if platform.mac_ver()[0]:
+            # self.error("Cannot run ELF binaries on macOS")
+
+        # Determine what architecture the binary is, and find the
+        # appropriate qemu binary to run it.
+        qemu = get_qemu_user(arch=binary.arch)
+        if qemu:
+            args = [qemu]
+            if self.argv:
+                args += ['-0', self.argv[0]]
+            args += ['--']
+            return [args, qemu]
+
+        # If we get here, we couldn't run the binary directly, and
+        # we don't have a qemu which can run it.
+        self.exception(exception)
+
     @property
     def program(self):
         """Alias for ``executable``, for backward compatibility"""
         return self.executable
 
-    @staticmethod
-    def _validate(cwd, executable, argv, env):
+    def _validate(self, cwd, executable, argv, env):
         """
         Perform extended validation on the executable path, argv, and envp.
 
@@ -398,14 +429,14 @@ class process(tube):
             argv = [argv]
 
         if not all(isinstance(arg, (str, unicode)) for arg in argv):
-            log.error("argv must be strings: %r" % argv)
+            self.error("argv must be strings: %r" % argv)
 
         # Create a duplicate so we can modify it
         argv = list(argv or [])
 
         for i, arg in enumerate(argv):
             if '\x00' in arg[:-1]:
-                log.error('Inappropriate nulls in argv[%i]: %r' % (i, arg))
+                self.error('Inappropriate nulls in argv[%i]: %r' % (i, arg))
 
             argv[i] = arg.rstrip('\x00')
 
@@ -417,7 +448,7 @@ class process(tube):
         #
         if not executable:
             if not argv:
-                log.error("Must specify argv or executable")
+                self.error("Must specify argv or executable")
             executable = argv[0]
 
         # Do not change absolute paths to binaries
@@ -437,11 +468,11 @@ class process(tube):
             executable = os.path.join(cwd, executable)
 
         if not os.path.exists(executable):
-            log.error("%r does not exist"  % executable)
+            self.error("%r does not exist"  % executable)
         if not os.path.isfile(executable):
-            log.error("%r is not a file" % executable)
+            self.error("%r is not a file" % executable)
         if not os.access(executable, os.X_OK):
-            log.error("%r is not marked as executable (+x)" % executable)
+            self.error("%r is not marked as executable (+x)" % executable)
 
         #
         # Validate environment
@@ -455,13 +486,13 @@ class process(tube):
 
         for k,v in env.items():
             if not isinstance(k, (str, unicode)):
-                log.error('Environment keys must be strings: %r' % k)
+                self.error('Environment keys must be strings: %r' % k)
             if not isinstance(k, (str, unicode)):
-                log.error('Environment values must be strings: %r=%r' % (k,v))
+                self.error('Environment values must be strings: %r=%r' % (k,v))
             if '\x00' in k[:-1]:
-                log.error('Inappropriate nulls in env key: %r' % (k))
+                self.error('Inappropriate nulls in env key: %r' % (k))
             if '\x00' in v[:-1]:
-                log.error('Inappropriate nulls in env value: %r=%r' % (k, v))
+                self.error('Inappropriate nulls in env value: %r=%r' % (k, v))
 
             env[k.rstrip('\x00')] = v.rstrip('\x00')
 
@@ -762,7 +793,7 @@ class process(tube):
         import pwnlib.elf.corefile
         return pwnlib.elf.corefile.Core(filename)
 
-    def leak(self, address, count=0):
+    def leak(self, address, count=1):
         """Leaks memory within the process at the specified address.
 
         Arguments:
@@ -771,7 +802,7 @@ class process(tube):
         """
         # If it's running under qemu-user, don't leak anything.
         if 'qemu-' in os.path.realpath('/proc/%i/exe' % self.pid):
-            log.error("Cannot use leaker on binaries under QEMU.")
+            self.error("Cannot use leaker on binaries under QEMU.")
 
         with open('/proc/%i/mem' % self.pid, 'rb') as mem:
             mem.seek(address)

pwnlib/tubes/remote.py

@@ -45,8 +45,8 @@ class remote(sock):
 
     def __init__(self, host, port,
                  fam = "any", typ = "tcp",
-                 timeout = Timeout.default, ssl=False, sock=None, level = None):
-        super(remote, self).__init__(timeout, level = level)
+                 ssl=False, sock=None, *args, **kwargs):
+        super(remote, self).__init__(*args, **kwargs)
 
         self.rport  = int(port)
         self.rhost  = host
@@ -108,38 +108,30 @@ class remote(sock):
         sock    = None
         timeout = self.timeout
 
-        h = self.waitfor('Opening connection to %s on port %d' % (self.rhost, self.rport))
+        with self.waitfor('Opening connection to %s on port %d' % (self.rhost, self.rport)) as h:
+            for res in socket.getaddrinfo(self.rhost, self.rport, fam, typ, 0, socket.AI_PASSIVE):
+                self.family, self.type, self.proto, _canonname, sockaddr = res
 
-        for res in socket.getaddrinfo(self.rhost, self.rport, fam, typ, 0, socket.AI_PASSIVE):
-            self.family, self.type, self.proto, _canonname, sockaddr = res
+                if self.type not in [socket.SOCK_STREAM, socket.SOCK_DGRAM]:
+                    continue
 
-            if self.type not in [socket.SOCK_STREAM, socket.SOCK_DGRAM]:
-                continue
+                h.status("Trying %s" % sockaddr[0])
 
-            h.status("Trying %s" % sockaddr[0])
+                sock = socket.socket(self.family, self.type, self.proto)
 
-            sock = socket.socket(self.family, self.type, self.proto)
+                if timeout != None and timeout <= 0:
+                    sock.setblocking(0)
+                else:
+                    sock.setblocking(1)
+                    sock.settimeout(timeout)
 
-            if timeout != None and timeout <= 0:
-                sock.setblocking(0)
-            else:
-                sock.setblocking(1)
-                sock.settimeout(timeout)
-
-            try:
-                sock.connect(sockaddr)
-                break
-            except socket.error:
-                pass
-        else:
-            h.failure()
+                try:
+                    sock.connect(sockaddr)
+                    return sock
+                except socket.error:
+                    pass
             self.error("Could not connect to %s on port %d" % (self.rhost, self.rport))
 
-        h.success()
-        return sock
-
-
-
     @classmethod
     def fromsocket(cls, socket):
         """
@@ -158,17 +150,13 @@ class remote(sock):
 
 class tcp(remote):
     __doc__ = remote.__doc__
-    def __init__(self, host, port,
-                 fam = "any", typ = "tcp",
-                 timeout = Timeout.default, ssl=False, sock=None, level = None):
-        return super(tcp, self).__init__(host, port, fam, typ, timeout, ssl, sock, level)
+    def __init__(self, host, port, *a, **kw):
+        return super(tcp, self).__init__(host, port, typ="tcp", *a, **kw)
 
 class udp(remote):
     __doc__ = remote.__doc__
-    def __init__(self, host, port,
-                 fam = "any", typ = "udp",
-                 timeout = Timeout.default, ssl=False, sock=None, level = None):
-        return super(udp, self).__init__(host, port, fam, typ, timeout, ssl, sock, level)
+    def __init__(self, host, port, *a, **kw):
+        return super(udp, self).__init__(host, port, typ="udp", *a, **kw)
 
 class connect(remote):
     __doc__ = remote.__doc__

pwnlib/tubes/serialtube.py

@@ -9,7 +9,6 @@ from . import tube
 from .. import context
 from .. import term
 from ..log import getLogger
-from ..timeout import Timeout
 
 log = getLogger(__name__)
 
@@ -18,10 +17,8 @@ class serialtube(tube.tube):
             self, port = None, baudrate = 115200,
             convert_newlines = True,
             bytesize = 8, parity='N', stopbits=1, xonxoff = False,
-            rtscts = False, dsrdtr = False,
-            timeout = Timeout.default,
-            level = None):
-        super(serialtube, self).__init__(timeout, level = level)
+            rtscts = False, dsrdtr = False, *a, **kw):
+        super(serialtube, self).__init__(*a, **kw)
 
         if port is None:
             if platform.system() == 'Darwin':
@@ -49,21 +46,14 @@ class serialtube(tube.tube):
         if not self.conn:
             raise EOFError
 
-        if self.timeout == None:
-            end = float('inf')
-        else:
-            end = time.time() + self.timeout
+        with self.countdown():
+            while self.conn and self.countdown_active():
+                data = self.conn.read(numb)
 
-        while self.conn:
-            data = self.conn.read(numb)
-            if data:
-                return data
+                if data:
+                    return data
 
-            delta = end - time.time()
-            if delta <= 0:
-                break
-            else:
-                time.sleep(min(delta, 0.1))
+                time.sleep(min(self.timeout, 0.1))
 
         return None
 
@@ -83,10 +73,11 @@ class serialtube(tube.tube):
         pass
 
     def can_recv_raw(self, timeout):
-        end = time.time()
-        while time.time() < end:
-            if self.conn.inWaiting():
-                return True
+        with self.countdown(timeout):
+            while self.conn and self.countdown_active():
+                if self.conn.inWaiting():
+                    return True
+                time.sleep(min(self.timeout, 0.1))
         return False
 
     def connected_raw(self, direction):
@@ -99,7 +90,7 @@ class serialtube(tube.tube):
 
     def fileno(self):
         if not self.connected():
-            self.error("A stopped program does not have a file number")
+            self.error("A closed serialtube does not have a file number")
 
         return self.conn.fileno()
 

pwnlib/tubes/sock.py

@@ -10,8 +10,8 @@ log = getLogger(__name__)
 class sock(tube):
     """Methods available exclusively to sockets."""
 
-    def __init__(self, timeout, level = None):
-        super(sock, self).__init__(timeout, level = level)
+    def __init__(self, *args, **kwargs):
+        super(sock, self).__init__(*args, **kwargs)
         self.closed = {"recv": False, "send": False}
 
     # Overwritten for better usability
@@ -26,13 +26,13 @@ class sock(tube):
         else:
             return super(sock, self).recvall(timeout)
 
-    def recv_raw(self, numb):
+    def recv_raw(self, numb, *a):
         if self.closed["recv"]:
             raise EOFError
 
         while True:
             try:
-                data = self.sock.recv(numb)
+                data = self.sock.recv(numb, *a)
                 break
             except socket.timeout:
                 return None
@@ -72,21 +72,89 @@ class sock(tube):
             self.sock.settimeout(timeout)
 
     def can_recv_raw(self, timeout):
+        """
+        Tests:
+
+            >>> l = listen()
+            >>> r = remote('localhost', l.lport)
+            >>> r.can_recv_raw(timeout=0)
+            False
+            >>> l.send('a')
+            >>> r.can_recv_raw(timeout=1)
+            True
+            >>> r.recv()
+            'a'
+            >>> r.can_recv_raw(timeout=0)
+            False
+            >>> l.close()
+            >>> r.can_recv_raw(timeout=1)
+            False
+            >>> r.closed['recv']
+            True
+        """
         if not self.sock or self.closed["recv"]:
             return False
 
-        return select.select([self.sock], [], [], timeout) == ([self.sock], [], [])
+        # select() will tell us data is available at EOF
+        can_recv = select.select([self.sock], [], [], timeout) == ([self.sock], [], [])
+
+        if not can_recv:
+            return False
+
+        # Ensure there's actually data, not just EOF
+        try:
+            self.recv_raw(1, socket.MSG_PEEK)
+        except EOFError:
+            return False
+
+        return True
 
     def connected_raw(self, direction):
+        """
+        Tests:
+
+            >>> l = listen()
+            >>> r = remote('localhost', l.lport)
+            >>> r.connected()
+            True
+            >>> l.close()
+            >>> time.sleep(1) # Avoid race condition
+            >>> r.connected()
+            False
+        """
+        # If there's no socket, it's definitely closed
         if not self.sock:
             return False
 
-        if direction == 'any':
-            return True
-        elif direction == 'recv':
-            return not self.closed['recv']
-        elif direction == 'send':
-            return not self.closed['send']
+        # If we have noticed a connection close in a given direction before,
+        # return fast.
+        if self.closed.get(direction, False):
+            return False
+
+        # If a connection is closed in all manners, return fast
+        if all(self.closed.values()):
+            return False
+
+        # Use poll() to determine the connection state
+        want = {
+            'recv': select.POLLIN,
+            'send': select.POLLOUT,
+            'any':  select.POLLIN | select.POLLOUT,
+        }[direction]
+
+        poll = select.poll()
+        poll.register(self, want | select.POLLHUP | select.POLLERR)
+
+        for fd, event in poll.poll(0):
+            if event & select.POLLHUP:
+                self.close()
+                return False
+            if event & select.POLLIN:
+                return True
+            if event & select.POLLOUT:
+                return True
+
+        return True
 
     def close(self):
         if not getattr(self, 'sock', None):

pwnlib/tubes/ssh.py

@@ -59,7 +59,7 @@ class ssh_channel(sock):
     #: Only valid when instantiated through :meth:`ssh.process`
     pid = None
 
-    #: Executable of the process
+    #: Executable of the procesks
     #: Only valid when instantiated through :meth:`ssh.process`
     executable = None
 
@@ -67,8 +67,8 @@ class ssh_channel(sock):
     #: Only valid when instantiated through :meth:`ssh.process`
     argv = None
 
-    def __init__(self, parent, process = None, tty = False, wd = None, env = None, timeout = Timeout.default, level = 0, raw = True):
-        super(ssh_channel, self).__init__(timeout, level=level)
+    def __init__(self, parent, process = None, tty = False, wd = None, env = None, raw = True, *args, **kwargs):
+        super(ssh_channel, self).__init__(*args, **kwargs)
 
         # keep the parent from being garbage collected in some cases
         self.parent = parent
@@ -191,11 +191,11 @@ class ssh_channel(sock):
         return self.returncode
 
     def can_recv_raw(self, timeout):
-        end = time.time() + timeout
-        while time.time() < end:
-            if self.sock.recv_ready():
-                return True
-            time.sleep(0.05)
+        with self.countdown(timeout):
+            while self.countdown_active():
+                if self.sock.recv_ready():
+                    return True
+                time.sleep(min(self.timeout, 0.05))
         return False
 
     def interactive(self, prompt = term.text.bold_red('$') + ' '):
@@ -366,8 +366,8 @@ class ssh_channel(sock):
                 return e
 
 class ssh_connecter(sock):
-    def __init__(self, parent, host, port, timeout = Timeout.default, level = None):
-        super(ssh_connecter, self).__init__(timeout, level = level)
+    def __init__(self, parent, host, port, *a, **kw):
+        super(ssh_connecter, self).__init__(*a, **kw)
 
         # keep the parent from being garbage collected in some cases
         self.parent = parent
@@ -398,8 +398,8 @@ class ssh_connecter(sock):
 
 
 class ssh_listener(sock):
-    def __init__(self, parent, bind_address, port, timeout = Timeout.default, level = None):
-        super(ssh_listener, self).__init__(timeout, level = level)
+    def __init__(self, parent, bind_address, port, *a, **kw):
+        super(ssh_listener, self).__init__(*a, **kw)
 
         # keep the parent from being garbage collected in some cases
         self.parent = parent
@@ -478,8 +478,7 @@ class ssh(Timeout, Logger):
 
     def __init__(self, user, host, port = 22, password = None, key = None,
                  keyfile = None, proxy_command = None, proxy_sock = None,
-                 timeout = Timeout.default, level = None, cache = True,
-                 ssh_agent = False):
+                 level = None, cache = True, ssh_agent = False, *a, **kw):
         """Creates a new ssh connection.
 
         Arguments:
@@ -498,7 +497,7 @@ class ssh(Timeout, Logger):
 
         NOTE: The proxy_command and proxy_sock arguments is only available if a
         fairly new version of paramiko is used."""
-        super(ssh, self).__init__(timeout)
+        super(ssh, self).__init__(*a, **kw)
 
         Logger.__init__(self)
         if level is not None:
@@ -623,7 +622,8 @@ class ssh(Timeout, Logger):
         return self.run(shell, tty, timeout = timeout)
 
     def process(self, argv=None, executable=None, tty=True, cwd=None, env=None, timeout=Timeout.default, run=True,
-                stdin=0, stdout=1, stderr=2, preexec_fn=None, preexec_args=[], raw=True, aslr=None, setuid=None):
+                stdin=0, stdout=1, stderr=2, preexec_fn=None, preexec_args=[], raw=True, aslr=None, setuid=None,
+                shell=False):
         r"""
         Executes a process on the remote server, in the same fashion
         as pwnlib.tubes.process.process.
@@ -680,6 +680,8 @@ class ssh(Timeout, Logger):
                 See ``pwnlib.tubes.process.process`` for more information.
             setuid(bool):
                 See ``pwnlib.tubes.process.process`` for more information.
+            shell(bool):
+                Pass the command-line arguments to the shell.
 
         Returns:
             A new SSH channel, or a path to a script if ``run=False``.
@@ -737,6 +739,9 @@ class ssh(Timeout, Logger):
             Traceback (most recent call last):
             ...
             NameError: global name 'bar' is not defined
+
+            >>> s.process('echo hello', shell=True).recvall()
+            'hello\n'
         """
         if not argv and not executable:
             self.error("Must specify argv or executable")
@@ -750,6 +755,11 @@ class ssh(Timeout, Logger):
         if not isinstance(argv, (list, tuple)):
             self.error('argv must be a list or tuple')
 
+        if shell:
+            if len(argv) != 1:
+                self.error('Cannot provide more than 1 argument if shell=True')
+            argv = ['/bin/sh', '-c'] + argv
+
         # Python doesn't like when an arg in argv contains '\x00'
         # -> execve() arg 2 must contain only strings
         for i, arg in enumerate(argv):
@@ -862,6 +872,13 @@ if not %(aslr)r:
 
     resource.setrlimit(resource.RLIMIT_STACK, (-1, -1))
 
+# Attempt to dump ALL core file regions
+try:
+    with open('/proc/self/coredump_filter', 'w') as core_filter:
+        core_filter.write('0x3f\n')
+except Exception:
+    pass
+
 # Assume that the user would prefer to have core dumps.
 resource.setrlimit(resource.RLIMIT_CORE, (-1, -1))
 
@@ -941,7 +958,7 @@ os.execve(exe, argv, os.environ)
 
         return result
 
-    def system(self, process, tty = True, wd = None, env = None, timeout = Timeout.default, raw = True):
+    def system(self, process, tty = True, wd = None, env = None, timeout = None, raw = True):
         r"""system(process, tty = True, wd = None, env = None, timeout = Timeout.default, raw = True) -> ssh_channel
 
         Open a new channel with a specific process inside. If `tty` is True,
@@ -967,7 +984,10 @@ os.execve(exe, argv, os.environ)
         if wd is None:
             wd = self.cwd
 
-        return ssh_channel(self, process, tty, wd, env, timeout, level = self.level, raw = raw)
+        if timeout is None:
+            timeout = self.timeout
+
+        return ssh_channel(self, process, tty, wd, env, timeout = timeout, level = self.level, raw = raw)
 
     #: Backward compatibility.  Use :meth:`system`
     run = system

pwnlib/tubes/tube.py

@@ -30,18 +30,18 @@ class tube(Timeout, Logger):
     #: and related functions.
     newline = '\n'
 
-    def __init__(self, timeout = default, level = None):
+    def __init__(self, timeout = default, level = None, *a, **kw):
         super(tube, self).__init__(timeout)
 
         Logger.__init__(self, None)
         if level is not None:
             self.setLevel(level)
 
-        self.buffer          = Buffer()
+        self.buffer = Buffer(*a, **kw)
         atexit.register(self.close)
 
     # Functions based on functions from subclasses
-    def recv(self, numb = 4096, timeout = default):
+    def recv(self, numb = None, timeout = default):
         r"""recv(numb = 4096, timeout = default) -> str
 
         Receives up to `numb` bytes of data from the tube, and returns
@@ -72,6 +72,7 @@ class tube(Timeout, Logger):
             [...] Received 0xc bytes:
                 'Hello, world'
         """
+        numb = self.buffer.get_fill_size(numb)
         return self._recv(numb, timeout) or ''
 
     def unrecv(self, data):
@@ -122,7 +123,7 @@ class tube(Timeout, Logger):
         data = ''
 
         with self.local(timeout):
-            data = self.recv_raw(4096)
+            data = self.recv_raw(self.buffer.get_fill_size())
 
         if data and self.isEnabledFor(logging.DEBUG):
             self.debug('Received %#x bytes:' % len(data))
@@ -141,12 +142,13 @@ class tube(Timeout, Logger):
         return data
 
 
-    def _recv(self, numb = 4096, timeout = default):
+    def _recv(self, numb = None, timeout = default):
         """_recv(numb = 4096, timeout = default) -> str
 
         Receives one chunk of from the internal buffer or from the OS if the
         buffer is empty.
         """
+        numb = self.buffer.get_fill_size(numb)
         data = ''
 
         # No buffered data, could not put anything in the buffer
@@ -826,6 +828,35 @@ class tube(Timeout, Logger):
         while t.is_alive():
             t.join(timeout = 0.1)
 
+    def stream(self, line_mode=True):
+        """stream()
+
+        Receive data until the tube exits, and print it to stdout.
+
+        Similar to :func:`interactive`, except that no input is sent.
+
+        Similar to ``print tube.recvall()`` except that data is printed
+        as it is received, rather than after all data is received.
+
+        Arguments:
+            line_mode(bool): Whether to receive line-by-line or raw data.
+
+        Returns:
+            All data printed.
+        """
+        buf = Buffer()
+        function = self.recvline if line_mode else self.recv
+        try:
+            while True:
+                buf.add(function())
+                sys.stdout.write(buf.data[-1])
+        except KeyboardInterrupt:
+            pass
+        except EOFError:
+            pass
+
+        return buf.get()
+
     def clean(self, timeout = 0.05):
         """clean(timeout = 0.05)
 

pwnlib/update.py

@@ -53,7 +53,11 @@ def available_on_pypi(prerelease=current_version.is_prerelease):
 
 def cache_file():
     """Returns the path of the file used to cache update data, and ensures that it exists."""
-    cache_dir  = os.path.expanduser('~/.pwntools-cache')
+    cache_dir = context.cache_dir
+
+    if not cache_dir:
+        return None
+
     cache_file = os.path.join(cache_dir, 'update')
 
     if not os.path.isdir(cache_dir):
@@ -66,12 +70,23 @@ def cache_file():
 
 def last_check():
     """Return the date of the last check"""
-    return os.path.getmtime(cache_file())
+    cache = cache_file()
+    if cache:
+        return os.path.getmtime(cache_file())
+
+    # Fallback
+    return time.time()
 
 def should_check():
     """Return True if we should check for an update"""
-    if read(cache_file()).strip() == 'never':
+    filename = cache_file()
+
+    if not filename:
         return False
+
+    if read(filename).strip() == 'never':
+        return False
+
     return time.time() > (last_check() + update_freq)
 
 def perform_check(prerelease=current_version.is_prerelease):
@@ -109,7 +124,10 @@ def perform_check(prerelease=current_version.is_prerelease):
     where = None
     command = None
 
-    os.utime(cache_file(), None)
+    cache = cache_file()
+
+    if cache:
+        os.utime(cache, None)
 
     if best == current_version:
         log.info("You have the latest version of Pwntools (%s)" % best)

pwnlib/util/crc/__init__.py

@@ -1,10 +1,10 @@
 """Module for calculating CRC-sums.
 
-Contains all crc implementations know on the interwebz. For most implementations it
-contains only the core crc algorithm and not e.g. padding schemes.
+Contains all crc implementations know on the interwebz. For most implementations
+it contains only the core crc algorithm and not e.g. padding schemes.
 
 It is horribly slow, as implements a naive algorithm working direclty on
-bit polynomials.
+bit polynomials. This class is exposed as `BitPolynom`.
 
 The current algorithm is super-linear and takes about 4 seconds to calculate
 the crc32-sum of ``'A'*40000``.
@@ -18,15 +18,70 @@ import types
 from . import known
 from .. import fiddling
 from .. import packing
+from .. import safeeval
 
 
 class BitPolynom(object):
+    """Class for representing GF(2)[X], i.e. the field of polynomials over
+    GF(2).
+
+    In practice the polynomials are represented as numbers such that `x**n`
+    corresponds to `1 << n`. In this representation calculations are easy: Just
+    do everything as normal, but forget about everything the carries.
+
+    Addition becomes xor and multiplication becomes carry-less multiplication.
+
+    Examples:
+
+        >>> p1 = BitPolynom("x**3 + x + 1")
+        >>> p1
+        BitPolynom('x**3 + x + 1')
+        >>> int(p1)
+        11
+        >>> p1 == BitPolynom(11)
+        True
+        >>> p2 = BitPolynom("x**2 + x + 1")
+        >>> p1 + p2
+        BitPolynom('x**3 + x**2')
+        >>> p1 * p2
+        BitPolynom('x**5 + x**4 + 1')
+        >>> p1 / p2
+        BitPolynom('x + 1')
+        >>> p1 % p2
+        BitPolynom('x')
+        >>> d, r = divmod(p1, p2)
+        >>> d * p2 + r == p1
+        True
+        >>> BitPolynom(-1)
+        Traceback (most recent call last):
+            ...
+        ValueError: Polynomials cannot be negative: -1
+        >>> BitPolynom('y')
+        Traceback (most recent call last):
+            ...
+        ValueError: Not a valid polynomial: y
+    """
+
+
     def __init__(self, n):
-        if not isinstance(n, (int, long)):
-            raise TypeError("Polynomial must be called with an integer or a list")
-        if n < 0:
-            raise ValueError("Polynomials cannot be negative: %d" % n)
-        self.n = n
+        if isinstance(n, (str, unicode)):
+            self.n = 0
+            x = BitPolynom(2)
+            try:
+                for p in n.split('+'):
+                    k = safeeval.values(p.strip(), {'x': x, 'X': x})
+                    assert isinstance(k, (BitPolynom, int, long))
+                    assert k >= 0
+                    self.n ^= int(k)
+            except (ValueError, NameError, AssertionError):
+                raise ValueError("Not a valid polynomial: %s" % n)
+        elif isinstance(n, (int, long)):
+            if n >= 0:
+                self.n = n
+            else:
+                raise ValueError("Polynomials cannot be negative: %d" % n)
+        else:
+            raise TypeError("Polynomial must be called with a string or integer")
 
     def __int__(self):
         return self.n
@@ -126,7 +181,30 @@ class BitPolynom(object):
     def __rrshift__(self, other):
         return BitPolynom(int(other) >> int(self))
 
+    def __pow__(self, other):
+        r = BitPolynom(1)
+        for _ in range(other):
+            r *= self
+        return r
+
     def degree(self):
+        """Returns the degree of the polynomial.
+
+        Examples:
+
+            >>> BitPolynom(0).degree()
+            0
+            >>> BitPolynom(1).degree()
+            0
+            >>> BitPolynom(2).degree()
+            1
+            >>> BitPolynom(7).degree()
+            2
+            >>> BitPolynom((1 << 10) - 1).degree()
+            9
+            >>> BitPolynom(1 << 10).degree()
+            10
+        """
         return max(0, int(self).bit_length()-1)
 
     def __repr__(self):
@@ -134,17 +212,20 @@ class BitPolynom(object):
             return '0'
 
         out = []
-        for n in range(self.degree(), 0, -1):
+        for n in range(self.degree(), 1, -1):
             if int(self) & (1 << n):
                 out.append("x**%d" % n)
+        if int(self) & 2:
+            out.append("x")
         if int(self) & 1:
             out.append("1")
-        return ' + '.join(out)
+        return 'BitPolynom(%r)' % ' + '.join(out)
 
 class Module(types.ModuleType):
     def __init__(self):
         super(Module, self).__init__(__name__)
         self._cached_crcs = None
+        self.BitPolynom = BitPolynom
         self.__dict__.update({
             '__file__'    : __file__,
             '__package__' : __package__,
@@ -154,7 +235,7 @@ class Module(types.ModuleType):
         crcs = known.all_crcs
 
         if attr == '__all__':
-            return ['generic_crc', 'cksum', 'find_crc_function'] + sorted(crcs.keys())
+            return ['BitPolynom', 'generic_crc', 'cksum', 'find_crc_function'] + sorted(crcs.keys())
 
         info = crcs.get(attr, None)
         if not info:

pwnlib/util/cyclic.py

@@ -43,8 +43,8 @@ def de_bruijn(alphabet = string.ascii_lowercase, n = None):
 def cyclic(length = None, alphabet = string.ascii_lowercase, n = None):
     """cyclic(length = None, alphabet = string.ascii_lowercase, n = 4) -> list/str
 
-    A simple wrapper over :func:`de_bruijn`. This function returns a
-    at most `length` elements.
+    A simple wrapper over :func:`de_bruijn`. This function returns at most
+    `length` elements.
 
     If the given alphabet is a string, a string is returned from this function. Otherwise
     a list is returned.
@@ -66,6 +66,10 @@ def cyclic(length = None, alphabet = string.ascii_lowercase, n = None):
     if n is None:
         n = 4
 
+    if len(alphabet) ** n < length:
+        log.error("Can't create a pattern length=%i with len(alphabet)==%i and n==%i" \
+                  % (length, len(alphabet), n))
+
     out = []
     for ndx, c in enumerate(de_bruijn(alphabet, n)):
         if length != None and ndx >= length:
@@ -93,8 +97,8 @@ def cyclic_find(subseq, alphabet = string.ascii_lowercase, n = None):
        https://www.sciencedirect.com/science/article/pii/S0012365X00001175
 
     Arguments:
-        subseq: The subsequence to look for. This can either be a string, a list
-                or an integer. If an integer is provided it will be packed as a
+        subseq: The subsequence to look for. This can be a string, a list or an
+                integer. If an integer is provided it will be packed as a
                 little endian integer.
         alphabet: List or string to generate the sequence over.
         n(int): The length of subsequences that should be unique.
@@ -124,8 +128,98 @@ def cyclic_find(subseq, alphabet = string.ascii_lowercase, n = None):
 
     return _gen_find(subseq, de_bruijn(alphabet, n))
 
+def metasploit_pattern(sets = None):
+    """metasploit_pattern(sets = [ string.ascii_uppercase, string.ascii_lowercase, string.digits ]) -> generator
+
+    Generator for a sequence of characters as per Metasploit Framework's
+    `Rex::Text.pattern_create` (aka `pattern_create.rb`).
+
+    The returned generator will yield up to
+    ``len(sets) * reduce(lambda x,y: x*y, map(len, sets))`` elements.
+
+    Arguments:
+        sets: List of strings to generate the sequence over.
+    """
+    sets = sets or [ string.ascii_uppercase, string.ascii_lowercase, string.digits ]
+    offsets = [ 0 ] * len(sets)
+    offsets_indexes_reversed = list(reversed(range(len(offsets))))
+
+    while True:
+        for i, j in zip(sets, offsets):
+            yield i[j]
+        # increment offsets with cascade
+        for i in offsets_indexes_reversed:
+            offsets[i] = (offsets[i] + 1) % len(sets[i])
+            if offsets[i] != 0:
+                break
+        # finish up if we've exhausted the sequence
+        if offsets == [ 0 ] * len(sets):
+            return
+
+def cyclic_metasploit(length = None, sets = None):
+    """cyclic_metasploit(length = None, sets = [ string.ascii_uppercase, string.ascii_lowercase, string.digits ]) -> str
+
+    A simple wrapper over :func:`metasploit_pattern`. This function returns a
+    string of length `length`.
+
+    Arguments:
+        length: The desired length of the string or None if the entire sequence is desired.
+        sets: List of strings to generate the sequence over.
+
+    Example:
+        >>> cyclic_metasploit(32)
+        'Aa0Aa1Aa2Aa3Aa4Aa5Aa6Aa7Aa8Aa9Ab'
+        >>> cyclic_metasploit(sets = ["AB","ab","12"])
+        'Aa1Aa2Ab1Ab2Ba1Ba2Bb1Bb2'
+        >>> cyclic_metasploit()[1337:1341]
+        '5Bs6'
+        >>> len(cyclic_metasploit())
+        20280
+    """
+    sets = sets or [ string.ascii_uppercase, string.ascii_lowercase, string.digits ]
+    out = []
+
+    for ndx, c in enumerate(metasploit_pattern(sets)):
+        if length != None and ndx >= length:
+            break
+        else:
+            out.append(c)
+
+    out = ''.join(out)
+
+    if len(out) < length:
+        log.error("Can't create a pattern of length %i with sets of lengths %s. Maximum pattern length is %i." \
+                  % (length, map(len, sets), len(out)))
+
+    return ''.join(out)
+
+def cyclic_metasploit_find(subseq, sets = None):
+    """cyclic_metasploit_find(subseq, sets = [ string.ascii_uppercase, string.ascii_lowercase, string.digits ]) -> int
+
+    Calculates the position of a substring into a Metasploit Pattern sequence.
+
+    Arguments:
+        subseq: The subsequence to look for. This can be a string or an
+                integer. If an integer is provided it will be packed as a
+                little endian integer.
+        sets: List of strings to generate the sequence over.
+
+    Examples:
+
+        >>> cyclic_metasploit_find(cyclic_metasploit(1000)[514:518])
+        514
+        >>> cyclic_metasploit_find(0x61413161)
+        4
+    """
+    sets = sets or [ string.ascii_uppercase, string.ascii_lowercase, string.digits ]
+
+    if isinstance(subseq, (int, long)):
+        subseq = packing.pack(subseq, 'all', 'little', False)
+
+    return _gen_find(subseq, metasploit_pattern(sets))
+
 def _gen_find(subseq, generator):
-    """Returns the first position of subseq in the generator or -1 if there is no such position."""
+    """Returns the first position of `subseq` in the generator or -1 if there is no such position."""
     subseq = list(subseq)
     pos = 0
     saved = []

pwnlib/util/fiddling.py

@@ -1,6 +1,7 @@
 # -*- coding: utf-8 -*-
 import base64
 import random
+import os
 import re
 import string
 import StringIO
@@ -10,7 +11,7 @@ from . import packing
 from ..context import context
 from ..log import getLogger
 from ..term import text
-from .cyclic import cyclic
+from .cyclic import de_bruijn
 from .cyclic import cyclic_find
 
 log = getLogger(__name__)
@@ -559,13 +560,15 @@ default_style = {
 }
 
 cyclic_pregen = ''
+de_bruijn_gen = de_bruijn()
 
 def sequential_lines(a,b):
     return (a+b) in cyclic_pregen
 
 def update_cyclic_pregenerated(size):
     global cyclic_pregen
-    cyclic_pregen = cyclic(size)
+    while size > len(cyclic_pregen):
+        cyclic_pregen += de_bruijn_gen.next()
 
 def hexdump_iter(fd, width=16, skip=True, hexii=False, begin=0, style=None,
                  highlight=None, cyclic=False):
@@ -627,30 +630,49 @@ def hexdump_iter(fd, width=16, skip=True, hexii=False, begin=0, style=None,
             return hbyte, abyte
         cache = [style_byte(chr(b)) for b in range(256)]
 
-    if cyclic:
-        update_cyclic_pregenerated(len(s))
-
     numb = 0
     while True:
         offset = begin + numb
-        chunk = fd.read(width)
+
+        # If a tube is passed in as fd, it will raise EOFError when it runs
+        # out of data, unlike a file or StringIO object, which return an empty
+        # string.
+        try:
+            chunk = fd.read(width)
+        except EOFError:
+            chunk = ''
+
+        # We have run out of data, exit the loop
         if chunk == '':
             break
+
+        # Advance the cursor by the number of bytes we actually read
         numb += len(chunk)
+
+        # Update the cyclic pattern in case
+        if cyclic:
+            update_cyclic_pregenerated(numb)
+
         # If this chunk is the same as the last unique chunk,
         # use a '*' instead.
-        if skip and \
-           (last_unique == chunk or \
-            (cyclic and sequential_lines(last_unique, chunk))):
+        if skip and last_unique:
+            same_as_last_line = (last_unique == chunk)
+            lines_are_sequential = (cyclic and sequential_lines(last_unique, chunk))
             last_unique = chunk
-            if not skipping:
-                yield '*'
-                skipping = True
-            continue
 
-        # Chunk is unique, save for next iteration
-        last_unique = chunk
+            if same_as_last_line or lines_are_sequential:
+
+                # If we have not already printed a "*", do so
+                if not skipping:
+                    yield '*'
+                    skipping = True
+
+                # Move on to the next chunk
+                continue
+
+        # Chunk is unique, no longer skipping
         skipping = False
+        last_unique = chunk
 
         # Generate contents for line
         hexbytes = ''

pwnlib/util/proc.py

@@ -2,19 +2,14 @@ import errno
 import socket
 import time
 
+import psutil
+
 from .. import tubes
 from ..log import getLogger
 
-try:
-    import psutil
-    _ok_import = True
-except ImportError:
-    _ok_import = False
-
 log = getLogger(__name__)
 
-if _ok_import:
-    all_pids = psutil.pids
+all_pids = psutil.pids
 
 def pidof(target):
     """pidof(target) -> int list
@@ -305,56 +300,3 @@ def wait_for_debugger(pid):
         while tracer(pid) is None:
             time.sleep(0.01)
         l.success()
-
-if not _ok_import:
-    def _make_stub(func):
-        func.__doc__ = 'Stubbed out function, because psutil is not available.'
-        return func
-
-    @_make_stub
-    def all_pids():
-        log.error("Called stubbed-out function. Get psutil to work on your platform, then come back.")
-
-    @_make_stub
-    def pidof(target):
-        log.error("Called stubbed-out function. Get psutil to work on your platform, then come back.")
-
-    @_make_stub
-    def pid_by_name(name):
-        log.error("Called stubbed-out function. Get psutil to work on your platform, then come back.")
-
-    @_make_stub
-    def name(pid):
-        log.error("Called stubbed-out function. Get psutil to work on your platform, then come back.")
-
-    @_make_stub
-    def parent(pid):
-        log.error("Called stubbed-out function. Get psutil to work on your platform, then come back.")
-
-    @_make_stub
-    def children(ppid):
-        log.error("Called stubbed-out function. Get psutil to work on your platform, then come back.")
-
-    @_make_stub
-    def ancestors(pid):
-        log.error("Called stubbed-out function. Get psutil to work on your platform, then come back.")
-
-    @_make_stub
-    def descendants(pid):
-        log.error("Called stubbed-out function. Get psutil to work on your platform, then come back.")
-
-    @_make_stub
-    def exe(pid):
-        log.error("Called stubbed-out function. Get psutil to work on your platform, then come back.")
-
-    @_make_stub
-    def cwd(pid):
-        log.error("Called stubbed-out function. Get psutil to work on your platform, then come back.")
-
-    @_make_stub
-    def cmdline(pid):
-        log.error("Called stubbed-out function. Get psutil to work on your platform, then come back.")
-
-    @_make_stub
-    def starttime(pid):
-        log.error("Called stubbed-out function. Get psutil to work on your platform, then come back.")

pwnlib/version.py

@@ -1 +1 @@
-__version__ = '3.2.0'
+__version__ = '3.4.1'

setup.py

@@ -54,11 +54,8 @@ install_requires     = ['paramiko>=1.15.2',
                         'pysocks',
                         'python-dateutil',
                         'pypandoc',
-                        'packaging']
-
-# This is a hack until somebody ports psutil to OpenBSD
-if platform.system() != 'OpenBSD':
-    install_requires.append('psutil>=2.1.3')
+                        'packaging',
+                        'psutil>=3.3.0']
 
 # Check that the user has installed the Python development headers
 PythonH = os.path.join(get_python_inc(), 'Python.h')
@@ -83,7 +80,7 @@ except ImportError:
 setup(
     name                 = 'pwntools',
     packages             = find_packages(),
-    version              = '3.2.0',
+    version              = '3.4.1',
     data_files           = [('',
                              glob.glob('*.md') + glob.glob('*.txt')),
                             ],

travis/install.sh

@@ -1,5 +1,6 @@
 #!/usr/bin/env bash -e
 set -e
+
 local_deb_extract()
 {
     wget $1
@@ -8,52 +9,54 @@ local_deb_extract()
     rm -f *.tar.* *deb*
 }
 
-get_binutils()
+install_deb()
 {
-    BINUTILS_PREFIX='https://launchpad.net/~pwntools/+archive/ubuntu/binutils/+files/binutils-'
-    BINUTILS_SUFFIX='-linux-gnu_2.22-6ubuntu1.1cross0.11pwntools12~precise_amd64.deb'
-    local_deb_extract "${BINUTILS_PREFIX}${1}${BINUTILS_SUFFIX}"
-}
-
-get_qemu()
-{
-    echo "Installing qemu"
-    QEMU_INDEX='http://packages.ubuntu.com/en/yakkety/amd64/qemu-user-static/download'
-    QEMU_URL=$(curl "$QEMU_INDEX" | grep kernel.org | grep -Eo 'http.*\.deb')
-    local_deb_extract "$QEMU_URL"
+    version=${2:-zesty}
+    package=$1
+    echo "Installing $package"
+    INDEX="http://packages.ubuntu.com/en/$version/amd64/$package/download"
+    URL=$(curl "$INDEX" | grep -Eo "https?://.*$package.*\.deb" | head -1)
+    local_deb_extract "$URL"
 }
 
 setup_travis()
 {
     export PATH=$PWD/usr/bin:$PATH
-    export LD_LIBRARY_PATH=$PWD/usr/lib
+    export LD_LIBRARY_PATH=$PWD/usr/lib:$LD_LIBRARY_PATH
+    export LD_LIBRARY_PATH=$PWD/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH
 
-    # Install libbfd-multiarch and libopcodes-multiarch if not found in the cache
-    if [ ! -f usr/lib/libbfd-2.22-multiarch.so ];
-    then
-        # Install the multiarch binutils
-        local_deb_extract http://mirrors.mit.edu/ubuntu/pool/universe/b/binutils/binutils-multiarch_2.22-6ubuntu1_amd64.deb
-        pushd usr/lib
-        ln -sf libbfd-2.22-multiarch.so libbfd-2.22.so
-        ln -sf libopcodes-2.22-multiarch.so libopcodes-2.22.so
-        popd
-    fi
+    # Install a more modern binutils, which is required for some of the tests
+    [[ -f usr/bin/objcopy ]] || install_deb binutils
 
     # Install/upgrade qemu
-    if ! (which qemu-arm-static && qemu-arm-static -version | grep 2.6.1); then
-        get_qemu
-    fi
+    [[ -f usr/bin/qemu-arm-static ]] || install_deb qemu-user-static xenial
 
-    # Install our custom binutils
-    which arm-linux-gnu-as     || get_binutils arm
-    which mips-linux-gnu-as    || get_binutils mips
-    which powerpc-linux-gnu-as || get_binutils powerpc
+    # Install cross-binutils
+    [[ -f usr/bin/x86_64-linux-gnu-ar ]]    || install_deb binutils-multiarch
+    [[ -f usr/bin/aarch64-linux-gnu-as ]]   || install_deb binutils-aarch64-linux-gnu
+    [[ -f usr/bin/arm-linux-gnueabihf-as ]] || install_deb binutils-arm-linux-gnueabihf
+    [[ -f usr/bin/mips-linux-gnu-as ]]      || install_deb binutils-mips-linux-gnu
+    [[ -f usr/bin/powerpc-linux-gnu-as ]]   || install_deb binutils-powerpc-linux-gnu
 
     # Test that the installs worked
-    which arm-linux-gnu-as
-    which mips-linux-gnu-as
-    which powerpc-linux-gnu-as
-    which qemu-arm-static
+    as                      --version
+    x86_64-linux-gnu-ar     --version
+    aarch64-linux-gnu-as    --version
+    arm-linux-gnueabihf-as  --version
+    mips-linux-gnu-as       --version
+    powerpc-linux-gnu-as    --version
+    qemu-arm-static         --version
+
+    # Force-install capstone because it's broken somehow
+    [[ -f usr/lib/libcapstone.so.3 ]] || install_deb libcapstone3
+
+    # Install a newer copy of GDB
+    if [[ ! -f usr/bin/gdb ]]; then
+        git clone --depth=1 https://github.com/zachriggle/pwntools-gdb-travis-ci.git
+        tar xf pwntools-gdb-travis-ci/gdb.tar.xz
+        which gdb
+        usr/bin/gdb --version
+    fi
 
     # Get rid of files we don't want cached
     rm -rf usr/share
@@ -72,14 +75,28 @@ setup_android_emulator()
     # If we are running on Travis CI, and there were no changes to Android
     # or ADB code, then we do not need the emulator
     if [ -n "$TRAVIS" ]; then
-        if ! (git log --stat "$TRAVIS_COMMIT_RANGE" | grep -E "android|adb"); then
+        if [ -z "$TRAVIS_COMMIT_RANGE" ]; then
+            echo "TRAVIS_COMMIT_RANGE is empty, forcing Android Emulator installation"
+        elif ! (git show "$TRAVIS_COMMIT_RANGE" >/dev/null) ; then
+            echo "TRAVIS_COMMIT_RANGE is invalid, forcing Android Emulator installation"
+        elif [[ "$TRAVIS_BRANCH" =~ "staging" ]]; then
+            echo "TRAVIS_BRANCH ($TRAVIS_BRANCH) indicates a branch we care about"
+            echo "Forcing Android Emulator installation"
+        elif [[ -n "$TRAVIS_TAG" ]]; then
+            echo "TRAVIS_TAG ($TRAVIS_TAG) indicates a new relase"
+            echo "Forcing Android Emulator installation"
+        elif (git log --stat "$TRAVIS_COMMIT_RANGE" | grep -iE "android|adb"); then
+            echo "Found Android-related commits, forcing Android Emulator installation"
+        else
             # In order to avoid running the doctests that require the Android
             # emulator, while still leaving the code intact, we remove the
             # RST file that Sphinx searches.
             rm -f 'docs/source/adb.rst'
+            rm -f 'docs/source/protocols/adb.rst'
 
             # However, the file needs to be present or else things break.
             touch 'docs/source/adb.rst'
+            touch 'docs/source/protocols/adb.rst' || true
 
             echo "Skipping Android emulator install, Android tests disabled."
             return
@@ -176,5 +193,4 @@ elif [[ "$(uname)" == "Linux" ]]; then
     setup_android_emulator
 fi
 
-
-dpkg -l
+set +e

travis/ssh_setup.sh

@@ -54,3 +54,5 @@ Host example.pwnme
 EOF
 
 ssh -o "StrictHostKeyChecking no" -vvvv [email protected] id
+
+set +e

travis/update_demo.sh

@@ -11,5 +11,5 @@ if [ "$TRAVIS_BRANCH" = "stable" ] && [ ! -z "$TRY_KEY" ]; then
     ssh-add           ~/.ssh/try_key
 
     # Update!
-    ssh [email protected] -- 'docker/update'
+    ssh -o "StrictHostKeyChecking no" [email protected] -- 'docker/update'
 fi