Release 3.4.0

This prevents silly things like tube.recv("abcdef") which will hang

.travis.yml

@@ -13,6 +13,7 @@ addons:
     - mksh
     - zsh
     - pandoc
+    - gdb
 cache:
     - pip
     - directories:
@@ -22,11 +23,14 @@ cache:
 python:
   - "2.7"
 before_install:
+  - echo $-
   - source travis/install.sh
   - source travis/ssh_setup.sh
+  - echo $-
 install:
+  - pip install --upgrade pip
   - pip install --upgrade --editable .
-  - pip install --requirement docs/requirements.txt
+  - pip install --upgrade --requirement docs/requirements.txt
 before_script:
   - PWNLIB_NOTERM=1 python -c 'from pwn import *; print pwnlib.term.term_mode'
   - PWNLIB_NOTERM=1 python -c 'from pwn import *; print pwnlib.term.term_mode'

CHANGELOG.md

@@ -9,9 +9,10 @@ The table below shows which release corresponds to each branch, and what date th
 
 | Version          | Branch   | Release Date           |
 | ---------------- | -------- | ---------------------- |
-| [3.5.0](#350)    | `dev`    | Mar 18, 2017 (planned)
-| [3.4.0](#340)    | `beta`   | Feb 4, 2017 (planned)
-| [3.3.4](#334)    | `stable` | Jan 12, 2016
+| [3.6.0](#350)    | `dev`    | May 6, 2017 (planned)
+| [3.5.0](#350)    | `beta`   | Mar 25, 2017 (planned)
+| [3.4.0](#340)    | `stable` | Feb 13, 2017
+| [3.3.4](#334)    |          | Jan 12, 2016
 | [3.3.3](#333)    |          | Jan 10, 2016
 | [3.3.2](#332)    |          | Jan 10, 2016
 | [3.3.1](#331)    |          | Jan 10, 2016
@@ -27,13 +28,32 @@ The table below shows which release corresponds to each branch, and what date th
 | [3.0.0](#300)    |          | Aug 20, 2016
 | [2.2.0](#220)    |          | Jan 5, 2015
 
+
+## 3.6.0
+
+To be released on May 6, 2017.
+
 ## 3.5.0
 
-To be released on Mar 18, 2017.
+To be released on Mar 25, 2017.
 
 ## 3.4.0
 
-To be released on Feb 4, 2017.
+- [#800][800] Add `shell=` option to `ssh.process()`
+- [#806][806] Add `context.buffer_size` for fine-tuning `tube` performance
+    + Also adds `buffer_fill_size=` argument for all tubes
+- [b83a6c7][b83a6c7] Fix undocumented `process.leak` function
+- [546061e][546061e] Modify `coredump_filter` of all spawned processes, so that core dumps are more complete
+- [#809][809] Add several functions to `adb` (`unlink`, `mkdir`, `makedirs`, `isdir`, `exists`)
+- [#817][817] Make disconnection detection more robust
+
+[800]: https://github.com/Gallopsled/pwntools/pull/800
+[806]: https://github.com/Gallopsled/pwntools/pull/806
+[809]: https://github.com/Gallopsled/pwntools/pull/809
+[817]: https://github.com/Gallopsled/pwntools/pull/817
+[5d9792f]: https://github.com/Gallopsled/pwntools/commit/5d9792f
+[b83a6c7]: https://github.com/Gallopsled/pwntools/commit/b83a6c7
+[546061e]: https://github.com/Gallopsled/pwntools/commit/546061e
 
 ## 3.3.4
 
@@ -154,7 +174,7 @@ Multiple bug fixes.
 - [1e414af][1e414af] Added `connect()` alias for `remote()`
     + For example, `io=connect('google.com', 80)`
     + This also works with `tcp(...)` and `udp(...)` aliases
-- [869ec42][869ec42] Added `ssh.read()` and `ssh.write()` aiases
+- [869ec42][869ec42] Added `ssh.read()` and `ssh.write()` aliases
 - [2af55c9][2af55c9] `AdbDevice` objects exposed via e.g. `adb.devices()` now offer scoped access to all `adb` module properties
     + It is now possible to e.g. `map(lambda d: d.process(['id']).recvall(), adb.devices())`
 
@@ -241,7 +261,7 @@ Android support via a new `adb` module, `context.device`, `context.adb_host`, an
 - Added `context.log_file` setting for sending logs to a file.  This can be set with the `LOG_FILE` magic command-line option.
 - Added `context.noptrace` setting for disabling actions which require `ptrace` support.  This is useful for turning all `gdb.debug` and `gdb.attach` options into no-ops, and can be set via the `NOPTRACE` magic command-line option.
 - Added `context.proxy` which hooks all connections and sends them to a SOCKS4/SOCKS5.  This can be set via the `PROXY` magic command-line option.
-- Added `context.randomize` to control randommization of settings like XOR keys and register ordering (default off).
+- Added `context.randomize` to control randomization of settings like XOR keys and register ordering (default off).
 - Added `context.terminal` for setting how to launch commands in a new terminal.
 
 #### DynELF and MemLeak Module
@@ -254,7 +274,7 @@ Android support via a new `adb` module, `context.device`, `context.adb_host`, an
 #### Encoders Module
 
 - Added a `pwnlib.encoders` module for assembled-shellcode encoders/decoders
-- Includes position-indepentent basic XOR encoders
+- Includes position-independent basic XOR encoders
 - Includes position-independent delta encoders
 - Includes non-position-independent alphanumeric encoders for Intel
 - Includes position-independent alphanumeric encoders for ARM/Thumb

README.md

@@ -2,7 +2,7 @@
 ![pwntools logo](https://github.com/Gallopsled/pwntools/blob/stable/docs/source/logo.png?raw=true)
 
 [![Docs](https://readthedocs.org/projects/pwntools/badge/?version=stable)](https://docs.pwntools.com/)
-[![PyPI](https://img.shields.io/badge/pypi-v3.3.4-green.svg?style=flat)](https://pypi.python.org/pypi/pwntools/)
+[![PyPI](https://img.shields.io/badge/pypi-v3.4.0-green.svg?style=flat)](https://pypi.python.org/pypi/pwntools/)
 [![Travis](https://travis-ci.org/Gallopsled/pwntools.svg)](https://travis-ci.org/Gallopsled/pwntools)
 [![Coveralls](https://img.shields.io/coveralls/Gallopsled/pwntools/dev.svg)](https://coveralls.io/github/Gallopsled/pwntools?branch=dev)
 [![Twitter](https://img.shields.io/badge/twitter-pwntools-4099FF.svg?style=flat)](https://twitter.com/pwntools)
@@ -38,7 +38,7 @@ Most of the functionality of pwntools is self-contained and Python-only.  You sh
 
 ```sh
 apt-get update
-apt-get install python2.7 python-pip python-dev git libssl-dev
+apt-get install python2.7 python-pip python-dev git libssl-dev libffi-dev
 pip install --upgrade pwntools
 ```
 

docs/source/install.rst

@@ -24,7 +24,8 @@ pwntools is available as a ``pip`` package.
 
 .. code-block:: bash
 
-    $ apt-get install python2.7 python2.7-dev python-pip
+    $ apt-get update
+    $ apt-get install python2.7 python-pip python-dev git libssl-dev libffi-dev
     $ pip install --upgrade pwntools
 
 Development

pwnlib/adb/adb.py

@@ -599,6 +599,150 @@ def write(path, data=''):
         misc.write(temp.name, data)
         push(temp.name, path)
 
+@context.quiet
+@with_device
+def mkdir(path):
+    """Create a directory on the target device.
+
+    Note:
+        Silently succeeds if the directory already exists.
+
+    Arguments:
+        path(str): Directory to create.
+
+    Examples:
+
+        >>> adb.mkdir('/')
+
+        >>> path = '/data/local/tmp/mkdir_test'
+        >>> adb.exists(path)
+        False
+        >>> adb.mkdir(path)
+        >>> adb.exists(path)
+        True
+
+        >>> adb.mkdir('/init')
+        Traceback (most recent call last):
+        ...
+        PwnlibException: mkdir failed for /init, File exists
+    """
+    if not path.startswith('/'):
+        log.error("Must provide an absolute path: %r" % path)
+
+    with Client() as c:
+        st = c.stat(path)
+
+        # Don't re-create existing directories
+        if st and stat.S_ISDIR(st['mode']):
+            return
+
+        result = process(['mkdir', path]).recvall()
+
+        # Any output at all is an error
+        if result:
+            log.error(result)
+
+@context.quiet
+@with_device
+def makedirs(path):
+    """Create a directory and all parent directories on the target device.
+
+    Note:
+        Silently succeeds if the directory already exists.
+
+    Examples:
+
+        >>> adb.makedirs('/data/local/tmp/this/is/a/directory/heirarchy')
+        >>> adb.listdir('/data/local/tmp/this/is/a/directory')
+        ['heirarchy']
+    """
+    if path != '/':
+        makedirs(os.path.dirname(path))
+
+    mkdir(path)
+
+@context.quiet
+@with_device
+def exists(path):
+    """Return ``True`` if ``path`` exists on the target device.
+
+    Examples:
+
+        >>> adb.exists('/')
+        True
+        >>> adb.exists('/init')
+        True
+        >>> adb.exists('/does/not/exist')
+        False
+    """
+    with Client() as c:
+        return bool(c.stat(path))
+
+@context.quiet
+@with_device
+def isdir(path):
+    """Return ``True`` if ``path`` is a on the target device.
+
+    Examples:
+
+        >>> adb.isdir('/')
+        True
+        >>> adb.isdir('/init')
+        False
+        >>> adb.isdir('/does/not/exist')
+        False
+    """
+    with Client() as c:
+        st = c.stat(path)
+        return bool(st and stat.S_ISDIR(st['mode']))
+
+@context.quiet
+@with_device
+def unlink(path, recursive=False):
+    """Unlinks a file or directory on the target device.
+
+    Examples:
+
+        >>> adb.unlink("/does/not/exist")
+        Traceback (most recent call last):
+        ...
+        PwnlibException: Could not unlink '/does/not/exist': Does not exist
+
+        >>> filename = '/data/local/tmp/unlink-test'
+        >>> adb.write(filename, 'hello')
+        >>> adb.exists(filename)
+        True
+        >>> adb.unlink(filename)
+        >>> adb.exists(filename)
+        False
+
+        >>> adb.mkdir(filename)
+        >>> adb.write(filename + '/contents', 'hello')
+        >>> adb.unlink(filename)
+        Traceback (most recent call last):
+        ...
+        PwnlibException: Cannot delete non-empty directory '/data/local/tmp/unlink-test' without recursive=True
+
+        >>> adb.unlink(filename, recursive=True)
+        >>> adb.exists(filename)
+        False
+    """
+    with Client() as c:
+        st = c.stat(path)
+        if not st:
+            log.error("Could not unlink %r: Does not exist" % path)
+
+        # If the directory is not empty, do not delete it
+        if isdir(path) and c.list(path) and not recursive:
+            log.error("Cannot delete non-empty directory %r without recursive=True" % path)
+
+        flags = '-rf' if recursive else '-r'
+
+        output = c.execute(['rm', flags, path]).recvall()
+
+        if output:
+            log.error(output)
+
 @with_device
 def process(argv, *a, **kw):
     """Execute a process on the device.
@@ -760,16 +904,15 @@ def listdir(directory='/'):
     """
     return list(sorted(Client().list(directory)))
 
+@with_device
 def fastboot(args, *a, **kw):
     """Executes a fastboot command.
 
     Returns:
         The command output.
     """
-    serial = context.device
-    if not serial:
-        log.error("Unknown device")
-    return tubes.process.process(['fastboot', '-s', serial] + list(args), **kw).recvall()
+    argv = ['fastboot', '-s', str(context.device)] + list(args)
+    return tubes.process.process(argv, *a, **kw).recvall()
 
 @with_device
 def fingerprint():

pwnlib/commandline/main.py

@@ -16,8 +16,8 @@ from . import scramble
 from . import shellcraft
 from . import unhex
 from . import update
-from .common import parser
 from ..context import context
+from .common import parser
 
 commands = {
     'asm': asm.main,

pwnlib/commandline/update.py

@@ -21,7 +21,7 @@ p.add_argument('--pre', action='store_true', help='''
 def main(a):
     result = pwnlib.update.perform_check(prerelease=a.pre)
     if a.install:
-        subprocess.check_call(result, shell=True)
+        subprocess.check_call(result, shell=False)
 
 if __name__ == '__main__':
     pwnlib.common.main(__file__)

pwnlib/context/__init__.py

@@ -330,6 +330,7 @@ class ContextType(object):
         'aslr': True,
         'binary': None,
         'bits': 32,
+        'buffer_size': 4096,
         'device': os.getenv('ANDROID_SERIAL', None) or None,
         'endian': 'little',
         'kernel': None,
@@ -1146,6 +1147,15 @@ class ContextType(object):
 
         return command
 
+    @_validator
+    def buffer_size(self, size):
+        """Internal buffer size to use for ``tube`` objects.
+
+        This is not the maximum size of the buffer, but this is the amount of data
+        which is passed to each raw ``read`` syscall (or equivalent).
+        """
+        return int(size)
+
     @property
     def cache_dir(self):
         """Directory used for caching data.
@@ -1185,7 +1195,6 @@ class ContextType(object):
 
         return cache
 
-
     #*************************************************************************
     #                               ALIASES
     #*************************************************************************

pwnlib/data/useragents/download-useragents.py

@@ -7,6 +7,7 @@ import re
 import urllib
 
 from bs4 import BeautifulSoup
+
 from pwn import *
 
 uas = set()

pwnlib/memleak.py

@@ -606,3 +606,14 @@ class MemLeak(object):
             return result
 
         return MemLeak(string_wrapper)
+
+    # Aliases for convenience
+    u64 = q
+    u32 = d
+    u16 = w
+    u8 = b
+
+    p64 = setq
+    p32 = setd
+    p16 = setw
+    p8 = setb

pwnlib/term/term.py

@@ -22,6 +22,7 @@ import threading
 import traceback
 
 from . import termcap
+from ..context import ContextType
 
 settings = None
 _graphics_mode = False
@@ -127,6 +128,11 @@ def init():
         sys.stdout = Wrapper(sys.stdout)
     if sys.stderr.isatty():
         sys.stderr = Wrapper(sys.stderr)
+
+    console = ContextType.defaults['log_console']
+    if console.isatty():
+        ContextType.defaults['log_console'] = Wrapper(console)
+
     # freeze all cells if an exception is thrown
     orig_hook = sys.excepthook
     def hook(*args):

pwnlib/tubes/buffer.py

@@ -1,5 +1,8 @@
 #!/usr/bin/env python2
 
+from ..context import context
+
+
 class Buffer(Exception):
     """
     List of strings with some helper routines.
@@ -28,10 +31,10 @@ class Buffer(Exception):
         The ``0th`` item in the buffer is the oldest item, and
         will be received first.
     """
-    def __init__(self):
+    def __init__(self, buffer_fill_size = None):
         self.data = [] # Buffer
         self.size = 0  # Length
-
+        self.buffer_fill_size = buffer_fill_size
 
     def __len__(self):
         """
@@ -169,3 +172,19 @@ class Buffer(Exception):
         self.size -= len(data)
 
         return data
+
+    def get_fill_size(self, size=None):
+        """
+        Retrieves the default fill size for this buffer class.
+
+        Arguments:
+            size (int): (Optional) If set and not None, returns the size variable back.
+
+        Returns:
+            Fill size as integer if size == None, else size.
+        """
+        if size is None:
+            size = self.buffer_fill_size
+
+        with context.local(buffer_size=size):
+            return context.buffer_size

pwnlib/tubes/listen.py

@@ -20,13 +20,11 @@ class listen(sock):
         bindaddr(str): The address to bind to.
         fam: The string "any", "ipv4" or "ipv6" or an integer to pass to :func:`socket.getaddrinfo`.
         typ: The string "tcp" or "udp" or an integer to pass to :func:`socket.getaddrinfo`.
-        timeout: A positive number, None
     """
 
     def __init__(self, port=0, bindaddr = "0.0.0.0",
-                 fam = "any", typ = "tcp",
-                 timeout = Timeout.default, level = None):
-        super(listen, self).__init__(timeout, level = level)
+                 fam = "any", typ = "tcp", *args, **kwargs):
+        super(listen, self).__init__(*args, **kwargs)
 
         port = int(port)
         fam  = {socket.AF_INET: 'ipv4',

pwnlib/tubes/process.py

@@ -44,8 +44,6 @@ class process(tube):
             Working directory.  Uses the current working directory by default.
         env(dict):
             Environment variables.  By default, inherits from Python's environment.
-        timeout(int):
-            Timeout to use on ``tube`` ``recv`` operations.
         stdin(int):
             File object or file descriptor number to use for ``stdin``.
             By default, a pipe is used.  A pty can be used instead by setting
@@ -206,11 +204,9 @@ class process(tube):
                  executable = None,
                  cwd = None,
                  env = None,
-                 timeout = Timeout.default,
                  stdin  = PIPE,
                  stdout = PTY,
                  stderr = STDOUT,
-                 level = None,
                  close_fds = True,
                  preexec_fn = lambda: None,
                  raw = True,
@@ -218,8 +214,11 @@ class process(tube):
                  setuid = None,
                  where = 'local',
                  display = None,
-                 alarm = None):
-        super(process, self).__init__(timeout, level = level)
+                 alarm = None,
+                 *args,
+                 **kwargs
+                 ):
+        super(process, self).__init__(*args,**kwargs)
 
         # Permit using context.binary
         if argv is None:
@@ -794,7 +793,7 @@ class process(tube):
         import pwnlib.elf.corefile
         return pwnlib.elf.corefile.Core(filename)
 
-    def leak(self, address, count=0):
+    def leak(self, address, count=1):
         """Leaks memory within the process at the specified address.
 
         Arguments:

pwnlib/tubes/remote.py

@@ -45,8 +45,8 @@ class remote(sock):
 
     def __init__(self, host, port,
                  fam = "any", typ = "tcp",
-                 timeout = Timeout.default, ssl=False, sock=None, level = None):
-        super(remote, self).__init__(timeout, level = level)
+                 ssl=False, sock=None, *args, **kwargs):
+        super(remote, self).__init__(*args, **kwargs)
 
         self.rport  = int(port)
         self.rhost  = host
@@ -150,17 +150,13 @@ class remote(sock):
 
 class tcp(remote):
     __doc__ = remote.__doc__
-    def __init__(self, host, port,
-                 fam = "any", typ = "tcp",
-                 timeout = Timeout.default, ssl=False, sock=None, level = None):
-        return super(tcp, self).__init__(host, port, fam, typ, timeout, ssl, sock, level)
+    def __init__(self, host, port, *a, **kw):
+        return super(tcp, self).__init__(host, port, typ="tcp", *a, **kw)
 
 class udp(remote):
     __doc__ = remote.__doc__
-    def __init__(self, host, port,
-                 fam = "any", typ = "udp",
-                 timeout = Timeout.default, ssl=False, sock=None, level = None):
-        return super(udp, self).__init__(host, port, fam, typ, timeout, ssl, sock, level)
+    def __init__(self, host, port, *a, **kw):
+        return super(udp, self).__init__(host, port, typ="udp", *a, **kw)
 
 class connect(remote):
     __doc__ = remote.__doc__

pwnlib/tubes/serialtube.py

@@ -9,7 +9,6 @@ from . import tube
 from .. import context
 from .. import term
 from ..log import getLogger
-from ..timeout import Timeout
 
 log = getLogger(__name__)
 
@@ -18,10 +17,8 @@ class serialtube(tube.tube):
             self, port = None, baudrate = 115200,
             convert_newlines = True,
             bytesize = 8, parity='N', stopbits=1, xonxoff = False,
-            rtscts = False, dsrdtr = False,
-            timeout = Timeout.default,
-            level = None):
-        super(serialtube, self).__init__(timeout, level = level)
+            rtscts = False, dsrdtr = False, *a, **kw):
+        super(serialtube, self).__init__(*a, **kw)
 
         if port is None:
             if platform.system() == 'Darwin':
@@ -49,21 +46,14 @@ class serialtube(tube.tube):
         if not self.conn:
             raise EOFError
 
-        if self.timeout == None:
-            end = float('inf')
-        else:
-            end = time.time() + self.timeout
+        with self.countdown():
+            while self.conn and self.countdown_active():
+                data = self.conn.read(numb)
 
-        while self.conn:
-            data = self.conn.read(numb)
-            if data:
-                return data
+                if data:
+                    return data
 
-            delta = end - time.time()
-            if delta <= 0:
-                break
-            else:
-                time.sleep(min(delta, 0.1))
+                time.sleep(min(self.timeout, 0.1))
 
         return None
 
@@ -83,10 +73,11 @@ class serialtube(tube.tube):
         pass
 
     def can_recv_raw(self, timeout):
-        end = time.time()
-        while time.time() < end:
-            if self.conn.inWaiting():
-                return True
+        with self.countdown(timeout):
+            while self.conn and self.countdown_active():
+                if self.conn.inWaiting():
+                    return True
+                time.sleep(min(self.timeout, 0.1))
         return False
 
     def connected_raw(self, direction):
@@ -99,7 +90,7 @@ class serialtube(tube.tube):
 
     def fileno(self):
         if not self.connected():
-            self.error("A stopped program does not have a file number")
+            self.error("A closed serialtube does not have a file number")
 
         return self.conn.fileno()
 

pwnlib/tubes/sock.py

@@ -10,8 +10,8 @@ log = getLogger(__name__)
 class sock(tube):
     """Methods available exclusively to sockets."""
 
-    def __init__(self, timeout, level = None):
-        super(sock, self).__init__(timeout, level = level)
+    def __init__(self, *args, **kwargs):
+        super(sock, self).__init__(*args, **kwargs)
         self.closed = {"recv": False, "send": False}
 
     # Overwritten for better usability
@@ -26,13 +26,13 @@ class sock(tube):
         else:
             return super(sock, self).recvall(timeout)
 
-    def recv_raw(self, numb):
+    def recv_raw(self, numb, *a):
         if self.closed["recv"]:
             raise EOFError
 
         while True:
             try:
-                data = self.sock.recv(numb)
+                data = self.sock.recv(numb, *a)
                 break
             except socket.timeout:
                 return None
@@ -72,21 +72,89 @@ class sock(tube):
             self.sock.settimeout(timeout)
 
     def can_recv_raw(self, timeout):
+        """
+        Tests:
+
+            >>> l = listen()
+            >>> r = remote('localhost', l.lport)
+            >>> r.can_recv_raw(timeout=0)
+            False
+            >>> l.send('a')
+            >>> r.can_recv_raw(timeout=1)
+            True
+            >>> r.recv()
+            'a'
+            >>> r.can_recv_raw(timeout=0)
+            False
+            >>> l.close()
+            >>> r.can_recv_raw(timeout=1)
+            False
+            >>> r.closed['recv']
+            True
+        """
         if not self.sock or self.closed["recv"]:
             return False
 
-        return select.select([self.sock], [], [], timeout) == ([self.sock], [], [])
+        # select() will tell us data is available at EOF
+        can_recv = select.select([self.sock], [], [], timeout) == ([self.sock], [], [])
+
+        if not can_recv:
+            return False
+
+        # Ensure there's actually data, not just EOF
+        try:
+            self.recv_raw(1, socket.MSG_PEEK)
+        except EOFError:
+            return False
+
+        return True
 
     def connected_raw(self, direction):
+        """
+        Tests:
+
+            >>> l = listen()
+            >>> r = remote('localhost', l.lport)
+            >>> r.connected()
+            True
+            >>> l.close()
+            >>> time.sleep(1) # Avoid race condition
+            >>> r.connected()
+            False
+        """
+        # If there's no socket, it's definitely closed
         if not self.sock:
             return False
 
-        if direction == 'any':
-            return True
-        elif direction == 'recv':
-            return not self.closed['recv']
-        elif direction == 'send':
-            return not self.closed['send']
+        # If we have noticed a connection close in a given direction before,
+        # return fast.
+        if self.closed.get(direction, False):
+            return False
+
+        # If a connection is closed in all manners, return fast
+        if all(self.closed.values()):
+            return False
+
+        # Use poll() to determine the connection state
+        want = {
+            'recv': select.POLLIN,
+            'send': select.POLLOUT,
+            'any':  select.POLLIN | select.POLLOUT,
+        }[direction]
+
+        poll = select.poll()
+        poll.register(self, want | select.POLLHUP | select.POLLERR)
+
+        for fd, event in poll.poll(0):
+            if event & select.POLLHUP:
+                self.close()
+                return False
+            if event & select.POLLIN:
+                return True
+            if event & select.POLLOUT:
+                return True
+
+        return True
 
     def close(self):
         if not getattr(self, 'sock', None):

pwnlib/tubes/ssh.py

@@ -59,7 +59,7 @@ class ssh_channel(sock):
     #: Only valid when instantiated through :meth:`ssh.process`
     pid = None
 
-    #: Executable of the process
+    #: Executable of the procesks
     #: Only valid when instantiated through :meth:`ssh.process`
     executable = None
 
@@ -67,8 +67,8 @@ class ssh_channel(sock):
     #: Only valid when instantiated through :meth:`ssh.process`
     argv = None
 
-    def __init__(self, parent, process = None, tty = False, wd = None, env = None, timeout = Timeout.default, level = 0, raw = True):
-        super(ssh_channel, self).__init__(timeout, level=level)
+    def __init__(self, parent, process = None, tty = False, wd = None, env = None, raw = True, *args, **kwargs):
+        super(ssh_channel, self).__init__(*args, **kwargs)
 
         # keep the parent from being garbage collected in some cases
         self.parent = parent
@@ -191,11 +191,11 @@ class ssh_channel(sock):
         return self.returncode
 
     def can_recv_raw(self, timeout):
-        end = time.time() + timeout
-        while time.time() < end:
-            if self.sock.recv_ready():
-                return True
-            time.sleep(0.05)
+        with self.countdown(timeout):
+            while self.countdown_active():
+                if self.sock.recv_ready():
+                    return True
+                time.sleep(min(self.timeout, 0.05))
         return False
 
     def interactive(self, prompt = term.text.bold_red('$') + ' '):
@@ -366,8 +366,8 @@ class ssh_channel(sock):
                 return e
 
 class ssh_connecter(sock):
-    def __init__(self, parent, host, port, timeout = Timeout.default, level = None):
-        super(ssh_connecter, self).__init__(timeout, level = level)
+    def __init__(self, parent, host, port, *a, **kw):
+        super(ssh_connecter, self).__init__(*a, **kw)
 
         # keep the parent from being garbage collected in some cases
         self.parent = parent
@@ -398,8 +398,8 @@ class ssh_connecter(sock):
 
 
 class ssh_listener(sock):
-    def __init__(self, parent, bind_address, port, timeout = Timeout.default, level = None):
-        super(ssh_listener, self).__init__(timeout, level = level)
+    def __init__(self, parent, bind_address, port, *a, **kw):
+        super(ssh_listener, self).__init__(*a, **kw)
 
         # keep the parent from being garbage collected in some cases
         self.parent = parent
@@ -478,8 +478,7 @@ class ssh(Timeout, Logger):
 
     def __init__(self, user, host, port = 22, password = None, key = None,
                  keyfile = None, proxy_command = None, proxy_sock = None,
-                 timeout = Timeout.default, level = None, cache = True,
-                 ssh_agent = False):
+                 level = None, cache = True, ssh_agent = False, *a, **kw):
         """Creates a new ssh connection.
 
         Arguments:
@@ -498,7 +497,7 @@ class ssh(Timeout, Logger):
 
         NOTE: The proxy_command and proxy_sock arguments is only available if a
         fairly new version of paramiko is used."""
-        super(ssh, self).__init__(timeout)
+        super(ssh, self).__init__(*a, **kw)
 
         Logger.__init__(self)
         if level is not None:
@@ -623,7 +622,8 @@ class ssh(Timeout, Logger):
         return self.run(shell, tty, timeout = timeout)
 
     def process(self, argv=None, executable=None, tty=True, cwd=None, env=None, timeout=Timeout.default, run=True,
-                stdin=0, stdout=1, stderr=2, preexec_fn=None, preexec_args=[], raw=True, aslr=None, setuid=None):
+                stdin=0, stdout=1, stderr=2, preexec_fn=None, preexec_args=[], raw=True, aslr=None, setuid=None,
+                shell=False):
         r"""
         Executes a process on the remote server, in the same fashion
         as pwnlib.tubes.process.process.
@@ -680,6 +680,8 @@ class ssh(Timeout, Logger):
                 See ``pwnlib.tubes.process.process`` for more information.
             setuid(bool):
                 See ``pwnlib.tubes.process.process`` for more information.
+            shell(bool):
+                Pass the command-line arguments to the shell.
 
         Returns:
             A new SSH channel, or a path to a script if ``run=False``.
@@ -737,6 +739,9 @@ class ssh(Timeout, Logger):
             Traceback (most recent call last):
             ...
             NameError: global name 'bar' is not defined
+
+            >>> s.process('echo hello', shell=True).recvall()
+            'hello\n'
         """
         if not argv and not executable:
             self.error("Must specify argv or executable")
@@ -750,6 +755,11 @@ class ssh(Timeout, Logger):
         if not isinstance(argv, (list, tuple)):
             self.error('argv must be a list or tuple')
 
+        if shell:
+            if len(argv) != 1:
+                self.error('Cannot provide more than 1 argument if shell=True')
+            argv = ['/bin/sh', '-c'] + argv
+
         # Python doesn't like when an arg in argv contains '\x00'
         # -> execve() arg 2 must contain only strings
         for i, arg in enumerate(argv):
@@ -862,6 +872,13 @@ if not %(aslr)r:
 
     resource.setrlimit(resource.RLIMIT_STACK, (-1, -1))
 
+# Attempt to dump ALL core file regions
+try:
+    with open('/proc/self/coredump_filter', 'w') as core_filter:
+        core_filter.write('0x3f\n')
+except Exception:
+    pass
+
 # Assume that the user would prefer to have core dumps.
 resource.setrlimit(resource.RLIMIT_CORE, (-1, -1))
 
@@ -941,7 +958,7 @@ os.execve(exe, argv, os.environ)
 
         return result
 
-    def system(self, process, tty = True, wd = None, env = None, timeout = Timeout.default, raw = True):
+    def system(self, process, tty = True, wd = None, env = None, timeout = None, raw = True):
         r"""system(process, tty = True, wd = None, env = None, timeout = Timeout.default, raw = True) -> ssh_channel
 
         Open a new channel with a specific process inside. If `tty` is True,
@@ -967,7 +984,10 @@ os.execve(exe, argv, os.environ)
         if wd is None:
             wd = self.cwd
 
-        return ssh_channel(self, process, tty, wd, env, timeout, level = self.level, raw = raw)
+        if timeout is None:
+            timeout = self.timeout
+
+        return ssh_channel(self, process, tty, wd, env, timeout = timeout, level = self.level, raw = raw)
 
     #: Backward compatibility.  Use :meth:`system`
     run = system

pwnlib/tubes/tube.py

@@ -30,18 +30,18 @@ class tube(Timeout, Logger):
     #: and related functions.
     newline = '\n'
 
-    def __init__(self, timeout = default, level = None):
+    def __init__(self, timeout = default, level = None, *a, **kw):
         super(tube, self).__init__(timeout)
 
         Logger.__init__(self, None)
         if level is not None:
             self.setLevel(level)
 
-        self.buffer          = Buffer()
+        self.buffer = Buffer(*a, **kw)
         atexit.register(self.close)
 
     # Functions based on functions from subclasses
-    def recv(self, numb = 4096, timeout = default):
+    def recv(self, numb = None, timeout = default):
         r"""recv(numb = 4096, timeout = default) -> str
 
         Receives up to `numb` bytes of data from the tube, and returns
@@ -72,6 +72,7 @@ class tube(Timeout, Logger):
             [...] Received 0xc bytes:
                 'Hello, world'
         """
+        numb = self.buffer.get_fill_size(numb)
         return self._recv(numb, timeout) or ''
 
     def unrecv(self, data):
@@ -122,7 +123,7 @@ class tube(Timeout, Logger):
         data = ''
 
         with self.local(timeout):
-            data = self.recv_raw(4096)
+            data = self.recv_raw(self.buffer.get_fill_size())
 
         if data and self.isEnabledFor(logging.DEBUG):
             self.debug('Received %#x bytes:' % len(data))
@@ -141,12 +142,13 @@ class tube(Timeout, Logger):
         return data
 
 
-    def _recv(self, numb = 4096, timeout = default):
+    def _recv(self, numb = None, timeout = default):
         """_recv(numb = 4096, timeout = default) -> str
 
         Receives one chunk of from the internal buffer or from the OS if the
         buffer is empty.
         """
+        numb = self.buffer.get_fill_size(numb)
         data = ''
 
         # No buffered data, could not put anything in the buffer

pwnlib/util/proc.py

@@ -2,19 +2,14 @@ import errno
 import socket
 import time
 
+import psutil
+
 from .. import tubes
 from ..log import getLogger
 
-try:
-    import psutil
-    _ok_import = True
-except ImportError:
-    _ok_import = False
-
 log = getLogger(__name__)
 
-if _ok_import:
-    all_pids = psutil.pids
+all_pids = psutil.pids
 
 def pidof(target):
     """pidof(target) -> int list
@@ -305,56 +300,3 @@ def wait_for_debugger(pid):
         while tracer(pid) is None:
             time.sleep(0.01)
         l.success()
-
-if not _ok_import:
-    def _make_stub(func):
-        func.__doc__ = 'Stubbed out function, because psutil is not available.'
-        return func
-
-    @_make_stub
-    def all_pids():
-        log.error("Called stubbed-out function. Get psutil to work on your platform, then come back.")
-
-    @_make_stub
-    def pidof(target):
-        log.error("Called stubbed-out function. Get psutil to work on your platform, then come back.")
-
-    @_make_stub
-    def pid_by_name(name):
-        log.error("Called stubbed-out function. Get psutil to work on your platform, then come back.")
-
-    @_make_stub
-    def name(pid):
-        log.error("Called stubbed-out function. Get psutil to work on your platform, then come back.")
-
-    @_make_stub
-    def parent(pid):
-        log.error("Called stubbed-out function. Get psutil to work on your platform, then come back.")
-
-    @_make_stub
-    def children(ppid):
-        log.error("Called stubbed-out function. Get psutil to work on your platform, then come back.")
-
-    @_make_stub
-    def ancestors(pid):
-        log.error("Called stubbed-out function. Get psutil to work on your platform, then come back.")
-
-    @_make_stub
-    def descendants(pid):
-        log.error("Called stubbed-out function. Get psutil to work on your platform, then come back.")
-
-    @_make_stub
-    def exe(pid):
-        log.error("Called stubbed-out function. Get psutil to work on your platform, then come back.")
-
-    @_make_stub
-    def cwd(pid):
-        log.error("Called stubbed-out function. Get psutil to work on your platform, then come back.")
-
-    @_make_stub
-    def cmdline(pid):
-        log.error("Called stubbed-out function. Get psutil to work on your platform, then come back.")
-
-    @_make_stub
-    def starttime(pid):
-        log.error("Called stubbed-out function. Get psutil to work on your platform, then come back.")

pwnlib/version.py

@@ -1 +1 @@
-__version__ = '3.3.4'
+__version__ = '3.4.0'

setup.py

@@ -54,11 +54,8 @@ install_requires     = ['paramiko>=1.15.2',
                         'pysocks',
                         'python-dateutil',
                         'pypandoc',
-                        'packaging']
-
-# This is a hack until somebody ports psutil to OpenBSD
-if platform.system() != 'OpenBSD':
-    install_requires.append('psutil>=2.1.3')
+                        'packaging',
+                        'psutil>=3.3.0']
 
 # Check that the user has installed the Python development headers
 PythonH = os.path.join(get_python_inc(), 'Python.h')
@@ -83,7 +80,7 @@ except ImportError:
 setup(
     name                 = 'pwntools',
     packages             = find_packages(),
-    version              = '3.3.4',
+    version              = '3.4.0',
     data_files           = [('',
                              glob.glob('*.md') + glob.glob('*.txt')),
                             ],

travis/install.sh

@@ -1,5 +1,6 @@
 #!/usr/bin/env bash -e
 set -e
+
 local_deb_extract()
 {
     wget $1
@@ -8,52 +9,54 @@ local_deb_extract()
     rm -f *.tar.* *deb*
 }
 
-get_binutils()
+install_deb()
 {
-    BINUTILS_PREFIX='https://launchpad.net/~pwntools/+archive/ubuntu/binutils/+files/binutils-'
-    BINUTILS_SUFFIX='-linux-gnu_2.22-6ubuntu1.1cross0.11pwntools12~precise_amd64.deb'
-    local_deb_extract "${BINUTILS_PREFIX}${1}${BINUTILS_SUFFIX}"
-}
-
-get_qemu()
-{
-    echo "Installing qemu"
-    QEMU_INDEX='http://packages.ubuntu.com/en/yakkety/amd64/qemu-user-static/download'
-    QEMU_URL=$(curl "$QEMU_INDEX" | grep kernel.org | grep -Eo 'http.*\.deb')
-    local_deb_extract "$QEMU_URL"
+    version=${2:-zesty}
+    package=$1
+    echo "Installing $package"
+    INDEX="http://packages.ubuntu.com/en/$version/amd64/$package/download"
+    URL=$(curl "$INDEX" | grep -Eo "https?://.*$package.*\.deb" | head -1)
+    local_deb_extract "$URL"
 }
 
 setup_travis()
 {
     export PATH=$PWD/usr/bin:$PATH
-    export LD_LIBRARY_PATH=$PWD/usr/lib
+    export LD_LIBRARY_PATH=$PWD/usr/lib:$LD_LIBRARY_PATH
+    export LD_LIBRARY_PATH=$PWD/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH
 
-    # Install libbfd-multiarch and libopcodes-multiarch if not found in the cache
-    if [ ! -f usr/lib/libbfd-2.22-multiarch.so ];
-    then
-        # Install the multiarch binutils
-        local_deb_extract http://mirrors.mit.edu/ubuntu/pool/universe/b/binutils/binutils-multiarch_2.22-6ubuntu1_amd64.deb
-        pushd usr/lib
-        ln -sf libbfd-2.22-multiarch.so libbfd-2.22.so
-        ln -sf libopcodes-2.22-multiarch.so libopcodes-2.22.so
-        popd
-    fi
+    # Install a more modern binutils, which is required for some of the tests
+    [[ -f usr/bin/objcopy ]] || install_deb binutils
 
     # Install/upgrade qemu
-    if ! (which qemu-arm-static && qemu-arm-static -version | grep 2.6.1); then
-        get_qemu
-    fi
+    [[ -f usr/bin/qemu-arm-static ]] || install_deb qemu-user-static xenial
 
-    # Install our custom binutils
-    which arm-linux-gnu-as     || get_binutils arm
-    which mips-linux-gnu-as    || get_binutils mips
-    which powerpc-linux-gnu-as || get_binutils powerpc
+    # Install cross-binutils
+    [[ -f usr/bin/x86_64-linux-gnu-ar ]]    || install_deb binutils-multiarch
+    [[ -f usr/bin/aarch64-linux-gnu-as ]]   || install_deb binutils-aarch64-linux-gnu
+    [[ -f usr/bin/arm-linux-gnueabihf-as ]] || install_deb binutils-arm-linux-gnueabihf
+    [[ -f usr/bin/mips-linux-gnu-as ]]      || install_deb binutils-mips-linux-gnu
+    [[ -f usr/bin/powerpc-linux-gnu-as ]]   || install_deb binutils-powerpc-linux-gnu
 
     # Test that the installs worked
-    which arm-linux-gnu-as
-    which mips-linux-gnu-as
-    which powerpc-linux-gnu-as
-    which qemu-arm-static
+    as                      --version
+    x86_64-linux-gnu-ar     --version
+    aarch64-linux-gnu-as    --version
+    arm-linux-gnueabihf-as  --version
+    mips-linux-gnu-as       --version
+    powerpc-linux-gnu-as    --version
+    qemu-arm-static         --version
+
+    # Force-install capstone because it's broken somehow
+    [[ -f usr/lib/libcapstone.so.3 ]] || install_deb libcapstone3
+
+    # Install a newer copy of GDB
+    if [[ ! -f usr/bin/gdb ]]; then
+        git clone --depth=1 https://github.com/zachriggle/pwntools-gdb-travis-ci.git
+        tar xf pwntools-gdb-travis-ci/gdb.tar.xz
+        which gdb
+        usr/bin/gdb --version
+    fi
 
     # Get rid of files we don't want cached
     rm -rf usr/share
@@ -74,8 +77,14 @@ setup_android_emulator()
     if [ -n "$TRAVIS" ]; then
         if [ -z "$TRAVIS_COMMIT_RANGE" ]; then
             echo "TRAVIS_COMMIT_RANGE is empty, forcing Android Emulator installation"
-        elif ! git show "$TRAVIS_COMMIT_RANGE" ; then
+        elif ! (git show "$TRAVIS_COMMIT_RANGE" >/dev/null) ; then
             echo "TRAVIS_COMMIT_RANGE is invalid, forcing Android Emulator installation"
+        elif [[ "$TRAVIS_BRANCH" =~ "staging" ]]; then
+            echo "TRAVIS_BRANCH ($TRAVIS_BRANCH) indicates a branch we care about"
+            echo "Forcing Android Emulator installation"
+        elif [[ -n "$TRAVIS_TAG" ]]; then
+            echo "TRAVIS_TAG ($TRAVIS_TAG) indicates a new relase"
+            echo "Forcing Android Emulator installation"
         elif (git log --stat "$TRAVIS_COMMIT_RANGE" | grep -iE "android|adb"); then
             echo "Found Android-related commits, forcing Android Emulator installation"
         else
@@ -83,9 +92,11 @@ setup_android_emulator()
             # emulator, while still leaving the code intact, we remove the
             # RST file that Sphinx searches.
             rm -f 'docs/source/adb.rst'
+            rm -f 'docs/source/protocols/adb.rst'
 
             # However, the file needs to be present or else things break.
             touch 'docs/source/adb.rst'
+            touch 'docs/source/protocols/adb.rst'
 
             echo "Skipping Android emulator install, Android tests disabled."
             return
@@ -182,5 +193,4 @@ elif [[ "$(uname)" == "Linux" ]]; then
     setup_android_emulator
 fi
 
-
-dpkg -l
+set +e

travis/ssh_setup.sh

@@ -54,3 +54,5 @@ Host example.pwnme
 EOF
 
 ssh -o "StrictHostKeyChecking no" -vvvv [email protected] id
+
+set +e