Release 3.3.0beta0

Also fix other incorrect log usage

Fixes Gallopsled/pwntools#786

.github/ISSUE_TEMPLATE.md

@@ -6,4 +6,4 @@ When reporting an issue, be sure that you are running the latest released versio
 
 Please verify that your issue occurs on 64-bit Ubuntu 14.04.  You can use the Dockerfile in `extra/docker/Dockerfile` to save time.
 
-If possible, provide a proof-of-concept which demonstrates the problem.  Include any binaries or scripts necessary to reproduce the issue.
+If possible, provide a proof-of-concept which demonstrates the problem.  Include any binaries or scripts necessary to reproduce the issue, and please include the full debug output via setting the environment variable `PWNLIB_DEBUG=1`.

CHANGELOG.md

@@ -9,9 +9,11 @@ The table below shows which release corresponds to each branch, and what date th
 
 | Version          | Branch   | Release Date           |
 | ---------------- | -------- | ---------------------- |
-| [3.3.0](#330)    | `dev`    | Dec 24, 2016 (planned)
-| [3.2.0](#320)    | `beta`   | Nov 12, 2016 (planned)
-| [3.1.0](#310)    | `stable` | Oct 2, 2016
+| [3.4.0](#340)    | `dev`    | Feb 4, 2017 (planned)
+| [3.3.0](#330)    | `beta`   | Dec 24, 2016 (planned)
+| [3.2.0](#320)    | `stable` | Nov 12, 2016
+| [3.1.1](#311)    |          | Oct 23, 2016
+| [3.1.0](#310)    |          | Oct 2, 2016
 | [3.0.4](#304)    |          | Sept 19, 2016
 | [3.0.3](#303)    |          | Sept 18, 2016
 | [3.0.2](#302)    |          | Sept 6, 2016
@@ -19,9 +21,22 @@ The table below shows which release corresponds to each branch, and what date th
 | [3.0.0](#300)    |          | Aug 20, 2016
 | [2.2.0](#220)    |          | Jan 5, 2015
 
+## 3.4.0
+
+To be released on Feb 4, 2017.
+
+## 3.3.0
+
+- [b198ec8][b198ec8] Added `tube.stream()` function, which is like `tube.interact()` without a prompt or keyboard input.
+
+[b198ec8]: https://github.com/Gallopsled/pwntools/commit/b198ec8
+
 ## 3.2.0
 
 - [#695][695] Fixed a performance regression in `phd`.
+- [452605e][452605e] Fixed [#629][#629] related to correct removal of temporary files.
+- [ea94ee4][ea94ee4] Disallows semi-colons in for the `run_in_terminal` function, since it did not work properly in all cases. 
+- [6376d07][6376d07] Added the mips shellcode `pushstr_array`.
 - [#700][700] Added missing MIPS shellcode documentation to readthedocs, and enabled unit tests
 - [#701][701] Command line tools refactored to have a common `pwn` entry point.
     + Added an option to *not* install the traditional `asm`, `disasm`, `checksec`, etc scripts
@@ -51,6 +66,7 @@ The table below shows which release corresponds to each branch, and what date th
 - [2af55c9][2af55c9] `AdbDevice` objects exposed via e.g. `adb.devices()` now offer scoped access to all `adb` module properties
     + It is now possible to e.g. `map(lambda d: d.process(['id']).recvall(), adb.devices())`
 
+
 [695]: https://github.com/Gallopsled/pwntools/pull/695
 [700]: https://github.com/Gallopsled/pwntools/pull/700
 [701]: https://github.com/Gallopsled/pwntools/pull/701
@@ -60,11 +76,18 @@ The table below shows which release corresponds to each branch, and what date th
 [709]: https://github.com/Gallopsled/pwntools/pull/709
 [705]: https://github.com/Gallopsled/pwntools/pull/705
 [703]: https://github.com/Gallopsled/pwntools/pull/703
+[452605e]: https://github.com/Gallopsled/pwntools/commit/452605e854f4870ef5ccfdf7fb110dfd75c50feb
+[ea94ee4]: https://github.com/Gallopsled/pwntools/commit/ea94ee4ca5a8060567cc9bd0dc33796a89ad0b95
+[6376d07]: https://github.com/Gallopsled/pwntools/commit/6376d072660fb2250f48bd22629bbd7e3c61c758
 [1e414af]: https://github.com/Gallopsled/pwntools/commit/1e414afbeb3a01242f4918f111febaa63b640eb7
 [869ec42]: https://github.com/Gallopsled/pwntools/commit/869ec42082b4b98958dfe85103da9b101dde7daa
 [4893819]: https://github.com/Gallopsled/pwntools/commit/4893819b4c23182da570e2f4ea4c14d73af2c0df
 [2af55c9]: https://github.com/Gallopsled/pwntools/commit/2af55c9bc382eca23f89bc0abc7a07c075521f94
 
+## 3.1.1
+
+Fixed a bug in `MemLeak.struct` (PR: #768).
+
 ## 3.1.0
 
 A number of smaller bugfixes and documentation tweaks.

README.md

@@ -2,7 +2,7 @@
 ![pwntools logo](https://github.com/Gallopsled/pwntools/blob/stable/docs/source/logo.png?raw=true)
 
 [![Docs](https://readthedocs.org/projects/pwntools/badge/?version=stable)](https://docs.pwntools.com/)
-[![PyPI](https://img.shields.io/badge/pypi-v3.1.0-green.svg?style=flat)](https://pypi.python.org/pypi/pwntools/)
+[![PyPI](https://img.shields.io/badge/pypi-v3.2.0-green.svg?style=flat)](https://pypi.python.org/pypi/pwntools/)
 [![Travis](https://travis-ci.org/Gallopsled/pwntools.svg)](https://travis-ci.org/Gallopsled/pwntools)
 [![Coveralls](https://img.shields.io/coveralls/Gallopsled/pwntools/dev.svg)](https://coveralls.io/github/Gallopsled/pwntools?branch=dev)
 [![Twitter](https://img.shields.io/badge/twitter-pwntools-4099FF.svg?style=flat)](https://twitter.com/pwntools)

TESTING.md

@@ -5,7 +5,7 @@ Pwntools makes extensive use of unit tests and integration tests to ensure every
 ## Test Suite
 
 
-To run the test suite, it is beste to use Ubuntu 12.04 or 14.04, and run the following commands.  **Be aware** that this will add a user to the machine, and create a public key for SSH login!
+To run the test suite, it is best to use Ubuntu 12.04 or 14.04, and run the following commands.  **Be aware** that this will add a user to the machine, and create a public key for SSH login!
 
 ```sh
 bash travis/install.sh

docs/requirements.txt

@@ -1,5 +1,5 @@
 sphinxcontrib-napoleon
-sphinxcontrib-autoprogram
+sphinxcontrib-autoprogram==0.1.2
 isort
 paramiko>=1.15.2
 mako>=1.0.0

docs/source/args.rst

@@ -0,0 +1,9 @@
+.. testsetup:: *
+
+   from pwn import *
+
+:mod:`pwnlib.args` --- Magic Command-Line Arguments
+=====================================================
+
+.. automodule:: pwnlib.args
+   :members:

docs/source/conf.py

@@ -55,6 +55,16 @@ pwnlib.context.ContextType.defaults['randomize'] = False
 pwnlib.term.text.when = 'never'
 pwnlib.log.install_default_handler()
 pwnlib.log.rootlogger.setLevel(1)
+
+# Sphinx modifies sys.stdout, and context.log_terminal has
+# a reference to the original instance.  We need to update
+# it for logging to be captured.
+class stdout(object):
+    def __getattr__(self, name):
+        return getattr(sys.stdout, name)
+    def __setattr__(self, name, value):
+        return setattr(sys.stdout, name, value)
+pwnlib.context.ContextType.defaults['log_console'] = stdout()
 '''
 
 autodoc_member_order = 'alphabetical'

docs/source/index.rst

@@ -43,6 +43,7 @@ Each of the ``pwntools`` modules is documented here.
    :glob:
 
    adb
+   args
    asm
    atexception
    atexit

extra/bash_completion.d/shellcraft

@@ -1,5 +1,6 @@
+#!/usr/bin/env bash
 # cache list of shellcodes
-_shellcraft_shellcodes=$(shellcraft)
+_shellcraft_shellcodes=$(pwn shellcraft -l)
 
 _shellcraft()
 {
@@ -44,3 +45,4 @@ _shellcraft()
 }
 
 complete -F _shellcraft pwn shellcraft
+complete -F _shellcraft shellcraft

pwn/__init__.py

@@ -13,4 +13,5 @@ if not platform.architecture()[0].startswith('64'):
     See Gallopsled/pwntools#518 for more information."""
     log.warn_once('Pwntools does not support 32-bit Python.  Use a 64-bit release.')
 
-pwnlib.update.check_automatically()
+with context.local(log_console=sys.stderr):
+    pwnlib.update.check_automatically()

pwn/toplevel.py

@@ -51,6 +51,7 @@ from pwnlib.util import iters
 from pwnlib.util import net
 from pwnlib.util import proc
 from pwnlib.util import safeeval
+from pwnlib.util.crc import BitPolynom
 from pwnlib.util.cyclic import *
 from pwnlib.util.fiddling import *
 from pwnlib.util.getdents import *

pwnlib/adb/adb.py

@@ -10,12 +10,38 @@ If you have multiple devices, you have a handful of options to select one, or it
 over the devices.
 
 First and most important is the ``context.device`` property, which declares the "currently"
-seelcted device in any scope.  It can be set manually to a serial number, or to a ``Device``
+selected device in any scope.  It can be set manually to a serial number, or to a ``Device``
 instance.
 
 .. code-block:: python
 
-    >>> context.device = ''
+    # Take the first available device
+    context.device = adb.wait_for_device()
+
+    # Set a device by serial number
+    context.device = 'ZX1G22LH8S'
+
+    # Set a device by its product name
+    for device in adb.devices():
+        if device.product == 'shamu':
+            break
+    else:
+        error("Could not find any shamus!")
+
+Once a device is selected, you can operate on it with any of the functions in
+the ``pwnlib.adb`` module.
+
+.. code-block:: python
+
+    # Get a process listing
+    print adb.process(['ps']).recvall()
+
+    # Fetch properties
+    print adb.properties.ro.build.fingerprint
+
+    # Read and write files
+    print adb.read('/proc/version')
+    adb.write('/data/local/tmp/foo', 'my data')
 
 """
 import functools
@@ -162,6 +188,29 @@ def reboot_bootloader():
     with Client() as c:
         c.reboot_bootloader()
 
+@with_device
+def uptime():
+    """uptime() -> float
+
+    Returns:
+        Uptime of the device, in seconds
+    """
+    up, idle = map(float, read('/proc/uptime').split())
+    return up
+
+@with_device
+def boot_time():
+    """boot_time() -> int
+
+    Returns:
+        Boot time of the device, in Unix time, rounded to the
+        nearest second.
+    """
+    for line in read('/proc/stat').splitlines():
+        name, value = line.split(None, 1)
+        if name == 'btime':
+            return int(value)
+
 class AdbDevice(Device):
     """Encapsulates information about a connected device."""
     def __init__(self, serial, type, port=None, product='unknown', model='unknown', device='unknown', features=None, **kw):
@@ -1036,13 +1085,12 @@ def uninstall(package, *arguments):
         package(str): Name of the package to uninstall (e.g. ``'com.foo.MyPackage'``)
         arguments: Supplementary arguments to ``'pm install'``, e.g. ``'-k'``.
     """
-    with log.progress("Installing package {}".format(package)):
+    with log.progress("Uninstalling package {}".format(package)):
         with context.quiet:
-            return process(['pm','uninstall',apk] + list(arguments)).recvall()
+            return process(['pm','uninstall',package] + list(arguments)).recvall()
 
 @context.quiet
 def packages():
     """Returns a list of packages installed on the system"""
     packages = process(['pm', 'list', 'packages']).recvall()
     return [line.split('package:', 1)[-1] for line in packages.splitlines()]
-

pwnlib/args.py

@@ -1,5 +1,43 @@
 #!/usr/bin/env python2
 """
+Pwntools exposes several magic command-line arguments and environment
+variables when operating in `from pwn import *` mode.
+
+The arguments extracted from the command-line and removed from ``sys.argv``.
+
+Arguments can be set by appending them to the command-line, or setting
+them in the environment prefixed by ``PWNLIB_``.
+
+The easiest example is to enable more verbose debugging.  Just set ``DEBUG``.
+
+.. code-block:: bash
+
+    $ PWNLIB_DEBUG=1 python exploit.py
+    $ python exploit.py DEBUG
+
+These arguments are automatically extracted, regardless of their name, and
+exposed via ``pwnlib.args.args``, which is exposed as the global variable
+``args``.  Arguments which ``pwntools`` reserves internally are not exposed
+this way.
+
+.. code-block:: bash
+
+    $ python -c 'from pwn import *; print args' A=1 B=Hello HOST=1.2.3.4 DEBUG
+    defaultdict(<type 'str'>, {'A': '1', 'HOST': '1.2.3.4', 'B': 'Hello'})
+
+This is very useful for conditional code, for example determining whether to
+run an exploit locally or to connect to a remote server.  Arguments which are
+not specified evaluate to an empty string.
+
+.. code-block:: python
+
+    if args['REMOTE']:
+        io = remote('exploitme.com', 4141)
+    else:
+        io = process('./pwnable')
+
+The full list of supported "magic arguments" and their effects are listed
+below.
 """
 import collections
 import logging
@@ -42,46 +80,67 @@ def asbool(s):
     else:
         raise ValueError('must be integer or boolean: %r' % s)
 
-def set_log_level(x):
+def LOG_LEVEL(x):
+    """Sets the logging verbosity used via ``context.log_level``,
+    e.g. ``LOG_LEVEL=debug``.
+    """
     with context.local(log_level=x):
         context.defaults['log_level']=context.log_level
 
-def set_log_file(x):
+def LOG_FILE(x):
+    """Sets a log file to be used via ``context.log_file``, e.g.
+    ``LOG_FILE=./log.txt``"""
     context.log_file=x
 
-def set_log_level_error(x):
-    set_log_level('error')
+def SILENT(x):
+    """Sets the logging verbosity to ``error`` which silences most
+    output."""
+    LOG_LEVEL('error')
 
-def set_log_level_debug(x):
-    set_log_level('debug')
+def DEBUG(x):
+    """Sets the logging verbosity to ``debug`` which displays much
+    more information, including logging each byte sent by tubes."""
+    LOG_LEVEL('debug')
 
-def set_noterm(v):
+def NOTERM(v):
+    """Disables pretty terminal settings and animations."""
     if asbool(v):
         global term_mode
         term_mode = False
 
-def set_timeout(v):
+def TIMEOUT(v):
+    """Sets a timeout for tube operations (in seconds) via
+    ``context.timeout``, e.g. ``TIMEOUT=30``"""
     context.defaults['timeout'] = int(v)
 
-def set_randomize(v):
+def RANDOMIZE(v):
+    """Enables randomization of various pieces via ``context.randomize``"""
     context.defaults['randomize'] = asbool(v)
 
-def set_aslr(v):
+def NOASLR(v):
+    """Disables ASLR via ``context.aslr``"""
     context.defaults['aslr'] = not asbool(v)
 
-def set_noptrace(v):
+def NOPTRACE(v):
+    """Disables facilities which require ``ptrace`` such as ``gdb.attach()``
+    statements, via ``context.noptrace``."""
     context.defaults['noptrace'] = asbool(v)
 
+def STDERR(v):
+    """Sends logging to ``stderr`` by default, instead of ``stdout``"""
+    context.log_console = sys.stderr
+
 hooks = {
-    'LOG_LEVEL': set_log_level,
-    'LOG_FILE': set_log_file,
-    'DEBUG': set_log_level_debug,
-    'NOTERM': set_noterm,
-    'SILENT': set_log_level_error,
-    'RANDOMIZE': set_randomize,
-    'TIMEOUT': set_timeout,
-    'NOASLR': set_aslr,
-    'NOPTRACE': set_noptrace,
+    'LOG_LEVEL': LOG_LEVEL,
+    'LOG_FILE': LOG_FILE,
+    'DEBUG': DEBUG,
+    'NOTERM': NOTERM,
+    'SILENT': SILENT,
+    'RANDOMIZE': RANDOMIZE,
+    'TIMEOUT': TIMEOUT,
+    'NOASLR': NOASLR,
+    'NOPTRACE': NOPTRACE,
+    'STDERR': STDERR,
 }
 
 def initialize():

pwnlib/commandline/common.py

@@ -5,8 +5,6 @@ import sys
 import pwnlib
 from pwnlib.context import context
 
-pwnlib.log.console.stream = sys.stderr
-
 choices = map(str, [16,32,64])
 choices += list(context.oses)
 choices += list(context.architectures)

pwnlib/commandline/main.py

@@ -1,3 +1,5 @@
+import sys
+
 from . import asm
 from . import checksec
 from . import common
@@ -15,6 +17,7 @@ from . import shellcraft
 from . import unhex
 from . import update
 from .common import parser
+from ..context import context
 
 commands = {
     'asm': asm.main,
@@ -36,7 +39,8 @@ commands = {
 
 def main():
     args = parser.parse_args()
-    commands[args.command](args)
+    with context.local(log_console = sys.stderr):
+        commands[args.command](args)
 
 if __name__ == '__main__':
     main()

pwnlib/commandline/shellcraft.py

@@ -57,9 +57,10 @@ p.add_argument(
                'p',
                'i', 'hexii',
                'e', 'elf',
+               'd', 'escaped',
                'default'],
     default = 'default',
-    help = 'Output format (default: hex), choose from {r}aw, {s}tring, {c}-style array, {h}ex string, hex{i}i, {a}ssembly code, {p}reprocssed code',
+    help = 'Output format (default: hex), choose from {e}lf, {r}aw, {s}tring, {c}-style array, {h}ex string, hex{i}i, {a}ssembly code, {p}reprocssed code, escape{d} hex string',
 )
 
 p.add_argument(
@@ -329,7 +330,8 @@ def main(args):
         code = pwnlib.util.fiddling.enhex(code) + '\n'
     elif args.format in ['i', 'hexii']:
         code = hexii(code) + '\n'
-
+    elif args.format in ['d', 'escaped']:
+        code = ''.join('\\x%02x' % ord(c) for c in code) + '\n'
     if not sys.stdin.isatty():
         args.out.write(sys.stdin.read())
 

pwnlib/commandline/unhex.py

@@ -20,7 +20,7 @@ def main(args):
             s = sys.stdin.read().translate(None, whitespace)
             sys.stdout.write(s.decode('hex'))
         else:
-            sys.stdout.write(''.join(sys.argv[1:]).decode('hex'))
+            sys.stdout.write(''.join(args.hex).decode('hex'))
     except TypeError, e:
         sys.stderr.write(str(e) + '\n')
 

pwnlib/context/__init__.py

@@ -334,6 +334,7 @@ class ContextType(object):
         'kernel': None,
         'log_level': logging.INFO,
         'log_file': _devnull(),
+        'log_console': sys.stdout,
         'randomize': False,
         'newline': '\n',
         'noptrace': False,
@@ -922,6 +923,24 @@ class ContextType(object):
         value.flush()
         return value
 
+    @_validator
+    def log_console(self, stream):
+        """
+        Sets the default logging console target.
+
+        Examples:
+
+            >>> context.log_level = 'warn'
+            >>> log.warn("Hello")
+            [!] Hello
+            >>> context.log_console=open('/dev/null', 'w')
+            >>> log.warn("Hello")
+            >>> context.clear()
+        """
+        if isinstance(stream, str):
+            stream = open(stream, 'wt')
+        return stream
+
     @property
     def mask(self):
         return (1 << self.bits) - 1

pwnlib/data/useragents/download-useragents.py

@@ -7,7 +7,6 @@ import re
 import urllib
 
 from bs4 import BeautifulSoup
-
 from pwn import *
 
 uas = set()

pwnlib/elf/corefile.py

@@ -6,6 +6,7 @@ from elftools.common.py3compat import bytes2str
 from elftools.common.utils import roundup
 from elftools.common.utils import struct_parse
 from elftools.construct import CString
+
 from ..context import context
 from ..log import getLogger
 from ..tubes.tube import tube

pwnlib/elf/elf.py

@@ -1,5 +1,6 @@
 """Exposes functionality for manipulating ELF files
 """
+import codecs
 import mmap
 import os
 import subprocess
@@ -30,7 +31,18 @@ log = getLogger(__name__)
 
 __all__ = ['load', 'ELF']
 
-Function = namedtuple('Function', 'address size')
+class Function(object):
+    def __init__(self, name, address, size):
+        self.name    = name
+        self.address = address
+        self.size    = size
+    def __repr__(self):
+        return '%s(name=%r, address=%#x, size=%#x)' % (
+            self.__class__.__name__,
+            self.name,
+            self.address,
+            self.size
+            )
 
 def load(*args, **kwargs):
     """Compatibility wrapper for pwntools v1"""
@@ -348,11 +360,15 @@ class ELF(ELFFile):
                     continue
                 if sym.entry.st_info['type'] == 'STT_FUNC' and sym.entry.st_size != 0:
                     name = sym.name
+                    try:
+                        name = codecs.encode(name, 'latin-1')
+                    except Exception:
+                        pass
                     if name not in self.symbols:
                         continue
                     addr = self.symbols[name]
                     size = sym.entry.st_size
-                    self.functions[name] = Function(addr, size)
+                    self.functions[name] = Function(name, addr, size)
 
     def _populate_symbols(self):
         """

pwnlib/flag/flag.py

@@ -3,8 +3,8 @@
 import os
 
 from ..args import args
-from ..tubes.remote import remote
 from ..log import getLogger
+from ..tubes.remote import remote
 
 env_server  = args.get('FLAG_HOST', 'flag-submission-server').strip()
 env_port    = args.get('FLAG_PORT', '31337').strip()

pwnlib/log.py

@@ -113,7 +113,7 @@ from .term import spinners
 from .term import text
 
 # list of prefixes to use for the different message types.  note that the `text`
-# module won't add any escape codes if `sys.stderr.isatty()` is `False`
+# module won't add any escape codes if `pwnlib.context.log_console.isatty()` is `False`
 _msgtype_prefixes = {
     'status'       : [text.magenta, 'x'],
     'success'      : [text.bold_green, '+'],
@@ -490,10 +490,14 @@ class Handler(logging.StreamHandler):
     logger will not be emitted but rather an animated progress line will be
     created.
 
-    This handler outputs to ``sys.stderr``.
-
     An instance of this handler is added to the ``'pwnlib'`` logger.
     """
+    @property
+    def stream(self):
+        return context.log_console
+    @stream.setter
+    def stream(self, value):
+        pass
     def emit(self, record):
         """
         Emit a log record or create/update an animated progress logger
@@ -652,7 +656,7 @@ log_file.setFormatter(logging.Formatter(fmt, iso_8601))
 #     logger.addHandler(myCoolPitchingHandler)
 #
 rootlogger = getLogger('pwnlib')
-console   = Handler(sys.stdout)
+console   = Handler()
 formatter = Formatter()
 console.setFormatter(formatter)
 
@@ -663,8 +667,6 @@ def install_default_handler():
     the ``pwnlib`` root logger.  This function is automatically called from when
     importing :mod:`pwn`.
     '''
-    console.stream = sys.stdout
-
     logger         = logging.getLogger('pwnlib')
 
     if console not in logger.handlers:

pwnlib/memleak.py

@@ -1,3 +1,4 @@
+import ctypes
 import functools
 import string
 
@@ -96,8 +97,18 @@ class MemLeak(object):
             struct(class): A ctypes structure to be instantiated with leaked data
         Return Value:
             An instance of the provided struct class, with the leaked data decoded
+
+        Examples:
+
+            >>> @pwnlib.memleak.MemLeak
+            ... def leaker(addr):
+            ...     return "A"
+            >>> e = leaker.struct(0, pwnlib.elf.Elf32_Phdr)
+            >>> hex(e.p_paddr)
+            '0x41414141'
+
         """
-        size = sizeof(struct)
+        size = ctypes.sizeof(struct)
         data = self.n(address, size)
         obj = struct.from_buffer_copy(data)
         return obj

pwnlib/rop/rop.py

@@ -700,6 +700,9 @@ class ROP(object):
         # If we can find a function with that name, just call it
         if isinstance(resolvable, str):
             addr = self.resolve(resolvable)
+        elif hasattr(resolvable, 'name') and hasattr(resolvable, 'address'):
+            addr = resolvable.address
+            resolvable = str(resolvable.name)
         else:
             addr = resolvable
             resolvable = ''

pwnlib/tubes/process.py

@@ -197,7 +197,7 @@ class process(tube):
     #: Have we seen the process stop?
     _stop_noticed = False
 
-    def __init__(self, argv,
+    def __init__(self, argv = None,
                  shell = False,
                  executable = None,
                  cwd = None,
@@ -217,6 +217,14 @@ class process(tube):
                  alarm = None):
         super(process, self).__init__(timeout, level = level)
 
+        # Permit using context.binary
+        if argv is None:
+            if context.binary:
+                argv = [context.binary.path]
+            else:
+                raise TypeError('Must provide argv or set context.binary')
+
+
         #: `subprocess.Popen` object
         self.proc = None
 

pwnlib/tubes/ssh.py

@@ -291,7 +291,7 @@ class ssh_channel(sock):
         """Retrieve the address of an environment variable in the remote process.
         """
         if not hasattr(self, 'argv'):
-            log.error("Can only call getenv() on ssh_channel objects created with ssh.process")
+            self.error("Can only call getenv() on ssh_channel objects created with ssh.process")
 
         argv0 = self.argv[0]
 
@@ -668,8 +668,12 @@ class ssh(Timeout, Logger):
                 See ``stdin``.
             preexec_fn(callable):
                 Function which is executed on the remote side before execve().
+                This **MUST** be a self-contained function -- it must perform
+                all of its own imports, and cannot refer to variables outside
+                its scope.
             preexec_args(object):
                 Argument passed to ``preexec_fn``.
+                This **MUST** only consist of native Python objects.
             raw(bool):
                 If ``True``, disable TTY control code interpretation.
             aslr(bool):
@@ -716,6 +720,23 @@ class ssh(Timeout, Logger):
             ''
             >>> s.process('/usr/bin/env', env={'A':'B'}).recvall()
             'A=B\n'
+
+            >>> s.process('false', preexec_fn=1234)
+            Traceback (most recent call last):
+            ...
+            PwnlibException: preexec_fn must be a function
+
+            >>> s.process('false', preexec_fn=lambda: 1234)
+            Traceback (most recent call last):
+            ...
+            PwnlibException: preexec_fn cannot be a lambda
+
+            >>> def uses_globals():
+            ...     foo = bar
+            >>> print s.process('false', preexec_fn=uses_globals).recvall().strip() # doctest: +ELLIPSIS
+            Traceback (most recent call last):
+            ...
+            NameError: global name 'bar' is not defined
         """
         if not argv and not executable:
             self.error("Must specify argv or executable")
@@ -767,15 +788,16 @@ class ssh(Timeout, Logger):
         # Allow the user to provide a self-contained function to run
         def func(): pass
         func      = preexec_fn or func
-        func_src  = inspect.getsource(func).strip()
-        func_name = func.__name__
         func_args = preexec_args
 
         if not isinstance(func, types.FunctionType):
-            log.error("preexec_fn must be a function")
-        if func_name == (lambda: 0).__name__:
-            log.error("preexec_fn cannot be a lambda")
+            self.error("preexec_fn must be a function")
 
+        func_name = func.__name__
+        if func_name == (lambda: 0).__name__:
+            self.error("preexec_fn cannot be a lambda")
+
+        func_src  = inspect.getsource(func).strip()
         setuid = setuid if setuid is None else bool(setuid)
 
         script = r"""
@@ -1454,7 +1476,7 @@ from ctypes import *; libc = CDLL('libc.so.6'); print(libc.getenv(%r))
         if os.path.isdir(file_or_directory):
             return self.upload_dir(file_or_directory, remote)
 
-        log.error('%r does not exist' % file_or_directory)
+        self.error('%r does not exist' % file_or_directory)
 
 
     def download(self, file_or_directory, remote=None):

pwnlib/tubes/tube.py

@@ -826,6 +826,35 @@ class tube(Timeout, Logger):
         while t.is_alive():
             t.join(timeout = 0.1)
 
+    def stream(self, line_mode=True):
+        """stream()
+
+        Receive data until the tube exits, and print it to stdout.
+
+        Similar to :func:`interactive`, except that no input is sent.
+
+        Similar to ``print tube.recvall()`` except that data is printed
+        as it is received, rather than after all data is received.
+
+        Arguments:
+            line_mode(bool): Whether to receive line-by-line or raw data.
+
+        Returns:
+            All data printed.
+        """
+        buf = Buffer()
+        function = self.recvline if line_mode else self.recv
+        try:
+            while True:
+                buf.add(function())
+                sys.stdout.write(buf.data[-1])
+        except KeyboardInterrupt:
+            pass
+        except EOFError:
+            pass
+
+        return buf.get()
+
     def clean(self, timeout = 0.05):
         """clean(timeout = 0.05)
 

pwnlib/update.py

@@ -34,34 +34,6 @@ package_name    = 'pwntools'
 package_repo    = 'Gallopsled/pwntools'
 update_freq     = datetime.timedelta(days=7).total_seconds()
 
-def available_on_github(prerelease=current_version.is_prerelease):
-    """Return True if an update is available on Github.
-
-    >>> available_on_github() # doctest: +ELLIPSIS
-    <Version('...')>
-    >>> available_on_github(prerelease=False).is_prerelease
-    False
-    """
-    url = 'https://api.github.com/repos/%s/tags' % package_repo
-
-    with context.quiet:
-        tags = json.loads(wget(url, timeout = 60))
-
-    # 'pwntools-ancient' is a tag, but not a valid version.
-    # Handle this here, and for all potential tags which cause
-    # issues.
-    versions = []
-    for tag in [t['name'] for t in tags]:
-        try:
-            versions.append(packaging.version.Version(tag))
-        except Exception:
-            pass
-
-    if not prerelease:
-        versions = filter(lambda v: not v.is_prerelease, versions)
-
-    return max(versions)
-
 def available_on_pypi(prerelease=current_version.is_prerelease):
     """Return True if an update is available on PyPI.
 
@@ -120,21 +92,12 @@ def perform_check(prerelease=current_version.is_prerelease):
     ['pip', 'install', '-U', ...]
 
     >>> def bail(*a): raise Exception()
-    >>> github = pwnlib.update.available_on_github
     >>> pypi   = pwnlib.update.available_on_pypi
 
-    >>> pwnlib.update.available_on_github = bail
     >>> perform_check(prerelease=False)
     ['pip', 'install', '-U', 'pwntools']
     >>> perform_check(prerelease=True)  # doctest: +ELLIPSIS
     ['pip', 'install', '-U', 'pwntools...']
-    >>> pwnlib.update.available_on_github = github
-
-    >>> pwnlib.update.available_on_pypi = bail
-    >>> perform_check(prerelease=False)
-    ['pip', 'install', '-U', 'git+https://github.com/Gallopsled/pwntools.git@...']
-    >>> perform_check(prerelease=True)  # doctest: +ELLIPSIS
-    ['pip', 'install', '-U', 'git+https://github.com/Gallopsled/pwntools.git@...']
     """
     pypi = current_version
     try:
@@ -142,13 +105,7 @@ def perform_check(prerelease=current_version.is_prerelease):
     except Exception:
         log.warning("An issue occurred while checking PyPI")
 
-    github = current_version
-    try:
-        github = available_on_github(prerelease)
-    except Exception:
-        log.warning("An issue occurred while checking Github")
-
-    best = max(pypi, github, current_version)
+    best = max(pypi, current_version)
     where = None
     command = None
 
@@ -170,9 +127,6 @@ def perform_check(prerelease=current_version.is_prerelease):
         if best.is_prerelease:
             pypi_package += '==%s' % (best)
         command += [pypi_package]
-    else:
-        where = 'GitHub'
-        command += ['git+https://github.com/%s.git@%s' % (package_repo, github)]
 
     command_str = ' '.join(command)
 

pwnlib/util/crc/__init__.py

@@ -1,10 +1,10 @@
 """Module for calculating CRC-sums.
 
-Contains all crc implementations know on the interwebz. For most implementations it
-contains only the core crc algorithm and not e.g. padding schemes.
+Contains all crc implementations know on the interwebz. For most implementations
+it contains only the core crc algorithm and not e.g. padding schemes.
 
 It is horribly slow, as implements a naive algorithm working direclty on
-bit polynomials.
+bit polynomials. This class is exposed as `BitPolynom`.
 
 The current algorithm is super-linear and takes about 4 seconds to calculate
 the crc32-sum of ``'A'*40000``.
@@ -18,15 +18,70 @@ import types
 from . import known
 from .. import fiddling
 from .. import packing
+from .. import safeeval
 
 
 class BitPolynom(object):
+    """Class for representing GF(2)[X], i.e. the field of polynomials over
+    GF(2).
+
+    In practice the polynomials are represented as numbers such that `x**n`
+    corresponds to `1 << n`. In this representation calculations are easy: Just
+    do everything as normal, but forget about everything the carries.
+
+    Addition becomes xor and multiplication becomes carry-less multiplication.
+
+    Examples:
+
+        >>> p1 = BitPolynom("x**3 + x + 1")
+        >>> p1
+        BitPolynom('x**3 + x + 1')
+        >>> int(p1)
+        11
+        >>> p1 == BitPolynom(11)
+        True
+        >>> p2 = BitPolynom("x**2 + x + 1")
+        >>> p1 + p2
+        BitPolynom('x**3 + x**2')
+        >>> p1 * p2
+        BitPolynom('x**5 + x**4 + 1')
+        >>> p1 / p2
+        BitPolynom('x + 1')
+        >>> p1 % p2
+        BitPolynom('x')
+        >>> d, r = divmod(p1, p2)
+        >>> d * p2 + r == p1
+        True
+        >>> BitPolynom(-1)
+        Traceback (most recent call last):
+            ...
+        ValueError: Polynomials cannot be negative: -1
+        >>> BitPolynom('y')
+        Traceback (most recent call last):
+            ...
+        ValueError: Not a valid polynomial: y
+    """
+
+
     def __init__(self, n):
-        if not isinstance(n, (int, long)):
-            raise TypeError("Polynomial must be called with an integer or a list")
-        if n < 0:
-            raise ValueError("Polynomials cannot be negative: %d" % n)
-        self.n = n
+        if isinstance(n, (str, unicode)):
+            self.n = 0
+            x = BitPolynom(2)
+            try:
+                for p in n.split('+'):
+                    k = safeeval.values(p.strip(), {'x': x, 'X': x})
+                    assert isinstance(k, (BitPolynom, int, long))
+                    assert k >= 0
+                    self.n ^= int(k)
+            except (ValueError, NameError, AssertionError):
+                raise ValueError("Not a valid polynomial: %s" % n)
+        elif isinstance(n, (int, long)):
+            if n >= 0:
+                self.n = n
+            else:
+                raise ValueError("Polynomials cannot be negative: %d" % n)
+        else:
+            raise TypeError("Polynomial must be called with a string or integer")
 
     def __int__(self):
         return self.n
@@ -126,7 +181,30 @@ class BitPolynom(object):
     def __rrshift__(self, other):
         return BitPolynom(int(other) >> int(self))
 
+    def __pow__(self, other):
+        r = BitPolynom(1)
+        for _ in range(other):
+            r *= self
+        return r
+
     def degree(self):
+        """Returns the degree of the polynomial.
+
+        Examples:
+
+            >>> BitPolynom(0).degree()
+            0
+            >>> BitPolynom(1).degree()
+            0
+            >>> BitPolynom(2).degree()
+            1
+            >>> BitPolynom(7).degree()
+            2
+            >>> BitPolynom((1 << 10) - 1).degree()
+            9
+            >>> BitPolynom(1 << 10).degree()
+            10
+        """
         return max(0, int(self).bit_length()-1)
 
     def __repr__(self):
@@ -134,17 +212,20 @@ class BitPolynom(object):
             return '0'
 
         out = []
-        for n in range(self.degree(), 0, -1):
+        for n in range(self.degree(), 1, -1):
             if int(self) & (1 << n):
                 out.append("x**%d" % n)
+        if int(self) & 2:
+            out.append("x")
         if int(self) & 1:
             out.append("1")
-        return ' + '.join(out)
+        return 'BitPolynom(%r)' % ' + '.join(out)
 
 class Module(types.ModuleType):
     def __init__(self):
         super(Module, self).__init__(__name__)
         self._cached_crcs = None
+        self.BitPolynom = BitPolynom
         self.__dict__.update({
             '__file__'    : __file__,
             '__package__' : __package__,
@@ -154,7 +235,7 @@ class Module(types.ModuleType):
         crcs = known.all_crcs
 
         if attr == '__all__':
-            return ['generic_crc', 'cksum', 'find_crc_function'] + sorted(crcs.keys())
+            return ['BitPolynom', 'generic_crc', 'cksum', 'find_crc_function'] + sorted(crcs.keys())
 
         info = crcs.get(attr, None)
         if not info:

pwnlib/util/cyclic.py

@@ -43,8 +43,8 @@ def de_bruijn(alphabet = string.ascii_lowercase, n = None):
 def cyclic(length = None, alphabet = string.ascii_lowercase, n = None):
     """cyclic(length = None, alphabet = string.ascii_lowercase, n = 4) -> list/str
 
-    A simple wrapper over :func:`de_bruijn`. This function returns a
-    at most `length` elements.
+    A simple wrapper over :func:`de_bruijn`. This function returns at most
+    `length` elements.
 
     If the given alphabet is a string, a string is returned from this function. Otherwise
     a list is returned.
@@ -66,6 +66,10 @@ def cyclic(length = None, alphabet = string.ascii_lowercase, n = None):
     if n is None:
         n = 4
 
+    if len(alphabet) ** n < length:
+        log.error("Can't create a pattern length=%i with len(alphabet)==%i and n==%i" \
+                  % (length, len(alphabet), n))
+
     out = []
     for ndx, c in enumerate(de_bruijn(alphabet, n)):
         if length != None and ndx >= length:
@@ -93,8 +97,8 @@ def cyclic_find(subseq, alphabet = string.ascii_lowercase, n = None):
        https://www.sciencedirect.com/science/article/pii/S0012365X00001175
 
     Arguments:
-        subseq: The subsequence to look for. This can either be a string, a list
-                or an integer. If an integer is provided it will be packed as a
+        subseq: The subsequence to look for. This can be a string, a list or an
+                integer. If an integer is provided it will be packed as a
                 little endian integer.
         alphabet: List or string to generate the sequence over.
         n(int): The length of subsequences that should be unique.
@@ -124,8 +128,98 @@ def cyclic_find(subseq, alphabet = string.ascii_lowercase, n = None):
 
     return _gen_find(subseq, de_bruijn(alphabet, n))
 
+def metasploit_pattern(sets = None):
+    """metasploit_pattern(sets = [ string.ascii_uppercase, string.ascii_lowercase, string.digits ]) -> generator
+
+    Generator for a sequence of characters as per Metasploit Framework's
+    `Rex::Text.pattern_create` (aka `pattern_create.rb`).
+
+    The returned generator will yield up to
+    ``len(sets) * reduce(lambda x,y: x*y, map(len, sets))`` elements.
+
+    Arguments:
+        sets: List of strings to generate the sequence over.
+    """
+    sets = sets or [ string.ascii_uppercase, string.ascii_lowercase, string.digits ]
+    offsets = [ 0 ] * len(sets)
+    offsets_indexes_reversed = list(reversed(range(len(offsets))))
+
+    while True:
+        for i, j in zip(sets, offsets):
+            yield i[j]
+        # increment offsets with cascade
+        for i in offsets_indexes_reversed:
+            offsets[i] = (offsets[i] + 1) % len(sets[i])
+            if offsets[i] != 0:
+                break
+        # finish up if we've exhausted the sequence
+        if offsets == [ 0 ] * len(sets):
+            return
+
+def cyclic_metasploit(length = None, sets = None):
+    """cyclic_metasploit(length = None, sets = [ string.ascii_uppercase, string.ascii_lowercase, string.digits ]) -> str
+
+    A simple wrapper over :func:`metasploit_pattern`. This function returns a
+    string of length `length`.
+
+    Arguments:
+        length: The desired length of the string or None if the entire sequence is desired.
+        sets: List of strings to generate the sequence over.
+
+    Example:
+        >>> cyclic_metasploit(32)
+        'Aa0Aa1Aa2Aa3Aa4Aa5Aa6Aa7Aa8Aa9Ab'
+        >>> cyclic_metasploit(sets = ["AB","ab","12"])
+        'Aa1Aa2Ab1Ab2Ba1Ba2Bb1Bb2'
+        >>> cyclic_metasploit()[1337:1341]
+        '5Bs6'
+        >>> len(cyclic_metasploit())
+        20280
+    """
+    sets = sets or [ string.ascii_uppercase, string.ascii_lowercase, string.digits ]
+    out = []
+
+    for ndx, c in enumerate(metasploit_pattern(sets)):
+        if length != None and ndx >= length:
+            break
+        else:
+            out.append(c)
+
+    out = ''.join(out)
+
+    if len(out) < length:
+        log.error("Can't create a pattern of length %i with sets of lengths %s. Maximum pattern length is %i." \
+                  % (length, map(len, sets), len(out)))
+
+    return ''.join(out)
+
+def cyclic_metasploit_find(subseq, sets = None):
+    """cyclic_metasploit_find(subseq, sets = [ string.ascii_uppercase, string.ascii_lowercase, string.digits ]) -> int
+
+    Calculates the position of a substring into a Metasploit Pattern sequence.
+
+    Arguments:
+        subseq: The subsequence to look for. This can be a string or an
+                integer. If an integer is provided it will be packed as a
+                little endian integer.
+        sets: List of strings to generate the sequence over.
+
+    Examples:
+
+        >>> cyclic_metasploit_find(cyclic_metasploit(1000)[514:518])
+        514
+        >>> cyclic_metasploit_find(0x61413161)
+        4
+    """
+    sets = sets or [ string.ascii_uppercase, string.ascii_lowercase, string.digits ]
+
+    if isinstance(subseq, (int, long)):
+        subseq = packing.pack(subseq, 'all', 'little', False)
+
+    return _gen_find(subseq, metasploit_pattern(sets))
+
 def _gen_find(subseq, generator):
-    """Returns the first position of subseq in the generator or -1 if there is no such position."""
+    """Returns the first position of `subseq` in the generator or -1 if there is no such position."""
     subseq = list(subseq)
     pos = 0
     saved = []

pwnlib/version.py

@@ -1 +1 @@
-__version__ = '3.2.0beta2'
+__version__ = '3.3.0beta0'

setup.py

@@ -83,7 +83,7 @@ except ImportError:
 setup(
     name                 = 'pwntools',
     packages             = find_packages(),
-    version              = '3.2.0beta2',
+    version              = '3.3.0beta0',
     data_files           = [('',
                              glob.glob('*.md') + glob.glob('*.txt')),
                             ],