Merge pull request #355 from Gallopsled/setup.py-openbsd

Ugly hack to be able to install on OpenBSD

.gitattributes

@@ -0,0 +1 @@
+pwnlib/data/* linguist-vendored

.gitignore

@@ -5,6 +5,7 @@
 *.pdf
 */auto/*
 */bin/*
+docs/build
 build
 MANIFEST
 dist

.travis.yml

@@ -0,0 +1,15 @@
+language: python
+python:
+  - "2.7"
+branches:
+  only:
+    - master
+install:
+  - source .travis_install.sh
+  - source .travis_ssh_setup.sh
+  - python -c 'from pwn import *; print pwnlib.term.term_mode' NOTERM
+  - python -c 'from pwn import *; print pwnlib.term.term_mode' NOTERM
+  - python -c 'from pwn import *; print pwnlib.term.term_mode' NOTERM
+  - python -c 'from pwn import *; print pwnlib.term.term_mode' NOTERM
+
+script: make -C docs doctest

.travis_install.sh

@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+
+setup_linux()
+{
+    sudo apt-get update
+    sudo apt-get install software-properties-common pwgen
+    sudo apt-add-repository --yes ppa:pwntools/binutils
+    sudo apt-get update
+    sudo apt-get install binutils-arm-linux-gnu binutils-mips-linux-gnu binutils-powerpc-linux-gnu
+    wget -nc http://www.capstone-engine.org/download/2.1.2/capstone-2.1.2_amd64.deb
+    (echo "a30bcb58fd82d32b135eec978eaa22230c44e722 *capstone-2.1.2_amd64.deb" | sha1sum -c) || exit
+    sudo dpkg -i *.deb
+}
+
+setup_osx()
+{
+    brew update
+    brew install binutils
+    brew install capstone
+}
+
+case $(uname) in
+Darwin) setup_osx   ;;
+Linux)  setup_linux ;;
+esac
+
+pip install -r requirements.txt
+pip install -r docs/requirements.txt
+pip install .

.travis_ssh_setup.sh

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+#
+# Install a demo user for SSH purposes
+#
+U=demouser
+H=/home/$U
+
+sudo useradd -m $U
+ssh-keygen -t rsa -f /tmp/$U -N ''
+
+eval $(ssh-agent -s)
+ssh-add    /tmp/$U
+
+sudo -u $U mkdir $H/.ssh
+sudo -u $U cp    /tmp/$U.pub $H/.ssh/authorized_keys
+sudo chmod -R 700 $H
+
+(echo -n "$U:"; pwgen 100 1) | sudo chpasswd

CHANGELOG.md

@@ -0,0 +1,18 @@
+# Changelog
+
+This changelog only includes added major features and changes. Bugfixes and
+minor changes are omitted.
+
+## 2.2.0 - 2014-08-28 (pre)
+
+This was a large release with a lot of bugfixes and changes. Only the most significant
+are mentioned here.
+
+- Added shellcodes
+- Added phd
+- Re-added our expansion of itertools
+- Added replacements for some semi-broken python standard library modules
+- Re-implemented the rop module
+- Added a serial tube
+- Huge performance gains in the buffering for tubes
+- Re-added user agents

CONTRIBUTING.md

@@ -0,0 +1,38 @@
+# Contributing
+
+We love pull requests!
+
+Github has a great guide for contributing to open source projects:
+
+- [Contributing to a project](https://guides.github.com/activities/forking/)
+- [Fork the repository](https://guides.github.com/activities/forking/#fork)
+- [Clone your fork](https://guides.github.com/activities/forking/#clone)
+- [Making and pushing changes](https://guides.github.com/activities/forking/#making-changes)
+- [Making a Pull Request](https://guides.github.com/activities/forking/#making-a-pull-request)
+- [Huzzah!](https://guides.github.com/activities/forking/#huzzah)
+
+## Pwntools Specifics
+
+In general, we like to keep things documented.  You should add documentation to any new functionality, and update it for any changed functionality.  Our docstrings use the [Google Style Python Docstrings](https://sphinxcontrib-napoleon.readthedocs.org/en/latest/example_google.html#example-google).
+
+After you have documentation, you should add a [doctest](https://docs.python.org/2/library/doctest.html).
+
+Finally, it is probably a good idea to run the test suite locally before doing
+the pull-request to make sure everything works, however this is not a
+requirement.
+
+Once you do the pull-request Travis CI will run the test-suite on it. Once it
+passes one of the core developers will look at your pull request, possibly
+comment on it and then hopefully merge it into master.
+
+## Test Suite
+
+To run the test suite, you should be running Ubuntu 12.04 or 14.04, and run the following commands.  **Be aware** that this will add a user to your machine, and create a public key for SSH login!
+
+```sh
+bash .travis_install.sh
+bash .travis_ssh_setup.sh
+cd docs
+pip install -r requirements.txt
+make doctest
+```

README.md

@@ -1,38 +1,56 @@
-# Introduction
+# pwntools - CTF toolkit
+[![Docs latest](https://readthedocs.org/projects/pwntools/badge/)](https://pwntools.readthedocs.org/en/latest)
+[![Docs 2.1.3](https://readthedocs.org/projects/pwntools/badge/?version=2.1.3)](https://pwntools.readthedocs.org/en/2.1.3)
+[![PyPI](https://img.shields.io/pypi/v/pwntools.svg)](https://pypi.python.org/pypi/pwntools/)
+[![Gittip](https://img.shields.io/gittip/gallopsled.svg)](https://www.gittip.com/gallopsled/)
+[![Travis](https://travis-ci.org/Gallopsled/pwntools.svg)](https://travis-ci.org/Gallopsled/pwntools)
 
 This is the CTF framework used by Gallopsled in every CTF.
 
-Most code is inside the pwnlib folder with some functionality inside pwn or
-bin. It is typically used as:
-
 ```python
 from pwn import *
 context(arch = 'i386', os = 'linux')
 
-# EXPLOIT HERE
+r = remote('exploitme.example.com', 31337)
+# EXPLOIT CODE GOES HERE
+r.send(asm(shellcraft.sh()))
+r.interactive()
 ```
 
 However we have made command-line frontends for some of the functionality
-inside the pwnlib. These are:
+in `pwnlib`. These are:
 
-* `asm`/`disasm`: Small wrapper for various assemblers
-* `constgrep`: Tool for finding constants defined in header files
-* `cyclic`: De Bruijn sequence generator and lookup tool
-* `hex`/`unhex`: Command line tools for doing common hexing/unhexing operations
-* `shellcraft`: Frontend to our shellcode
+* `asm`/`disasm`: Small wrapper for various assemblers.
+* `constgrep`: Tool for finding constants defined in header files.
+* `cyclic`: De Bruijn sequence generator and lookup tool.
+* `hex`/`unhex`: Command line tools for doing common hexing/unhexing operations.
+* `shellcraft`: Frontend to our shellcode.
+* `phd`: Replacement for `hexdump` with colors.
 
 # Documentation
-Our documentation is available on http://pwntools.readthedocs.org/
+Our documentation is available at [pwntools.readthedocs.org](https://pwntools.readthedocs.org/en/latest/)
+
+To get you started, we've provided some example solutions for past CTF challenges in our [write-ups repository](https://github.com/Gallopsled/pwntools-write-ups).
 
 # Installation
-Pwntools is available as a pip package. You can install it by running
-`pip install pwntools`.
 
-Alternatively if you prefer to have the latest version in git, you can
-simply clone this repository, run `pip install -r requirements.txt`
-and add entries in your `PATH` and `PYTHONPATH` variables. The script
-`install_local.sh` will help you do so, in case you are using bash.
+pwntools is best supported on Ubuntu 12.04 and 14.04, but most functionality should work on any Posix-like distribution (Debian, Arch, FreeBSD, OSX, etc.).
+
+Most of the functionality of pwntools is self-contained and Python-only.  You should be able to get running quickly with
+
+```sh
+pip install pwntools
+```
+
+However, some of the features (ROP generation and assembling/disassembling foreign architectures) require non-Python dependencies.  For more information, see the [complete installation instructions here](https://pwntools.readthedocs.org/en/latest/install.html).
+
+
+# Contribution
+
+See [CONTRIBUTING.md](CONTRIBUTING.md)
 
 # Contact
-If you have any questions not worthy of a bug report, feel free to join us
-at `#gallopsled` on Freenode and ask away.
+If you have any questions not worthy of a [bug report](https://github.com/Gallopsled/pwntools/issues), feel free to join us
+at [`#gallopsled` on Freenode](irc://irc.freenode.net/gallopsled) and ask away.
+Click [here](https://kiwiirc.com/client/irc.freenode.net/gallopsled) to connect.
+

bin/constgrep

@@ -1,136 +0,0 @@
-#!/usr/bin/env python2
-
-import argparse, sys, os, types, re
-from pwnlib import asm, util, constants
-from pwnlib.util import safeeval
-from pwnlib.context import __possible__ as contexts
-
-p = argparse.ArgumentParser(
-    description = "Looking up constants from header files.\n\nExample: constgrep -c freebsd -m  ^PROT_ '3 + 4'",
-    formatter_class = argparse.RawDescriptionHelpFormatter,
-)
-
-group = p.add_mutually_exclusive_group()
-group.add_argument(
-    '-e', '--exact',
-    metavar = '<constant name>',
-    # nargs = 1,
-    default = None,
-    help = 'Do an exact match for a constant instead of searching for a regex',
-)
-group.add_argument(
-    'regex',
-    nargs = '?',
-    default = '',
-    help = 'The regex matching constant you want to find',
-)
-
-p.add_argument(
-    'constant',
-    nargs = '?',
-    default = None,
-    help = 'The constant to find',
-)
-
-p.add_argument(
-    '-i', '--case-insensitive',
-    action = 'store_true',
-    help = 'Search case insensitive',
-)
-
-p.add_argument(
-    '-m', '--mask-mode',
-    action = 'store_true',
-    help = 'Instead of searching for a specific constant value, search for values not containing strictly less bits that the given value.',
-)
-
-p.add_argument(
-    '-c', '--context',
-    metavar = '<opt>',
-    choices = contexts['os'] + contexts['arch'],
-    default = ['i386','linux'],
-    action = 'append',
-    help = 'The os/architecture to find constants for (default: linux/i386), choose from: %s' % \
-    ', '.join(dir(constants) + dir(constants.linux))
-)
-
-args = p.parse_args()
-
-# Find the architecture and os from the list of contexts
-for arch in args.context[::-1]:
-    if arch in contexts['arch']: break
-for os in args.context[::-1]:
-    if os in contexts['os']: break
-
-if args.exact:
-    # This is the simple case
-    print asm.cpp(args.exact, os = os, arch = arch).strip()
-else:
-    # New we search in the right module.
-    # But first: We find the right module
-    if os == 'freebsd':
-        mod = constants.freebsd
-    else:
-        mod = getattr(getattr(constants, os), arch)
-
-    # Compile the given regex, for optimized lookup
-    if args.case_insensitive:
-        matcher = re.compile(args.regex, re.IGNORECASE)
-    else:
-        matcher = re.compile(args.regex)
-
-    # Evaluate the given constant
-    if args.constant:
-        constant = safeeval.expr_eval(args.constant)
-    else:
-        constant = None
-
-    # The found matching constants and the length of the longest string
-    out    = []
-    maxlen = 0
-
-    for k in dir(mod):
-        # No python stuff
-        if k.endswith('__') and k.startswith('__'):
-            continue
-
-        # Run the regex
-        if not matcher.search(k):
-            continue
-
-        # Check the constant
-        if constant != None:
-            val = getattr(mod, k)
-            if args.mask_mode:
-                if constant & val != val:
-                    continue
-            else:
-                if constant != val:
-                    continue
-
-        # Append it
-        out.append((getattr(mod, k), k))
-        maxlen = max(len(k), maxlen)
-
-    # Output all matching constants
-    for _, k in sorted(out):
-        print '#define %s %s' % (k.ljust(maxlen), asm.cpp(k, os = os, arch = arch).strip())
-
-    # If we are in match_mode, then try to find a combination of
-    # constants that yield the exact given value
-    # We do not want to find combinations using the value 0.
-    if not (constant == None or constant == 0) and args.mask_mode:
-        mask = constant
-        good = []
-        out = [(v, k) for v, k in out if v != 0]
-
-        while mask and out:
-            cur = out.pop()
-            mask &= ~cur[0]
-            good.append(cur)
-
-            out = [(v, k) for v, k in out if mask & v == v]
-
-        if reduce(lambda x, cur: x | cur[0], good, 0) == constant:
-            print
-            print '(%s) == %s' % (' | '.join(k for v, k in good), args.constant)

bin/cyclic

@@ -1,69 +0,0 @@
-#!/usr/bin/env python2
-import argparse, string
-from pwnlib.util import cyclic, packing
-from pwnlib import log
-
-parser = argparse.ArgumentParser(
-    description = "Cyclic pattern creator/finder"
-)
-
-parser.add_argument(
-    '-a', '--alphabet',
-    metavar = 'alphabet',
-    default = string.ascii_lowercase,
-    help = 'The alphabet to use in the cyclic pattern (defaults to all lower case letters)',
-)
-
-parser.add_argument(
-    '-n', '--length',
-    metavar = 'length',
-    default = 4,
-    type = int,
-    help = 'Size of the unique subsequences (defaults to 4).'
-)
-
-group = parser.add_mutually_exclusive_group(required = True)
-group.add_argument(
-    '-l', '-o', '--offset', '--lookup',
-    metavar = '<lookup value>',
-    help = 'Do a lookup instead printing the alphabet',
-)
-
-group.add_argument(
-    'count',
-    type = int,
-    nargs = '?',
-    help = 'Number of characters to print'
-)
-
-args = parser.parse_args()
-alphabet = args.alphabet
-subsize  = args.length
-
-if args.lookup:
-    pat = args.lookup
-
-    if pat.startswith('0x'):
-        pat = packing.pack(int(pat[2:], 16), subsize*8, 'little', 'unsigned')
-    elif pat.isdigit():
-        pat = packing.pack(int(pat, 10), subsize*8, 'little', 'unsigned')
-
-    if len(pat) != 4:
-        log.fatal('Subpattern must be 4 bytes', 1)
-
-    if not all(c in alphabet for c in pat):
-        log.fatal('Pattern contains characters not present in the alphabet', 1)
-
-    offset = cyclic.cyclic_find(pat, alphabet, subsize)
-
-    if offset == -1:
-        log.fatal('Given pattern does not exist in cyclic pattern', 1)
-    else:
-        print offset
-else:
-    want   = args.count
-    result = cyclic.cyclic(want, alphabet, subsize)
-    got    = len(result)
-    if got < want:
-        log.failure("Alphabet too small (max length = %i)" % got)
-    print result

bin/disasm

@@ -1,38 +0,0 @@
-#!/usr/bin/env python2
-
-import argparse, sys
-from pwnlib import asm, context
-from string import whitespace, hexdigits
-
-parser = argparse.ArgumentParser(
-    description = 'Disassemble bytes into text format'
-)
-
-parser.add_argument(
-    'hex',
-    metavar = 'hex',
-    nargs = '*',
-    help = 'Hex-string to disasemble. If none are supplied, then it uses stdin in non-hex mode.'
-)
-
-parser.add_argument(
-    '-c', '--context',
-    metavar = '<opt>',
-    choices = context.__possible__['arch'],
-    default = 'i386',
-    help = 'The architecture of the shellcode (default: i386), choose from:\n%s' % ', '.join(context.__possible__['arch'])
-)
-
-args = parser.parse_args()
-
-if len(args.hex) > 0:
-    dat = ''.join(args.hex)
-    dat = dat.translate(None, whitespace)
-    if not set(hexdigits) >= set(dat):
-        print "This is not a hex string"
-        exit(-1)
-    dat = dat.decode('hex')
-else:
-    dat = sys.stdin.read()
-
-print asm.disasm(dat, arch = args.context)

bin/hex

@@ -1,7 +0,0 @@
-#!/usr/bin/env python2
-import sys
-
-if len(sys.argv) == 1:
-    print sys.stdin.read().encode('hex')
-else:
-    print ' '.join(sys.argv[1:]).encode('hex')

bin/shellcraft

@@ -1,173 +0,0 @@
-#!/usr/bin/env python2
-import argparse, sys, os, types
-import pwnlib
-from pwnlib import asm, log, util
-from pwnlib.context import __possible__ as contexts
-import pwnlib.term.text as text
-
-r = text.red
-g = text.green
-b = text.blue
-banner = '\n'.join(['  ' + r('____') + '  ' + g('_') + '          ' + r('_') + ' ' + g('_') + '                 ' + b('__') + ' ' + r('_'),
-                    ' ' + r('/ ___|') + g('| |__') + '   ' + b('___') + r('| |') + ' ' + g('|') + ' ' + b('___') + ' ' + r('_ __') + ' ' + g('__ _') + ' ' + b('/ _|') + ' ' + r('|_'),
-                    ' ' + r('\___ \\') + g('| \'_ \\') + ' ' + b('/ _ \\') + ' ' + r('|') + ' ' + g('|') + b('/ __|') + ' ' + r('\'__/') + ' ' + g('_` |') + ' ' + b('|_') + r('| __|'),
-                    '  ' + r('___) |') + ' ' + g('| | |') + '  ' + b('__/') + ' ' + r('|') + ' ' + g('|') + ' ' + b('(__') + r('| |') + ' ' + g('| (_| |') + '  ' + b('_|') + ' ' + r('|_'),
-                    ' ' + r('|____/') + g('|_| |_|') + b('\\___|') + r('_|') + g('_|') + b('\\___|') + r('_|') + '  ' + g('\\__,_|') + b('_|') + '  ' + r('\\__|'),
-                    ''
-                    ])
-#  ____  _          _ _                 __ _
-# / ___|| |__   ___| | | ___ _ __ __ _ / _| |_
-# \___ \| '_ \ / _ \ | |/ __| '__/ _` | |_| __|
-#  ___) | | | |  __/ | | (__| | | (_| |  _| |_
-# |____/|_| |_|\___|_|_|\___|_|  \__,_|_|  \__|
-
-def _string(s):
-    out = []
-    for c in s:
-        co = ord(c)
-        if co >= 0x20 and co <= 0x7e and c not in '/$\'"`':
-            out.append(c)
-        else:
-            out.append('\\x%02x' % co)
-    return '"' + ''.join(out) + '"\n'
-
-def _carray(s):
-    out = []
-    for c in s:
-        out.append('0x' + util.fiddling.enhex(c))
-    return '{' + ', '.join(out) + '};\n'
-
-def _hex(s):
-    return pwnlib.util.fiddling.enhex(s) + '\n'
-
-p = argparse.ArgumentParser(
-    description = banner + '\nMicrowave shellcode -- Easy, fast and delicious',
-    formatter_class = argparse.RawDescriptionHelpFormatter,
-)
-
-p.add_argument(
-    'shellcode',
-    nargs = '?',
-    default = '',
-    metavar = '<shellcode>',
-    help = 'The shellcode you want',
-)
-
-p.add_argument(
-    'args',
-    nargs = '*',
-    metavar = '<arg>',
-    default = (),
-    help = 'Argument to the chosen shellcode',
-)
-
-p.add_argument(
-    '-?', '--show',
-    action = 'store_true',
-    help = 'Show shellcode documentation',
-)
-
-p.add_argument(
-    '-o', '--out',
-    metavar = '<file>',
-    type = argparse.FileType('w'),
-    default = sys.stdout,
-    help = 'Output file (default: stdout)',
-)
-
-p.add_argument(
-    '-f', '--format',
-    metavar = '<format>',
-    choices = ['r', 'raw',
-               's', 'str', 'string',
-               'c',
-               'h', 'hex',
-               'a', 'asm', 'assembly',
-               'p',
-               'i', 'hexii'],
-    default = 'hex',
-    help = 'Output format (default: hex), choose from {r}aw, {s}tring, {c}-style array, {h}ex string, hex{i}i, {a}ssembly code, {p}reprocssed code',
-)
-
-args = p.parse_args()
-
-def get_tree(path, val, result):
-    if path:
-        path += '.'
-
-    if path.startswith('common.'):
-        return
-
-    mods = []
-
-    for k in sorted(dir(val)):
-        if k and k[0] != '_':
-            cur = getattr(val, k)
-            if isinstance(cur, types.ModuleType):
-                mods.append((path + k, cur))
-            else:
-                result.append((path + k, cur))
-
-    for path, val in mods:
-        get_tree(path, val, result)
-    return result
-
-vals = get_tree('', pwnlib.shellcraft, [])
-if args.shellcode:
-    vals = [(k, val) for k, val in vals if k.startswith(args.shellcode + '.') or k == args.shellcode]
-
-if len(vals) == 0:
-    log.fatal("Cannot find subtree by the name of %r" % args.shellcode)
-elif len(vals) > 1:
-    for k, _ in vals:
-        print k
-    exit()
-else:
-    func = vals[0][1]
-
-if args.show:
-    print func.__doc__
-    exit()
-
-defargs = len(func.func_defaults or ())
-reqargs = func.func_code.co_argcount - defargs
-if len(args.args) < reqargs:
-    if defargs > 0:
-        log.fatal('%s takes at least %d arguments' % (args.shellcode, reqargs))
-    else:
-        log.fatal('%s takes exactly %d arguments' % (args.shellcode, reqargs))
-
-# Captain uglyness saves the day!
-for i in range(len(args.args)):
-    if args.args[i] in ['False', 'false']:
-        args.args[i] = False
-
-# And he strikes again!
-os = arch = None
-for k in args.shellcode.split('.')[:-1]:
-    if k in contexts['arch']:
-        arch = k
-    elif k in contexts['os']:
-        os = k
-
-code = func(*args.args)
-
-if args.format in ['a', 'asm', 'assembly']:
-    print code
-    exit()
-if args.format == 'p':
-    print pwnlib.asm.cpp(code, arch = arch, os = os)
-    exit()
-
-code = pwnlib.asm.asm(code, arch = arch, os = os)
-
-if args.format in ['s', 'str', 'string']:
-    code = _string(code)
-elif args.format == 'c':
-    code = _carray(code)
-elif args.format in ['h', 'hex']:
-    code = _hex(code)
-elif args.format in ['i', 'hexii']:
-    code = pwnlib.util.fiddling.hexii(code) + '\n'
-
-sys.stdout.write(code)

bin/unhex

@@ -1,11 +0,0 @@
-#!/usr/bin/env python2
-import sys
-from string import whitespace
-try:
-    if len(sys.argv) == 1:
-        s = sys.stdin.read().translate(None, whitespace)
-        sys.stdout.write(s.decode('hex'))
-    else:
-        sys.stdout.write(''.join(sys.argv[1:]).decode('hex'))
-except TypeError, e:
-    sys.stderr.write(str(e) + '\n')

docs/.gitignore

@@ -1 +1,2 @@
 build
+robots.txt

docs/requirements.txt

@@ -3,3 +3,4 @@ argparse
 mako
 pyelftools
 sphinxcontrib-napoleon
+sphinxcontrib-autoprogram

docs/source/about.rst

@@ -0,0 +1,56 @@
+About pwntools
+========================
+
+Whether you're using it to write exploits, or as part
+of another software project will dictate how you use it.
+
+Historically pwntools was used as a sort of exploit-writing DSL. Simply doing
+``from pwn import *`` in a previous version of pwntools would bring all sorts of
+nice side-effects.
+
+When redesigning pwntools for 2.0, we noticed two contrary goals:
+
+* We would like to have a "normal" python module structure, to allow other
+  people to faster get familiar with how pwntools works.
+* We would like to have even more side-effects, especially by putting the
+  terminal in raw-mode.
+
+To make this possible, we decided to have two different modules. :mod:`pwnlib`
+would be our nice, clean Python module, while :mod:`pwn` would be used during
+CTFs.
+
+:mod:`pwn` --- Toolbox optimized for CTFs
+-----------------------------------------
+
+.. module:: pwn
+
+As stated, we would also like to have the ability to get a lot of these
+side-effects by default. That is the purpose of this module. It does
+the following:
+
+* Imports everything from the toplevel :mod:`pwnlib` along with
+  functions from a lot of submodules. This means that if you do
+  ``import pwn`` or ``from pwn import *``, you will access to
+  everything you need to write an exploit.
+* Calls :func:`pwnlib.term.init` to put your terminal in raw mode
+  and implementing functionality to make it look like it is not.
+* Setting the :data:`pwnlib.context.log_level` to `"info"`.
+* Tries to parse some of the values in :data:`sys.argv` and every
+  value it succeeds in parsing it removes.
+
+:mod:`pwnlib` --- Normal python library
+---------------------------------------
+
+.. module:: pwnlib
+
+This module is our "clean" python-code. As a rule, we do not think that
+importing :mod:`pwnlib` or any of the submodules should have any significant
+side-effects (besides e.g. caching).
+
+For the most part, you will also only get the bits you import. You for instance
+not get access to :mod:`pwnlib.util.packing` simply by doing ``import
+pwnlib.util``.
+
+Though there are a few exceptions (such as :mod:`pwnlib.shellcraft`), that does
+not quite fit the goals of being simple and clean, but they can still be
+imported without implicit side-effects.

docs/source/asm.rst

@@ -1,6 +1,7 @@
 .. testsetup:: *
 
    from pwnlib.asm import *
+   from pwnlib import shellcraft
 
 :mod:`pwnlib.asm` --- Assembler functions
 =========================================

docs/source/atexception.rst

@@ -0,0 +1,5 @@
+:mod:`pwnlib.atexception` --- Callbacks on unhandled exception
+==============================================================
+
+.. automodule:: pwnlib.atexception
+   :members:

docs/source/atexit.rst

@@ -0,0 +1,5 @@
+:mod:`pwnlib.atexit` --- Replacement for atexit
+===============================================
+
+.. automodule:: pwnlib.atexit
+   :members:

docs/source/commandline.rst

@@ -0,0 +1,46 @@
+.. testsetup:: *
+
+   from pwn import *
+   old = context.defaults.copy()
+
+.. testcleanup:: *
+
+    context.defaults.copy = old
+
+Command Line Tools
+========================
+
+pwntools comes with a handful of useful command-line utilities which serve as wrappers for some of the internal functionality.
+
+.. toctree::
+
+.. autoprogram:: pwnlib.commandline.asm:parser
+   :prog: asm
+
+.. autoprogram:: pwnlib.commandline.constgrep:p
+   :prog: constgrep
+
+.. autoprogram:: pwnlib.commandline.cyclic:parser
+   :prog: cyclic
+
+.. autoprogram:: pwnlib.commandline.disasm:parser
+   :prog: disasm
+
+.. autoprogram:: pwnlib.commandline.elfdiff:p
+   :prog: elfdiff
+
+.. autoprogram:: pwnlib.commandline.elfpatch:p
+   :prog: elfpatch
+
+.. autoprogram:: pwnlib.commandline.hex:parser
+   :prog: hex
+
+.. autoprogram:: pwnlib.commandline.phd:parser
+   :prog: phd
+
+.. autoprogram:: pwnlib.commandline.shellcraft:p
+   :prog: shellcraft
+
+.. autoprogram:: pwnlib.commandline.unhex:parser
+   :prog: unhex
+

docs/source/conf.py

@@ -11,7 +11,7 @@
 # All configuration values have a default; values that are commented out
 # serve to show the default.
 
-import sys, os
+import sys, os, subprocess
 
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
@@ -31,17 +31,22 @@ extensions = [
     'sphinx.ext.doctest',
     'sphinx.ext.linkcode',
     'sphinx.ext.autosummary',
+    'sphinx.ext.coverage',
     'sphinx.ext.todo',
     'sphinx.ext.intersphinx',
+    'sphinxcontrib.autoprogram',
     'sphinxcontrib.napoleon'
 ]
 
 doctest_global_setup = '''
 import pwnlib
-pwnlib.context.reset_local()
+pwnlib.context.context.reset_local()
+pwnlib.context.ContextType.defaults['log_level'] = 'ERROR'
+pwnlib.term.text.when = 'never'
+
 '''
 
-autodoc_member_order = 'bysource'
+autodoc_member_order = 'alphabetical'
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
@@ -52,7 +57,7 @@ doctest_test_doctest_blocks = True
 source_suffix = '.rst'
 
 # The encoding of source files.
-#source_encoding = 'utf-8-sig'
+source_encoding = 'utf-8-sig'
 
 # The master toctree document.
 master_doc = 'index'
@@ -205,7 +210,8 @@ latex_documents = [
    u'Gallopsled et al', 'manual'),
 ]
 
-intersphinx_mapping = {'python': ('http://docs.python.org/2.7', None)}
+intersphinx_mapping = {'python': ('https://docs.python.org/2.7', None),
+                       'paramiko': ('https://paramiko-docs.readthedocs.org/en/1.15/', None)}
 
 # The name of an image file (relative to this directory) to place at the top of
 # the title page.
@@ -261,6 +267,19 @@ texinfo_documents = [
 # How to display URL addresses: 'footnote', 'no', or 'inline'.
 #texinfo_show_urls = 'footnote'
 
+branch = release
+
+try:
+    git_branch = subprocess.check_output('git describe --tags', shell = True)
+except subprocess.CalledProcessError:
+    git_branch = '-'
+
+try:
+    if '-' in git_branch:
+        branch = subprocess.check_output('git rev-parse HEAD', shell = True).strip()[:10]
+except subprocess.CalledProcessError:
+    pass
+
 def linkcode_resolve(domain, info):
     if domain != 'py':
         return None
@@ -285,21 +304,11 @@ def linkcode_resolve(domain, info):
     else:
         filename = info['module'].replace('.', '/') + '.py'
 
-        # Fixup for context properties
-        if info['module'].startswith('pwnlib.context'):
-            filename = 'pwnlib/context.py'
-
-            if not isinstance(val, (types.FunctionType, types.MethodType)):
-                import pwnlib.context.defaults
-                val = getattr(pwnlib.context.defaults.__class__, info['fullname'], None)
-                if isinstance(val, property):
-                    val = val.fget
-                    val = getattr(val, '_inner', val)
-
         if isinstance(val, (types.ModuleType, types.ClassType, types.MethodType, types.FunctionType, types.TracebackType, types.FrameType, types.CodeType)):
             try:
                 lines, first = inspect.getsourcelines(val)
                 filename += '#L%d-%d' % (first, first + len(lines) - 1)
             except IOError:
                 pass
-    return "https://github.com/Gallopsled/pwntools/blob/%s/%s" % (release, filename)
+
+    return "https://github.com/Gallopsled/pwntools/blob/%s/%s" % (branch, filename)

docs/source/constants.rst

@@ -1,6 +1,7 @@
 .. testsetup:: *
 
-   from pwnlib import context, constants
+   from pwnlib import constants
+   from pwnlib.context import context
 
 :mod:`pwnlib.constants` --- Easy access to header file constants
 ================================================================

docs/source/context.rst

@@ -1,49 +1,16 @@
 .. testsetup:: *
 
-   from pwnlib import context
+   import pwnlib
+   from   pwnlib.context import context
+   import threading, logging
 
 
 :mod:`pwnlib.context` --- Setting runtime variables
 =====================================================
 
+.. autodata:: pwnlib.context.context
 
-:mod:`pwnlib.context`
----------------------
+.. autoclass:: pwnlib.context.ContextType
+    :members:
 
-.. automodule:: pwnlib.context
-
-   .. exec::
-      import pwnlib
-      from inspect import cleandoc
-      mod = pwnlib.context.defaults.__class__
-      for key in sorted(dir(mod)):
-          val = getattr(mod, key)
-          if isinstance(val, property):
-              print ".. data:: " + key
-              print ""
-              print cleandoc(val.__doc__)
-              print ""
-
-
-   .. autofunction:: pwnlib.context.__call__(**kwargs)
-   .. autofunction:: pwnlib.context.local(**kwargs)
-   .. autofunction:: pwnlib.context.reset_local()
-
-:mod:`pwnlib.context.defaults`
-------------------------------
-
-.. automodule:: pwnlib.context.defaults
-
-   .. exec::
-      import pwnlib
-      mod = pwnlib.context.defaults.__class__
-      for key in sorted(dir(mod)):
-          val = getattr(mod, key)
-          if isinstance(val, property):
-              print ".. data:: " + key
-              print ""
-              print "This is the global version of :data:`pwnlib.context.%s`." % key
-              print
-
-
-   .. autofunction:: pwnlib.context.defaults.__call__(**kwargs)
+.. autoclass:: pwnlib.context.Thread

docs/source/elf.rst

@@ -1,6 +1,7 @@
 .. testsetup:: *
 
    from pwnlib.elf import *
+   from pwnlib.util.misc import which
 
 :mod:`pwnlib.elf` --- Working with ELF binaries
 ===============================================

docs/source/globals.rst

@@ -0,0 +1,102 @@
+.. testsetup:: *
+
+   from pwn import *
+
+``from pwn import *``
+========================
+
+The most common way that you'll see pwntools used is
+
+    >>> from pwn import *
+
+Which imports a bazillion things into the global namespace to make your life easier.
+
+This is a quick list of most of the objects and routines imported, in rough order of importance and frequency of use.
+
+- ``context``
+    - :data:`pwnlib.context.context`
+    - Responsible for most of the pwntools convenience settings
+    - Set `context.log_level = 'debug'` when troubleshooting your exploit
+    - Scope-aware, so you can disable logging for a subsection of code via ``pwnlib.context.ContextType.local``
+- ``remote``, ``listen``, ``ssh``, ``process``
+    - :mod:`pwnlib.tubes`
+    - Super convenient wrappers around all of the common functionality for CTF challenges
+    - Connect to anything, anywhere, and it works the way you want it to
+    - Helpers for common tasks like ``recvline``, ``recvuntil``, ``clean``, etc.
+    - Interact directly with the application via ``.interactive()``
+- ``p32`` and ``u32``
+    - :mod:`pwnlib.util.packing`
+    - Useful functions to make sure you never have to remember if ``'>'`` means signed or unsigned for ``struct.pack``, and no more ugly ``[0]`` index at the end.
+    - Set ``signed`` and ``endian`` in sane manners (also these can be set once on ``context`` and not bothered with again)
+    - Most common sizes are pre-defined (``u8``, ``u64``, etc), and :func:`pwnlib.util.packing.pack` lets you define your own.
+- ``log``
+    - :mod:`pwnlib.log`
+    - Make your output pretty!
+- ``cyclic`` and ``cyclic_func``
+    - :mod:`pwnlib.util.cyclic`
+    - Utilities for generating strings such that you can find the offset of any given substring given only N (usually 4) bytes.  This is super useful for straight buffer overflows.  Instead of looking at 0x41414141, you could know that 0x61616171 means you control EIP at offset 64 in your buffer.
+- ``asm`` and ``disasm``
+    - :mod:`pwnlib.asm`
+    - Quickly turn assembly into some bytes, or vice-versa, without mucking about
+    - Supports any architecture for which you have a binutils installed
+    - Over 20 different architectures have pre-built binaries at `ppa:pwntools/binutils <https://launchpad.net/~pwntools/+archive/ubuntu/binutils>`_.
+- ``shellcraft``
+    - :mod:`pwnlib.shellcraft`
+    - Library of shellcode ready to go
+    - ``asm(shellcraft.sh())`` gives you a shell
+    - Templating library for reusability of shellcode fragments
+- ``ELF``
+    - :mod:`pwnlib.elf`
+    - ELF binary manipulation tools, including symbol lookup, virtual memory to file offset helpers, and the ability to modify and save binaries back to disk
+- ``DynELF``
+    - :mod:`pwnlib.dynelf`
+    - Dynamically resolve functions given only a pointer to any loaded module, and a function which can leak data at any address
+- ``ROP``
+    - :mod:`pwnlib.rop`
+    - Automatically generate ROP chains using a DSL to describe what you want to do, rather than raw addresses
+- ``gdb.debug`` and ``gdb.attach``
+    - :mod:`pwnlib.gdb`
+    - Launch a binary under GDB and pop up a new terminal to interact with it.  Automates setting breakpoints and makes iteration on exploits MUCH faster.
+    - Alternately, attach to a running process given a PID, ``pwnlib.tubes`` object, or even just a socket that's connected to it
+- ``args``
+    - Dictionary contining all-caps command-line arguments for quick access
+    - Run via ``python foo.py REMOTE=1`` and ``args['REMOTE'] == '1'``.
+    - Can also control logging verbosity and terminal fancyness
+        - `NOTERM`
+        - `SILENT`
+        - `DEBUG`
+- ``randoms``, ``rol``, ``ror``, ``xor``, ``bits``
+    - :mod:`pwnlib.util.fiddling`
+    - Useful utilities for generating random data from a given alphabet, or simplifying math operations that usually require masking off with `0xffffffff` or calling `ord` and `chr` an ugly number of times
+- ``net``
+    - :mod:`pwnlib.util.net`
+    - Routines for querying about network interfaces
+- ``proc``
+    - :mod:`pwnlib.util.proc`
+    - Routines for querying about processes
+- ``pause``
+    - It's the new ``getch``
+- ``safeeval``
+    - :mod:`pwnlib.util.safeeval`
+    - Functions for safely evalutaing python code without nasty side-effects.
+
+These are all pretty self explanatory, but are useful to have in the global namespace.
+
+- ``hexdump``
+- ``read`` and ``write``
+- ``enhex`` and ``unhex``
+- ``more``
+- ``group``
+- ``align`` and ``align_down``
+- ``urlencode`` and ``urldecode``
+- ``which``
+- ``wget``
+
+Additionally, all of the following modules are auto-imported for you.  You were going to do it anyway.
+
+- ``os``
+- ``sys``
+- ``time``
+- ``requests``
+- ``re``
+- ``random``

docs/source/index.rst

@@ -1,14 +1,36 @@
-Welcome to pwntools's documentation!
+pwntools
 ====================================
 
-Contents:
+``pwntools`` is a CTF framework and exploit development library.
+Written in Python, it is designed for rapid prototyping and development,
+and intended to make exploit writing as simple as possible.
+
+Getting Started
+---------------
 
 .. toctree::
    :maxdepth: 3
    :glob:
 
+   about
+   install
    intro
+   globals
+   commandline
+
+
+Module Index
+------------
+
+Each of the ``pwntools`` modules is documented here.
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
    asm
+   atexception
+   atexit
    constants
    context
    dynelf
@@ -17,10 +39,14 @@ Contents:
    gdb
    log
    memleak
-   tubes
+   replacements
+   rop
    shellcraft
    term
+   timeout
+   tubes
    ui
+   useragents
    util/*
 
 Indices and tables

docs/source/install.rst

@@ -0,0 +1,46 @@
+Installation
+============
+
+pwntools is best supported on Ubuntu 12.04 and 14.04, but most
+functionality should work on any Posix-like distribution (Debian, Arch,
+FreeBSD, OSX, etc.).
+
+Prerequisites
+-------------
+
+In order to get the most out of ``pwntools``, you should have the
+following system libraries installed.
+
+.. toctree::
+   :maxdepth: 3
+   :glob:
+
+   install/*
+
+Released Version
+-----------------
+
+Pwntools is available as a ``pip`` package.
+
+.. code-block:: bash
+
+    $ apt-get install python2.7 python2.7-dev python-pip
+    $ pip install pwntools
+
+Latest Version
+--------------
+
+Alternatively if you prefer to use the latest version from the
+repository:
+
+.. code-block:: bash
+
+    $ git clone https://github.com/Gallopsled/pwntools
+    $ cd pwntools
+    $ pip install -e .
+
+.. _Ubuntu: https://launchpad.net/~pwntools/+archive/ubuntu/binutils
+.. _libcapstone 2.1: http://www.capstone-engine.org
+.. _i386: http://www.capstone-engine.org/download/2.1.2/capstone-2.1.2_i386.deb
+.. _amd64: http://www.capstone-engine.org/download/2.1.2/capstone-2.1.2_amd64.deb
+

docs/source/install/binutils.rst

@@ -0,0 +1,82 @@
+Binutils
+-------------
+
+Assembly of foreign architectures (e.g. assembling Sparc shellcode on
+Mac OS X) requires cross-compiled versions of ``binutils`` to be
+installed. We've made this process as smooth as we can.
+
+In these examples, replace ``$ARCH`` with your target architecture (e.g., arm, mips64, vax, etc.).
+
+Ubuntu
+^^^^^^^^^^^^^^^^
+First, add our `Personal Package Archive repository <http://binutils.pwntools.com>`__.
+
+.. code-block:: bash
+
+    $ apt-get install software-properties-common
+    $ apt-add-repository ppa:pwntools/binutils
+    $ apt-get updatez
+
+Then, install the binutils for your architecture.
+
+.. code-block:: bash
+
+    $ apt-get install binutils-$ARCH-linux-gnu
+
+Mac OS X
+^^^^^^^^^^^^^^^^
+
+Mac OS X is just as easy, but requires building binutils from source.
+However, we've made ``homebrew`` recipes to make this a single command.
+After installing `brew <http://brew.sh>`__, grab the appropriate
+recipe from our `binutils
+repo <https://github.com/Gallopsled/pwntools-binutils/>`__.
+
+.. code-block:: bash
+
+    $ brew install https://raw.githubusercontent.com/Gallopsled/pwntools-binutils/master/osx/binutils-$ARCH.rb
+
+Alternate OSes
+^^^^^^^^^^^^^^^^
+
+If you want to build everything by hand, or don't use any of the above
+OSes, ``binutils`` is simple to build by hand.
+
+.. code-block:: bash
+
+    #!/usr/bin/env bash
+
+    V=2.25   # Binutils Version
+    ARCH=arm # Target architecture
+
+    cd /tmp
+    wget -nc https://ftp.gnu.org/gnu/binutils/binutils-$V.tar.gz
+    wget -nc https://ftp.gnu.org/gnu/binutils/binutils-$V.tar.gz.sig
+
+    gpg --keyserver keys.gnupg.net --recv-keys 4AE55E93
+    gpg --verify binutils-$V.tar.gz.sig
+
+    rm -rf binutils-*
+
+    tar xf binutils-$V.tar.gz
+
+    mkdir binutils-build
+    cd binutils-build
+
+    export AR=ar
+    export AS=as
+
+    ../binutils-$V/configure \
+        --prefix=/usr/local \
+        --target=$ARCH-unknown-linux-gnu \
+        --disable-static \
+        --disable-multilib \
+        --disable-werror \
+        --disable-nls
+
+    MAKE=gmake
+    hash gmake || MAKE=make
+
+    $MAKE -j
+    sudo $MAKE install
+

docs/source/install/capstone.rst

@@ -0,0 +1,27 @@
+Capstone
+-------------
+
+Capstone is a disassembly library required for gathering ROP gadgets
+and ROP chain generation.
+
+It's a separate requirement from ``binutils`` because it's used by
+Jon Salwan's ``ROPgadget`` tool which we use under the covers.
+
+In particular, version ``2.1.2`` should be used.  Capstone can be downloaded here_, or installed with the steps below.
+
+Ubuntu
+^^^^^^^^^^^^^^^^
+
+.. code-block:: bash
+
+    $ wget -nc http://www.capstone-engine.org/download/2.1.2/capstone-2.1.2_amd64.deb
+    $ apt-get install capstone-2.1.2_amd64.deb
+
+Mac OS X
+^^^^^^^^^^^^^^^^
+
+.. code-block:: bash
+
+    $ brew install capstone
+
+.. _here:  http://www.capstone-engine.org/download.html

docs/source/install/headers.rst

@@ -0,0 +1,18 @@
+Python Development Headers
+-------------
+
+Some of pwntools' Python dependencies require native extensions (for example, Paramiko requires PyCrypto).
+
+In order to build these native extensions, the development headers for Python must be installed.
+
+Ubuntu
+^^^^^^^^^^^^^^^^
+
+.. code-block:: bash
+
+    $ apt-get install python-dev
+
+Mac OS X
+^^^^^^^^^^^^^^^^
+
+No action needed.

docs/source/intro.rst

@@ -1,53 +1,221 @@
-Introduction to pwntools
+.. testsetup:: *
+
+   from pwn import *
+
+Getting Started
 ========================
 
-Historically pwntools was used as a sort of exploit-writing DSL. Simply doing
-``from pwn import *`` in a previous version of pwntools would bring all sorts of
-nice side-effects.
+To get your feet wet with pwntools, let's first go through a few examples.
 
-When redesigning pwntools for 2.0, we noticed two contrary goals:
+When writing exploits, pwntools generally follows the "kitchen sink" approach.
 
-* We would like to have a "normal" python module structure, to allow other
-  people to faster get familiar with how pwntools works.
-* We would like to have even more side-effects, especially by putting the
-  terminal in raw-mode.
+    >>> from pwn import *
 
-To make this possible, we decided to have two different modules. :mod:`pwnlib`
-would be our nice, clean python module, while :mod:`pwn` would be used during
-CTFs.
+This imports a lot of functionality into the global namespace.  You can now
+assemble, disassemble, pack, unpack, and many other things with a single function.
 
-:mod:`pwnlib` --- Normal python library
----------------------------------------
+A full list of everything that is imported is available on :doc:`globals`.
 
-.. module:: pwnlib
 
-This module is our "clean" python-code. As a rule, we do not think that
-importing :mod:`pwnlib` or any of the submodules should have any significant
-side-effects (besides e.g. caching).
+Making Connections
+------------------
 
-For the most part, you will also only get the bits you import. You for instance
-not get access to :mod:`pwnlib.util.packing` simply by doing ``import
-pwnlib.util``.
+You need to talk to the challenge binary in order to pwn it, right?
+pwntools makes this stupid simple with its :mod:`pwnlib.tubes` module.
 
-Though there are a few exceptions (such as :mod:`pwnlib.shellcraft`), that does
-not quite fit the goals of being simple and clean, but they can still be
-imported without implicit side-effects.
+This exposes a standard interface to talk to processes, sockets, serial ports,
+and all manner of things, along with some nifty helpers for common tasks.
+For example, remote connections via :mod:`pwnlib.tubes.remote`.
 
-:mod:`pwn` --- Toolbox optimized for CTFs
------------------------------------------
+    >>> conn = remote('ftp.debian.org',21)
+    >>> conn.recvline() # doctest: +ELLIPSIS
+    '220 ...'
+    >>> conn.send('USER anonymous\r\n')
+    >>> conn.recvuntil(' ', drop=True)
+    '331'
+    >>> conn.recvline()
+    'Please specify the password.\r\n'
+    >>> conn.close()
 
-.. module:: pwn
+It's also easy to spin up a listener
 
-As stated, we would also like to have the ability to get a lot of these
-side-effects by default. That is the purpose of this module. It does
-the following:
+    >>> l = listen()
+    >>> r = remote('localhost', l.lport)
+    >>> c = l.wait_for_connection()
+    >>> r.send('hello')
+    >>> c.recv()
+    'hello'
+
+Interacting with processes is easy thanks to :mod:`pwnlib.tubes.process`.
+
+::
+
+    >>> sh = process('/bin/sh')
+    >>> sh.sendline('sleep 3; echo hello world;')
+    >>> sh.recvline(timeout=1)
+    ''
+    >>> sh.recvline(timeout=5)
+    'hello world\n'
+    >>> sh.close()
+
+Not only can you interact with processes programmatically, but you can
+actually **interact** with processes.
+
+    >>> sh.interactive() # doctest: +SKIP
+    $ whoami
+    user
+
+There's even an SSH module for when you've got to SSH into a box to perform
+a local/setuid exploit with :mod:`pwnlib.tubes.ssh`.  You can quickly spawn
+processes and grab the output, or spawn a process and interact iwth it like
+a ``process`` tube.
+
+::
+
+    >>> shell = ssh('bandit0', 'bandit.labs.overthewire.org', password='bandit0')
+    >>> shell['whoami']
+    'bandit0'
+    >>> shell.download_file('/etc/motd')
+    >>> sh = shell.run('sh')
+    >>> sh.sendline('sleep 3; echo hello world;') # doctest: +SKIP
+    >>> sh.recvline(timeout=1)
+    ''
+    >>> sh.recvline(timeout=5)
+    'hello world\n'
+    >>> shell.close()
+
+Packing Integers
+------------------
+
+A common task for exploit-writing is converting between integers as Python
+sees them, and their representation as a sequence of bytes.
+Usually folks resort to the built-in ``struct`` module.
+
+pwntools makes this easier with :mod:`pwnlib.util.packing`.  No more remembering
+unpacking codes, and littering your code with helper routines.
+
+    >>> import struct
+    >>> p32(0xdeadbeef) == struct.pack('I', 0xdeadbeef)
+    True
+    >>> leet = '37130000'.decode('hex')
+    >>> u32('abcd') == struct.unpack('I', 'abcd')[0]
+    True
+
+The packing/unpacking operations are defined for many common bit-widths.
+
+    >>> u8('A') == 0x41
+    True
+
+Setting the Target Architecture and OS
+--------------------------------------
+
+The target architecture can generally be specified as an argument to the routine that requires it.
+
+    >>> asm('nop')
+    '\x90'
+    >>> asm('nop', arch='arm')
+    '\x00\xf0 \xe3'
+
+However, it can also be set once in the global ``context``.  The operating system, word size, and endianness can also be set here.
+
+    >>> context.arch      = 'i386'
+    >>> context.os        = 'linux'
+    >>> context.endian    = 'little'
+    >>> context.word_size = 32
+
+Additionally, you can use a shorthand to set all of the values at once.
+
+    >>> asm('nop')
+    '\x90'
+    >>> context(arch='arm', os='linux', endian='big', word_size=32)
+    >>> asm('nop')
+    '\xe3 \xf0\x00'
+
+.. doctest::
+   :hide:
+
+    >>> context.clear()
+
+Setting Logging Verbosity
+-------------------------
+
+You can control the verbosity of the standard pwntools logging via ``context``.
+
+For example, setting
+
+    >>> context.log_level = 'debug'
+
+Will cause all of the data sent and received by a ``tube`` to be printed to the screen.
+
+.. doctest::
+   :hide:
+
+    >>> context.clear()
+
+Assembly and Disassembly
+------------------------
+
+Never again will you need to run some already-assembled pile of shellcode
+from the internet!  The :mod:`pwnlib.asm` module is full of awesome.
+
+    >>> asm('mov eax, 0').encode('hex')
+    'b800000000'
+
+But if you do, it's easy to suss out!
+
+    >>> print disasm('6a0258cd80ebf9'.decode('hex'))
+       0:   6a 02                   push   0x2
+       2:   58                      pop    eax
+       3:   cd 80                   int    0x80
+       5:   eb f9                   jmp    0x0
+
+However, you shouldn't even need to write your own shellcode most of the
+time!  Pwntools comes with the :mod:`pwnlib.shellcraft` module, which is
+loaded with useful time-saving shellcodes.
+
+Let's say that we want to `setreuid(getuid(), getuid())` followed by `dup`ing
+file descriptor 4 to `stdin`, `stdout`, and `stderr`, and then pop a shell!
+
+    >>> asm(shellcraft.setreuid() + shellcraft.dupsh(4)).encode('hex')
+    '6a3158cd8089c389d96a4658cd806a045b6a0359496a3f58cd8075f86a68682f2f2f73682f62696e89e331c96a0b5899cd80'
+
+
+Misc Tools
+----------------------
+
+Never write another hexdump, thanks to :mod:`pwnlib.util.fiddling`.
+
+
+Find offsets in your buffer that cause a crash, thanks to :mod:`pwnlib.cyclic`.
+
+    >>> print cyclic(20)
+    aaaabaaacaaadaaaeaaa
+    >>> # Assume EIP = 0x62616166 ('faab') at crash time
+    >>> print cyclic_find('faab')
+    120
+
+ELF Manipulation
+----------------
+
+Stop hard-coding things!  Look them up at runtime with :mod:`pwnlib.elf`.
+
+    >>> e = ELF('/bin/cat')
+    >>> print hex(e.address) #doctest: +SKIP
+    0x400000
+    >>> print hex(e.symbols['write']) #doctest: +SKIP
+    0x401680
+    >>> print hex(e.got['write']) #doctest: +SKIP
+    0x60b070
+    >>> print hex(e.plt['write']) #doctest: +SKIP
+    0x401680
+
+You can even patch and save the files.
+
+    >>> e = ELF('/bin/cat')
+    >>> e.read(e.address+1, 3)
+    'ELF'
+    >>> e.asm(e.address, 'ret')
+    >>> e.save('/tmp/quiet-cat')
+    >>> disasm(file('/tmp/quiet-cat','rb').read(1))
+    '   0:   c3                      ret'
 
-* Imports everything from the toplevel :mod:`pwnlib` along with
-  functions from a lot of submodules. This means that if you do
-  ``import pwn`` or ``from pwn import *``, you will access to
-  everything you need to write an exploit.
-* Calls :func:`pwnlib.term.init` to put your terminal in raw mode
-  and implementing functionality to make it look like it is not.
-* Setting the :data:`pwnlib.context.log_level` to `"info"`.
-* Tries to parse some of the values in :data:`sys.argv` and every
-  value it succeeds in parsing it removes.

docs/source/log.rst

@@ -2,10 +2,6 @@
 =================================================================
 
 .. automodule:: pwnlib.log
-   :members:
 
-:mod:`pwnlib.log_levels` --- Possible log levels
-------------------------------------------------
-
-.. automodule:: pwnlib.log_levels
-   :members:
+.. autoclass:: pwnlib.log.Logger
+   :members: debug, info, warn, error, progress, success, failure, indented, info_once, warn_once

docs/source/memleak.rst

@@ -1,3 +1,7 @@
+.. testsetup:: *
+
+   from pwnlib.memleak import *
+
 :mod:`pwnlib.memleak` --- Helper class for leaking memory
 =========================================================
 

docs/source/replacements.rst

@@ -0,0 +1,5 @@
+:mod:`pwnlib.replacements` --- Replacements for various functions
+=================================================================
+
+.. automodule:: pwnlib.replacements
+   :members:

docs/source/rop.rst

@@ -0,0 +1,11 @@
+.. testsetup:: *
+
+   from pwnlib.elf import *
+   from pwnlib.rop import *
+   from pwnlib.util.misc import which
+
+:mod:`pwnlib.rop` --- Return Oriented Programming
+=================================================
+
+.. automodule:: pwnlib.rop
+   :members:

docs/source/shellcraft/amd64.rst

@@ -0,0 +1,18 @@
+.. testsetup:: *
+
+   from pwn import *
+
+:mod:`pwnlib.shellcraft.amd64` --- Shellcode for AMD64
+===========================================================
+
+:mod:`pwnlib.shellcraft.amd64`
+---------------------------------------
+
+.. automodule:: pwnlib.shellcraft.amd64
+   :members:
+
+:mod:`pwnlib.shellcraft.amd64.linux`
+---------------------------------------
+
+.. automodule:: pwnlib.shellcraft.amd64.linux
+   :members:

docs/source/shellcraft/arm.rst

@@ -0,0 +1,18 @@
+.. testsetup:: *
+
+   from pwn import *
+
+:mod:`pwnlib.shellcraft.arm` --- Shellcode for ARM
+===========================================================
+
+:mod:`pwnlib.shellcraft.arm`
+-----------------------------
+
+.. automodule:: pwnlib.shellcraft.arm
+   :members:
+
+:mod:`pwnlib.shellcraft.arm.linux`
+-----------------------------------
+
+.. automodule:: pwnlib.shellcraft.arm.linux
+   :members:

docs/source/shellcraft/i386.rst

@@ -1,3 +1,7 @@
+.. testsetup:: *
+
+   from pwn import *
+
 :mod:`pwnlib.shellcraft.i386` --- Shellcode for Intel 80386
 ===========================================================
 

docs/source/shellcraft/thumb.rst

@@ -0,0 +1,18 @@
+.. testsetup:: *
+
+   from pwn import *
+
+:mod:`pwnlib.shellcraft.thumb` --- Shellcode for Thumb Mode
+===========================================================
+
+:mod:`pwnlib.shellcraft.thumb`
+-------------------------------
+
+.. automodule:: pwnlib.shellcraft.thumb
+   :members:
+
+:mod:`pwnlib.shellcraft.thumb.linux`
+---------------------------------------
+
+.. automodule:: pwnlib.shellcraft.thumb.linux
+   :members:

docs/source/timeout.rst

@@ -0,0 +1,11 @@
+.. testsetup:: *
+
+   from pwnlib.context import context
+   from pwnlib.timeout import Timeout
+   import time
+
+:mod:`pwnlib.timeout` --- Timeout handling
+========================================
+
+.. automodule:: pwnlib.timeout
+   :members:

docs/source/tubes.rst

@@ -1,3 +1,11 @@
+.. testsetup:: *
+
+   from pwnlib.tubes.tube import tube
+   from pwnlib.tubes.ssh import *
+   from pwnlib.tubes.remote import remote
+   from pwnlib.util.misc import which
+   from pwnlib.context import context
+
 :mod:`pwnlib.tubes` --- Talking to the World!
 =============================================
 
@@ -8,13 +16,13 @@
 
    .. automodule:: pwnlib.tubes.remote
 
-      .. autoclass:: pwnlib.tubes.remote.remote(host, port, fam = None, typ = None, proto = 0, timeout = 'default', log_level = INFO)
+      .. autoclass:: pwnlib.tubes.remote.remote
          :members:
          :show-inheritance:
 
    .. automodule:: pwnlib.tubes.listen
 
-      .. autoclass:: pwnlib.tubes.listen.listen(port, bindaddr = "0.0.0.0", fam = "any", typ = "tcp", timeout = 'default', log_level = INFO)
+      .. autoclass:: pwnlib.tubes.listen.listen
          :members:
          :show-inheritance:
 
@@ -28,7 +36,7 @@
 
    .. automodule:: pwnlib.tubes.process
 
-      .. autoclass:: pwnlib.tubes.process.process(args, shell = False, executable = None, env = None, timeout = 'default', log_level = INFO)
+      .. autoclass:: pwnlib.tubes.process.process
          :members: kill, poll, communicate
          :show-inheritance:
 
@@ -37,13 +45,18 @@
 
    .. automodule:: pwnlib.tubes.ssh
 
-      .. autoclass:: pwnlib.tubes.ssh.ssh(user, host, port = 22, password = None, key = None, keyfile = None, proxy_command = None, proxy_sock = None, timeout = 'default', log_level = INFO)
+      .. autoclass:: pwnlib.tubes.ssh.ssh
          :members:
 
       .. autoclass:: pwnlib.tubes.ssh.ssh_channel()
          :members: kill, poll, interactive
          :show-inheritance:
 
+      .. autoclass:: pwnlib.tubes.ssh.ssh_connecter()
+         :show-inheritance:
+
+      .. autoclass:: pwnlib.tubes.ssh.ssh_listener()
+         :show-inheritance:
 
    Common functionality
    --------------------

docs/source/useragents.rst

@@ -0,0 +1,9 @@
+.. testsetup:: *
+
+   from pwnlib.useragents import *
+
+:mod:`pwnlib.useragents` --- A database of useragent strings
+============================================================
+
+.. automodule:: pwnlib.useragents
+   :members:

docs/source/util/hashes.rst

@@ -0,0 +1,5 @@
+:mod:`pwnlib.util.hashes` --- Hashing functions
+===============================================
+
+.. automodule:: pwnlib.util.hashes
+   :members:

docs/source/util/iters.rst

@@ -0,0 +1,113 @@
+.. testsetup:: *
+
+   import string, operator
+   from pwnlib.util.iters import *
+
+
+:mod:`pwnlib.util.iters` --- Extension of standard module :mod:`itertools`
+==========================================================================
+
+.. automodule:: pwnlib.util.iters
+   :members:
+   :exclude-members:
+     chain                                  ,
+     combinations                           ,
+     combinations_with_replacement          ,
+     compress                               ,
+     count                                  ,
+     cycle                                  ,
+     dropwhile                              ,
+     groupby                                ,
+     ifilter                                ,
+     ifilterfalse                           ,
+     imap                                   ,
+     islice                                 ,
+     izip                                   ,
+     izip_longest                           ,
+     permutations                           ,
+     product                                ,
+     repeat                                 ,
+     starmap                                ,
+     takewhile                              ,
+     tee
+
+.. function:: chain
+
+   Alias for :func:`itertools.chain`.
+
+.. function:: combinations
+
+   Alias for :func:`itertools.combinations`
+
+.. function:: combinations_with_replacement
+
+   Alias for :func:`itertools.combinations_with_replacement`
+
+.. function:: compress
+
+   Alias for :func:`itertools.compress`
+
+.. function:: count
+
+   Alias for :func:`itertools.count`
+
+.. function:: cycle
+
+   Alias for :func:`itertools.cycle`
+
+.. function:: dropwhile
+
+   Alias for :func:`itertools.dropwhile`
+
+.. function:: groupby
+
+   Alias for :func:`itertools.groupby`
+
+.. function:: ifilter
+
+   Alias for :func:`itertools.ifilter`
+
+.. function:: ifilterfalse
+
+   Alias for :func:`itertools.ifilterfalse`
+
+.. function:: imap
+
+   Alias for :func:`itertools.imap`
+
+.. function:: islice
+
+   Alias for :func:`itertools.islice`
+
+.. function:: izip
+
+   Alias for :func:`itertools.izip`
+
+.. function:: izip_longest
+
+   Alias for :func:`itertools.izip_longest`
+
+.. function:: permutations
+
+   Alias for :func:`itertools.permutations`
+
+.. function:: product
+
+   Alias for :func:`itertools.product`
+
+.. function:: repeat
+
+   Alias for :func:`itertools.repeat`
+
+.. function:: starmap
+
+   Alias for :func:`itertools.starmap`
+
+.. function:: takewhile
+
+   Alias for :func:`itertools.takewhile`
+
+.. function:: tee
+
+   Alias for :func:`itertools.tee`
+

docs/source/util/misc.rst

@@ -1,7 +1,7 @@
 .. testsetup:: *
 
    from pwnlib.util.misc import *
-   import os
+   import os, subprocess
    os.chdir("..")
 
 :mod:`pwnlib.util.misc` --- We could not fit it any other place

docs/source/util/proc.rst

@@ -1,6 +1,7 @@
 .. testsetup:: *
 
    from pwnlib.util.proc import *
+   import os, sys
 
 
 :mod:`pwnlib.util.proc` --- Working with ``/proc/``

docs/source/util/web.rst

@@ -0,0 +1,10 @@
+.. testsetup:: *
+
+   from pwnlib.context import context
+   from pwnlib.util.web import *
+
+:mod:`pwnlib.util.web` --- Utilities for working with the WWW
+=============================================================
+
+.. automodule:: pwnlib.util.web
+   :members:

examples/README

@@ -1,6 +0,0 @@
-While these examples should all work, they are not very representative of
-the pwntools project.
-
-We have a plan to create a separate repository with examples, primarily
-exploits. Until we do so, we recommend new users to look at
-http://pwntools.readthedocs.org, as this is a better overview of our features.

examples/README.md

@@ -0,0 +1,109 @@
+# Examples
+While these examples should all work, they are not very representative of
+the pwntools project.
+
+We have a plan to create a separate repository with examples, primarily
+exploits. Until we do so, we recommend new users to look at
+https://pwntools.readthedocs.org, as this is a better overview of our features.
+
+In no particular order the docstrings for each example:
+
+* `args.py`
+```
+When not in lib-mode (import `pwn` rather than `pwnlib`) we parse the
+commandline for variables definitions.  A variable definition has the form::
+
+  <var>=<val>
+
+where ``<var>`` contains only uppercase letters, digits and underscores and
+doesn't start with a digit.
+
+Try running this example with::
+
+  $ python args.py RHOST=localhost RPORT=1337
+```
+* `asm.py`
+```
+Example showing the interface to `pwnlib.asm.asm` and `pwnlib.shellcraft`.
+```
+* `clean_and_log.py`
+```
+Use case for `pwnlib.tubes.tube.clean_and_log`.
+
+Sometimes you will have a solution to a challenge but you don't know what it
+will look like when you get the flag.  Sometimes that will leave you with a
+top-level exception, no flag, and angry team members.
+
+Solution:
+ 1. Always run wireshark or tcpdump.  Always.
+ 2. Register <your socket>.clean or <your socket>.clean_and_log to run at exit.
+```
+* `indented.py`
+```
+When running in term-mode (import `pwn` rather than `pwnlib`, stdout is a TTY
+and not running in a REPL), we can do proper indentation where lines too long to
+fit on a screen are split into multiple individually indented lines.
+
+Too see the difference try running with::
+
+  $ python indented.py
+
+and
+
+  $ python -i indented.py
+
+Also notice that `pause()` can react on any key when in `term_mode`.
+```
+* `listen_uroboros.py`
+```
+An example showing interconnection of sockets.  This script will wait for three
+connections on port 1337, then connect them like a three-way Uroboros.
+```
+* `options.py`
+```
+Example showing `pwnlib.ui.options()`
+```
+* `port_forward.py`
+```
+A very simple port forwarder using `pwnlib.tubes.tube.connect_both()`.  Notice
+that `<>` is just a shorthand.
+```
+* `readline_completers.py`
+```
+Example showing pwnlib's readline implementation and a few completers.  This
+part of pwnlib will probably see some major changes soon, but we wanted to show
+off some proof-of-concepts.
+```
+* `remote_gdb_debugging.py`
+```
+Simple example showing how to use the remote
+gdb debugging features available in pwntools.
+```
+* `spinners.py`
+```
+Just a lot of spinners!
+```
+* `splash.py`
+```
+"Easteregg"
+```
+* `ssh.py`
+```
+Example showing how to use the ssh class.
+```
+* `text.py`
+```
+Example showing how to use `pwnlib.term.text`.
+
+Try running with::
+
+  $ TERM=xterm python text.py
+
+and::
+
+  $ TERM=xterm-256color python text.py
+```
+* `remote.py`
+```
+Example showing how to use the remote class.
+```

examples/args.py

@@ -0,0 +1,18 @@
+"""
+When not in lib-mode (import `pwn` rather than `pwnlib`) we parse the
+commandline for variables definitions.  A variable definition has the form::
+
+  <var>=<val>
+
+where ``<var>`` contains only uppercase letters, digits and underscores and
+doesn't start with a digit.
+
+Try running this example with::
+
+  $ python args.py RHOST=localhost RPORT=1337
+"""
+
+from pwn import *
+
+print args['RHOST'] or 'RHOST is not set'
+print args['RPORT'] or 'RPORT is not set'

examples/argstest.py

@@ -1,4 +0,0 @@
-import pwn
-
-print pwn.args['RHOST'],
-print pwn.args['RPORT']

examples/asm.py

@@ -1,3 +1,7 @@
+"""
+Example showing the interface to `pwnlib.asm.asm` and `pwnlib.shellcraft`.
+"""
+
 from pwn import *
 context(arch='i386', os='linux', net='tcp4')
 

examples/clean_and_log.py

@@ -0,0 +1,29 @@
+"""
+Use case for `pwnlib.tubes.tube.clean_and_log`.
+
+Sometimes you will have a solution to a challenge but you don't know what it
+will look like when you get the flag.  Sometimes that will leave you with a
+top-level exception, no flag, and angry team members.
+
+Solution:
+ 1. Always run wireshark or tcpdump.  Always.
+ 2. Register <your socket>.clean or <your socket>.clean_and_log to run at exit.
+"""
+
+from pwn import *
+
+os.system('''((
+echo prefix sometext ;
+echo prefix someothertext ;
+echo here comes the flag ;
+echo LostInTheInterTubes
+) | nc -l 1337) &
+''')
+
+r = remote('localhost', 1337)
+atexit.register(r.clean_and_log)
+
+while True:
+    line = r.recvline()
+    print re.findall('^prefix (\S+)$', line)[0]
+

examples/completertest.py

@@ -1,15 +0,0 @@
-import pwn
-import pwnlib.term.completer
-
-c1 = pwn.term.completer.LongestPrefixCompleter([
-    'foobar',
-    'foobaz',
-    'fooquux',
-    ])
-
-c2 = pwn.term.completer.PathCompleter(mask = '*.py')
-
-with c1:
-    while True:
-        s = pwn.term.readline.readline(prompt = '> ')
-        print 'You wrote', s

examples/contexttest.py

@@ -1,6 +0,0 @@
-import pwn
-
-print pwn.context
-
-pwn.context.arch = 'i386'
-pwn.context.arch = 'yolo'

examples/gen-README.py

@@ -0,0 +1,41 @@
+"""
+Script to generate README.md
+"""
+
+from pwn import *
+
+out = '''# Examples
+While these examples should all work, they are not very representative of
+the pwntools project.
+
+We have a plan to create a separate repository with examples, primarily
+exploits. Until we do so, we recommend new users to look at
+https://pwntools.readthedocs.org, as this is a better overview of our features.
+
+In no particular order the docstrings for each example:
+
+'''
+
+def append_example(_arg, top, names):
+    global out
+    for name in names:
+        if not (name.endswith('.py') and name != __file__):
+            continue
+        path = os.path.join(top, name)[2:] # strip './'
+        log.info('-> %s' % path)
+        data = read(path).strip()
+        if data[0:3] not in ('"""', "'''"):
+            log.warning('  Has no docstring!')
+            continue
+        try:
+            i = data.index(data[0:3], 3)
+        except ValueError:
+            log.warning('  Docstring is weird')
+            continue
+        doc = util.safeeval.const(data[0:i + 3])
+        out += '* `%s`\n' % path
+        out += '```%s```\n' % doc
+
+os.path.walk('.', append_example, None)
+
+write('README.md', out)

examples/indented.py

@@ -0,0 +1,25 @@
+"""
+When running in term-mode (import `pwn` rather than `pwnlib`, stdout is a TTY
+and not running in a REPL), we can do proper indentation where lines too long to
+fit on a screen are split into multiple individually indented lines.
+
+Too see the difference try running with::
+
+  $ python indented.py
+
+and
+
+  $ python -i indented.py
+
+Also notice that `pause()` can react on any key when in `term_mode`.
+"""
+
+from pwn import *
+
+context.log_level = 'info'
+
+log.indented('A' * 100)
+log.indented('B' * 100)
+log.indented('C' * 100)
+
+pause()

examples/indenttest.py

@@ -1,8 +0,0 @@
-import pwn
-
-pwn.context.log_level = 'info'
-
-# print pwn2.lib
-pwn.log.info(('A' * 100 + '\n') * 4)
-
-raw_input('>')

examples/inputtest.py

@@ -1,48 +0,0 @@
-# raise Exception('1234')
-
-# import sys
-# _write = sys.stdout.write
-# class Proxy:
-#     def __init__(self, fd):
-#         self._fd = fd
-#     def readline(self):
-#         _write('Input plox: ')
-#         return self._fd.readline()
-#     def read(self, n):
-#         _write('xxx')
-#         return self._fd.read(n)
-#     def __getattr__(self, k):
-#         # _write('[%s]\n' % k)
-#         return self._fd.__getattribute__(k)
-
-# # sys.stdout = Proxy(sys.stdout)
-# sys.stdin = Proxy(sys.stdin)
-
-# print raw_input()
-
-
-import pwn
-import pwnlib.term.readline
-
-h = pwn.term.output('yolo\n')
-i = 0
-def foo(_):
-    global i
-    i += 1
-    pwn.term.output('foooooo%d\n' % i, frozen = True, before = h)
-
-def bar():
-    print
-
-def baz(_):
-    import sys
-    sys.exit()
-
-pwnlib.term.readline.startup_hook = bar
-pwnlib.term.readline.keymap.register({
-    'C-x C-a': foo,
-    'C-x C-s': baz,
-    })
-
-while True:
-    print raw_input('> ')

examples/iotest.py

@@ -1,70 +0,0 @@
-import termios, tty, sys, string, time
-
-def get_term_size():
-    fd = os.open(os.ctermid(), os.O_RDONLY)
-    height, width = struct.unpack('hh', fcntl.ioctl(fd, termios.TIOCGWINSZ, '1234'))
-    os.close(fd)
-    return width, height
-
-def get_cursor_pos():
-    sys.stdout.write("\033[6n")
-    s = ''
-    while True:
-        c = sys.stdin.read(1)
-        if c == 'R':
-            break
-        s += c
-    col, row = map(int, s[2:].split(';'))
-    return col, row
-
-fd = sys.stdin.fileno()
-old_settings = termios.tcgetattr(fd)
-tty.setraw(sys.stdin.fileno())
-
-while True:
-    # print 'At (%d, %d)' % get_cursor_pos()
-    c = sys.stdin.read(1)
-    if c == 'q':
-        break
-    if c == '\r':
-        print
-    if c == 'w':
-        s = '\x01\x1b[s\x1b[50Ayolo\x1b[u\x02'
-        sys.stdout.write(s)
-    if c == 'e':
-        sys.stdout.write('\x1b[s')
-        for i in range(100):
-            sys.stdout.write('\x1b[0;15f%s' % '/-\\|'[i % 4])
-            sys.stdout.flush()
-            time.sleep(0.1)
-        sys.stdout.write('\x1b[u')
-    if c == 'E':
-        sys.stdout.write('\x1b[s')
-        for i in range(100):
-            sys.stdout.write('\x1b[0;15H%s' % '.oO*'[i % 4])
-            sys.stdout.flush()
-            time.sleep(0.1)
-        sys.stdout.write('\x1b[u')
-    if c == 'r':
-        sys.stdout.write('\x1b[S')
-    if c == 't':
-        sys.stdout.write('\x1b[T')
-    if c == 'y':
-        sys.stdout.write('\x1b[syolo\r\n\x1b[ufoo')
-    if c == 'u':
-        x = ord('a')
-        for i in range(10):
-            sys.stdout.write('\x1b[A' + chr(x))
-            sys.stdout.flush()
-            time.sleep(0.3)
-            x += 1
-            sys.stdout.write('\x1b[B' + chr(x))
-            sys.stdout.flush()
-            time.sleep(0.3)
-            x += 1
-    if c in string.printable:
-        sys.stdout.write(c)
-    else:
-        sys.stdout.write('\\x%02x' % ord(c))
-
-termios.tcsetattr(fd, termios.TCSADRAIN, old_settings)

examples/keymaptest.py

@@ -1,27 +0,0 @@
-import pwn
-
-def match(t):
-    print 'match', t
-
-def nomatch(t):
-    print 'nomatch', t
-
-def key(t):
-    print 'key', t
-
-def tab(t):
-    print 'tab', t
-
-def tabtab(t):
-    print 'tabtab', t
-
-import pwnlib.term.keymap
-km = pwnlib.term.keymap.Keymap({
-    '<match>'     : match,
-    '<nomatch>'   : nomatch,
-    '<any>'       : key,
-    '<tab>'       : tab,
-    '<tab> <tab>' : tabtab,
-    })
-
-km.handle_input()

examples/listen_uroboros.py

@@ -0,0 +1,14 @@
+"""
+An example showing interconnection of sockets.  This script will wait for three
+connections on port 1337, then connect them like a three-way Uroboros.
+"""
+
+from pwn import *
+
+cs = [listen(1337).wait_for_connection() for _ in range(3)]
+
+cs[0] << cs[1] << cs[2] << cs[0]
+
+cs[0].wait_for_close()
+cs[1].wait_for_close()
+cs[2].wait_for_close()

examples/logtest.py

@@ -1,55 +0,0 @@
-# import pwn2.lib as lib
-import pwn
-import time
-
-pwn.context.log_level = 0
-
-n = 0
-
-h = pwn.log.waitfor('spinners running', status = '0')
-
-hs = []
-while True:
-    s = raw_input('> ')
-    if s == 'q':
-        break
-    hs.append(pwn.log.waitfor(s, status = 'running'))
-    n += 1
-    h.status(str(n))
-
-h.success()
-
-for h in hs:
-    h.failure()
-
-# pwn.log.waitfor('waiting for stuff')
-# time.sleep(0.5)
-# pwn.log.info('something happened')
-# time.sleep(0.5)
-# pwn.log.success('something went well')
-# time.sleep(0.5)
-# pwn.log.waitfor('waiting for some minor stuff')
-# time.sleep(0.5)
-# pwn.log.failure('went wrong')
-# raw_input('please hit <enter> ')
-# time.sleep(0.5)
-# pwn.log.done_failure()
-# time.sleep(0.5)
-# pwn.log.info('all in all it went ok')
-# time.sleep(0.5)
-# pwn.log.done_success()
-
-
-# def foo():
-#     plib.log.stub()
-
-# foo()
-
-# try:
-#     try:
-#         raise KeyError
-#     except:
-#         plib.log.die('no reason')
-# except Exception as e:
-#     print `e`
-#     # raise

examples/memleaktest.py

@@ -1,12 +0,0 @@
-from pwn import *
-
-context(endianness='little', sign='unsigned')
-
-def legit_leak(addr):
-    return p32(addr)
-
-mem = MemLeak(legit_leak)
-print mem.b(0x08040001) # prints 1
-mem.setd(0x08040000, 0xaabbccdd)
-print hex(mem.b(0x08040001))   # prints 0xcc
-print mem.raw(0x08040000, 20)  # prints stuff

examples/options.py

@@ -0,0 +1,8 @@
+"""
+Example showing `pwnlib.ui.options()`
+"""
+
+from pwn import *
+
+opts = [string.letters[x] for x in range(10)]
+print 'You choose "%s"' % opts[options('Pick one:', opts)]

examples/port_forward.py

@@ -0,0 +1,11 @@
+"""
+A very simple port forwarder using `pwnlib.tubes.tube.connect_both()`.  Notice
+that `<>` is just a shorthand.
+"""
+
+from pwn import *
+
+while True:
+    listen(1337).wait_for_connection() <> remote('google.com', 80)
+
+# now point your browser (or curl(1)) to http://localhost:1337

examples/ptytest.py

@@ -1,37 +0,0 @@
-import pty, os, select, sys, time, pwn
-
-pid, fd = pty.fork()
-
-if pid == 0:
-    argv = ('/usr/bin/vim',)
-    argv = ('/bin/bash',)
-    os.execlp(argv[0], *argv)
-else:
-    # time.sleep(1)
-    # sys.stdout.write(os.read(fd, 4096))
-    while True:
-        rfds, wfds, xfds = select.select([fd, 0], [], [])
-        if fd in rfds:
-            s = os.read(fd, 4096)
-            if not s:
-                raise EOFError
-            sys.stdout.write(s)
-            sys.stdout.flush()
-        if 0 in rfds:
-            s = os.read(0, 4096)
-            if not s:
-                raise EOFError
-            os.write(fd, s)
-
-
-# def master_read(fd):
-#     import os
-#     s = os.read(fd, 4096)
-#     return s
-
-# def stdin_read(fd):
-#     import os
-#     s = os.read(fd, 4096)
-#     return s
-
-# pty.spawn('/bin/sh', master_read = master_read, stdin_read = stdin_read)

examples/readline_completers.py

@@ -0,0 +1,31 @@
+"""
+Example showing pwnlib's readline implementation and a few completers.  This
+part of pwnlib will probably see some major changes soon, but we wanted to show
+off some proof-of-concepts.
+"""
+
+from pwn import *
+from pwnlib.term.completer import LongestPrefixCompleter, \
+                                  PathCompleter
+
+c1 = LongestPrefixCompleter([
+    'foobar',
+    'foobaz',
+    'fooqux',
+    'exit',
+    'enough!',
+    ])
+
+c2 = PathCompleter(mask = '*.py')
+
+with c1:
+    print 'type "exit" to exit'
+    while True:
+        s = term.readline.readline(prompt = '> ').strip()
+        if s in ('exit', 'enough!'):
+            break
+        print 'You wrote', s
+with c2:
+    print 'choose a file'
+    s = term.readline.readline(prompt = text.bold_green('$ ')).strip()
+    print 'You picked', s

examples/readlineexc.py

@@ -1,15 +0,0 @@
-
-# import sys
-# class Wrapper:
-#     def __init__(self, fd):
-#         self._fd = fd
-#     def readline(self, size = None):
-#         return ''
-#     def __getattr__(self, k):
-#         return self._fd.__getattribute__(k)
-# sys.stdin = Wrapper(sys.stdin)
-# print `raw_input()`
-
-import readline
-import pwn
-raw_input('> ')

examples/remote.py

@@ -0,0 +1,11 @@
+"""
+Example showing how to use the remote class.
+"""
+
+from pwn import *
+
+sock = remote('127.0.0.1', 9001)
+
+print sock.recvline()
+sock.send('foo')
+sock.interactive()

examples/remote_gdb_debugging.py

@@ -0,0 +1,16 @@
+"""
+Simple example showing how to use the remote
+gdb debugging features available in pwntools.
+"""
+
+from pwn import *
+import getpass
+
+s = ssh(getpass.getuser(), '127.0.0.1', port = 22, keyfile = "~/.ssh/id_rsa")
+c = gdb.ssh_gdb(s, '/bin/sh', execute = '''
+p/x $pc
+c''')
+
+c.sendline('ls -la')
+c.sendline('exit')
+print c.recvall()

examples/spinners.py

@@ -0,0 +1,25 @@
+"""
+Just a lot of spinners!
+"""
+
+from pwn import *
+
+context.log_level = 0
+
+n = 1
+h = log.waitfor('spinners running', status = str(n))
+
+hs = []
+print 'type "q" to quit'
+while True:
+    s = raw_input('> ')
+    if s == 'q':
+        break
+    hs.append(log.waitfor(s, status = 'running'))
+    n += 1
+    h.status(str(n))
+
+h.success()
+
+for h in hs:
+    h.failure()

examples/splash.py

@@ -1,9 +1,12 @@
-import pwn
+"""
+"Easteregg"
+"""
 
-pwn.splash()
-pwn.pause()
+from pwn import *
 
-h = pwn.log.waitfor("Inputting:")
+splash()
+
+h = log.waitfor("You wrote", status = "--")
 
 while True:
     l = raw_input('> ')

examples/ssh.py

@@ -0,0 +1,40 @@
+"""
+Example showing how to use the ssh class.
+"""
+
+from pwn import *
+
+shell = ssh(host='bandit.labs.overthewire.org',user='bandit0',password='bandit0')
+
+# Show basic command syntax
+log.info("username: %s" % shell.whoami())
+log.info("pwd: %s" % shell.pwd())
+
+# Show full tube syntax
+tube = shell.run('cat')
+tube.send("Hello, cat")
+tube.shutdown("out")
+print tube.recvall()
+
+# Show automatic working directories
+shell.set_working_directory()
+log.info("pwd: %s" % shell.pwd())
+
+shell.upload_data("""
+#include <stdio.h>
+int main() {
+    return printf("Hello, world");
+}
+""", 'example.c')
+
+shell.gcc(['example.c','-o','example'])
+
+print shell['./example']
+
+# Show the different styles of calling
+print shell.echo("single string")
+print shell.echo(["list","of","strings"])
+print shell["echo single statement"]
+
+# Show off the interactive shell
+shell.interactive()

examples/termtest.py

@@ -1,93 +0,0 @@
-import pwn
-import pwnlib.term
-from pwnlib.term import *
-import time, os, sys, threading
-
-from pwnlib.log import *
-
-try:
-    # output(os.urandom(10))
-    # print 'foo'
-    # print 'bar'
-    # h = output('')
-    # for i in range(250):
-    #     time.sleep(0.005)
-    #     print 'foo%d' % i
-    #     # print 'xx'
-    #     # time.sleep(1)
-    #     # update(h, 'hello \x1b[3%dmthere\x1b[m\n' % (i % 6 + 1))
-    # # freeze(h)
-    # # print pwn2.nonlib.term.parse('hello \x1b[32mthere\x1b[m\n')
-    # # output('hello \x1b[32mthere\x1b[m\n')
-    # # update(h, 'asdf')
-    # print 'hello'
-
-    # output('fooxxx')
-    # h1 = output('')
-    # h2 = output('bar')
-    # output(' baz\n')
-
-    # # update(h2, 'BAR')
-    # update(h1, '\x1b[K\x1b[2A\x1b[1;31mYOLO\x1b[m\x1b[2B')
-
-    # print 'fooxxx\x1b[2Dbar', 'baz'
-
-    clock = output(float = True, frozen = False)
-    def loop():
-        while True:
-            clock.update('\n' + time.asctime(time.localtime()) + '\n')
-            time.sleep(1)
-    t = threading.Thread(target = loop)
-    t.daemon = True
-    t.start()
-    i = 0
-    # numlines = output()
-    numlines = info(frozen = False)
-    while True:
-        s = ''
-        numlines.update('You have written %d lines' % i)
-        i += 1
-        s = raw_input('> ')
-        output('you wrote: ')
-        output('%s\n' % s, indent = 20)
-        if s == 'q':
-            break
-
-    # output('\xc3\xb8' * 85)
-    # output('\x1b[44m \x1b[m')
-    # # output('\x1b[J')
-    # time.sleep(1)
-
-    # h = output('last\n', float = True)
-    # output('foo')
-    # # update(h, 'LAST\n')
-    # h = output('bar')
-    # output('baz\n')
-    # # delete(h)
-    # update(h, 'BAR')
-
-
-    # output('\x1b[s\x1b[20;40H')
-    # h = output('yolo')
-    # update(h, 'YOLO\x1b[u')
-
-    # output('\n')
-    # pwn2.nonlib.term.goto((1, 0))
-    # pwn2.nonlib.term.put('%d' % pwn2.nonlib.term.scroll)
-    # pwn2.nonlib.term.flush()
-    # time.sleep(1)
-
-    # pwn2.nonlib.term.put('\n'*10)
-
-    pass
-
-except:
-    # pwn2.nonlib.term.fini()
-    # raise e
-    # pwn2.nonlib.term.put(str(e))
-    pass
-    raise
-
-pwn2.nonlib.term.put('\n' * 20)
-for c in pwn2.nonlib.term.cells:
-    pwn2.nonlib.term.put('%s %s %s %s\n' % (c.start, c.end, c.float, `c.content`))

examples/text.py

@@ -0,0 +1,19 @@
+'''
+Example showing how to use `pwnlib.term.text`.
+
+Try running with::
+
+  $ TERM=xterm python text.py
+
+and::
+
+  $ TERM=xterm-256color python text.py
+'''
+from pwn import *
+
+s = 'hello from pwntools'
+print text.black_on_green(s)
+print text.black_on_bright_green(s)
+print text.green_on_black(s)
+print text.bright_green_on_black(s)
+print text.bold_green_on_black(s)

install_local.sh

@@ -1,5 +1,5 @@
 #!/bin/sh
-sudo pip install -r requirements.txt
+sudo pip2 install -r requirements.txt
 
 echo "================================================"
 echo "Now please put this directory in your PYTHONPATH"

pwn/__init__.py

@@ -1,15 +1,9 @@
 # Promote useful stuff to toplevel
-import sys
 from .toplevel import *
 
-# Put the terminal in raw mode
-term.init()
-
-# default log level in non-lib mode
-context.defaults.log_level = 'info'
-
 # look for special args in argv
 def closure():
+    term_mode = True
     import sys
     if not hasattr(sys, 'argv'):
         return
@@ -26,18 +20,55 @@ def closure():
         if not all(c in body for c in s[1:]):
             return False
         return True
+    def asbool(s):
+        if   s.lower() == 'true':
+            return True
+        elif s.lower() == 'false':
+            return False
+        elif s.isdigit():
+            return bool(int(s))
+        else:
+            raise ValueError('must be integer or boolean')
+    # parse environtment variables
+    for k, v in os.environ.items():
+        if not k.startswith('PWNLIB_'):
+            continue
+        k = k[7:]
+        if   k == 'DEBUG':
+            if asbool(v):
+                context.log_level = 'debug'
+        elif k == 'SILENT':
+            if asbool(v):
+                context.log_level = 'ERROR'
+        elif k == 'NOTERM':
+            if asbool(v):
+                term_mode = False
+        elif k == 'LOG_LEVEL':
+            context.log_level = v
+        elif isident(k):
+            args[k] = v
+    # parse command line
     for arg in sys.argv[:]:
         if   arg == 'DEBUG':
             sys.argv.remove(arg)
             context.log_level = 'debug'
-        elif arg == 'NOINFO':
+        elif arg == 'SILENT':
             sys.argv.remove(arg)
-            context.log_level = 'silent'
+            context.log_level = 'ERROR'
+        elif arg == 'NOTERM':
+            term_mode = False
         elif arg.find('=') > 0:
             k, v = arg.split('=', 1)
             if not isident(k):
                 continue
             sys.argv.remove(arg)
             args[k] = v
+    # put the terminal in rawmode unless NOTERM was specified
+    if term_mode:
+        term.init()
+    else:
+        term.text.when = 'never'
 closure()
 del closure
+
+log = logging.getLogger('pwnlib.exploit')

pwn/toplevel.py

@@ -1,26 +1,36 @@
 # Get all the modules from pwnlib
-from pwnlib                 import *
+from pwnlib                  import *
 
 # Promote functions from these modules to toplevel
-from pwnlib.asm             import *
-from pwnlib.dynelf          import DynELF
-from pwnlib.elf             import ELF, load
-from pwnlib.log_levels      import *
-from pwnlib.memleak         import MemLeak
-from pwnlib.tubes.listen    import *
-from pwnlib.tubes.process   import *
-from pwnlib.tubes.remote    import *
-from pwnlib.tubes.ssh       import *
-from pwnlib.ui              import *
-from pwnlib.util            import crc, net, proc
-from pwnlib.util.cyclic     import *
-from pwnlib.util.fiddling   import *
-from pwnlib.util.hashes     import *
-from pwnlib.util.lists      import *
-from pwnlib.util.misc       import *
-from pwnlib.util.packing    import *
-from pwnlib.util.proc       import pidof
-from pwnlib.util.splash     import *
+from pwnlib.asm              import asm, disasm, cpp
+from pwnlib.context          import context, Thread
+from pwnlib.dynelf           import DynELF
+from pwnlib.elf              import ELF, load
+from pwnlib.exception        import PwnlibException
+from pwnlib.memleak          import MemLeak
+from pwnlib.replacements     import *
+from pwnlib.rop              import ROP
+from pwnlib.timeout          import Timeout
+from pwnlib.tubes.listen     import listen
+from pwnlib.tubes.process    import process
+from pwnlib.tubes.remote     import remote
+from pwnlib.tubes.serialtube import serialtube
+from pwnlib.tubes.ssh        import ssh
+from pwnlib.tubes.tube       import tube
+from pwnlib.ui               import *
+from pwnlib.util             import crc, net, proc, iters, safeeval
+from pwnlib.util.cyclic      import *
+from pwnlib.util.fiddling    import *
+from pwnlib.util.hashes      import *
+from pwnlib.util.lists       import *
+from pwnlib.util.misc        import *
+from pwnlib.util.packing     import *
+from pwnlib.util.proc        import pidof
+from pwnlib.util.splash      import *
+from pwnlib.util.web         import *
 
 # Promote these modules, so that "from pwn import *" will let you access them
-import pwn, pwnlib, re, time, sys, os, string, threading, subprocess, struct
+import \
+    collections   , operator      , os            , pwnlib        , \
+    re            , string        , struct        , subprocess    , \
+    sys           , threading     , time          , tempfile

pwnlib/__init__.py

@@ -1,12 +1,19 @@
 __all__ = [
-    'asm', 'constants', 'context', 'dynelf',
-    'elf', 'exception', 'gdb', 'log_levels',
-    'log', 'memleak', 'shellcraft', 'term',
-    'tubes', 'ui', 'util'
+    'atexception' , 'atexit'      , 'asm'         , 'constants'   ,
+    'context'     , 'dynelf'      , 'elf'         , 'exception'   ,
+    'gdb'                         , 'log'         , 'memleak'     ,
+    'replacements', 'rop'         , 'shellcraft'  , 'term'        ,
+    'tubes'       , 'ui'          , 'useragents'  , 'util'
 ]
-from . import asm, constants, context, dynelf
-from . import elf, exception, gdb, log_levels
-from . import log, memleak, shellcraft, term
-from . import tubes, ui, util
 
+
+from . import \
+    atexception   , atexit        , asm           , constants     , \
+                    dynelf        , elf           , exception     , \
+    gdb                           , log           , memleak       , \
+    replacements  , rop           , shellcraft    , term          , \
+    tubes         , ui            , useragents    , util,           \
+    pep237
+
+# from .context import context
 from .version import __version__

pwnlib/asm.py

@@ -1,95 +1,208 @@
-from . import context, log
-import tempfile, subprocess, shutil, tempfile, errno
-from os import path
+# -*- coding: utf-8 -*-
+r"""
+Utilities for assembling and disassembling code.
 
-__all__ = ['asm', 'cpp', 'disasm']
+Architecture Selection
+------------------------
+
+    Architecture, endianness, and word size are selected by using :mod:`pwnlib.context`.
+
+    Any parameters which can be specified to ``context`` can also be specified as
+    keyword arguments to either :func:`asm` or :func:`disasm`.
+
+Assembly
+------------------------
+
+    To assemble code, simply invoke :func:`asm` on the code to assemble.
+
+        >>> asm('mov eax, 0')
+        '\xb8\x00\x00\x00\x00'
+
+    Additionally, you can use constants as defined in the :mod:`pwnlib.constants`
+    module.
+
+        >>> asm('mov eax, SYS_execve')
+        '\xb8\x0b\x00\x00\x00'
+
+    Finally, :func:`asm` is used to assemble shellcode provided by ``pwntools``
+    in the :mod:`shellcraft` module.
+
+        >>> asm(shellcraft.sh())
+        'jhh///sh/bin\x89\xe31\xc9j\x0bX\x99\xcd\x80'
+
+Disassembly
+------------------------
+
+    To disassemble code, simply invoke :func:`disasm` on the bytes to disassemble.
+
+    >>> disasm('\xb8\x0b\x00\x00\x00')
+    '   0:   b8 0b 00 00 00          mov    eax,0xb'
+
+"""
+import tempfile, subprocess, shutil, tempfile, errno, logging, platform
+from os import path, environ
+from glob import glob
+from . import log
+from .context import context
+
+log = logging.getLogger(__name__)
+
+__all__ = ['asm', 'cpp', 'disasm', 'which_binutils']
 
 _basedir = path.split(__file__)[0]
-_bindir  = path.join(_basedir, 'data', 'binutils')
 _incdir  = path.join(_basedir, 'data', 'includes')
 
-def _assembler(arch):
+def which_binutils(util, **kwargs):
+    """
+    Finds a binutils in the PATH somewhere.
+    Expects that the utility is prefixed with the architecture name.
+
+    Examples:
+
+        >>> import platform
+        >>> which_binutils('as', arch=platform.machine())
+        '.../bin/as'
+        >>> which_binutils('as', arch='arm') #doctest: +ELLIPSIS
+        '.../bin/arm-...-as'
+        >>> which_binutils('as', arch='powerpc') #doctest: +ELLIPSIS
+        '.../bin/powerpc...-as'
+        >>> which_binutils('as', arch='msp430') #doctest: +SKIP
+        ...
+        Traceback (most recent call last):
+        ...
+        Exception: Could not find 'as' installed for ContextType(arch = 'msp430')
+    """
+    with context.local(**kwargs):
+        arch = context.arch
+        bits = context.bits
+
+        # Fix up pwntools vs Debian triplet naming, and account
+        # for 'thumb' being its own pwntools architecture.
+        arches = [arch] + {
+            'thumb':  ['arm',    'aarch64'],
+            'i386':   ['x86_64', 'amd64'],
+            'amd64':  ['x86_64', 'i386'],
+        }.get(arch, [])
+
+        # If one of the candidate architectures matches the native
+        # architecture, use that as a last resort.
+        machine = platform.machine()
+        try:
+            with context.local(arch = machine):
+                if context.arch in arches:
+                    arches.append(None)
+        except AttributeError:
+            log.warn_once("Your local binutils won't be used because architecture %r is not supported." % machine)
+
+        for arch in arches:
+            # hack for homebrew-installed binutils on mac
+            for gutil in ['g'+util, util]:
+                # e.g. objdump
+                if arch is None: pattern = gutil
+
+                # e.g. aarch64-linux-gnu-objdump
+                else:       pattern = '%s*linux*-%s' % (arch,gutil)
+
+                for dir in environ['PATH'].split(':'):
+                    res = glob(path.join(dir, pattern))
+                    if res:
+                        return res[0]
+
+        locals()['context'] = context
+        log.warning("""
+Could not find %(util)r installed for %(context)s
+Try installing binutils for this architecture:
+    https://pwntools.readthedocs.org/en/latest/install/binutils.html
+""".strip() % locals())
+        raise Exception('Could not find %(util)r installed for %(context)s' % locals())
+
+def _assembler():
+    gas = which_binutils('as')
+
+    E = {
+        'big':    '-EB',
+        'little': '-EL'
+    }[context.endianness]
+
     assemblers = {
-        'i386'   : ['nasm', '-Ox'],
-        'amd64'  : ['nasm', '-Ox'],
-        'thumb'  : [path.join(_bindir, 'arm-as')],
-        'mips'   : [path.join(_bindir, 'mips-as' ), '--EB'],
-        'mipsel' : [path.join(_bindir, 'mips-as' ), '--EL'],
+        'i386'   : [gas, '--32'],
+        'amd64'  : [gas, '--64'],
+
+        # Most architectures accept -EL or -EB
+        'thumb'  : [gas, '-mthumb', E],
+        'arm'    : [gas, E],
+        'aarch64': [gas, E],
+        'mips'   : [gas, E],
+        'mips64' : [gas, E],
+        'sparc':   [gas, E],
+        'sparc64': [gas, E],
+
+        # Powerpc wants -mbig or -mlittle
+        'powerpc':   [gas, '-m%s' % context.endianness],
+        'powerpc64': [gas, '-m%s' % context.endianness],
+
+        # ia64 only accepts -mbe or -mle
+        'ia64':    [gas, '-m%ce' % context.endianness[0]]
     }
 
-    if arch in assemblers:
-        return assemblers[arch]
-    else:
-        return [path.join(_bindir, arch + '-as')]
+    return assemblers.get(context.arch, [gas])
 
+def _objcopy():
+    return [which_binutils('objcopy')]
 
-def _objcopy(arch):
-    if arch in ['i386', 'amd64']:
-        return 'objcopy'
-    else:
-        return path.join(_bindir, 'objcopy')
+def _objdump():
+    path = [which_binutils('objdump')]
 
+    if context.arch in ('i386', 'amd64'):
+        path += ['-Mintel']
 
-def _objdump(arch):
-    if arch in ['i386', 'amd64']:
-        return ['objdump', '-Mintel']
-    else:
-        return [path.join(_bindir, 'objdump')]
+    return path
 
+def _include_header():
+    os   = context.os
+    arch = context.arch
 
-def _include_header(arch, os):
     if os == 'freebsd':
-        return '#include <freebsd.h>\n'
-    elif os == 'linux' and arch:
-        return '#include <linux/%s.h>\n' % arch
-    else:
-        return ''
+        include = 'freebsd.h'
+    elif os == 'linux':
+        include = 'linux/%s.h' % arch
+
+    if not include or not path.exists(path.join(_incdir, include)):
+        log.warn_once("Could not find system include headers for %s-%s" % (arch,os))
+        return '\n'
+
+    return '#include <%s>\n' % include
 
 
-def _arch_header(arch):
+def _arch_header():
+    prefix  = ['.section .shellcode,"ax"']
     headers = {
-        'i386'  : ['bits 32'],
-        'amd64' : ['bits 64'],
-        'arm'   : [
-            '.section .shellcode,"ax"',
-            '.syntax unified',
-            '.arch armv7-a',
-            '.arm'
-        ],
-        'thumb' : [
-            '.section .shellcode,"ax"',
-            '.syntax unified',
-            '.arch armv7-a',
-            '.thumb'
-        ],
-        'mips'  : [
-            '.section .shellcode,"ax"',
-            '.set mips2',
-            '.set noreorder',
-        ],
-        'mipsel'  : [
-            '.section .shellcode,"ax"',
-            '.set mips2',
-            '.set noreorder',
-        ],
+        'i386'  :  ['.intel_syntax noprefix'],
+        'amd64' :  ['.intel_syntax noprefix'],
+        'arm'   : ['.syntax unified',
+                   '.arch armv7-a',
+                   '.arm'],
+        'thumb' : ['.syntax unified',
+                   '.arch armv7-a',
+                   '.thumb'],
+        'mips'  : ['.set mips2',
+                   '.set noreorder'],
     }
 
-    if arch in headers:
-        return '\n'.join(headers[arch]) + '\n'
-    else:
-        return ''
+    return '\n'.join(prefix + headers.get(context.arch, [])) + '\n'
+
+def _bfdname():
+    arch = context.arch
 
-def _bfdname(arch):
     bfdnames = {
         'i386'    : 'elf32-i386',
         'amd64'   : 'elf64-x86-64',
         'arm'     : 'elf32-littlearm',
         'thumb'   : 'elf32-littlearm',
-        'mips'    : 'elf32-bigmips',
-        'mipsel'  : 'elf32-littlemips',
+        'mips'    : 'elf32-%smips' % context.endianness,
         'alpha'   : 'elf64-alpha',
         'cris'    : 'elf32-cris',
-        'ia64'    : 'elf64-ia64-little',
+        'ia64'    : 'elf64-ia64-%s' % context.endianness,
         'm68k'    : 'elf32-m68k',
         'powerpc' : 'elf32-powerpc',
         'vax'     : 'elf32-vax',
@@ -98,21 +211,25 @@ def _bfdname(arch):
     if arch in bfdnames:
         return bfdnames[arch]
     else:
-        log.error("Cannot find bfd name for architecture %r" % arch)
+        raise Exception("Cannot find bfd name for architecture %r" % arch)
 
 
-def _bfdarch(arch):
-    if arch == 'amd64':
-        return 'i386:x86-64'
-    elif arch == 'thumb':
-        return 'arm'
-    elif arch == 'ia64':
-        return 'ia64-elf64'
-    else:
-        return arch
+def _bfdarch():
+    arch = context.arch
+    convert = {
+    'i386': 'i386',
+    'amd64': 'i386:x86-64',
+    'thumb': 'arm',
+    'ia64': 'ia64-elf64'
+    }
 
+    if arch in convert:
+        return convert[arch]
+
+    return arch
 
 def _run(cmd, stdin = None):
+    log.debug(subprocess.list2cmdline(cmd))
     try:
         proc = subprocess.Popen(
             cmd,
@@ -124,60 +241,67 @@ def _run(cmd, stdin = None):
         exitcode = proc.wait()
     except OSError as e:
         if e.errno == errno.ENOENT:
-            log.error('Could not run %r the program' % cmd[0])
+            log.exception('Could not run %r the program' % cmd[0])
         else:
             raise
 
     if (exitcode, stderr) != (0, ''):
-        msg = 'There was an error running %s:\n' % repr(cmd[0])
+        msg = 'There was an error running %s:\n' % repr(cmd)
         if exitcode != 0:
             msg += 'It had the exitcode %d.\n' % exitcode
         if stderr != '':
             msg += 'It had this on stdout:\n%s\n' % stderr
         log.error(msg)
 
-
     return stdout
 
-def cpp(shellcode, arch = None, os = None):
-    """cpp(shellcode, arch = None, os = None) -> str
+def cpp(shellcode, **kwargs):
+    r"""cpp(shellcode, ...) -> str
 
     Runs CPP over the given shellcode.
 
     The output will always contain exactly one newline at the end.
 
+    Arguments:
+        shellcode(str): Shellcode to preprocess
+
+    Kwargs:
+        Any arguments/properties that can be set on ``context``
+
     Examples:
-      >>> cpp("testing: SYS_setresuid")
-      'testing: SYS_setresuid\\n'
-      >>> cpp("mov al, SYS_setresuid", arch = "i386", os = "linux")
-      'mov al, 164\\n'
-      >>> cpp("weee SYS_setresuid", arch = "arm", os = "linux")
-      'weee (0x900000+164)\\n'
-      >>> cpp("SYS_setresuid", arch = "thumb", os = "linux")
-      '(0+164)\\n'
-      >>> cpp("SYS_setresuid", os = "freebsd")
-      '311\\n'
+
+        .. doctest::
+
+            >>> cpp("mov al, SYS_setresuid", arch = "i386", os = "linux")
+            'mov al, 164\n'
+            >>> cpp("weee SYS_setresuid", arch = "arm", os = "linux")
+            'weee (0x900000+164)\n'
+            >>> cpp("SYS_setresuid", arch = "thumb", os = "linux")
+            '(0+164)\n'
+            >>> cpp("SYS_setresuid", os = "freebsd")
+            '311\n'
     """
 
-    arch = arch or context.arch
-    os   = os   or context.os
-    code = _include_header(arch, os) + shellcode
-    cmd  = [
-        'cpp',
-        '-C',
-        '-nostdinc',
-        '-undef',
-        '-P',
-        '-I' + _incdir,
-        '/dev/stdin'
-    ]
-    return _run(cmd, code).strip('\n').rstrip() + '\n'
+    with context.local(**kwargs):
+        arch = context.arch
+        os   = context.os
+        code = _include_header() + shellcode
+        cmd  = [
+            'cpp',
+            '-C',
+            '-nostdinc',
+            '-undef',
+            '-P',
+            '-I' + _incdir,
+            '/dev/stdin'
+        ]
+        return _run(cmd, code).strip('\n').rstrip() + '\n'
 
 
-def asm(shellcode, arch = None, os = None):
-    """asm(code, arch = None, os = None) -> str
+def asm(shellcode, vma = 0, **kwargs):
+    r"""asm(code, vma = 0, ...) -> str
 
-    Runs CPP over a given shellcode and then assembles it into bytes.
+    Runs :func:`cpp` over a given shellcode and then assembles it into bytes.
 
     To see which architectures or operating systems are supported,
     look in :mod:`pwnlib.contex`.
@@ -187,63 +311,67 @@ def asm(shellcode, arch = None, os = None):
 
     Args:
       shellcode(str): Assembler code to assemble.
-      arch: A supported architecture or None. In case of None,
-            :data:`pwnlib.context.arch` will be used.
-      os: A supported operating system or None. In case of None,
-          :data:`pwnlib.context.os` will be used.
+      vma(int):       Virtual memory address of the beginning of assembly
+
+    Kwargs:
+        Any arguments/properties that can be set on ``context``
 
     Examples:
-      >>> asm("mov eax, SYS_select", arch = 'i386', os = 'freebsd')
-      '\\xb8]\\x00\\x00\\x00'
-      >>> asm("mov rax, SYS_select", arch = 'amd64', os = 'linux')
-      '\\xb8\\x17\\x00\\x00\\x00'
-      >>> asm("ldr r0, =SYS_select", arch = 'arm', os = 'linux')
-      '\\x04\\x00\\x1f\\xe5R\\x00\\x90\\x00'
+
+        .. doctest::
+
+            >>> asm("mov eax, SYS_select", arch = 'i386', os = 'freebsd')
+            '\xb8]\x00\x00\x00'
+            >>> asm("mov eax, SYS_select", arch = 'amd64', os = 'linux')
+            '\xb8\x17\x00\x00\x00'
+            >>> asm("mov rax, SYS_select", arch = 'amd64', os = 'linux')
+            'H\xc7\xc0\x17\x00\x00\x00'
+            >>> asm("ldr r0, =SYS_select", arch = 'arm', os = 'linux', bits=32)
+            '\x04\x00\x1f\xe5R\x00\x90\x00'
     """
+    result = ''
 
-    arch = arch or context.arch
-    os   = os   or context.os
-    if not arch:
-        raise ValueError(
-            "asm() needs to get 'arch' through an argument or the context"
-        )
+    with context.local(**kwargs):
+        assembler = _assembler()
+        objcopy   = _objcopy() + ['-j', '.shellcode', '-Obinary']
+        code      = '.org %#x\n' % vma
+        code      += _arch_header()
+        code      += cpp(shellcode)
 
-    tmpdir  = tempfile.mkdtemp(prefix = 'pwn-asm-')
-    def tmp(name):
-        return path.join(tmpdir, name)
+        log.debug('Assembling\n%s' % code)
 
-    assembler = _assembler(arch)
-    objcopy   = [_objcopy(arch), '-j.shellcode', '-Obinary']
-    code      = _arch_header(arch) + cpp(shellcode, arch, os)
+        tmpdir    = tempfile.mkdtemp(prefix = 'pwn-asm-')
+        step1     = path.join(tmpdir, 'step1')
+        step2     = path.join(tmpdir, 'step2')
+        step3     = path.join(tmpdir, 'step3')
 
-    try:
-        with open(tmp('step1'), 'w') as fd:
-            fd.write(code)
+        try:
+            with open(step1, 'w') as fd:
+                fd.write(code)
 
-        _run(assembler + ['-o', tmp('step2'), tmp('step1')])
+            _run(assembler + ['-o', step2, step1])
 
-        if arch in ['i386', 'amd64']:
-            with open(tmp('step2')) as fd:
-                return fd.read()
+            if file(step2,'rb').read(4) == '\x7fELF':
+                # Sanity check for seeing if the output has relocations
+                relocs = subprocess.check_output(
+                    [which_binutils('readelf'), '-r', step2]
+                ).strip()
+                if len(relocs.split('\n')) > 1:
+                    log.error('Shellcode contains relocations:\n%s' % relocs)
 
-        # Sanity check for seeing if the output has relocations
-        relocs = subprocess.check_output(
-            ['readelf', '-r', tmp('step2')]
-        ).strip()
-        if len(relocs.split('\n')) > 1:
-            raise Exception(
-                'There were relocations in the shellcode:\n\n%s' % relocs
-            )
+            _run(objcopy + [step2, step3])
 
-        _run(objcopy + [tmp('step2'), tmp('step3')])
+            with open(step3) as fd:
+                result = fd.read()
 
-        with open(tmp('step3')) as fd:
-            return fd.read()
-    finally:
-        shutil.rmtree(tmpdir)
+        except:
+            log.exception("An error occurred while assembling:\n%s" % code)
+        else:
+            shutil.rmtree(tmpdir)
+            return result
 
-def disasm(data, arch = None):
-    """disasm(data, arch = None) -> str
+def disasm(data, vma = 0, **kwargs):
+    """disasm(data, ...) -> str
 
     Disassembles a bytestring into human readable assembler.
 
@@ -255,60 +383,70 @@ def disasm(data, arch = None):
 
     Args:
       data(str): Bytestring to disassemble.
-      arch: A supported architecture or None. In case of None,
-            :data:`pwnlib.context.arch` will be used.
+      vma(int): Passed through to the --adjust-vma argument of objdump
+
+    Kwargs:
+      Any arguments/properties that can be set on ``context``
 
     Examples:
-      >>> print disasm('b85d000000'.decode('hex'), arch = 'i386')
-         0:   b8 5d 00 00 00          mov    eax,0x5d
-      >>> print disasm('b817000000'.decode('hex'), arch = 'amd64')
-         0:   b8 17 00 00 00          mov    eax,0x17
-      >>> print disasm('04001fe552009000'.decode('hex'), arch = 'arm')
-         0:   e51f0004        ldr     r0, [pc, #-4]   ; 0x4
-         4:   00900052        addseq  r0, r0, r2, asr r0
-      >>> print disasm('4ff00500'.decode('hex'), arch = 'thumb')
-         0:   f04f 0005       mov.w   r0, #5
+
+        .. doctest::
+
+          >>> print disasm('b85d000000'.decode('hex'), arch = 'i386')
+             0:   b8 5d 00 00 00          mov    eax,0x5d
+          >>> print disasm('b817000000'.decode('hex'), arch = 'amd64')
+             0:   b8 17 00 00 00          mov    eax,0x17
+          >>> print disasm('48c7c017000000'.decode('hex'), arch = 'amd64')
+             0:   48 c7 c0 17 00 00 00    mov    rax,0x17
+          >>> print disasm('04001fe552009000'.decode('hex'), arch = 'arm')
+             0:   e51f0004        ldr     r0, [pc, #-4]   ; 0x4
+             4:   00900052        addseq  r0, r0, r2, asr r0
+          >>> print disasm('4ff00500'.decode('hex'), arch = 'thumb', bits=32)
+             0:   f04f 0005       mov.w   r0, #5
     """
+    result = ''
 
-    arch = arch or context.arch
-    if not arch:
-        raise ValueError(
-            "asm() needs to get 'arch' through an argument or the context"
-        )
+    with context.local(**kwargs):
+        arch   = context.arch
+        os     = context.os
 
-    tmpdir = tempfile.mkdtemp(prefix = 'pwn-disasm-')
-    def tmp(name):
-        return path.join(tmpdir, name)
+        tmpdir = tempfile.mkdtemp(prefix = 'pwn-disasm-')
+        step1  = path.join(tmpdir, 'step1')
+        step2  = path.join(tmpdir, 'step2')
 
-    bfdarch = _bfdarch(arch)
-    bfdname = _bfdname(arch)
-    objdump = _objdump(arch) + ['-d']
-    objcopy = [
-        _objcopy(arch),
-        '-I', 'binary',
-        '-O', bfdname,
-        '-B', bfdarch,
-        '--set-section-flags', '.data=code',
-        '--rename-section', '.data=.text',
-    ]
+        bfdarch = _bfdarch()
+        bfdname = _bfdname()
+        objdump = _objdump() + ['-d', '--adjust-vma', str(vma), '-b', bfdname]
+        objcopy = _objcopy() + [
+            '-I', 'binary',
+            '-O', bfdname,
+            '-B', bfdarch,
+            '--set-section-flags', '.data=code',
+            '--rename-section', '.data=.text',
+        ]
 
-    if arch == 'thumb':
-        objcopy += ['--prefix-symbol=$t.']
-    else:
-        objcopy += ['-w', '-N', '*']
-
-    try:
-        with open(tmp('step1'), 'w') as fd:
-            fd.write(data)
-        _run(objcopy + [tmp('step1'), tmp('step2')])
-
-        output0 = subprocess.check_output(objdump + [tmp('step2')])
-        output1 = output0.split('<.text>:\n')
-        if len(output1) != 2:
-            raise IOError(
-                'Something went wrong with objdump:\n\n%s' % output0
-            )
+        if arch == 'thumb':
+            objcopy += ['--prefix-symbol=$t.']
         else:
-            return output1[1].strip('\n').rstrip().expandtabs()
-    finally:
-        shutil.rmtree(tmpdir)
+            objcopy += ['-w', '-N', '*']
+
+        try:
+
+            with open(step1, 'w') as fd:
+                fd.write(data)
+
+            res = _run(objcopy + [step1, step2])
+
+            output0 = subprocess.check_output(objdump + [step2])
+            output1 = output0.split('<.text>:\n')
+
+            if len(output1) != 2:
+                log.error('Could not find .text in objdump output:\n%s' % output0)
+
+            result = output1[1].strip('\n').rstrip().expandtabs()
+        except:
+            log.exception("An error occurred while disassembling:\n%s" % data)
+        else:
+            shutil.rmtree(tmpdir)
+            return result
+

pwnlib/atexception.py

@@ -0,0 +1,101 @@
+"""
+Analogous to atexit, this module allows the programmer to register functions to
+be run if an unhandled exception occurs.
+"""
+
+import sys, traceback, threading
+from .context import context
+
+__all__ = ['register', 'unregister']
+
+_lock = threading.Lock()
+_ident = 0
+_handlers = {}
+
+def register(func, *args, **kwargs):
+    """register(func, *args, **kwargs)
+
+    Registers a function to be called when an unhandled exception occurs.  The
+    function will be called with positional arguments `args` and keyword
+    arguments `kwargs`, i.e. ``func(*args, **kwargs)``.  The current `context`
+    is recorded and will be the one used when the handler is run.
+
+    E.g. to suppress logging output from an exception-handler one could write::
+
+      with context.local(log_level = 'error'):
+        atexception.register(handler)
+
+    An identifier is returned which can be used to unregister the
+    exception-handler.
+
+    This function can be used as a decorator::
+
+      @atexception.register
+      def handler():
+        ...
+
+    Notice however that this will bind ``handler`` to the identifier and not the
+    actual exception-handler.  The exception-handler can then be unregistered
+    with::
+
+      atexception.unregister(handler)
+
+    This function is thread safe.
+
+    """
+    global _ident
+    with _lock:
+        ident = _ident
+        _ident += 1
+    _handlers[ident] = (func, args, kwargs, vars(context))
+    return ident
+
+def unregister(func):
+    """unregister(func)
+
+    Remove `func` from the collection of registered functions.  If `func` isn't
+    registered this is a no-op.
+    """
+    if func in _handlers:
+        del _handlers[func]
+
+def _run_handlers():
+    """_run_handlers()
+
+    Run registered handlers.  They run in the reverse order of which they were
+    registered.
+
+    If a handler raises an exception, it will be printed but nothing else
+    happens, i.e. other handlers will be run.
+    """
+    context.clear()
+    for _ident, (func, args, kwargs, ctx) in \
+        sorted(_handlers.items(), reverse = True):
+        try:
+            with context.local(**ctx):
+                func(*args, **kwargs)
+        except SystemExit:
+            pass
+        except:
+            # extract the current exception and rewind the traceback to where it
+            # originated
+            typ, val, tb = sys.exc_info()
+            traceback.print_exception(typ, val, tb.tb_next)
+
+# we rely on the existing excepthook to print exceptions
+if hasattr(sys, 'excepthook'):
+    _oldhook = sys.excepthook
+else:
+    _oldhook = None
+
+def _newhook(typ, val, tb):
+    """_newhook(typ, val, tb)
+
+    Our excepthook replacement.  First the original hook is called to print the
+    exception, then each handler is called.
+    """
+    if _oldhook:
+        _oldhook(typ, val, tb)
+    _run_handlers()
+
+sys.excepthook = _newhook

pwnlib/atexit.py

@@ -0,0 +1,93 @@
+"""
+Replacement for the Python standard library's atexit.py.
+
+Whereas the standard :mod:`atexit` module only defines :func:`atexit.register`,
+this replacement module also defines :func:`unregister`.
+
+This module also fixes a the issue that exceptions raised by an exit handler is
+printed twice when the standard :mod:`atexit` is used.
+"""
+
+import sys, traceback, threading
+from .context import context
+
+__all__ = ['register', 'unregister']
+
+_lock = threading.Lock()
+_ident = 0
+_handlers = {}
+
+def register(func, *args, **kwargs):
+    """register(func, *args, **kwargs)
+
+    Registers a function to be called on program termination.  The function will
+    be called with positional arguments `args` and keyword arguments `kwargs`,
+    i.e. ``func(*args, **kwargs)``.  The current `context` is recorded and will
+    be the one used when the handler is run.
+
+    E.g. to suppress logging output from an exit-handler one could write::
+
+      with context.local(log_level = 'error'):
+        atexit.register(handler)
+
+    An identifier is returned which can be used to unregister the exit-handler.
+
+    This function can be used as a decorator::
+
+      @atexit.register
+      def handler():
+        ...
+
+    Notice however that this will bind ``handler`` to the identifier and not the
+    actual exit-handler.  The exit-handler can then be unregistered with::
+
+      atexit.unregister(handler)
+
+    This function is thread safe.
+
+    """
+    global _ident
+    with _lock:
+        ident = _ident
+        _ident += 1
+    _handlers[ident] = (func, args, kwargs, vars(context))
+    return ident
+
+def unregister(ident):
+    """unregister(ident)
+
+    Remove the exit-handler identified by `ident` from the list of registered
+    handlers.  If `ident` isn't registered this is a no-op.
+    """
+    if ident in _handlers:
+        del _handlers[ident]
+
+def _run_handlers():
+    """_run_handlers()
+
+    Run registered exit-handlers.  They run in the reverse order of which they
+    were registered.
+
+    If a handler raises an exception, it will be printed but nothing else
+    happens, i.e. other handlers will be run and `sys.excepthook` will not be
+    called for that reason.
+    """
+    context.clear()
+    for _ident, (func, args, kwargs, ctx) in \
+        sorted(_handlers.items(), reverse = True):
+        try:
+            with context.local(**ctx):
+                func(*args, **kwargs)
+        except SystemExit:
+            pass
+        except:
+            # extract the current exception and rewind the traceback to where it
+            # originated
+            typ, val, tb = sys.exc_info()
+            traceback.print_exception(typ, val, tb.tb_next)
+
+# if there's already an exitfunc registered be sure to run that too
+if hasattr(sys, "exitfunc"):
+    register(sys.exitfunc)
+
+sys.exitfunc = _run_handlers

pwnlib/commandline/asm.py

@@ -2,7 +2,7 @@
 
 import argparse, sys
 from pwnlib.asm           import asm
-from pwnlib.context       import __possible__ as contexts
+from pwnlib.context       import context
 from pwnlib.util.fiddling import enhex
 
 parser = argparse.ArgumentParser(
@@ -33,27 +33,30 @@ parser.add_argument(
 parser.add_argument(
     '-c', '--context',
     metavar = '<opt>',
-    choices = contexts['os'] + contexts['arch'],
+    choices = context.oses + list(context.architectures),
     default = ['i386','linux'],
     action = 'append',
     help = 'The os/architecture the shellcode will run in (default: linux/i386), choose from: %s' % \
-      ', '.join(contexts['os'] + contexts['arch'])
+      ', '.join(sorted(context.oses + list(context.architectures)))
 )
 
-args   = parser.parse_args()
-tty    = args.output.isatty()
+def main():
+    args   = parser.parse_args()
+    tty    = args.output.isatty()
 
-for arch in args.context[::-1]:
-    if arch in contexts['arch']: break
-for os in args.context[::-1]:
-    if os in contexts['os']: break
+    for arch in args.context[::-1]:
+        if arch in context.architectures: break
+    for os in args.context[::-1]:
+        if os in context.oses: break
 
-data   = '\n'.join(args.lines) or sys.stdin.read()
-output = asm(data.replace(';', '\n'), arch = arch, os = os)
-fmt    = args.format or ('hex' if tty else 'raw')
-formatters = {'r':str, 'h':enhex, 's':repr}
+    data   = '\n'.join(args.lines) or sys.stdin.read()
+    output = asm(data.replace(';', '\n'), arch = arch, os = os)
+    fmt    = args.format or ('hex' if tty else 'raw')
+    formatters = {'r':str, 'h':enhex, 's':repr}
 
-args.output.write(formatters[fmt[0]](output))
+    args.output.write(formatters[fmt[0]](output))
 
-if tty and fmt is not 'raw':
-    args.output.write('\n')
+    if tty and fmt is not 'raw':
+        args.output.write('\n')
+
+if __name__ == '__main__': main()

pwnlib/commandline/constgrep.py

@@ -0,0 +1,140 @@
+#!/usr/bin/env python2
+
+import argparse, os, re
+from pwnlib import asm, constants
+from pwnlib.util import safeeval
+# from pwnlib.context import __possible__ as contexts
+from pwnlib.context import context
+
+p = argparse.ArgumentParser(
+    description = "Looking up constants from header files.\n\nExample: constgrep -c freebsd -m  ^PROT_ '3 + 4'",
+    formatter_class = argparse.RawDescriptionHelpFormatter,
+)
+
+group = p.add_mutually_exclusive_group()
+group.add_argument(
+    '-e', '--exact',
+    metavar = '<constant name>',
+    # nargs = 1,
+    default = None,
+    help = 'Do an exact match for a constant instead of searching for a regex',
+)
+group.add_argument(
+    'regex',
+    nargs = '?',
+    default = '',
+    help = 'The regex matching constant you want to find',
+)
+
+p.add_argument(
+    'constant',
+    nargs = '?',
+    default = None,
+    help = 'The constant to find',
+)
+
+p.add_argument(
+    '-i', '--case-insensitive',
+    action = 'store_true',
+    help = 'Search case insensitive',
+)
+
+p.add_argument(
+    '-m', '--mask-mode',
+    action = 'store_true',
+    help = 'Instead of searching for a specific constant value, search for values not containing strictly less bits that the given value.',
+)
+
+p.add_argument(
+    '-c', '--context',
+    metavar = '<opt>',
+    choices = context.oses + list(context.architectures),
+    default = ['i386','linux'],
+    action = 'append',
+    help = 'The os/architecture to find constants for (default: linux/i386), choose from: %s' % \
+    ', '.join(sorted(context.oses + context.architectures.keys()))
+)
+
+def main():
+    args = p.parse_args()
+
+    # Find the architecture and os from the list of contexts
+    for arch in args.context[::-1]:
+        if arch in context.architectures: break
+    for os in args.context[::-1]:
+        if os in context.oses: break
+
+    if args.exact:
+        # This is the simple case
+        print asm.cpp(args.exact, os = os, arch = arch).strip()
+    else:
+        # New we search in the right module.
+        # But first: We find the right module
+        if os == 'freebsd':
+            mod = constants.freebsd
+        else:
+            mod = getattr(getattr(constants, os), arch)
+
+        # Compile the given regex, for optimized lookup
+        if args.case_insensitive:
+            matcher = re.compile(args.regex, re.IGNORECASE)
+        else:
+            matcher = re.compile(args.regex)
+
+        # Evaluate the given constant
+        if args.constant:
+            constant = safeeval.expr(args.constant)
+        else:
+            constant = None
+
+        # The found matching constants and the length of the longest string
+        out    = []
+        maxlen = 0
+
+        for k in dir(mod):
+            # No python stuff
+            if k.endswith('__') and k.startswith('__'):
+                continue
+
+            # Run the regex
+            if not matcher.search(k):
+                continue
+
+            # Check the constant
+            if constant != None:
+                val = getattr(mod, k)
+                if args.mask_mode:
+                    if constant & val != val:
+                        continue
+                else:
+                    if constant != val:
+                        continue
+
+            # Append it
+            out.append((getattr(mod, k), k))
+            maxlen = max(len(k), maxlen)
+
+        # Output all matching constants
+        for _, k in sorted(out):
+            print '#define %s %s' % (k.ljust(maxlen), asm.cpp(k, os = os, arch = arch).strip())
+
+        # If we are in match_mode, then try to find a combination of
+        # constants that yield the exact given value
+        # We do not want to find combinations using the value 0.
+        if not (constant == None or constant == 0) and args.mask_mode:
+            mask = constant
+            good = []
+            out = [(v, k) for v, k in out if v != 0]
+
+            while mask and out:
+                cur = out.pop()
+                mask &= ~cur[0]
+                good.append(cur)
+
+                out = [(v, k) for v, k in out if mask & v == v]
+
+            if reduce(lambda x, cur: x | cur[0], good, 0) == constant:
+                print
+                print '(%s) == %s' % (' | '.join(k for v, k in good), args.constant)
+
+if __name__ == '__main__': main()

pwnlib/commandline/cyclic.py

@@ -0,0 +1,77 @@
+#!/usr/bin/env python2
+import argparse, string, sys
+from pwnlib.util import cyclic, packing
+from pwnlib import log
+
+parser = argparse.ArgumentParser(
+    description = "Cyclic pattern creator/finder"
+)
+
+parser.add_argument(
+    '-a', '--alphabet',
+    metavar = 'alphabet',
+    default = string.ascii_lowercase,
+    help = 'The alphabet to use in the cyclic pattern (defaults to all lower case letters)',
+)
+
+parser.add_argument(
+    '-n', '--length',
+    metavar = 'length',
+    default = 4,
+    type = int,
+    help = 'Size of the unique subsequences (defaults to 4).'
+)
+
+group = parser.add_mutually_exclusive_group(required = True)
+group.add_argument(
+    '-l', '-o', '--offset', '--lookup',
+    dest = 'lookup',
+    metavar = '<lookup value>',
+    help = 'Do a lookup instead printing the alphabet',
+)
+
+group.add_argument(
+    'count',
+    type = int,
+    nargs = '?',
+    help = 'Number of characters to print'
+)
+
+def main():
+    args = parser.parse_args()
+    alphabet = args.alphabet
+    subsize  = args.length
+
+    if args.lookup:
+        pat = args.lookup
+
+        if pat.startswith('0x'):
+            pat = packing.pack(int(pat[2:], 16), subsize*8, 'little', 'unsigned')
+        elif pat.isdigit():
+            pat = packing.pack(int(pat, 10), subsize*8, 'little', 'unsigned')
+
+        if len(pat) != 4:
+            log.fatal('Subpattern must be 4 bytes', 1)
+
+        if not all(c in alphabet for c in pat):
+            log.fatal('Pattern contains characters not present in the alphabet', 1)
+
+        offset = cyclic.cyclic_find(pat, alphabet, subsize)
+
+        if offset == -1:
+            log.fatal('Given pattern does not exist in cyclic pattern', 1)
+        else:
+            print offset
+    else:
+        want   = args.count
+        result = cyclic.cyclic(want, alphabet, subsize)
+        got    = len(result)
+        if got < want:
+            log.failure("Alphabet too small (max length = %i)" % got)
+
+        sys.stdout.write(result)
+
+        if sys.stdout.isatty():
+            sys.stdout.write('\n')
+
+if __name__ == '__main__': main()

pwnlib/commandline/disasm.py

@@ -0,0 +1,42 @@
+#!/usr/bin/env python2
+
+import argparse, sys
+from pwnlib import asm
+from pwnlib.context import context
+from string import whitespace, hexdigits
+
+parser = argparse.ArgumentParser(
+    description = 'Disassemble bytes into text format'
+)
+
+parser.add_argument(
+    'hex',
+    metavar = 'hex',
+    nargs = '*',
+    help = 'Hex-string to disasemble. If none are supplied, then it uses stdin in non-hex mode.'
+)
+
+parser.add_argument(
+    '-c', '--context',
+    metavar = '<opt>',
+    choices = context.architectures,
+    default = 'i386',
+    help = 'The architecture of the shellcode (default: i386), choose from:\n%s' % ', '.join(context.architectures)
+)
+
+def main():
+    args = parser.parse_args()
+
+    if len(args.hex) > 0:
+        dat = ''.join(args.hex)
+        dat = dat.translate(None, whitespace)
+        if not set(hexdigits) >= set(dat):
+            print "This is not a hex string"
+            exit(-1)
+        dat = dat.decode('hex')
+    else:
+        dat = sys.stdin.read()
+
+    print asm.disasm(dat, arch = args.context)
+
+if __name__ == '__main__': main()

pwnlib/commandline/elfdiff.py

@@ -0,0 +1,27 @@
+#!/usr/bin/env python2
+from argparse import ArgumentParser
+from subprocess import check_output, CalledProcessError
+from tempfile import NamedTemporaryFile
+
+def dump(x):
+    n = NamedTemporaryFile(delete=False)
+    o = check_output(['objdump','-d','-x','-s',x])
+    n.write(o)
+    n.flush()
+    return n.name
+
+def diff(a,b):
+    try: return check_output(['diff',a,b])
+    except CalledProcessError as e:
+        return e.output
+
+
+p = ArgumentParser()
+p.add_argument('a')
+p.add_argument('b')
+
+def main():
+    a = p.parse_args()
+    print diff(dump(a.a), dump(a.b))
+
+if __name__ == '__main__': main()

pwnlib/commandline/elfpatch.py

@@ -0,0 +1,25 @@
+#!/usr/bin/env python2
+
+import argparse, sys
+from pwnlib.elf import ELF
+from pwnlib.util.fiddling import unhex
+p = argparse.ArgumentParser()
+p.add_argument('elf',help="File to patch")
+p.add_argument('offset',help="Offset to patch in virtual address (hex encoded)")
+p.add_argument('bytes',help='Bytes to patch (hex encoded)')
+
+
+def main():
+    a = p.parse_args()
+
+    if not a.offset.startswith('0x'):
+        a.offset = '0x' + a.offset
+
+    offset = int(a.offset, 16)
+    bytes  = unhex(a.bytes)
+    elf    = ELF(a.elf)
+
+    elf.write(offset, bytes)
+    sys.stdout.write(elf.get_data())
+
+if __name__ == '__main__': main()

pwnlib/commandline/hex.py

@@ -0,0 +1,17 @@
+#!/usr/bin/env python2
+import sys, argparse
+
+parser = argparse.ArgumentParser(description='''
+Hex-encodes data provided on the command line or via stdin.
+''')
+parser.add_argument('data', nargs='*',
+    help='Data to convert into hex')
+
+def main():
+    args = parser.parse_args()
+    if not args.data:
+        print sys.stdin.read().encode('hex')
+    else:
+        print ' '.join(args.data).encode('hex')
+
+if __name__ == '__main__': main()

pwnlib/commandline/phd.py

@@ -0,0 +1,99 @@
+#!/usr/bin/env python2
+
+import argparse, sys, os
+import pwnlib.term.text as text
+from pwnlib.util.fiddling import hexdump_iter
+
+parser = argparse.ArgumentParser(
+    description = 'Pwnlib HexDump'
+)
+
+parser.add_argument(
+    'file',
+    metavar='file',
+    nargs='?',
+    help='File to hexdump.  Reads from stdin if missing.',
+    type=argparse.FileType('r'),
+    default=sys.stdin
+)
+
+parser.add_argument(
+    "-w", "--width",
+    help="Number of bytes per line.",
+    default='16',
+)
+
+parser.add_argument(
+    "-l", "--highlight",
+    help="Byte to highlight.",
+    nargs="*",
+)
+
+parser.add_argument(
+    "-s", "--skip",
+    help="Skip this many initial bytes.",
+    default='0',
+)
+
+parser.add_argument(
+    "-c", "--count",
+    help="Only show this many bytes.",
+    default='-1',
+)
+
+parser.add_argument(
+    "-o", "--offset",
+    help="Addresses in left hand column starts at this address.",
+    default='0',
+)
+
+parser.add_argument(
+    "--color",
+    nargs='?',
+    help="Colorize the output.  When 'auto' output is colorized exactly when stdout is a TTY.  Default is 'auto'.",
+    choices = ('always', 'never', 'auto'),
+    default='auto',
+)
+
+def asint(s):
+    if   s.startswith('0x'):
+        return int(s, 16)
+    elif s.startswith('0'):
+        return int(s, 8)
+    else:
+        return int(s, 10)
+
+def main():
+    args = parser.parse_args()
+
+    infile = args.file
+    width  = asint(args.width)
+    skip   = asint(args.skip)
+    count  = asint(args.count)
+    offset = asint(args.offset)
+
+    # if `--color` has no argument it is `None`
+    color = args.color or 'always'
+    text.when = color
+
+    if skip:
+        if infile == sys.stdin:
+            infile.read(skip)
+        else:
+            infile.seek(skip, os.SEEK_CUR)
+
+    data = infile.read(count)
+
+    hl = []
+    if args.highlight:
+        for hs in args.highlight:
+            for h in hs.split(','):
+                hl.append(asint(h))
+
+    try:
+        for line in hexdump_iter(data, width, highlight = hl, begin = offset + skip):
+            print line
+    except (KeyboardInterrupt, IOError):
+        pass
+
+if __name__ == '__main__': main()

pwnlib/commandline/shellcraft.py

@@ -0,0 +1,215 @@
+#!/usr/bin/env python2
+import argparse, sys, os, types
+import pwnlib
+from pwnlib import log, util
+import pwnlib.term.text as text
+from pwnlib.context import context
+
+r = text.red
+g = text.green
+b = text.blue
+
+banner = '\n'.join(['  ' + r('____') + '  ' + g('_') + '          ' + r('_') + ' ' + g('_') + '                 ' + b('__') + ' ' + r('_'),
+                    ' ' + r('/ ___|') + g('| |__') + '   ' + b('___') + r('| |') + ' ' + g('|') + ' ' + b('___') + ' ' + r('_ __') + ' ' + g('__ _') + ' ' + b('/ _|') + ' ' + r('|_'),
+                    ' ' + r('\___ \\') + g('| \'_ \\') + ' ' + b('/ _ \\') + ' ' + r('|') + ' ' + g('|') + b('/ __|') + ' ' + r('\'__/') + ' ' + g('_` |') + ' ' + b('|_') + r('| __|'),
+                    '  ' + r('___) |') + ' ' + g('| | |') + '  ' + b('__/') + ' ' + r('|') + ' ' + g('|') + ' ' + b('(__') + r('| |') + ' ' + g('| (_| |') + '  ' + b('_|') + ' ' + r('|_'),
+                    ' ' + r('|____/') + g('|_| |_|') + b('\\___|') + r('_|') + g('_|') + b('\\___|') + r('_|') + '  ' + g('\\__,_|') + b('_|') + '  ' + r('\\__|'),
+                    '\n'
+                    ])
+
+
+#  ____  _          _ _                 __ _
+# / ___|| |__   ___| | | ___ _ __ __ _ / _| |_
+# \___ \| '_ \ / _ \ | |/ __| '__/ _` | |_| __|
+#  ___) | | | |  __/ | | (__| | | (_| |  _| |_
+# |____/|_| |_|\___|_|_|\___|_|  \__,_|_|  \__|
+
+def _string(s):
+    out = []
+    for c in s:
+        co = ord(c)
+        if co >= 0x20 and co <= 0x7e and c not in '/$\'"`':
+            out.append(c)
+        else:
+            out.append('\\x%02x' % co)
+    return '"' + ''.join(out) + '"\n'
+
+def _carray(s):
+    out = []
+    for c in s:
+        out.append('0x' + util.fiddling.enhex(c))
+    return '{' + ', '.join(out) + '};\n'
+
+def _hex(s):
+    return pwnlib.util.fiddling.enhex(s) + '\n'
+
+p = argparse.ArgumentParser(
+    description = 'Microwave shellcode -- Easy, fast and delicious',
+    formatter_class = argparse.RawDescriptionHelpFormatter,
+)
+
+
+p.add_argument(
+    '-?', '--show',
+    action = 'store_true',
+    help = 'Show shellcode documentation',
+)
+
+p.add_argument(
+    '-o', '--out',
+    metavar = '<file>',
+    type = argparse.FileType('w'),
+    default = sys.stdout,
+    help = 'Output file (default: stdout)',
+)
+
+p.add_argument(
+    '-f', '--format',
+    metavar = '<format>',
+    choices = ['r', 'raw',
+               's', 'str', 'string',
+               'c',
+               'h', 'hex',
+               'a', 'asm', 'assembly',
+               'p',
+               'i', 'hexii',
+               'default'],
+    default = 'default',
+    help = 'Output format (default: hex), choose from {r}aw, {s}tring, {c}-style array, {h}ex string, hex{i}i, {a}ssembly code, {p}reprocssed code',
+)
+
+
+class NoDefaultContextValues(object):
+    def __enter__(self):
+        self.old = context.defaults.copy()
+        context.defaults['os'] = None
+        context.defaults['arch'] = None
+    def __exit__(self, *a):
+        context.defaults.update(self.old)
+
+
+
+def get_tree(path, val, result):
+    with NoDefaultContextValues():
+        if path:
+            path += '.'
+
+        if path.startswith('common.'):
+            return
+
+        mods = []
+
+        for k in sorted(dir(val)):
+            if k and k[0] != '_':
+                cur = getattr(val, k)
+                if isinstance(cur, types.ModuleType):
+                    mods.append((path + k, cur))
+                else:
+                    result.append((path + k, cur))
+
+        for path, val in mods:
+            get_tree(path, val, result)
+        return result
+
+# Enumearte all of the shellcode names
+all_shellcodes = get_tree('', pwnlib.shellcraft, [])
+names = '\n'.join('    ' + sc[0] for sc in all_shellcodes)
+
+p.add_argument(
+    'shellcode',
+    nargs = '?',
+    default = '',
+    metavar = '<shellcode>',
+    help = 'The shellcode you want',
+)
+
+p.epilog = 'Available shellcodes are:\n' + names
+
+
+p.add_argument(
+    'args',
+    nargs = '*',
+    metavar = '<arg>',
+    default = (),
+    help = 'Argument to the chosen shellcode',
+)
+
+def main():
+    # Banner must be added here so that it doesn't appear in the autodoc
+    # generation for command line tools
+    p.description = banner + p.description
+    args = p.parse_args()
+
+    if args.format == 'default':
+        if sys.stdout.isatty():
+            args.format = 'hex'
+        else:
+            args.format = 'raw'
+
+
+    vals = get_tree('', pwnlib.shellcraft, [])
+    if args.shellcode:
+        vals = [(k, val) for k, val in vals if k.startswith(args.shellcode + '.') or k == args.shellcode]
+
+    if len(vals) == 0:
+        log.fatal("Cannot find subtree by the name of %r" % args.shellcode)
+    elif len(vals) > 1:
+        for k, _ in vals:
+            print k
+        exit()
+    else:
+        func = vals[0][1]
+
+    if args.show:
+        print func.__doc__
+        exit()
+
+    defargs = len(func.func_defaults or ())
+    reqargs = func.func_code.co_argcount - defargs
+    if len(args.args) < reqargs:
+        if defargs > 0:
+            log.fatal('%s takes at least %d arguments' % (args.shellcode, reqargs))
+        else:
+            log.fatal('%s takes exactly %d arguments' % (args.shellcode, reqargs))
+
+    # Captain uglyness saves the day!
+    for i, val in enumerate(args.args):
+        try:
+            args.args[i] = util.safeeval.expr(val)
+        except ValueError:
+            pass
+
+    # And he strikes again!
+    os = arch = None
+    for k in args.shellcode.split('.')[:-1]:
+        if k in context.architectures:
+            arch = k
+        elif k in context.oses:
+            os = k
+
+    code = func(*args.args)
+
+    if args.format in ['a', 'asm', 'assembly']:
+        print code
+        exit()
+    if args.format == 'p':
+        print pwnlib.asm.cpp(code, arch = arch, os = os)
+        exit()
+
+    code = pwnlib.asm.asm(code, arch = arch, os = os)
+
+    if args.format in ['s', 'str', 'string']:
+        code = _string(code)
+    elif args.format == 'c':
+        code = _carray(code)
+    elif args.format in ['h', 'hex']:
+        code = _hex(code)
+    elif args.format in ['i', 'hexii']:
+        code = pwnlib.util.fiddling.hexii(code) + '\n'
+
+    if not sys.stdin.isatty():
+        sys.stdout.write(sys.stdin.read())
+
+    sys.stdout.write(code)
+
+if __name__ == '__main__': main()

pwnlib/commandline/unhex.py

@@ -0,0 +1,22 @@
+#!/usr/bin/env python2
+import sys, argparse
+from string import whitespace
+
+parser = argparse.ArgumentParser(description='''
+Decodes hex-encoded data provided on the command line or via stdin.
+''')
+parser.add_argument('hex', nargs='*',
+    help='Hex bytes to decode')
+
+def main():
+    args = parser.parse_args()
+    try:
+        if not args.hex:
+            s = sys.stdin.read().translate(None, whitespace)
+            sys.stdout.write(s.decode('hex'))
+        else:
+            sys.stdout.write(''.join(sys.argv[1:]).decode('hex'))
+    except TypeError, e:
+        sys.stderr.write(str(e) + '\n')
+
+if __name__ == '__main__': main()

pwnlib/constants/__init__.py

@@ -34,7 +34,7 @@ Example:
 """
 from types import ModuleType
 import importlib, sys
-from .. import context
+from ..context import context
 
 class module(ModuleType):
     def __init__(self, submodules):

pwnlib/constants/linux/__init__.py

@@ -1,6 +1,6 @@
 from types import ModuleType
 import importlib, sys
-from ... import context
+from ...context import context
 
 class module(ModuleType):
     def __init__(self, submodules):

pwnlib/context.py

@@ -1,382 +0,0 @@
-import types, sys, threading
-from . import log_levels
-
-# These attributes are set on the defaults module after it have been constructed
-# If you change any of these values, remember to update the docstring.
-defaults = {
-    'endianness': 'little',
-    'sign': 'unsigned',
-    'word_size': 32,
-    'log_level': 'error',
-    '__doc__': '''The global default-version of :mod:`pwnlib.context`.
-
-For at description see :mod:`pwnlib.context`. This is the global defaults, that
-act as a "base" for the thread-local values.
-'''}
-
-# These are the possiblities for arch and os
-__possible__ = {
-    'arch': (
-        'alpha', 'amd64', 'arm', 'armeb',
-        'cris', 'i386', 'm68k', 'mips',
-        'mipsel', 'powerpc', 'thumb'
-    ),
-    'arch32': ('arm', 'armeb', 'cris', 'i386', 'm68k', 'mips', 'mipsel', 'thumb'),
-    'arch64': ('alpha', 'amd64'),
-    'os': ('linux', 'freebsd')
-}
-
-class Local(object):
-    def __init__(self, args):
-        self.args = args
-
-    def __enter__(self):
-        self.saved = context._thread_ctx().__dict__.copy()
-        for k, v in self.args.items():
-            setattr(context, k, v)
-
-    def __exit__(self, *args):
-        context._thread_ctx().__dict__.clear()
-        context._thread_ctx().__dict__.update(self.saved)
-
-def _updater(updater, name = None, doc = None):
-    name = name or updater.__name__
-    doc  = doc  or updater.__doc__
-
-    def getter(self):
-        if hasattr(self, '_' + name):
-            return getattr(self, '_' + name)
-        elif self.defaults:
-            return getattr(self.defaults, name)
-        else:
-            return None
-
-    def setter(self, val):
-        setattr(self, '_' + name, updater(self, val))
-
-    # Setting _inner is a slight hack only used to get better documentation
-    res = property(getter, setter, doc = doc)
-    res.fget._inner = updater
-    return res
-
-def _validator(validator, name = None, doc = None):
-    name = name or validator.__name__
-    doc  = doc  or validator.__doc__
-
-    def updater(self, val):
-        if val == None or validator(self, val):
-            return val
-        else:
-            raise AttributeError(
-                'Cannot set context-key %s to %s, did not validate' % \
-                  (name, val)
-            )
-
-    # Setting _inner is a slight hack only used to get better documentation
-    res = _updater(updater, name, doc)
-    res.fget._inner = validator
-    return res
-
-def properties():
-    keys = [k for k in dir(ContextModule) if k[0] != '_']
-    return {k: getattr(ContextModule, k) for k in keys}
-
-class ContextModule(types.ModuleType):
-    def __init__(self, defaults = None):
-        super(ContextModule, self).__init__(__name__)
-        self.defaults     = defaults
-        self.__possible__ = __possible__
-        self.__dict__.update({
-            '__all__'     : [],
-            '__file__'    : __file__,
-            '__package__' : __package__,
-        })
-
-    def __call__(self, **kwargs):
-        """This function is the global equivalent of
-        :func:`pwnlib.context.__call__`.
-
-        Args:
-          kwargs: Variables to be assigned in the environment."""
-
-        for k, v in kwargs.items():
-            setattr(self, k, v)
-
-    @_validator
-    def arch(self, value):
-        """Variable for the current architecture. This is useful e.g. to make
-        :mod:`pwnlib.shellcraft` easier to use. Allowed values:
-
-        * ``alpha``
-        * ``amd64``
-        * ``arm``
-        * ``armeb``
-        * ``cris``
-        * ``i386``
-        * ``m68k``
-        * ``mips``
-        * ``mipsel``
-        * ``powerpc``
-        * ``thumb``
-
-        Setting this will also update :data:`pwnlib.context.word_size`.
-        """
-
-        if value in self.__possible__['arch']:
-            if value in self.__possible__['arch32']:
-                self.word_size = 32
-            elif value in self.__possible__['arch64']:
-                self.word_size = 64
-            return value
-
-    @_validator
-    def os(self, value):
-        """Variable for the current operating system. This is useful e.g. for
-        choosing the right constants for syscall numbers.
-
-        Allowed values:
-
-        * ``linux``
-        * ``freebsd``"""
-
-        if value in self.__possible__['os']:
-            return value
-
-    @_validator
-    def endianness(self, value):
-        """The default endianness used for e.g. the
-        :func:`pwnlib.util.packing.pack` function. Defaults to ``little``.
-
-        Allowed values:
-
-        * ``little``
-        * ``big``"""
-
-        return value in ('big', 'little')
-
-    @_validator
-    def sign(self, value):
-        """The default signedness used for e.g. the
-        :func:`pwnlib.util.packing.pack` function. Defaults to ``unsigned``.
-
-        Allowed values:
-
-        * ``unsigned``
-        * ``signed``"""
-
-        return value in ('unsigned', 'signed')
-
-    @_validator
-    def timeout(self, value):
-        """The default timeout used by e.g. :class:`pwnlib.tubes.ssh`.
-
-        Defaults to None, meaning no timeout.
-
-        Allowed values are any strictly positive number or None."""
-
-        return type(value) in [types.IntType,
-                               types.LongType,
-                               types.FloatType] and value >= 0
-
-    @_validator
-    def word_size(self, value):
-        """The default word size used for e.g. the
-        :func:`pwnlib.util.packing.pack` function. Defaults to ``32``.
-
-        Allowed values are any strictly positive number."""
-
-        return type(value) in [types.IntType, types.LongType] and value > 0
-
-    @_updater
-    def log_level(self, value):
-        """The amount of output desired from the :mod:`pwnlib.log` module.
-
-        Allowed values are any numbers or a string.
-
-        If a string is given, we uppercase the string and lookup it
-        up in the log module.
-
-        E.g if ``'debug'`` is specified, then the result is ``10``, as
-        :data:`pwnlib.log_levels.DEBUG` is ``10``.
-"""
-
-        if type(value) in [types.IntType, types.LongType, types.NoneType]:
-            return value
-        elif type(value) == types.StringType:
-            if hasattr(log_levels, value.upper()):
-                return getattr(log_levels, value.upper())
-
-        raise AttributeError(
-            'Cannot set context-key log_level, ' +
-            'as the value %r did not validate' % value
-        )
-
-    def __dir__(self):
-        res = set(dir(super(ContextModule, self))) | set(properties().keys())
-        if self.defaults:
-            res |= set(dir(self.defaults))
-        return sorted(res)
-
-
-class MainModule(types.ModuleType):
-    '''The module for thread-local context variables.
-
-The purpose of this module is to store runtime configuration of pwntools, such
-as the level of logging or the default architecture for shellcode.
-
-It is implemented as a restricted dictionary, with a predefined number of
-keys and with each key having restrictions of which values it will allow.
-
-The values are available both in a thread-local version and as a global
-default. You are able to read or write each version separately. If you try to
-read from the thread-local version, and no value is found, then the global
-default is checked.
-
-The module :mod:`pwnlib.context` is for accessing the thread-local version,
-while the global defaults are available in :mod:`pwnlib.context.defaults`.
-
-.. note::
-
-   Ideally, we would want to clone the thread-local context on thread creation,
-   but do not know of a way to hook thread creation.
-
-The variables in this module can be read or written directly. If you try to
-write an invalid value, an exception is thrown:
-
-.. doctest:: context_example
-
-   >>> print context.arch
-   None
-   >>> context.arch = 'i386'
-   >>> print context.arch
-   i386
-   >>> context.arch = 'mill'
-   Traceback (most recent call last):
-       ...
-   AttributeError: Cannot set context-key arch, as the value 'mill' did not validate
-
-For a few variables, a slight translation occur when you try to set the
-variable. An example of this is :data:`pwnlib.context.log_level`:
-
-.. doctest:: context_log_level
-
-   >>> context.log_level = 33
-   >>> print context.log_level
-   33
-   >>> context.log_level = 'debug'
-   >>> print context.log_level
-   10
-
-In this case the translation is done by looking up the string in
-:mod:`pwnlib.log`, so the result happens because :data:`pwnlib.log_levels.DEBUG`
-is ``10``.
-
-A read can never throw an exception. If there is no result in the thread-local
-dictionary, the global dictionary is queried. If it has no results either,
-``None`` is returned.
-'''
-
-    def __init__(self):
-        super(MainModule, self).__init__(__name__)
-        sys.modules[self.__name__] = self
-        self.__dict__.update({
-            '__all__'     : ['defaults', 'local', 'reset_local'],
-            '__doc__'     : MainModule.__doc__,
-            '__file__'    : __file__,
-            '__package__' : __package__,
-            'defaults'    : ContextModule(),
-            '_ctxs'       : {},
-        })
-        sys.modules[self.__name__ + '.defaults'] = self.defaults
-        for k, v in defaults.items():
-            setattr(self.defaults, k, v)
-
-    def __call__(self, **kwargs):
-        """Convenience function, which is shorthand for setting multiple
-        variables at once.
-
-        It is a simple shorthand such that::
-
-            context(a = b, c = d, ...)
-
-        is equivalent to::
-
-            context.a = b
-            context.c = d
-            ...
-
-        Args:
-          kwargs: Variables to be assigned in the environment.
-
-        Examples:
-
-          .. doctest:: context
-
-             >>> context(arch = 'i386', os = 'linux')
-             >>> print context.arch
-             i386
-"""
-
-        for k, v in kwargs.items():
-            setattr(self, k, v)
-
-    def _thread_ctx(self):
-        return self._ctxs.setdefault(
-            threading.current_thread().ident, ContextModule(self.defaults)
-        )
-
-    def __getattr__(self, key):
-        return getattr(self._thread_ctx(), key)
-
-    def __setattr__(self, key, value):
-        setattr(self._thread_ctx(), key, value)
-
-    def local(self, **kwargs):
-        '''Create a new thread-local context.
-
-        This function creates a `context manager <https://docs.python.org/2/reference/compound_stmts.html#the-with-statement>`_,
-        which will create a new environment upon entering and restore the old
-        environment upon exiting.
-
-        As a convenience, it also accepts a number of kwarg-style arguments for
-        settings variables in the newly created environment.
-
-        Args:
-          kwargs: Variables to be assigned in the new environment.
-
-        Returns:
-          Context manager for managing the old and new environment.
-
-        Examples:
-
-          .. doctest:: context_local
-
-             >>> print context.arch
-             None
-             >>> with context.local(arch = 'i386'):
-             ...     print context.arch
-             ...     context.arch = 'mips'
-             ...     print context.arch
-             i386
-             mips
-             >>> print context.arch
-             None
-'''
-
-        return Local(kwargs)
-
-    def reset_local(self):
-        '''Completely clears the current thread-local context, thus making the
-        value from :mod:`pwnlib.context.defaults` "shine through".'''
-        ctx = self._thread_ctx()
-        for k in dir(ctx):
-            if k[0] == '_' and k[:2] != '__' and hasattr(ctx, k):
-                delattr(ctx, k)
-
-    def __dir__(self):
-        res = set(dir(super(MainModule, self))) | set(dir(self._thread_ctx()))
-        return sorted(res)
-
-# prevent this scope from being GC'ed
-tether = sys.modules[__name__]
-context = MainModule()

pwnlib/context/__init__.py

@@ -0,0 +1,860 @@
+#!/usr/bin/env python2
+# -*- coding: utf-8 -*-
+"""
+Implements context management so that nested/scoped contexts and threaded
+contexts work properly and as expected.
+"""
+import threading, collections, string, logging
+from ..timeout import Timeout
+
+class _defaultdict(dict):
+    """
+    Dictionary which loads missing keys from another dictionary.
+
+    This is neccesary because the ``default_factory`` method of
+    :class:`collections.defaultdict` does not provide the key.
+
+    Examples:
+
+        >>> a = {'foo': 'bar'}
+        >>> b = pwnlib.context._defaultdict(a)
+        >>> b['foo']
+        'bar'
+        >>> 'foo' in b
+        False
+        >>> b['foo'] = 'baz'
+        >>> b['foo']
+        'baz'
+        >>> del b['foo']
+        >>> b['foo']
+        'bar'
+
+        >>> a = {'foo': 'bar'}
+        >>> b = pwnlib.context._defaultdict(a)
+        >>> b['baz'] #doctest: +ELLIPSIS
+        Traceback (most recent call last):
+        ...
+        KeyError: 'baz'
+    """
+    def __init__(self, default=None):
+        super(_defaultdict, self).__init__()
+        if default is None:
+            default = {}
+
+        self.default = default
+
+
+    def __missing__(self, key):
+        return self.default[key]
+
+class _DictStack(object):
+    """
+    Manages a dictionary-like object, permitting saving and restoring from
+    a stack of states via :func:`push` and :func:`pop`.
+
+    The underlying object used as ``default`` must implement ``copy``, ``clear``,
+    and ``update``.
+
+    Examples:
+
+        >>> t = pwnlib.context._DictStack(default={})
+        >>> t['key'] = 'value'
+        >>> t
+        {'key': 'value'}
+        >>> t.push()
+        >>> t
+        {'key': 'value'}
+        >>> t['key'] = 'value2'
+        >>> t
+        {'key': 'value2'}
+        >>> t.pop()
+        >>> t
+        {'key': 'value'}
+    """
+    def __init__(self, default):
+        self._current = _defaultdict(default)
+        self.__stack  = []
+
+    def push(self):
+        self.__stack.append(self._current.copy())
+
+    def pop(self):
+        self._current.clear()
+        self._current.update(self.__stack.pop())
+
+    def copy(self):
+        return self._current.copy()
+
+    # Pass-through container emulation routines
+    def __len__(self):              return self._current.__len__()
+    def __delitem__(self, k):       return self._current.__delitem__(k)
+    def __getitem__(self, k):       return self._current.__getitem__(k)
+    def __setitem__(self, k, v):    return self._current.__setitem__(k, v)
+    def __contains__(self, k):      return self._current.__contains__(k)
+    def __iter__(self):             return self._current.__iter__()
+    def __repr__(self):             return self._current.__repr__()
+    def __eq__(self, other):        return self._current.__eq__(other)
+
+    # Required for keyword expansion operator ** to work
+    def keys(self):                 return self._current.keys()
+    def values(self):               return self._current.values()
+    def items(self):                return self._current.items()
+
+
+class _Tls_DictStack(threading.local, _DictStack):
+    """
+    Per-thread implementation of :class:`_DictStack`.
+
+    Examples:
+
+        >>> t = pwnlib.context._Tls_DictStack({})
+        >>> t['key'] = 'value'
+        >>> print t
+        {'key': 'value'}
+        >>> def p(): print t
+        >>> thread = threading.Thread(target=p)
+        >>> _ = (thread.start(), thread.join())
+        {}
+    """
+    pass
+
+
+def _validator(validator):
+    """
+    Validator that tis tightly coupled to the implementation
+    of the classes here.
+
+    This expects that the object has a ._tls property which
+    is of type _DictStack.
+    """
+
+    name = validator.__name__
+    doc  = validator.__doc__
+
+    def fget(self):
+        return self._tls[name]
+
+    def fset(self, val):
+        self._tls[name] = validator(self, val)
+
+    def fdel(self):
+        self._tls._current.pop(name,None)
+
+    return property(fget, fset, fdel, doc)
+
+class Thread(threading.Thread):
+    """
+    Instantiates a context-aware thread, which inherit its context when it is
+    instantiated. The class can be accessed both on the context module as
+    `pwnlib.context.Thread` and on the context singleton object inside the
+    context module as `pwnlib.context.context.Thread`.
+
+    Threads created by using the native :class`threading`.Thread` will have a
+    clean (default) context.
+
+    Regardless of the mechanism used to create any thread, the context
+    is de-coupled from the parent thread, so changes do not cascade
+    to child or parent.
+
+    Saves a copy of the context when instantiated (at ``__init__``)
+    and updates the new thread's context before passing control
+    to the user code via ``run`` or ``target=``.
+
+    Examples:
+
+        >>> context.clear()
+        >>> context.update(arch='arm')
+        >>> def p():
+        ...     print context.arch
+        ...     context.arch = 'mips'
+        ...     print context.arch
+        >>> # Note that a normal Thread starts with a clean context
+        >>> # (i386 is the default architecture)
+        >>> t = threading.Thread(target=p)
+        >>> _=(t.start(), t.join())
+        i386
+        mips
+        >>> # Note that the main Thread's context is unchanged
+        >>> print context.arch
+        arm
+        >>> # Note that a context-aware Thread receives a copy of the context
+        >>> t = pwnlib.context.Thread(target=p)
+        >>> _=(t.start(), t.join())
+        arm
+        mips
+        >>> # Again, the main thread is unchanged
+        >>> print context.arch
+        arm
+
+    Implementation Details:
+
+        This class implemented by hooking the private function
+        :func:`threading.Thread._Thread_bootstrap`, which is called before
+        passing control to :func:`threading.Thread.run`.
+
+        This could be done by overriding ``run`` itself, but we would have to
+        ensure that all uses of the class would only ever use the keyword
+        ``target=`` for ``__init__``, or that all subclasses invoke
+        ``super(Subclass.self).set_up_context()`` or similar.
+    """
+    def __init__(self, *args, **kwargs):
+        super(Thread, self).__init__(*args, **kwargs)
+        self.old = context.copy()
+
+    def __bootstrap(self):
+        """
+        Implementation Details:
+            This only works because the class is named ``Thread``.
+            If its name is changed, we have to implement this hook
+            differently.
+        """
+        context.update(**self.old)
+        super(Thread, self).__bootstrap()
+
+def _longest(d):
+    """
+    Returns an OrderedDict with the contents of the input dictionary ``d``
+    sorted by the length of the keys, in descending order.
+
+    This is useful for performing substring matching via ``str.startswith``,
+    as it ensures the most complete match will be found.
+
+    >>> data = {'a': 1, 'bb': 2, 'ccc': 3}
+    >>> _longest(data) == data
+    True
+    >>> for i in _longest(data): print i
+    ccc
+    bb
+    a
+    """
+    return collections.OrderedDict((k,d[k]) for k in sorted(d, key=len, reverse=True))
+
+def TlsProperty(object):
+    def __get__(self, obj, objtype=None):
+        return obj._tls
+
+class ContextType(object):
+    r"""
+    Class for specifying information about the target machine.
+    Intended for use as a pseudo-singleton through the global
+    variable ``pwnlib.context.context``, available via
+    ``from pwn import *`` as ``context``.
+
+    The context is usually specified at the top of the Python file for clarity. ::
+
+        #!/usr/bin/env python
+        context.update(arch='i386', os='linux')
+
+    Currently supported properties and their defaults are listed below.
+    The defaults are inherited from :data:`pwnlib.context.ContextType.defaults`.
+
+    Additionally, the context is thread-aware when using
+    :class:`pwnlib.context.Thread` instead of :class:`threading.Thread`
+    (all internal ``pwntools`` threads use the former).
+
+    The context is also scope-aware by using the ``with`` keyword.
+
+    Examples:
+
+        >>> context.clear()
+        >>> context.update(os='linux') # doctest: +ELLIPSIS
+        >>> context.os == 'linux'
+        True
+        >>> context.arch = 'arm'
+        >>> vars(context) == {'arch': 'arm', 'bits': 32, 'endian': 'little', 'os': 'linux'}
+        True
+        >>> context.endian
+        'little'
+        >>> context.bits
+        32
+        >>> def nop():
+        ...   print pwnlib.asm.asm('nop').encode('hex')
+        >>> nop()
+        00f020e3
+        >>> with context.local(arch = 'i386'):
+        ...   nop()
+        90
+        >>> from pwnlib.context import Thread as PwnThread
+        >>> from threading      import Thread as NormalThread
+        >>> with context.local(arch = 'mips'):
+        ...     pwnthread = PwnThread(target=nop)
+        ...     thread    = NormalThread(target=nop)
+        >>> # Normal thread uses the default value for arch, 'i386'
+        >>> _=(thread.start(), thread.join())
+        90
+        >>> # Pwnthread uses the correct context from creation-time
+        >>> _=(pwnthread.start(), pwnthread.join())
+        00000000
+        >>> nop()
+        00f020e3
+    """
+
+    #
+    # Use of 'slots' is a heavy-handed way to prevent accidents
+    # like 'context.architecture=' instead of 'context.arch='.
+    #
+    # Setting any properties on a ContextType object will throw an
+    # exception.
+    #
+    __slots__ = '_tls',
+
+    #: Default values for :class:`pwnlib.context.ContextType`
+    defaults = {
+        'arch': 'i386',
+        'bits': 32,
+        'endian': 'little',
+        'log_level': logging.INFO,
+        'newline': '\n',
+        'os': 'linux',
+        'signed': False,
+        'timeout': Timeout.maximum,
+    }
+
+    #: Valid values for :meth:`pwnlib.context.ContextType.os`
+    oses = sorted(('linux','freebsd','windows'))
+
+    big_32    = {'endian': 'big', 'bits': 32}
+    big_64    = {'endian': 'big', 'bits': 64}
+    little_8  = {'endian': 'little', 'bits': 8}
+    little_16 = {'endian': 'little', 'bits': 16}
+    little_32 = {'endian': 'little', 'bits': 32}
+    little_64 = {'endian': 'little', 'bits': 64}
+
+    #: Keys are valid values for :meth:`pwnlib.context.ContextType.arch`.
+    #
+    #: Values are defaults which are set when
+    #: :attr:`pwnlib.context.ContextType.arch` is set
+    architectures = _longest({
+        'aarch64':   little_64,
+        'alpha':     little_64,
+        'avr':       little_8,
+        'amd64':     little_64,
+        'arm':       little_32,
+        'cris':      little_32,
+        'i386':      little_32,
+        'ia64':      big_64,
+        'm68k':      big_32,
+        'mips':      little_32,
+        'mips64':    little_64,
+        'msp430':    little_16,
+        'powerpc':   big_32,
+        'powerpc64': big_64,
+        's390':      big_32,
+        'sparc':     big_32,
+        'sparc64':   big_64,
+        'thumb':     little_32,
+        'vax':       little_32,
+    })
+
+    #: Valid values for :attr:`endian`
+    endiannesses = _longest({
+        'be':     'big',
+        'eb':     'big',
+        'big':    'big',
+        'le':     'little',
+        'el':     'little',
+        'little': 'little'
+    })
+
+    #: Valid string values for :attr:`signed`
+    signednesses = {
+        'unsigned': False,
+        'no':       False,
+        'yes':      True,
+        'signed':   True
+    }
+
+    valid_signed = sorted(signednesses)
+
+    def __init__(self, **kwargs):
+        """
+        Initialize the ContextType structure.
+
+        All keyword arguments are passed to :func:`update`.
+        """
+        self._tls = _Tls_DictStack(_defaultdict(ContextType.defaults))
+        self.update(**kwargs)
+
+
+    def copy(self):
+        """copy() -> dict
+        Returns a copy of the current context as a dictionary.
+
+        Examples:
+
+            >>> context.clear()
+            >>> context.os   = 'linux'
+            >>> vars(context) == {'os': 'linux'}
+            True
+        """
+        return self._tls.copy()
+
+
+    @property
+    def __dict__(self):
+        return self.copy()
+
+    def update(self, *args, **kwargs):
+        """
+        Convenience function, which is shorthand for setting multiple
+        variables at once.
+
+        It is a simple shorthand such that::
+
+            context.update(os = 'linux', arch = 'arm', ...)
+
+        is equivalent to::
+
+            context.os   = 'linux'
+            context.arch = 'arm'
+            ...
+
+        The following syntax is also valid::
+
+            context.update({'os': 'linux', 'arch': 'arm'})
+
+        Args:
+          kwargs: Variables to be assigned in the environment.
+
+        Examples:
+
+            >>> context.clear()
+            >>> context.update(arch = 'i386', os = 'linux')
+            >>> context.arch, context.os
+            ('i386', 'linux')
+        """
+        for arg in args:
+            self.update(**arg)
+
+        for k,v in kwargs.items():
+            setattr(self,k,v)
+
+    def __repr__(self):
+        v = sorted("%s = %r" % (k,v) for k,v in self._tls._current.items())
+        return '%s(%s)' % (self.__class__.__name__, ', '.join(v))
+
+    def local(self, **kwargs):
+        """local(**kwargs) -> context manager
+
+        Create a context manager for use with the ``with`` statement.
+
+        For more information, see the example below or PEP 343.
+
+        Args:
+          kwargs: Variables to be assigned in the new environment.
+
+        Returns:
+          ContextType manager for managing the old and new environment.
+
+        Examples:
+
+            >>> context.clear()
+            >>> context.timeout = 1
+            >>> context.timeout == 1
+            True
+            >>> print context.timeout
+            1.0
+            >>> with context.local(timeout = 2):
+            ...     print context.timeout
+            ...     context.timeout = 3
+            ...     print context.timeout
+            2.0
+            3.0
+            >>> print context.timeout
+            1.0
+        """
+        class LocalContext(object):
+            def __enter__(a):
+                self._tls.push()
+                self.update(**{k:v for k,v in kwargs.items() if v is not None})
+                return self
+
+            def __exit__(a, *b, **c):
+                self._tls.pop()
+
+        return LocalContext()
+
+    def clear(self):
+        """
+        Clears the contents of the context.
+        All values are set to their defaults.
+
+        Examples:
+
+            >>> # Default value
+            >>> context.arch == 'i386'
+            True
+            >>> context.arch = 'arm'
+            >>> context.arch == 'i386'
+            False
+            >>> context.clear()
+            >>> context.arch == 'i386'
+            True
+        """
+        self._tls._current.clear()
+
+    @_validator
+    def arch(self, arch):
+        """
+        Target machine architecture.
+
+        Allowed values are listed in :attr:`pwnlib.context.ContextType.architectures`.
+
+        Side Effects:
+
+            If an architecture is specified which also implies additional
+            attributes (e.g. 'amd64' implies 64-bit words, 'powerpc' implies
+            big-endian), these attributes will be set on the context if a
+            user has not already set a value.
+
+            The following properties may be modified.
+
+            - :attr:`bits`
+            - :attr:`endian`
+
+        Raises:
+            AttributeError: An invalid architecture was specified
+
+        Examples:
+
+            >>> context.clear()
+            >>> context.arch == 'i386' # Default architecture
+            True
+
+            >>> context.arch = 'mips'
+            >>> context.arch == 'mips'
+            True
+
+            >>> context.arch = 'doge' #doctest: +ELLIPSIS
+            Traceback (most recent call last):
+             ...
+            AttributeError: arch must be one of ['aarch64', ..., 'thumb']
+
+            >>> context.arch = 'ppc'
+            >>> context.arch == 'powerpc' # Aliased architecture
+            True
+
+            >>> context.clear()
+            >>> context.bits == 32 # Default value
+            True
+            >>> context.arch = 'amd64'
+            >>> context.bits == 64 # New value
+            True
+
+            Note that expressly setting :attr:`bits` means that we use
+            that value instead of the default
+
+            >>> context.clear()
+            >>> context.bits = 32
+            >>> context.arch = 'amd64'
+            >>> context.bits == 32
+            True
+
+            Setting the architecture can override the defaults for
+            both :attr:`endian` and :attr:`bits`
+
+            >>> context.clear()
+            >>> context.arch = 'powerpc64'
+            >>> vars(context) == {'arch': 'powerpc64', 'bits': 64, 'endian': 'big'}
+            True
+        """
+
+        # Lowercase, remove everything non-alphanumeric
+        arch = arch.lower()
+        arch = arch.replace(string.punctuation, '')
+
+        # Attempt to perform convenience and legacy compatibility
+        # transformations.
+        transform = {'x86':'i386', 'ppc': 'powerpc', 'x86_64': 'amd64'}
+        for k, v in transform.items():
+            if arch.startswith(k):
+                arch = arch.replace(k,v,1)
+
+        try:
+            defaults = ContextType.architectures[arch]
+        except KeyError:
+            raise AttributeError('AttributeError: arch must be one of %r' % sorted(ContextType.architectures))
+
+        for k,v in ContextType.architectures[arch].items():
+            if k not in self._tls:
+                self._tls[k] = v
+
+        return arch
+
+    @_validator
+    def bits(self, bits):
+        """
+        Target machine word size, in bits (i.e. the size of general purpose registers).
+
+        The default value is ``32``, but changes according to :attr:`arch`.
+
+        Examples:
+            >>> context.clear()
+            >>> context.bits == 32
+            True
+            >>> context.bits = 64
+            >>> context.bits == 64
+            True
+            >>> context.bits = -1 #doctest: +ELLIPSIS
+            Traceback (most recent call last):
+            ...
+            AttributeError: bits must be >= 0 (-1)
+        """
+        bits = int(bits)
+
+        if bits <= 0:
+            raise AttributeError("bits must be >= 0 (%r)" % bits)
+
+        return bits
+
+    @property
+    def bytes(self):
+        """
+        Target machine word size, in bytes (i.e. the size of general purpose registers).
+
+        This is a convenience wrapper around ``bits / 8``.
+
+        Examples:
+
+            >>> context.bytes = 1
+            >>> context.bits == 8
+            True
+
+            >>> context.bytes = 0 #doctest: +ELLIPSIS
+            Traceback (most recent call last):
+            ...
+            AttributeError: bits must be >= 0 (0)
+        """
+        return self.bits/8
+    @bytes.setter
+    def bytes(self, value):
+        self.bits = value*8
+
+
+    @_validator
+    def endian(self, endianness):
+        """
+        Endianness of the target machine.
+
+        The default value is ``'little'``, but changes according to :attr:`arch`.
+
+        Raises:
+            AttributeError: An invalid endianness was provided
+
+        Examples:
+
+            >>> context.clear()
+            >>> context.endian == 'little'
+            True
+
+            >>> context.endian = 'big'
+            >>> context.endian
+            'big'
+
+            >>> context.endian = 'be'
+            >>> context.endian == 'big'
+            True
+
+            >>> context.endian = 'foobar' #doctest: +ELLIPSIS
+            Traceback (most recent call last):
+             ...
+            AttributeError: endian must be one of ['be', 'big', 'eb', 'el', 'le', 'little']
+        """
+        endian = endianness.lower()
+
+        if endian not in ContextType.endiannesses:
+            raise AttributeError("endian must be one of %r" % sorted(ContextType.endiannesses))
+
+        return ContextType.endiannesses[endian]
+
+
+    @_validator
+    def log_level(self, value):
+        """
+        Sets the verbosity of ``pwntools`` logging mechanism.
+
+        Valid values are specified by the standard Python ``logging`` module.
+
+        Default value is set to ``INFO``.
+
+        Examples:
+
+            >>> context.log_level = 'error'
+            >>> context.log_level == logging.ERROR
+            True
+            >>> context.log_level = 10
+            >>> context.log_level = 'foobar' #doctest: +ELLIPSIS
+            Traceback (most recent call last):
+            ...
+            AttributeError: log_level must be an integer or one of ['CRITICAL', 'DEBUG', 'ERROR', 'INFO', 'NOTSET', 'WARN', 'WARNING']
+        """
+        # If it can be converted into an int, success
+        try:                    return int(value)
+        except ValueError:  pass
+
+        # If it is defined in the logging module, success
+        try:                    return getattr(logging, value.upper())
+        except AttributeError:  pass
+
+        # Otherwise, fail
+        level_names = filter(lambda x: isinstance(x,str), logging._levelNames)
+        permitted = sorted(level_names)
+        raise AttributeError('log_level must be an integer or one of %r' % permitted)
+
+
+    @_validator
+    def os(self, os):
+        """
+        Operating system of the target machine.
+
+        The default value is ``linux``.
+
+        Allowed values are listed in :attr:`pwnlib.context.ContextType.oses`.
+
+        Examples:
+
+            >>> context.os = 'linux'
+            >>> context.os = 'foobar' #doctest: +ELLIPSIS
+            Traceback (most recent call last):
+            ...
+            AttributeError: os must be one of ['freebsd', 'linux', 'windows']
+        """
+        os = os.lower()
+
+        if os not in ContextType.oses:
+            raise AttributeError("os must be one of %r" % sorted(ContextType.oses))
+
+        return os
+
+
+
+    @_validator
+    def signed(self, signed):
+        """
+        Signed-ness for packing operation when it's not explicitly set.
+
+        Can be set to any non-string truthy value, or the specific string
+        values ``'signed'`` or ``'unsigned'`` which are converted into
+        ``True`` and ``False`` correspondingly.
+
+        Examples:
+
+            >>> context.signed
+            False
+            >>> context.signed = 1
+            >>> context.signed
+            True
+            >>> context.signed = 'signed'
+            >>> context.signed
+            True
+            >>> context.signed = 'unsigned'
+            >>> context.signed
+            False
+            >>> context.signed = 'foobar' #doctest: +ELLIPSIS
+            Traceback (most recent call last):
+            ...
+            AttributeError: signed must be one of ['no', 'signed', 'unsigned', 'yes'] or a non-string truthy value
+        """
+        try:             signed = ContextType.signednesses[signed]
+        except KeyError: pass
+
+        if isinstance(signed, str):
+            raise AttributeError('signed must be one of %r or a non-string truthy value' % sorted(ContextType.signednesses))
+
+        return bool(signed)
+
+    @_validator
+    def timeout(self, value=Timeout.default):
+        """
+        Default amount of time to wait for a blocking operation before it times out,
+        specified in seconds.
+
+        The default value is to have an infinite timeout.
+
+        See :class:`pwnlib.timeout.Timeout` for additional information on
+        valid values.
+        """
+        return Timeout(value).timeout
+
+    #*************************************************************************
+    #                               ALIASES
+    #*************************************************************************
+    #
+    # These fields are aliases for fields defined above, either for
+    # convenience or compatibility.
+    #
+    #*************************************************************************
+
+    def __call__(self, **kwargs):
+        """
+        Alias for :meth:`pwnlib.context.ContextType.update`
+        """
+        return self.update(**kwargs)
+
+    def reset_local(self):
+        """
+        Deprecated.  Use :meth:`clear`.
+        """
+        self.clear()
+
+    @property
+    def endianness(self):
+        """
+        Legacy alias for :attr:`endian`.
+
+        Examples:
+
+            >>> context.endian == context.endianness
+            True
+        """
+        return self.endian
+    @endianness.setter
+    def endianness(self, value):
+        self.endian = value
+
+
+    @property
+    def sign(self):
+        """
+        Alias for :attr:`signed`
+        """
+        return self.signed
+
+    @sign.setter
+    def sign(self, value):
+        self.signed = value
+
+    @property
+    def signedness(self):
+        """
+        Alias for :attr:`signed`
+        """
+        return self.signed
+
+    @signedness.setter
+    def signedness(self, value):
+        self.signed = value
+
+
+    @property
+    def word_size(self):
+        """
+        Alias for :attr:`bits`
+        """
+        return self.bits
+
+    @word_size.setter
+    def word_size(self, value):
+        self.bits = value
+
+    Thread = Thread
+
+
+#: Global ``context`` object, used to store commonly-used pwntools settings.
+#: In most cases, the context is used to infer default variables values.
+#: For example, :meth:`pwnlib.asm.asm` can take an ``os`` parameter as a
+#: keyword argument.  If it is not supplied, the ``os`` specified by
+#: ``context`` is used instead.
+#: Consider it a shorthand to passing ``os=`` and ``arch=`` to every single
+#: function call.
+context = ContextType()

pwnlib/data/binutils/LICENSE.txt

@@ -1,14 +0,0 @@
-These files all come from the GNU Binutils package.
-
-The source code is available here:
-https://www.gnu.org/software/binutils/
-
-We have made no alterations to the software, except
-for changing the build-system slightly to allow the
-creating of a disassembler with a horrifically large
-number of supported architectures.
-
-The binaries are to the best of our knowledge redistributable
-under GPL, which is available here:
-
-https://www.gnu.org/licenses/gpl.html