Merge pull request #355 from Gallopsled/setup.py-openbsd

Ugly hack to be able to install on OpenBSD

.gitattributes

@@ -0,0 +1 @@
+pwnlib/data/* linguist-vendored

.gitignore

@@ -5,6 +5,8 @@
 *.pdf
 */auto/*
 */bin/*
+docs/build
 build
 MANIFEST
 dist
+pwntools.egg-info

.travis.yml

@@ -0,0 +1,15 @@
+language: python
+python:
+  - "2.7"
+branches:
+  only:
+    - master
+install:
+  - source .travis_install.sh
+  - source .travis_ssh_setup.sh
+  - python -c 'from pwn import *; print pwnlib.term.term_mode' NOTERM
+  - python -c 'from pwn import *; print pwnlib.term.term_mode' NOTERM
+  - python -c 'from pwn import *; print pwnlib.term.term_mode' NOTERM
+  - python -c 'from pwn import *; print pwnlib.term.term_mode' NOTERM
+
+script: make -C docs doctest

.travis_install.sh

@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+
+setup_linux()
+{
+    sudo apt-get update
+    sudo apt-get install software-properties-common pwgen
+    sudo apt-add-repository --yes ppa:pwntools/binutils
+    sudo apt-get update
+    sudo apt-get install binutils-arm-linux-gnu binutils-mips-linux-gnu binutils-powerpc-linux-gnu
+    wget -nc http://www.capstone-engine.org/download/2.1.2/capstone-2.1.2_amd64.deb
+    (echo "a30bcb58fd82d32b135eec978eaa22230c44e722 *capstone-2.1.2_amd64.deb" | sha1sum -c) || exit
+    sudo dpkg -i *.deb
+}
+
+setup_osx()
+{
+    brew update
+    brew install binutils
+    brew install capstone
+}
+
+case $(uname) in
+Darwin) setup_osx   ;;
+Linux)  setup_linux ;;
+esac
+
+pip install -r requirements.txt
+pip install -r docs/requirements.txt
+pip install .

.travis_ssh_setup.sh

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+#
+# Install a demo user for SSH purposes
+#
+U=demouser
+H=/home/$U
+
+sudo useradd -m $U
+ssh-keygen -t rsa -f /tmp/$U -N ''
+
+eval $(ssh-agent -s)
+ssh-add    /tmp/$U
+
+sudo -u $U mkdir $H/.ssh
+sudo -u $U cp    /tmp/$U.pub $H/.ssh/authorized_keys
+sudo chmod -R 700 $H
+
+(echo -n "$U:"; pwgen 100 1) | sudo chpasswd

CHANGELOG.md

@@ -0,0 +1,18 @@
+# Changelog
+
+This changelog only includes added major features and changes. Bugfixes and
+minor changes are omitted.
+
+## 2.2.0 - 2014-08-28 (pre)
+
+This was a large release with a lot of bugfixes and changes. Only the most significant
+are mentioned here.
+
+- Added shellcodes
+- Added phd
+- Re-added our expansion of itertools
+- Added replacements for some semi-broken python standard library modules
+- Re-implemented the rop module
+- Added a serial tube
+- Huge performance gains in the buffering for tubes
+- Re-added user agents

CONTRIBUTING.md

@@ -0,0 +1,38 @@
+# Contributing
+
+We love pull requests!
+
+Github has a great guide for contributing to open source projects:
+
+- [Contributing to a project](https://guides.github.com/activities/forking/)
+- [Fork the repository](https://guides.github.com/activities/forking/#fork)
+- [Clone your fork](https://guides.github.com/activities/forking/#clone)
+- [Making and pushing changes](https://guides.github.com/activities/forking/#making-changes)
+- [Making a Pull Request](https://guides.github.com/activities/forking/#making-a-pull-request)
+- [Huzzah!](https://guides.github.com/activities/forking/#huzzah)
+
+## Pwntools Specifics
+
+In general, we like to keep things documented.  You should add documentation to any new functionality, and update it for any changed functionality.  Our docstrings use the [Google Style Python Docstrings](https://sphinxcontrib-napoleon.readthedocs.org/en/latest/example_google.html#example-google).
+
+After you have documentation, you should add a [doctest](https://docs.python.org/2/library/doctest.html).
+
+Finally, it is probably a good idea to run the test suite locally before doing
+the pull-request to make sure everything works, however this is not a
+requirement.
+
+Once you do the pull-request Travis CI will run the test-suite on it. Once it
+passes one of the core developers will look at your pull request, possibly
+comment on it and then hopefully merge it into master.
+
+## Test Suite
+
+To run the test suite, you should be running Ubuntu 12.04 or 14.04, and run the following commands.  **Be aware** that this will add a user to your machine, and create a public key for SSH login!
+
+```sh
+bash .travis_install.sh
+bash .travis_ssh_setup.sh
+cd docs
+pip install -r requirements.txt
+make doctest
+```

LICENSE-pwntools.txt

@@ -1,7 +1,6 @@
 TL;DR version:
-   As far as we can tell, everything in pwntools is open source. Most is
-   under an MIT license, but a few pieces are under GPL or a BSD 2-clause
-   licence.
+   Everything in pwntools is open source. Most is under an MIT license, but a
+   few pieces are under GPL or a BSD 2-clause licence.
 
 This license covers everything within this project, except for a few pieces
 of code that we either did not write ourselves or which we derived from code

MANIFEST.in

@@ -0,0 +1 @@
+include *.txt

README.md

@@ -1,40 +1,56 @@
-# Introduction
+# pwntools - CTF toolkit
+[![Docs latest](https://readthedocs.org/projects/pwntools/badge/)](https://pwntools.readthedocs.org/en/latest)
+[![Docs 2.1.3](https://readthedocs.org/projects/pwntools/badge/?version=2.1.3)](https://pwntools.readthedocs.org/en/2.1.3)
+[![PyPI](https://img.shields.io/pypi/v/pwntools.svg)](https://pypi.python.org/pypi/pwntools/)
+[![Gittip](https://img.shields.io/gittip/gallopsled.svg)](https://www.gittip.com/gallopsled/)
+[![Travis](https://travis-ci.org/Gallopsled/pwntools.svg)](https://travis-ci.org/Gallopsled/pwntools)
 
-This is the CTF framework used by pwnies in every CTF.
-
-Most code is inside the pwnlib folder with some functionality inside pwn or bin. It is typically used as:
+This is the CTF framework used by Gallopsled in every CTF.
 
 ```python
 from pwn import *
 context(arch = 'i386', os = 'linux')
 
-# EXPLOIT HERE
+r = remote('exploitme.example.com', 31337)
+# EXPLOIT CODE GOES HERE
+r.send(asm(shellcraft.sh()))
+r.interactive()
 ```
 
-However we have made command-line frontends for much of the functionality
-inside the pwnlib. These are:
+However we have made command-line frontends for some of the functionality
+in `pwnlib`. These are:
 
-* `asm`/`disasm`: Small wrapper for nasm
-* `cyclic`: De Bruijn sequence generator and lookup tool
-* `shellcraft`: Frontend to our shellcode
-
-We also have the following tools, not dependent on the pwnlib:
-
-*  binutils directory: Assemblers and disassemblers for various architectures
-* `hex`/`unhex`: Command line tools for doing common hexing/unhexing operations
+* `asm`/`disasm`: Small wrapper for various assemblers.
+* `constgrep`: Tool for finding constants defined in header files.
+* `cyclic`: De Bruijn sequence generator and lookup tool.
+* `hex`/`unhex`: Command line tools for doing common hexing/unhexing operations.
+* `shellcraft`: Frontend to our shellcode.
+* `phd`: Replacement for `hexdump` with colors.
 
 # Documentation
-Our documentation is available on
+Our documentation is available at [pwntools.readthedocs.org](https://pwntools.readthedocs.org/en/latest/)
+
+To get you started, we've provided some example solutions for past CTF challenges in our [write-ups repository](https://github.com/Gallopsled/pwntools-write-ups).
 
 # Installation
-Pwntools is available as a pip package. You can install it by running
-`pip install pwntools`.
 
-Alternatively if you prefer to have the latest version in git, you can
-simply clone this repository, run `pip install -r requirements.txt`
-and entries in your `PATH` and `PYTHONPATH` variables. The script
-`install_local.sh` will help you do so, in case you are using bash.
+pwntools is best supported on Ubuntu 12.04 and 14.04, but most functionality should work on any Posix-like distribution (Debian, Arch, FreeBSD, OSX, etc.).
+
+Most of the functionality of pwntools is self-contained and Python-only.  You should be able to get running quickly with
+
+```sh
+pip install pwntools
+```
+
+However, some of the features (ROP generation and assembling/disassembling foreign architectures) require non-Python dependencies.  For more information, see the [complete installation instructions here](https://pwntools.readthedocs.org/en/latest/install.html).
+
+
+# Contribution
+
+See [CONTRIBUTING.md](CONTRIBUTING.md)
 
 # Contact
-If you have any questions not worthy of a bug report, feel free to join us
-at `#gallopsled` on Freenode and ask away.
+If you have any questions not worthy of a [bug report](https://github.com/Gallopsled/pwntools/issues), feel free to join us
+at [`#gallopsled` on Freenode](irc://irc.freenode.net/gallopsled) and ask away.
+Click [here](https://kiwiirc.com/client/irc.freenode.net/gallopsled) to connect.
+

bin/constgrep

@@ -1,136 +0,0 @@
-#!/usr/bin/env python2
-
-import argparse, sys, os, types, re
-from pwnlib import asm, util, constants
-from pwnlib.util import safeeval
-from pwnlib.context import __possible__ as contexts
-
-p = argparse.ArgumentParser(
-    description = "Looking up constants from header files.\n\nExample: constgrep -c freebsd -m  ^PROT_ '3 + 4'",
-    formatter_class = argparse.RawDescriptionHelpFormatter,
-)
-
-group = p.add_mutually_exclusive_group()
-group.add_argument(
-    '-e', '--exact',
-    metavar = '<constant name>',
-    # nargs = 1,
-    default = None,
-    help = 'Do an exact match for a constant instead of searching for a regex',
-)
-group.add_argument(
-    'regex',
-    nargs = '?',
-    default = '',
-    help = 'The regex matching constant you want to find',
-)
-
-p.add_argument(
-    'constant',
-    nargs = '?',
-    default = None,
-    help = 'The constant to find',
-)
-
-p.add_argument(
-    '-i', '--case-insensitive',
-    action = 'store_true',
-    help = 'Search case insensitive',
-)
-
-p.add_argument(
-    '-m', '--mask-mode',
-    action = 'store_true',
-    help = 'Instead of searching for a specific constant value, search for values not containing strictly less bits that the given value.',
-)
-
-p.add_argument(
-    '-c', '--context',
-    metavar = '<opt>',
-    choices = contexts['os'] + contexts['arch'],
-    default = ['i386','linux'],
-    action = 'append',
-    help = 'The os/architecture to find constants for (default: linux/i386), choose from: %s' % \
-    ', '.join(dir(constants) + dir(constants.linux))
-)
-
-args = p.parse_args()
-
-# Find the architecture and os from the list of contexts
-for arch in args.context[::-1]:
-    if arch in contexts['arch']: break
-for os in args.context[::-1]:
-    if os in contexts['os']: break
-
-if args.exact:
-    # This is the simple case
-    print asm.cpp(args.exact, os = os, arch = arch).strip()
-else:
-    # New we search in the right module.
-    # But first: We find the right module
-    if os == 'freebsd':
-        mod = constants.freebsd
-    else:
-        mod = getattr(getattr(constants, os), arch)
-
-    # Compile the given regex, for optimized lookup
-    if args.case_insensitive:
-        matcher = re.compile(args.regex, re.IGNORECASE)
-    else:
-        matcher = re.compile(args.regex)
-
-    # Evaluate the given constant
-    if args.constant:
-        constant = safeeval.expr_eval(args.constant)
-    else:
-        constant = None
-
-    # The found matching constants and the length of the longest string
-    out    = []
-    maxlen = 0
-
-    for k in dir(mod):
-        # No python stuff
-        if k.endswith('__') and k.startswith('__'):
-            continue
-
-        # Run the regex
-        if not matcher.search(k):
-            continue
-
-        # Check the constant
-        if constant != None:
-            val = getattr(mod, k)
-            if args.mask_mode:
-                if constant & val != val:
-                    continue
-            else:
-                if constant != val:
-                    continue
-
-        # Append it
-        out.append((getattr(mod, k), k))
-        maxlen = max(len(k), maxlen)
-
-    # Output all matching constants
-    for _, k in sorted(out):
-        print '#define %s %s' % (k.ljust(maxlen), asm.cpp(k, os = os, arch = arch).strip())
-
-    # If we are in match_mode, then try to find a combination of
-    # constants that yield the exact given value
-    # We do not want to find combinations using the value 0.
-    if not (constant == None or constant == 0) and args.mask_mode:
-        mask = constant
-        good = []
-        out = [(v, k) for v, k in out if v != 0]
-
-        while mask and out:
-            cur = out.pop()
-            mask &= ~cur[0]
-            good.append(cur)
-
-            out = [(v, k) for v, k in out if mask & v == v]
-
-        if reduce(lambda x, cur: x | cur[0], good, 0) == constant:
-            print
-            print '(%s) == %s' % (' | '.join(k for v, k in good), args.constant)

bin/cyclic

@@ -1,69 +0,0 @@
-#!/usr/bin/env python2
-import argparse, string
-from pwnlib.util import cyclic, packing
-from pwnlib import log
-
-parser = argparse.ArgumentParser(
-    description = "Cyclic pattern creator/finder"
-)
-
-parser.add_argument(
-    '-a', '--alphabet',
-    metavar = 'alphabet',
-    default = string.ascii_lowercase,
-    help = 'The alphabet to use in the cyclic pattern (defaults to all lower case letters)',
-)
-
-parser.add_argument(
-    '-n', '--length',
-    metavar = 'length',
-    default = 4,
-    type = int,
-    help = 'Size of the unique subsequences (defaults to 4).'
-)
-
-group = parser.add_mutually_exclusive_group(required = True)
-group.add_argument(
-    '-l', '-o', '--offset', '--lookup',
-    metavar = '<lookup value>',
-    help = 'Do a lookup instead printing the alphabet',
-)
-
-group.add_argument(
-    'count',
-    type = int,
-    nargs = '?',
-    help = 'Number of characters to print'
-)
-
-args = parser.parse_args()
-alphabet = args.alphabet
-subsize  = args.length
-
-if args.lookup:
-    pat = args.lookup
-
-    if pat.startswith('0x'):
-        pat = packing.pack(int(pat[2:], 16), subsize*8, 'little', 'unsigned')
-    elif pat.isdigit():
-        pat = packing.pack(int(pat, 10), subsize*8, 'little', 'unsigned')
-
-    if len(pat) != 4:
-        log.fatal('Subpattern must be 4 bytes', 1)
-
-    if not all(c in alphabet for c in pat):
-        log.fatal('Pattern contains characters not present in the alphabet', 1)
-
-    offset = cyclic.cyclic_find(pat, alphabet, subsize)
-
-    if offset == -1:
-        log.fatal('Given pattern does not exist in cyclic pattern', 1)
-    else:
-        print offset
-else:
-    want   = args.count
-    result = cyclic.cyclic(want, alphabet, subsize)
-    got    = len(result)
-    if got < want:
-        log.failure("Alphabet too small (max length = %i)" % got)
-    print result

bin/disasm

@@ -1,38 +0,0 @@
-#!/usr/bin/env python2
-
-import argparse, sys
-from pwnlib import asm, context
-from string import whitespace, hexdigits
-
-parser = argparse.ArgumentParser(
-    description = 'Disassemble bytes into text format'
-)
-
-parser.add_argument(
-    'hex',
-    metavar = 'hex',
-    nargs = '*',
-    help = 'Hex-string to disasemble. If none are supplied, then it uses stdin in non-hex mode.'
-)
-
-parser.add_argument(
-    '-c', '--context',
-    metavar = '<opt>',
-    choices = context.__possible__['arch'],
-    default = 'i386',
-    help = 'The architecture of the shellcode (default: i386), choose from:\n%s' % ', '.join(context.__possible__['arch'])
-)
-
-args = parser.parse_args()
-
-if len(args.hex) > 0:
-    dat = ''.join(args.hex)
-    dat = dat.translate(None, whitespace)
-    if not set(hexdigits) >= set(dat):
-        print "This is not a hex string"
-        exit(-1)
-    dat = dat.decode('hex')
-else:
-    dat = sys.stdin.read()
-
-print asm.disasm(dat, arch = args.context)

bin/hex

@@ -1,7 +0,0 @@
-#!/usr/bin/env python2
-import sys
-
-if len(sys.argv) == 1:
-    print sys.stdin.read().encode('hex')
-else:
-    print ' '.join(sys.argv[1:]).encode('hex')

bin/shellcraft

@@ -1,173 +0,0 @@
-#!/usr/bin/env python2
-import argparse, sys, os, types
-import pwnlib
-from pwnlib import asm, log, util
-from pwnlib.context import __possible__ as contexts
-import pwnlib.term.text as text
-
-r = text.red
-g = text.green
-b = text.blue
-banner = '\n'.join(['  ' + r('____') + '  ' + g('_') + '          ' + r('_') + ' ' + g('_') + '                 ' + b('__') + ' ' + r('_'),
-                    ' ' + r('/ ___|') + g('| |__') + '   ' + b('___') + r('| |') + ' ' + g('|') + ' ' + b('___') + ' ' + r('_ __') + ' ' + g('__ _') + ' ' + b('/ _|') + ' ' + r('|_'),
-                    ' ' + r('\___ \\') + g('| \'_ \\') + ' ' + b('/ _ \\') + ' ' + r('|') + ' ' + g('|') + b('/ __|') + ' ' + r('\'__/') + ' ' + g('_` |') + ' ' + b('|_') + r('| __|'),
-                    '  ' + r('___) |') + ' ' + g('| | |') + '  ' + b('__/') + ' ' + r('|') + ' ' + g('|') + ' ' + b('(__') + r('| |') + ' ' + g('| (_| |') + '  ' + b('_|') + ' ' + r('|_'),
-                    ' ' + r('|____/') + g('|_| |_|') + b('\\___|') + r('_|') + g('_|') + b('\\___|') + r('_|') + '  ' + g('\\__,_|') + b('_|') + '  ' + r('\\__|'),
-                    ''
-                    ])
-#  ____  _          _ _                 __ _
-# / ___|| |__   ___| | | ___ _ __ __ _ / _| |_
-# \___ \| '_ \ / _ \ | |/ __| '__/ _` | |_| __|
-#  ___) | | | |  __/ | | (__| | | (_| |  _| |_
-# |____/|_| |_|\___|_|_|\___|_|  \__,_|_|  \__|
-
-def _string(s):
-    out = []
-    for c in s:
-        co = ord(c)
-        if co >= 0x20 and co <= 0x7e and c not in '/$\'"`':
-            out.append(c)
-        else:
-            out.append('\\x%02x' % co)
-    return '"' + ''.join(out) + '"\n'
-
-def _carray(s):
-    out = []
-    for c in s:
-        out.append('0x' + util.fiddling.enhex(c))
-    return '{' + ', '.join(out) + '};\n'
-
-def _hex(s):
-    return pwnlib.util.fiddling.enhex(s) + '\n'
-
-p = argparse.ArgumentParser(
-    description = banner + '\nMicrowave shellcode -- Easy, fast and delicious',
-    formatter_class = argparse.RawDescriptionHelpFormatter,
-)
-
-p.add_argument(
-    'shellcode',
-    nargs = '?',
-    default = '',
-    metavar = '<shellcode>',
-    help = 'The shellcode you want',
-)
-
-p.add_argument(
-    'args',
-    nargs = '*',
-    metavar = '<arg>',
-    default = (),
-    help = 'Argument to the chosen shellcode',
-)
-
-p.add_argument(
-    '-?', '--show',
-    action = 'store_true',
-    help = 'Show shellcode documentation',
-)
-
-p.add_argument(
-    '-o', '--out',
-    metavar = '<file>',
-    type = argparse.FileType('w'),
-    default = sys.stdout,
-    help = 'Output file (default: stdout)',
-)
-
-p.add_argument(
-    '-f', '--format',
-    metavar = '<format>',
-    choices = ['r', 'raw',
-               's', 'str', 'string',
-               'c',
-               'h', 'hex',
-               'a', 'asm', 'assembly',
-               'p',
-               'i', 'hexii'],
-    default = 'hex',
-    help = 'Output format (default: hex), choose from {r}aw, {s}tring, {c}-style array, {h}ex string, hex{i}i, {a}ssembly code, {p}reprocssed code',
-)
-
-args = p.parse_args()
-
-def get_tree(path, val, result):
-    if path:
-        path += '.'
-
-    if path.startswith('common.'):
-        return
-
-    mods = []
-
-    for k in sorted(dir(val)):
-        if k and k[0] != '_':
-            cur = getattr(val, k)
-            if isinstance(cur, types.ModuleType):
-                mods.append((path + k, cur))
-            else:
-                result.append((path + k, cur))
-
-    for path, val in mods:
-        get_tree(path, val, result)
-    return result
-
-vals = get_tree('', pwnlib.shellcraft, [])
-if args.shellcode:
-    vals = [(k, val) for k, val in vals if k.startswith(args.shellcode + '.') or k == args.shellcode]
-
-if len(vals) == 0:
-    log.fatal("Cannot find subtree by the name of %r" % args.shellcode)
-elif len(vals) > 1:
-    for k, _ in vals:
-        print k
-    exit()
-else:
-    func = vals[0][1]
-
-if args.show:
-    print func.__doc__
-    exit()
-
-defargs = len(func.func_defaults or ())
-reqargs = func.func_code.co_argcount - defargs
-if len(args.args) < reqargs:
-    if defargs > 0:
-        log.fatal('%s takes at least %d arguments' % (args.shellcode, reqargs))
-    else:
-        log.fatal('%s takes exactly %d arguments' % (args.shellcode, reqargs))
-
-# Captain uglyness saves the day!
-for i in range(len(args.args)):
-    if args.args[i] in ['False', 'false']:
-        args.args[i] = False
-
-# And he strikes again!
-os = arch = None
-for k in args.shellcode.split('.')[:-1]:
-    if k in contexts['arch']:
-        arch = k
-    elif k in contexts['os']:
-        os = k
-
-code = func(*args.args)
-
-if args.format in ['a', 'asm', 'assembly']:
-    print code
-    exit()
-if args.format == 'p':
-    print pwnlib.asm.cpp(code, arch = arch, os = os)
-    exit()
-
-code = pwnlib.asm.asm(code, arch = arch, os = os)
-
-if args.format in ['s', 'str', 'string']:
-    code = _string(code)
-elif args.format == 'c':
-    code = _carray(code)
-elif args.format in ['h', 'hex']:
-    code = _hex(code)
-elif args.format in ['i', 'hexii']:
-    code = pwnlib.util.fiddling.hexii(code) + '\n'
-
-sys.stdout.write(code)

bin/unhex

@@ -1,11 +0,0 @@
-#!/usr/bin/env python2
-import sys
-from string import whitespace
-try:
-    if len(sys.argv) == 1:
-        s = sys.stdin.read().translate(None, whitespace)
-        sys.stdout.write(s.decode('hex'))
-    else:
-        sys.stdout.write(''.join(sys.argv[1:]).decode('hex'))
-except TypeError, e:
-    sys.stderr.write(str(e) + '\n')

docs/.gitignore

@@ -1 +1,2 @@
 build
+robots.txt

docs/requirements.txt

@@ -0,0 +1,6 @@
+paramiko
+argparse
+mako
+pyelftools
+sphinxcontrib-napoleon
+sphinxcontrib-autoprogram

docs/source/about.rst

@@ -0,0 +1,56 @@
+About pwntools
+========================
+
+Whether you're using it to write exploits, or as part
+of another software project will dictate how you use it.
+
+Historically pwntools was used as a sort of exploit-writing DSL. Simply doing
+``from pwn import *`` in a previous version of pwntools would bring all sorts of
+nice side-effects.
+
+When redesigning pwntools for 2.0, we noticed two contrary goals:
+
+* We would like to have a "normal" python module structure, to allow other
+  people to faster get familiar with how pwntools works.
+* We would like to have even more side-effects, especially by putting the
+  terminal in raw-mode.
+
+To make this possible, we decided to have two different modules. :mod:`pwnlib`
+would be our nice, clean Python module, while :mod:`pwn` would be used during
+CTFs.
+
+:mod:`pwn` --- Toolbox optimized for CTFs
+-----------------------------------------
+
+.. module:: pwn
+
+As stated, we would also like to have the ability to get a lot of these
+side-effects by default. That is the purpose of this module. It does
+the following:
+
+* Imports everything from the toplevel :mod:`pwnlib` along with
+  functions from a lot of submodules. This means that if you do
+  ``import pwn`` or ``from pwn import *``, you will access to
+  everything you need to write an exploit.
+* Calls :func:`pwnlib.term.init` to put your terminal in raw mode
+  and implementing functionality to make it look like it is not.
+* Setting the :data:`pwnlib.context.log_level` to `"info"`.
+* Tries to parse some of the values in :data:`sys.argv` and every
+  value it succeeds in parsing it removes.
+
+:mod:`pwnlib` --- Normal python library
+---------------------------------------
+
+.. module:: pwnlib
+
+This module is our "clean" python-code. As a rule, we do not think that
+importing :mod:`pwnlib` or any of the submodules should have any significant
+side-effects (besides e.g. caching).
+
+For the most part, you will also only get the bits you import. You for instance
+not get access to :mod:`pwnlib.util.packing` simply by doing ``import
+pwnlib.util``.
+
+Though there are a few exceptions (such as :mod:`pwnlib.shellcraft`), that does
+not quite fit the goals of being simple and clean, but they can still be
+imported without implicit side-effects.

docs/source/asm.rst

@@ -1,6 +1,7 @@
 .. testsetup:: *
 
    from pwnlib.asm import *
+   from pwnlib import shellcraft
 
 :mod:`pwnlib.asm` --- Assembler functions
 =========================================

docs/source/atexception.rst

@@ -0,0 +1,5 @@
+:mod:`pwnlib.atexception` --- Callbacks on unhandled exception
+==============================================================
+
+.. automodule:: pwnlib.atexception
+   :members:

docs/source/atexit.rst

@@ -0,0 +1,5 @@
+:mod:`pwnlib.atexit` --- Replacement for atexit
+===============================================
+
+.. automodule:: pwnlib.atexit
+   :members:

docs/source/commandline.rst

@@ -0,0 +1,46 @@
+.. testsetup:: *
+
+   from pwn import *
+   old = context.defaults.copy()
+
+.. testcleanup:: *
+
+    context.defaults.copy = old
+
+Command Line Tools
+========================
+
+pwntools comes with a handful of useful command-line utilities which serve as wrappers for some of the internal functionality.
+
+.. toctree::
+
+.. autoprogram:: pwnlib.commandline.asm:parser
+   :prog: asm
+
+.. autoprogram:: pwnlib.commandline.constgrep:p
+   :prog: constgrep
+
+.. autoprogram:: pwnlib.commandline.cyclic:parser
+   :prog: cyclic
+
+.. autoprogram:: pwnlib.commandline.disasm:parser
+   :prog: disasm
+
+.. autoprogram:: pwnlib.commandline.elfdiff:p
+   :prog: elfdiff
+
+.. autoprogram:: pwnlib.commandline.elfpatch:p
+   :prog: elfpatch
+
+.. autoprogram:: pwnlib.commandline.hex:parser
+   :prog: hex
+
+.. autoprogram:: pwnlib.commandline.phd:parser
+   :prog: phd
+
+.. autoprogram:: pwnlib.commandline.shellcraft:p
+   :prog: shellcraft
+
+.. autoprogram:: pwnlib.commandline.unhex:parser
+   :prog: unhex
+

docs/source/conf.py

@@ -11,7 +11,7 @@
 # All configuration values have a default; values that are commented out
 # serve to show the default.
 
-import sys, os
+import sys, os, subprocess
 
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
@@ -31,22 +31,22 @@ extensions = [
     'sphinx.ext.doctest',
     'sphinx.ext.linkcode',
     'sphinx.ext.autosummary',
+    'sphinx.ext.coverage',
     'sphinx.ext.todo',
     'sphinx.ext.intersphinx',
+    'sphinxcontrib.autoprogram',
+    'sphinxcontrib.napoleon'
 ]
 
-try:
-    import sphinxcontrib.napoleon
-    extensions.append('sphinxcontrib.napoleon')
-except ImportError:
-    pass
-
 doctest_global_setup = '''
 import pwnlib
-pwnlib.context.reset_local()
+pwnlib.context.context.reset_local()
+pwnlib.context.ContextType.defaults['log_level'] = 'ERROR'
+pwnlib.term.text.when = 'never'
+
 '''
 
-autodoc_member_order = 'bysource'
+autodoc_member_order = 'alphabetical'
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
@@ -57,7 +57,7 @@ doctest_test_doctest_blocks = True
 source_suffix = '.rst'
 
 # The encoding of source files.
-#source_encoding = 'utf-8-sig'
+source_encoding = 'utf-8-sig'
 
 # The master toctree document.
 master_doc = 'index'
@@ -71,9 +71,9 @@ copyright = u'2014, Gallopsled et al'
 # built documents.
 #
 # The short X.Y version.
-version = '2.0'
-# The full version, including alpha/beta/rc tags.
-release = '2.0'
+import pwnlib
+release = pwnlib.__version__
+version = release.rsplit('.', 1)[0]
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
@@ -210,7 +210,8 @@ latex_documents = [
    u'Gallopsled et al', 'manual'),
 ]
 
-intersphinx_mapping = {'python': ('http://docs.python.org/2.7', None)}
+intersphinx_mapping = {'python': ('https://docs.python.org/2.7', None),
+                       'paramiko': ('https://paramiko-docs.readthedocs.org/en/1.15/', None)}
 
 # The name of an image file (relative to this directory) to place at the top of
 # the title page.
@@ -266,6 +267,19 @@ texinfo_documents = [
 # How to display URL addresses: 'footnote', 'no', or 'inline'.
 #texinfo_show_urls = 'footnote'
 
+branch = release
+
+try:
+    git_branch = subprocess.check_output('git describe --tags', shell = True)
+except subprocess.CalledProcessError:
+    git_branch = '-'
+
+try:
+    if '-' in git_branch:
+        branch = subprocess.check_output('git rev-parse HEAD', shell = True).strip()[:10]
+except subprocess.CalledProcessError:
+    pass
+
 def linkcode_resolve(domain, info):
     if domain != 'py':
         return None
@@ -290,21 +304,11 @@ def linkcode_resolve(domain, info):
     else:
         filename = info['module'].replace('.', '/') + '.py'
 
-        # Fixup for context properties
-        if info['module'].startswith('pwnlib.context'):
-            filename = 'pwnlib/context.py'
-
-            if not isinstance(val, (types.FunctionType, types.MethodType)):
-                import pwnlib.context.defaults
-                val = getattr(pwnlib.context.defaults.__class__, info['fullname'], None)
-                if isinstance(val, property):
-                    val = val.fget
-                    val = getattr(val, '_inner', val)
-
         if isinstance(val, (types.ModuleType, types.ClassType, types.MethodType, types.FunctionType, types.TracebackType, types.FrameType, types.CodeType)):
             try:
                 lines, first = inspect.getsourcelines(val)
                 filename += '#L%d-%d' % (first, first + len(lines) - 1)
             except IOError:
                 pass
-    return "https://github.com/pwnies/pwntools/blob/master/%s" % filename
+
+    return "https://github.com/Gallopsled/pwntools/blob/%s/%s" % (branch, filename)

docs/source/constants.rst

@@ -0,0 +1,9 @@
+.. testsetup:: *
+
+   from pwnlib import constants
+   from pwnlib.context import context
+
+:mod:`pwnlib.constants` --- Easy access to header file constants
+================================================================
+
+.. automodule:: pwnlib.constants

docs/source/context.rst

@@ -1,49 +1,16 @@
 .. testsetup:: *
 
-   from pwnlib import context
+   import pwnlib
+   from   pwnlib.context import context
+   import threading, logging
 
 
 :mod:`pwnlib.context` --- Setting runtime variables
 =====================================================
 
+.. autodata:: pwnlib.context.context
 
-:mod:`pwnlib.context`
----------------------
+.. autoclass:: pwnlib.context.ContextType
+    :members:
 
-.. automodule:: pwnlib.context
-
-   .. exec::
-      import pwnlib
-      from inspect import cleandoc
-      mod = pwnlib.context.defaults.__class__
-      for key in sorted(dir(mod)):
-          val = getattr(mod, key)
-          if isinstance(val, property):
-              print ".. data:: " + key
-              print ""
-              print cleandoc(val.__doc__)
-              print ""
-
-
-   .. autofunction:: pwnlib.context.__call__(**kwargs)
-   .. autofunction:: pwnlib.context.local(**kwargs)
-   .. autofunction:: pwnlib.context.reset_local()
-
-:mod:`pwnlib.context.defaults`
-------------------------------
-
-.. automodule:: pwnlib.context.defaults
-
-   .. exec::
-      import pwnlib
-      mod = pwnlib.context.defaults.__class__
-      for key in sorted(dir(mod)):
-          val = getattr(mod, key)
-          if isinstance(val, property):
-              print ".. data:: " + key
-              print ""
-              print "This is the global version of :data:`pwnlib.context.%s`." % key
-              print
-
-
-   .. autofunction:: pwnlib.context.defaults.__call__(**kwargs)
+.. autoclass:: pwnlib.context.Thread

docs/source/dynelf.rst

@@ -0,0 +1,5 @@
+:mod:`pwnlib.dynelf` --- Resolving remote functions using leaks
+===============================================================
+
+.. automodule:: pwnlib.dynelf
+   :members:

docs/source/elf.rst

@@ -0,0 +1,10 @@
+.. testsetup:: *
+
+   from pwnlib.elf import *
+   from pwnlib.util.misc import which
+
+:mod:`pwnlib.elf` --- Working with ELF binaries
+===============================================
+
+.. automodule:: pwnlib.elf
+   :members:

docs/source/gdb.rst

@@ -0,0 +1,5 @@
+:mod:`pwnlib.gdb` --- Working with GDB
+======================================
+
+.. automodule:: pwnlib.gdb
+   :members:

docs/source/globals.rst

@@ -0,0 +1,102 @@
+.. testsetup:: *
+
+   from pwn import *
+
+``from pwn import *``
+========================
+
+The most common way that you'll see pwntools used is
+
+    >>> from pwn import *
+
+Which imports a bazillion things into the global namespace to make your life easier.
+
+This is a quick list of most of the objects and routines imported, in rough order of importance and frequency of use.
+
+- ``context``
+    - :data:`pwnlib.context.context`
+    - Responsible for most of the pwntools convenience settings
+    - Set `context.log_level = 'debug'` when troubleshooting your exploit
+    - Scope-aware, so you can disable logging for a subsection of code via ``pwnlib.context.ContextType.local``
+- ``remote``, ``listen``, ``ssh``, ``process``
+    - :mod:`pwnlib.tubes`
+    - Super convenient wrappers around all of the common functionality for CTF challenges
+    - Connect to anything, anywhere, and it works the way you want it to
+    - Helpers for common tasks like ``recvline``, ``recvuntil``, ``clean``, etc.
+    - Interact directly with the application via ``.interactive()``
+- ``p32`` and ``u32``
+    - :mod:`pwnlib.util.packing`
+    - Useful functions to make sure you never have to remember if ``'>'`` means signed or unsigned for ``struct.pack``, and no more ugly ``[0]`` index at the end.
+    - Set ``signed`` and ``endian`` in sane manners (also these can be set once on ``context`` and not bothered with again)
+    - Most common sizes are pre-defined (``u8``, ``u64``, etc), and :func:`pwnlib.util.packing.pack` lets you define your own.
+- ``log``
+    - :mod:`pwnlib.log`
+    - Make your output pretty!
+- ``cyclic`` and ``cyclic_func``
+    - :mod:`pwnlib.util.cyclic`
+    - Utilities for generating strings such that you can find the offset of any given substring given only N (usually 4) bytes.  This is super useful for straight buffer overflows.  Instead of looking at 0x41414141, you could know that 0x61616171 means you control EIP at offset 64 in your buffer.
+- ``asm`` and ``disasm``
+    - :mod:`pwnlib.asm`
+    - Quickly turn assembly into some bytes, or vice-versa, without mucking about
+    - Supports any architecture for which you have a binutils installed
+    - Over 20 different architectures have pre-built binaries at `ppa:pwntools/binutils <https://launchpad.net/~pwntools/+archive/ubuntu/binutils>`_.
+- ``shellcraft``
+    - :mod:`pwnlib.shellcraft`
+    - Library of shellcode ready to go
+    - ``asm(shellcraft.sh())`` gives you a shell
+    - Templating library for reusability of shellcode fragments
+- ``ELF``
+    - :mod:`pwnlib.elf`
+    - ELF binary manipulation tools, including symbol lookup, virtual memory to file offset helpers, and the ability to modify and save binaries back to disk
+- ``DynELF``
+    - :mod:`pwnlib.dynelf`
+    - Dynamically resolve functions given only a pointer to any loaded module, and a function which can leak data at any address
+- ``ROP``
+    - :mod:`pwnlib.rop`
+    - Automatically generate ROP chains using a DSL to describe what you want to do, rather than raw addresses
+- ``gdb.debug`` and ``gdb.attach``
+    - :mod:`pwnlib.gdb`
+    - Launch a binary under GDB and pop up a new terminal to interact with it.  Automates setting breakpoints and makes iteration on exploits MUCH faster.
+    - Alternately, attach to a running process given a PID, ``pwnlib.tubes`` object, or even just a socket that's connected to it
+- ``args``
+    - Dictionary contining all-caps command-line arguments for quick access
+    - Run via ``python foo.py REMOTE=1`` and ``args['REMOTE'] == '1'``.
+    - Can also control logging verbosity and terminal fancyness
+        - `NOTERM`
+        - `SILENT`
+        - `DEBUG`
+- ``randoms``, ``rol``, ``ror``, ``xor``, ``bits``
+    - :mod:`pwnlib.util.fiddling`
+    - Useful utilities for generating random data from a given alphabet, or simplifying math operations that usually require masking off with `0xffffffff` or calling `ord` and `chr` an ugly number of times
+- ``net``
+    - :mod:`pwnlib.util.net`
+    - Routines for querying about network interfaces
+- ``proc``
+    - :mod:`pwnlib.util.proc`
+    - Routines for querying about processes
+- ``pause``
+    - It's the new ``getch``
+- ``safeeval``
+    - :mod:`pwnlib.util.safeeval`
+    - Functions for safely evalutaing python code without nasty side-effects.
+
+These are all pretty self explanatory, but are useful to have in the global namespace.
+
+- ``hexdump``
+- ``read`` and ``write``
+- ``enhex`` and ``unhex``
+- ``more``
+- ``group``
+- ``align`` and ``align_down``
+- ``urlencode`` and ``urldecode``
+- ``which``
+- ``wget``
+
+Additionally, all of the following modules are auto-imported for you.  You were going to do it anyway.
+
+- ``os``
+- ``sys``
+- ``time``
+- ``requests``
+- ``re``
+- ``random``

docs/source/index.rst

@@ -1,22 +1,52 @@
-Welcome to pwntools's documentation!
+pwntools
 ====================================
 
-Contents:
+``pwntools`` is a CTF framework and exploit development library.
+Written in Python, it is designed for rapid prototyping and development,
+and intended to make exploit writing as simple as possible.
+
+Getting Started
+---------------
 
 .. toctree::
    :maxdepth: 3
    :glob:
 
+   about
+   install
    intro
+   globals
+   commandline
+
+
+Module Index
+------------
+
+Each of the ``pwntools`` modules is documented here.
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
    asm
+   atexception
+   atexit
+   constants
    context
+   dynelf
+   elf
    exception
+   gdb
    log
    memleak
-   tubes
+   replacements
+   rop
    shellcraft
    term
+   timeout
+   tubes
    ui
+   useragents
    util/*
 
 Indices and tables

docs/source/install.rst

@@ -0,0 +1,46 @@
+Installation
+============
+
+pwntools is best supported on Ubuntu 12.04 and 14.04, but most
+functionality should work on any Posix-like distribution (Debian, Arch,
+FreeBSD, OSX, etc.).
+
+Prerequisites
+-------------
+
+In order to get the most out of ``pwntools``, you should have the
+following system libraries installed.
+
+.. toctree::
+   :maxdepth: 3
+   :glob:
+
+   install/*
+
+Released Version
+-----------------
+
+Pwntools is available as a ``pip`` package.
+
+.. code-block:: bash
+
+    $ apt-get install python2.7 python2.7-dev python-pip
+    $ pip install pwntools
+
+Latest Version
+--------------
+
+Alternatively if you prefer to use the latest version from the
+repository:
+
+.. code-block:: bash
+
+    $ git clone https://github.com/Gallopsled/pwntools
+    $ cd pwntools
+    $ pip install -e .
+
+.. _Ubuntu: https://launchpad.net/~pwntools/+archive/ubuntu/binutils
+.. _libcapstone 2.1: http://www.capstone-engine.org
+.. _i386: http://www.capstone-engine.org/download/2.1.2/capstone-2.1.2_i386.deb
+.. _amd64: http://www.capstone-engine.org/download/2.1.2/capstone-2.1.2_amd64.deb
+

docs/source/install/binutils.rst

@@ -0,0 +1,82 @@
+Binutils
+-------------
+
+Assembly of foreign architectures (e.g. assembling Sparc shellcode on
+Mac OS X) requires cross-compiled versions of ``binutils`` to be
+installed. We've made this process as smooth as we can.
+
+In these examples, replace ``$ARCH`` with your target architecture (e.g., arm, mips64, vax, etc.).
+
+Ubuntu
+^^^^^^^^^^^^^^^^
+First, add our `Personal Package Archive repository <http://binutils.pwntools.com>`__.
+
+.. code-block:: bash
+
+    $ apt-get install software-properties-common
+    $ apt-add-repository ppa:pwntools/binutils
+    $ apt-get updatez
+
+Then, install the binutils for your architecture.
+
+.. code-block:: bash
+
+    $ apt-get install binutils-$ARCH-linux-gnu
+
+Mac OS X
+^^^^^^^^^^^^^^^^
+
+Mac OS X is just as easy, but requires building binutils from source.
+However, we've made ``homebrew`` recipes to make this a single command.
+After installing `brew <http://brew.sh>`__, grab the appropriate
+recipe from our `binutils
+repo <https://github.com/Gallopsled/pwntools-binutils/>`__.
+
+.. code-block:: bash
+
+    $ brew install https://raw.githubusercontent.com/Gallopsled/pwntools-binutils/master/osx/binutils-$ARCH.rb
+
+Alternate OSes
+^^^^^^^^^^^^^^^^
+
+If you want to build everything by hand, or don't use any of the above
+OSes, ``binutils`` is simple to build by hand.
+
+.. code-block:: bash
+
+    #!/usr/bin/env bash
+
+    V=2.25   # Binutils Version
+    ARCH=arm # Target architecture
+
+    cd /tmp
+    wget -nc https://ftp.gnu.org/gnu/binutils/binutils-$V.tar.gz
+    wget -nc https://ftp.gnu.org/gnu/binutils/binutils-$V.tar.gz.sig
+
+    gpg --keyserver keys.gnupg.net --recv-keys 4AE55E93
+    gpg --verify binutils-$V.tar.gz.sig
+
+    rm -rf binutils-*
+
+    tar xf binutils-$V.tar.gz
+
+    mkdir binutils-build
+    cd binutils-build
+
+    export AR=ar
+    export AS=as
+
+    ../binutils-$V/configure \
+        --prefix=/usr/local \
+        --target=$ARCH-unknown-linux-gnu \
+        --disable-static \
+        --disable-multilib \
+        --disable-werror \
+        --disable-nls
+
+    MAKE=gmake
+    hash gmake || MAKE=make
+
+    $MAKE -j
+    sudo $MAKE install
+

docs/source/install/capstone.rst

@@ -0,0 +1,27 @@
+Capstone
+-------------
+
+Capstone is a disassembly library required for gathering ROP gadgets
+and ROP chain generation.
+
+It's a separate requirement from ``binutils`` because it's used by
+Jon Salwan's ``ROPgadget`` tool which we use under the covers.
+
+In particular, version ``2.1.2`` should be used.  Capstone can be downloaded here_, or installed with the steps below.
+
+Ubuntu
+^^^^^^^^^^^^^^^^
+
+.. code-block:: bash
+
+    $ wget -nc http://www.capstone-engine.org/download/2.1.2/capstone-2.1.2_amd64.deb
+    $ apt-get install capstone-2.1.2_amd64.deb
+
+Mac OS X
+^^^^^^^^^^^^^^^^
+
+.. code-block:: bash
+
+    $ brew install capstone
+
+.. _here:  http://www.capstone-engine.org/download.html

docs/source/install/headers.rst

@@ -0,0 +1,18 @@
+Python Development Headers
+-------------
+
+Some of pwntools' Python dependencies require native extensions (for example, Paramiko requires PyCrypto).
+
+In order to build these native extensions, the development headers for Python must be installed.
+
+Ubuntu
+^^^^^^^^^^^^^^^^
+
+.. code-block:: bash
+
+    $ apt-get install python-dev
+
+Mac OS X
+^^^^^^^^^^^^^^^^
+
+No action needed.

docs/source/intro.rst

@@ -1,53 +1,221 @@
-Introduction to pwntools
+.. testsetup:: *
+
+   from pwn import *
+
+Getting Started
 ========================
 
-Historically pwntools was used as a sort of exploit-writing DSL. Simply doing
-``from pwn import *`` in a previous version of pwntools would bring all sorts of
-nice side-effects.
+To get your feet wet with pwntools, let's first go through a few examples.
 
-When redesigning pwntools for 2.0, we noticed two contrary goals:
+When writing exploits, pwntools generally follows the "kitchen sink" approach.
 
-* We would like to have a "normal" python module structure, to allow other
-  people to faster get familiar with how pwntools works.
-* We would like to have even more side-effects, especially by putting the
-  terminal in raw-mode.
+    >>> from pwn import *
 
-To make this possible, we decided to have two different modules. :mod:`pwnlib`
-would be our nice, clean python module, while :mod:`pwn` would be used during
-CTFs.
+This imports a lot of functionality into the global namespace.  You can now
+assemble, disassemble, pack, unpack, and many other things with a single function.
 
-:mod:`pwnlib` --- Normal python library
----------------------------------------
+A full list of everything that is imported is available on :doc:`globals`.
 
-.. module:: pwnlib
 
-This module is our "clean" python-code. As a rule, we do not think that
-importing :mod:`pwnlib` or any of the submodules should have any significant
-side-effects (besides e.g. caching).
+Making Connections
+------------------
 
-For the most part, you will also only get the bits you import. You for instance
-not get access to :mod:`pwnlib.util.packing` simply by doing ``import
-pwnlib.util``.
+You need to talk to the challenge binary in order to pwn it, right?
+pwntools makes this stupid simple with its :mod:`pwnlib.tubes` module.
 
-Though there are a few exceptions (such as :mod:`pwnlib.shellcraft`), that does
-not quite fit the goals of being simple and clean, but they can still be
-imported without implicit side-effects.
+This exposes a standard interface to talk to processes, sockets, serial ports,
+and all manner of things, along with some nifty helpers for common tasks.
+For example, remote connections via :mod:`pwnlib.tubes.remote`.
 
-:mod:`pwn` --- Toolbox optimized for CTFs
------------------------------------------
+    >>> conn = remote('ftp.debian.org',21)
+    >>> conn.recvline() # doctest: +ELLIPSIS
+    '220 ...'
+    >>> conn.send('USER anonymous\r\n')
+    >>> conn.recvuntil(' ', drop=True)
+    '331'
+    >>> conn.recvline()
+    'Please specify the password.\r\n'
+    >>> conn.close()
 
-.. module:: pwn
+It's also easy to spin up a listener
 
-As stated, we would also like to have the ability to get a lot of these
-side-effects by default. That is the purpose of this module. It does
-the following:
+    >>> l = listen()
+    >>> r = remote('localhost', l.lport)
+    >>> c = l.wait_for_connection()
+    >>> r.send('hello')
+    >>> c.recv()
+    'hello'
+
+Interacting with processes is easy thanks to :mod:`pwnlib.tubes.process`.
+
+::
+
+    >>> sh = process('/bin/sh')
+    >>> sh.sendline('sleep 3; echo hello world;')
+    >>> sh.recvline(timeout=1)
+    ''
+    >>> sh.recvline(timeout=5)
+    'hello world\n'
+    >>> sh.close()
+
+Not only can you interact with processes programmatically, but you can
+actually **interact** with processes.
+
+    >>> sh.interactive() # doctest: +SKIP
+    $ whoami
+    user
+
+There's even an SSH module for when you've got to SSH into a box to perform
+a local/setuid exploit with :mod:`pwnlib.tubes.ssh`.  You can quickly spawn
+processes and grab the output, or spawn a process and interact iwth it like
+a ``process`` tube.
+
+::
+
+    >>> shell = ssh('bandit0', 'bandit.labs.overthewire.org', password='bandit0')
+    >>> shell['whoami']
+    'bandit0'
+    >>> shell.download_file('/etc/motd')
+    >>> sh = shell.run('sh')
+    >>> sh.sendline('sleep 3; echo hello world;') # doctest: +SKIP
+    >>> sh.recvline(timeout=1)
+    ''
+    >>> sh.recvline(timeout=5)
+    'hello world\n'
+    >>> shell.close()
+
+Packing Integers
+------------------
+
+A common task for exploit-writing is converting between integers as Python
+sees them, and their representation as a sequence of bytes.
+Usually folks resort to the built-in ``struct`` module.
+
+pwntools makes this easier with :mod:`pwnlib.util.packing`.  No more remembering
+unpacking codes, and littering your code with helper routines.
+
+    >>> import struct
+    >>> p32(0xdeadbeef) == struct.pack('I', 0xdeadbeef)
+    True
+    >>> leet = '37130000'.decode('hex')
+    >>> u32('abcd') == struct.unpack('I', 'abcd')[0]
+    True
+
+The packing/unpacking operations are defined for many common bit-widths.
+
+    >>> u8('A') == 0x41
+    True
+
+Setting the Target Architecture and OS
+--------------------------------------
+
+The target architecture can generally be specified as an argument to the routine that requires it.
+
+    >>> asm('nop')
+    '\x90'
+    >>> asm('nop', arch='arm')
+    '\x00\xf0 \xe3'
+
+However, it can also be set once in the global ``context``.  The operating system, word size, and endianness can also be set here.
+
+    >>> context.arch      = 'i386'
+    >>> context.os        = 'linux'
+    >>> context.endian    = 'little'
+    >>> context.word_size = 32
+
+Additionally, you can use a shorthand to set all of the values at once.
+
+    >>> asm('nop')
+    '\x90'
+    >>> context(arch='arm', os='linux', endian='big', word_size=32)
+    >>> asm('nop')
+    '\xe3 \xf0\x00'
+
+.. doctest::
+   :hide:
+
+    >>> context.clear()
+
+Setting Logging Verbosity
+-------------------------
+
+You can control the verbosity of the standard pwntools logging via ``context``.
+
+For example, setting
+
+    >>> context.log_level = 'debug'
+
+Will cause all of the data sent and received by a ``tube`` to be printed to the screen.
+
+.. doctest::
+   :hide:
+
+    >>> context.clear()
+
+Assembly and Disassembly
+------------------------
+
+Never again will you need to run some already-assembled pile of shellcode
+from the internet!  The :mod:`pwnlib.asm` module is full of awesome.
+
+    >>> asm('mov eax, 0').encode('hex')
+    'b800000000'
+
+But if you do, it's easy to suss out!
+
+    >>> print disasm('6a0258cd80ebf9'.decode('hex'))
+       0:   6a 02                   push   0x2
+       2:   58                      pop    eax
+       3:   cd 80                   int    0x80
+       5:   eb f9                   jmp    0x0
+
+However, you shouldn't even need to write your own shellcode most of the
+time!  Pwntools comes with the :mod:`pwnlib.shellcraft` module, which is
+loaded with useful time-saving shellcodes.
+
+Let's say that we want to `setreuid(getuid(), getuid())` followed by `dup`ing
+file descriptor 4 to `stdin`, `stdout`, and `stderr`, and then pop a shell!
+
+    >>> asm(shellcraft.setreuid() + shellcraft.dupsh(4)).encode('hex')
+    '6a3158cd8089c389d96a4658cd806a045b6a0359496a3f58cd8075f86a68682f2f2f73682f62696e89e331c96a0b5899cd80'
+
+
+Misc Tools
+----------------------
+
+Never write another hexdump, thanks to :mod:`pwnlib.util.fiddling`.
+
+
+Find offsets in your buffer that cause a crash, thanks to :mod:`pwnlib.cyclic`.
+
+    >>> print cyclic(20)
+    aaaabaaacaaadaaaeaaa
+    >>> # Assume EIP = 0x62616166 ('faab') at crash time
+    >>> print cyclic_find('faab')
+    120
+
+ELF Manipulation
+----------------
+
+Stop hard-coding things!  Look them up at runtime with :mod:`pwnlib.elf`.
+
+    >>> e = ELF('/bin/cat')
+    >>> print hex(e.address) #doctest: +SKIP
+    0x400000
+    >>> print hex(e.symbols['write']) #doctest: +SKIP
+    0x401680
+    >>> print hex(e.got['write']) #doctest: +SKIP
+    0x60b070
+    >>> print hex(e.plt['write']) #doctest: +SKIP
+    0x401680
+
+You can even patch and save the files.
+
+    >>> e = ELF('/bin/cat')
+    >>> e.read(e.address+1, 3)
+    'ELF'
+    >>> e.asm(e.address, 'ret')
+    >>> e.save('/tmp/quiet-cat')
+    >>> disasm(file('/tmp/quiet-cat','rb').read(1))
+    '   0:   c3                      ret'
 
-* Imports everything from the toplevel :mod:`pwnlib` along with
-  functions from a lot of submodules. This means that if you do
-  ``import pwn`` or ``from pwn import *``, you will access to
-  everything you need to write an exploit.
-* Calls :func:`pwnlib.term.init` to put your terminal in raw mode
-  and implementing functionality to make it look like it is not.
-* Setting the :data:`pwnlib.context.log_level` to `"info"`.
-* Tries to parse some of the values in :data:`sys.argv` and every
-  value it succeeds in parsing it removes.

docs/source/log.rst

@@ -2,10 +2,6 @@
 =================================================================
 
 .. automodule:: pwnlib.log
-   :members:
 
-:mod:`pwnlib.log_levels` --- Possible log levels
-------------------------------------------------
-
-.. automodule:: pwnlib.log_levels
-   :members:
+.. autoclass:: pwnlib.log.Logger
+   :members: debug, info, warn, error, progress, success, failure, indented, info_once, warn_once

docs/source/memleak.rst

@@ -1,3 +1,7 @@
+.. testsetup:: *
+
+   from pwnlib.memleak import *
+
 :mod:`pwnlib.memleak` --- Helper class for leaking memory
 =========================================================
 

docs/source/replacements.rst

@@ -0,0 +1,5 @@
+:mod:`pwnlib.replacements` --- Replacements for various functions
+=================================================================
+
+.. automodule:: pwnlib.replacements
+   :members:

docs/source/rop.rst

@@ -0,0 +1,11 @@
+.. testsetup:: *
+
+   from pwnlib.elf import *
+   from pwnlib.rop import *
+   from pwnlib.util.misc import which
+
+:mod:`pwnlib.rop` --- Return Oriented Programming
+=================================================
+
+.. automodule:: pwnlib.rop
+   :members:

docs/source/shellcraft.rst

@@ -1,14 +1,20 @@
+.. testsetup:: *
+
+   from pwnlib import shellcraft
+
 :mod:`pwnlib.shellcraft` --- Shellcode generation
 =================================================
 
 .. automodule:: pwnlib.shellcraft
 
+.. TODO:
+
+   Write a guide to adding more shellcode.
+
+Submodules
+----------
+
 .. toctree::
    :glob:
 
    shellcraft/*
-
-Writing shellcode
------------------
-
-Foo bar

docs/source/shellcraft/amd64.rst

@@ -0,0 +1,18 @@
+.. testsetup:: *
+
+   from pwn import *
+
+:mod:`pwnlib.shellcraft.amd64` --- Shellcode for AMD64
+===========================================================
+
+:mod:`pwnlib.shellcraft.amd64`
+---------------------------------------
+
+.. automodule:: pwnlib.shellcraft.amd64
+   :members:
+
+:mod:`pwnlib.shellcraft.amd64.linux`
+---------------------------------------
+
+.. automodule:: pwnlib.shellcraft.amd64.linux
+   :members:

docs/source/shellcraft/arm.rst

@@ -0,0 +1,18 @@
+.. testsetup:: *
+
+   from pwn import *
+
+:mod:`pwnlib.shellcraft.arm` --- Shellcode for ARM
+===========================================================
+
+:mod:`pwnlib.shellcraft.arm`
+-----------------------------
+
+.. automodule:: pwnlib.shellcraft.arm
+   :members:
+
+:mod:`pwnlib.shellcraft.arm.linux`
+-----------------------------------
+
+.. automodule:: pwnlib.shellcraft.arm.linux
+   :members:

docs/source/shellcraft/i386.rst

@@ -1,11 +1,24 @@
+.. testsetup:: *
+
+   from pwn import *
+
 :mod:`pwnlib.shellcraft.i386` --- Shellcode for Intel 80386
 ===========================================================
 
+:mod:`pwnlib.shellcraft.i386`
+-----------------------------
+
 .. automodule:: pwnlib.shellcraft.i386
    :members:
 
+:mod:`pwnlib.shellcraft.i386.linux`
+-----------------------------------
+
 .. automodule:: pwnlib.shellcraft.i386.linux
    :members:
 
+:mod:`pwnlib.shellcraft.i386.freebsd`
+-------------------------------------
+
 .. automodule:: pwnlib.shellcraft.i386.freebsd
    :members:

docs/source/shellcraft/thumb.rst

@@ -0,0 +1,18 @@
+.. testsetup:: *
+
+   from pwn import *
+
+:mod:`pwnlib.shellcraft.thumb` --- Shellcode for Thumb Mode
+===========================================================
+
+:mod:`pwnlib.shellcraft.thumb`
+-------------------------------
+
+.. automodule:: pwnlib.shellcraft.thumb
+   :members:
+
+:mod:`pwnlib.shellcraft.thumb.linux`
+---------------------------------------
+
+.. automodule:: pwnlib.shellcraft.thumb.linux
+   :members:

docs/source/timeout.rst

@@ -0,0 +1,11 @@
+.. testsetup:: *
+
+   from pwnlib.context import context
+   from pwnlib.timeout import Timeout
+   import time
+
+:mod:`pwnlib.timeout` --- Timeout handling
+========================================
+
+.. automodule:: pwnlib.timeout
+   :members:

docs/source/tubes.rst

@@ -1,3 +1,11 @@
+.. testsetup:: *
+
+   from pwnlib.tubes.tube import tube
+   from pwnlib.tubes.ssh import *
+   from pwnlib.tubes.remote import remote
+   from pwnlib.util.misc import which
+   from pwnlib.context import context
+
 :mod:`pwnlib.tubes` --- Talking to the World!
 =============================================
 
@@ -8,20 +16,19 @@
 
    .. automodule:: pwnlib.tubes.remote
 
-      .. autoclass:: pwnlib.tubes.remote.remote(host, port, fam = None, typ = None, proto = 0, timeout = 'default', log_level = INFO)
+      .. autoclass:: pwnlib.tubes.remote.remote
          :members:
          :show-inheritance:
 
    .. automodule:: pwnlib.tubes.listen
 
-      .. autoclass:: pwnlib.tubes.listen.listen(port, bindaddr = "0.0.0.0", fam = "any", typ = "tcp", timeout = 'default', log_level = INFO)
+      .. autoclass:: pwnlib.tubes.listen.listen
          :members:
          :show-inheritance:
 
    .. automodule:: pwnlib.tubes.sock
 
       .. autoclass:: pwnlib.tubes.sock.sock()
-         :members: shutdown
          :show-inheritance:
 
    Processes
@@ -29,11 +36,27 @@
 
    .. automodule:: pwnlib.tubes.process
 
-      .. autoclass:: pwnlib.tubes.process.process(args, shell = False, executable = None, env = None, timeout = 'default', log_level = INFO)
-         :members: kill, shutdown, poll, communicate
+      .. autoclass:: pwnlib.tubes.process.process
+         :members: kill, poll, communicate
          :show-inheritance:
 
+   SSH
+   ---
 
+   .. automodule:: pwnlib.tubes.ssh
+
+      .. autoclass:: pwnlib.tubes.ssh.ssh
+         :members:
+
+      .. autoclass:: pwnlib.tubes.ssh.ssh_channel()
+         :members: kill, poll, interactive
+         :show-inheritance:
+
+      .. autoclass:: pwnlib.tubes.ssh.ssh_connecter()
+         :show-inheritance:
+
+      .. autoclass:: pwnlib.tubes.ssh.ssh_listener()
+         :show-inheritance:
 
    Common functionality
    --------------------

docs/source/useragents.rst

@@ -0,0 +1,9 @@
+.. testsetup:: *
+
+   from pwnlib.useragents import *
+
+:mod:`pwnlib.useragents` --- A database of useragent strings
+============================================================
+
+.. automodule:: pwnlib.useragents
+   :members:

docs/source/util/hashes.rst

@@ -0,0 +1,5 @@
+:mod:`pwnlib.util.hashes` --- Hashing functions
+===============================================
+
+.. automodule:: pwnlib.util.hashes
+   :members:

docs/source/util/iters.rst

@@ -0,0 +1,113 @@
+.. testsetup:: *
+
+   import string, operator
+   from pwnlib.util.iters import *
+
+
+:mod:`pwnlib.util.iters` --- Extension of standard module :mod:`itertools`
+==========================================================================
+
+.. automodule:: pwnlib.util.iters
+   :members:
+   :exclude-members:
+     chain                                  ,
+     combinations                           ,
+     combinations_with_replacement          ,
+     compress                               ,
+     count                                  ,
+     cycle                                  ,
+     dropwhile                              ,
+     groupby                                ,
+     ifilter                                ,
+     ifilterfalse                           ,
+     imap                                   ,
+     islice                                 ,
+     izip                                   ,
+     izip_longest                           ,
+     permutations                           ,
+     product                                ,
+     repeat                                 ,
+     starmap                                ,
+     takewhile                              ,
+     tee
+
+.. function:: chain
+
+   Alias for :func:`itertools.chain`.
+
+.. function:: combinations
+
+   Alias for :func:`itertools.combinations`
+
+.. function:: combinations_with_replacement
+
+   Alias for :func:`itertools.combinations_with_replacement`
+
+.. function:: compress
+
+   Alias for :func:`itertools.compress`
+
+.. function:: count
+
+   Alias for :func:`itertools.count`
+
+.. function:: cycle
+
+   Alias for :func:`itertools.cycle`
+
+.. function:: dropwhile
+
+   Alias for :func:`itertools.dropwhile`
+
+.. function:: groupby
+
+   Alias for :func:`itertools.groupby`
+
+.. function:: ifilter
+
+   Alias for :func:`itertools.ifilter`
+
+.. function:: ifilterfalse
+
+   Alias for :func:`itertools.ifilterfalse`
+
+.. function:: imap
+
+   Alias for :func:`itertools.imap`
+
+.. function:: islice
+
+   Alias for :func:`itertools.islice`
+
+.. function:: izip
+
+   Alias for :func:`itertools.izip`
+
+.. function:: izip_longest
+
+   Alias for :func:`itertools.izip_longest`
+
+.. function:: permutations
+
+   Alias for :func:`itertools.permutations`
+
+.. function:: product
+
+   Alias for :func:`itertools.product`
+
+.. function:: repeat
+
+   Alias for :func:`itertools.repeat`
+
+.. function:: starmap
+
+   Alias for :func:`itertools.starmap`
+
+.. function:: takewhile
+
+   Alias for :func:`itertools.takewhile`
+
+.. function:: tee
+
+   Alias for :func:`itertools.tee`
+

docs/source/util/misc.rst

@@ -1,7 +1,7 @@
 .. testsetup:: *
 
    from pwnlib.util.misc import *
-   import os
+   import os, subprocess
    os.chdir("..")
 
 :mod:`pwnlib.util.misc` --- We could not fit it any other place

docs/source/util/proc.rst

@@ -1,6 +1,7 @@
 .. testsetup:: *
 
    from pwnlib.util.proc import *
+   import os, sys
 
 
 :mod:`pwnlib.util.proc` --- Working with ``/proc/``

docs/source/util/web.rst

@@ -0,0 +1,10 @@
+.. testsetup:: *
+
+   from pwnlib.context import context
+   from pwnlib.util.web import *
+
+:mod:`pwnlib.util.web` --- Utilities for working with the WWW
+=============================================================
+
+.. automodule:: pwnlib.util.web
+   :members:

examples/README.md

@@ -0,0 +1,109 @@
+# Examples
+While these examples should all work, they are not very representative of
+the pwntools project.
+
+We have a plan to create a separate repository with examples, primarily
+exploits. Until we do so, we recommend new users to look at
+https://pwntools.readthedocs.org, as this is a better overview of our features.
+
+In no particular order the docstrings for each example:
+
+* `args.py`
+```
+When not in lib-mode (import `pwn` rather than `pwnlib`) we parse the
+commandline for variables definitions.  A variable definition has the form::
+
+  <var>=<val>
+
+where ``<var>`` contains only uppercase letters, digits and underscores and
+doesn't start with a digit.
+
+Try running this example with::
+
+  $ python args.py RHOST=localhost RPORT=1337
+```
+* `asm.py`
+```
+Example showing the interface to `pwnlib.asm.asm` and `pwnlib.shellcraft`.
+```
+* `clean_and_log.py`
+```
+Use case for `pwnlib.tubes.tube.clean_and_log`.
+
+Sometimes you will have a solution to a challenge but you don't know what it
+will look like when you get the flag.  Sometimes that will leave you with a
+top-level exception, no flag, and angry team members.
+
+Solution:
+ 1. Always run wireshark or tcpdump.  Always.
+ 2. Register <your socket>.clean or <your socket>.clean_and_log to run at exit.
+```
+* `indented.py`
+```
+When running in term-mode (import `pwn` rather than `pwnlib`, stdout is a TTY
+and not running in a REPL), we can do proper indentation where lines too long to
+fit on a screen are split into multiple individually indented lines.
+
+Too see the difference try running with::
+
+  $ python indented.py
+
+and
+
+  $ python -i indented.py
+
+Also notice that `pause()` can react on any key when in `term_mode`.
+```
+* `listen_uroboros.py`
+```
+An example showing interconnection of sockets.  This script will wait for three
+connections on port 1337, then connect them like a three-way Uroboros.
+```
+* `options.py`
+```
+Example showing `pwnlib.ui.options()`
+```
+* `port_forward.py`
+```
+A very simple port forwarder using `pwnlib.tubes.tube.connect_both()`.  Notice
+that `<>` is just a shorthand.
+```
+* `readline_completers.py`
+```
+Example showing pwnlib's readline implementation and a few completers.  This
+part of pwnlib will probably see some major changes soon, but we wanted to show
+off some proof-of-concepts.
+```
+* `remote_gdb_debugging.py`
+```
+Simple example showing how to use the remote
+gdb debugging features available in pwntools.
+```
+* `spinners.py`
+```
+Just a lot of spinners!
+```
+* `splash.py`
+```
+"Easteregg"
+```
+* `ssh.py`
+```
+Example showing how to use the ssh class.
+```
+* `text.py`
+```
+Example showing how to use `pwnlib.term.text`.
+
+Try running with::
+
+  $ TERM=xterm python text.py
+
+and::
+
+  $ TERM=xterm-256color python text.py
+```
+* `remote.py`
+```
+Example showing how to use the remote class.
+```

examples/args.py

@@ -0,0 +1,18 @@
+"""
+When not in lib-mode (import `pwn` rather than `pwnlib`) we parse the
+commandline for variables definitions.  A variable definition has the form::
+
+  <var>=<val>
+
+where ``<var>`` contains only uppercase letters, digits and underscores and
+doesn't start with a digit.
+
+Try running this example with::
+
+  $ python args.py RHOST=localhost RPORT=1337
+"""
+
+from pwn import *
+
+print args['RHOST'] or 'RHOST is not set'
+print args['RPORT'] or 'RPORT is not set'

examples/argstest.py

@@ -1,4 +0,0 @@
-import pwn
-
-print pwn.args['RHOST'],
-print pwn.args['RPORT']

examples/asm.py

@@ -1,3 +1,7 @@
+"""
+Example showing the interface to `pwnlib.asm.asm` and `pwnlib.shellcraft`.
+"""
+
 from pwn import *
 context(arch='i386', os='linux', net='tcp4')
 

examples/clean_and_log.py

@@ -0,0 +1,29 @@
+"""
+Use case for `pwnlib.tubes.tube.clean_and_log`.
+
+Sometimes you will have a solution to a challenge but you don't know what it
+will look like when you get the flag.  Sometimes that will leave you with a
+top-level exception, no flag, and angry team members.
+
+Solution:
+ 1. Always run wireshark or tcpdump.  Always.
+ 2. Register <your socket>.clean or <your socket>.clean_and_log to run at exit.
+"""
+
+from pwn import *
+
+os.system('''((
+echo prefix sometext ;
+echo prefix someothertext ;
+echo here comes the flag ;
+echo LostInTheInterTubes
+) | nc -l 1337) &
+''')
+
+r = remote('localhost', 1337)
+atexit.register(r.clean_and_log)
+
+while True:
+    line = r.recvline()
+    print re.findall('^prefix (\S+)$', line)[0]
+

examples/completertest.py

@@ -1,15 +0,0 @@
-import pwn
-import pwnlib.term.completer
-
-c1 = pwn.term.completer.LongestPrefixCompleter([
-    'foobar',
-    'foobaz',
-    'fooquux',
-    ])
-
-c2 = pwn.term.completer.PathCompleter(mask = '*.py')
-
-with c1:
-    while True:
-        s = pwn.term.readline.readline(prompt = '> ')
-        print 'You wrote', s

examples/contexttest.py

@@ -1,6 +0,0 @@
-import pwn
-
-print pwn.context
-
-pwn.context.arch = 'i386'
-pwn.context.arch = 'yolo'

examples/gen-README.py

@@ -0,0 +1,41 @@
+"""
+Script to generate README.md
+"""
+
+from pwn import *
+
+out = '''# Examples
+While these examples should all work, they are not very representative of
+the pwntools project.
+
+We have a plan to create a separate repository with examples, primarily
+exploits. Until we do so, we recommend new users to look at
+https://pwntools.readthedocs.org, as this is a better overview of our features.
+
+In no particular order the docstrings for each example:
+
+'''
+
+def append_example(_arg, top, names):
+    global out
+    for name in names:
+        if not (name.endswith('.py') and name != __file__):
+            continue
+        path = os.path.join(top, name)[2:] # strip './'
+        log.info('-> %s' % path)
+        data = read(path).strip()
+        if data[0:3] not in ('"""', "'''"):
+            log.warning('  Has no docstring!')
+            continue
+        try:
+            i = data.index(data[0:3], 3)
+        except ValueError:
+            log.warning('  Docstring is weird')
+            continue
+        doc = util.safeeval.const(data[0:i + 3])
+        out += '* `%s`\n' % path
+        out += '```%s```\n' % doc
+
+os.path.walk('.', append_example, None)
+
+write('README.md', out)

examples/indented.py

@@ -0,0 +1,25 @@
+"""
+When running in term-mode (import `pwn` rather than `pwnlib`, stdout is a TTY
+and not running in a REPL), we can do proper indentation where lines too long to
+fit on a screen are split into multiple individually indented lines.
+
+Too see the difference try running with::
+
+  $ python indented.py
+
+and
+
+  $ python -i indented.py
+
+Also notice that `pause()` can react on any key when in `term_mode`.
+"""
+
+from pwn import *
+
+context.log_level = 'info'
+
+log.indented('A' * 100)
+log.indented('B' * 100)
+log.indented('C' * 100)
+
+pause()

examples/indenttest.py

@@ -1,8 +0,0 @@
-import pwn
-
-pwn.context.log_level = 'info'
-
-# print pwn2.lib
-pwn.log.info(('A' * 100 + '\n') * 4)
-
-raw_input('>')

examples/inputtest.py

@@ -1,48 +0,0 @@
-# raise Exception('1234')
-
-# import sys
-# _write = sys.stdout.write
-# class Proxy:
-#     def __init__(self, fd):
-#         self._fd = fd
-#     def readline(self):
-#         _write('Input plox: ')
-#         return self._fd.readline()
-#     def read(self, n):
-#         _write('xxx')
-#         return self._fd.read(n)
-#     def __getattr__(self, k):
-#         # _write('[%s]\n' % k)
-#         return self._fd.__getattribute__(k)
-
-# # sys.stdout = Proxy(sys.stdout)
-# sys.stdin = Proxy(sys.stdin)
-
-# print raw_input()
-
-
-import pwn
-import pwnlib.term.readline
-
-h = pwn.term.output('yolo\n')
-i = 0
-def foo(_):
-    global i
-    i += 1
-    pwn.term.output('foooooo%d\n' % i, frozen = True, before = h)
-
-def bar():
-    print
-
-def baz(_):
-    import sys
-    sys.exit()
-
-pwnlib.term.readline.startup_hook = bar
-pwnlib.term.readline.keymap.register({
-    'C-x C-a': foo,
-    'C-x C-s': baz,
-    })
-
-while True:
-    print raw_input('> ')

examples/iotest.py

@@ -1,70 +0,0 @@
-import termios, tty, sys, string, time
-
-def get_term_size():
-    fd = os.open(os.ctermid(), os.O_RDONLY)
-    height, width = struct.unpack('hh', fcntl.ioctl(fd, termios.TIOCGWINSZ, '1234'))
-    os.close(fd)
-    return width, height
-
-def get_cursor_pos():
-    sys.stdout.write("\033[6n")
-    s = ''
-    while True:
-        c = sys.stdin.read(1)
-        if c == 'R':
-            break
-        s += c
-    col, row = map(int, s[2:].split(';'))
-    return col, row
-
-fd = sys.stdin.fileno()
-old_settings = termios.tcgetattr(fd)
-tty.setraw(sys.stdin.fileno())
-
-while True:
-    # print 'At (%d, %d)' % get_cursor_pos()
-    c = sys.stdin.read(1)
-    if c == 'q':
-        break
-    if c == '\r':
-        print
-    if c == 'w':
-        s = '\x01\x1b[s\x1b[50Ayolo\x1b[u\x02'
-        sys.stdout.write(s)
-    if c == 'e':
-        sys.stdout.write('\x1b[s')
-        for i in range(100):
-            sys.stdout.write('\x1b[0;15f%s' % '/-\\|'[i % 4])
-            sys.stdout.flush()
-            time.sleep(0.1)
-        sys.stdout.write('\x1b[u')
-    if c == 'E':
-        sys.stdout.write('\x1b[s')
-        for i in range(100):
-            sys.stdout.write('\x1b[0;15H%s' % '.oO*'[i % 4])
-            sys.stdout.flush()
-            time.sleep(0.1)
-        sys.stdout.write('\x1b[u')
-    if c == 'r':
-        sys.stdout.write('\x1b[S')
-    if c == 't':
-        sys.stdout.write('\x1b[T')
-    if c == 'y':
-        sys.stdout.write('\x1b[syolo\r\n\x1b[ufoo')
-    if c == 'u':
-        x = ord('a')
-        for i in range(10):
-            sys.stdout.write('\x1b[A' + chr(x))
-            sys.stdout.flush()
-            time.sleep(0.3)
-            x += 1
-            sys.stdout.write('\x1b[B' + chr(x))
-            sys.stdout.flush()
-            time.sleep(0.3)
-            x += 1
-    if c in string.printable:
-        sys.stdout.write(c)
-    else:
-        sys.stdout.write('\\x%02x' % ord(c))
-
-termios.tcsetattr(fd, termios.TCSADRAIN, old_settings)

examples/keymaptest.py

@@ -1,27 +0,0 @@
-import pwn
-
-def match(t):
-    print 'match', t
-
-def nomatch(t):
-    print 'nomatch', t
-
-def key(t):
-    print 'key', t
-
-def tab(t):
-    print 'tab', t
-
-def tabtab(t):
-    print 'tabtab', t
-
-import pwnlib.term.keymap
-km = pwnlib.term.keymap.Keymap({
-    '<match>'     : match,
-    '<nomatch>'   : nomatch,
-    '<any>'       : key,
-    '<tab>'       : tab,
-    '<tab> <tab>' : tabtab,
-    })
-
-km.handle_input()

examples/listen_uroboros.py

@@ -0,0 +1,14 @@
+"""
+An example showing interconnection of sockets.  This script will wait for three
+connections on port 1337, then connect them like a three-way Uroboros.
+"""
+
+from pwn import *
+
+cs = [listen(1337).wait_for_connection() for _ in range(3)]
+
+cs[0] << cs[1] << cs[2] << cs[0]
+
+cs[0].wait_for_close()
+cs[1].wait_for_close()
+cs[2].wait_for_close()

examples/logtest.py

@@ -1,55 +0,0 @@
-# import pwn2.lib as lib
-import pwn
-import time
-
-pwn.context.log_level = 0
-
-n = 0
-
-h = pwn.log.waitfor('spinners running', status = '0')
-
-hs = []
-while True:
-    s = raw_input('> ')
-    if s == 'q':
-        break
-    hs.append(pwn.log.waitfor(s, status = 'running'))
-    n += 1
-    h.status(str(n))
-
-h.success()
-
-for h in hs:
-    h.failure()
-
-# pwn.log.waitfor('waiting for stuff')
-# time.sleep(0.5)
-# pwn.log.info('something happened')
-# time.sleep(0.5)
-# pwn.log.success('something went well')
-# time.sleep(0.5)
-# pwn.log.waitfor('waiting for some minor stuff')
-# time.sleep(0.5)
-# pwn.log.failure('went wrong')
-# raw_input('please hit <enter> ')
-# time.sleep(0.5)
-# pwn.log.done_failure()
-# time.sleep(0.5)
-# pwn.log.info('all in all it went ok')
-# time.sleep(0.5)
-# pwn.log.done_success()
-
-
-# def foo():
-#     plib.log.stub()
-
-# foo()
-
-# try:
-#     try:
-#         raise KeyError
-#     except:
-#         plib.log.die('no reason')
-# except Exception as e:
-#     print `e`
-#     # raise

examples/memleaktest.py

@@ -1,12 +0,0 @@
-from pwn import *
-
-context(endianness='little', sign='unsigned')
-
-def legit_leak(addr):
-    return p32(addr)
-
-mem = MemLeak(legit_leak)
-print mem.b(0x08040001) # prints 1
-mem.setd(0x08040000, 0xaabbccdd)
-print hex(mem.b(0x08040001))   # prints 0xcc
-print mem.raw(0x08040000, 20)  # prints stuff

examples/options.py

@@ -0,0 +1,8 @@
+"""
+Example showing `pwnlib.ui.options()`
+"""
+
+from pwn import *
+
+opts = [string.letters[x] for x in range(10)]
+print 'You choose "%s"' % opts[options('Pick one:', opts)]

examples/port_forward.py

@@ -0,0 +1,11 @@
+"""
+A very simple port forwarder using `pwnlib.tubes.tube.connect_both()`.  Notice
+that `<>` is just a shorthand.
+"""
+
+from pwn import *
+
+while True:
+    listen(1337).wait_for_connection() <> remote('google.com', 80)
+
+# now point your browser (or curl(1)) to http://localhost:1337

examples/ptytest.py

@@ -1,37 +0,0 @@
-import pty, os, select, sys, time, pwn
-
-pid, fd = pty.fork()
-
-if pid == 0:
-    argv = ('/usr/bin/vim',)
-    argv = ('/bin/bash',)
-    os.execlp(argv[0], *argv)
-else:
-    # time.sleep(1)
-    # sys.stdout.write(os.read(fd, 4096))
-    while True:
-        rfds, wfds, xfds = select.select([fd, 0], [], [])
-        if fd in rfds:
-            s = os.read(fd, 4096)
-            if not s:
-                raise EOFError
-            sys.stdout.write(s)
-            sys.stdout.flush()
-        if 0 in rfds:
-            s = os.read(0, 4096)
-            if not s:
-                raise EOFError
-            os.write(fd, s)
-
-
-# def master_read(fd):
-#     import os
-#     s = os.read(fd, 4096)
-#     return s
-
-# def stdin_read(fd):
-#     import os
-#     s = os.read(fd, 4096)
-#     return s
-
-# pty.spawn('/bin/sh', master_read = master_read, stdin_read = stdin_read)

examples/readline_completers.py

@@ -0,0 +1,31 @@
+"""
+Example showing pwnlib's readline implementation and a few completers.  This
+part of pwnlib will probably see some major changes soon, but we wanted to show
+off some proof-of-concepts.
+"""
+
+from pwn import *
+from pwnlib.term.completer import LongestPrefixCompleter, \
+                                  PathCompleter
+
+c1 = LongestPrefixCompleter([
+    'foobar',
+    'foobaz',
+    'fooqux',
+    'exit',
+    'enough!',
+    ])
+
+c2 = PathCompleter(mask = '*.py')
+
+with c1:
+    print 'type "exit" to exit'
+    while True:
+        s = term.readline.readline(prompt = '> ').strip()
+        if s in ('exit', 'enough!'):
+            break
+        print 'You wrote', s
+with c2:
+    print 'choose a file'
+    s = term.readline.readline(prompt = text.bold_green('$ ')).strip()
+    print 'You picked', s

examples/readlineexc.py

@@ -1,15 +0,0 @@
-
-# import sys
-# class Wrapper:
-#     def __init__(self, fd):
-#         self._fd = fd
-#     def readline(self, size = None):
-#         return ''
-#     def __getattr__(self, k):
-#         return self._fd.__getattribute__(k)
-# sys.stdin = Wrapper(sys.stdin)
-# print `raw_input()`
-
-import readline
-import pwn
-raw_input('> ')

examples/remote.py

@@ -0,0 +1,11 @@
+"""
+Example showing how to use the remote class.
+"""
+
+from pwn import *
+
+sock = remote('127.0.0.1', 9001)
+
+print sock.recvline()
+sock.send('foo')
+sock.interactive()

examples/remote_gdb_debugging.py

@@ -0,0 +1,16 @@
+"""
+Simple example showing how to use the remote
+gdb debugging features available in pwntools.
+"""
+
+from pwn import *
+import getpass
+
+s = ssh(getpass.getuser(), '127.0.0.1', port = 22, keyfile = "~/.ssh/id_rsa")
+c = gdb.ssh_gdb(s, '/bin/sh', execute = '''
+p/x $pc
+c''')
+
+c.sendline('ls -la')
+c.sendline('exit')
+print c.recvall()

examples/spinners.py

@@ -0,0 +1,25 @@
+"""
+Just a lot of spinners!
+"""
+
+from pwn import *
+
+context.log_level = 0
+
+n = 1
+h = log.waitfor('spinners running', status = str(n))
+
+hs = []
+print 'type "q" to quit'
+while True:
+    s = raw_input('> ')
+    if s == 'q':
+        break
+    hs.append(log.waitfor(s, status = 'running'))
+    n += 1
+    h.status(str(n))
+
+h.success()
+
+for h in hs:
+    h.failure()

examples/splash.py

@@ -0,0 +1,13 @@
+"""
+"Easteregg"
+"""
+
+from pwn import *
+
+splash()
+
+h = log.waitfor("You wrote", status = "--")
+
+while True:
+    l = raw_input('> ')
+    h.status(l.upper())

examples/ssh.py

@@ -0,0 +1,40 @@
+"""
+Example showing how to use the ssh class.
+"""
+
+from pwn import *
+
+shell = ssh(host='bandit.labs.overthewire.org',user='bandit0',password='bandit0')
+
+# Show basic command syntax
+log.info("username: %s" % shell.whoami())
+log.info("pwd: %s" % shell.pwd())
+
+# Show full tube syntax
+tube = shell.run('cat')
+tube.send("Hello, cat")
+tube.shutdown("out")
+print tube.recvall()
+
+# Show automatic working directories
+shell.set_working_directory()
+log.info("pwd: %s" % shell.pwd())
+
+shell.upload_data("""
+#include <stdio.h>
+int main() {
+    return printf("Hello, world");
+}
+""", 'example.c')
+
+shell.gcc(['example.c','-o','example'])
+
+print shell['./example']
+
+# Show the different styles of calling
+print shell.echo("single string")
+print shell.echo(["list","of","strings"])
+print shell["echo single statement"]
+
+# Show off the interactive shell
+shell.interactive()

examples/termtest.py

@@ -1,93 +0,0 @@
-import pwn
-import pwnlib.term
-from pwnlib.term import *
-import time, os, sys, threading
-
-from pwnlib.log import *
-
-try:
-    # output(os.urandom(10))
-    # print 'foo'
-    # print 'bar'
-    # h = output('')
-    # for i in range(250):
-    #     time.sleep(0.005)
-    #     print 'foo%d' % i
-    #     # print 'xx'
-    #     # time.sleep(1)
-    #     # update(h, 'hello \x1b[3%dmthere\x1b[m\n' % (i % 6 + 1))
-    # # freeze(h)
-    # # print pwn2.nonlib.term.parse('hello \x1b[32mthere\x1b[m\n')
-    # # output('hello \x1b[32mthere\x1b[m\n')
-    # # update(h, 'asdf')
-    # print 'hello'
-
-    # output('fooxxx')
-    # h1 = output('')
-    # h2 = output('bar')
-    # output(' baz\n')
-
-    # # update(h2, 'BAR')
-    # update(h1, '\x1b[K\x1b[2A\x1b[1;31mYOLO\x1b[m\x1b[2B')
-
-    # print 'fooxxx\x1b[2Dbar', 'baz'
-
-    clock = output(float = True, frozen = False)
-    def loop():
-        while True:
-            clock.update('\n' + time.asctime(time.localtime()) + '\n')
-            time.sleep(1)
-    t = threading.Thread(target = loop)
-    t.daemon = True
-    t.start()
-    i = 0
-    # numlines = output()
-    numlines = info(frozen = False)
-    while True:
-        s = ''
-        numlines.update('You have written %d lines' % i)
-        i += 1
-        s = raw_input('> ')
-        output('you wrote: ')
-        output('%s\n' % s, indent = 20)
-        if s == 'q':
-            break
-
-    # output('\xc3\xb8' * 85)
-    # output('\x1b[44m \x1b[m')
-    # # output('\x1b[J')
-    # time.sleep(1)
-
-    # h = output('last\n', float = True)
-    # output('foo')
-    # # update(h, 'LAST\n')
-    # h = output('bar')
-    # output('baz\n')
-    # # delete(h)
-    # update(h, 'BAR')
-
-
-    # output('\x1b[s\x1b[20;40H')
-    # h = output('yolo')
-    # update(h, 'YOLO\x1b[u')
-
-    # output('\n')
-    # pwn2.nonlib.term.goto((1, 0))
-    # pwn2.nonlib.term.put('%d' % pwn2.nonlib.term.scroll)
-    # pwn2.nonlib.term.flush()
-    # time.sleep(1)
-
-    # pwn2.nonlib.term.put('\n'*10)
-
-    pass
-
-except:
-    # pwn2.nonlib.term.fini()
-    # raise e
-    # pwn2.nonlib.term.put(str(e))
-    pass
-    raise
-
-pwn2.nonlib.term.put('\n' * 20)
-for c in pwn2.nonlib.term.cells:
-    pwn2.nonlib.term.put('%s %s %s %s\n' % (c.start, c.end, c.float, `c.content`))

examples/text.py

@@ -0,0 +1,19 @@
+'''
+Example showing how to use `pwnlib.term.text`.
+
+Try running with::
+
+  $ TERM=xterm python text.py
+
+and::
+
+  $ TERM=xterm-256color python text.py
+'''
+from pwn import *
+
+s = 'hello from pwntools'
+print text.black_on_green(s)
+print text.black_on_bright_green(s)
+print text.green_on_black(s)
+print text.bright_green_on_black(s)
+print text.bold_green_on_black(s)

install_local.sh

@@ -1,5 +1,5 @@
 #!/bin/sh
-sudo pip install -r requirements.txt
+sudo pip2 install -r requirements.txt
 
 echo "================================================"
 echo "Now please put this directory in your PYTHONPATH"

pwn/__init__.py

@@ -1,15 +1,9 @@
 # Promote useful stuff to toplevel
-import sys
 from .toplevel import *
 
-# Put the terminal in raw mode
-term.init()
-
-# default log level in non-lib mode
-context.defaults.log_level = 'info'
-
 # look for special args in argv
 def closure():
+    term_mode = True
     import sys
     if not hasattr(sys, 'argv'):
         return
@@ -26,18 +20,55 @@ def closure():
         if not all(c in body for c in s[1:]):
             return False
         return True
+    def asbool(s):
+        if   s.lower() == 'true':
+            return True
+        elif s.lower() == 'false':
+            return False
+        elif s.isdigit():
+            return bool(int(s))
+        else:
+            raise ValueError('must be integer or boolean')
+    # parse environtment variables
+    for k, v in os.environ.items():
+        if not k.startswith('PWNLIB_'):
+            continue
+        k = k[7:]
+        if   k == 'DEBUG':
+            if asbool(v):
+                context.log_level = 'debug'
+        elif k == 'SILENT':
+            if asbool(v):
+                context.log_level = 'ERROR'
+        elif k == 'NOTERM':
+            if asbool(v):
+                term_mode = False
+        elif k == 'LOG_LEVEL':
+            context.log_level = v
+        elif isident(k):
+            args[k] = v
+    # parse command line
     for arg in sys.argv[:]:
         if   arg == 'DEBUG':
             sys.argv.remove(arg)
             context.log_level = 'debug'
-        elif arg == 'NOINFO':
+        elif arg == 'SILENT':
             sys.argv.remove(arg)
-            context.log_level = 'silent'
+            context.log_level = 'ERROR'
+        elif arg == 'NOTERM':
+            term_mode = False
         elif arg.find('=') > 0:
             k, v = arg.split('=', 1)
             if not isident(k):
                 continue
             sys.argv.remove(arg)
             args[k] = v
+    # put the terminal in rawmode unless NOTERM was specified
+    if term_mode:
+        term.init()
+    else:
+        term.text.when = 'never'
 closure()
 del closure
+
+log = logging.getLogger('pwnlib.exploit')

pwn/toplevel.py

@@ -1,24 +1,36 @@
 # Get all the modules from pwnlib
-from pwnlib                 import *
+from pwnlib                  import *
 
 # Promote functions from these modules to toplevel
-from pwnlib.asm             import *
-from pwnlib.ui              import *
-from pwnlib.util            import crc, proc
-from pwnlib.util.cyclic     import *
-from pwnlib.util.fiddling   import *
-from pwnlib.util.hashes     import *
-from pwnlib.util.lists      import *
-from pwnlib.util.misc       import *
-from pwnlib.util.packing    import *
-from pwnlib.util.proc       import pidof
-from pwnlib.util.splash     import *
-from pwnlib.tubes.remote    import *
-from pwnlib.tubes.listen    import *
-from pwnlib.tubes.process   import *
-from pwnlib.memleak         import MemLeak
-from pwnlib.elf             import ELF, load
-from pwnlib.dynelf          import DynELF
+from pwnlib.asm              import asm, disasm, cpp
+from pwnlib.context          import context, Thread
+from pwnlib.dynelf           import DynELF
+from pwnlib.elf              import ELF, load
+from pwnlib.exception        import PwnlibException
+from pwnlib.memleak          import MemLeak
+from pwnlib.replacements     import *
+from pwnlib.rop              import ROP
+from pwnlib.timeout          import Timeout
+from pwnlib.tubes.listen     import listen
+from pwnlib.tubes.process    import process
+from pwnlib.tubes.remote     import remote
+from pwnlib.tubes.serialtube import serialtube
+from pwnlib.tubes.ssh        import ssh
+from pwnlib.tubes.tube       import tube
+from pwnlib.ui               import *
+from pwnlib.util             import crc, net, proc, iters, safeeval
+from pwnlib.util.cyclic      import *
+from pwnlib.util.fiddling    import *
+from pwnlib.util.hashes      import *
+from pwnlib.util.lists       import *
+from pwnlib.util.misc        import *
+from pwnlib.util.packing     import *
+from pwnlib.util.proc        import pidof
+from pwnlib.util.splash      import *
+from pwnlib.util.web         import *
 
 # Promote these modules, so that "from pwn import *" will let you access them
-import pwn, pwnlib, re, time, sys, os, string, threading, subprocess, struct
+import \
+    collections   , operator      , os            , pwnlib        , \
+    re            , string        , struct        , subprocess    , \
+    sys           , threading     , time          , tempfile

pwnlib/__init__.py

@@ -1,10 +1,19 @@
 __all__ = [
-    'asm', 'constants', 'context', 'dynelf',
-    'elf', 'exception', 'log', 'memleak',
-    'shellcraft', 'term', 'tubes', 'ui',
-    'util'
+    'atexception' , 'atexit'      , 'asm'         , 'constants'   ,
+    'context'     , 'dynelf'      , 'elf'         , 'exception'   ,
+    'gdb'                         , 'log'         , 'memleak'     ,
+    'replacements', 'rop'         , 'shellcraft'  , 'term'        ,
+    'tubes'       , 'ui'          , 'useragents'  , 'util'
 ]
-from . import asm, constants, context, dynelf
-from . import elf, exception, log, memleak
-from . import shellcraft, term, tubes, ui
-from . import util
+
+
+from . import \
+    atexception   , atexit        , asm           , constants     , \
+                    dynelf        , elf           , exception     , \
+    gdb                           , log           , memleak       , \
+    replacements  , rop           , shellcraft    , term          , \
+    tubes         , ui            , useragents    , util,           \
+    pep237
+
+# from .context import context
+from .version import __version__

pwnlib/asm.py

@@ -1,95 +1,208 @@
-from . import context, log
-import tempfile, subprocess, shutil, tempfile, errno
-from os import path
+# -*- coding: utf-8 -*-
+r"""
+Utilities for assembling and disassembling code.
 
-__all__ = ['asm', 'cpp', 'disasm']
+Architecture Selection
+------------------------
+
+    Architecture, endianness, and word size are selected by using :mod:`pwnlib.context`.
+
+    Any parameters which can be specified to ``context`` can also be specified as
+    keyword arguments to either :func:`asm` or :func:`disasm`.
+
+Assembly
+------------------------
+
+    To assemble code, simply invoke :func:`asm` on the code to assemble.
+
+        >>> asm('mov eax, 0')
+        '\xb8\x00\x00\x00\x00'
+
+    Additionally, you can use constants as defined in the :mod:`pwnlib.constants`
+    module.
+
+        >>> asm('mov eax, SYS_execve')
+        '\xb8\x0b\x00\x00\x00'
+
+    Finally, :func:`asm` is used to assemble shellcode provided by ``pwntools``
+    in the :mod:`shellcraft` module.
+
+        >>> asm(shellcraft.sh())
+        'jhh///sh/bin\x89\xe31\xc9j\x0bX\x99\xcd\x80'
+
+Disassembly
+------------------------
+
+    To disassemble code, simply invoke :func:`disasm` on the bytes to disassemble.
+
+    >>> disasm('\xb8\x0b\x00\x00\x00')
+    '   0:   b8 0b 00 00 00          mov    eax,0xb'
+
+"""
+import tempfile, subprocess, shutil, tempfile, errno, logging, platform
+from os import path, environ
+from glob import glob
+from . import log
+from .context import context
+
+log = logging.getLogger(__name__)
+
+__all__ = ['asm', 'cpp', 'disasm', 'which_binutils']
 
 _basedir = path.split(__file__)[0]
-_bindir  = path.join(_basedir, 'data', 'binutils')
 _incdir  = path.join(_basedir, 'data', 'includes')
 
-def _assembler(arch):
+def which_binutils(util, **kwargs):
+    """
+    Finds a binutils in the PATH somewhere.
+    Expects that the utility is prefixed with the architecture name.
+
+    Examples:
+
+        >>> import platform
+        >>> which_binutils('as', arch=platform.machine())
+        '.../bin/as'
+        >>> which_binutils('as', arch='arm') #doctest: +ELLIPSIS
+        '.../bin/arm-...-as'
+        >>> which_binutils('as', arch='powerpc') #doctest: +ELLIPSIS
+        '.../bin/powerpc...-as'
+        >>> which_binutils('as', arch='msp430') #doctest: +SKIP
+        ...
+        Traceback (most recent call last):
+        ...
+        Exception: Could not find 'as' installed for ContextType(arch = 'msp430')
+    """
+    with context.local(**kwargs):
+        arch = context.arch
+        bits = context.bits
+
+        # Fix up pwntools vs Debian triplet naming, and account
+        # for 'thumb' being its own pwntools architecture.
+        arches = [arch] + {
+            'thumb':  ['arm',    'aarch64'],
+            'i386':   ['x86_64', 'amd64'],
+            'amd64':  ['x86_64', 'i386'],
+        }.get(arch, [])
+
+        # If one of the candidate architectures matches the native
+        # architecture, use that as a last resort.
+        machine = platform.machine()
+        try:
+            with context.local(arch = machine):
+                if context.arch in arches:
+                    arches.append(None)
+        except AttributeError:
+            log.warn_once("Your local binutils won't be used because architecture %r is not supported." % machine)
+
+        for arch in arches:
+            # hack for homebrew-installed binutils on mac
+            for gutil in ['g'+util, util]:
+                # e.g. objdump
+                if arch is None: pattern = gutil
+
+                # e.g. aarch64-linux-gnu-objdump
+                else:       pattern = '%s*linux*-%s' % (arch,gutil)
+
+                for dir in environ['PATH'].split(':'):
+                    res = glob(path.join(dir, pattern))
+                    if res:
+                        return res[0]
+
+        locals()['context'] = context
+        log.warning("""
+Could not find %(util)r installed for %(context)s
+Try installing binutils for this architecture:
+    https://pwntools.readthedocs.org/en/latest/install/binutils.html
+""".strip() % locals())
+        raise Exception('Could not find %(util)r installed for %(context)s' % locals())
+
+def _assembler():
+    gas = which_binutils('as')
+
+    E = {
+        'big':    '-EB',
+        'little': '-EL'
+    }[context.endianness]
+
     assemblers = {
-        'i386'   : ['nasm', '-Ox'],
-        'amd64'  : ['nasm', '-Ox'],
-        'thumb'  : [path.join(_bindir, 'arm-as')],
-        'mips'   : [path.join(_bindir, 'mips-as' ), '--EB'],
-        'mipsel' : [path.join(_bindir, 'mips-as' ), '--EL'],
+        'i386'   : [gas, '--32'],
+        'amd64'  : [gas, '--64'],
+
+        # Most architectures accept -EL or -EB
+        'thumb'  : [gas, '-mthumb', E],
+        'arm'    : [gas, E],
+        'aarch64': [gas, E],
+        'mips'   : [gas, E],
+        'mips64' : [gas, E],
+        'sparc':   [gas, E],
+        'sparc64': [gas, E],
+
+        # Powerpc wants -mbig or -mlittle
+        'powerpc':   [gas, '-m%s' % context.endianness],
+        'powerpc64': [gas, '-m%s' % context.endianness],
+
+        # ia64 only accepts -mbe or -mle
+        'ia64':    [gas, '-m%ce' % context.endianness[0]]
     }
 
-    if arch in assemblers:
-        return assemblers[arch]
-    else:
-        return [path.join(_bindir, arch + '-as')]
+    return assemblers.get(context.arch, [gas])
 
+def _objcopy():
+    return [which_binutils('objcopy')]
 
-def _objcopy(arch):
-    if arch in ['i386', 'amd64']:
-        return 'objcopy'
-    else:
-        return path.join(_bindir, 'objcopy')
+def _objdump():
+    path = [which_binutils('objdump')]
 
+    if context.arch in ('i386', 'amd64'):
+        path += ['-Mintel']
 
-def _objdump(arch):
-    if arch in ['i386', 'amd64']:
-        return ['objdump', '-Mintel']
-    else:
-        return [path.join(_bindir, 'objdump')]
+    return path
 
+def _include_header():
+    os   = context.os
+    arch = context.arch
 
-def _include_header(arch, os):
     if os == 'freebsd':
-        return '#include <freebsd.h>\n'
-    elif os == 'linux' and arch:
-        return '#include <linux/%s.h>\n' % arch
-    else:
-        return ''
+        include = 'freebsd.h'
+    elif os == 'linux':
+        include = 'linux/%s.h' % arch
+
+    if not include or not path.exists(path.join(_incdir, include)):
+        log.warn_once("Could not find system include headers for %s-%s" % (arch,os))
+        return '\n'
+
+    return '#include <%s>\n' % include
 
 
-def _arch_header(arch):
+def _arch_header():
+    prefix  = ['.section .shellcode,"ax"']
     headers = {
-        'i386'  : ['bits 32'],
-        'amd64' : ['bits 64'],
-        'arm'   : [
-            '.section .shellcode,"ax"',
-            '.syntax unified',
-            '.arch armv7-a',
-            '.arm'
-        ],
-        'thumb' : [
-            '.section .shellcode,"ax"',
-            '.syntax unified',
-            '.arch armv7-a',
-            '.thumb'
-        ],
-        'mips'  : [
-            '.section .shellcode,"ax"',
-            '.set mips2',
-            '.set noreorder',
-        ],
-        'mipsel'  : [
-            '.section .shellcode,"ax"',
-            '.set mips2',
-            '.set noreorder',
-        ],
+        'i386'  :  ['.intel_syntax noprefix'],
+        'amd64' :  ['.intel_syntax noprefix'],
+        'arm'   : ['.syntax unified',
+                   '.arch armv7-a',
+                   '.arm'],
+        'thumb' : ['.syntax unified',
+                   '.arch armv7-a',
+                   '.thumb'],
+        'mips'  : ['.set mips2',
+                   '.set noreorder'],
     }
 
-    if arch in headers:
-        return '\n'.join(headers[arch]) + '\n'
-    else:
-        return ''
+    return '\n'.join(prefix + headers.get(context.arch, [])) + '\n'
+
+def _bfdname():
+    arch = context.arch
 
-def _bfdname(arch):
     bfdnames = {
         'i386'    : 'elf32-i386',
         'amd64'   : 'elf64-x86-64',
         'arm'     : 'elf32-littlearm',
         'thumb'   : 'elf32-littlearm',
-        'mips'    : 'elf32-bigmips',
-        'mipsel'  : 'elf32-littlemips',
+        'mips'    : 'elf32-%smips' % context.endianness,
         'alpha'   : 'elf64-alpha',
         'cris'    : 'elf32-cris',
-        'ia64'    : 'elf64-ia64-little',
+        'ia64'    : 'elf64-ia64-%s' % context.endianness,
         'm68k'    : 'elf32-m68k',
         'powerpc' : 'elf32-powerpc',
         'vax'     : 'elf32-vax',
@@ -98,21 +211,25 @@ def _bfdname(arch):
     if arch in bfdnames:
         return bfdnames[arch]
     else:
-        log.error("Cannot find bfd name for architecture %r" % arch)
+        raise Exception("Cannot find bfd name for architecture %r" % arch)
 
 
-def _bfdarch(arch):
-    if arch == 'amd64':
-        return 'i386:x86-64'
-    elif arch == 'thumb':
-        return 'arm'
-    elif arch == 'ia64':
-        return 'ia64-elf64'
-    else:
-        return arch
+def _bfdarch():
+    arch = context.arch
+    convert = {
+    'i386': 'i386',
+    'amd64': 'i386:x86-64',
+    'thumb': 'arm',
+    'ia64': 'ia64-elf64'
+    }
 
+    if arch in convert:
+        return convert[arch]
+
+    return arch
 
 def _run(cmd, stdin = None):
+    log.debug(subprocess.list2cmdline(cmd))
     try:
         proc = subprocess.Popen(
             cmd,
@@ -124,60 +241,67 @@ def _run(cmd, stdin = None):
         exitcode = proc.wait()
     except OSError as e:
         if e.errno == errno.ENOENT:
-            log.error('Could not run %r the program' % cmd[0])
+            log.exception('Could not run %r the program' % cmd[0])
         else:
             raise
 
     if (exitcode, stderr) != (0, ''):
-        msg = 'There was an error running %s:\n' % repr(cmd[0])
+        msg = 'There was an error running %s:\n' % repr(cmd)
         if exitcode != 0:
             msg += 'It had the exitcode %d.\n' % exitcode
         if stderr != '':
             msg += 'It had this on stdout:\n%s\n' % stderr
         log.error(msg)
 
-
     return stdout
 
-def cpp(shellcode, arch = None, os = None):
-    """cpp(shellcode, arch = None, os = None) -> str
+def cpp(shellcode, **kwargs):
+    r"""cpp(shellcode, ...) -> str
 
     Runs CPP over the given shellcode.
 
     The output will always contain exactly one newline at the end.
 
+    Arguments:
+        shellcode(str): Shellcode to preprocess
+
+    Kwargs:
+        Any arguments/properties that can be set on ``context``
+
     Examples:
-      >>> cpp("testing: SYS_setresuid")
-      'testing: SYS_setresuid\\n'
-      >>> cpp("mov al, SYS_setresuid", arch = "i386", os = "linux")
-      'mov al, 164\\n'
-      >>> cpp("weee SYS_setresuid", arch = "arm", os = "linux")
-      'weee (0x900000+164)\\n'
-      >>> cpp("SYS_setresuid", arch = "thumb", os = "linux")
-      '(0+164)\\n'
-      >>> cpp("SYS_setresuid", os = "freebsd")
-      '311\\n'
+
+        .. doctest::
+
+            >>> cpp("mov al, SYS_setresuid", arch = "i386", os = "linux")
+            'mov al, 164\n'
+            >>> cpp("weee SYS_setresuid", arch = "arm", os = "linux")
+            'weee (0x900000+164)\n'
+            >>> cpp("SYS_setresuid", arch = "thumb", os = "linux")
+            '(0+164)\n'
+            >>> cpp("SYS_setresuid", os = "freebsd")
+            '311\n'
     """
 
-    arch = arch or context.arch
-    os   = os   or context.os
-    code = _include_header(arch, os) + shellcode
-    cmd  = [
-        'cpp',
-        '-C',
-        '-nostdinc',
-        '-undef',
-        '-P',
-        '-I' + _incdir,
-        '/dev/stdin'
-    ]
-    return _run(cmd, code).strip('\n').rstrip() + '\n'
+    with context.local(**kwargs):
+        arch = context.arch
+        os   = context.os
+        code = _include_header() + shellcode
+        cmd  = [
+            'cpp',
+            '-C',
+            '-nostdinc',
+            '-undef',
+            '-P',
+            '-I' + _incdir,
+            '/dev/stdin'
+        ]
+        return _run(cmd, code).strip('\n').rstrip() + '\n'
 
 
-def asm(shellcode, arch = None, os = None):
-    """asm(code, arch = None, os = None) -> str
+def asm(shellcode, vma = 0, **kwargs):
+    r"""asm(code, vma = 0, ...) -> str
 
-    Runs CPP over a given shellcode and then assembles it into bytes.
+    Runs :func:`cpp` over a given shellcode and then assembles it into bytes.
 
     To see which architectures or operating systems are supported,
     look in :mod:`pwnlib.contex`.
@@ -187,63 +311,67 @@ def asm(shellcode, arch = None, os = None):
 
     Args:
       shellcode(str): Assembler code to assemble.
-      arch: A supported architecture or None. In case of None,
-            :data:`pwnlib.context.arch` will be used.
-      os: A supported operating system or None. In case of None,
-          :data:`pwnlib.context.os` will be used.
+      vma(int):       Virtual memory address of the beginning of assembly
+
+    Kwargs:
+        Any arguments/properties that can be set on ``context``
 
     Examples:
-      >>> asm("mov eax, SYS_select", arch = 'i386', os = 'freebsd')
-      '\\xb8]\\x00\\x00\\x00'
-      >>> asm("mov rax, SYS_select", arch = 'amd64', os = 'linux')
-      '\\xb8\\x17\\x00\\x00\\x00'
-      >>> asm("ldr r0, =SYS_select", arch = 'arm', os = 'linux')
-      '\\x04\\x00\\x1f\\xe5R\\x00\\x90\\x00'
+
+        .. doctest::
+
+            >>> asm("mov eax, SYS_select", arch = 'i386', os = 'freebsd')
+            '\xb8]\x00\x00\x00'
+            >>> asm("mov eax, SYS_select", arch = 'amd64', os = 'linux')
+            '\xb8\x17\x00\x00\x00'
+            >>> asm("mov rax, SYS_select", arch = 'amd64', os = 'linux')
+            'H\xc7\xc0\x17\x00\x00\x00'
+            >>> asm("ldr r0, =SYS_select", arch = 'arm', os = 'linux', bits=32)
+            '\x04\x00\x1f\xe5R\x00\x90\x00'
     """
+    result = ''
 
-    arch = arch or context.arch
-    os   = os   or context.os
-    if not arch:
-        raise ValueError(
-            "asm() needs to get 'arch' through an argument or the context"
-        )
+    with context.local(**kwargs):
+        assembler = _assembler()
+        objcopy   = _objcopy() + ['-j', '.shellcode', '-Obinary']
+        code      = '.org %#x\n' % vma
+        code      += _arch_header()
+        code      += cpp(shellcode)
 
-    tmpdir  = tempfile.mkdtemp(prefix = 'pwn-asm-')
-    def tmp(name):
-        return path.join(tmpdir, name)
+        log.debug('Assembling\n%s' % code)
 
-    assembler = _assembler(arch)
-    objcopy   = [_objcopy(arch), '-j.shellcode', '-Obinary']
-    code      = _arch_header(arch) + cpp(shellcode, arch, os)
+        tmpdir    = tempfile.mkdtemp(prefix = 'pwn-asm-')
+        step1     = path.join(tmpdir, 'step1')
+        step2     = path.join(tmpdir, 'step2')
+        step3     = path.join(tmpdir, 'step3')
 
-    try:
-        with open(tmp('step1'), 'w') as fd:
-            fd.write(code)
+        try:
+            with open(step1, 'w') as fd:
+                fd.write(code)
 
-        _run(assembler + ['-o', tmp('step2'), tmp('step1')])
+            _run(assembler + ['-o', step2, step1])
 
-        if arch in ['i386', 'amd64']:
-            with open(tmp('step2')) as fd:
-                return fd.read()
+            if file(step2,'rb').read(4) == '\x7fELF':
+                # Sanity check for seeing if the output has relocations
+                relocs = subprocess.check_output(
+                    [which_binutils('readelf'), '-r', step2]
+                ).strip()
+                if len(relocs.split('\n')) > 1:
+                    log.error('Shellcode contains relocations:\n%s' % relocs)
 
-        # Sanity check for seeing if the output has relocations
-        relocs = subprocess.check_output(
-            ['readelf', '-r', tmp('step2')]
-        ).strip()
-        if len(relocs.split('\n')) > 1:
-            raise Exception(
-                'There were relocations in the shellcode:\n\n%s' % relocs
-            )
+            _run(objcopy + [step2, step3])
 
-        _run(objcopy + [tmp('step2'), tmp('step3')])
+            with open(step3) as fd:
+                result = fd.read()
 
-        with open(tmp('step3')) as fd:
-            return fd.read()
-    finally:
-        shutil.rmtree(tmpdir)
+        except:
+            log.exception("An error occurred while assembling:\n%s" % code)
+        else:
+            shutil.rmtree(tmpdir)
+            return result
 
-def disasm(data, arch = None):
-    """disasm(data, arch = None) -> str
+def disasm(data, vma = 0, **kwargs):
+    """disasm(data, ...) -> str
 
     Disassembles a bytestring into human readable assembler.
 
@@ -255,60 +383,70 @@ def disasm(data, arch = None):
 
     Args:
       data(str): Bytestring to disassemble.
-      arch: A supported architecture or None. In case of None,
-            :data:`pwnlib.context.arch` will be used.
+      vma(int): Passed through to the --adjust-vma argument of objdump
+
+    Kwargs:
+      Any arguments/properties that can be set on ``context``
 
     Examples:
-      >>> print disasm('b85d000000'.decode('hex'), arch = 'i386')
-         0:   b8 5d 00 00 00          mov    eax,0x5d
-      >>> print disasm('b817000000'.decode('hex'), arch = 'amd64')
-         0:   b8 17 00 00 00          mov    eax,0x17
-      >>> print disasm('04001fe552009000'.decode('hex'), arch = 'arm')
-         0:   e51f0004        ldr     r0, [pc, #-4]   ; 0x4
-         4:   00900052        addseq  r0, r0, r2, asr r0
-      >>> print disasm('4ff00500'.decode('hex'), arch = 'thumb')
-         0:   f04f 0005       mov.w   r0, #5
+
+        .. doctest::
+
+          >>> print disasm('b85d000000'.decode('hex'), arch = 'i386')
+             0:   b8 5d 00 00 00          mov    eax,0x5d
+          >>> print disasm('b817000000'.decode('hex'), arch = 'amd64')
+             0:   b8 17 00 00 00          mov    eax,0x17
+          >>> print disasm('48c7c017000000'.decode('hex'), arch = 'amd64')
+             0:   48 c7 c0 17 00 00 00    mov    rax,0x17
+          >>> print disasm('04001fe552009000'.decode('hex'), arch = 'arm')
+             0:   e51f0004        ldr     r0, [pc, #-4]   ; 0x4
+             4:   00900052        addseq  r0, r0, r2, asr r0
+          >>> print disasm('4ff00500'.decode('hex'), arch = 'thumb', bits=32)
+             0:   f04f 0005       mov.w   r0, #5
     """
+    result = ''
 
-    arch = arch or context.arch
-    if not arch:
-        raise ValueError(
-            "asm() needs to get 'arch' through an argument or the context"
-        )
+    with context.local(**kwargs):
+        arch   = context.arch
+        os     = context.os
 
-    tmpdir = tempfile.mkdtemp(prefix = 'pwn-disasm-')
-    def tmp(name):
-        return path.join(tmpdir, name)
+        tmpdir = tempfile.mkdtemp(prefix = 'pwn-disasm-')
+        step1  = path.join(tmpdir, 'step1')
+        step2  = path.join(tmpdir, 'step2')
 
-    bfdarch = _bfdarch(arch)
-    bfdname = _bfdname(arch)
-    objdump = _objdump(arch) + ['-d']
-    objcopy = [
-        _objcopy(arch),
-        '-I', 'binary',
-        '-O', bfdname,
-        '-B', bfdarch,
-        '--set-section-flags', '.data=code',
-        '--rename-section', '.data=.text',
-    ]
+        bfdarch = _bfdarch()
+        bfdname = _bfdname()
+        objdump = _objdump() + ['-d', '--adjust-vma', str(vma), '-b', bfdname]
+        objcopy = _objcopy() + [
+            '-I', 'binary',
+            '-O', bfdname,
+            '-B', bfdarch,
+            '--set-section-flags', '.data=code',
+            '--rename-section', '.data=.text',
+        ]
 
-    if arch == 'thumb':
-        objcopy += ['--prefix-symbol=$t.']
-    else:
-        objcopy += ['-w', '-N', '*']
-
-    try:
-        with open(tmp('step1'), 'w') as fd:
-            fd.write(data)
-        _run(objcopy + [tmp('step1'), tmp('step2')])
-
-        output0 = subprocess.check_output(objdump + [tmp('step2')])
-        output1 = output0.split('<.text>:\n')
-        if len(output1) != 2:
-            raise IOError(
-                'Something went wrong with objdump:\n\n%s' % output0
-            )
+        if arch == 'thumb':
+            objcopy += ['--prefix-symbol=$t.']
         else:
-            return output1[1].strip('\n').rstrip().expandtabs()
-    finally:
-        shutil.rmtree(tmpdir)
+            objcopy += ['-w', '-N', '*']
+
+        try:
+
+            with open(step1, 'w') as fd:
+                fd.write(data)
+
+            res = _run(objcopy + [step1, step2])
+
+            output0 = subprocess.check_output(objdump + [step2])
+            output1 = output0.split('<.text>:\n')
+
+            if len(output1) != 2:
+                log.error('Could not find .text in objdump output:\n%s' % output0)
+
+            result = output1[1].strip('\n').rstrip().expandtabs()
+        except:
+            log.exception("An error occurred while disassembling:\n%s" % data)
+        else:
+            shutil.rmtree(tmpdir)
+            return result
+

pwnlib/atexception.py

@@ -0,0 +1,101 @@
+"""
+Analogous to atexit, this module allows the programmer to register functions to
+be run if an unhandled exception occurs.
+"""
+
+import sys, traceback, threading
+from .context import context
+
+__all__ = ['register', 'unregister']
+
+_lock = threading.Lock()
+_ident = 0
+_handlers = {}
+
+def register(func, *args, **kwargs):
+    """register(func, *args, **kwargs)
+
+    Registers a function to be called when an unhandled exception occurs.  The
+    function will be called with positional arguments `args` and keyword
+    arguments `kwargs`, i.e. ``func(*args, **kwargs)``.  The current `context`
+    is recorded and will be the one used when the handler is run.
+
+    E.g. to suppress logging output from an exception-handler one could write::
+
+      with context.local(log_level = 'error'):
+        atexception.register(handler)
+
+    An identifier is returned which can be used to unregister the
+    exception-handler.
+
+    This function can be used as a decorator::
+
+      @atexception.register
+      def handler():
+        ...
+
+    Notice however that this will bind ``handler`` to the identifier and not the
+    actual exception-handler.  The exception-handler can then be unregistered
+    with::
+
+      atexception.unregister(handler)
+
+    This function is thread safe.
+
+    """
+    global _ident
+    with _lock:
+        ident = _ident
+        _ident += 1
+    _handlers[ident] = (func, args, kwargs, vars(context))
+    return ident
+
+def unregister(func):
+    """unregister(func)
+
+    Remove `func` from the collection of registered functions.  If `func` isn't
+    registered this is a no-op.
+    """
+    if func in _handlers:
+        del _handlers[func]
+
+def _run_handlers():
+    """_run_handlers()
+
+    Run registered handlers.  They run in the reverse order of which they were
+    registered.
+
+    If a handler raises an exception, it will be printed but nothing else
+    happens, i.e. other handlers will be run.
+    """
+    context.clear()
+    for _ident, (func, args, kwargs, ctx) in \
+        sorted(_handlers.items(), reverse = True):
+        try:
+            with context.local(**ctx):
+                func(*args, **kwargs)
+        except SystemExit:
+            pass
+        except:
+            # extract the current exception and rewind the traceback to where it
+            # originated
+            typ, val, tb = sys.exc_info()
+            traceback.print_exception(typ, val, tb.tb_next)
+
+# we rely on the existing excepthook to print exceptions
+if hasattr(sys, 'excepthook'):
+    _oldhook = sys.excepthook
+else:
+    _oldhook = None
+
+def _newhook(typ, val, tb):
+    """_newhook(typ, val, tb)
+
+    Our excepthook replacement.  First the original hook is called to print the
+    exception, then each handler is called.
+    """
+    if _oldhook:
+        _oldhook(typ, val, tb)
+    _run_handlers()
+
+sys.excepthook = _newhook

pwnlib/atexit.py

@@ -0,0 +1,93 @@
+"""
+Replacement for the Python standard library's atexit.py.
+
+Whereas the standard :mod:`atexit` module only defines :func:`atexit.register`,
+this replacement module also defines :func:`unregister`.
+
+This module also fixes a the issue that exceptions raised by an exit handler is
+printed twice when the standard :mod:`atexit` is used.
+"""
+
+import sys, traceback, threading
+from .context import context
+
+__all__ = ['register', 'unregister']
+
+_lock = threading.Lock()
+_ident = 0
+_handlers = {}
+
+def register(func, *args, **kwargs):
+    """register(func, *args, **kwargs)
+
+    Registers a function to be called on program termination.  The function will
+    be called with positional arguments `args` and keyword arguments `kwargs`,
+    i.e. ``func(*args, **kwargs)``.  The current `context` is recorded and will
+    be the one used when the handler is run.
+
+    E.g. to suppress logging output from an exit-handler one could write::
+
+      with context.local(log_level = 'error'):
+        atexit.register(handler)
+
+    An identifier is returned which can be used to unregister the exit-handler.
+
+    This function can be used as a decorator::
+
+      @atexit.register
+      def handler():
+        ...
+
+    Notice however that this will bind ``handler`` to the identifier and not the
+    actual exit-handler.  The exit-handler can then be unregistered with::
+
+      atexit.unregister(handler)
+
+    This function is thread safe.
+
+    """
+    global _ident
+    with _lock:
+        ident = _ident
+        _ident += 1
+    _handlers[ident] = (func, args, kwargs, vars(context))
+    return ident
+
+def unregister(ident):
+    """unregister(ident)
+
+    Remove the exit-handler identified by `ident` from the list of registered
+    handlers.  If `ident` isn't registered this is a no-op.
+    """
+    if ident in _handlers:
+        del _handlers[ident]
+
+def _run_handlers():
+    """_run_handlers()
+
+    Run registered exit-handlers.  They run in the reverse order of which they
+    were registered.
+
+    If a handler raises an exception, it will be printed but nothing else
+    happens, i.e. other handlers will be run and `sys.excepthook` will not be
+    called for that reason.
+    """
+    context.clear()
+    for _ident, (func, args, kwargs, ctx) in \
+        sorted(_handlers.items(), reverse = True):
+        try:
+            with context.local(**ctx):
+                func(*args, **kwargs)
+        except SystemExit:
+            pass
+        except:
+            # extract the current exception and rewind the traceback to where it
+            # originated
+            typ, val, tb = sys.exc_info()
+            traceback.print_exception(typ, val, tb.tb_next)
+
+# if there's already an exitfunc registered be sure to run that too
+if hasattr(sys, "exitfunc"):
+    register(sys.exitfunc)
+
+sys.exitfunc = _run_handlers

pwnlib/commandline/asm.py

@@ -2,7 +2,7 @@
 
 import argparse, sys
 from pwnlib.asm           import asm
-from pwnlib.context       import __possible__ as contexts
+from pwnlib.context       import context
 from pwnlib.util.fiddling import enhex
 
 parser = argparse.ArgumentParser(
@@ -33,27 +33,30 @@ parser.add_argument(
 parser.add_argument(
     '-c', '--context',
     metavar = '<opt>',
-    choices = contexts['os'] + contexts['arch'],
+    choices = context.oses + list(context.architectures),
     default = ['i386','linux'],
     action = 'append',
     help = 'The os/architecture the shellcode will run in (default: linux/i386), choose from: %s' % \
-      ', '.join(contexts['os'] + contexts['arch'])
+      ', '.join(sorted(context.oses + list(context.architectures)))
 )
 
-args   = parser.parse_args()
-tty    = args.output.isatty()
+def main():
+    args   = parser.parse_args()
+    tty    = args.output.isatty()
 
-for arch in args.context[::-1]:
-    if arch in contexts['arch']: break
-for os in args.context[::-1]:
-    if os in contexts['os']: break
+    for arch in args.context[::-1]:
+        if arch in context.architectures: break
+    for os in args.context[::-1]:
+        if os in context.oses: break
 
-data   = '\n'.join(args.lines) or sys.stdin.read()
-output = asm(data.replace(';', '\n'), arch = arch, os = os)
-fmt    = args.format or ('hex' if tty else 'raw')
-formatters = {'r':str, 'h':enhex, 's':repr}
+    data   = '\n'.join(args.lines) or sys.stdin.read()
+    output = asm(data.replace(';', '\n'), arch = arch, os = os)
+    fmt    = args.format or ('hex' if tty else 'raw')
+    formatters = {'r':str, 'h':enhex, 's':repr}
 
-args.output.write(formatters[fmt[0]](output))
+    args.output.write(formatters[fmt[0]](output))
 
-if tty and fmt is not 'raw':
-    args.output.write('\n')
+    if tty and fmt is not 'raw':
+        args.output.write('\n')
+
+if __name__ == '__main__': main()

pwnlib/commandline/constgrep.py

@@ -0,0 +1,140 @@
+#!/usr/bin/env python2
+
+import argparse, os, re
+from pwnlib import asm, constants
+from pwnlib.util import safeeval
+# from pwnlib.context import __possible__ as contexts
+from pwnlib.context import context
+
+p = argparse.ArgumentParser(
+    description = "Looking up constants from header files.\n\nExample: constgrep -c freebsd -m  ^PROT_ '3 + 4'",
+    formatter_class = argparse.RawDescriptionHelpFormatter,
+)
+
+group = p.add_mutually_exclusive_group()
+group.add_argument(
+    '-e', '--exact',
+    metavar = '<constant name>',
+    # nargs = 1,
+    default = None,
+    help = 'Do an exact match for a constant instead of searching for a regex',
+)
+group.add_argument(
+    'regex',
+    nargs = '?',
+    default = '',
+    help = 'The regex matching constant you want to find',
+)
+
+p.add_argument(
+    'constant',
+    nargs = '?',
+    default = None,
+    help = 'The constant to find',
+)
+
+p.add_argument(
+    '-i', '--case-insensitive',
+    action = 'store_true',
+    help = 'Search case insensitive',
+)
+
+p.add_argument(
+    '-m', '--mask-mode',
+    action = 'store_true',
+    help = 'Instead of searching for a specific constant value, search for values not containing strictly less bits that the given value.',
+)
+
+p.add_argument(
+    '-c', '--context',
+    metavar = '<opt>',
+    choices = context.oses + list(context.architectures),
+    default = ['i386','linux'],
+    action = 'append',
+    help = 'The os/architecture to find constants for (default: linux/i386), choose from: %s' % \
+    ', '.join(sorted(context.oses + context.architectures.keys()))
+)
+
+def main():
+    args = p.parse_args()
+
+    # Find the architecture and os from the list of contexts
+    for arch in args.context[::-1]:
+        if arch in context.architectures: break
+    for os in args.context[::-1]:
+        if os in context.oses: break
+
+    if args.exact:
+        # This is the simple case
+        print asm.cpp(args.exact, os = os, arch = arch).strip()
+    else:
+        # New we search in the right module.
+        # But first: We find the right module
+        if os == 'freebsd':
+            mod = constants.freebsd
+        else:
+            mod = getattr(getattr(constants, os), arch)
+
+        # Compile the given regex, for optimized lookup
+        if args.case_insensitive:
+            matcher = re.compile(args.regex, re.IGNORECASE)
+        else:
+            matcher = re.compile(args.regex)
+
+        # Evaluate the given constant
+        if args.constant:
+            constant = safeeval.expr(args.constant)
+        else:
+            constant = None
+
+        # The found matching constants and the length of the longest string
+        out    = []
+        maxlen = 0
+
+        for k in dir(mod):
+            # No python stuff
+            if k.endswith('__') and k.startswith('__'):
+                continue
+
+            # Run the regex
+            if not matcher.search(k):
+                continue
+
+            # Check the constant
+            if constant != None:
+                val = getattr(mod, k)
+                if args.mask_mode:
+                    if constant & val != val:
+                        continue
+                else:
+                    if constant != val:
+                        continue
+
+            # Append it
+            out.append((getattr(mod, k), k))
+            maxlen = max(len(k), maxlen)
+
+        # Output all matching constants
+        for _, k in sorted(out):
+            print '#define %s %s' % (k.ljust(maxlen), asm.cpp(k, os = os, arch = arch).strip())
+
+        # If we are in match_mode, then try to find a combination of
+        # constants that yield the exact given value
+        # We do not want to find combinations using the value 0.
+        if not (constant == None or constant == 0) and args.mask_mode:
+            mask = constant
+            good = []
+            out = [(v, k) for v, k in out if v != 0]
+
+            while mask and out:
+                cur = out.pop()
+                mask &= ~cur[0]
+                good.append(cur)
+
+                out = [(v, k) for v, k in out if mask & v == v]
+
+            if reduce(lambda x, cur: x | cur[0], good, 0) == constant:
+                print
+                print '(%s) == %s' % (' | '.join(k for v, k in good), args.constant)
+
+if __name__ == '__main__': main()

pwnlib/commandline/cyclic.py

@@ -0,0 +1,77 @@
+#!/usr/bin/env python2
+import argparse, string, sys
+from pwnlib.util import cyclic, packing
+from pwnlib import log
+
+parser = argparse.ArgumentParser(
+    description = "Cyclic pattern creator/finder"
+)
+
+parser.add_argument(
+    '-a', '--alphabet',
+    metavar = 'alphabet',
+    default = string.ascii_lowercase,
+    help = 'The alphabet to use in the cyclic pattern (defaults to all lower case letters)',
+)
+
+parser.add_argument(
+    '-n', '--length',
+    metavar = 'length',
+    default = 4,
+    type = int,
+    help = 'Size of the unique subsequences (defaults to 4).'
+)
+
+group = parser.add_mutually_exclusive_group(required = True)
+group.add_argument(
+    '-l', '-o', '--offset', '--lookup',
+    dest = 'lookup',
+    metavar = '<lookup value>',
+    help = 'Do a lookup instead printing the alphabet',
+)
+
+group.add_argument(
+    'count',
+    type = int,
+    nargs = '?',
+    help = 'Number of characters to print'
+)
+
+def main():
+    args = parser.parse_args()
+    alphabet = args.alphabet
+    subsize  = args.length
+
+    if args.lookup:
+        pat = args.lookup
+
+        if pat.startswith('0x'):
+            pat = packing.pack(int(pat[2:], 16), subsize*8, 'little', 'unsigned')
+        elif pat.isdigit():
+            pat = packing.pack(int(pat, 10), subsize*8, 'little', 'unsigned')
+
+        if len(pat) != 4:
+            log.fatal('Subpattern must be 4 bytes', 1)
+
+        if not all(c in alphabet for c in pat):
+            log.fatal('Pattern contains characters not present in the alphabet', 1)
+
+        offset = cyclic.cyclic_find(pat, alphabet, subsize)
+
+        if offset == -1:
+            log.fatal('Given pattern does not exist in cyclic pattern', 1)
+        else:
+            print offset
+    else:
+        want   = args.count
+        result = cyclic.cyclic(want, alphabet, subsize)
+        got    = len(result)
+        if got < want:
+            log.failure("Alphabet too small (max length = %i)" % got)
+
+        sys.stdout.write(result)
+
+        if sys.stdout.isatty():
+            sys.stdout.write('\n')
+
+if __name__ == '__main__': main()

pwnlib/commandline/disasm.py

@@ -0,0 +1,42 @@
+#!/usr/bin/env python2
+
+import argparse, sys
+from pwnlib import asm
+from pwnlib.context import context
+from string import whitespace, hexdigits
+
+parser = argparse.ArgumentParser(
+    description = 'Disassemble bytes into text format'
+)
+
+parser.add_argument(
+    'hex',
+    metavar = 'hex',
+    nargs = '*',
+    help = 'Hex-string to disasemble. If none are supplied, then it uses stdin in non-hex mode.'
+)
+
+parser.add_argument(
+    '-c', '--context',
+    metavar = '<opt>',
+    choices = context.architectures,
+    default = 'i386',
+    help = 'The architecture of the shellcode (default: i386), choose from:\n%s' % ', '.join(context.architectures)
+)
+
+def main():
+    args = parser.parse_args()
+
+    if len(args.hex) > 0:
+        dat = ''.join(args.hex)
+        dat = dat.translate(None, whitespace)
+        if not set(hexdigits) >= set(dat):
+            print "This is not a hex string"
+            exit(-1)
+        dat = dat.decode('hex')
+    else:
+        dat = sys.stdin.read()
+
+    print asm.disasm(dat, arch = args.context)
+
+if __name__ == '__main__': main()

pwnlib/commandline/elfdiff.py

@@ -0,0 +1,27 @@
+#!/usr/bin/env python2
+from argparse import ArgumentParser
+from subprocess import check_output, CalledProcessError
+from tempfile import NamedTemporaryFile
+
+def dump(x):
+    n = NamedTemporaryFile(delete=False)
+    o = check_output(['objdump','-d','-x','-s',x])
+    n.write(o)
+    n.flush()
+    return n.name
+
+def diff(a,b):
+    try: return check_output(['diff',a,b])
+    except CalledProcessError as e:
+        return e.output
+
+
+p = ArgumentParser()
+p.add_argument('a')
+p.add_argument('b')
+
+def main():
+    a = p.parse_args()
+    print diff(dump(a.a), dump(a.b))
+
+if __name__ == '__main__': main()

pwnlib/commandline/elfpatch.py

@@ -0,0 +1,25 @@
+#!/usr/bin/env python2
+
+import argparse, sys
+from pwnlib.elf import ELF
+from pwnlib.util.fiddling import unhex
+p = argparse.ArgumentParser()
+p.add_argument('elf',help="File to patch")
+p.add_argument('offset',help="Offset to patch in virtual address (hex encoded)")
+p.add_argument('bytes',help='Bytes to patch (hex encoded)')
+
+
+def main():
+    a = p.parse_args()
+
+    if not a.offset.startswith('0x'):
+        a.offset = '0x' + a.offset
+
+    offset = int(a.offset, 16)
+    bytes  = unhex(a.bytes)
+    elf    = ELF(a.elf)
+
+    elf.write(offset, bytes)
+    sys.stdout.write(elf.get_data())
+
+if __name__ == '__main__': main()

pwnlib/commandline/hex.py

@@ -0,0 +1,17 @@
+#!/usr/bin/env python2
+import sys, argparse
+
+parser = argparse.ArgumentParser(description='''
+Hex-encodes data provided on the command line or via stdin.
+''')
+parser.add_argument('data', nargs='*',
+    help='Data to convert into hex')
+
+def main():
+    args = parser.parse_args()
+    if not args.data:
+        print sys.stdin.read().encode('hex')
+    else:
+        print ' '.join(args.data).encode('hex')
+
+if __name__ == '__main__': main()

pwnlib/commandline/phd.py

@@ -0,0 +1,99 @@
+#!/usr/bin/env python2
+
+import argparse, sys, os
+import pwnlib.term.text as text
+from pwnlib.util.fiddling import hexdump_iter
+
+parser = argparse.ArgumentParser(
+    description = 'Pwnlib HexDump'
+)
+
+parser.add_argument(
+    'file',
+    metavar='file',
+    nargs='?',
+    help='File to hexdump.  Reads from stdin if missing.',
+    type=argparse.FileType('r'),
+    default=sys.stdin
+)
+
+parser.add_argument(
+    "-w", "--width",
+    help="Number of bytes per line.",
+    default='16',
+)
+
+parser.add_argument(
+    "-l", "--highlight",
+    help="Byte to highlight.",
+    nargs="*",
+)
+
+parser.add_argument(
+    "-s", "--skip",
+    help="Skip this many initial bytes.",
+    default='0',
+)
+
+parser.add_argument(
+    "-c", "--count",
+    help="Only show this many bytes.",
+    default='-1',
+)
+
+parser.add_argument(
+    "-o", "--offset",
+    help="Addresses in left hand column starts at this address.",
+    default='0',
+)
+
+parser.add_argument(
+    "--color",
+    nargs='?',
+    help="Colorize the output.  When 'auto' output is colorized exactly when stdout is a TTY.  Default is 'auto'.",
+    choices = ('always', 'never', 'auto'),
+    default='auto',
+)
+
+def asint(s):
+    if   s.startswith('0x'):
+        return int(s, 16)
+    elif s.startswith('0'):
+        return int(s, 8)
+    else:
+        return int(s, 10)
+
+def main():
+    args = parser.parse_args()
+
+    infile = args.file
+    width  = asint(args.width)
+    skip   = asint(args.skip)
+    count  = asint(args.count)
+    offset = asint(args.offset)
+
+    # if `--color` has no argument it is `None`
+    color = args.color or 'always'
+    text.when = color
+
+    if skip:
+        if infile == sys.stdin:
+            infile.read(skip)
+        else:
+            infile.seek(skip, os.SEEK_CUR)
+
+    data = infile.read(count)
+
+    hl = []
+    if args.highlight:
+        for hs in args.highlight:
+            for h in hs.split(','):
+                hl.append(asint(h))
+
+    try:
+        for line in hexdump_iter(data, width, highlight = hl, begin = offset + skip):
+            print line
+    except (KeyboardInterrupt, IOError):
+        pass
+
+if __name__ == '__main__': main()

pwnlib/commandline/shellcraft.py

@@ -0,0 +1,215 @@
+#!/usr/bin/env python2
+import argparse, sys, os, types
+import pwnlib
+from pwnlib import log, util
+import pwnlib.term.text as text
+from pwnlib.context import context
+
+r = text.red
+g = text.green
+b = text.blue
+
+banner = '\n'.join(['  ' + r('____') + '  ' + g('_') + '          ' + r('_') + ' ' + g('_') + '                 ' + b('__') + ' ' + r('_'),
+                    ' ' + r('/ ___|') + g('| |__') + '   ' + b('___') + r('| |') + ' ' + g('|') + ' ' + b('___') + ' ' + r('_ __') + ' ' + g('__ _') + ' ' + b('/ _|') + ' ' + r('|_'),
+                    ' ' + r('\___ \\') + g('| \'_ \\') + ' ' + b('/ _ \\') + ' ' + r('|') + ' ' + g('|') + b('/ __|') + ' ' + r('\'__/') + ' ' + g('_` |') + ' ' + b('|_') + r('| __|'),
+                    '  ' + r('___) |') + ' ' + g('| | |') + '  ' + b('__/') + ' ' + r('|') + ' ' + g('|') + ' ' + b('(__') + r('| |') + ' ' + g('| (_| |') + '  ' + b('_|') + ' ' + r('|_'),
+                    ' ' + r('|____/') + g('|_| |_|') + b('\\___|') + r('_|') + g('_|') + b('\\___|') + r('_|') + '  ' + g('\\__,_|') + b('_|') + '  ' + r('\\__|'),
+                    '\n'
+                    ])
+
+
+#  ____  _          _ _                 __ _
+# / ___|| |__   ___| | | ___ _ __ __ _ / _| |_
+# \___ \| '_ \ / _ \ | |/ __| '__/ _` | |_| __|
+#  ___) | | | |  __/ | | (__| | | (_| |  _| |_
+# |____/|_| |_|\___|_|_|\___|_|  \__,_|_|  \__|
+
+def _string(s):
+    out = []
+    for c in s:
+        co = ord(c)
+        if co >= 0x20 and co <= 0x7e and c not in '/$\'"`':
+            out.append(c)
+        else:
+            out.append('\\x%02x' % co)
+    return '"' + ''.join(out) + '"\n'
+
+def _carray(s):
+    out = []
+    for c in s:
+        out.append('0x' + util.fiddling.enhex(c))
+    return '{' + ', '.join(out) + '};\n'
+
+def _hex(s):
+    return pwnlib.util.fiddling.enhex(s) + '\n'
+
+p = argparse.ArgumentParser(
+    description = 'Microwave shellcode -- Easy, fast and delicious',
+    formatter_class = argparse.RawDescriptionHelpFormatter,
+)
+
+
+p.add_argument(
+    '-?', '--show',
+    action = 'store_true',
+    help = 'Show shellcode documentation',
+)
+
+p.add_argument(
+    '-o', '--out',
+    metavar = '<file>',
+    type = argparse.FileType('w'),
+    default = sys.stdout,
+    help = 'Output file (default: stdout)',
+)
+
+p.add_argument(
+    '-f', '--format',
+    metavar = '<format>',
+    choices = ['r', 'raw',
+               's', 'str', 'string',
+               'c',
+               'h', 'hex',
+               'a', 'asm', 'assembly',
+               'p',
+               'i', 'hexii',
+               'default'],
+    default = 'default',
+    help = 'Output format (default: hex), choose from {r}aw, {s}tring, {c}-style array, {h}ex string, hex{i}i, {a}ssembly code, {p}reprocssed code',
+)
+
+
+class NoDefaultContextValues(object):
+    def __enter__(self):
+        self.old = context.defaults.copy()
+        context.defaults['os'] = None
+        context.defaults['arch'] = None
+    def __exit__(self, *a):
+        context.defaults.update(self.old)
+
+
+
+def get_tree(path, val, result):
+    with NoDefaultContextValues():
+        if path:
+            path += '.'
+
+        if path.startswith('common.'):
+            return
+
+        mods = []
+
+        for k in sorted(dir(val)):
+            if k and k[0] != '_':
+                cur = getattr(val, k)
+                if isinstance(cur, types.ModuleType):
+                    mods.append((path + k, cur))
+                else:
+                    result.append((path + k, cur))
+
+        for path, val in mods:
+            get_tree(path, val, result)
+        return result
+
+# Enumearte all of the shellcode names
+all_shellcodes = get_tree('', pwnlib.shellcraft, [])
+names = '\n'.join('    ' + sc[0] for sc in all_shellcodes)
+
+p.add_argument(
+    'shellcode',
+    nargs = '?',
+    default = '',
+    metavar = '<shellcode>',
+    help = 'The shellcode you want',
+)
+
+p.epilog = 'Available shellcodes are:\n' + names
+
+
+p.add_argument(
+    'args',
+    nargs = '*',
+    metavar = '<arg>',
+    default = (),
+    help = 'Argument to the chosen shellcode',
+)
+
+def main():
+    # Banner must be added here so that it doesn't appear in the autodoc
+    # generation for command line tools
+    p.description = banner + p.description
+    args = p.parse_args()
+
+    if args.format == 'default':
+        if sys.stdout.isatty():
+            args.format = 'hex'
+        else:
+            args.format = 'raw'
+
+
+    vals = get_tree('', pwnlib.shellcraft, [])
+    if args.shellcode:
+        vals = [(k, val) for k, val in vals if k.startswith(args.shellcode + '.') or k == args.shellcode]
+
+    if len(vals) == 0:
+        log.fatal("Cannot find subtree by the name of %r" % args.shellcode)
+    elif len(vals) > 1:
+        for k, _ in vals:
+            print k
+        exit()
+    else:
+        func = vals[0][1]
+
+    if args.show:
+        print func.__doc__
+        exit()
+
+    defargs = len(func.func_defaults or ())
+    reqargs = func.func_code.co_argcount - defargs
+    if len(args.args) < reqargs:
+        if defargs > 0:
+            log.fatal('%s takes at least %d arguments' % (args.shellcode, reqargs))
+        else:
+            log.fatal('%s takes exactly %d arguments' % (args.shellcode, reqargs))
+
+    # Captain uglyness saves the day!
+    for i, val in enumerate(args.args):
+        try:
+            args.args[i] = util.safeeval.expr(val)
+        except ValueError:
+            pass
+
+    # And he strikes again!
+    os = arch = None
+    for k in args.shellcode.split('.')[:-1]:
+        if k in context.architectures:
+            arch = k
+        elif k in context.oses:
+            os = k
+
+    code = func(*args.args)
+
+    if args.format in ['a', 'asm', 'assembly']:
+        print code
+        exit()
+    if args.format == 'p':
+        print pwnlib.asm.cpp(code, arch = arch, os = os)
+        exit()
+
+    code = pwnlib.asm.asm(code, arch = arch, os = os)
+
+    if args.format in ['s', 'str', 'string']:
+        code = _string(code)
+    elif args.format == 'c':
+        code = _carray(code)
+    elif args.format in ['h', 'hex']:
+        code = _hex(code)
+    elif args.format in ['i', 'hexii']:
+        code = pwnlib.util.fiddling.hexii(code) + '\n'
+
+    if not sys.stdin.isatty():
+        sys.stdout.write(sys.stdin.read())
+
+    sys.stdout.write(code)
+
+if __name__ == '__main__': main()

pwnlib/commandline/unhex.py

@@ -0,0 +1,22 @@
+#!/usr/bin/env python2
+import sys, argparse
+from string import whitespace
+
+parser = argparse.ArgumentParser(description='''
+Decodes hex-encoded data provided on the command line or via stdin.
+''')
+parser.add_argument('hex', nargs='*',
+    help='Hex bytes to decode')
+
+def main():
+    args = parser.parse_args()
+    try:
+        if not args.hex:
+            s = sys.stdin.read().translate(None, whitespace)
+            sys.stdout.write(s.decode('hex'))
+        else:
+            sys.stdout.write(''.join(sys.argv[1:]).decode('hex'))
+    except TypeError, e:
+        sys.stderr.write(str(e) + '\n')
+
+if __name__ == '__main__': main()

pwnlib/constants/__init__.py

@@ -1,6 +1,40 @@
+"""Module containing constants extracted from header files.
+
+The purpose of this module is to provide quick access to constants from
+different architectures and operating systems.
+
+Example:
+
+    >>> print constants.freebsd.SYS_stat
+    188
+    >>> print constants.linux.i386.SYS_stat
+    106
+    >>> print constants.linux.amd64.SYS_stat
+    4
+
+The submodule ``freebsd`` contains all constants for FreeBSD, while the
+constants for Linux have been split up by architecture.
+
+The variables of the submodules will be "lifted up" by setting the
+:data:`pwnlib.context.arch` or :data:`pwnlib.context.os` in a manner similar to
+what happens in :mod:`pwnlib.shellcraft`.
+
+Example:
+
+    >>> with context.local(os = 'freebsd'):
+    ...     print constants.SYS_stat
+    188
+    >>> with context.local(os = 'linux', arch = 'i386'):
+    ...     print constants.SYS_stat
+    106
+    >>> with context.local(os = 'linux', arch = 'amd64'):
+    ...     print constants.SYS_stat
+    4
+
+"""
 from types import ModuleType
 import importlib, sys
-from .. import context
+from ..context import context
 
 class module(ModuleType):
     def __init__(self, submodules):
@@ -8,6 +42,7 @@ class module(ModuleType):
 
         # Insert nice properties
         self.__dict__.update({
+            '__doc__':     __doc__,
             '__file__':    __file__,
             '__package__': __package__,
             '__path__':    __path__,