Release 3.8.0

* Fix config types for integer values

Fixes Gallopsled/pwntools#988

* Add warning for unsupported configuration types

CHANGELOG.md

@@ -9,9 +9,11 @@ The table below shows which release corresponds to each branch, and what date th
 
 | Version          | Branch   | Release Date           |
 | ---------------- | -------- | ---------------------- |
-| [3.9.0](#390)    | `dev`    | Sep 8, 2017 (planned)
-| [3.8.0](#380)    | `beta`   | Jul 28, 2017 (planned)
-| [3.7.0](#370)    | `stable` | Jun 19, 2017
+| [3.10.0](#3100)  | `dev`    | Oct 21, 2017 (planned)
+| [3.9.0](#390)    | `beta`   | Sep 9, 2017 (planned)
+| [3.8.0](#380)    | `stable` | Jul 29, 2017
+| [3.7.1](#371)    |          | Jul 14, 2017
+| [3.7.0](#370)    |          | Jun 19, 2017
 | [3.6.1](#361)    |          | May 12, 2017
 | [3.6.0](#360)    |          | May 8, 2017
 | [3.5.1](#351)    |          | Apr 15, 2017
@@ -34,19 +36,29 @@ The table below shows which release corresponds to each branch, and what date th
 | [3.0.0](#300)    |          | Aug 20, 2016
 | [2.2.0](#220)    |          | Jan 5, 2015
 
+## 3.10.0
+
+To be released on Sep 8, 2017.
+
 ## 3.9.0
 
 To be released on Sep 8, 2017.
 
 ## 3.8.0
 
-To be released on Jul 28, 2017.
-
 - [#981][981] Fixed RELRO detection logic
 - [#986][986] Enhancements to DynELF for controlling usage of LibcDB
+- A few documentation fixes
+- A few fixes for the Docker image
 
-[981]: https://github.com/Gallopsled/pwntools/issues/981
-[986]: https://github.com/Gallopsled/pwntools/issues/986
+[981]: https://github.com/Gallopsled/pwntools/pull/981
+[986]: https://github.com/Gallopsled/pwntools/pull/986
+
+## 3.7.1
+
+- [#998][998] Fix a bug where integer values could not be set in `.pwn.conf`.
+
+[998]: https://github.com/Gallopsled/pwntools/pull/998
 
 ## 3.7.0
 
@@ -54,8 +66,8 @@ To be released on Jul 28, 2017.
 - [#952][952] A few small fixes were made to `pwn template`, and the CRC database was updated.
 - [5c72d62c][5c72d62c] Updated the CRC database
 
-[933]: https://github.com/Gallopsled/pwntools/issues/933
-[952]: https://github.com/Gallopsled/pwntools/issues/952
+[933]: https://github.com/Gallopsled/pwntools/pull/933
+[952]: https://github.com/Gallopsled/pwntools/pull/952
 [5c72d62c]: https://github.com/Gallopsled/pwntools/commit/5c72d62c
 
 ## 3.6.1

README.md

@@ -2,13 +2,13 @@
 ![pwntools logo](https://github.com/Gallopsled/pwntools/blob/stable/docs/source/logo.png?raw=true)
 
 [![Docs](https://readthedocs.org/projects/pwntools/badge/?version=stable)](https://docs.pwntools.com/)
-[![PyPI](https://img.shields.io/badge/pypi-v3.7.0-green.svg?style=flat)](https://pypi.python.org/pypi/pwntools/)
+[![PyPI](https://img.shields.io/badge/pypi-v3.8.0-green.svg?style=flat)](https://pypi.python.org/pypi/pwntools/)
 [![Travis](https://travis-ci.org/Gallopsled/pwntools.svg)](https://travis-ci.org/Gallopsled/pwntools)
 [![Coveralls](https://img.shields.io/coveralls/Gallopsled/pwntools/dev.svg)](https://coveralls.io/github/Gallopsled/pwntools?branch=dev)
 [![Twitter](https://img.shields.io/badge/twitter-pwntools-4099FF.svg?style=flat)](https://twitter.com/pwntools)
 [![MIT License](https://img.shields.io/badge/license-MIT-blue.svg?style=flat)](http://choosealicense.com/licenses/mit/)
 
-pwntools is a CTF framework and exploit development library. Written in Python, it is designed for rapid prototyping and development, and intended to make exploit writing as simple as possible.
+Pwntools is a CTF framework and exploit development library. Written in Python, it is designed for rapid prototyping and development, and intended to make exploit writing as simple as possible.
 
 ```python
 from pwn import *
@@ -32,7 +32,7 @@ To get you started, we've provided some example solutions for past CTF challenge
 
 # Installation
 
-pwntools is best supported on 64-bit Ubuntu 12.04 and 14.04, but most functionality should work on any Posix-like distribution (Debian, Arch, FreeBSD, OSX, etc.).  Python 2.7 is required.
+Pwntools is best supported on 64-bit Ubuntu LTE releases (12.04, 14.04, and 16.04).  Most functionality should work on any Posix-like distribution (Debian, Arch, FreeBSD, OSX, etc.).  Python 2.7 is required.
 
 Most of the functionality of pwntools is self-contained and Python-only.  You should be able to get running quickly with
 

docs/source/asm.rst

@@ -15,7 +15,7 @@
    :members:
 
 Internal Functions
-=========================================
+-----------------------------------------
 
 These are only included so that their tests are run.
 

docs/source/index.rst

@@ -57,6 +57,7 @@ Each of the ``pwntools`` modules is documented here.
    flag
    fmtstr
    gdb
+   libcdb
    log
    memleak
    protocols

docs/source/libcdb.rst

@@ -0,0 +1,10 @@
+.. testsetup:: *
+
+   from pwn import *
+   from pwnlib.libcdb import *
+
+:mod:`pwnlib.libcdb` --- Libc Database
+===========================================
+
+.. automodule:: pwnlib.libcdb
+   :members:

extra/docker/Makefile

@@ -1,6 +1,6 @@
 subdirs=xenial stable $(wildcard */)
 makefiles=$(wildcard */Makefile)
-tags=stable beta dev
+tags=base stable beta dev
 
 all: $(makefiles) $(subdirs)
 

extra/docker/base

@@ -0,0 +1 @@
+xenial

extra/docker/kali/Makefile

@@ -1,6 +1,6 @@
 all: Dockerfile
 
-Dockerfile: ../stable/Dockerfile FORCE
+Dockerfile: ../xenial/Dockerfile FORCE
 	sed "s|ubuntu:xenial|kalilinux/kali-linux-docker|" < "$<" > "$@"
 
 FORCE:

extra/docker/stable/Dockerfile

@@ -1,26 +1,6 @@
-############################################################
-# Dockerfile to build Pwntools container
-# Based on Ubuntu
-############################################################
+FROM pwntools/pwntools:base
 
-FROM ubuntu:xenial
-MAINTAINER Maintainer Gallopsled et al.
-
-RUN apt-get update
-RUN apt-get install -y build-essential
-RUN apt-get install -y git
-RUN apt-get install -y libssl-dev
-RUN apt-get install -y libffi-dev
-RUN apt-get install -y python2.7
-RUN apt-get install -y python-pip
-RUN apt-get install -y python-dev
-RUN pip install --upgrade pip
-RUN python -m pip install --upgrade pwntools
+USER root
+RUN pip install --upgrade git+https://github.com/Gallopsled/pwntools@stable
 RUN PWNLIB_NOTERM=1 pwn update
-
-RUN apt-get install -y sudo
-RUN useradd -m pwntools
-RUN passwd --delete --unlock pwntools
-RUN echo "pwntools ALL=(ALL:ALL) NOPASSWD: ALL" > /etc/sudoers.d/pwntools
 USER pwntools
-WORKDIR /home/pwntools

extra/docker/stable/Makefile

@@ -1,8 +0,0 @@
-all: Dockerfile
-
-Dockerfile: ../xenial/Dockerfile FORCE
-	cp "$<" "$@"
-
-FORCE:
-
-.PHONY: all

pwnlib/adb/adb.py

@@ -328,7 +328,7 @@ class AdbDevice(Device):
             kwargs['port'] = 'emulator'
 
         for field in fields[2:]:
-            k,v = field.split(':')
+            k,v = field.split(':', 1)
             kwargs[k] = v
 
         return AdbDevice(serial, type, **kwargs)

pwnlib/args.py

@@ -37,8 +37,18 @@ not specified evaluate to an empty string.
     else:
         io = process('./pwnable')
 
+Arguments can also be accessed directly with the dot operator, e.g.:
+
+.. code-block:: python
+
+    if args.REMOTE:
+        ...
+
+Any undefined arguments evaluate to an empty string, ``''``.
+
 The full list of supported "magic arguments" and their effects are listed
 below.
+
 """
 from __future__ import absolute_import
 

pwnlib/context/__init__.py

@@ -1359,9 +1359,14 @@ def update_context_defaults(section):
         if key not in ContextType.defaults:
             log.warn("Unknown configuration option %r in section %r" % (key, 'context'))
             continue
-        if isinstance(ContextType.defaults[key], (str, unicode, tuple)):
-            value = safeeval.expr(value)
 
-        ContextType.defaults[key] = value
+        default = ContextType.defaults[key]
+
+        if isinstance(default, (str, unicode, tuple, int, long, list, dict)):
+            value = safeeval.expr(value)
+        else:
+            log.warn("Unsupported configuration option %r in section %r" % (key, 'context'))
+
+        ContextType.defaults[key] = type(default)(value)
 
 register_config('context', update_context_defaults)

pwnlib/data/elf/__init__.py

@@ -1,3 +1,6 @@
+from __future__ import absolute_import
+from pwnlib.data.elf import relro
+
 import os
 path = os.path.dirname(__file__)
 

pwnlib/data/elf/relro/__init__.py

@@ -0,0 +1,5 @@
+import os
+path = os.path.dirname(__file__)
+
+def get(x):
+    return os.path.join(path, x)

pwnlib/dynelf.py

@@ -145,7 +145,7 @@ class DynELF(object):
     .. _DT_PLTGOT: http://refspecs.linuxfoundation.org/ELF/zSeries/lzsabi0_zSeries/x2251.html
     '''
 
-    def __init__(self, leak, pointer=None, elf=None):
+    def __init__(self, leak, pointer=None, elf=None, libcdb=True):
         '''
         Instantiates an object which can resolve symbols in a running binary
         given a :class:`pwnlib.memleak.MemLeak` leaker and a pointer inside
@@ -155,7 +155,9 @@ class DynELF(object):
             leak(MemLeak): Instance of pwnlib.memleak.MemLeak for leaking memory
             pointer(int):  A pointer into a loaded ELF file
             elf(str,ELF):  Path to the ELF file on disk, or a loaded :class:`pwnlib.elf.ELF`.
+            libcdb(bool):  Attempt to use libcdb to speed up libc lookups
         '''
+        self.libcdb    = libcdb
         self._elfclass = None
         self._elftype  = None
         self._link_map = None
@@ -560,7 +562,7 @@ class DynELF(object):
         #
         # If we are resolving a symbol in the library, find it.
         #
-        if symb:
+        if symb and self.libcdb:
             # Try a quick lookup by build ID
             self.status("Trying lookup based on Build ID")
             build_id = dynlib._lookup_build_id(lib=lib)

pwnlib/elf/datatypes.py

@@ -252,6 +252,40 @@ class constants:
     AT_L2_CACHESHAPE        = 36
     AT_L3_CACHESHAPE        = 37
 
+    # Legal flags used in the d_val field of the DT_FLAGS dynamic entry.
+    DF_ORIGIN               = 0x01
+    DF_SYMBOLIC             = 0x02
+    DF_TEXTREL              = 0x04
+    DF_BIND_NOW             = 0x08
+    DF_STATIC_TLS           = 0x10
+
+    # Legal flags used in the d_val field of the DT_FLAGS_1 dynamic entry.
+    DF_1_NOW                = 0x00000001
+    DF_1_GLOBAL             = 0x00000002
+    DF_1_GROUP              = 0x00000004
+    DF_1_NODELETE           = 0x00000008
+    DF_1_LOADFLTR           = 0x00000010
+    DF_1_INITFIRST          = 0x00000020
+    DF_1_NOOPEN             = 0x00000040
+    DF_1_ORIGIN             = 0x00000080
+    DF_1_DIRECT             = 0x00000100
+    DF_1_TRANS              = 0x00000200
+    DF_1_INTERPOSE          = 0x00000400
+    DF_1_NODEFLIB           = 0x00000800
+    DF_1_NODUMP             = 0x00001000
+    DF_1_CONFALT            = 0x00002000
+    DF_1_ENDFILTEE          = 0x00004000
+    DF_1_DISPRELDNE         = 0x00008000
+    DF_1_DISPRELPND         = 0x00010000
+    DF_1_NODIRECT           = 0x00020000
+    DF_1_IGNMULDEF          = 0x00040000
+    DF_1_NOKSYMS            = 0x00080000
+    DF_1_NOHDR              = 0x00100000
+    DF_1_EDITED             = 0x00200000
+    DF_1_NORELOC            = 0x00400000
+    DF_1_SYMINTPOSE         = 0x00800000
+    DF_1_GLOBAUDIT          = 0x01000000
+    DF_1_SINGLETON          = 0x02000000
 
 
 class Elf32_Ehdr(ctypes.Structure):

pwnlib/elf/elf.py

@@ -64,6 +64,7 @@ from pwnlib.context import LocalContext
 from pwnlib.context import context
 from pwnlib.elf.config import kernel_configuration
 from pwnlib.elf.config import parse_kconfig
+from pwnlib.elf.datatypes import constants
 from pwnlib.elf.plt import emulate_plt_instructions
 from pwnlib.log import getLogger
 from pwnlib.qemu import get_qemu_arch
@@ -1327,14 +1328,28 @@ class ELF(ELFFile):
         .. _PT_GNU_RELRO: https://refspecs.linuxbase.org/LSB_3.1.1/LSB-Core-generic/LSB-Core-generic.html#PROGHEADER
         .. _DF_BIND_NOW: http://refspecs.linuxbase.org/elf/gabi4+/ch5.dynamic.html#df_bind_now
 
+        >>> path = pwnlib.data.elf.relro.path
+        >>> for test in glob(os.path.join(path, 'test-*')):
+        ...     e = ELF(test)
+        ...     expected = os.path.basename(test).split('-')[2]
+        ...     actual = str(e.relro).lower()
+        ...     assert actual == expected
         """
+        if not any('GNU_RELRO' in str(s.header.p_type) for s in self.segments):
+            return None
+
         if self.dynamic_by_tag('DT_BIND_NOW'):
             return "Full"
 
-        if any('GNU_RELRO' in str(s.header.p_type) for s in self.segments):
-            return "Partial"
+        flags = self.dynamic_value_by_tag('DT_FLAGS')
+        if flags and flags & constants.DF_BIND_NOW:
+            return "Full"
 
-        return None
+        flags_1 = self.dynamic_value_by_tag('DT_FLAGS_1')
+        if flags_1 and flags_1 & constants.DF_1_NOW:
+            return "Full"
+
+        return "Partial"
 
     @property
     def nx(self):

pwnlib/libcdb.py

@@ -21,14 +21,63 @@ from pwnlib.util.web import wget
 
 log = getLogger(__name__)
 
-cache_dir = os.path.join(tempfile.gettempdir(), 'pwn')
+HASHES = ['build_id', 'sha1', 'sha256', 'md5']
+
+def search_by_hash(hex_encoded_id, hash_type='build_id'):
+    assert hash_type in HASHES, hash_type
+
+    # Ensure that the libcdb cache directory exists
+    cache_dir = os.path.join(context.cache_dir, 'libcdb', hash_type)
+
+    if not os.path.isdir(cache_dir):
+        os.makedirs(cache_dir)
+
+    # If we already downloaded the file, and it looks even passingly like
+    # a valid ELF file, return it.
+    cache = os.path.join(cache_dir, hex_encoded_id)
+
+    if os.path.exists(cache):
+        log.debug("Found existing cached libc at %r", cache)
+
+        data = read(cache)
+        if data.startswith('\x7FELF'):
+            log.info_once("Using cached data from %r", cache)
+            return cache
+        else:
+            log.info_once("Skipping unavialable libc %s", hex_encoded_id)
+            return None
+
+    # Build the URL using the requested hash type
+    url_base = "https://gitlab.com/libcdb/libcdb/raw/master/hashes/%s/" % hash_type
+    url      = urlparse.urljoin(url_base, hex_encoded_id)
+
+    data   = ""
+    while not data.startswith('\x7fELF'):
+        log.debug("Downloading data from LibcDB: %s", url)
+        data = wget(url)
+
+        if not data:
+            log.warn_once("Could not fetch libc for build_id %s", hex_encoded_id)
+            break
+
+        # GitLab serves up symlinks with
+        if data.startswith('..'):
+            url = os.path.dirname(url) + '/'
+            url = urlparse.urljoin(url, data)
+
+    # Save whatever we got to the cache
+    write(cache, data or '')
+
+    # Return ``None`` if we did not get a valid ELF file
+    if not data or not data.startswith('\x7FELF'):
+        return None
+
+    return cache
+
 
 def search_by_build_id(hex_encoded_id):
     """
-    Given a hex-encoded Build ID, return the path to an ELF with that Build ID
-    only the local system.
-
-    If it can't be found, return None.
+    Given a hex-encoded Build ID, attempt to download a matching libc from libcdb.
 
     Arguments:
         hex_encoded_id(str):
@@ -36,31 +85,79 @@ def search_by_build_id(hex_encoded_id):
 
     Returns:
         Path to the downloaded library on disk, or :const:`None`.
+
+    Examples:
+        >>> filename = search_by_build_id('fe136e485814fee2268cf19e5c124ed0f73f4400')
+        >>> hex(ELF(filename).symbols.read)
+        '0xda260'
+        >>> None == search_by_build_id('XX')
+        True
     """
-    cache = cache_dir + '-libc.so.' + hex_encoded_id
+    return search_by_hash(hex_encoded_id, 'build_id')
 
-    if os.path.exists(cache) and read(cache).startswith('\x7FELF'):
-        log.info_once("Using cached data from %r" % cache)
-        return cache
+def search_by_md5(hex_encoded_id):
+    """
+    Given a hex-encoded md5sum, attempt to download a matching libc from libcdb.
 
-    log.info("Downloading data from GitHub")
+    Arguments:
+        hex_encoded_id(str):
+            Hex-encoded Build ID (e.g. 'ABCDEF...') of the library
 
-    url_base = "https://gitlab.com/libcdb/libcdb/raw/master/hashes/build_id/"
-    url      = urlparse.urljoin(url_base, hex_encoded_id)
+    Returns:
+        Path to the downloaded library on disk, or :const:`None`.
 
-    data   = ""
-    while not data.startswith('\x7fELF'):
-        data = wget(url)
+    Examples:
+        >>> filename = search_by_md5('7a71dafb87606f360043dcd638e411bd')
+        >>> hex(ELF(filename).symbols.read)
+        '0xda260'
+        >>> None == search_by_build_id('XX')
+        True
+    """
+    return search_by_hash(hex_encoded_id, 'md5')
+
+def search_by_sha1(hex_encoded_id):
+    """
+    Given a hex-encoded sha1, attempt to download a matching libc from libcdb.
+
+    Arguments:
+        hex_encoded_id(str):
+            Hex-encoded Build ID (e.g. 'ABCDEF...') of the library
+
+    Returns:
+        Path to the downloaded library on disk, or :const:`None`.
+
+    Examples:
+        >>> filename = search_by_sha1('34471e355a5e71400b9d65e78d2cd6ce7fc49de5')
+        >>> hex(ELF(filename).symbols.read)
+        '0xda260'
+        >>> None == search_by_sha1('XX')
+        True
+    """
+    return search_by_hash(hex_encoded_id, 'sha1')
+
+
+def search_by_sha256(hex_encoded_id):
+    """
+    Given a hex-encoded sha256, attempt to download a matching libc from libcdb.
+
+    Arguments:
+        hex_encoded_id(str):
+            Hex-encoded Build ID (e.g. 'ABCDEF...') of the library
+
+    Returns:
+        Path to the downloaded library on disk, or :const:`None`.
+
+    Examples:
+        >>> filename = search_by_sha256('5e877a8272da934812d2d1f9ee94f73c77c790cbc5d8251f5322389fc9667f21')
+        >>> hex(ELF(filename).symbols.read)
+        '0xda260'
+        >>> None == search_by_sha256('XX')
+        True
+    """
+    return search_by_hash(hex_encoded_id, 'sha256')
 
-        if not data:
-            return None
 
-        if data.startswith('..'):
-            url = os.path.dirname(url) + '/'
-            url = urlparse.urljoin(url, data)
 
-    write(cache, data)
-    return cache
 
 def get_build_id_offsets():
     """
@@ -114,4 +211,4 @@ def get_build_id_offsets():
     }.get(context.arch, [])
 
 
-__all__ = ['get_build_id_offsets', 'search_by_build_id']
+__all__ = ['get_build_id_offsets', 'search_by_build_id', 'search_by_sha1', 'search_by_sha256', 'search_by_md5']

pwnlib/tubes/tube.py

@@ -622,7 +622,7 @@ class tube(Timeout, Logger):
         return self.recvline_pred(pred, keepends = keepends, timeout = timeout)
 
     def recvrepeat(self, timeout = default):
-        """recvrepeat()
+        """recvrepeat(timeout = default) -> str
 
         Receives data until a timeout or EOF is reached.
 

pwnlib/version.py

@@ -1 +1 @@
-__version__ = '3.7.0'
+__version__ = '3.8.0'

setup.py

@@ -86,7 +86,7 @@ except Exception as e:
 setup(
     name                 = 'pwntools',
     packages             = find_packages(),
-    version              = '3.7.0',
+    version              = '3.8.0',
     data_files           = [('',
                              glob.glob('*.md') + glob.glob('*.txt')),
                             ],
@@ -105,7 +105,7 @@ setup(
     description          = "Pwntools CTF framework and exploit development library.",
     long_description     = long_description,
     author               = "Gallopsled et al.",
-    author_email         = "#pwntools @ freenode.net",
+    author_email         = "pwntools[email protected]",
     url                  = 'https://pwntools.com',
     download_url         = "https://github.com/Gallopsled/pwntools/releases",
     install_requires     = install_requires,