Release 3.4.0

We cannot determine the length of the file if passed e.g. a tube or a pipe or PTY or character device (e.g. stdin or /dev/null).

Supercedes: 9c2f0f4e
(cherry picked from commit 14f717b7b4)

.travis.yml

@@ -13,6 +13,7 @@ addons:
     - mksh
     - zsh
     - pandoc
+    - gdb
 cache:
     - pip
     - directories:
@@ -22,11 +23,14 @@ cache:
 python:
   - "2.7"
 before_install:
+  - echo $-
   - source travis/install.sh
   - source travis/ssh_setup.sh
+  - echo $-
 install:
+  - pip install --upgrade pip
   - pip install --upgrade --editable .
-  - pip install --requirement docs/requirements.txt
+  - pip install --upgrade --requirement docs/requirements.txt
 before_script:
   - PWNLIB_NOTERM=1 python -c 'from pwn import *; print pwnlib.term.term_mode'
   - PWNLIB_NOTERM=1 python -c 'from pwn import *; print pwnlib.term.term_mode'

CHANGELOG.md

@@ -9,9 +9,12 @@ The table below shows which release corresponds to each branch, and what date th
 
 | Version          | Branch   | Release Date           |
 | ---------------- | -------- | ---------------------- |
-| [3.5.0](#350)    | `dev`    | Mar 18, 2017 (planned)
-| [3.4.0](#340)    | `beta`   | Feb 4, 2017 (planned)
-| [3.3.2](#332)    | `stable` | Jan 10, 2016
+| [3.6.0](#350)    | `dev`    | May 6, 2017 (planned)
+| [3.5.0](#350)    | `beta`   | Mar 25, 2017 (planned)
+| [3.4.0](#340)    | `stable` | Feb 13, 2017
+| [3.3.4](#334)    |          | Jan 12, 2016
+| [3.3.3](#333)    |          | Jan 10, 2016
+| [3.3.2](#332)    |          | Jan 10, 2016
 | [3.3.1](#331)    |          | Jan 10, 2016
 | [3.3.0](#330)    |          | Dec 24, 2016
 | [3.2.1](#321)    |          | Dec 24, 2016
@@ -25,31 +28,55 @@ The table below shows which release corresponds to each branch, and what date th
 | [3.0.0](#300)    |          | Aug 20, 2016
 | [2.2.0](#220)    |          | Jan 5, 2015
 
+
+## 3.6.0
+
+To be released on May 6, 2017.
+
 ## 3.5.0
 
-To be released on Mar 18, 2017.
+To be released on Mar 25, 2017.
 
 ## 3.4.0
 
-To be released on Feb 4, 2017.
-
 - [#800][800] Add `shell=` option to `ssh.process()`
-- [5d9792f][5d9792f] Add `context.buffer_size` for fine-tuning `tube` performance
+- [#806][806] Add `context.buffer_size` for fine-tuning `tube` performance
     + Also adds `buffer_fill_size=` argument for all tubes
 - [b83a6c7][b83a6c7] Fix undocumented `process.leak` function
 - [546061e][546061e] Modify `coredump_filter` of all spawned processes, so that core dumps are more complete
-
+- [#809][809] Add several functions to `adb` (`unlink`, `mkdir`, `makedirs`, `isdir`, `exists`)
+- [#817][817] Make disconnection detection more robust
 
 [800]: https://github.com/Gallopsled/pwntools/pull/800
+[806]: https://github.com/Gallopsled/pwntools/pull/806
+[809]: https://github.com/Gallopsled/pwntools/pull/809
+[817]: https://github.com/Gallopsled/pwntools/pull/817
 [5d9792f]: https://github.com/Gallopsled/pwntools/commit/5d9792f
 [b83a6c7]: https://github.com/Gallopsled/pwntools/commit/b83a6c7
 [546061e]: https://github.com/Gallopsled/pwntools/commit/546061e
 
+## 3.3.4
+
+- [#850][850] and [#846][846] fix issues with `hexdump` and the `phd` command-line utility, when using pipes (e.g. `echo foo | phd`)
+- [#852][852] Fixes register ordering in `regsort`
+- [#853][853] Fixes the registers restored in `shellcraft.amd64.popad`
+
+[846]: https://github.com/gallopsled/pwntools/pull/846
+[850]: https://github.com/gallopsled/pwntools/pull/850
+[852]: https://github.com/gallopsled/pwntools/pull/852
+[853]: https://github.com/gallopsled/pwntools/pull/853
+
+## 3.3.3
+
+- [#843][843] fixed a bug in `amd64.mov`.
+
+[843]: https://github.com/gallopsled/pwntools/pull/843
+
 ## 3.3.2
 
-- [#840][840] Fixed a regression introduced by [#837][837].
+- [#840][840] fixed a regression introduced by [#837][837].
 
-[840]: https://github.com/Gallopsled/pwntools/pull/840
+[840]: https://github.com/gallopsled/pwntools/pull/840
 
 ## 3.3.1
 
@@ -147,7 +174,7 @@ Multiple bug fixes.
 - [1e414af][1e414af] Added `connect()` alias for `remote()`
     + For example, `io=connect('google.com', 80)`
     + This also works with `tcp(...)` and `udp(...)` aliases
-- [869ec42][869ec42] Added `ssh.read()` and `ssh.write()` aiases
+- [869ec42][869ec42] Added `ssh.read()` and `ssh.write()` aliases
 - [2af55c9][2af55c9] `AdbDevice` objects exposed via e.g. `adb.devices()` now offer scoped access to all `adb` module properties
     + It is now possible to e.g. `map(lambda d: d.process(['id']).recvall(), adb.devices())`
 
@@ -234,7 +261,7 @@ Android support via a new `adb` module, `context.device`, `context.adb_host`, an
 - Added `context.log_file` setting for sending logs to a file.  This can be set with the `LOG_FILE` magic command-line option.
 - Added `context.noptrace` setting for disabling actions which require `ptrace` support.  This is useful for turning all `gdb.debug` and `gdb.attach` options into no-ops, and can be set via the `NOPTRACE` magic command-line option.
 - Added `context.proxy` which hooks all connections and sends them to a SOCKS4/SOCKS5.  This can be set via the `PROXY` magic command-line option.
-- Added `context.randomize` to control randommization of settings like XOR keys and register ordering (default off).
+- Added `context.randomize` to control randomization of settings like XOR keys and register ordering (default off).
 - Added `context.terminal` for setting how to launch commands in a new terminal.
 
 #### DynELF and MemLeak Module
@@ -247,7 +274,7 @@ Android support via a new `adb` module, `context.device`, `context.adb_host`, an
 #### Encoders Module
 
 - Added a `pwnlib.encoders` module for assembled-shellcode encoders/decoders
-- Includes position-indepentent basic XOR encoders
+- Includes position-independent basic XOR encoders
 - Includes position-independent delta encoders
 - Includes non-position-independent alphanumeric encoders for Intel
 - Includes position-independent alphanumeric encoders for ARM/Thumb

README.md

@@ -2,7 +2,7 @@
 ![pwntools logo](https://github.com/Gallopsled/pwntools/blob/stable/docs/source/logo.png?raw=true)
 
 [![Docs](https://readthedocs.org/projects/pwntools/badge/?version=stable)](https://docs.pwntools.com/)
-[![PyPI](https://img.shields.io/badge/pypi-v3.3.2-green.svg?style=flat)](https://pypi.python.org/pypi/pwntools/)
+[![PyPI](https://img.shields.io/badge/pypi-v3.4.0-green.svg?style=flat)](https://pypi.python.org/pypi/pwntools/)
 [![Travis](https://travis-ci.org/Gallopsled/pwntools.svg)](https://travis-ci.org/Gallopsled/pwntools)
 [![Coveralls](https://img.shields.io/coveralls/Gallopsled/pwntools/dev.svg)](https://coveralls.io/github/Gallopsled/pwntools?branch=dev)
 [![Twitter](https://img.shields.io/badge/twitter-pwntools-4099FF.svg?style=flat)](https://twitter.com/pwntools)

pwnlib/regsort.py

@@ -224,6 +224,8 @@ def regsort(in_out, all_regs, tmp = None, xchg = True, randomize = None):
          ('mov', 'b', 'c'),
          ('mov', 'c', 'x'),
          ('mov', 'x', '1')]
+         >>> regsort({'a': 'b', 'b': 'c'}, ['a','b','c'], xchg=0)
+         [('mov', 'a', 'b'), ('mov', 'b', 'c')]
     """
     if randomize is None:
         randomize = context.randomize
@@ -333,7 +335,9 @@ def regsort(in_out, all_regs, tmp = None, xchg = True, randomize = None):
     # If XCHG is expressly disabled, and there is no temporary register,
     # try to see if there is any register which can be used as a temp
     # register instead.
-    if not (xchg or tmp):
+    #
+    # If there are no cycles, there's no need for a temporary register.
+    if in_cycle and not (xchg or tmp):
         for reg in in_out:
             if not depends_on_cycle(reg, in_out, in_cycle):
                 tmp = reg

pwnlib/shellcraft/templates/amd64/mov.asm

@@ -61,17 +61,26 @@ Example:
         mov rax, 0x1010110dfac01fe
         xor [rsp], rax
         pop rax
-   >>> with context.local(os = 'linux'):
-   ...     print shellcraft.amd64.mov('eax', 'SYS_read').rstrip()
-       xor eax, eax /* (SYS_read) */
-   >>> with context.local(os = 'freebsd'):
-   ...     print shellcraft.amd64.mov('eax', 'SYS_read').rstrip()
-       push (SYS_read) /* 3 */
-       pop rax
-   >>> with context.local(os = 'linux'):
-   ...     print shellcraft.amd64.mov('eax', 'PROT_READ | PROT_WRITE | PROT_EXEC').rstrip()
-       push (PROT_READ | PROT_WRITE | PROT_EXEC) /* 7 */
-       pop rax
+    >>> print shellcraft.amd64.mov('rax', 0xffffffff).rstrip()
+        mov eax, 0xffffffff
+    >>> print shellcraft.amd64.mov('rax', 0x7fffffff).rstrip()
+        mov eax, 0x7fffffff
+    >>> print shellcraft.amd64.mov('rax', 0x80010101).rstrip()
+        mov eax, 0x80010101
+    >>> print shellcraft.amd64.mov('rax', 0x80000000).rstrip()
+        mov eax, 0x1010101 /* 2147483648 == 0x80000000 */
+        xor eax, 0x81010101
+    >>> with context.local(os = 'linux'):
+    ...     print shellcraft.amd64.mov('eax', 'SYS_read').rstrip()
+        xor eax, eax /* (SYS_read) */
+    >>> with context.local(os = 'freebsd'):
+    ...     print shellcraft.amd64.mov('eax', 'SYS_read').rstrip()
+        push (SYS_read) /* 3 */
+        pop rax
+    >>> with context.local(os = 'linux'):
+    ...     print shellcraft.amd64.mov('eax', 'PROT_READ | PROT_WRITE | PROT_EXEC').rstrip()
+        push (PROT_READ | PROT_WRITE | PROT_EXEC) /* 7 */
+        pop rax
 
 Args:
   dest (str): The destination register.
@@ -99,7 +108,6 @@ if get_register(src):
     if dest.size == 64 and src.size <= 32:
         dest = get_register(dest.native32)
 
-    src_size = src.size
 else:
     with ctx.local(arch = 'amd64'):
         src = eval(src)
@@ -107,9 +115,9 @@ else:
     if not dest.fits(src):
         log.error("cannot mov %s, %r: dest is smaller than src" % (dest, src))
 
-    src_size = bits_required(src)
+    orig_dest = dest
 
-    if dest.size == 64 and src_size <= 32:
+    if dest.size == 64 and bits_required(src) <= 32:
         dest = get_register(dest.native32)
 
     # Calculate the packed version
@@ -117,7 +125,15 @@ else:
 
     # Calculate the unsigned and signed versions
     srcu = packing.unpack(srcp, dest.size, sign=False)
-    srcs = packing.unpack(srcp, dest.size, sign=True)
+
+    # N.B.: We may have downsized the register for e.g. mov('rax', 0xffffffff)
+    #       In this case, srcp is now a 4-byte packed value, which will expand
+    #       to "-1", which isn't correct.
+    if orig_dest.size != dest.size:
+        srcs = src
+    else:
+        srcs = packing.unpack(srcp, dest.size, sign=True)
+
 %>\
 % if is_register(src):
     % if src == dest:

pwnlib/shellcraft/templates/amd64/popad.asm

@@ -6,8 +6,8 @@ in the same order.
     pop rdi
     pop rsi
     pop rbp
-    pop rsp
-    pop rbp
+    pop rbx /* add rsp, 8 */
+    pop rbx
     pop rdx
     pop rcx
     pop rax

pwnlib/util/fiddling.py

@@ -1,6 +1,7 @@
 # -*- coding: utf-8 -*-
 import base64
 import random
+import os
 import re
 import string
 import StringIO
@@ -10,7 +11,7 @@ from . import packing
 from ..context import context
 from ..log import getLogger
 from ..term import text
-from .cyclic import cyclic
+from .cyclic import de_bruijn
 from .cyclic import cyclic_find
 
 log = getLogger(__name__)
@@ -559,13 +560,15 @@ default_style = {
 }
 
 cyclic_pregen = ''
+de_bruijn_gen = de_bruijn()
 
 def sequential_lines(a,b):
     return (a+b) in cyclic_pregen
 
 def update_cyclic_pregenerated(size):
     global cyclic_pregen
-    cyclic_pregen = cyclic(size)
+    while size > len(cyclic_pregen):
+        cyclic_pregen += de_bruijn_gen.next()
 
 def hexdump_iter(fd, width=16, skip=True, hexii=False, begin=0, style=None,
                  highlight=None, cyclic=False):
@@ -608,22 +611,6 @@ def hexdump_iter(fd, width=16, skip=True, hexii=False, begin=0, style=None,
     spacer      = ' '
     marker      = (style.get('marker') or (lambda s:s))('│')
 
-    # Total length of the input stream
-    total = 0
-
-    if hasattr(fd, 'len'):
-        total = fd.len
-    else:
-        # Save the current file offset
-        cur = fd.seek(0, os.SEEK_CUR)
-
-        # Determine the total size of the file
-        fd.seek(0, os.SEEK_END)
-        total = fd.tell()
-
-        # Restore the file offset
-        fd.seek(cur, os.SEEK_SET)
-
     if hexii:
         column_sep = ''
         line_fmt   = '%%(offset)08x  %%(hexbytes)-%is│' % (len(column_sep)+(width*byte_width))
@@ -643,17 +630,29 @@ def hexdump_iter(fd, width=16, skip=True, hexii=False, begin=0, style=None,
             return hbyte, abyte
         cache = [style_byte(chr(b)) for b in range(256)]
 
-    if cyclic:
-        update_cyclic_pregenerated(total)
-
     numb = 0
     while True:
         offset = begin + numb
-        chunk = fd.read(width)
+
+        # If a tube is passed in as fd, it will raise EOFError when it runs
+        # out of data, unlike a file or StringIO object, which return an empty
+        # string.
+        try:
+            chunk = fd.read(width)
+        except EOFError:
+            chunk = ''
+
+        # We have run out of data, exit the loop
         if chunk == '':
             break
+
+        # Advance the cursor by the number of bytes we actually read
         numb += len(chunk)
 
+        # Update the cyclic pattern in case
+        if cyclic:
+            update_cyclic_pregenerated(numb)
+
         # If this chunk is the same as the last unique chunk,
         # use a '*' instead.
         if skip and last_unique:

pwnlib/version.py

@@ -1 +1 @@
-__version__ = '3.4.0beta2'
+__version__ = '3.4.0'

setup.py

@@ -80,7 +80,7 @@ except ImportError:
 setup(
     name                 = 'pwntools',
     packages             = find_packages(),
-    version              = '3.4.0beta2',
+    version              = '3.4.0',
     data_files           = [('',
                              glob.glob('*.md') + glob.glob('*.txt')),
                             ],

travis/install.sh

@@ -1,5 +1,6 @@
 #!/usr/bin/env bash -e
 set -e
+
 local_deb_extract()
 {
     wget $1
@@ -8,52 +9,54 @@ local_deb_extract()
     rm -f *.tar.* *deb*
 }
 
-get_binutils()
+install_deb()
 {
-    BINUTILS_PREFIX='https://launchpad.net/~pwntools/+archive/ubuntu/binutils/+files/binutils-'
-    BINUTILS_SUFFIX='-linux-gnu_2.22-6ubuntu1.1cross0.11pwntools12~precise_amd64.deb'
-    local_deb_extract "${BINUTILS_PREFIX}${1}${BINUTILS_SUFFIX}"
-}
-
-get_qemu()
-{
-    echo "Installing qemu"
-    QEMU_INDEX='http://packages.ubuntu.com/en/yakkety/amd64/qemu-user-static/download'
-    QEMU_URL=$(curl "$QEMU_INDEX" | grep kernel.org | grep -Eo 'http.*\.deb')
-    local_deb_extract "$QEMU_URL"
+    version=${2:-zesty}
+    package=$1
+    echo "Installing $package"
+    INDEX="http://packages.ubuntu.com/en/$version/amd64/$package/download"
+    URL=$(curl "$INDEX" | grep -Eo "https?://.*$package.*\.deb" | head -1)
+    local_deb_extract "$URL"
 }
 
 setup_travis()
 {
     export PATH=$PWD/usr/bin:$PATH
-    export LD_LIBRARY_PATH=$PWD/usr/lib
+    export LD_LIBRARY_PATH=$PWD/usr/lib:$LD_LIBRARY_PATH
+    export LD_LIBRARY_PATH=$PWD/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH
 
-    # Install libbfd-multiarch and libopcodes-multiarch if not found in the cache
-    if [ ! -f usr/lib/libbfd-2.22-multiarch.so ];
-    then
-        # Install the multiarch binutils
-        local_deb_extract http://mirrors.mit.edu/ubuntu/pool/universe/b/binutils/binutils-multiarch_2.22-6ubuntu1_amd64.deb
-        pushd usr/lib
-        ln -sf libbfd-2.22-multiarch.so libbfd-2.22.so
-        ln -sf libopcodes-2.22-multiarch.so libopcodes-2.22.so
-        popd
-    fi
+    # Install a more modern binutils, which is required for some of the tests
+    [[ -f usr/bin/objcopy ]] || install_deb binutils
 
     # Install/upgrade qemu
-    if ! (which qemu-arm-static && qemu-arm-static -version | grep 2.6.1); then
-        get_qemu
-    fi
+    [[ -f usr/bin/qemu-arm-static ]] || install_deb qemu-user-static xenial
 
-    # Install our custom binutils
-    which arm-linux-gnu-as     || get_binutils arm
-    which mips-linux-gnu-as    || get_binutils mips
-    which powerpc-linux-gnu-as || get_binutils powerpc
+    # Install cross-binutils
+    [[ -f usr/bin/x86_64-linux-gnu-ar ]]    || install_deb binutils-multiarch
+    [[ -f usr/bin/aarch64-linux-gnu-as ]]   || install_deb binutils-aarch64-linux-gnu
+    [[ -f usr/bin/arm-linux-gnueabihf-as ]] || install_deb binutils-arm-linux-gnueabihf
+    [[ -f usr/bin/mips-linux-gnu-as ]]      || install_deb binutils-mips-linux-gnu
+    [[ -f usr/bin/powerpc-linux-gnu-as ]]   || install_deb binutils-powerpc-linux-gnu
 
     # Test that the installs worked
-    which arm-linux-gnu-as
-    which mips-linux-gnu-as
-    which powerpc-linux-gnu-as
-    which qemu-arm-static
+    as                      --version
+    x86_64-linux-gnu-ar     --version
+    aarch64-linux-gnu-as    --version
+    arm-linux-gnueabihf-as  --version
+    mips-linux-gnu-as       --version
+    powerpc-linux-gnu-as    --version
+    qemu-arm-static         --version
+
+    # Force-install capstone because it's broken somehow
+    [[ -f usr/lib/libcapstone.so.3 ]] || install_deb libcapstone3
+
+    # Install a newer copy of GDB
+    if [[ ! -f usr/bin/gdb ]]; then
+        git clone --depth=1 https://github.com/zachriggle/pwntools-gdb-travis-ci.git
+        tar xf pwntools-gdb-travis-ci/gdb.tar.xz
+        which gdb
+        usr/bin/gdb --version
+    fi
 
     # Get rid of files we don't want cached
     rm -rf usr/share
@@ -74,8 +77,14 @@ setup_android_emulator()
     if [ -n "$TRAVIS" ]; then
         if [ -z "$TRAVIS_COMMIT_RANGE" ]; then
             echo "TRAVIS_COMMIT_RANGE is empty, forcing Android Emulator installation"
-        elif ! git show "$TRAVIS_COMMIT_RANGE" ; then
+        elif ! (git show "$TRAVIS_COMMIT_RANGE" >/dev/null) ; then
             echo "TRAVIS_COMMIT_RANGE is invalid, forcing Android Emulator installation"
+        elif [[ "$TRAVIS_BRANCH" =~ "staging" ]]; then
+            echo "TRAVIS_BRANCH ($TRAVIS_BRANCH) indicates a branch we care about"
+            echo "Forcing Android Emulator installation"
+        elif [[ -n "$TRAVIS_TAG" ]]; then
+            echo "TRAVIS_TAG ($TRAVIS_TAG) indicates a new relase"
+            echo "Forcing Android Emulator installation"
         elif (git log --stat "$TRAVIS_COMMIT_RANGE" | grep -iE "android|adb"); then
             echo "Found Android-related commits, forcing Android Emulator installation"
         else
@@ -83,9 +92,11 @@ setup_android_emulator()
             # emulator, while still leaving the code intact, we remove the
             # RST file that Sphinx searches.
             rm -f 'docs/source/adb.rst'
+            rm -f 'docs/source/protocols/adb.rst'
 
             # However, the file needs to be present or else things break.
             touch 'docs/source/adb.rst'
+            touch 'docs/source/protocols/adb.rst'
 
             echo "Skipping Android emulator install, Android tests disabled."
             return
@@ -182,5 +193,4 @@ elif [[ "$(uname)" == "Linux" ]]; then
     setup_android_emulator
 fi
 
-
-dpkg -l
+set +e

travis/ssh_setup.sh

@@ -54,3 +54,5 @@ Host example.pwnme
 EOF
 
 ssh -o "StrictHostKeyChecking no" -vvvv [email protected] id
+
+set +e