Release 3.3.3

Fixes Gallopsled/pwntools#841
Caused by: Gallopsled/pwntools#504 (9ca377c)

CHANGELOG.md

@@ -9,9 +9,13 @@ The table below shows which release corresponds to each branch, and what date th
 
 | Version          | Branch   | Release Date           |
 | ---------------- | -------- | ---------------------- |
-| [3.4.0](#340)    | `dev`    | Feb 4, 2017 (planned)
-| [3.3.0](#330)    | `beta`   | Dec 24, 2016 (planned)
-| [3.2.0](#320)    | `stable` | Nov 12, 2016
+| [3.5.0](#350)    | `dev`    | Mar 18, 2017 (planned)
+| [3.4.0](#340)    | `beta`   | Feb 4, 2017 (planned)
+| [3.3.2](#332)    | `stable` | Jan 10, 2016
+| [3.3.1](#331)    |          | Jan 10, 2016
+| [3.3.0](#330)    |          | Dec 24, 2016
+| [3.2.1](#321)    |          | Dec 24, 2016
+| [3.2.0](#320)    |          | Nov 12, 2016
 | [3.1.1](#311)    |          | Oct 23, 2016
 | [3.1.0](#310)    |          | Oct 2, 2016
 | [3.0.4](#304)    |          | Sept 19, 2016
@@ -21,21 +25,96 @@ The table below shows which release corresponds to each branch, and what date th
 | [3.0.0](#300)    |          | Aug 20, 2016
 | [2.2.0](#220)    |          | Jan 5, 2015
 
+## 3.5.0
+
+To be released on Mar 18, 2017.
+
 ## 3.4.0
 
 To be released on Feb 4, 2017.
 
+## 3.3.3
+
+- [#843][843] fixed a bug in `amd64.mov`.
+
+[843]: https://github.com/gallopsled/pwntools/pull/843
+
+## 3.3.2
+
+- [#840][840] fixed a regression introduced by [#837][837].
+
+[840]: https://github.com/gallopsled/pwntools/pull/840
+
+## 3.3.1
+
+- [#833][833] Fixed a performance-impacting bug in the adb module.
+- [#837][837] Fixed a bug([#836][836]) causing `hexdump(cyclic=True)` to throw an exception.
+
+[833]: https://github.com/Gallopsled/pwntools/pull/833
+[837]: https://github.com/Gallopsled/pwntools/pull/837
+[836]: https://github.com/Gallopsled/pwntools/issues/836
+
 ## 3.3.0
 
 - [b198ec8][b198ec8] Added `tube.stream()` function, which is like `tube.interact()` without a prompt or keyboard input.
+    + Effectively, this is similar to `cat file` and just prints data as fast as it is received.
+- [aec3fa6][aec3fa6] Disable update checks against GitHub
+    + These checks frequently broke due to GitHub query limits
+- [#757][757] Fixed `adb.wait_for_device()` re-use of the same connection
+- [f9133b1][f9133b1] Add a `STDERR` magic argument to make logging go to `stderr` instead of `stdout`
+    + Usage is e.g. `python foo.py STDERR` or `PWNLIB_STDERR=1 python foo.py`
+    + Also adds `context.log_console` to log to any file or terminal
+- [67e11a9][67e11a9] Add faster error checking to `cyclic()` when provided very large values
+- [5fda658][5fda658] Expose BitPolynom in `globals()`
+- [#765][765] Added `-d` option for hex-escaped output for `shellcraft` command-line tool
+- [#772][772] Fixed bash completion regressions
+- [30c34b7][30c34b7] Fix `ROP.call()` with `Function` objects from `ELF.functions`
+- [fa402ce][fa402ce] Add `adb.uptime` and `adb.boot_time`
+- [82312ba][82312ba] Add `cyclic_metasploit` and `cyclic_metasploit_find`
 
+[757]: https://github.com/Gallopsled/pwntools/pull/757
+[765]: https://github.com/Gallopsled/pwntools/pull/765
+[772]: https://github.com/Gallopsled/pwntools/pull/772
 [b198ec8]: https://github.com/Gallopsled/pwntools/commit/b198ec8
+[aec3fa6]: https://github.com/Gallopsled/pwntools/commit/aec3fa6
+[f9133b1]: https://github.com/Gallopsled/pwntools/commit/f9133b1
+[67e11a9]: https://github.com/Gallopsled/pwntools/commit/67e11a9
+[5fda658]: https://github.com/Gallopsled/pwntools/commit/5fda658
+[30c34b7]: https://github.com/Gallopsled/pwntools/commit/30c34b7
+[fa402ce]: https://github.com/Gallopsled/pwntools/commit/fa402ce
+[82312ba]: https://github.com/Gallopsled/pwntools/commit/82312ba
+
+## 3.2.1
+
+Multiple bug fixes.
+
+- [#783][783] Fix `adb.uninstall` typo
+- [#787][787] Added error handling for `ssh.process` argument `preexec_fn`
+- [#793][793] Fixed progress message in `remote()` when connections failed
+- [#802][802] Fixed partition listing in `adb.partitions`, which accidentally shelled out to the `adb` binary
+- [#804][804] Fix error message for 32-bit distributions
+- [#805][805] Fix exception in `Core.segments` when a segment has no name
+- [#811][811] Fixes and performance improvements for `adb.wait_for_device()`
+- [#813][813] Fixed a release script
+- [#814][814] Fixed exceptions thrown if the `$HOME` directory is not writable
+- [#815][815] Properly handle `None` in `MemLeak`
+
+[783]: https://github.com/Gallopsled/pwntools/pull/783
+[787]: https://github.com/Gallopsled/pwntools/pull/787
+[793]: https://github.com/Gallopsled/pwntools/pull/793
+[802]: https://github.com/Gallopsled/pwntools/pull/802
+[804]: https://github.com/Gallopsled/pwntools/pull/804
+[805]: https://github.com/Gallopsled/pwntools/pull/805
+[811]: https://github.com/Gallopsled/pwntools/pull/811
+[813]: https://github.com/Gallopsled/pwntools/pull/813
+[814]: https://github.com/Gallopsled/pwntools/pull/814
+[815]: https://github.com/Gallopsled/pwntools/pull/815
 
 ## 3.2.0
 
 - [#695][695] Fixed a performance regression in `phd`.
-- [452605e][452605e] Fixed [#629][#629] related to correct removal of temporary files.
-- [ea94ee4][ea94ee4] Disallows semi-colons in for the `run_in_terminal` function, since it did not work properly in all cases. 
+- [452605e][452605e] Fixed [#629][629] related to correct removal of temporary files.
+- [ea94ee4][ea94ee4] Disallows semi-colons in for the `run_in_terminal` function, since it did not work properly in all cases.
 - [6376d07][6376d07] Added the mips shellcode `pushstr_array`.
 - [#700][700] Added missing MIPS shellcode documentation to readthedocs, and enabled unit tests
 - [#701][701] Command line tools refactored to have a common `pwn` entry point.
@@ -67,6 +146,7 @@ To be released on Feb 4, 2017.
     + It is now possible to e.g. `map(lambda d: d.process(['id']).recvall(), adb.devices())`
 
 
+[629]: https://github.com/Gallopsled/pwntools/issues/629
 [695]: https://github.com/Gallopsled/pwntools/pull/695
 [700]: https://github.com/Gallopsled/pwntools/pull/700
 [701]: https://github.com/Gallopsled/pwntools/pull/701

README.md

@@ -2,7 +2,7 @@
 ![pwntools logo](https://github.com/Gallopsled/pwntools/blob/stable/docs/source/logo.png?raw=true)
 
 [![Docs](https://readthedocs.org/projects/pwntools/badge/?version=stable)](https://docs.pwntools.com/)
-[![PyPI](https://img.shields.io/badge/pypi-v3.2.0-green.svg?style=flat)](https://pypi.python.org/pypi/pwntools/)
+[![PyPI](https://img.shields.io/badge/pypi-v3.3.3-green.svg?style=flat)](https://pypi.python.org/pypi/pwntools/)
 [![Travis](https://travis-ci.org/Gallopsled/pwntools.svg)](https://travis-ci.org/Gallopsled/pwntools)
 [![Coveralls](https://img.shields.io/coveralls/Gallopsled/pwntools/dev.svg)](https://coveralls.io/github/Gallopsled/pwntools?branch=dev)
 [![Twitter](https://img.shields.io/badge/twitter-pwntools-4099FF.svg?style=flat)](https://twitter.com/pwntools)

pwnlib/adb/adb.py

@@ -212,7 +212,22 @@ def boot_time():
             return int(value)
 
 class AdbDevice(Device):
-    """Encapsulates information about a connected device."""
+    """Encapsulates information about a connected device.
+
+    Example:
+
+        >>> device = adb.wait_for_device()
+        >>> device.arch
+        'arm'
+        >>> device.bits
+        32
+        >>> device.os
+        'android'
+        >>> device.product
+        'sdk_phone_armv7'
+        >>> device.serial
+        'emulator-5554'
+    """
     def __init__(self, serial, type, port=None, product='unknown', model='unknown', device='unknown', features=None, **kw):
         self.serial  = serial
         self.type    = type
@@ -225,13 +240,45 @@ class AdbDevice(Device):
         if product == 'unknown':
             return
 
-        with context.local(device=serial):
+        # Deferred fields
+        self._initialized = False
+        self._arch = None
+        self._bits = None
+        self._endian = None
+        self._avd = None
+
+    @property
+    def arch(self):
+        self.__do_deferred_initialization()
+        return self._arch
+
+    @property
+    def avd(self):
+        self.__do_deferred_initialization()
+        return self._avd
+
+    @property
+    def bits(self):
+        self.__do_deferred_initialization()
+        return self._bits
+
+    @property
+    def endian(self):
+        self.__do_deferred_initialization()
+        return self._endian
+
+
+    def __do_deferred_initialization(self):
+        if self._initialized:
+            return
+
+        with context.local(device=self.serial):
             abi = str(properties.ro.product.cpu.abi)
             context.clear()
             context.arch = str(abi)
-            self.arch = context.arch
-            self.bits = context.bits
-            self.endian = context.endian
+            self._arch = context.arch
+            self._bits = context.bits
+            self._endian = context.endian
 
         if self.port == 'emulator':
             emulator, port = self.serial.split('-')
@@ -244,7 +291,8 @@ class AdbDevice(Device):
                     self.avd = r.recvline().strip()
             except:
                 pass
-            # r = remote('localhost')
+
+        self._initialized = True
 
     def __str__(self):
         return self.serial
@@ -345,7 +393,9 @@ def wait_for_device(kick=False):
         for device in devices():
             if context.device == device:
                 return device
-            break
+
+            if not serial:
+                break
         else:
             log.error("Could not find any devices")
 
@@ -899,7 +949,7 @@ _android_mk_template = '''
 LOCAL_PATH := $(call my-dir)
 
 include $(CLEAR_VARS)
-LOCAL_MODULE := poc
+LOCAL_MODULE := %(local_module)s
 LOCAL_SRC_FILES := %(local_src_files)s
 
 include $(BUILD_EXECUTABLE)
@@ -929,6 +979,8 @@ def _generate_ndk_project(file_list, abi='arm-v7a', platform_version=21):
     # Create the directories
 
     # Populate Android.mk
+    local_module = os.path.basename(file_list[0])
+    local_module, _ = os.path.splitext(local_module)
     local_src_files = ' '.join(list(map(os.path.basename, file_list)))
     Android_mk = os.path.join(jni_directory, 'Android.mk')
     with open(Android_mk, 'w+') as f:
@@ -963,7 +1015,7 @@ def compile(source):
         abi = 'armeabi-v7a'
         sdk = '21'
 
-        # If we have an atatched device, use its settings.
+        # If we have an attached device, use its settings.
         if context.device:
             abi = str(properties.ro.product.cpu.abi)
             sdk = str(properties.ro.build.version.sdk)
@@ -1000,12 +1052,51 @@ class Partition(object):
         with log.waitfor('Fetching %r partition (%s)' % (self.name, self.path)):
             return read(self.path)
 
+@with_device
+def walk(top, topdown=True):
+    join = os.path.join
+    isdir = lambda x: stat.S_ISDIR(x['mode'])
+    client = Client()
+    names = client.list(top)
+
+    dirs, nondirs = [], []
+    for name, metadata in names.items():
+        if isdir(metadata):
+            dirs.append(name)
+        else:
+            nondirs.append(name)
+
+    if topdown:
+        yield top, dirs, nondirs
+    for name in dirs:
+        new_path = join(top, name)
+        for x in walk(new_path, topdown):
+            yield x
+    if not topdown:
+        yield top, dirs, nondirs
+
+@with_device
+def find(top, name):
+    for root, dirs, files in walk(top):
+        if name in files or name in dirs:
+            yield os.path.join(root, name)
+
+@with_device
+def readlink(path):
+    path = process(['readlink', path]).recvall()
+
+    # Readlink will emit a single newline
+    # We can't use the '-n' flag since old versions don't support it
+    if path.endswith('\n'):
+        path = path[:-1]
+
+    return path
+
 class Partitions(object):
     @property
     @context.quiet
     def by_name_dir(self):
-        cmd = ['shell','find /dev/block/platform -type d -name by-name']
-        return adb(cmd).strip()
+        return next(find('/dev/block/platform','by-name'))
 
     @context.quiet
     def __dir__(self):
@@ -1021,6 +1112,7 @@ class Partitions(object):
             yield name
 
     @context.quiet
+    @with_device
     def __getattr__(self, attr):
         for name in self:
             if name == attr:
@@ -1031,7 +1123,7 @@ class Partitions(object):
         path = os.path.join(self.by_name_dir, name)
 
         # Find the actual path of the device
-        devpath = process(['readlink', '-n', path]).recvall()
+        devpath = readlink(path)
         devname = os.path.basename(devpath)
 
         # Get the size of the partition

pwnlib/adb/protocol.py

@@ -5,6 +5,7 @@ Documentation is available here:
 https://android.googlesource.com/platform/system/core/+/master/adb/protocol.txt
 """
 import functools
+import stat
 import time
 
 from ..context import context
@@ -42,14 +43,7 @@ class Message(object):
 class Connection(remote):
     """Connection to the ADB server"""
     def __init__(self, host, port, level=None, *a, **kw):
-
-        # Try to make sure ADB is running if it's on the default host and port.
-        if host == context.defaults['adb_host'] \
-        and port == context.defaults['adb_port']:
-            process(context.adb + ['start-server'], level='error').wait_for_close()
-
-        with context.quiet:
-            super(Connection, self).__init__(host, port, level=level, *a, **kw)
+        super(Connection, self).__init__(host, port, level=level, *a, **kw)
 
         self._executable = None
         self._argv       = None
@@ -90,6 +84,20 @@ class Client(Logger):
     @property
     def c(self):
         """Client's connection to the ADB server"""
+        if not self._c:
+            try:
+                self._c = Connection(self.host, self.port, level=self.level)
+            except Exception:
+                # If the connection fails, try starting a server on that port
+                # as long as it's the *default* port.
+                if self.host == context.defaults['adb_host'] \
+                and self.port == context.defaults['adb_port']:
+                    log.warn("Could not connect to ADB server, trying to start it")
+                    process(context.adb + ['start-server']).recvall()
+                else:
+                    log.exception('Could not connect to ADB server')
+
+        # Final attempt...
         if not self._c:
             self._c = Connection(self.host, self.port, level=self.level)
         return self._c
@@ -134,12 +142,23 @@ class Client(Logger):
 
     @_autoclose
     def kill(self):
-        """Kills the remote ADB server"""
+        """Kills the remote ADB server"
+
+        >>> c=adb.protocol.Client()
+        >>> c.kill()
+
+        The server is automatically re-started on the next request,
+        if the default host/port are used.
+
+        >>> c.version() > (4,0)
+        True
+        """
         try:
             self.send('host:kill')
         except EOFError:
             pass
 
+    @_autoclose
     def version(self):
         """
         Returns:
@@ -296,8 +315,6 @@ class Client(Logger):
             return rv
         return wrapper
 
-    @_with_transport
-    @_sync
     def list(self, path):
         """Execute the ``LIST`` command of the ``SYNC`` API.
 
@@ -328,8 +345,23 @@ class Client(Logger):
             >>> pprint(adb.Client().list('/data/user'))
             {'0': {'mode': 41471, 'size': 11, 'time': ...}}
             >>> adb.Client().list('/does/not/exist')
-            {}
+            Traceback (most recent call last):
+            ...
+            PwnlibException: Cannot list directory '/does/not/exist': Does not exist
         """
+        st = self.stat(path)
+
+        if not st:
+            log.error("Cannot list directory %r: Does not exist" % path)
+
+        if not stat.S_ISDIR(st['mode']):
+            log.error("Cannot list directory %r: Path is not a directory" % path)
+
+        return self._list(path)
+
+    @_with_transport
+    @_sync
+    def _list(self, path):
         self.c.flat('LIST', len(path), path)
         files = {}
         while True:
@@ -389,9 +421,35 @@ class Client(Logger):
 
         return {'mode': mode, 'size': size, 'time': time}
 
+    def write(self, path, data, mode=0o755, timestamp=None, callback=None):
+        """Execute the ``WRITE`` command of the ``SYNC`` API.
+
+        Arguments:
+            path(str): Path to the file to write
+            data(str): Data to write to the file
+            mode(int): File mode to set (e.g. ``0o755``)
+            timestamp(int): Unix timestamp to set the file date to
+            callback(callable): Callback function invoked as data
+                is written.  Arguments provided are:
+
+                - File path
+                - All data
+                - Expected size of all data
+                - Current chunk
+                - Expected size of chunk
+        """
+        # We must ensure that 'path' is not a directory
+        # Writing to a directory is supported, but creates a temporary file
+        st = self.stat(path)
+
+        if st and stat.S_ISDIR(st['mode']):
+            log.error("Cannot write to %r: Path is a directory" % path)
+
+        return self._write(path, data, mode=0o755, timestamp=None, callback=None)
+
     @_with_transport
     @_sync
-    def write(self, path, data, mode=0o755, timestamp=None, callback=None):
+    def _write(self, path, data, mode=0o755, timestamp=None, callback=None):
         path += ',' + str(mode)
         self.c.flat('SEND', len(path), path)
 

pwnlib/context/__init__.py

@@ -10,6 +10,7 @@ import logging
 import os
 import platform
 import socket
+import stat
 import string
 import subprocess
 import sys
@@ -1126,8 +1127,13 @@ class ContextType(object):
 
     @property
     def adb(self):
-        """Returns an argument array for connecting to adb."""
-        command = ['adb']
+        """Returns an argument array for connecting to adb.
+
+        Unless ``$ADB_PATH`` is set, uses the default ``adb`` binary in ``$PATH``.
+        """
+        ADB_PATH = os.environ.get('ADB_PATH', 'adb')
+
+        command = [ADB_PATH]
 
         if self.adb_host != self.defaults['adb_host']:
             command += ['-H', self.adb_host]
@@ -1140,6 +1146,45 @@ class ContextType(object):
 
         return command
 
+    @property
+    def cache_dir(self):
+        """Directory used for caching data.
+
+        Note:
+            May be either a path string, or ``None``.
+
+        Example:
+
+            >>> cache_dir = context.cache_dir
+            >>> cache_dir is not None
+            True
+            >>> os.chmod(cache_dir, 0o000)
+            >>> context.cache_dir is None
+            True
+            >>> os.chmod(cache_dir, 0o755)
+            >>> cache_dir == context.cache_dir
+            True
+        """
+        home = os.path.expanduser('~')
+
+        if not os.access(home, os.W_OK):
+            return None
+
+        cache = os.path.join(home, '.pwntools-cache')
+
+        if not os.path.exists(cache):
+            try:
+                os.mkdir(cache)
+            except OSError:
+                return None
+
+        # Some wargames e.g. pwnable.kr have created dummy directories
+        # which cannot be modified by the user account (owned by root).
+        if not os.access(cache, os.W_OK):
+            return None
+
+        return cache
+
 
     #*************************************************************************
     #                               ALIASES

pwnlib/elf/corefile.py

@@ -236,7 +236,7 @@ class Core(ELF):
     def libc(self):
         """Return the first mapping in libc"""
         for m in self.mappings:
-            if m.name.startswith('libc') and m.name.endswith('.so'):
+            if m.name and m.name.startswith('libc') and m.name.endswith('.so'):
                 return m
 
     @property

pwnlib/elf/datatypes.py

@@ -539,7 +539,8 @@ class elf_prstatus_i386(ctypes.Structure):
 assert ctypes.sizeof(elf_prstatus_i386) == 0x90
 
 class elf_prstatus_amd64(ctypes.Structure):
-    _fields_ = generate_prstatus_common(64, user_regs_struct_amd64)
+    _fields_ = generate_prstatus_common(64, user_regs_struct_amd64) \
+             + [('padding', ctypes.c_uint32)]
 
 assert ctypes.sizeof(elf_prstatus_amd64) == 0x150
 

pwnlib/log.py

@@ -166,7 +166,6 @@ class Progress(object):
     using :meth:`Logger.progress`.
     """
     def __init__(self, logger, msg, status, level, args, kwargs):
-        global _progressid
         self._logger = logger
         self._msg = msg
         self._status = status
@@ -264,7 +263,7 @@ class Logger(object):
             # This is a minor hack to permit user-defined classes which inherit
             # from a tube (which do not actually reside in the pwnlib library)
             # to receive logging abilities that behave as they would expect from
-            # the resto f the library
+            # the rest of the library
             module = self.__module__
             if not module.startswith('pwnlib'):
                 module = 'pwnlib.' + module

pwnlib/memleak.py

@@ -129,6 +129,8 @@ class MemLeak(object):
         size   = obj.size
         offset = obj.offset
         data   = self.n(address + offset, size)
+        if not data:
+            return None
         return unpack(data, size*8)
 
     def field_compare(self, address, obj, expected):

pwnlib/shellcraft/internal.py

@@ -1,6 +1,8 @@
 import os
 from collections import defaultdict
 
+from ..context import context
+
 __all__ = ['make_function']
 
 loaded = {}
@@ -33,10 +35,14 @@ def init_mako():
 
     render_global = IsInside()
 
+    cache  = context.cache_dir
+    if cache:
+        cache = os.path.join(cache, 'mako')
+
     curdir = os.path.dirname(os.path.abspath(__file__))
     lookup = TemplateLookup(
         directories      = [os.path.join(curdir, 'templates')],
-        module_directory = os.path.expanduser('~/.pwntools-cache/mako')
+        module_directory = cache
     )
 
     # The purpose of this definition is to create a new Tag.

pwnlib/shellcraft/templates/amd64/mov.asm

@@ -61,17 +61,26 @@ Example:
         mov rax, 0x1010110dfac01fe
         xor [rsp], rax
         pop rax
-   >>> with context.local(os = 'linux'):
-   ...     print shellcraft.amd64.mov('eax', 'SYS_read').rstrip()
-       xor eax, eax /* (SYS_read) */
-   >>> with context.local(os = 'freebsd'):
-   ...     print shellcraft.amd64.mov('eax', 'SYS_read').rstrip()
-       push (SYS_read) /* 3 */
-       pop rax
-   >>> with context.local(os = 'linux'):
-   ...     print shellcraft.amd64.mov('eax', 'PROT_READ | PROT_WRITE | PROT_EXEC').rstrip()
-       push (PROT_READ | PROT_WRITE | PROT_EXEC) /* 7 */
-       pop rax
+    >>> print shellcraft.amd64.mov('rax', 0xffffffff).rstrip()
+        mov eax, 0xffffffff
+    >>> print shellcraft.amd64.mov('rax', 0x7fffffff).rstrip()
+        mov eax, 0x7fffffff
+    >>> print shellcraft.amd64.mov('rax', 0x80010101).rstrip()
+        mov eax, 0x80010101
+    >>> print shellcraft.amd64.mov('rax', 0x80000000).rstrip()
+        mov eax, 0x1010101 /* 2147483648 == 0x80000000 */
+        xor eax, 0x81010101
+    >>> with context.local(os = 'linux'):
+    ...     print shellcraft.amd64.mov('eax', 'SYS_read').rstrip()
+        xor eax, eax /* (SYS_read) */
+    >>> with context.local(os = 'freebsd'):
+    ...     print shellcraft.amd64.mov('eax', 'SYS_read').rstrip()
+        push (SYS_read) /* 3 */
+        pop rax
+    >>> with context.local(os = 'linux'):
+    ...     print shellcraft.amd64.mov('eax', 'PROT_READ | PROT_WRITE | PROT_EXEC').rstrip()
+        push (PROT_READ | PROT_WRITE | PROT_EXEC) /* 7 */
+        pop rax
 
 Args:
   dest (str): The destination register.
@@ -99,7 +108,6 @@ if get_register(src):
     if dest.size == 64 and src.size <= 32:
         dest = get_register(dest.native32)
 
-    src_size = src.size
 else:
     with ctx.local(arch = 'amd64'):
         src = eval(src)
@@ -107,9 +115,9 @@ else:
     if not dest.fits(src):
         log.error("cannot mov %s, %r: dest is smaller than src" % (dest, src))
 
-    src_size = bits_required(src)
+    orig_dest = dest
 
-    if dest.size == 64 and src_size <= 32:
+    if dest.size == 64 and bits_required(src) <= 32:
         dest = get_register(dest.native32)
 
     # Calculate the packed version
@@ -117,7 +125,15 @@ else:
 
     # Calculate the unsigned and signed versions
     srcu = packing.unpack(srcp, dest.size, sign=False)
-    srcs = packing.unpack(srcp, dest.size, sign=True)
+
+    # N.B.: We may have downsized the register for e.g. mov('rax', 0xffffffff)
+    #       In this case, srcp is now a 4-byte packed value, which will expand
+    #       to "-1", which isn't correct.
+    if orig_dest.size != dest.size:
+        srcs = src
+    else:
+        srcs = packing.unpack(srcp, dest.size, sign=True)
+
 %>\
 % if is_register(src):
     % if src == dest:

pwnlib/tubes/process.py

@@ -3,6 +3,7 @@ import errno
 import fcntl
 import logging
 import os
+import platform
 import pty
 import resource
 import select
@@ -190,6 +191,9 @@ class process(tube):
         >>> io = process(['sh','-c','sleep 10; exit 7'], alarm=2)
         >>> io.poll(block=True) == -signal.SIGALRM
         True
+
+        >>> binary = ELF.from_assembly('nop', arch='mips')
+        >>> p = process(binary.path)
     """
 
     PTY = PTY
@@ -284,7 +288,7 @@ class process(tube):
         with self.progress(message) as p:
 
             if not self.aslr:
-                log.warn_once("ASLR is disabled!")
+                self.warn_once("ASLR is disabled!")
 
             # In the event the binary is a foreign architecture,
             # and binfmt is not installed (e.g. when running on
@@ -294,12 +298,6 @@ class process(tube):
             executables = [executable]
             exception = None
 
-            try:
-                if not context.native:
-                    qemu = get_qemu_user()
-                    prefixes.append(([qemu], qemu))
-            except: pass
-
             for prefix, executable in prefixes:
                 try:
                     self.proc = subprocess.Popen(args = prefix + argv,
@@ -316,11 +314,7 @@ class process(tube):
                 except OSError as exception:
                     if exception.errno != errno.ENOEXEC:
                         raise
-            else:
-                try:
-                    raise exception
-                except:
-                    log.exception(str(prefixes))
+                    prefixes.append(self.__on_enoexec(exception))
 
         if self.pty is not None:
             if stdin is slave:
@@ -357,7 +351,7 @@ class process(tube):
 
                 resource.setrlimit(resource.RLIMIT_STACK, (-1, -1))
             except:
-                log.exception("Could not disable ASLR")
+                self.exception("Could not disable ASLR")
 
         # Assume that the user would prefer to have core dumps.
         resource.setrlimit(resource.RLIMIT_CORE, (-1, -1))
@@ -381,13 +375,43 @@ class process(tube):
 
         self.preexec_fn()
 
+    def __on_enoexec(self, exception):
+        """We received an 'exec format' error (ENOEXEC)
+
+        This implies that the user tried to execute e.g.
+        an ARM binary on a non-ARM system, and does not have
+        binfmt helpers installed for QEMU.
+        """
+        # Get the ELF binary for the target executable
+        with context.quiet:
+            # XXX: Cyclic imports :(
+            from ..elf import ELF
+            binary = ELF(self.executable)
+
+        # If we're on macOS, this will never work.  Bail now.
+        # if platform.mac_ver()[0]:
+            # self.error("Cannot run ELF binaries on macOS")
+
+        # Determine what architecture the binary is, and find the
+        # appropriate qemu binary to run it.
+        qemu = get_qemu_user(arch=binary.arch)
+        if qemu:
+            args = [qemu]
+            if self.argv:
+                args += ['-0', self.argv[0]]
+            args += ['--']
+            return [args, qemu]
+
+        # If we get here, we couldn't run the binary directly, and
+        # we don't have a qemu which can run it.
+        self.exception(exception)
+
     @property
     def program(self):
         """Alias for ``executable``, for backward compatibility"""
         return self.executable
 
-    @staticmethod
-    def _validate(cwd, executable, argv, env):
+    def _validate(self, cwd, executable, argv, env):
         """
         Perform extended validation on the executable path, argv, and envp.
 
@@ -406,14 +430,14 @@ class process(tube):
             argv = [argv]
 
         if not all(isinstance(arg, (str, unicode)) for arg in argv):
-            log.error("argv must be strings: %r" % argv)
+            self.error("argv must be strings: %r" % argv)
 
         # Create a duplicate so we can modify it
         argv = list(argv or [])
 
         for i, arg in enumerate(argv):
             if '\x00' in arg[:-1]:
-                log.error('Inappropriate nulls in argv[%i]: %r' % (i, arg))
+                self.error('Inappropriate nulls in argv[%i]: %r' % (i, arg))
 
             argv[i] = arg.rstrip('\x00')
 
@@ -425,7 +449,7 @@ class process(tube):
         #
         if not executable:
             if not argv:
-                log.error("Must specify argv or executable")
+                self.error("Must specify argv or executable")
             executable = argv[0]
 
         # Do not change absolute paths to binaries
@@ -445,11 +469,11 @@ class process(tube):
             executable = os.path.join(cwd, executable)
 
         if not os.path.exists(executable):
-            log.error("%r does not exist"  % executable)
+            self.error("%r does not exist"  % executable)
         if not os.path.isfile(executable):
-            log.error("%r is not a file" % executable)
+            self.error("%r is not a file" % executable)
         if not os.access(executable, os.X_OK):
-            log.error("%r is not marked as executable (+x)" % executable)
+            self.error("%r is not marked as executable (+x)" % executable)
 
         #
         # Validate environment
@@ -463,13 +487,13 @@ class process(tube):
 
         for k,v in env.items():
             if not isinstance(k, (str, unicode)):
-                log.error('Environment keys must be strings: %r' % k)
+                self.error('Environment keys must be strings: %r' % k)
             if not isinstance(k, (str, unicode)):
-                log.error('Environment values must be strings: %r=%r' % (k,v))
+                self.error('Environment values must be strings: %r=%r' % (k,v))
             if '\x00' in k[:-1]:
-                log.error('Inappropriate nulls in env key: %r' % (k))
+                self.error('Inappropriate nulls in env key: %r' % (k))
             if '\x00' in v[:-1]:
-                log.error('Inappropriate nulls in env value: %r=%r' % (k, v))
+                self.error('Inappropriate nulls in env value: %r=%r' % (k, v))
 
             env[k.rstrip('\x00')] = v.rstrip('\x00')
 
@@ -779,7 +803,7 @@ class process(tube):
         """
         # If it's running under qemu-user, don't leak anything.
         if 'qemu-' in os.path.realpath('/proc/%i/exe' % self.pid):
-            log.error("Cannot use leaker on binaries under QEMU.")
+            self.error("Cannot use leaker on binaries under QEMU.")
 
         with open('/proc/%i/mem' % self.pid, 'rb') as mem:
             mem.seek(address)

pwnlib/tubes/remote.py

@@ -108,38 +108,30 @@ class remote(sock):
         sock    = None
         timeout = self.timeout
 
-        h = self.waitfor('Opening connection to %s on port %d' % (self.rhost, self.rport))
+        with self.waitfor('Opening connection to %s on port %d' % (self.rhost, self.rport)) as h:
+            for res in socket.getaddrinfo(self.rhost, self.rport, fam, typ, 0, socket.AI_PASSIVE):
+                self.family, self.type, self.proto, _canonname, sockaddr = res
 
-        for res in socket.getaddrinfo(self.rhost, self.rport, fam, typ, 0, socket.AI_PASSIVE):
-            self.family, self.type, self.proto, _canonname, sockaddr = res
+                if self.type not in [socket.SOCK_STREAM, socket.SOCK_DGRAM]:
+                    continue
 
-            if self.type not in [socket.SOCK_STREAM, socket.SOCK_DGRAM]:
-                continue
+                h.status("Trying %s" % sockaddr[0])
 
-            h.status("Trying %s" % sockaddr[0])
+                sock = socket.socket(self.family, self.type, self.proto)
 
-            sock = socket.socket(self.family, self.type, self.proto)
+                if timeout != None and timeout <= 0:
+                    sock.setblocking(0)
+                else:
+                    sock.setblocking(1)
+                    sock.settimeout(timeout)
 
-            if timeout != None and timeout <= 0:
-                sock.setblocking(0)
-            else:
-                sock.setblocking(1)
-                sock.settimeout(timeout)
-
-            try:
-                sock.connect(sockaddr)
-                break
-            except socket.error:
-                pass
-        else:
-            h.failure()
+                try:
+                    sock.connect(sockaddr)
+                    return sock
+                except socket.error:
+                    pass
             self.error("Could not connect to %s on port %d" % (self.rhost, self.rport))
 
-        h.success()
-        return sock
-
-
-
     @classmethod
     def fromsocket(cls, socket):
         """

pwnlib/update.py

@@ -53,7 +53,11 @@ def available_on_pypi(prerelease=current_version.is_prerelease):
 
 def cache_file():
     """Returns the path of the file used to cache update data, and ensures that it exists."""
-    cache_dir  = os.path.expanduser('~/.pwntools-cache')
+    cache_dir = context.cache_dir
+
+    if not cache_dir:
+        return None
+
     cache_file = os.path.join(cache_dir, 'update')
 
     if not os.path.isdir(cache_dir):
@@ -66,12 +70,23 @@ def cache_file():
 
 def last_check():
     """Return the date of the last check"""
-    return os.path.getmtime(cache_file())
+    cache = cache_file()
+    if cache:
+        return os.path.getmtime(cache_file())
+
+    # Fallback
+    return time.time()
 
 def should_check():
     """Return True if we should check for an update"""
-    if read(cache_file()).strip() == 'never':
+    filename = cache_file()
+
+    if not filename:
         return False
+
+    if read(filename).strip() == 'never':
+        return False
+
     return time.time() > (last_check() + update_freq)
 
 def perform_check(prerelease=current_version.is_prerelease):
@@ -109,7 +124,10 @@ def perform_check(prerelease=current_version.is_prerelease):
     where = None
     command = None
 
-    os.utime(cache_file(), None)
+    cache = cache_file()
+
+    if cache:
+        os.utime(cache, None)
 
     if best == current_version:
         log.info("You have the latest version of Pwntools (%s)" % best)

pwnlib/util/fiddling.py

@@ -608,6 +608,22 @@ def hexdump_iter(fd, width=16, skip=True, hexii=False, begin=0, style=None,
     spacer      = ' '
     marker      = (style.get('marker') or (lambda s:s))('│')
 
+    # Total length of the input stream
+    total = 0
+
+    if hasattr(fd, 'len'):
+        total = fd.len
+    else:
+        # Save the current file offset
+        cur = fd.seek(0, os.SEEK_CUR)
+
+        # Determine the total size of the file
+        fd.seek(0, os.SEEK_END)
+        total = fd.tell()
+
+        # Restore the file offset
+        fd.seek(cur, os.SEEK_SET)
+
     if hexii:
         column_sep = ''
         line_fmt   = '%%(offset)08x  %%(hexbytes)-%is│' % (len(column_sep)+(width*byte_width))
@@ -628,7 +644,7 @@ def hexdump_iter(fd, width=16, skip=True, hexii=False, begin=0, style=None,
         cache = [style_byte(chr(b)) for b in range(256)]
 
     if cyclic:
-        update_cyclic_pregenerated(len(s))
+        update_cyclic_pregenerated(total)
 
     numb = 0
     while True:
@@ -637,20 +653,27 @@ def hexdump_iter(fd, width=16, skip=True, hexii=False, begin=0, style=None,
         if chunk == '':
             break
         numb += len(chunk)
+
         # If this chunk is the same as the last unique chunk,
         # use a '*' instead.
-        if skip and \
-           (last_unique == chunk or \
-            (cyclic and sequential_lines(last_unique, chunk))):
+        if skip and last_unique:
+            same_as_last_line = (last_unique == chunk)
+            lines_are_sequential = (cyclic and sequential_lines(last_unique, chunk))
             last_unique = chunk
-            if not skipping:
-                yield '*'
-                skipping = True
-            continue
 
-        # Chunk is unique, save for next iteration
-        last_unique = chunk
+            if same_as_last_line or lines_are_sequential:
+
+                # If we have not already printed a "*", do so
+                if not skipping:
+                    yield '*'
+                    skipping = True
+
+                # Move on to the next chunk
+                continue
+
+        # Chunk is unique, no longer skipping
         skipping = False
+        last_unique = chunk
 
         # Generate contents for line
         hexbytes = ''

pwnlib/version.py

@@ -1 +1 @@
-__version__ = '3.3.0beta0'
+__version__ = '3.3.3'

setup.py

@@ -83,7 +83,7 @@ except ImportError:
 setup(
     name                 = 'pwntools',
     packages             = find_packages(),
-    version              = '3.3.0beta0',
+    version              = '3.3.3',
     data_files           = [('',
                              glob.glob('*.md') + glob.glob('*.txt')),
                             ],

travis/install.sh

@@ -72,7 +72,13 @@ setup_android_emulator()
     # If we are running on Travis CI, and there were no changes to Android
     # or ADB code, then we do not need the emulator
     if [ -n "$TRAVIS" ]; then
-        if ! (git log --stat "$TRAVIS_COMMIT_RANGE" | grep -E "android|adb"); then
+        if [ -z "$TRAVIS_COMMIT_RANGE" ]; then
+            echo "TRAVIS_COMMIT_RANGE is empty, forcing Android Emulator installation"
+        elif ! git show "$TRAVIS_COMMIT_RANGE" ; then
+            echo "TRAVIS_COMMIT_RANGE is invalid, forcing Android Emulator installation"
+        elif (git log --stat "$TRAVIS_COMMIT_RANGE" | grep -iE "android|adb"); then
+            echo "Found Android-related commits, forcing Android Emulator installation"
+        else
             # In order to avoid running the doctests that require the Android
             # emulator, while still leaving the code intact, we remove the
             # RST file that Sphinx searches.

travis/update_demo.sh

@@ -11,5 +11,5 @@ if [ "$TRAVIS_BRANCH" = "stable" ] && [ ! -z "$TRY_KEY" ]; then
     ssh-add           ~/.ssh/try_key
 
     # Update!
-    ssh [email protected] -- 'docker/update'
+    ssh -o "StrictHostKeyChecking no" [email protected] -- 'docker/update'
 fi