Release 3.10.0

* Fix corefile environment variables and speed up ELF.string()

* Even more doctests

* Fix doctest arch

* No longer split on = since it isn't necessary

* Fix arch again

* Delete intermediate corefiles

* Do not flag for extended testing when the commit SHA is what matches

.github/ISSUE_TEMPLATE.md

@@ -4,6 +4,11 @@ Thanks for contributing to Pwntools!
 
 When reporting an issue, be sure that you are running the latest released version of pwntools (`pip install --upgrade pwntools`).
 
-Please verify that your issue occurs on 64-bit Ubuntu 14.04.  You can use the Dockerfile in `extra/docker/Dockerfile` to save time.
+Please verify that your issue occurs on 64-bit Ubuntu 14.04.  You can use the Dockerfile on `docker.io` for quick testing.
+
+```
+$ docker pull pwntools/pwntools:stable
+$ docker run -it pwntools/pwntools:stable
+```
 
 If possible, provide a proof-of-concept which demonstrates the problem.  Include any binaries or scripts necessary to reproduce the issue, and please include the full debug output via setting the environment variable `PWNLIB_DEBUG=1`.

.travis.yml

@@ -1,10 +1,9 @@
-sudo: false
+dist: precise
 language: python
 addons:
   apt:
     packages:
     - gcc-multilib
-    - gcc-4.6-arm-linux-gnueabihf
     - lib32stdc++6
     - ash
     - bash
@@ -18,8 +17,8 @@ cache:
     - pip
     - directories:
         - usr
-        - /home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/
-        - /home/travis/virtualenv/python2.7.9/bin/
+        - /home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/
+        - /home/travis/virtualenv/python2.7.13/bin/
 python:
   - "2.7"
 before_install:

CHANGELOG.md

@@ -9,9 +9,13 @@ The table below shows which release corresponds to each branch, and what date th
 
 | Version          | Branch   | Release Date           |
 | ---------------- | -------- | ---------------------- |
-| [3.10.0](#3100)  | `dev`    | Oct 21, 2017 (planned)
-| [3.9.0](#390)    | `beta`   | Sep 9, 2017 (planned)
-| [3.8.0](#380)    | `stable` | Jul 29, 2017
+| [3.12.0](#3120)  | `dev`    | Jan 13, 2018 (planned)
+| [3.11.0](#3110)  | `beta`   | Dec 2, 2017 (planned)
+| [3.10.0](#3100)  | `stable` | Oct 25, 2017
+| [3.9.2](#392)    |          | Oct 5, 2017
+| [3.9.1](#391)    |          | Sep 28, 2017
+| [3.9.0](#390)    |          | Sep 11, 2017
+| [3.8.0](#380)    |          | Jul 29, 2017
 | [3.7.1](#371)    |          | Jul 14, 2017
 | [3.7.0](#370)    |          | Jun 19, 2017
 | [3.6.1](#361)    |          | May 12, 2017
@@ -36,13 +40,65 @@ The table below shows which release corresponds to each branch, and what date th
 | [3.0.0](#300)    |          | Aug 20, 2016
 | [2.2.0](#220)    |          | Jan 5, 2015
 
+## 3.12.0
+
+To be released on Jan 13, 2018.
+
+## 3.11.0
+
+To be released on Dec 2, 2017.
+
+- [#1044][1044] Enhancements to ROP
+    + Much better support for 64-bit Intel (amd64) ROP
+    + ROP gadget selection is optimized to favor multi-pops instead of multiple single-pop gadgets
+    + Added support for blacklisting byte values in ROP gadget addresses
+- [#1049][1049] Enhancements to `cyclic`
+    + `context` now has two additional attributes, `cyclic_alphabet` and `cyclic_length`, which correspond to the arguments `alphabet` and `n` to `cyclic()` and `cyclic_find()` and related routines.
+    + The motivation for this change is to allow setting the `alphabet` globally, so that any padding / patterns generated internally to pwntools can be controlled.  The specific motivation is blacklisting values in ROP padding.
+- [#1052][1052] Enhancements for detecting `QEMU_LD_PREFIX` used by QEMU user-mode emulation for sysroots
+
+[1044]: https://github.com/Gallopsled/pwntools/pull/1044
+[1049]: https://github.com/Gallopsled/pwntools/pull/1049
+[1052]: https://github.com/Gallopsled/pwntools/pull/1052
+
 ## 3.10.0
 
-To be released on Sep 8, 2017.
+- [#1007][1007] Add support for setting a `gdbinit` file in the context
+- [#1055][1055] Fixes for `Corefile` stack parsing, speed up `ELF.string()`
+- [#1057][1057] Fix a variable name typo in `DynELF` logging which results in an exception being thrown
+- [#1058][1058] Fix an edge case in `ssh_process.exe`
+
+[1007]: https://github.com/Gallopsled/pwntools/pull/1007
+[1055]: https://github.com/Gallopsled/pwntools/pull/1055
+[1057]: https://github.com/Gallopsled/pwntools/pull/1057
+[1058]: https://github.com/Gallopsled/pwntools/pull/1058
+
+
+## 3.9.2
+
+- [#1043][1043] Do not attempt to populate the libraries used by statically-linked binaries
+
+[1043]: https://github.com/Gallopsled/pwntools/pull/1043
+
+## 3.9.1
+
+- [#1038][1038] Fix an issue with `process()` where glibc would buffer data internally, causing a hang on `select()`
+- [#1036][1036] Fix Travis CI logging verbosity
+- [#1029][1029] Fix some `unicode` issues when using the `readline` command history in `tube.interactive()`
+
+[1038]: https://github.com/Gallopsled/pwntools/pull/1038
+[1036]: https://github.com/Gallopsled/pwntools/pull/1036
+[1029]: https://github.com/Gallopsled/pwntools/pull/1029
 
 ## 3.9.0
 
-To be released on Sep 8, 2017.
+- [#1003][1003] Make `concat_all` faster while also simplifying it's logic
+- [#1014][1014] Fix for overwritten env when parsing core file
+- [#1023][1023] Fixes to Travis CI
+
+[1003]: https://github.com/Gallopsled/pwntools/pull/1003
+[1014]: https://github.com/Gallopsled/pwntools/pull/1014
+[1023]: https://github.com/Gallopsled/pwntools/pull/1023
 
 ## 3.8.0
 

README.md

@@ -2,7 +2,7 @@
 ![pwntools logo](https://github.com/Gallopsled/pwntools/blob/stable/docs/source/logo.png?raw=true)
 
 [![Docs](https://readthedocs.org/projects/pwntools/badge/?version=stable)](https://docs.pwntools.com/)
-[![PyPI](https://img.shields.io/badge/pypi-v3.8.0-green.svg?style=flat)](https://pypi.python.org/pypi/pwntools/)
+[![PyPI](https://img.shields.io/badge/pypi-v3.10.0-green.svg?style=flat)](https://pypi.python.org/pypi/pwntools/)
 [![Travis](https://travis-ci.org/Gallopsled/pwntools.svg)](https://travis-ci.org/Gallopsled/pwntools)
 [![Coveralls](https://img.shields.io/coveralls/Gallopsled/pwntools/dev.svg)](https://coveralls.io/github/Gallopsled/pwntools?branch=dev)
 [![Twitter](https://img.shields.io/badge/twitter-pwntools-4099FF.svg?style=flat)](https://twitter.com/pwntools)

docs/source/conf.py

@@ -44,6 +44,23 @@ extensions = [
     'sphinxcontrib.napoleon'
 ]
 
+# Disable "info" logging directly to stdout by Sphinx
+import logging
+
+class SphinxPwnlibFilter(logging.Filter):
+    def filter(self, record):
+        if record.name.startswith('pwn'):
+            return False
+        if record.name.startswith('paramiko'):
+            return False
+        return True
+
+log_filter = SphinxPwnlibFilter()
+
+for i, handler in enumerate(logging.root.handlers):
+    print("Filtering Sphinx handler", handler)
+    handler.addFilter(log_filter)
+
 # Napoleon settings
 napoleon_use_ivar = True
 napoleon_use_rtype = False

pwnlib/context/__init__.py

@@ -150,7 +150,7 @@ class _Tls_DictStack(threading.local, _DictStack):
 
 def _validator(validator):
     """
-    Validator that tis tightly coupled to the implementation
+    Validator that is tightly coupled to the implementation
     of the classes here.
 
     This expects that the object has a ._tls property which
@@ -340,6 +340,7 @@ class ContextType(object):
         'delete_corefiles': False,
         'device': os.getenv('ANDROID_SERIAL', None) or None,
         'endian': 'little',
+        'gdbinit': "",
         'kernel': None,
         'log_level': logging.INFO,
         'log_file': _devnull(),
@@ -1231,6 +1232,23 @@ class ContextType(object):
         """
         return bool(v)
 
+
+    @_validator
+    def gdbinit(self, value):
+        """Path to the gdbinit that is used when running GDB locally.
+
+        This is useful if you want pwntools-launched GDB to include some additional modules,
+        like PEDA but you do not want to have GDB include them by default.
+
+        The setting will only apply when GDB is launched locally since remote hosts may not have
+        the necessary requirements for the gdbinit.
+
+        If set to an empty string, GDB will use the default `~/.gdbinit`.
+
+        Default value is ``""``.
+        """
+        return str(value)
+
     #*************************************************************************
     #                               ALIASES
     #*************************************************************************

pwnlib/dynelf.py

@@ -417,7 +417,7 @@ class DynELF(object):
     def _find_linkmap(self, pltgot=None, debug=None):
         """
         The linkmap is a chained structure created by the loader at runtime
-        which contains information on the names and load addresses osf all
+        which contains information on the names and load addresses of all
         libraries.
 
         For non-RELRO binaries, a pointer to this is stored in the .got.plt
@@ -431,7 +431,7 @@ class DynELF(object):
         Got     = {32: elf.Elf_i386_GOT, 64: elf.Elf_x86_64_GOT}[self.elfclass]
         r_debug = {32: elf.Elf32_r_debug, 64: elf.Elf64_r_debug}[self.elfclass]
 
-        result = None
+        linkmap = None
 
         if not pltgot:
             w.status("Finding linkmap: DT_PLTGOT")
@@ -439,24 +439,24 @@ class DynELF(object):
 
         if pltgot:
             w.status("GOT.linkmap")
-            result = self.leak.field(pltgot, Got.linkmap)
-            w.status("GOT.linkmap %#x" % result)
+            linkmap = self.leak.field(pltgot, Got.linkmap)
+            w.status("GOT.linkmap %#x" % linkmap)
 
-        if not result:
+        if not linkmap:
             debug = debug or self._find_dt(constants.DT_DEBUG)
             if debug:
                 w.status("r_debug.linkmap")
-                result = self.leak.field(debug, r_debug.r_map)
-                w.status("r_debug.linkmap %#x" % result)
+                linkmap = self.leak.field(debug, r_debug.r_map)
+                w.status("r_debug.linkmap %#x" % linkmap)
 
-        if not (pltgot or debug):
+        if not linkmap:
             w.failure("Could not find DT_PLTGOT or DT_DEBUG")
             return None
 
-        result = self._make_absolute_ptr(result)
+        linkmap = self._make_absolute_ptr(linkmap)
 
-        w.success('%#x' % result)
-        return result
+        w.success('%#x' % linkmap)
+        return linkmap
 
     def waitfor(self, msg):
         if not self._waitfor:
@@ -523,6 +523,7 @@ class DynELF(object):
 
         Arguments:
             symb(str): Named routine to look up
+              If omitted, the base address of the library will be returned.
             lib(str): Substring to match for the library name.
               If omitted, the current library is searched.
               If set to ``'libc'``, ``'libc.so'`` is assumed.
@@ -566,7 +567,6 @@ class DynELF(object):
             # Try a quick lookup by build ID
             self.status("Trying lookup based on Build ID")
             build_id = dynlib._lookup_build_id(lib=lib)
-            result   = None
             if build_id:
                 log.info("Trying lookup based on Build ID: %s" % build_id)
                 path = libcdb.search_by_build_id(build_id)
@@ -575,11 +575,10 @@ class DynELF(object):
                         e = ELF(path)
                         e.address = dynlib.libbase
                         result = e.symbols[symb]
-
-            if not result:
-                self.status("Trying remote lookup")
-                result = dynlib._lookup(symb)
-        else:
+        if symb and not result:
+            self.status("Trying remote lookup")
+            result = dynlib._lookup(symb)
+        if not symb:
             result = dynlib.libbase
 
         #
@@ -839,6 +838,9 @@ class DynELF(object):
     def _lookup_build_id(self, lib = None):
 
         libbase = self.libbase
+        if not self.link_map:
+            self.status("No linkmap found")
+            return None
 
         if lib is not None:
             libbase = self.lookup(symb = None, lib = lib)

pwnlib/elf/corefile.py

@@ -389,7 +389,7 @@ class Corefile(ELF):
 
         >>> core.vdso.data[:4]
         '\x7fELF'
-        >>> core.libc # docteset: +ELLIPSIS
+        >>> core.libc # doctest: +ELLIPSIS
         Mapping('/lib/x86_64-linux-gnu/libc-...', ...)
 
         The corefile also contains a :attr:`.Corefile.stack` property, which gives
@@ -421,7 +421,7 @@ class Corefile(ELF):
 
         >>> s = ssh('travis', 'example.pwnme')
         >>> _ = s.set_working_directory()
-        >>> elf = ELF.from_assembly('int3')
+        >>> elf = ELF.from_assembly(shellcraft.trap())
         >>> path = s.upload(elf.path)
         >>> _ =s.chmod('+x', path)
         >>> io = s.process(path)
@@ -429,6 +429,46 @@ class Corefile(ELF):
         -1
         >>> io.corefile.signal == signal.SIGTRAP # doctest: +SKIP
         True
+
+    Tests:
+
+        These are extra tests not meant to serve as examples.
+
+        Corefile.getenv() works correctly, even if the environment variable's
+        value contains embedded '='. Corefile is able to find the stack, even
+        if the stack pointer doesn't point at the stack.
+
+        >>> elf = ELF.from_assembly(shellcraft.crash())
+        >>> io = elf.process(env={'FOO': 'BAR=BAZ'})
+        >>> io.wait()
+        >>> core = io.corefile
+        >>> core.getenv('FOO')
+        'BAR=BAZ'
+        >>> core.sp == 0
+        True
+        >>> core.sp in core.stack
+        False
+
+        Corefile gracefully handles the stack being filled with garbage, including
+        argc / argv / envp being overwritten.
+
+        >>> context.clear(arch='i386')
+        >>> assembly = '''
+        ... LOOP:
+        ...   mov dword ptr [esp], 0x41414141
+        ...   pop eax
+        ...   jmp LOOP
+        ... '''
+        >>> elf = ELF.from_assembly(assembly)
+        >>> io = elf.process()
+        >>> io.wait()
+        >>> core = io.corefile
+        >>> core.argc, core.argv, core.env
+        (0, [], {})
+        >>> core.stack.data.endswith('AAAA')
+        True
+        >>> core.fault_addr == core.sp
+        True
     """
 
     _fill_gaps = False
@@ -454,6 +494,9 @@ class Corefile(ELF):
         #: variable.
         #:
         #: Note: Use with the :meth:`.ELF.string` method to extract them.
+        #:
+        #: Note: If FOO=BAR is in the environment, self.env['FOO'] is the
+        #:       address of the string "BAR\x00".
         self.env = {}
 
         #: :class:`list`: List of addresses of arguments on the stack.
@@ -545,7 +588,7 @@ class Corefile(ELF):
                         log.warn('Could not find the stack!')
                         self.stack = None
 
-            with context.local(bytes=self.bytes, log_level='error'):
+            with context.local(bytes=self.bytes, log_level='warn'):
                 try:
                     self._parse_stack()
                 except ValueError:
@@ -664,7 +707,22 @@ class Corefile(ELF):
 
     @property
     def signal(self):
-        """:class:`int`: Signal which caused the core to be dumped."""
+        """:class:`int`: Signal which caused the core to be dumped.
+
+        Example:
+
+            >>> elf = ELF.from_assembly(shellcraft.trap())
+            >>> io = elf.process()
+            >>> io.wait()
+            >>> io.corefile.signal == signal.SIGTRAP
+            True
+
+            >>> elf = ELF.from_assembly(shellcraft.crash())
+            >>> io = elf.process()
+            >>> io.wait()
+            >>> io.corefile.signal == signal.SIGSEGV
+            True
+        """
         if self.siginfo:
             return int(self.siginfo.si_signo)
         if self.prstatus:
@@ -675,7 +733,17 @@ class Corefile(ELF):
         """:class:`int`: Address which generated the fault, for the signals
             SIGILL, SIGFPE, SIGSEGV, SIGBUS.  This is only available in native
             core dumps created by the kernel.  If the information is unavailable,
-            this returns the address of the instruction pointer."""
+            this returns the address of the instruction pointer.
+
+
+        Example:
+
+            >>> elf = ELF.from_assembly('mov eax, 0xdeadbeef; jmp eax', arch='i386')
+            >>> io = elf.process()
+            >>> io.wait()
+            >>> io.corefile.fault_addr == io.corefile.eax == 0xdeadbeef
+            True
+        """
         if self.siginfo:
             return int(self.siginfo.sigfault_addr)
 
@@ -826,6 +894,10 @@ class Corefile(ELF):
         # find a pointer to this address, followed by a NULL.
         last_env_addr = address + 1
         p_last_env_addr = stack.find(pack(last_env_addr), None, last_env_addr)
+        if p_last_env_addr < 0:
+            # Something weird is happening.  Just don't touch it.
+            log.warn_once("Found bad environment at %#x", last_env_addr)
+            return
 
         # Sanity check that we did correctly find the envp NULL terminator.
         envp_nullterm = p_last_env_addr+context.bytes
@@ -844,11 +916,31 @@ class Corefile(ELF):
         # Now we can fill in the environment
         env_pointer_data = stack[start_of_envp:p_last_env_addr+self.bytes]
         for pointer in unpack_many(env_pointer_data):
-            end = stack.find('=', last_env_addr)
 
-            if pointer in stack and end in stack:
-                name = stack[pointer:end]
-                self.env[name] = pointer
+            # If the stack is corrupted, the pointer will be outside of
+            # the stack.
+            if pointer not in stack:
+                continue
+
+            try:
+                name_value = self.string(pointer)
+            except Exception:
+                continue
+
+            name, value = name_value.split('=', 1)
+
+            # "end" points at the byte after the null terminator
+            end = pointer + len(name_value) + 1
+
+            # Do not mark things as environment variables if they point
+            # outside of the stack itself, or we had to cross into a different
+            # mapping (after the stack) to read it.
+            # This may occur when the entire stack is filled with non-NUL bytes,
+            # and we NULL-terminate on a read failure in .string().
+            if end not in stack:
+                continue
+
+            self.env[name] = pointer + len(name) + len('=')
 
         # May as well grab the arguments off the stack as well.
         # argc comes immediately before argv[0] on the stack, but
@@ -900,15 +992,32 @@ class Corefile(ELF):
 
         Returns:
             :class:`str`: The contents of the environment variable.
+
+        Example:
+
+            >>> elf = ELF.from_assembly(shellcraft.trap())
+            >>> io = elf.process(env={'GREETING': 'Hello!'})
+            >>> io.wait()
+            >>> io.corefile.getenv('GREETING')
+            'Hello!'
         """
         if name not in self.env:
             log.error("Environment variable %r not set" % name)
 
-        return self.string(self.env[name]).split('=',1)[-1]
+        return self.string(self.env[name])
 
     @property
     def registers(self):
-        """:class:`dict`: All available registers in the coredump."""
+        """:class:`dict`: All available registers in the coredump.
+
+        Example:
+
+            >>> elf = ELF.from_assembly('mov eax, 0xdeadbeef;' + shellcraft.trap(), arch='i386')
+            >>> io = elf.process()
+            >>> io.wait()
+            >>> io.corefile.registers['eax'] == 0xdeadbeef
+            True
+        """
         if not self.prstatus:
             return {}
 
@@ -1016,6 +1125,7 @@ class CorefileFinder(object):
             new_path = 'core.%i' % core_pid
             if core_pid > 0 and new_path != self.core_path:
                 write(new_path, self.read(self.core_path))
+                self.unlink(self.core_path)
                 self.core_path = new_path
 
         # Check the PID

pwnlib/elf/elf.py

@@ -55,6 +55,7 @@ from elftools.elf.enums import ENUM_P_TYPE
 from elftools.elf.gnuversions import GNUVerDefSection
 from elftools.elf.relocation import RelocationSection
 from elftools.elf.sections import SymbolTableSection
+from elftools.elf.segments import InterpSegment
 
 import intervaltree
 
@@ -605,10 +606,20 @@ class ELF(ELFFile):
         >>> any(map(lambda x: 'libc' in x, bash.libs.keys()))
         True
         """
-        if not self.get_section_by_name('.dynamic'):
+
+        # We need a .dynamic section for dynamically linked libraries
+        if not self.get_section_by_name('.dynamic') or self.statically_linked:
             self.libs= {}
             return
 
+        # We must also specify a 'PT_INTERP', otherwise it's a 'statically-linked'
+        # binary which is also position-independent (and as such has a .dynamic).
+        for segment in self.iter_segments_by_type('PT_INTERP'):
+            break
+        else:
+            self.libs = {}
+            return
+
         try:
             cmd = 'ulimit -s unlimited; LD_TRACE_LOADED_OBJECTS=1 LD_WARN=1 LD_BIND_NOW=1 %s 2>/dev/null' % sh_string(self.path)
 
@@ -1675,16 +1686,33 @@ class ELF(ELFFile):
         return packing.unpack(self.read(address, context.bytes), *a, **kw)
 
     def string(self, address):
-        """Reads a null-terminated string from the specified ``address``"""
+        """string(address) -> str
+
+        Reads a null-terminated string from the specified ``address``
+
+        Returns:
+            A ``str`` with the string contents (NUL terminator is omitted),
+            or an empty string if no NUL terminator could be found.
+        """
         data = ''
         while True:
-            c = self.read(address, 1)
+            read_size = 0x1000
+            partial_page = address & 0xfff
+
+            if partial_page:
+                read_size -= partial_page
+
+            c = self.read(address, read_size)
+
             if not c:
                 return ''
-            if c == '\x00':
-                return data
+
             data += c
-            address += 1
+
+            if '\x00' in c:
+                return data[:data.index('\x00')]
+
+            address += len(c)
 
     def flat(self, address, *a, **kw):
         """Writes a full array of values to the specified address.

pwnlib/gdb.py

@@ -646,6 +646,10 @@ def attach(target, gdbscript = None, exe = None, need_ptrace_scope = True, gdb_a
         cmd += ' '
         cmd += ' '.join(gdb_args)
 
+    if context.gdbinit:
+        cmd += ' -nh '                     # ignore ~/.gdbinit
+        cmd += ' -x %s ' % context.gdbinit # load custom gdbinit
+
     cmd += ' -q '
 
     if exe and context.native:

pwnlib/shellcraft/templates/arm/trap.asm

@@ -1,8 +1,2 @@
 <%docstring>A trap instruction.</%docstring>
-    nop
-    nop
-    nop
-    mov  r0, 12
-    mov  r1, sp
-    BKPT 0xab
-    B .
+    BKPT

pwnlib/shellcraft/templates/i386/linux/connect.asm

@@ -24,7 +24,7 @@ Examples:
     'Hello'
 
     >>> l = listen(fam='ipv6', timeout=5)
-    >>> assembly   = shellcraft.i386.linux.connect('ip6-localhost', l.lport, 'ipv6')
+    >>> assembly = shellcraft.i386.linux.connect('::1', l.lport, 'ipv6')
     >>> p = run_assembly(assembly)
     >>> assert l.wait_for_connection()
 

pwnlib/term/readline.py

@@ -1,3 +1,5 @@
+#!/usr/bin/env python2
+# -*- coding: utf-8 -*-
 from __future__ import absolute_import
 
 from pwnlib.term import keyconsts as kc
@@ -7,7 +9,7 @@ from pwnlib.term import text
 
 cursor = text.reverse
 
-buffer_left, buffer_right = u'', u''
+buffer_left, buffer_right = '', ''
 saved_buffer = None
 history = []
 history_idx = None
@@ -29,6 +31,14 @@ suggest_hook = None
 
 tabs = 0
 
+def force_to_bytes(data):
+    if isinstance(data, bytes):
+        return data
+    try:
+        return data.encode('utf-8')
+    except Exception:
+        return data.encode('latin-1')
+
 def set_completer(completer):
     global complete_hook, suggest_hook
     if completer is None:
@@ -80,7 +90,7 @@ def handle_keypress(trace):
 
 def clear():
     global buffer_left, buffer_right, history_idx, search_idx
-    buffer_left, buffer_right = u'', u''
+    buffer_left, buffer_right = '', ''
     history_idx = None
     search_idx = None
     redisplay()
@@ -141,7 +151,7 @@ def cancel_search(*_):
 def commit_search():
     global search_idx
     if search_idx is not None and search_results:
-        set_buffer(history[search_results[search_idx][0]], u'')
+        set_buffer(history[search_results[search_idx][0]], '')
         search_idx = None
         redisplay()
 
@@ -169,7 +179,7 @@ def update_search_results():
 def search_history(*_):
     global buffer_left, buffer_right, history_idx, search_idx
     if search_idx is None:
-        buffer_left, buffer_right = buffer_left + buffer_right, u''
+        buffer_left, buffer_right = buffer_left + buffer_right, ''
         history_idx = None
         search_idx = 0
         update_search_results()
@@ -187,7 +197,7 @@ def history_prev(*_):
         history_idx = -1
     if history_idx < len(history) - 1:
         history_idx += 1
-        set_buffer(history[history_idx], u'')
+        set_buffer(history[history_idx], '')
 
 def history_next(*_):
     global history_idx, saved_buffer
@@ -200,7 +210,7 @@ def history_next(*_):
         saved_buffer = None
     else:
         history_idx -= 1
-        set_buffer(history[history_idx], u'')
+        set_buffer(history[history_idx], '')
 
 def backward_char(*_):
     global buffer_left, buffer_right
@@ -319,11 +329,11 @@ def forward_word(*_):
 
 def go_beginning(*_):
     commit_search()
-    set_buffer(u'', buffer_left + buffer_right)
+    set_buffer('', buffer_left + buffer_right)
 
 def go_end(*_):
     commit_search()
-    set_buffer(buffer_left + buffer_right, u'')
+    set_buffer(buffer_left + buffer_right, '')
 
 keymap = km.Keymap({
     '<nomatch>'   : self_insert,
@@ -382,10 +392,10 @@ def readline(_size = None, prompt = '', float = True, priority = 10):
                 if eof:
                     return ''
                 else:
-                    buffer = (buffer_left + buffer_right).encode('utf-8')
+                    buffer = (buffer_left + buffer_right)
                     if buffer:
                         history.insert(0, buffer)
-                    return buffer + '\n'
+                    return force_to_bytes(buffer) + '\n'
             except KeyboardInterrupt:
                 control_c()
     finally:

pwnlib/tubes/process.py

@@ -336,11 +336,11 @@ class process(tube):
 
         if self.pty is not None:
             if stdin is slave:
-                self.proc.stdin = os.fdopen(os.dup(master), 'r+')
+                self.proc.stdin = os.fdopen(os.dup(master), 'r+', 0)
             if stdout is slave:
-                self.proc.stdout = os.fdopen(os.dup(master), 'r+')
+                self.proc.stdout = os.fdopen(os.dup(master), 'r+', 0)
             if stderr is slave:
-                self.proc.stderr = os.fdopen(os.dup(master), 'r+')
+                self.proc.stderr = os.fdopen(os.dup(master), 'r+', 0)
 
             os.close(master)
             os.close(slave)

pwnlib/tubes/ssh.py

@@ -341,7 +341,8 @@ class ssh_process(ssh_channel):
         libs = self.parent.libs(self.executable)
 
         for lib in libs:
-            if self.executable in lib:
+            # Cannot just check "executable in lib", see issue #1047
+            if lib.endswith(self.executable):
                 return pwnlib.elf.elf.ELF(lib)
 
 

pwnlib/util/lists.py

@@ -114,17 +114,15 @@ def concat_all(*args):
        [0, 1, 2, 3, 4, 5, 6]
     """
 
-    if len(args) != 1:
-        return concat_all(list(args))
+    def go(arg, output):
+        if isinstance(arg, (tuple, list)):
+            for e in arg:
+                go(e, output)
+        else:
+            output.append(arg)
+        return output
 
-    if not isinstance(args[0], (tuple, list)):
-        return [args[0]]
-
-    res = []
-    for k in args[0]:
-        res.extend(concat_all(k))
-
-    return res
+    return go(args, [])
 
 def ordlist(s):
     """ordlist(s) -> list

pwnlib/version.py

@@ -1 +1 @@
-__version__ = '3.8.0'
+__version__ = '3.10.0'

setup.py

@@ -86,7 +86,7 @@ except Exception as e:
 setup(
     name                 = 'pwntools',
     packages             = find_packages(),
-    version              = '3.8.0',
+    version              = '3.10.0',
     data_files           = [('',
                              glob.glob('*.md') + glob.glob('*.txt')),
                             ],

travis/docker/Dockerfile

@@ -8,7 +8,10 @@ MAINTAINER Maintainer Gallopsled et al.
 
 USER root
 
+RUN apt-get update
+
 # Use UTF-8
+RUN apt-get install -y locales
 RUN locale-gen en_US.UTF-8
 ENV LANG en_US.UTF-8
 ENV LANGUAGE en_US:en

travis/install.sh

@@ -1,5 +1,5 @@
 #!/usr/bin/env bash -e
-set -e
+set -ex
 
 local_deb_extract()
 {
@@ -15,7 +15,7 @@ install_deb()
     package=$1
     echo "Installing $package"
     INDEX="http://packages.ubuntu.com/en/$version/amd64/$package/download"
-    URL=$(curl "$INDEX" | grep -Eo "https?://.*$package.*\.deb" | head -1)
+    URL=$(curl -L "$INDEX" | grep -Eo "https?://.*$package.*\.deb" | head -1)
     local_deb_extract "$URL"
 }
 
@@ -90,7 +90,7 @@ setup_android_emulator()
         elif [[ -n "$TRAVIS_TAG" ]]; then
             echo "TRAVIS_TAG ($TRAVIS_TAG) indicates a new relase"
             echo "Forcing Android Emulator installation"
-        elif (git log --stat "$TRAVIS_COMMIT_RANGE" | grep -iE "android|adb"); then
+        elif (git log --stat "$TRAVIS_COMMIT_RANGE" | grep -iE "android|adb" | grep -v "commit "); then
             echo "Found Android-related commits, forcing Android Emulator installation"
         else
             # In order to avoid running the doctests that require the Android
@@ -198,4 +198,4 @@ elif [[ "$(uname)" == "Linux" ]]; then
     setup_android_emulator
 fi
 
-set +e
+set +ex

travis/ssh_setup.sh

@@ -4,7 +4,7 @@
 #
 # All of the "conditional sudo" is to do container-based builds on
 # Travis which are much, much faster.
-set -e
+set -ex
 
 U=travis
 H=/home/$U
@@ -55,4 +55,4 @@ EOF
 
 ssh -o "StrictHostKeyChecking no" -vvvv [email protected] id
 
-set +e
+set +ex