Release 4.8.0

This fixes a misbehaviour in environments where `XDG_CACHE_DIR` is not
set and `~/.cache` does not exist, e.g. in the docker image. We simply
attempt to create a temporary directory which will be registered with
`atexit` to be deleted on exit using `shutil.rmtree()`. A Python 3
approach for future would be to use a `TemporaryDirectory` instead which
would get cleaned up when the held object gets dropped out of the TLS
stack.

It is still possible for the `context.cache_dir` to be `None` if the
default directory exists but is not writable, which means some code
which uses this property is liable to blow up in such a circumstance.

.github/workflows/android.yml

@@ -3,7 +3,7 @@ name: Android Tests
 on: [push, pull_request]
 
 jobs:
-  test:
+  android-test:
     strategy:
       matrix:
         python-version: [3.8]

.github/workflows/ci.yml

@@ -60,6 +60,31 @@ jobs:
           binutils-sparc64-linux-gnu \
           gcc-multilib \
           libc6-dbg
+      
+    - name: Cache for elfutils
+      uses: actions/cache@v1
+      id: cache-elfutils
+      with:
+        path: ~/.cache/elfutils
+        key: ${{ matrix.os }}-cache-elfutils
+    
+    # Install newer elfutils version due to regression in 0.176 available in focal.
+    - name: Install newer elfutils
+      env:
+        ELFUTILS_VERSION: 0.181
+      run: |
+        if [[ ! -d ~/.cache/elfutils/elfutils-${ELFUTILS_VERSION} ]]
+        then
+          wget -O /tmp/elfutils-${ELFUTILS_VERSION}.tar.bz2 https://sourceware.org/elfutils/ftp/${ELFUTILS_VERSION}/elfutils-${ELFUTILS_VERSION}.tar.bz2
+          mkdir -p ~/.cache/elfutils && cd ~/.cache/elfutils
+          tar -xf /tmp/elfutils-${ELFUTILS_VERSION}.tar.bz2
+          cd elfutils-${ELFUTILS_VERSION}
+          LDFLAGS=-Wl,-rpath=/usr/local/lib,--enable-new-dtags ./configure --disable-libdebuginfod --disable-debuginfod && make && sudo make install
+        else
+          cd ~/.cache/elfutils/elfutils-${ELFUTILS_VERSION}
+          sudo make install
+        fi
+        eu-unstrip --version
 
     - name: Install RPyC for GDB
       run: |
@@ -81,6 +106,7 @@ jobs:
     - name: Install dependencies
       run: |
         pip install --upgrade pip
+        pip install --upgrade wheel
         pip install --upgrade flake8 appdirs
         python setup.py egg_info
         pip install --upgrade --editable .

.travis.yml

@@ -25,6 +25,7 @@ addons:
     - binutils-sparc64-linux-gnu
     - bash-static
     - python3-pip
+    - elfutils
 cache:
     - pip
     - directories:

CHANGELOG.md

@@ -9,9 +9,11 @@ The table below shows which release corresponds to each branch, and what date th
 
 | Version          | Branch   | Release Date           |
 | ---------------- | -------- | ---------------------- |
-| [4.9.0](#490)    | `dev`    |
-| [4.8.0](#480)    | `beta`   |
-| [4.7.0](#470)    | `stable` | Nov 15, 2021
+| [4.10.0](#4100)  | `dev`    |
+| [4.9.0](#490)    | `beta`   |
+| [4.8.0](#480)    | `stable` | Apr 21, 2022
+| [4.7.1](#471)    |          | Apr 20, 2022
+| [4.7.0](#470)    |          | Nov 15, 2021
 | [4.6.0](#460)    |          | Jul 12, 2021
 | [4.5.1](#451)    |          | May 30, 2021
 | [4.5.0](#450)    |          | Apr 30, 2021
@@ -62,11 +64,31 @@ The table below shows which release corresponds to each branch, and what date th
 | [3.0.0](#300)    |          | Aug 20, 2016
 | [2.2.0](#220)    |          | Jan 5, 2015
 
-## 4.9.0 (`dev`)
+## 4.10.0 (`dev`)
 
 
 
-## 4.8.0 (`beta`)
+## 4.9.0 (`beta`)
+
+- [#1975][1975] Add libcdb commandline tool
+- [#1979][1979] Add `js_escape()` and `js_unescape()` to `util.fiddling`
+- [#2011][2011] Fix tube's debug output of same byte compression
+- [#2023][2023] Support KDE Konsole in run_in_new_terminal function
+- [#2027][2027] Fix ELF.libc_start_main_return with glibc 2.34
+- [#2033][2033] Quote file and core path in generated GDB script
+- [#2035][2035] Change Buffer's parent class to object
+- [#2037][2037] Allow SSH tunnel to be treated like a TCP socket (with 'raw=True')
+
+[1975]: https://github.com/Gallopsled/pwntools/pull/1975
+[1979]: https://github.com/Gallopsled/pwntools/pull/1979
+[2011]: https://github.com/Gallopsled/pwntools/pull/2011
+[2023]: https://github.com/Gallopsled/pwntools/pull/2023
+[2027]: https://github.com/Gallopsled/pwntools/pull/2027
+[2033]: https://github.com/Gallopsled/pwntools/pull/2033
+[2035]: https://github.com/Gallopsled/pwntools/pull/2035
+[2037]: https://github.com/Gallopsled/pwntools/pull/2037
+
+## 4.8.0 (`stable`)
 
 - [#1922][1922] Fix logic in `wait_for_debugger`
 - [#1828][1828] libcdb: Load debug info and unstrip libc binary
@@ -78,7 +100,24 @@ The table below shows which release corresponds to each branch, and what date th
 [1939]: https://github.com/Gallopsled/pwntools/pull/1939
 [1981]: https://github.com/Gallopsled/pwntools/pull/1981
 
-## 4.7.0 (`stable`)
+## 4.7.1
+
+- [#1784][1784] Use temporary cache directory when persistent cache cannot be used
+- [#1973][1973] ELF symbols can be looked up by bytes values
+- several bugfixes ([#2012][2012], [#2031][2031], [#1912][1912], [#1961][1961],
+[#2007][2007], [#2040][2040], [#2051][2051])
+
+[1784]: https://github.com/Gallopsled/pwntools/pull/1784
+[1912]: https://github.com/Gallopsled/pwntools/pull/1912
+[1961]: https://github.com/Gallopsled/pwntools/pull/1961
+[1973]: https://github.com/Gallopsled/pwntools/pull/1973
+[2007]: https://github.com/Gallopsled/pwntools/pull/2007
+[2012]: https://github.com/Gallopsled/pwntools/pull/2012
+[2031]: https://github.com/Gallopsled/pwntools/pull/2031
+[2040]: https://github.com/Gallopsled/pwntools/pull/2040
+[2051]: https://github.com/Gallopsled/pwntools/pull/2051
+
+## 4.7.0
 
 - [#1733][1733] Update libc headers -> more syscalls available!
 - [#1876][1876] add `self.message` and change `sys.exc_type` to `sys.exec_info()` in PwnlibException

docs/requirements.txt

@@ -12,7 +12,8 @@ pypandoc
 pyserial>=2.7
 requests>=2.5.1
 ropgadget>=5.3
-sphinx<3.4
+sphinx==1.8.6; python_version < '3.0.0'
+sphinx==4.5.0; python_version >= '3.0.0'
 sphinx_rtd_theme
 sphinxcontrib-napoleon
 sphinxcontrib-autoprogram<=0.1.5

extra/docker/base/Dockerfile

@@ -16,6 +16,7 @@ RUN apt-get update \
     && locale-gen en_US.UTF-8 \
     && apt-get install -y \
         build-essential \
+        elfutils \
         git \
         libssl-dev \
         libffi-dev \

pwnlib/asm.py

@@ -779,6 +779,11 @@ def disasm(data, vma = 0, byte = True, offset = True, instructions = True):
            0:   60 00 00 00     nop
         >>> print(disasm(unhex('00000000'), arch='mips64'))
            0:   00000000        nop
+        >>> print(disasm(unhex('48b84141414141414100c3'), arch='amd64'))
+           0:   48 b8 41 41 41 41 41 41 41 00   movabs rax, 0x41414141414141
+           a:   c3                      ret
+        >>> print(disasm(unhex('00000000'), vma=0x80000000, arch='mips'))
+        80000000:       00000000        nop
     """
     result = ''
 
@@ -790,7 +795,7 @@ def disasm(data, vma = 0, byte = True, offset = True, instructions = True):
 
     bfdarch = _bfdarch()
     bfdname = _bfdname()
-    objdump = _objdump() + ['-d', '--adjust-vma', str(vma), '-b', bfdname]
+    objdump = _objdump() + ['-w', '-d', '--adjust-vma', str(vma), '-b', bfdname]
     objcopy = _objcopy() + [
         '-I', 'binary',
         '-O', bfdname,
@@ -815,7 +820,7 @@ def disasm(data, vma = 0, byte = True, offset = True, instructions = True):
         _run(objcopy + [step1, step2])
 
         output0 = _run(objdump + [step2])
-        output1 = output0.split('<.text>:\n')
+        output1 = re.split(r'<\.text(?:\+0x0)?>:\n', output0, flags=re.MULTILINE)
 
         if len(output1) != 2:
             log.error('Could not find .text in objdump output:\n%s' % output0)

pwnlib/config.py

@@ -55,8 +55,8 @@ def initialize():
     from pwnlib.log import getLogger
     log = getLogger(__name__)
 
-    xdg_config_home = (os.environ.get('XDG_CONFIG_HOME') or
-                       os.path.expanduser("~/.config"))
+    xdg_config_home = os.environ.get('XDG_CONFIG_HOME',
+                                     os.path.expanduser("~/.config"))
 
     c = configparser.ConfigParser()
     c.read(['/etc/pwn.conf',
@@ -65,7 +65,7 @@ def initialize():
 
     for section in c.sections():
         if section not in registered_configs:
-            log.warn("Unknown configuration section %r" % section)
+            log.warn("Unknown configuration section %r", section)
             continue
         settings = dict(c.items(section))
         registered_configs[section](settings)

pwnlib/context/__init__.py

@@ -6,17 +6,20 @@ contexts work properly and as expected.
 from __future__ import absolute_import
 from __future__ import division
 
+import atexit
 import collections
+import errno
 import functools
 import logging
 import os
+import os.path
 import platform
+import shutil
 import six
 import socket
-import stat
 import string
-import subprocess
 import sys
+import tempfile
 import threading
 import time
 
@@ -345,6 +348,10 @@ class ContextType(object):
         'binary': None,
         'bits': 32,
         'buffer_size': 4096,
+        'cache_dir_base': os.environ.get(
+            'XDG_CACHE_HOME',
+            os.path.join(os.path.expanduser('~'), '.cache')
+        ),
         'cyclic_alphabet': string.ascii_lowercase.encode(),
         'cyclic_size': 4,
         'delete_corefiles': False,
@@ -972,7 +979,10 @@ class ContextType(object):
         except AttributeError:  pass
 
         # Otherwise, fail
-        level_names = filter(lambda x: isinstance(x,str), logging._levelNames)
+        try:
+            level_names = logging._levelToName.values()
+        except AttributeError:
+            level_names = filter(lambda x: isinstance(x,str), logging._levelNames)
         permitted = sorted(level_names)
         raise AttributeError('log_level must be an integer or one of %r' % permitted)
 
@@ -1269,6 +1279,21 @@ class ContextType(object):
         """
         return int(size)
 
+    @_validator
+    def cache_dir_base(self, new_base):
+        """Base directory to use for caching content.
+
+        Changing this to a different value will clear the `cache_dir` path
+        stored in TLS since a new path will need to be generated to respect the
+        new `cache_dir_base` value.
+        """
+
+        if new_base != self.cache_dir_base:
+            del self._tls["cache_dir"]
+        if os.access(new_base, os.F_OK) and not os.access(new_base, os.W_OK):
+            raise OSError(errno.EPERM, "Cache base dir is not writable")
+        return new_base
+
     @property
     def cache_dir(self):
         """Directory used for caching data.
@@ -1282,32 +1307,59 @@ class ContextType(object):
             >>> cache_dir is not None
             True
             >>> os.chmod(cache_dir, 0o000)
+            >>> del context._tls['cache_dir']
             >>> context.cache_dir is None
             True
             >>> os.chmod(cache_dir, 0o755)
             >>> cache_dir == context.cache_dir
             True
         """
-        xdg_cache_home = os.environ.get('XDG_CACHE_HOME') or \
-                         os.path.join(os.path.expanduser('~'), '.cache')
+        try:
+            # If the TLS already has a cache directory path, we return it
+            # without any futher checks since it must have been valid when it
+            # was set and if that has changed, hiding the TOCTOU here would be
+            # potentially confusing
+            return self._tls["cache_dir"]
+        except KeyError:
+            pass
 
-        if not os.access(xdg_cache_home, os.W_OK):
+        # Attempt to create a Python version specific cache dir and its parents
+        cache_dirname = '.pwntools-cache-%d.%d' % sys.version_info[:2]
+        cache_dirpath = os.path.join(self.cache_dir_base, cache_dirname)
+        try:
+            os.makedirs(cache_dirpath)
+        except OSError as exc:
+            # If we failed for any reason other than the cache directory
+            # already existing then we'll fall back to a temporary directory
+            # object which doesn't respect the `cache_dir_base`
+            if exc.errno != errno.EEXIST:
+                try:
+                    cache_dirpath = tempfile.mkdtemp(prefix=".pwntools-tmp")
+                except IOError:
+                    # This implies no good candidates for temporary files so we
+                    # have to return `None`
+                    return None
+                else:
+                    # Ensure the temporary cache dir is cleaned up on exit. A
+                    # `TemporaryDirectory` would do this better upon garbage
+                    # collection but this is necessary for Python 2 support.
+                    atexit.register(shutil.rmtree, cache_dirpath)
+        # By this time we have a cache directory which exists but we don't know
+        # if it is actually writable. Some wargames e.g. pwnable.kr have
+        # created dummy directories which cannot be modified by the user
+        # account (owned by root).
+        if os.access(cache_dirpath, os.W_OK):
+            # Stash this in TLS for later reuse
+            self._tls["cache_dir"] = cache_dirpath
+            return cache_dirpath
+        else:
             return None
 
-        cache = os.path.join(xdg_cache_home, '.pwntools-cache-%d.%d' % sys.version_info[:2])
-
-        if not os.path.exists(cache):
-            try:
-                os.mkdir(cache)
-            except OSError:
-                return None
-
-        # Some wargames e.g. pwnable.kr have created dummy directories
-        # which cannot be modified by the user account (owned by root).
-        if not os.access(cache, os.W_OK):
-            return None
-
-        return cache
+    @cache_dir.setter
+    def cache_dir(self, v):
+        if os.access(v, os.W_OK):
+            # Stash this in TLS for later reuse
+            self._tls["cache_dir"] = v
 
     @_validator
     def delete_corefiles(self, v):

pwnlib/dynelf.py

@@ -15,7 +15,7 @@ Example
     # leaks at least one byte at that address.
     def leak(address):
         data = p.read(address, 4)
-        log.debug("%#x => %s" % (address, enhex(data or '')))
+        log.debug("%#x => %s", address, enhex(data or ''))
         return data
 
     # For the sake of this example, let's say that we
@@ -62,8 +62,7 @@ from pwnlib.elf import constants
 from pwnlib.log import getLogger
 from pwnlib.memleak import MemLeak
 from pwnlib.util.fiddling import enhex
-from pwnlib.util.packing import unpack, _need_bytes
-from pwnlib.util.web import wget
+from pwnlib.util.packing import _need_bytes
 
 log    = getLogger(__name__)
 sizeof = ctypes.sizeof
@@ -562,7 +561,7 @@ class DynELF(object):
         else:   dynlib = self
 
         if dynlib is None:
-            log.failure("Could not find %r" % lib)
+            log.failure("Could not find %r", lib)
             return None
 
         #
@@ -573,7 +572,7 @@ class DynELF(object):
             self.status("Trying lookup based on Build ID")
             build_id = dynlib._lookup_build_id(lib=lib)
             if build_id:
-                log.info("Trying lookup based on Build ID: %s" % build_id)
+                log.info("Trying lookup based on Build ID: %s", build_id)
                 path = libcdb.search_by_build_id(build_id)
                 if path:
                     with context.local(log_level='error'):
@@ -615,7 +614,7 @@ class DynELF(object):
                 addr   = leak.field(cur, LinkMap.l_addr)
                 cur    = leak.field(cur, LinkMap.l_next)
 
-                log.debug('Found %r @ %#x' % (name, addr))
+                log.debug('Found %r @ %#x', name, addr)
 
                 self._bases[name] = addr
 

pwnlib/elf/elf.py

@@ -157,6 +157,12 @@ class dotdict(dict):
         >>> x.bar.baz
         4
     """
+    def __missing__(self, name):
+        if isinstance(name, (bytes, bytearray)):
+            name = packing._decode(name)
+            return self[name]
+        raise KeyError(name)
+
     def __getattr__(self, name):
         if name in self:
             return self[name]

pwnlib/encoders/arm/alphanumeric/ARM_Instructions.py

@@ -1,3 +1,5 @@
+# Copyright (c) 2013 Pratik Kumar Sahu, Nagendra Chowdary, Anish Mathuria
+# Ported to Python by Gallopsled
 from __future__ import division
 
 # +------------------------------------------------------------------------+ 

pwnlib/encoders/arm/alphanumeric/__init__.py

@@ -1,3 +1,5 @@
+# Copyright (c) 2013 Pratik Kumar Sahu, Nagendra Chowdary, Anish Mathuria
+# Ported to Python by Gallopsled
 from __future__ import absolute_import
 from __future__ import division
 

pwnlib/encoders/arm/alphanumeric/alphanum_byte.py

@@ -1,3 +1,5 @@
+# Copyright (c) 2013 Pratik Kumar Sahu, Nagendra Chowdary, Anish Mathuria
+# Ported to Python by Gallopsled
 from __future__ import division
 from __future__ import absolute_import
 

pwnlib/encoders/arm/alphanumeric/builder.py

@@ -1,3 +1,5 @@
+# Copyright (c) 2013 Pratik Kumar Sahu, Nagendra Chowdary, Anish Mathuria
+# Ported to Python by Gallopsled
 from __future__ import absolute_import
 from __future__ import division
 

pwnlib/encoders/arm/alphanumeric/random_funcs.py

@@ -1,3 +1,5 @@
+# Copyright (c) 2013 Pratik Kumar Sahu, Nagendra Chowdary, Anish Mathuria
+# Ported to Python by Gallopsled
 from __future__ import division
 
 import os

pwnlib/filepointer.py

@@ -193,7 +193,8 @@ class FileStructure(object):
             if isinstance(self.__getattr__(val), bytes):
                 structure += self.__getattr__(val).ljust(context.bytes, b'\x00')
             else:
-                structure += pack(self.__getattr__(val), self.length[val]*8)
+                if self.length[val] > 0:
+                    structure += pack(self.__getattr__(val), self.length[val]*8)
         return structure
 
     def struntil(self,v):

pwnlib/gdb.py

@@ -303,7 +303,7 @@ def _gdbserver_args(pid=None, path=None, args=None, which=None, env=None):
                 env_args.append(b'%s=%s' % (key, env.pop(key)))
             else:
                 env_args.append(b'%s=%s' % (key, env[key]))
-        gdbserver_args += ['--wrapper', 'env', '-i'] + env_args + ['--']
+        gdbserver_args += ['--wrapper', which('env'), '-i'] + env_args + ['--']
 
     gdbserver_args += ['localhost:0']
     gdbserver_args += args
@@ -915,7 +915,7 @@ def attach(target, gdbscript = '', exe = None, gdb_args = None, ssh = None, sysr
             cmd = ['sshpass', '-p', shell.password] + cmd
         if shell.keyfile:
             cmd += ['-i', shell.keyfile]
-        cmd += ['gdb', '-q', target.executable, target.pid, '-x', tmpfile]
+        cmd += ['gdb', '-q', target.executable, str(target.pid), '-x', tmpfile]
 
         misc.run_in_new_terminal(cmd)
         return

pwnlib/libcdb.py

@@ -12,9 +12,12 @@ import tempfile
 from pwnlib.context import context
 from pwnlib.elf import ELF
 from pwnlib.log import getLogger
+from pwnlib.tubes.process import process
 from pwnlib.util.fiddling import b64d
+from pwnlib.util.fiddling import enhex
 from pwnlib.util.fiddling import hexdump
 from pwnlib.util.misc import read
+from pwnlib.util.misc import which
 from pwnlib.util.misc import write
 from pwnlib.util.safeeval import const
 from pwnlib.util.web import wget
@@ -22,6 +25,7 @@ from pwnlib.util.web import wget
 log = getLogger(__name__)
 
 HASHES = ['build_id', 'sha1', 'sha256', 'md5']
+DEBUGINFOD_SERVERS = ['https://debuginfod.systemtap.org/']
 
 # https://gitlab.com/libcdb/libcdb wasn't updated after 2019,
 # but still is a massive database of older libc binaries.
@@ -88,29 +92,13 @@ def provider_libc_rip(hex_encoded_id, hash_type):
 
 PROVIDERS = [provider_libcdb, provider_libc_rip]
 
-def search_by_hash(hex_encoded_id, hash_type='build_id'):
+def search_by_hash(hex_encoded_id, hash_type='build_id', unstrip=True):
     assert hash_type in HASHES, hash_type
 
     # Ensure that the libcdb cache directory exists
-    cache_dir = os.path.join(context.cache_dir, 'libcdb', hash_type)
-
-    if not os.path.isdir(cache_dir):
-        os.makedirs(cache_dir)
-
-    # If we already downloaded the file, and it looks even passingly like
-    # a valid ELF file, return it.
-    cache = os.path.join(cache_dir, hex_encoded_id)
-
-    if os.path.exists(cache):
-        log.debug("Found existing cached libc at %r", cache)
-
-        data = read(cache)
-        if data.startswith(b'\x7FELF'):
-            log.info_once("Using cached data from %r", cache)
-            return cache
-        else:
-            log.info_once("Skipping unavailable libc %s", hex_encoded_id)
-            return None
+    cache, cache_valid = _check_elf_cache('libcdb', hex_encoded_id, hash_type)
+    if cache_valid:
+        return cache
 
     # Run through all available libc database providers to see if we have a match.
     for provider in PROVIDERS:
@@ -128,16 +116,146 @@ def search_by_hash(hex_encoded_id, hash_type='build_id'):
     if not data or not data.startswith(b'\x7FELF'):
         return None
 
+    # Try to find debug info for this libc.
+    if unstrip:
+        unstrip_libc(cache)
+
     return cache
 
+def _search_debuginfo_by_hash(base_url, hex_encoded_id):
+    # Deferred import because it's slow
+    import requests
+    from six.moves import urllib
 
-def search_by_build_id(hex_encoded_id):
+    # Check if we tried this buildid before.
+    cache, cache_valid = _check_elf_cache('libcdb_dbg', hex_encoded_id, 'build_id')
+    if cache_valid:
+        return cache
+
+    # Try to find separate debuginfo.
+    url  = '/buildid/{}/debuginfo'.format(hex_encoded_id)
+    url  = urllib.parse.urljoin(base_url, url)
+    data = b""
+    log.debug("Downloading data from debuginfod: %s", url)
+    try:
+        data = wget(url, timeout=20)
+    except requests.RequestException as e:
+        log.warn_once("Failed to fetch libc debuginfo for build_id %s from %s: %s", hex_encoded_id, base_url, e)
+    
+    # Save whatever we got to the cache
+    write(cache, data or b'')
+
+    # Return ``None`` if we did not get a valid ELF file
+    if not data or not data.startswith(b'\x7FELF'):
+        log.warn_once("Could not fetch libc debuginfo for build_id %s from %s", hex_encoded_id, base_url)
+        return None
+
+    return cache
+
+def _check_elf_cache(cache_type, hex_encoded_id, hash_type):
+    """
+    Check if there already is an ELF file for this hash in the cache.
+
+    >>> cache, _ = _check_elf_cache('libcdb', '2d1c5e0b85cb06ff47fa6fa088ec22cb6e06074e', 'build_id')
+    >>> os.unlink(cache) if os.path.exists(cache)
+    >>> filename = search_by_hash('2d1c5e0b85cb06ff47fa6fa088ec22cb6e06074e', 'build_id', unstrip=False)
+    >>> hex(ELF(filename).symbols.read)
+    '0xe56c0'
+    >>> filename == cache
+    True
+    """
+    # Ensure that the cache directory exists
+    cache_dir = os.path.join(context.cache_dir, cache_type, hash_type)
+
+    if not os.path.isdir(cache_dir):
+        os.makedirs(cache_dir)
+
+    # If we already downloaded the file, and it looks even passingly like
+    # a valid ELF file, return it.
+    cache = os.path.join(cache_dir, hex_encoded_id)
+
+    if not os.path.exists(cache):
+        return cache, False
+    
+    log.debug("Found existing cached ELF at %r", cache)
+
+    data = read(cache)
+    if not data.startswith(b'\x7FELF'):
+        log.info_once("Skipping unavailable ELF %s", hex_encoded_id)
+        return cache, False
+
+    log.info_once("Using cached data from %r", cache)
+    return cache, True
+
+def unstrip_libc(filename):
+    """
+    Given a path to a libc binary, attempt to download matching debug info
+    and add them back to the given binary.
+
+    This modifies the given file.
+
+    Arguments:
+        filename(str):
+            Path to the libc binary to unstrip.
+
+    Returns:
+        :const:`True` if binary was unstripped, :const:`False` otherwise.
+
+    Examples:
+        >>> filename = search_by_build_id('2d1c5e0b85cb06ff47fa6fa088ec22cb6e06074e', unstrip=False)
+        >>> libc = ELF(filename)
+        >>> hex(libc.symbols.read)
+        '0xe56c0'
+        >>> 'main_arena' in libc.symbols
+        False
+        >>> unstrip_libc(filename)
+        True
+        >>> libc = ELF(filename)
+        >>> hex(libc.symbols.main_arena)
+        '0x1d57a0'
+        >>> unstrip_libc(which('python'))
+        False
+        >>> filename = search_by_build_id('06a8004be6e10c4aeabbe0db74423ace392a2d6b', unstrip=True)
+        >>> 'main_arena' in ELF(filename).symbols
+        True
+    """
+    if not which('eu-unstrip'):
+        log.warn_once('Couldn\'t find "eu-unstrip" in PATH. Install elfutils first.')
+        return False
+
+    libc = ELF(filename, checksec=False)
+    if not libc.buildid:
+        log.warn_once('Given libc does not have a buildid. Cannot look for debuginfo to unstrip.')
+        return False
+
+    for server_url in DEBUGINFOD_SERVERS:
+        libc_dbg = _search_debuginfo_by_hash(server_url, enhex(libc.buildid))
+        if libc_dbg:
+            break
+    else:
+        log.warn_once('Couldn\'t find debug info for libc with build_id %s on any debuginfod server.', enhex(libc.buildid))
+        return False
+
+    # Add debug info to given libc binary inplace.
+    p = process(['eu-unstrip', '-o', filename, filename, libc_dbg])
+    output = p.recvall()
+    p.close()
+
+    if output:
+        log.error('Failed to unstrip libc binary: %r', output)
+        return False
+
+    return True
+
+def search_by_build_id(hex_encoded_id, unstrip=True):
     """
     Given a hex-encoded Build ID, attempt to download a matching libc from libcdb.
 
     Arguments:
         hex_encoded_id(str):
             Hex-encoded Build ID (e.g. 'ABCDEF...') of the library
+        unstrip(bool):
+            Try to fetch debug info for the libc and apply it to the downloaded file.
 
     Returns:
         Path to the downloaded library on disk, or :const:`None`.
@@ -152,15 +270,17 @@ def search_by_build_id(hex_encoded_id):
         >>> hex(ELF(filename).symbols.read)
         '0xeef40'
     """
-    return search_by_hash(hex_encoded_id, 'build_id')
+    return search_by_hash(hex_encoded_id, 'build_id', unstrip)
 
-def search_by_md5(hex_encoded_id):
+def search_by_md5(hex_encoded_id, unstrip=True):
     """
     Given a hex-encoded md5sum, attempt to download a matching libc from libcdb.
 
     Arguments:
         hex_encoded_id(str):
             Hex-encoded md5sum (e.g. 'ABCDEF...') of the library
+        unstrip(bool):
+            Try to fetch debug info for the libc and apply it to the downloaded file.
 
     Returns:
         Path to the downloaded library on disk, or :const:`None`.
@@ -175,15 +295,17 @@ def search_by_md5(hex_encoded_id):
         >>> hex(ELF(filename).symbols.read)
         '0xeef40'
     """
-    return search_by_hash(hex_encoded_id, 'md5')
+    return search_by_hash(hex_encoded_id, 'md5', unstrip)
 
-def search_by_sha1(hex_encoded_id):
+def search_by_sha1(hex_encoded_id, unstrip=True):
     """
     Given a hex-encoded sha1, attempt to download a matching libc from libcdb.
 
     Arguments:
         hex_encoded_id(str):
             Hex-encoded sha1sum (e.g. 'ABCDEF...') of the library
+        unstrip(bool):
+            Try to fetch debug info for the libc and apply it to the downloaded file.
 
     Returns:
         Path to the downloaded library on disk, or :const:`None`.
@@ -198,16 +320,18 @@ def search_by_sha1(hex_encoded_id):
         >>> hex(ELF(filename).symbols.read)
         '0xeef40'
     """
-    return search_by_hash(hex_encoded_id, 'sha1')
+    return search_by_hash(hex_encoded_id, 'sha1', unstrip)
 
 
-def search_by_sha256(hex_encoded_id):
+def search_by_sha256(hex_encoded_id, unstrip=True):
     """
     Given a hex-encoded sha256, attempt to download a matching libc from libcdb.
 
     Arguments:
         hex_encoded_id(str):
             Hex-encoded sha256sum (e.g. 'ABCDEF...') of the library
+        unstrip(bool):
+            Try to fetch debug info for the libc and apply it to the downloaded file.
 
     Returns:
         Path to the downloaded library on disk, or :const:`None`.
@@ -222,7 +346,7 @@ def search_by_sha256(hex_encoded_id):
         >>> hex(ELF(filename).symbols.read)
         '0xeef40'
     """
-    return search_by_hash(hex_encoded_id, 'sha256')
+    return search_by_hash(hex_encoded_id, 'sha256', unstrip)
 
 
 
@@ -279,4 +403,4 @@ def get_build_id_offsets():
     }.get(context.arch, [])
 
 
-__all__ = ['get_build_id_offsets', 'search_by_build_id', 'search_by_sha1', 'search_by_sha256', 'search_by_md5']
+__all__ = ['get_build_id_offsets', 'search_by_build_id', 'search_by_sha1', 'search_by_sha256', 'search_by_md5', 'unstrip_libc']

pwnlib/rop/rop.py

@@ -1477,8 +1477,6 @@ class ROP(object):
         # Find an appropriate, non-library ELF.
         # Prioritise non-PIE binaries so we can use _fini
         exes = (elf for elf in self.elfs if not elf.library and elf.bits == 64)
-        if not exes:
-            log.error('No non-library binaries in [elfs]')
 
         nonpie = csu = None
         for elf in exes:
@@ -1488,6 +1486,8 @@ class ROP(object):
                 nonpie = elf
             elif '__libc_csu_init' in elf.symbols:
                 csu = elf
+        else:
+            log.error('No non-library binaries in [elfs]')
 
         if elf.pie:
             if nonpie:

pwnlib/tubes/ssh.py

@@ -1523,29 +1523,24 @@ from ctypes import *; libc = CDLL('libc.so.6'); print(libc.getenv(%r))
             local: Local directory
             remote: Remote directory
         """
-        remote   = remote or self.cwd
-
+        remote = packing._encode(remote or self.cwd)
 
         if self.sftp:
-            remote = str(self.sftp.normalize(remote))
+            remote = packing._encode(self.sftp.normalize(remote))
         else:
             with context.local(log_level='error'):
-                remote = self.system('readlink -f ' + sh_string(remote))
+                remote = self.system(b'readlink -f ' + sh_string(remote))
 
-        basename = os.path.basename(remote)
+        local = local or '.'
+        local = os.path.expanduser(local)
 
-
-        local    = local or '.'
-        local    = os.path.expanduser(local)
-
-        self.info("Downloading %r to %r" % (basename,local))
+        self.info("Downloading %r to %r" % (remote, local))
 
         with context.local(log_level='error'):
             remote_tar = self.mktemp()
-            cmd = 'tar -C %s -czf %s %s' % \
+            cmd = b'tar -C %s -czf %s .' % \
                   (sh_string(remote),
-                   sh_string(remote_tar),
-                   sh_string(basename))
+                   sh_string(remote_tar))
             tar = self.system(cmd)
 
             if 0 != tar.wait():
@@ -1686,10 +1681,8 @@ from ctypes import *; libc = CDLL('libc.so.6'); print(libc.getenv(%r))
             local(str): Local path to store the data.
                 By default, uses the current directory.
         """
-        if not self.sftp:
-            self.error("Cannot determine remote file type without SFTP")
-
-        with self.system('test -d ' + sh_string(file_or_directory)) as io:
+        file_or_directory = packing._encode(file_or_directory)
+        with self.system(b'test -d ' + sh_string(file_or_directory)) as io:
             is_dir = io.wait()
 
         if 0 == is_dir:
@@ -1827,6 +1820,9 @@ from ctypes import *; libc = CDLL('libc.so.6'); print(libc.getenv(%r))
             symlink = os.path.join(self.pwd(), b'*')
         if not hasattr(symlink, 'encode') and hasattr(symlink, 'decode'):
             symlink = symlink.decode('utf-8')
+            
+        if isinstance(wd, six.text_type):
+            wd = packing._need_bytes(wd, 2, 0x80)
 
         if not wd:
             wd, status = self.run_to_end('x=$(mktemp -d) && cd $x && chmod +x . && echo $PWD', wd='.')

pwnlib/util/cyclic.py

@@ -197,6 +197,16 @@ def cyclic_find(subseq, alphabet = None, n = None):
         >>> cyclic_find(0x61616162, endian='big')
         1
 
+        Similarly to the case where you can provide a bytes value that is longer
+        than four bytes, you can provided an integer value that is larger than
+        what can be held in four bytes. If such a large value is given, it is
+        automatically truncated.
+
+        >>> cyclic_find(0x6161616361616162)
+        4
+        >>> cyclic_find(0x6261616163616161, endian='big')
+        4
+
         You can use anything for the cyclic pattern, including non-printable
         characters.
 
@@ -208,14 +218,32 @@ def cyclic_find(subseq, alphabet = None, n = None):
         n = context.cyclic_size
 
     if isinstance(subseq, six.integer_types):
+        if subseq >= 2**(8*n):
+            # Assumption: The user has given an integer that is more than 2**(8n) bits, but would otherwise fit within
+            #  a register of size 2**(8m) where m is a multiple of four
+            notice = ("cyclic_find() expected an integer argument <= {cap:#x}, you gave {gave:#x}\n"
+                      "Unless you specified cyclic(..., n={fits}), you probably just want the first {n} bytes.\n"
+                      "Truncating the data at {n} bytes.  Specify cyclic_find(..., n={fits}) to override this.").format(
+                cap=2**(8*n)-1,
+                gave=subseq,
+                # The number of bytes needed to represent subseq, rounded to the next 4
+                fits=int(round(float(subseq.bit_length()) / 32 + 0.5) * 32) // 8,
+                n=n,
+            )
+            log.warn_once(notice)
+            if context.endian == 'little':
+                subseq &= 2**(8*n) - 1
+            else:
+                while subseq >= 2**(8*n):
+                    subseq >>= 8*n
         subseq = packing.pack(subseq, bytes=n)
     subseq = packing._need_bytes(subseq, 2, 0x80)
 
     if len(subseq) != n:
-        log.warn_once("cyclic_find() expects %i-byte subsequences by default, you gave %r\n"
-            "Unless you specified cyclic(..., n=%i), you probably just want the first 4 bytes.\n"
-            "Truncating the data at 4 bytes.  Specify cyclic_find(..., n=%i) to override this.",
-            n, subseq, len(subseq), len(subseq))
+        log.warn_once("cyclic_find() expected a %i-byte subsequence, you gave %r\n"
+            "Unless you specified cyclic(..., n=%i), you probably just want the first %d bytes.\n"
+            "Truncating the data at %d bytes.  Specify cyclic_find(..., n=%i) to override this.",
+            n, subseq, len(subseq), n, n, len(subseq))
         subseq = subseq[:n]
 
     if alphabet is None:

pwnlib/util/misc.py

@@ -100,13 +100,13 @@ def size(n, abbrev = 'B', si = False):
 
     return '%.02fP%s' % (n / base, abbrev)
 
-KB = 1024
-MB = 1024 * KB
-GB = 1024 * MB
+KB = 1000
+MB = 1000 * KB
+GB = 1000 * MB
 
-KiB = 1000
-MiB = 1000 * KB
-GiB = 1000 * MB
+KiB = 1024
+MiB = 1024 * KiB
+GiB = 1024 * MiB
 
 def read(path, count=-1, skip=0):
     r"""read(path, count=-1, skip=0) -> str

pwnlib/util/packing.py

@@ -1040,7 +1040,7 @@ def _need_text(s, level=1):
     if encoding == 'auto':
         for encoding in 'ASCII', 'UTF-8', 'ISO-8859-1':
             try:
-                s = s.decode(encoding)
+                s.decode(encoding)
             except UnicodeDecodeError:
                 pass
             else:

pwnlib/util/proc.py

@@ -333,9 +333,9 @@ def wait_for_debugger(pid, debugger_pid=None):
     t = Timeout()
     with t.countdown(timeout=15):
         with log.waitfor('Waiting for debugger') as l:
-            while debugger_pid:
-                debugger = psutil.Process(debugger_pid)
-                while t.timeout and tracer(pid) is None:
+            while t.timeout and tracer(pid) is None:
+                if debugger_pid:
+                    debugger = psutil.Process(debugger_pid)
                     try:
                         debugger.wait(0.01)
                     except psutil.TimeoutExpired:
@@ -343,8 +343,7 @@ def wait_for_debugger(pid, debugger_pid=None):
                     else:
                         debugger_pid = 0
                         break
-            else:
-                while t.timeout and tracer(pid) is None:
+                else:
                     time.sleep(0.01)
 
         if tracer(pid):

pwnlib/version.py

@@ -1 +1 @@
-__version__ = '4.7.0'
+__version__ = '4.8.0'

setup.py

@@ -89,7 +89,7 @@ setup(
     name                 = 'pwntools',
     python_requires      = '>=2.7',
     packages             = find_packages(),
-    version              = '4.7.0',
+    version              = '4.8.0',
     data_files           = [('pwntools-doc',
                              glob.glob('*.md') + glob.glob('*.txt')),
                             ],