Release 3.13.0

Merge branch 'stable' into beta
Reduce cache filename length on rop.py (fix #1180 ) (#1204 )
2026-11-11 10:12:19 -10:00 · 2019-11-05 18:39:55 +01:00 · 2019-11-05 10:40:48 +01:00 · 2019-11-04 17:11:16 +01:00 · 2019-10-30 15:49:06 +01:00 · 2019-10-30 15:46:22 +01:00
56 changed files with 1023 additions and 378 deletions
@@ -1,4 +1,4 @@
-dist: precise
+dist: xenial
 language: python
 addons:
  apt:
@@ -13,14 +13,22 @@ addons:
    - zsh
    - pandoc
    - gdb
+    - binutils
+    - qemu-user-static
+    - binutils-multiarch
+    - binutils-aarch64-linux-gnu
+    - binutils-arm-linux-gnueabihf
+    - binutils-mips-linux-gnu
+    - binutils-powerpc-linux-gnu
 cache:
    - pip
    - directories:
        - usr
-        - /home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/
-        - /home/travis/virtualenv/python2.7.13/bin/
+        - /home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/
+        - /home/travis/virtualenv/python2.7.15/bin/
+sudo: required
 python:
-  - "2.7"
+  - "2.7.15"
 before_install:
  - echo $-
  - source travis/install.sh
@@ -29,6 +37,7 @@ before_install:
 install:
  - pip install --upgrade pip
  - pip install --upgrade appdirs # https://github.com/ActiveState/appdirs/issues/89#issuecomment-282326570
+  - python setup.py egg_info      # pypa/pip#6275
  - pip install --upgrade --editable .
  - pip install --upgrade --requirement docs/requirements.txt
 before_script:
@@ -9,9 +9,11 @@ The table below shows which release corresponds to each branch, and what date th

 | Version          | Branch   | Release Date           |
 | ---------------- | -------- | ---------------------- |
-| [3.13.0](#3130)  | `dev`    | Mar 31, 2018 (planned)
-| [3.12.0](#3120)  | `beta`   | Feb 17, 2018 (planned)
-| [3.11.0](#3110)  | `stable` | Jan 3, 2018
+| [4.1.0](#410)    | `dev`    | Jan 20, 2020 (planned)
+| [4.0.0](#400)    | `beta`   | Dec 20, 2019 (planned)
+| [3.13.0](#3130)  | `stable` | Nov 5, 2019
+| [3.12.0](#3120)  |          | Feb 22, 2018
+| [3.11.0](#3110)  |          | Jan 3, 2018
 | [3.10.0](#3100)  |          | Oct 25, 2017
 | [3.9.2](#392)    |          | Oct 5, 2017
 | [3.9.1](#391)    |          | Sep 28, 2017
@@ -41,14 +43,66 @@ The table below shows which release corresponds to each branch, and what date th
 | [3.0.0](#300)    |          | Aug 20, 2016
 | [2.2.0](#220)    |          | Jan 5, 2015

-## 3.12.0
+## 4.1.0

-To be released on Feb 17, 2018.
+To be released on Jan 20, 2020.
+
+## 4.0.0
+
+To be released on Dec 20, 2019.
+
+- **Python 3 support! <3**
+- [#1317][1317] Tubes with `context.encoding`
+- [#1216][1216] Improve format string generator
+- [#1285][1285] Add freebsd generic syscall templates
+- [76413f][76413f] Add pwnlib.adb.bootimg for 'ANDROID!' format boot.img images
+- [#1202][1202] Docker: Kill 14 layers in pwntools base images
+- [#1182][1182] shellcraft.dupio() for mips
+
+[1317]: https://github.com/Gallopsled/pwntools/pulls/1317
+[1285]: https://github.com/Gallopsled/pwntools/pulls/1285
+[1216]: https://github.com/Gallopsled/pwntools/pulls/1216
+[1202]: https://github.com/Gallopsled/pwntools/pulls/1202
+[1182]: https://github.com/Gallopsled/pwntools/pulls/1182
+[76413f]: https://github.com/Gallopsled/pwntools/commit/76413f
+
+## 3.13.0
+
+- [#1204][1204] Reduce ROP cache filename length
+- [#1175][1175] Fix nested SSH connectors
+- [#1355][1355] Fix 'break' syscall
+- [#1277][1277] Fix timeout parameter passing in sendlineafter and other similar functions
+- [#1292][1292] Provide correct arch name to gdb for sparc64
+
+[1175]: https://github.com/Gallopsled/pwntools/pulls/1175
+[1204]: https://github.com/Gallopsled/pwntools/pulls/1355
+[1277]: https://github.com/Gallopsled/pwntools/pulls/1277
+[1292]: https://github.com/Gallopsled/pwntools/pulls/1292
+[1355]: https://github.com/Gallopsled/pwntools/pulls/1355
+
+## 3.12.1
+
+- [#1104][1104] Add `DynELF.dump()` for dumping remote ELF files
+- [#1101][1101] Set `context.os` via `context.binary`, useful for Android exploitation
+- [5fdc08][5fdc08] Work around broken `pidof` on Android
+- [63dfed][63dfed] Print warning when Corefile deletion fails instead of throwing an exception
+- [#1094][1094] Make hexdump output alignment more consistent
+- [#1096][1096] `flat()` and `fit()` are now the same function
+
+[1104]: https://github.com/Gallopsled/pwntools/pull/1104
+[1101]: https://github.com/Gallopsled/pwntools/pull/1101
+[1094]: https://github.com/Gallopsled/pwntools/pull/1094
+[1096]: https://github.com/Gallopsled/pwntools/pull/1096
+[5fdc08]: https://github.com/Gallopsled/pwntools/commit/5fdc08
+[63dfed]: https://github.com/Gallopsled/pwntools/commit/63dfed
+
+## 3.12.0

 - [#1083][1083] Better error messages for `gdb` when `LD_PRELOAD` is incorrect
 - [#1085][1085] Add support for extracting Android `BOOTLDR!` images
 - [#1075][1075] Add support for detecting GNU Screen for `run_in_new_terminal`
 - [#1074][1074] Add support for running `pwntools-gdb` wrapper script instead of `gdb`
+- [#1068][1068] Work around very old OpenSSL versions which don't have sha256 support *AND* don't exit with an error code when trying to use it
 - [#1067][1067] Add `pwnlib.tubes.server` module, which adds a reusable `server` listener
 - [#1063][1063] Add support for labels in `fit()`, allowing dynamic contents to be injected.  (This feature is really cool, check out the pull request!)

@@ -56,10 +110,10 @@ To be released on Feb 17, 2018.
 [1085]: https://github.com/Gallopsled/pwntools/pull/1085
 [1075]: https://github.com/Gallopsled/pwntools/pull/1075
 [1074]: https://github.com/Gallopsled/pwntools/pull/1074
+[1068]: https://github.com/Gallopsled/pwntools/pull/1068
 [1067]: https://github.com/Gallopsled/pwntools/pull/1067
 [1063]: https://github.com/Gallopsled/pwntools/pull/1063

-
 ## 3.11.0

 - [#1044][1044] Enhancements to ROP
@@ -95,7 +149,6 @@ To be released on Feb 17, 2018.
 [1057]: https://github.com/Gallopsled/pwntools/pull/1057
 [1058]: https://github.com/Gallopsled/pwntools/pull/1058

-
 ## 3.9.2

 - [#1043][1043] Do not attempt to populate the libraries used by statically-linked binaries
@@ -2,7 +2,7 @@
 ![pwntools logo](https://github.com/Gallopsled/pwntools/blob/stable/docs/source/logo.png?raw=true)

 [![Docs](https://readthedocs.org/projects/pwntools/badge/?version=stable)](https://docs.pwntools.com/)
-[![PyPI](https://img.shields.io/badge/pypi-v3.11.0-green.svg?style=flat)](https://pypi.python.org/pypi/pwntools/)
+[![PyPI](https://img.shields.io/badge/pypi-v3.13.0-green.svg?style=flat)](https://pypi.python.org/pypi/pwntools/)
 [![Travis](https://travis-ci.org/Gallopsled/pwntools.svg)](https://travis-ci.org/Gallopsled/pwntools)
 [![Coveralls](https://img.shields.io/coveralls/Gallopsled/pwntools/dev.svg)](https://coveralls.io/github/Gallopsled/pwntools?branch=dev)
 [![Twitter](https://img.shields.io/badge/twitter-pwntools-4099FF.svg?style=flat)](https://twitter.com/pwntools)
@@ -1,5 +1,5 @@
 sphinxcontrib-napoleon
-sphinxcontrib-autoprogram==0.1.2
+sphinxcontrib-autoprogram==0.1.4
 isort
 paramiko>=1.15.2
 mako>=1.0.0
@@ -14,3 +14,4 @@ pygments>=2.0
 sphinx_rtd_theme
 doc2dash
 coveralls
+sphinx<1.8.0
@@ -24,8 +24,6 @@ sys.path.insert(0, os.path.abspath('../..'))

 import pwnlib

-# -- General configuration -----------------------------------------------------
-
 # If your documentation needs a minimal Sphinx version, state it here.
 #needs_sphinx = '1.0'

@@ -380,27 +378,25 @@ from sphinx.util.inspect import safe_getmembers, safe_getattr
 def dont_skip_any_doctests(app, what, name, obj, skip, options):
    return False

-# Test non-exported members
-class ModuleDocumenter(sphinx.ext.autodoc.ModuleDocumenter):
-    def get_object_members(self, want_all):
-        if want_all:
-            # if not hasattr(self.object, '__all__'):
-            #     for implicit module members, check __module__ to avoid
-            #     documenting imported objects
-                return True, safe_getmembers(self.object)
-            # else:
-            #     memberlist = self.object.__all__
-            #     # Sometimes __all__ is broken...
-            #     if not isinstance(memberlist, (list, tuple)) or not \
-            #        all(isinstance(entry, string_types) for entry in memberlist):
-            #         self.directive.warn(
-            #             '__all__ should be a list of strings, not %r '
-            #             '(in module %s) -- ignoring __all__' %
-            #             (memberlist, self.fullname))
-            #         # fall back to all members
-            #         return True, safe_getmembers(self.object)
-        else:
-            memberlist = self.options.members or []
+def get_object_members_all(self, want_all):
+    if want_all:
+        # if not hasattr(self.object, '__all__'):
+        #     for implicit module members, check __module__ to avoid
+        #     documenting imported objects
+        return True, safe_getmembers(self.object)
+    # else:
+    #     memberlist = self.object.__all__
+    #     # Sometimes __all__ is broken...
+    #     if not isinstance(memberlist, (list, tuple)) or not \
+        #        all(isinstance(entry, string_types) for entry in memberlist):
+    #         self.directive.warn(
+    #             '__all__ should be a list of strings, not %r '
+    #             '(in module %s) -- ignoring __all__' %
+    #             (memberlist, self.fullname))
+    #         # fall back to all members
+    #         return True, safe_getmembers(self.object)
+    else:
+        memberlist = self.options.members or []
        ret = []
        for mname in memberlist:
            try:
@@ -416,4 +412,4 @@ if 'doctest' in sys.argv:
    def setup(app):
        app.connect('autodoc-skip-member', dont_skip_any_doctests)

-    sphinx.ext.autodoc.ModuleDocumenter = ModuleDocumenter
+    sphinx.ext.autodoc.ModuleDocumenter.get_object_members = get_object_members_all
@@ -1,5 +1,6 @@
 .. testsetup:: *

+   from glob import glob
   from pwn import *

 :mod:`pwnlib.elf.corefile` --- Core Files
@@ -27,7 +27,7 @@ This exposes a standard interface to talk to processes, sockets, serial ports,
 and all manner of things, along with some nifty helpers for common tasks.
 For example, remote connections via :mod:`pwnlib.tubes.remote`.

-    >>> conn = remote('ftp.ubuntu.org',21)
+    >>> conn = remote('ftp.ubuntu.com',21)
    >>> conn.recvline() # doctest: +ELLIPSIS
    '220 ...'
    >>> conn.send('USER anonymous\r\n')
@@ -1,6 +1,7 @@
 .. testsetup:: *

   from pwn import *
+   from pwnlib.tubes.server import server

 :mod:`pwnlib.tubes.sock` --- Sockets
 ===========================================================
@@ -23,3 +24,9 @@
   .. autoclass:: pwnlib.tubes.listen.listen
      :members:
      :show-inheritance:
+
+.. automodule:: pwnlib.tubes.server
+
+   .. autoclass:: pwnlib.tubes.server.server
+      :members:
+      :show-inheritance:
@@ -1,6 +1,7 @@
 .. testsetup:: *

   from pwnlib.util.proc import *
+   from pwnlib.tubes.process import process
   import os, sys


@@ -43,6 +43,7 @@ from pwnlib.tubes.listen import listen
 from pwnlib.tubes.process import process, PTY, PIPE, STDOUT
 from pwnlib.tubes.remote import remote, tcp, udp, connect
 from pwnlib.tubes.serialtube import serialtube
+from pwnlib.tubes.server import server
 from pwnlib.tubes.ssh import ssh
 from pwnlib.tubes.tube import tube
 from pwnlib.ui import *
@@ -124,8 +124,3 @@ linux_aarch64_sigreturn = SigreturnABI('sp', ['x8'], 16, 0)

 windows_i386  = ABI('esp', [], 4, 0)
 windows_amd64 = ABI('rsp', ['rcx','rdx','r8','r9'], 32, 32)
-
-# Fake ABIs used by SROP
-linux_i386_srop = ABI('esp', ['eax'], 4, 0)
-linux_amd64_srop = ABI('rsp', ['rax'], 4, 0)
-linux_arm_srop = ABI('sp', ['r7'], 4, 0)
@@ -820,6 +820,7 @@ echo $PATH | while read -d: directory; do
    [ -x "$directory/{name}" ] || continue;
    echo -n "$directory/{name}\\x00";
 done
+[ -x "{name}" ] && echo -n "$PWD/{name}\\x00"
 '''.format(name=name)

    which_cmd = which_cmd.strip()
@@ -880,9 +881,16 @@ def logcat(stream=False):
 def pidof(name):
    """Returns a list of PIDs for the named process."""
    with context.quiet:
-        io = process(['pidof', name])
-        data = io.recvall().split()
-    return list(map(int, data))
+        # Older devices have a broken 'pidof', apparently.
+        # Try pgrep first.
+        io = process(['pgrep', name])
+        data = io.recvall()
+
+        if 'not found' in data:
+            io = process(['pidof', name])
+            data = io.recvall()
+
+    return list(map(int, data.split()))

@with_device
 def proc_exe(pid):
@@ -0,0 +1,103 @@
+from __future__ import unicode_literals
+
+import ctypes
+import io
+import os
+import sys
+
+class img_info(ctypes.Structure):
+    _fields_ = [
+        ('name', ctypes.c_char * 64),
+        ('size', ctypes.c_uint32)
+    ]
+
+class bootloader_images_header(ctypes.Structure):
+    _fields_ = [
+        ('magic', ctypes.c_char * 8),
+        ('num_images', ctypes.c_uint32),
+        ('start_offset', ctypes.c_uint32),
+        ('bootldr_size', ctypes.c_uint32),
+    ]
+
+    def __init__(self, *a, **kw):
+        super(bootloader_images_header, self).__init__(*a, **kw)
+        if self.magic != self.MAGIC:
+            raise ValueError("Incorrect magic (%r, expected %r)" % (self.magic, self.MAGIC))
+    MAGIC = 'BOOTLDR!'
+
+class BootloaderImage(object):
+    def __init__(self, data):
+        """Android Bootloader image
+
+        Arguments:
+            data(str): Binary data from the image file.
+        """
+        self.data = data
+        self.header = bootloader_images_header.from_buffer_copy(data)
+
+        imgarray = ctypes.ARRAY(img_info, self.header.num_images)
+        self.img_info = imgarray.from_buffer_copy(data, ctypes.sizeof(self.header))
+
+    def extract(self, index_or_name):
+        """extract(index_or_name) -> bytes
+
+        Extract the contents of an image.
+
+        Arguments:
+            index_or_name(str,int): Either an image index or name.
+
+        Returns:
+            Contents of the image.
+        """
+        if isinstance(index_or_name, int):
+            index = index_or_name
+        else:
+            for i in range(len(self.img_info)):
+                if self.img_info[i].name == index_or_name:
+                    index = i
+                    break
+            else:
+                raise ValueError("Invalid img name: %r" % index_or_name)
+
+        if index >= len(self.img_info):
+            raise ValueError("index out of range (%s, max %s)" % (index, len(self.img_info)))
+
+        offset = self.header.start_offset
+
+        for i in range(index):
+            offset += self.img_info[i].size
+
+        return self.data[offset:offset + self.img_info[index].size]
+
+    def extract_all(self, path):
+        """extract_all(path)
+
+        Extracts all images to the provided path.  The filenames are taken
+        from the image name, with '.img' appended.
+        """
+        if not os.path.isdir(path):
+            raise ValueError("%r does not exist or is not a directory" % path)
+
+        for img in self.img_info:
+            imgpath = os.path.join(path, img.name + '.img')
+            with open(imgpath, 'wb+') as f:
+                data = self.extract(img.name)
+                f.write(data)
+
+    def __str__(self):
+        rv = []
+        rv.append("Bootloader")
+        rv.append("  Magic:  %r" % self.header.magic)
+        rv.append("  Offset: %#x" % self.header.start_offset)
+        rv.append("  Size:   %#x" % self.header.bootldr_size)
+        rv.append("  Images: %s" % self.header.num_images)
+        for img in self.img_info:
+            rv.append("    Name: %s" % img.name)
+            rv.append("    Size: %#x" % img.size)
+            rv.append("    Data: %r..." % self.extract(img.name)[:32])
+        return '\n'.join(rv)
+
+if __name__ == '__main__':
+    # Easy sanity checking
+    b = BootloaderImage(open(sys.argv[1]).read())
+    print(b)
@@ -194,7 +194,7 @@ def initialize():
        value = 'True'

        if '=' in arg:
-            arg, value = arg.split('=')
+            arg, value = arg.split('=', 1)

        if arg in hooks:
            sys.argv.remove(orig)
@@ -36,7 +36,7 @@ Disassembly
    To disassemble code, simply invoke :func:`disasm` on the bytes to disassemble.

    >>> disasm('\xb8\x0b\x00\x00\x00')
-    '   0:   b8 0b 00 00 00          mov    eax,0xb'
+    '   0:   b8 0b 00 00 00          mov    eax, 0xb'

 """
 from __future__ import absolute_import
@@ -614,7 +614,7 @@ def asm(shellcode, vma = 0, extract = True, shared = False):
    Runs :func:`cpp` over a given shellcode and then assembles it into bytes.

    To see which architectures or operating systems are supported,
-    look in :mod:`pwnlib.contex`.
+    look in :mod:`pwnlib.context`.

    Assembling shellcode requires that the GNU assembler is installed
    for the target architecture.
@@ -737,15 +737,15 @@ def disasm(data, vma = 0, byte = True, offset = True, instructions = True):
    Examples:

        >>> print disasm('b85d000000'.decode('hex'), arch = 'i386')
-           0:   b8 5d 00 00 00          mov    eax,0x5d
+           0:   b8 5d 00 00 00          mov    eax, 0x5d
        >>> print disasm('b85d000000'.decode('hex'), arch = 'i386', byte = 0)
-           0:   mov    eax,0x5d
+           0:   mov    eax, 0x5d
        >>> print disasm('b85d000000'.decode('hex'), arch = 'i386', byte = 0, offset = 0)
-        mov    eax,0x5d
+        mov    eax, 0x5d
        >>> print disasm('b817000000'.decode('hex'), arch = 'amd64')
-           0:   b8 17 00 00 00          mov    eax,0x17
+           0:   b8 17 00 00 00          mov    eax, 0x17
        >>> print disasm('48c7c017000000'.decode('hex'), arch = 'amd64')
-           0:   48 c7 c0 17 00 00 00    mov    rax,0x17
+           0:   48 c7 c0 17 00 00 00    mov    rax, 0x17
        >>> print disasm('04001fe552009000'.decode('hex'), arch = 'arm')
           0:   e51f0004        ldr     r0, [pc, #-4]   ; 0x4
           4:   00900052        addseq  r0, r0, r2, asr r0
@@ -816,4 +816,4 @@ def disasm(data, vma = 0, byte = True, offset = True, instructions = True):
            line += i
        lines.append(line)

-    return '\n'.join(lines)
+    return re.sub(',([^ ])', r', \1', '\n'.join(lines))
@@ -30,13 +30,25 @@ parser.add_argument(
    help = 'The os/architecture/endianness/bits the shellcode will run in (default: linux/i386), choose from: %s' % common.choices,
 )
 parser.add_argument(
-    '--exec', type=file, dest='executable',
+    '--exec',
+
+    # NOTE: Type cannot be "file" because we may be referring to a remote
+    #       file, or a file on an Android device.
+    type=str,
+
+    dest='executable',
    help='File to debug'
 )
 parser.add_argument(
    '--process', metavar='PROCESS_NAME',
    help='Name of the process to attach to (e.g. "bash")'
 )
+parser.add_argument(
+    '--sysroot', metavar='SYSROOT',
+    type=str,
+    default='',
+    help="GDB sysroot path"
+)

 def main(args):
    gdbscript = ''
@@ -47,23 +59,46 @@ def main(args):
        context.device = adb.wait_for_device()

    if args.executable:
-        context.binary = ELF(args.executable.name)
-        target = context.binary.path
+        if os.path.exists(args.executable):
+            context.binary = ELF(args.executable)
+            target = context.binary.path
+
+        # This path does nothing, but avoids the "print_usage()"
+        # path below.
+        elif context.os == 'android':
+            target = args.executable
    elif args.pid:
        target = int(args.pid)
    elif args.process:
        if context.os == 'android':
-            target = adb.pidof(process)
+            target = adb.pidof(args.process)
        else:
-            target = pidof(process)
+            target = pidof(args.process)
+
+        # pidof() returns a list
+        if not target:
+            log.error("Could not find a PID for %r", args.process)
+
+        target = target[0]
+
+        # pidof will sometimes return all PIDs, including init
+        if target == 1:
+            log.error("Got PID 1 from pidof.  Check the process name, or use --pid 1 to debug init")
    else:
        parser.print_usage()
        return 1

    if args.pid or args.process:
-        gdb.attach(target, gdbscript=gdbscript)
+        pid = gdb.attach(target, gdbscript=gdbscript, sysroot=args.sysroot)
+
+        # Since we spawned the gdbserver process, and process registers an
+        # atexit handler to close itself, gdbserver will be terminated when
+        # we exit.  This will manifest as a "remote connected ended" or
+        # similar error message.  Hold it open for the user.
+        log.info("GDB connection forwarding will terminate when you press enter")
+        pause()
    else:
-        gdb.debug(target, gdbscript=gdbscript).interactive()
+        gdb.debug(target, gdbscript=gdbscript, sysroot=args.sysroot).interactive()

 if __name__ == '__main__':
    pwnlib.commandline.common.main(__file__)
@@ -777,6 +777,7 @@ class ContextType(object):
        self.arch   = binary.arch
        self.bits   = binary.bits
        self.endian = binary.endian
+        self.os     = binary.os

        return binary

@@ -1,13 +1,12 @@
-This directory contains a script `download-useragents.py' which downloads a list
-of user agent strings from the Internet.
+This directory contains a script `download-useragents.py' which downloads a
+list of user agent strings from the Internet.

 The user agent strings are downloaded from:
- - http://www.useragentstring.com/
- - http://www.user-agents.org/
+ - http://www.user-agents.org
 - http://techpatterns.com/downloads/firefox/useragentswitcher.xml

-The latter list is part of the Firefox `User Agent Switcher' add-on and released
-under BSD 2 Clause.
+The latter list is part of the Firefox `User Agent Switcher' add-on and
+released under BSD 2 Clause.


 As mentioned elsewhere we are not lawyers and do not know the legal status of
@@ -1,13 +1,9 @@
 #!/usr/bin/env python2
-"""
-Script for downloading lists of user agent strings
-"""
+"""Script for downloading lists of user agent strings."""
 import os
-import re
 import urllib

 from bs4 import BeautifulSoup
-
 from pwn import *

 uas = set()
@@ -23,39 +19,9 @@ def getxml(url):
    f.close()
    return xml

-with log.waitfor('Fetching from from http://www.useragentstring.com') as l:
-    html = getxml('http://www.useragentstring.com/pages/All/')
-    soup = BeautifulSoup(html)
-    l.success()
-
-with log.waitfor('Parsing list') as l:
-    liste = soup.select('#liste a')
-    for a in liste:
-        href = a.get('href')
-        if not href or href[0] != '/':
-            continue
-        m = re.match('More (.+) user agents strings -->>', a.string)
-        if m:
-            target = m.group(1)
-            with log.waitfor('Fetching additional user agents for %s' % target) as l:
-                html = getxml('http://www.useragentstring.com' + href)
-                soup = BeautifulSoup(html)
-                subliste = soup.select('#liste a')
-                for a in subliste:
-                    ua = a.string.strip()
-                    try:
-                        ua = ua.decode('utf-8')
-                    except UnicodeEncodeError:
-                        continue
-                    uas.add(ua)
-                l.success()
-        else:
-            uas.add(a.string.strip())
-    l.success()
-
 with log.waitfor('Fetching from from http://techpatterns.com') as l:
    xml = getxml('http://techpatterns.com/downloads/firefox/useragentswitcher.xml')
-    soup = BeautifulSoup(xml)
+    soup = BeautifulSoup(xml, 'html.parser')
    l.success()

 def loop(xml):
@@ -67,16 +33,16 @@ def loop(xml):
            uas.add(item['useragent'].strip())

 with log.waitfor('Parsing list') as l:
-    loop(soup.body.useragentswitcher)
+    loop(soup.useragentswitcher)
    l.success()

 with log.waitfor('Fetching from from http://www.user-agents.org') as l:
    xml = getxml('http://www.user-agents.org/allagents.xml')
-    soup = BeautifulSoup(xml)
+    soup = BeautifulSoup(xml, 'html.parser')
    l.success()

 with log.waitfor('Parsing list') as l:
-    for item in soup.body.__getattr__('user-agents'):
+    for item in soup.__getattr__('user-agents'):
        if item.name == 'user-agent':
            ua = item.select('string')[0].string.strip()
            try:
@@ -8211,6 +8211,7 @@ Mozilla/5.0 (BlackBerry; U; BlackBerry 9850; en-US) AppleWebKit/534.11+ (KHTML,
 Mozilla/5.0 (BlackBerry; U; BlackBerry 9850; en-US) AppleWebKit/534.11+ (KHTML, like Gecko) Version/7.0.0.254 Mobile Safari/534.11+
 Mozilla/5.0 (BlackBerry; U; BlackBerry 9860; en-US) AppleWebKit/534.11+ (KHTML, like Gecko) Version/7.0.0.254 Mobile Safari/534.11+
 Mozilla/5.0 (BlackBerry; U; BlackBerry 9900; en) AppleWebKit/534.11+ (KHTML, like Gecko) Version/7.1.0.346 Mobile Safari/534.11+
+Mozilla/5.0 (BlackBerry; U; BlackBerry 9930; en-US) AppleWebKit/534.11+ (KHTML, like Gecko) Version/7.1.0.267 Mobile Safari/534.11+
 Mozilla/5.0 (Clustered-Search-Bot/1.0; [email protected]; http://www.clush.com/)
 Mozilla/5.0 (Darwin; FreeBSD 5.6; en-GB; rv:1.8.1.17pre) Gecko/20080716 K-Meleon/1.5.0
 Mozilla/5.0 (Darwin; FreeBSD 5.6; en-GB; rv:1.9.1b3pre)Gecko/20081211 K-Meleon/1.5.2
@@ -8244,13 +8245,26 @@ Mozilla/5.0 (Linux; Android 4.4.4; XT1032 Build/KXB21.14-L1.61) AppleWebKit/537.
 Mozilla/5.0 (Linux; Android 4.4; Nexus 5 Build/BuildID) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/30.0.0.0 Mobile Safari/537.36
 Mozilla/5.0 (Linux; Android 5.0.1; SCH-R970 Build/LRX22C) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.84 Mobile Safari/537.36
 Mozilla/5.0 (Linux; Android 5.0.2; SAMSUNG SM-T530NU Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/3.2 Chrome/38.0.2125.102 Safari/537.36
+Mozilla/5.0 (Linux; Android 5.1.1; Coolpad 3622A Build/LMY47V) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.83 Mobile Safari/537.36
 Mozilla/5.0 (Linux; Android 5.1.1; Nexus 7 Build/LMY47V) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.78 Safari/537.36 OPR/30.0.1856.93524
 Mozilla/5.0 (Linux; Android 5.1; C6740N Build/LMY47O) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.111 Mobile Safari/537.36
 Mozilla/5.0 (Linux; Android 6.0.1; SM-G900H Build/MMB29K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.98 Mobile Safari/537.36
+Mozilla/5.0 (Linux; Android 6.0; ALE-L21 Build/HuaweiALE-L21) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.89 Mobile Safari/537.36
 Mozilla/5.0 (Linux; Android 6.0; HTC One M9 Build/MRA58K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.98 Mobile Safari/537.36
 Mozilla/5.0 (Linux; Android 6.0; LG-D850 Build/MRA58K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.97 Mobile Safari/537.36
 Mozilla/5.0 (Linux; Android 6.0; Nexus 5X Build/MDB08L) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.124 Mobile Safari/537.36
+Mozilla/5.0 (Linux; Android 7.0; LG-H918 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Mobile Safari/537.36
+Mozilla/5.0 (Linux; Android 7.0; LGL84VL Build/NRD90U) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36
+Mozilla/5.0 (Linux; Android 7.0; LGUS997 Build/NRD90U) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36
+Mozilla/5.0 (Linux; Android 7.0; Moto G (5) Plus Build/NPNS25.137-35-5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.107 Mobile Safari/537.36
 Mozilla/5.0 (Linux; Android 7.0; Nexus 9 Build/NRD90R) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.124 Safari/537.36
+Mozilla/5.0 (Linux; Android 7.0; SAMSUNG SM-G925R6 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/5.4 Chrome/51.0.2704.106 Mobile Safari/537.36
+Mozilla/5.0 (Linux; Android 7.1.1; BBB100-1 Build/NMF26F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36
+Mozilla/5.0 (Linux; Android 7.1.1; Coolpad 3632A Build/NMF26F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36
+Mozilla/5.0 (Linux; Android 7.1.1; XT1710-02 Build/NDS26.74-36) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36
+Mozilla/5.0 (Linux; Android 7.1.2; Nexus 6P Build/N2G48C) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.107 Mobile Safari/537.36
+Mozilla/5.0 (Linux; Android 7.1.2; Pixel Build/NHG47N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.83 Mobile Safari/537.36
+Mozilla/5.0 (Linux; Android 8.0.0; Pixel XL Build/OPR6.170623.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.107 Mobile Safari/537.36
 Mozilla/5.0 (Linux; U) Opera 6.02  [en]
 Mozilla/5.0 (Linux; U; Android 0.5; en-us) AppleWebKit/522  (KHTML, like Gecko) Safari/419.3
 Mozilla/5.0 (Linux; U; Android 1.0; en-us; dream) AppleWebKit/525.10  (KHTML, like Gecko) Version/3.0.4 Mobile Safari/523.12.2
@@ -8369,6 +8383,7 @@ Mozilla/5.0 (Macintosh; I; Intel Mac OS X 11_7_9; de-LI; rv:1.9b4) Gecko/2012010
 Mozilla/5.0 (Macintosh; I; PPC Mac OS X Mach-O; en-US; rv:1.9a1) Gecko/20061204 Firefox/3.0a1
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:40.0) Gecko/20100101 Firefox/40.0
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:49.0) Gecko/20100101 Firefox/49.0
+Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:55.0) Gecko/20100101 Firefox/55.0
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10.5; rv:10.0.1) Gecko/20100101 Firefox/10.0.1 SeaMonkey/2.7.1
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10.5; rv:5.0) Gecko/20100101 Firefox/5.0
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:15.0) Gecko/20120909 Firefox/15.0.1 SeaMonkey/2.12.1
@@ -8405,17 +8420,20 @@ Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10; rv:33.0) Gecko/20100101 Firefox/33
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.1 Safari/537.36
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
+Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.1.25 (KHTML, like Gecko)
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36 OPR/28.0.1750.51
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.82 Safari/537.36 OPR/29.0.1795.41
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.105 Safari/537.36 Vivaldi/1.0.162.9
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2859.0 Safari/537.36
+Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.49 Safari/537.36
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/600.8.9 (KHTML, like Gecko) Maxthon/4.5.2
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/600.8.9 (KHTML, like Gecko) Version/8.0.8 Safari/600.8.9
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Safari/602.1.50
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11) AppleWebKit/601.1.56 (KHTML, like Gecko) Version/9.0 Safari/601.1.56
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36
+Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/601.7.8 (KHTML, like Gecko) Version/10.1 Safari/603.1.30
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_5_8) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.68 Safari/534.24
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_5_8) AppleWebKit/534.31 (KHTML, like Gecko) Chrome/13.0.748.0 Safari/534.31
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_5_8) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.801.0 Safari/535.1
@@ -10005,6 +10023,8 @@ Mozilla/5.0 (OS/2; Warp 4.5; rv:31.0) Gecko/20100101 Firefox/31.0
 Mozilla/5.0 (OS/2; Warp 4.5; rv:31.0) Gecko/20100101 Firefox/31.0 SeaMonkey/2.28
 Mozilla/5.0 (OS/2; Warp 4.5; rv:38.0) Gecko/20100101 Firefox/38.0
 Mozilla/5.0 (OS/2; Warp 4.5; rv:38.0) Gecko/20100101 Firefox/38.0 SeaMonkey/2.35
+Mozilla/5.0 (OS/2; Warp 4.5; rv:45.0) Gecko/20100101 Firefox/45.0
+Mozilla/5.0 (OS/2; Warp 4.5; rv:45.0) Gecko/20100101 Firefox/45.0 SeaMonkey/2.42.9esr
 Mozilla/5.0 (PLAYSTATION 3; 1.0)
 Mozilla/5.0 (PLAYSTATION 3; 1.00)
 Mozilla/5.0 (PLAYSTATION 3; 1.10)
@@ -10018,6 +10038,7 @@ Mozilla/5.0 (PlayBook; U; RIM Tablet OS 2.1.0; en-US) AppleWebKit/536.2+ (KHTML
 Mozilla/5.0 (S60; SymbOS; Opera Mobi/1181; U; en-GB; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Opera 10.1
 Mozilla/5.0 (S60; SymbOS; Opera Mobi/1209; U; it; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Opera 10.1
 Mozilla/5.0 (S60; SymbOS; Opera Mobi/SYB-1103211396; U; es-LA; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Opera 11.00
+Mozilla/5.0 (SMART-TV; X11; Linux armv7l) AppleWebkit/537.42 (KHTML, like Gecko) Chromium/25.0.1349.2 Chrome/25.0.1349.2 Safari/537.42
 Mozilla/5.0 (Sage)
 Mozilla/5.0 (Slurp/cat; [email protected]; http://www.inktomi.com/slurp.html)
 Mozilla/5.0 (Slurp/si; [email protected]; http://www.inktomi.com/slurp.html)
@@ -10072,18 +10093,27 @@ Mozilla/5.0 (Windows ME; U) Opera 6.05  [de]
 Mozilla/5.0 (Windows ME; U; en) Opera 8.51
 Mozilla/5.0 (Windows ME; U; en; rv:1.8.1) Gecko/20061208 Firefox/2.0.0 Opera 9.63
 Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.10240
+Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063
+Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.91 Safari/537.36 Vivaldi/1.92.917.39
 Mozilla/5.0 (Windows NT 10.0; ARM; Lumia 950 Dual SIM) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.14393
 Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.36
 Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 UBrowser/5.6.13705.206 Safari/537.36
 Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36 OPR/36.0.2130.46
 Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36
 Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.90 Safari/537.36 Vivaldi/1.4.589.11
+Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 AOL/11.0 AOLBUILD/11.0.1305 Safari/537.36
+Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.78 Safari/537.36 OPR/47.0.2631.55
+Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Maxthon/5.0.4.3000 Chrome/47.0.2526.73 Safari/537.36
 Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; MATBJS; rv:11.0) like Gecko
+Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; Touch; MALNJS; rv:11.0) like Gecko
 Mozilla/5.0 (Windows NT 10.0; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0
 Mozilla/5.0 (Windows NT 10.0; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0
 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36 Edge/12.0
 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2869.0 Safari/537.36
+Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3191.0 Safari/537.36
 Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:52.0) Gecko/20100101 Firefox/52.0
+Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0
+Mozilla/5.0 (Windows NT 10.0; rv:45.9) Gecko/20100101 Goanna/3.2 Firefox/45.9 PaleMoon/27.4.0
 Mozilla/5.0 (Windows NT 4.0; U) Opera 6.05  [en]
 Mozilla/5.0 (Windows NT 4.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2049.0 Safari/537.36
 Mozilla/5.0 (Windows NT 5.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1150.1  Iron/20.0.1150.1 Safari/536.11
@@ -10145,6 +10175,8 @@ Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.
 Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.3319.102 Safari/537.36
 Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36
 Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2224.3 Safari/537.36
+Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 YaBrowser/17.3.0.1785 Yowser/2.5 Safari/537.36
+Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) MxBrowser/4.5.10.7000 Chrome/30.0.1551.0 Safari/537.36
 Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1250.0 Iron/22.0.2150.0 Safari/537.4
 Mozilla/5.0 (Windows NT 5.1) Gecko/20100101 Firefox/14.0 Opera/12.0
 Mozilla/5.0 (Windows NT 5.1; U) Opera 7.03  [de]
@@ -15192,6 +15224,7 @@ Mozilla/5.0 (X11; ; Linux i686; en-US; rv:1.8.1.3) Gecko
 Mozilla/5.0 (X11; ; Linux i686; rv:1.9.2.20) Gecko/20110805
 Mozilla/5.0 (X11; ; Linux x86_64; rv:1.8.1.6) Gecko/20070802 Firefox
 Mozilla/5.0 (X11; Arch Linux i686; rv:2.0) Gecko/20110321 Firefox/4.0
+Mozilla/5.0 (X11; CentOS; Linux x86_64; rv:36.0) Gecko/20100101 Firefox/36.0
 Mozilla/5.0 (X11; CrOS i686 0.13.507) AppleWebKit/534.35 (KHTML, like Gecko) Chrome/13.0.763.0 Safari/534.35
 Mozilla/5.0 (X11; CrOS i686 0.13.587) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.14 Safari/535.1
 Mozilla/5.0 (X11; CrOS i686 1193.158.0) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7
@@ -15205,6 +15238,7 @@ Mozilla/5.0 (X11; CrOS i686 3912.101.0) AppleWebKit/537.36 (KHTML, like Gecko) C
 Mozilla/5.0 (X11; CrOS i686 4319.74.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.57 Safari/537.36
 Mozilla/5.0 (X11; CrOS x86_64 5841.83.0) AppleWebKit/537.36 (KHTML like Gecko) Chrome/36.0.1985.138 Safari/537.36
 Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
+Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3178.0 Safari/537.36
 Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0
 Mozilla/5.0 (X11; FreeBSD 7.1-RELEASE i386; U; en; rv:1.8.1) Gecko/20061208 Firefox/2.0.0 Opera 9.63
 Mozilla/5.0 (X11; FreeBSD amd64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.65 Safari/535.11
@@ -15215,6 +15249,7 @@ Mozilla/5.0 (X11; FreeBSD amd64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/3
 Mozilla/5.0 (X11; FreeBSD amd64) AppleWebKit/537.4 (KHTML like Gecko) Chrome/22.0.1229.79 Safari/537.4
 Mozilla/5.0 (X11; FreeBSD amd64; rv:27.0) Gecko/20100101 Firefox/27.0 SeaMonkey/2.24
 Mozilla/5.0 (X11; FreeBSD amd64; rv:5.0) Gecko/20100101 Firefox/5.0
+Mozilla/5.0 (X11; FreeBSD amd64; rv:54.0) Gecko/20100101 Firefox/54.0
 Mozilla/5.0 (X11; FreeBSD amd64; rv:6.0) Gecko/20110818 Firefox/6.0 SeaMonkey/2.3
 Mozilla/5.0 (X11; FreeBSD i386) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2
 Mozilla/5.0 (X11; FreeBSD i386; rv:28.0) Gecko/20100101 Firefox/28.0 SeaMonkey/2.25
@@ -15239,6 +15274,7 @@ Mozilla/5.0 (X11; Linux armv7l; rv:2.0b4pre) Gecko/20100818 Firefox/4.0b4pre Fen
 Mozilla/5.0 (X11; Linux i386; U) Opera 7.60  [en-GB]
 Mozilla/5.0 (X11; Linux i586; rv:31.0) Gecko/20100101 Firefox/31.0
 Mozilla/5.0 (X11; Linux i686 (x86_64)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.130 Safari/537.36
+Mozilla/5.0 (X11; Linux i686 (x86_64)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3187.0 Safari/537.366
 Mozilla/5.0 (X11; Linux i686 on x86_64; rv:10.0.6) Gecko/20100101 Firefox/10.0.6 Iceweasel/10.0.6
 Mozilla/5.0 (X11; Linux i686 on x86_64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1
 Mozilla/5.0 (X11; Linux i686 on x86_64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1 Fennec/2.0.1
@@ -15284,6 +15320,7 @@ Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0
 Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2166.2 Safari/537.36
 Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.101 Safari/537.36 OPR/40.0.2308.62
 Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/51.0.2704.79 Chrome/51.0.2704.79 Safari/537.36
+Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/60.0.3112.78 Chrome/60.0.3112.78 Safari/537.36
 Mozilla/5.0 (X11; Linux i686) AppleWebKit/538.1 (KHTML, like Gecko) QupZilla/1.8.6 Safari/538.1
 Mozilla/5.0 (X11; Linux i686; Debian Testing; rv:7.0.1) Gecko/20100101 Firefox/7.0.1 Iceweasel/7.0.1
 Mozilla/5.0 (X11; Linux i686; U) Opera 7.54  [en]
@@ -15387,6 +15424,7 @@ Mozilla/5.0 (X11; Linux i686; rv:40.0) Gecko/20100101 Firefox/40.0
 Mozilla/5.0 (X11; Linux i686; rv:43.0) Gecko/20100101 Firefox/43.0
 Mozilla/5.0 (X11; Linux i686; rv:46.0) Gecko/20100101 Firefox/46.0
 Mozilla/5.0 (X11; Linux i686; rv:49.0) Gecko/20100101 Firefox/49.0
+Mozilla/5.0 (X11; Linux i686; rv:49.0) Gecko/20100101 Firefox/49.0 SeaMonkey/2.46
 Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/20100101 Firefox/5.0
 Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/20100101 Firefox/5.0 Iceweasel/5.0
 Mozilla/5.0 (X11; Linux i686; rv:6.0) Gecko/20100101 Firefox/6.0
@@ -15453,6 +15491,7 @@ Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55
 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/33.0.1750.152 Chrome/33.0.1750.152 Safari/537.36
 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML like Gecko) Chrome/22.0.1229.56 Safari/537.4
 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) QupZilla/1.9.0 Safari/538.1
+Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/604.1 (KHTML, like Gecko) Version/11.0 Safari/604.1 Ubuntu/17.04 (3.24.1-0ubuntu1) Epiphany/3.24.1
 Mozilla/5.0 (X11; Linux x86_64) Gecko Firefox/5.0
 Mozilla/5.0 (X11; Linux x86_64; U; de; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Opera 10.62
 Mozilla/5.0 (X11; Linux x86_64; U; en; rv:1.8.1) Gecko/20061208 Firefox/2.0.0
@@ -15516,6 +15555,7 @@ Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0 Iceweasel/3
 Mozilla/5.0 (X11; Linux x86_64; rv:5.0) Gecko/20100101 Firefox/5.0 FirePHP/0.5
 Mozilla/5.0 (X11; Linux x86_64; rv:5.0) Gecko/20100101 Firefox/5.0 Firefox/5.0
 Mozilla/5.0 (X11; Linux x86_64; rv:5.0) Gecko/20100101 Firefox/5.0 Iceweasel/5.0
+Mozilla/5.0 (X11; Linux x86_64; rv:55.0) Gecko/20100101 Firefox/55.0
 Mozilla/5.0 (X11; Linux x86_64; rv:6.0.1) Gecko/20110831 conkeror/0.9.3
 Mozilla/5.0 (X11; Linux x86_64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1 Iceweasel/7.0.1 Debian
 Mozilla/5.0 (X11; Linux x86_64; rv:7.0a1) Gecko/20110602 Firefox/7.0a1 SeaMonkey/2.2a1pre Lightning/1.1a1pre
@@ -18563,6 +18603,7 @@ Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:21.0) Gecko/20130331 Firefox/21.0
 Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0
 Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:35.0) Gecko/20100101 Firefox/35.0
 Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0
+Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:55.0) Gecko/20100101 Firefox/55.0
 Mozilla/5.0 (X11; debian; Linux x86_64; rv:15.0) Gecko/20100101 Iceweasel/15.0
 Mozilla/5.0 (X11;U; Linux i686; en-GB; rv:1.9.1) Gecko/20090624 Ubuntu/9.04 (jaunty) Firefox/3.5
 Mozilla/5.0 (X11;U;Linux i686;en-US;rv:1.8.1) Gecko/2006101022 Firefox/2.0
@@ -19300,15 +19341,19 @@ Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
 Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
 Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
 Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
+Mozilla/5.0 (compatible; YandexNews/4.0; +http://yandex.com/bots)
 Mozilla/5.0 (compatible; YesupBot/1.0; +http://www.yesup.net/bot.html)
 Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
 Mozilla/5.0 (compatible; Yoono; http://www.yoono.com/)
 Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )
 Mozilla/5.0 (compatible; Zenbot/1.3; +http://zen.co.za/webmasters/)
 Mozilla/5.0 (compatible; aiHitBot/1.0; +http://www.aihit.com/)
+Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; )
+Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)
 Mozilla/5.0 (compatible; archive.org_bot/1.10.0 +http://www.loc.gov/minerva/crawl.html)
 Mozilla/5.0 (compatible; archive.org_bot/1.13.1x http://crawler.archive.org)
 Mozilla/5.0 (compatible; archive.org_bot/1.5.0-200506132127 http://crawler.archive.org) Hurricane Katrina
+Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)
 Mozilla/5.0 (compatible; bingbot/2.0  http://www.bing.com/bingbot.htm)
 Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm)
 Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
@@ -19352,6 +19397,7 @@ Mozilla/5.0 (en-US;rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2
 Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13
 Mozilla/5.0 (hp-tablet; Linux; hpwOS/3.0.2; U; de-DE) AppleWebKit/534.6 (KHTML, like Gecko) wOSBrowser/234.40.1 Safari/534.6 TouchPad/1.0
 Mozilla/5.0 (iPad; CPU OS 10_0 like Mac OS X) AppleWebKit/601.1 (KHTML, like Gecko) CriOS/49.0.2623.109 Mobile/14A5335b Safari/601.1.46
+Mozilla/5.0 (iPad; CPU OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A5362a Safari/604.1
 Mozilla/5.0 (iPad; CPU OS 5_0_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Mobile/9B176
 Mozilla/5.0 (iPad; CPU OS 5_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko ) Version/5.1 Mobile/9B176 Safari/7534.48.3
 Mozilla/5.0 (iPad; CPU OS 5_1_1 like Mac OS X; ru-ru) AppleWebKit/534.46.0 (KHTML, like Gecko) CriOS/21.0.1180.80 Mobile/9B206 Safari/7534.48.3
@@ -19390,6 +19436,8 @@ Mozilla/5.0 (iPhone Simulator; U; CPU iPhone OS 3_2 like Mac OS X; en-us) AppleW
 Mozilla/5.0 (iPhone; CPU  iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7A341 Safari/528.16
 Mozilla/5.0 (iPhone; CPU iPhone OS 10_0 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) GSA/18.0.130791545 Mobile/14A5345a Safari/600.1.4
 Mozilla/5.0 (iPhone; CPU iPhone OS 10_0 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Mobile/14A346 Safari/602.1
+Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_3 like Mac OS X) AppleWebKit/603.3.8 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1
+Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A5362a Safari/604.1
 Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A403 Safari/8536.25
 Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53
 Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4
@@ -20490,6 +20538,7 @@ Opera/9.80 (Android; Opera Mini/7.29530/27.1407; U; en) Presto/2.8.119 Version/1
 Opera/9.80 (Android; Opera Mini/7.5.33361/31.1350; U; en) Presto/2.8.119 Version/11.10
 Opera/9.80 (Android; Opera Mini/7.5.33361/31.1543; U; en) Presto/2.8.119 Version/11.1010
 Opera/9.80 (Android; Opera Mini/7.6.35766/35.5706; U; en) Presto/2.8.119 Version/11.10
+Opera/9.80 (Android; Opera Mini/9.0.1829/66.318; U; en) Presto/2.12.423 Version/12.16
 Opera/9.80 (Android;Opera Mini/6.0.24212/24.746 U;en) Presto/2.5.25 Version/10.5454
 Opera/9.80 (BlackBerry; Opera Mini/5.1.22303/22.387; U; en) Presto/2.5.25 Version/10.54
 Opera/9.80 (BlackBerry; Opera Mini/6.24209/27.1366; U; en) Presto/2.8.119 Version/11.10
@@ -21055,6 +21104,7 @@ Scooter_trk3-3.0.3
 Scope (Mars+)
 ScoutAbout
 ScoutAnt/0.1; +http://www.ant.com/what_is_ant.com/
+Screaming Frog SEO Spider/8.1
 Scrubby/2.1 (http://www.scrubtheweb.com/)
 Scrubby/2.2 (http://www.scrubtheweb.com/)
 Scrubby/2.x (http://www.scrubtheweb.com/)
@@ -21311,6 +21361,7 @@ Trampelpfad-Spider-v0.1
 TulipChain/5.x (http://ostermiller.org/tulipchain/) Java/1.x.1_0x (http://java.sun.com/) Linux/2.4.17
 TulipChain/5.xx (http://ostermiller.org/tulipchain/) Java/1.x.1_0x (http://apple.com/) Mac_OS_X/10.2.8
 Tumblr/1.0 RSS syndication (+http://www.tumblr.com/) ([email protected])
+TurnitinBot (https://turnitin.com/robot/crawlerinfo.html)
 TurnitinBot/1.5 (http://www.turnitin.com/robot/crawlerinfo.html)
 TurnitinBot/1.5 http://www.turnitin.com/robot/crawlerinfo.html
 TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html
@@ -21323,6 +21374,7 @@ TwengaBot
 Twiceler www.cuill.com/robots.html
 Twiceler-0.9 http://www.cuill.com/twiceler/robot.html
 Twisted PageGetter
+Twitterbot/1.0
 Twitturly / v0.x
 Twotrees Reactive Filter V2.0
 Tycoon Agent/Nutch-1.0-dev
@@ -21584,6 +21636,7 @@ Xenu Link Sleuth 1.2h
 Xenu Link Sleuth 1.2i
 Xenu Link Sleuth 1.xx
 Xenu Link Sleuth/1.3.7
+Xenu Link Sleuth/1.3.8
 Xenu's Link Sleuth 1.x[a-z]
 Xerka WebBot v1.0.0 [UPVOpenDir]
 Xylix
@@ -910,3 +910,65 @@ class DynELF(object):
        self.success('*curbrk: %#x' % brk)

        return brk
+
+    def _find_mapped_pages(self, readonly = False, page_size = 0x1000):
+        """
+        A generator of all mapped pages, as found using the Program Headers.
+
+        Yields tuples of the form: (virtual address, memory size)
+        """
+        leak  = self.leak
+        base  = self.libbase
+
+        Ehdr  = {32: elf.Elf32_Ehdr, 64: elf.Elf64_Ehdr}[self.elfclass]
+        Phdr  = {32: elf.Elf32_Phdr, 64: elf.Elf64_Phdr}[self.elfclass]
+
+        phead = base + leak.field(base, Ehdr.e_phoff)
+        phnum = leak.field(base, Ehdr.e_phnum)
+
+        for i in range(phnum):
+            if leak.field_compare(phead, Phdr.p_type, constants.PT_LOAD) :
+                # the interesting pages are those that are aligned to PAGE_SIZE
+                if leak.field_compare(phead, Phdr.p_align, page_size) and \
+                    (readonly or leak.field(phead, Phdr.p_flags) & 0x02 != 0):
+                    vaddr = leak.field(phead, Phdr.p_vaddr)
+                    memsz = leak.field(phead, Phdr.p_memsz)
+                    # fix relative offsets
+                    if vaddr < base :
+                        vaddr += base
+                    yield vaddr, memsz
+            phead += sizeof(Phdr)
+
+    def dump(self, libs = False, readonly = False):
+        """dump(libs = False, readonly = False)
+
+        Dumps the ELF's memory pages to allow further analysis.
+
+        Arguments:
+            libs(bool, optional): True if should dump the libraries too (False by default)
+            readonly(bool, optional): True if should dump read-only pages (False by default)
+
+        Returns:
+            a dictionary of the form: { address : bytes }
+        """
+        leak      = self.leak
+        page_size = 0x1000
+        pages     = {}
+
+        for vaddr, memsz in self._find_mapped_pages(readonly, page_size) :
+            offset    = vaddr % page_size
+            if offset != 0 :
+                memsz += offset
+                vaddr -= offset
+            memsz += (page_size - (memsz % page_size)) % page_size
+            pages[vaddr] = leak.n(vaddr, memsz)
+
+        if libs :
+            for lib_name in self.bases() :
+                if len(lib_name) == 0 :
+                    continue
+                dyn_lib = self._dynamic_load_dynelf(lib_name)
+                if dyn_lib is not None :
+                    pages.update(dyn_lib.dump(readonly = readonly))
+
+        return pages
@@ -87,6 +87,7 @@ from pwnlib.tubes.process import process
 from pwnlib.tubes.ssh import ssh_channel
 from pwnlib.tubes.tube import tube
 from pwnlib.util.fiddling import b64d
+from pwnlib.util.fiddling import enhex
 from pwnlib.util.fiddling import unhex
 from pwnlib.util.misc import read
 from pwnlib.util.misc import write
@@ -223,6 +224,8 @@ class Mapping(object):
        return self._core.read(item, 1)

    def __contains__(self, item):
+        if isinstance(item, Mapping):
+            return (self.start <= item.start) and (item.stop <= self.stop)
        return self.start <= item < self.stop

    def find(self, sub, start=None, end=None):
@@ -472,6 +475,7 @@ class Corefile(ELF):
        >>> io = elf.process()
        >>> io.wait()
        >>> core = io.corefile
+        [!] End of the stack is corrupted, skipping stack parsing (got: 4141414141414141)
        >>> core.argc, core.argv, core.env
        (0, [], {})
        >>> core.stack.data.endswith('AAAA')
@@ -545,36 +549,40 @@ class Corefile(ELF):
            for segment in self.segments:
                if not isinstance(segment, elftools.elf.segments.NoteSegment):
                    continue
+
+
+                # Note that older versions of pyelftools (<=0.24) are missing enum values
+                # for NT_PRSTATUS, NT_PRPSINFO, NT_AUXV, etc.
+                # For this reason, we have to check if note.n_type is any of several values.
                for note in iter_notes(segment):
-                    # Try to find NT_PRSTATUS.  Note that pyelftools currently
-                    # mis-identifies the enum name as 'NT_GNU_ABI_TAG'.
+                    # Try to find NT_PRSTATUS.
                    if prstatus_type and \
                       note.n_descsz == ctypes.sizeof(prstatus_type) and \
-                       note.n_type == 'NT_GNU_ABI_TAG':
+                       note.n_type in ('NT_GNU_ABI_TAG', 'NT_PRSTATUS'):
                        self.NT_PRSTATUS = note
                        self.prstatus = prstatus_type.from_buffer_copy(note.n_desc)

                    # Try to find NT_PRPSINFO
-                    # Note that pyelftools currently mis-identifies the enum name
-                    # as 'NT_GNU_BUILD_ID'
-                    if note.n_descsz == ctypes.sizeof(prpsinfo_type) and \
-                       note.n_type == 'NT_GNU_BUILD_ID':
+                    if prpsinfo_type and \
+                       note.n_descsz == ctypes.sizeof(prpsinfo_type) and \
+                       note.n_type in ('NT_GNU_ABI_TAG', 'NT_PRPSINFO'):
                        self.NT_PRPSINFO = note
                        self.prpsinfo = prpsinfo_type.from_buffer_copy(note.n_desc)

                    # Try to find NT_SIGINFO so we can see the fault
-                    if note.n_type == 0x53494749:
+                    if note.n_type in (0x53494749, 'NT_SIGINFO'):
                        self.NT_SIGINFO = note
                        self.siginfo = siginfo_type.from_buffer_copy(note.n_desc)

                    # Try to find the list of mapped files
-                    if note.n_type == constants.NT_FILE:
+                    if note.n_type in (constants.NT_FILE, 'NT_FILE'):
                        with context.local(bytes=self.bytes):
                            self._parse_nt_file(note)

                    # Try to find the auxiliary vector, which will tell us
                    # where the top of the stack is.
-                    if note.n_type == constants.NT_AUXV:
+                    if note.n_type in (constants.NT_AUXV, 'NT_AUXV'):
+                        self.NT_AUXV = note
                        with context.local(bytes=self.bytes):
                            self._parse_auxv(note)

@@ -583,19 +591,13 @@ class Corefile(ELF):

            if self.stack and self.mappings:
                for mapping in self.mappings:
-                    if mapping.stop == self.stack:
+                    if self.stack in mapping or self.stack == mapping.stop:
                        mapping.name = '[stack]'
                        self.stack   = mapping
                        break
                else:
-                    for mapping in self.mappings:
-                        if self.stack in mapping:
-                            mapping.name = '[stack]'
-                            self.stack   = mapping
-                            break
-                    else:
-                        log.warn('Could not find the stack!')
-                        self.stack = None
+                    log.warn('Could not find the stack!')
+                    self.stack = None

            with context.local(bytes=self.bytes, log_level='warn'):
                try:
@@ -753,30 +755,38 @@ class Corefile(ELF):
            >>> io.corefile.fault_addr == io.corefile.eax == 0xdeadbeef
            True
        """
-        fault_addr = 0
-        if self.siginfo:
-            fault_addr = int(self.siginfo.sigfault_addr)
+        if not self.siginfo:
+            return getattr(self, 'pc', 0)

-            # The fault_addr on AMD64 is zero if the crash occurs
-            # after a "ret" instruction, if the "ret" would return
-            # to an invalid address.  We need to extract the address
-            # manually, as a convenience.
-            if fault_addr == 0 and self.arch == 'amd64' and self.pc and self.sp:
-                try:
-                    code = self.read(self.pc, 1)
-                    if code != '\xc3':
-                        return fault_addr
-                    address = self.unpack(self.sp)
-                    return address
-                except Exception:
-                    # Could not read $rsp or $rip
-                    pass
+        fault_addr = int(self.siginfo.sigfault_addr)

-            return fault_addr
+        # The fault_addr is zero if the crash occurs due to a
+        # "protection fault", e.g. a dereference of 0x4141414141414141
+        # because this is technically a kernel address.
+        #
+        # A protection fault does not set "fault_addr" in the siginfo.
+        # (http://elixir.free-electrons.com/linux/v4.14-rc8/source/kernel/signal.c#L1052)
+        #
+        # Since a common use for corefiles is to spray the stack with a
+        # cyclic pattern to find the offset to get control of $PC,
+        # check for a "ret" instruction ("\xc3").
+        #
+        # If we find a RET at $PC, extract the "return address" from the
+        # top of the stack.
+        if fault_addr == 0 and self.siginfo.si_code == 0x80:
+            try:
+                code = self.read(self.pc, 1)
+                RET = '\xc3'
+                if code == RET:
+                    fault_addr = self.unpack(self.sp)
+            except Exception:
+                # Could not read $rsp or $rip
+                pass
+
+        return fault_addr

        # No embedded siginfo structure, so just return the
        # current instruction pointer.
-        return getattr(self, 'pc', 0)

    @property
    def _pc_register(self):
@@ -819,8 +829,8 @@ class Corefile(ELF):
        fields = [
            repr(self.path),
            '%-10s %s' % ('Arch:', gnu_triplet),
-            '%-10s %#x' % ('%s:' % self._pc_register.upper(), getattr(self, 'pc', 0)),
-            '%-10s %#x' % ('%s:' % self._sp_register.upper(), getattr(self, 'sp', 0)),
+            '%-10s %#x' % ('%s:' % self._pc_register.upper(), self.pc or 0),
+            '%-10s %#x' % ('%s:' % self._sp_register.upper(), self.sp or 0),
        ]

        if self.exe and self.exe.name:
@@ -887,18 +897,25 @@ class Corefile(ELF):
        if not stack:
            return

+        # If the stack does not end with zeroes, something is very wrong.
+        if not stack.data.endswith('\x00' * 8):
+            log.warn_once("End of the stack is corrupted, skipping stack parsing (got: %s)",
+                          enhex(self.data[-8:]))
+            return
+
        # AT_EXECFN is the start of the filename, e.g. '/bin/sh'
        # Immediately preceding is a NULL-terminated environment variable string.
        # We want to find the beginning of it
-        if self.at_execfn:
-            address = self.at_execfn-1
-        else:
-            log.debug('No AT_EXECFN')
+        if not self.at_execfn:
            address = stack.stop
            address -= 2*self.bytes
            address -= 1
            address = stack.rfind('\x00', None, address)
            address += 1
+            self.at_execfn = address
+
+        address = self.at_execfn-1
+

        # Sanity check!
        try:
@@ -1094,7 +1111,7 @@ class Coredump(Corefile):
 class CorefileFinder(object):
    def __init__(self, proc):
        if proc.poll() is None:
-            log.error("Process %i has not exited" % (process.pid))
+            log.error("Process %i has not exited" % (proc.pid))

        self.process = proc
        self.pid = proc.pid
@@ -1154,7 +1171,10 @@ class CorefileFinder(object):
            new_path = 'core.%i' % core_pid
            if core_pid > 0 and new_path != self.core_path:
                write(new_path, self.read(self.core_path))
-                self.unlink(self.core_path)
+                try:
+                    self.unlink(self.core_path)
+                except (IOError, OSError):
+                    log.warn("Could not delete %r" % self.core_path)
                self.core_path = new_path

        # Check the PID
@@ -51,12 +51,17 @@ from elftools.elf.constants import P_FLAGS
 from elftools.elf.constants import SHN_INDICES
 from elftools.elf.descriptions import describe_e_type
 from elftools.elf.elffile import ELFFile
-from elftools.elf.enums import ENUM_P_TYPE
 from elftools.elf.gnuversions import GNUVerDefSection
 from elftools.elf.relocation import RelocationSection
 from elftools.elf.sections import SymbolTableSection
 from elftools.elf.segments import InterpSegment

+# See https://github.com/Gallopsled/pwntools/issues/1189
+try:
+    from elftools.elf.enums import ENUM_P_TYPE
+except ImportError:
+    from elftools.elf.enums import ENUM_P_TYPE_BASE as ENUM_P_TYPE
+
 import intervaltree

 from pwnlib import adb
@@ -183,6 +188,7 @@ class ELF(ELFFile):
    functions = {}
    endian = 'little'
    address = 0x400000
+    linker = None

    # Whether to fill gaps in memory with zeroed pages
    _fill_gaps = True
@@ -262,9 +268,13 @@ class ELF(ELFFile):
                self.bits = 64

        # Is this a native binary? Should we be checking QEMU?
-        with context.local(arch=self.arch):
-            #: Whether this ELF should be able to run natively
-            self.native = context.native
+        try:
+            with context.local(arch=self.arch):
+                #: Whether this ELF should be able to run natively
+                self.native = context.native
+        except AttributeError:
+            # The architecture may not be supported in pwntools
+            self.native = False

        self._address = 0
        if self.elftype != 'DYN':
@@ -304,8 +314,20 @@ class ELF(ELFFile):

        for seg in self.iter_segments_by_type('PT_INTERP'):
            self.executable = True
+
+            #: ``True`` if the ELF is statically linked
            self.statically_linked = False

+            #: Path to the linker for the ELF
+            self.linker = self.read(seg.header.p_vaddr, seg.header.p_memsz)
+            self.linker = self.linker.rstrip('\x00')
+
+        #: Operating system of the ELF
+        self.os = 'linux'
+
+        if self.linker and self.linker.startswith('/system/bin/linker'):
+            self.os = 'android'
+
        #: ``True`` if the ELF is a shared library
        self.library = not self.executable and self.elftype == 'DYN'

@@ -1763,4 +1785,3 @@ class ELF(ELFFile):
                return

        log.error("Could not find PT_GNU_STACK, stack should already be executable")
-
@@ -229,7 +229,7 @@ def _gdbserver_args(pid=None, path=None, args=None, which=None):

    orig_args = args

-    gdbserver_args = [gdbserver]
+    gdbserver_args = [gdbserver, '--multi']
    if context.aslr:
        gdbserver_args += ['--no-disable-randomization']
    else:
@@ -249,6 +249,12 @@ def _gdbserver_port(gdbserver, ssh):
    # Process /bin/bash created; pid = 14366
    # Listening on port 34816
    process_created = gdbserver.recvline()
+
+    if process_created.startswith('ERROR:'):
+        raise ValueError(
+            'Failed to spawn process under gdbserver. gdbserver error message: %s' % process_created
+        )
+
    gdbserver.pid   = int(process_created.split()[-1], 0)

    listening_on = ''
@@ -287,7 +293,7 @@ def _get_runner(ssh=None):
    else:                          return tubes.process.process

@LocalContext
-def debug(args, gdbscript=None, exe=None, ssh=None, env=None, **kwargs):
+def debug(args, gdbscript=None, exe=None, ssh=None, env=None, sysroot=None, **kwargs):
    """debug(args) -> tube

    Launch a GDB server with the specified command line,
@@ -299,6 +305,8 @@ def debug(args, gdbscript=None, exe=None, ssh=None, env=None, **kwargs):
        exe(str): Path to the executable on disk
        env(dict): Environment to start the binary in
        ssh(:class:`.ssh`): Remote ssh session to use to launch the process.
+        sysroot(str): Foreign-architecture sysroot, used for QEMU-emulated binaries
+            and Android targets.

    Returns:
        :class:`.process` or :class:`.ssh_channel`: A tube connected to the target process
@@ -404,7 +412,6 @@ def debug(args, gdbscript=None, exe=None, ssh=None, env=None, **kwargs):

    runner = _get_runner(ssh)
    which  = _get_which(ssh)
-    sysroot = None
    gdbscript = gdbscript or ''

    if context.noptrace:
@@ -416,11 +423,15 @@ def debug(args, gdbscript=None, exe=None, ssh=None, env=None, **kwargs):
    else:
        qemu_port = random.randint(1024, 65535)
        qemu_user = qemu.user_path()
-        sysroot = qemu.ld_prefix(env)
+        sysroot = sysroot or qemu.ld_prefix(env=env)
        if not qemu_user:
            log.error("Cannot debug %s binaries without appropriate QEMU binaries" % context.arch)
        args = [qemu_user, '-g', str(qemu_port)] + args

+    # Use a sane default sysroot for Android
+    if not sysroot and context.os == 'android':
+        sysroot = 'remote:/'
+
    # Make sure gdbserver/qemu is installed
    if not which(args[0]):
        log.error("%s is not installed" % args[0])
@@ -464,7 +475,8 @@ def get_gdb_arch():
        'powerpc': 'powerpc:common',
        'powerpc64': 'powerpc:common64',
        'mips64': 'mips:isa64',
-        'thumb': 'arm'
+        'thumb': 'arm',
+        'sparc64': 'sparc:v9'
    }.get(context.arch, context.arch)

 def binary():
@@ -478,7 +490,7 @@ def binary():
        >>> gdb.binary() # doctest: +SKIP
        '/usr/bin/gdb'
    """
-    gdb = misc.which('gdb')
+    gdb = misc.which('pwntools-gdb') or misc.which('gdb')

    if not context.native:
        multiarch = misc.which('gdb-multiarch')
@@ -508,6 +520,7 @@ def attach(target, gdbscript = None, exe = None, need_ptrace_scope = True, gdb_a
          detect the architechture automatically (if it is supported).
        gdb_args(list): List of additional arguments to pass to GDB.
        sysroot(str): Foreign-architecture sysroot, used for QEMU-emulated binaries
+            and Android targets.

    Returns:
        PID of the GDB process (or the window which it is running in).
@@ -607,6 +620,10 @@ def attach(target, gdbscript = None, exe = None, need_ptrace_scope = True, gdb_a
    if gdbscript and not gdbscript.endswith('\n'):
        gdbscript += '\n'

+    # Use a sane default sysroot for Android
+    if not sysroot and context.os == 'android':
+        sysroot = 'remote:/'
+
    # gdb script to run before `gdbscript`
    pre = ''
    if not context.native:
@@ -671,7 +688,15 @@ def attach(target, gdbscript = None, exe = None, need_ptrace_scope = True, gdb_a
        exe = exe or target.executable
    elif isinstance(target, tuple) and len(target) == 2:
        host, port = target
-        pre += 'target remote %s:%d\n' % (host, port)
+
+        if context.os != 'android':
+            pre += 'target remote %s:%d\n' % (host, port)
+        else:
+            # Android debugging is done over gdbserver, which can't follow
+            # new inferiors (tldr; follow-fork-mode child) unless it is run
+            # in extended-remote mode.
+            pre += 'target extended-remote %s:%d\n' % (host, port)
+            pre += 'set detach-on-fork off\n'

        def findexe():
            for spid in proc.pidof(target):
@@ -699,7 +724,8 @@ def attach(target, gdbscript = None, exe = None, need_ptrace_scope = True, gdb_a
        log.error('could not find target process')

    if exe:
-        pre += 'file %s\n' % exe
+        # The 'file' statement should go first
+        pre = 'file %s\n%s' % (exe, pre)

    cmd = binary()

@@ -731,7 +757,11 @@ def attach(target, gdbscript = None, exe = None, need_ptrace_scope = True, gdb_a
        gdbserver = runner(gdb_cmd)
        port    = _gdbserver_port(gdbserver, None)
        host    = context.adb_host
-        pre    += 'target remote %s:%i' % (context.adb_host, port)
+        pre    += 'target extended-remote %s:%i\n' % (context.adb_host, port)
+
+        # gdbserver on Android sets 'detach-on-fork on' which breaks things
+        # when you're trying to debug anything that forks.
+        pre += 'set detach-on-fork off\n'

    gdbscript = pre + (gdbscript or '')

@@ -759,7 +789,7 @@ def ssh_gdb(ssh, argv, gdbscript = None, arch = None, **kwargs):
        argv = [argv]

    exe = argv[0]
-    argv = ["gdbserver", "127.0.0.1:0"] + argv
+    argv = ["gdbserver", "--multi", "127.0.0.1:0"] + argv

    # Download the executable
    local_exe = os.path.basename(exe)
@@ -3,7 +3,7 @@ from os.path import basename

 from docutils import nodes
 from docutils import statemachine
-from sphinx.util.compat import Directive
+from docutils.parsers.rst import Directive

 try:
    from StringIO import StringIO
@@ -363,7 +363,7 @@ class Logger(object):
        """
        m = message % args
        if m not in self._one_time_warnings:
-            if self.isEnabledFor(logging.INFO):
+            if self.isEnabledFor(logging.WARNING):
                self._one_time_warnings.add(m)
            self._log(logging.WARNING, message, args, kwargs, 'warning_once')

@@ -233,4 +233,12 @@ class Call(object):
                args.extend(map(repr, arg.values))
            else:
                args.append(arg)
-        return '%s(%s)' % (self.name or fmt % self.target, ', '.join(map(str, args)))
+
+        name = self.name or (fmt % self.target)
+        arg_str = []
+        for arg in args:
+            if isinstance(arg, (int,long)) and arg > 0x100:
+                arg_str.append(hex(arg))
+            else:
+                arg_str.append(str(arg))
+        return '%s(%s)' % (name, ', '.join(arg_str))
@@ -105,7 +105,7 @@ Finally, let's build our ROP stack
    >>> rop.write(c.STDOUT_FILENO, binary.symbols['flag'], 8)
    >>> rop.exit()
    >>> print rop.dump()
-    0x0000:       0x10000012 write(STDOUT_FILENO, 268435494, 8)
+    0x0000:       0x10000012 write(STDOUT_FILENO, 0x10000026, 8)
    0x0004:       0x1000000e <adjust @0x18> add esp, 0x10; ret
    0x0008:              0x1 arg0
    0x000c:       0x10000026 flag
@@ -168,6 +168,21 @@ requirements so that everything "just works".
    0x0078:       'faabgaab' <pad 0x8>
    0x0080:       0x10000008 target

+Pwntools will also filter out some bad instructions while setting the registers
+( e.g. syscall, int 0x80... )
+
+    >>> assembly = 'syscall; pop rdx; pop rsi; ret ; pop rdi ; int 0x80; pop rsi; pop rdx; ret ; pop rdi ; ret'
+    >>> binary = ELF.from_assembly(assembly)
+    >>> rop = ROP(binary)
+    >>> rop.call(0xdeadbeef, [1, 2, 3])
+    >>> print rop.dump()
+    0x0000:       0x1000000b pop rdi; ret
+    0x0008:              0x1 [arg0] rdi = 1
+    0x0010:       0x10000008 pop rsi; pop rdx; ret
+    0x0018:              0x2 [arg1] rsi = 2
+    0x0020:              0x3 [arg2] rdx = 3
+    0x0028:       0xdeadbeef
+
 ROP + Sigreturn
 -----------------------

@@ -495,15 +510,20 @@ class ROP(object):

        regset = set(registers)

+        bad_instructions = set(('syscall', 'sysenter', 'int 0x80'))
+        
        # Collect all gadgets which use these registers
        # Also collect the "best" gadget for each combination of registers
        gadgets = []
        best_gadgets = {}

        for gadget in self.gadgets.values():
-            # Do not use gadgets which end in e.g. "int 0x80"
+            # Do not use gadgets which doesn't end with 'ret'
            if gadget.insns[-1] != 'ret':
                continue
+            # Do not use gadgets which contain 'syscall' or 'int'
+            if set(gadget.insns) & bad_instructions:
+                continue

            touched = tuple(regset & set(gadget.regs))

@@ -512,10 +532,14 @@ class ROP(object):

            old = best_gadgets.get(touched, gadget)

-            if old is gadget or old.move > gadget.move:
+            # if we have a new gadget for the touched registers, choose it
+            # if the new gadget requires less stack space, choose it
+            # if both gadgets require same stack space, choose the one with less instructions
+            if (old is gadget) \
+              or (old.move > gadget.move) \
+              or (old.move == gadget.move and len(old.insns) > len(gadget.insns)):
                best_gadgets[touched] = gadget

-
        winner = None
        budget = 999999999

@@ -987,13 +1011,11 @@ class ROP(object):
        if isinstance(files, ELF):
            files = [files]

-        hashes = []
+        sha256 = hashlib.sha256()
+        for elf_data in sorted(elf.get_data() for elf in self.elfs):
+            sha256.update(elf_data)

-        for elf in self.elfs:
-            sha256 = hashlib.sha256(elf.get_data()).hexdigest()
-            hashes.append(sha256)
-
-        return os.path.join(cachedir, '_'.join(hashes))
+        return os.path.join(cachedir, sha256.hexdigest())

    def __cache_load(self, elf):
        filename = self.__get_cachefile_name(elf)
@@ -1156,6 +1178,8 @@ class ROP(object):
        regs = set(regs or ())

        for addr, gadget in self.gadgets.items():
+            addr_bytes = set(pack(gadget.address))
+            if addr_bytes & self._badchars:     continue
            if gadget.insns[-1] != 'ret':        continue
            if gadget.move < move:               continue
            if not (regs <= set(gadget.regs)):   continue
@@ -165,7 +165,7 @@ tether = sys.modules[__name__]
 shellcraft = module(__name__, '')

 class LazyImporter:
-    def find_module(self, fullname, path):
+    def find_module(self, fullname, path=None):
        if not fullname.startswith('pwnlib.shellcraft.'):
            return None

@@ -40,6 +40,8 @@ mips = {
    '$31': 31, '$ra': 31,
 }

+mips_list = list(mips)
+
 arm = map('r{}'.format, range(13))
 arm += ["sp", "lr", "pc", "cpsr"]

@@ -211,7 +213,7 @@ def current():
        'arm': arm,
        'thumb': arm,
        'aarch64': aarch64,
-        'mips': mips,
+        'mips': mips_list,
        'powerpc': powerpc
    }[context.arch]

@@ -1,7 +1,7 @@
 <%
  from pwnlib import shellcraft
 %>
-<%page args="filename, mode='O_RDONLY'"/>
+<%page args="filename, flags='O_RDONLY', mode=0"/>
 <%docstring>
 Opens a file
 </%docstring>
@@ -9,4 +9,4 @@ Opens a file
  AT_FDCWD=-100
 %>
    ${shellcraft.pushstr(filename)}
-    ${shellcraft.syscall('SYS_openat', AT_FDCWD, 'sp', mode, 0)}
+    ${shellcraft.syscall('SYS_openat', AT_FDCWD, 'sp', flags, mode)}
@@ -8,5 +8,5 @@ Opens a file and writes its contents to the specified file descriptor.
 </%docstring>

    ${pushstr(filename)}
-    ${syscall('SYS_open', 'rsp', 0, 'O_RDONLY')}
+    ${syscall('SYS_open', 'rsp', 'O_RDONLY', 0)}
    ${syscall('SYS_sendfile', fd, 'rax', 0, 0x7fffffff)}
@@ -10,7 +10,7 @@ Args: [sock (imm/reg) = rbp]
  after     = common.label("after")
 %>

-
+    /* dup() file descriptor ${sock} into stdin/stdout/stderr */
 ${dup}:
    ${amd64.mov('rbp', sock)}

@@ -15,5 +15,5 @@ Example:

 </%docstring>
    ${arm.pushstr(filename)}
-    ${arm.linux.open('sp', 0, int(constants.O_RDONLY))}
+    ${arm.linux.open('sp', int(constants.O_RDONLY), 0)}
    ${arm.linux.sendfile(fd, 'r0', 0, 0x7fffffff)}
@@ -4,7 +4,7 @@ import pwnlib.abi
 import pwnlib.constants
 import pwnlib.shellcraft
 %>
-<%docstring>break(vararg_0, vararg_1, vararg_2, vararg_3, vararg_4) -> str
+<%docstring>break_(vararg_0, vararg_1, vararg_2, vararg_3, vararg_4) -> str

 Invokes the syscall break.

@@ -83,7 +83,7 @@ Returns:
    else:
        raise Exception("Could not locate any syscalls: %r" % syscalls)
 %>
-    /* break(${', '.join(syscall_repr)}) */
+    /* break_(${', '.join(syscall_repr)}) */
 %for name, arg in string_arguments.items():
    ${pwnlib.shellcraft.pushstr(arg, append_null=('\x00' not in arg))}
    ${pwnlib.shellcraft.mov(regs[argument_names.index(name)], abi.stack)}
@@ -95,4 +95,4 @@ Returns:
    ${pwnlib.shellcraft.push(arg)}
 %endfor
    ${pwnlib.shellcraft.setregs(register_arguments)}
-    ${pwnlib.shellcraft.syscall(syscall)}
+    ${pwnlib.shellcraft.syscall(syscall)}
@@ -19,5 +19,5 @@ label = common.label("sendfile_loop")
 %>

    ${sc.pushstr(filename)}
-    ${sc.open('esp', 0, 'O_RDONLY')}
+    ${sc.open('esp', 'O_RDONLY', 0)}
    ${sc.sendfile(fd, 'eax', 0, 0x7fffffff)}
@@ -11,6 +11,7 @@ Args: [sock (imm/reg) = ebp]
  looplabel = common.label("loop")
 %>

+    /* dup() file descriptor ${sock} into stdin/stdout/stderr */
 ${dup}:
    ${mov('ebx', sock)}
    ${mov('ecx', 3)}
@@ -22,5 +22,5 @@ label = common.label("sendfile_loop")
 %>

    ${mips.pushstr(filename)}
-    ${mips.open('$sp', 0, int(constants.O_RDONLY))}
+    ${mips.open('$sp', int(constants.O_RDONLY), 0)}
    ${mips.sendfile(fd, '$v0', 0, 0x7fffffff)}
@@ -1,4 +1,13 @@
 <% from pwnlib.shellcraft import mips %>
-<%docstring>Execute /bin/sh</%docstring>
+<%docstring>Execute /bin/sh
+
+Example:
+
+    >>> p = run_assembly(shellcraft.mips.linux.sh())
+    >>> p.sendline('echo Hello')
+    >>> p.recv()
+    'Hello\n'
+
+</%docstring>

 ${mips.execve('//bin/sh', ['sh'], {})}
@@ -20,6 +20,6 @@ label = common.label("sendfile_loop")
 %>

    ${thumb.pushstr(filename)}
-    ${thumb.linux.open('sp', 0, constants.O_RDONLY)}
+    ${thumb.linux.open('sp', constants.O_RDONLY, 0)}
    ${thumb.mov('r5', 'r0')}
    ${thumb.linux.sendfile(fd, 'r5', 0, 0x7fffffff)}
@@ -9,6 +9,7 @@ Args: [sock (imm/reg) = r6]
  dup       = common.label("dup")
  looplabel = common.label("loop")
 %>
+    /* dup() file descriptor ${sock} into stdin/stdout/stderr */
 ${dup}:
        ${mov('r1', 2)}
        ${mov('r7', 'SYS_dup2')}
@@ -1,4 +1,4 @@
-<% from pwnlib.shellcraft.thumb.linux import mov %>
+<% from pwnlib.shellcraft.thumb import mov %>
 <% from socket import htons %>
 <%page args="port = None"/>
 <%docstring>
@@ -6,6 +6,10 @@

    Finds a connected socket. If port is specified it is checked
    against the peer port. Resulting socket is left in r6.
+
+    Example:
+        >>> enhex(asm(shellcraft.findpeer(1337)))
+        '6ff00006ee4606f101064ff001074fea072707f11f07f54630461fb401a96a4601df0130efdd01994fea11414ff039024fea022202f105029142e4d1'
 </%docstring>
 findpeer:
    /* File descriptor in r6 */
@@ -52,7 +56,7 @@ compare_port:
    ldr r1, [sp, #4]
    lsr r1, #16

-    /* Put the port (${port}) to search for into r1 */
+    /* Put the port (${port}) to search for into r2 */
    ${mov('r2', htons(int(port)))}

    /* Is it the one we have been searching for? */
@@ -1,4 +1,4 @@
-<% from pwnlib.shellcraft.thumb.linux import mov %>
+<% from pwnlib.shellcraft.thumb import mov %>
 <% from socket import htons %>
 <%page args="port, network='ipv4'"/>
 <%docstring>
@@ -6,6 +6,10 @@

    Listens on a TCP port, accept a client and leave his socket in r6.
    Port is the TCP port to listen on, network is either 'ipv4' or 'ipv6'.
+
+    Example:
+        >>> enhex(asm(shellcraft.listen(1337, 'ipv4')))
+        '4ff001074fea072707f119074ff002004ff0010182ea020201df0646004901e00200053906b469464ff0100207f1010701df30464ff0010107f1020701df304681ea010182ea020207f1010701df0646'
 </%docstring>
    /* First create listening socket */
    ${mov('r7', 'SYS_socket')}
@@ -15,8 +15,9 @@ from pwnlib.tubes import listen
 from pwnlib.tubes import process
 from pwnlib.tubes import remote
 from pwnlib.tubes import serialtube
+from pwnlib.tubes import server
 from pwnlib.tubes import sock
 from pwnlib.tubes import ssh
 from pwnlib.tubes import tube

-__all__ = ['tube', 'sock', 'remote', 'listen', 'process', 'serialtube', 'ssh']
+__all__ = ['tube', 'sock', 'remote', 'listen', 'process', 'serialtube', 'server', 'ssh']
@@ -434,11 +434,11 @@ class process(tube):
        # appropriate qemu binary to run it.
        qemu_path = qemu.user_path(arch=binary.arch)

-        if not qemu:
+        if not qemu_path:
            raise exception

        qemu_path = which(qemu_path)
-        if qemu:
+        if qemu_path:
            self._qemu = qemu_path

            args = [qemu_path]
@@ -565,7 +565,7 @@ class process(tube):
        #

        # Create a duplicate so we can modify it safely
-        env = dict(os.environ if env is None else env)
+        env = (os.environ if env is None else env).copy()

        for k,v in env.items():
            if not isinstance(k, (str, unicode)):
@@ -875,7 +875,7 @@ class process(tube):
        from pwnlib.elf import ELF

        for lib, address in self.libs().items():
-            if 'libc.so' in lib:
+            if 'libc.so' in lib or 'libc-' in lib:
                e = ELF(lib)
                e.address = address
                return e
@@ -931,7 +931,7 @@ class process(tube):

        Example:

-            >>> e = ELF('/bin/sh')
+            >>> e = ELF('/bin/bash')
            >>> p = process(e.path)

            In order to make sure there's not a race condition against
@@ -0,0 +1,162 @@
+from __future__ import absolute_import
+
+import errno
+import socket
+import threading
+
+from pwnlib.context import context
+from pwnlib.log import getLogger
+from pwnlib.tubes.sock import sock
+from pwnlib.tubes.remote import remote
+
+log = getLogger(__name__)
+
+class server(sock):
+    r"""Creates an TCP or UDP-server to listen for connections. It supports
+    both IPv4 and IPv6.
+
+    Arguments:
+        port(int): The port to connect to.
+            Defaults to a port auto-selected by the operating system.
+        bindaddr(str): The address to bind to.
+            Defaults to ``0.0.0.0`` / `::`.
+        fam: The string "any", "ipv4" or "ipv6" or an integer to pass to :func:`socket.getaddrinfo`.
+        typ: The string "tcp" or "udp" or an integer to pass to :func:`socket.getaddrinfo`.
+        callback: A function to be started on incoming connections. It should take a :class:`pwnlib.tubes.remote` as its only argument.
+
+    Examples:
+
+        >>> s = server(8888)
+        >>> client_conn = remote('localhost', s.lport)
+        >>> server_conn = s.next_connection()
+        >>> client_conn.sendline('Hello')
+        >>> server_conn.recvline()
+        'Hello\n'
+        >>> def cb(r):
+        ...     client_input = r.readline()
+        ...     r.send(client_input[::-1])
+        ...
+        >>> t = server(8889, callback=cb)
+        >>> client_conn = remote('localhost', t.lport)
+        >>> client_conn.sendline('callback')
+        >>> client_conn.recv()
+        '\nkcabllac'
+    """
+
+    #: Local port
+    lport = 0
+
+    #: Local host
+    lhost = None
+
+    #: Socket type (e.g. socket.SOCK_STREAM)
+    type = None
+
+    #: Socket family
+    family = None
+
+    #: Socket protocol
+    protocol = None
+
+    #: Canonical name of the listening interface
+    canonname = None
+
+    #: Sockaddr structure that is being listened on
+    sockaddr = None
+
+    _accepter = None
+
+    def __init__(self, port=0, bindaddr = "0.0.0.0", fam = "any", typ = "tcp",
+                 callback = None, blocking = False, *args, **kwargs):
+        super(server, self).__init__(*args, **kwargs)
+
+        port = int(port)
+        fam  = {socket.AF_INET: 'ipv4',
+                socket.AF_INET6: 'ipv6'}.get(fam, fam)
+
+        fam = self._get_family(fam)
+        typ = self._get_type(typ)
+
+        if fam == socket.AF_INET6 and bindaddr == '0.0.0.0':
+            bindaddr = '::'
+
+        h = self.waitfor('Trying to bind to %s on port %d' % (bindaddr, port))
+
+        for res in socket.getaddrinfo(bindaddr, port, fam, typ, 0, socket.AI_PASSIVE):
+            self.family, self.type, self.proto, self.canonname, self.sockaddr = res
+
+            if self.type not in [socket.SOCK_STREAM, socket.SOCK_DGRAM]:
+                continue
+
+            h.status("Trying %s" % self.sockaddr[0])
+            listen_sock = socket.socket(self.family, self.type, self.proto)
+            listen_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+            listen_sock.bind(self.sockaddr)
+            self.lhost, self.lport = listen_sock.getsockname()[:2]
+            if self.type == socket.SOCK_STREAM:
+                listen_sock.listen(1)
+            break
+        else:
+            h.failure()
+            self.error("Could not bind to %s on port %d" % (bindaddr, port))
+
+        h.success()
+
+        self.sock = listen_sock
+        self.connections_waiting = threading.Event()
+        self.connections = []
+        def accepter():
+            while True:
+                h = self.waitfor('Waiting for connections on %s:%s' % (self.lhost, self.lport))
+                while True:
+                    try:
+                        if self.type == socket.SOCK_STREAM:
+                            sock, rhost = listen_sock.accept()
+                        else:
+                            data, rhost = listen_sock.recvfrom(4096)
+                            listen_sock.connect(rhost)
+                            sock = listen_sock
+                            sock.unrecv(data)
+                        sock.settimeout(self.timeout)
+                        break
+                    except socket.error as e:
+                        if e.errno == errno.EINTR:
+                            continue
+                        h.failure()
+                        self.exception("Socket failure while waiting for connection")
+                        sock = None
+                        return
+
+                self.rhost, self.rport = rhost[:2]
+                r = remote(self.rhost, self.rport, sock = sock)
+                h.success('Got connection from %s on port %d' % (self.rhost, self.rport))
+                if callback:
+                    if not blocking:
+                        t = context.Thread(target = callback, args = (r,))
+                        t.daemon = True
+                        t.start()
+                    else:
+                        callback(r)
+                else:
+                    self.connections.append(r)
+                    if not self.connections_waiting.is_set():
+                        self.connections_waiting.set()
+
+        self._accepter = context.Thread(target = accepter)
+        self._accepter.daemon = True
+        self._accepter.start()
+
+    def next_connection(self):
+        if not self.connections_waiting.is_set():
+            self.connections_waiting.wait()
+        conn = self.connections.pop(0)
+        if not self.connections:
+            self.connections_waiting.clear()
+        return conn
+
+    def close(self):
+        # since `close` is scheduled to run on exit we must check that we got
+        # a connection or the program will hang in the `join` call above
+        if self._accepter and self._accepter.is_alive():
+            return
+        super(server, self).close()
@@ -146,13 +146,17 @@ class ssh_channel(sock):
        # However, we need to wait for the return value to propagate,
        # which may not happen by the time .close() is called by tube.recvall()
        tmp_sock = self.sock
+        tmp_close = self.close
+        self.close = lambda: None

        timeout = self.maximum if self.timeout is self.forever else self.timeout
        data = super(ssh_channel, self).recvall(timeout)

        # Restore self.sock to be able to call wait()
+        self.close = tmp_close
        self.sock = tmp_sock
        self.wait()
+        self.close()

        # Again set self.sock to None
        self.sock = None
@@ -418,9 +422,18 @@ class ssh_connecter(sock):
                self.exception(e.message)
                raise

-            sockname = self.sock.get_transport().sock.getsockname()
-            self.lhost = sockname[0]
-            self.lport = sockname[1]
+            try:
+                # Iterate all layers of proxying to get to base-level Socket object
+                curr = self.sock.get_transport().sock
+                while getattr(curr, "get_transport", None):
+                    curr = curr.get_transport().sock
+
+                sockname = curr.getsockname()
+                self.lhost = sockname[0]
+                self.lport = sockname[1]
+            except Exception as e:
+                self.exception("Could not find base-level Socket object.")
+                raise e

            h.success()

@@ -530,7 +543,21 @@ class ssh(Timeout, Logger):
            ssh_agent: If :const:`True`, enable usage of keys via ssh-agent

        NOTE: The proxy_command and proxy_sock arguments is only available if a
-        fairly new version of paramiko is used."""
+        fairly new version of paramiko is used.
+
+        Example proxying:
+
+            >>> s1 = ssh(host='example.pwnme',
+            ...          user='travis',
+            ...          password='demopass')
+            >>> r1 = s1.remote('localhost', 22)
+            >>> s2 = ssh(host='example.pwnme',
+            ...          user='travis',
+            ...          password='demopass',
+            ...          proxy_sock=r1.sock)
+            >>> r2 = s2.remote('localhost', 22) # and so on...
+            >>> for x in r2, s2, r1, s1: x.close()
+        """
        super(ssh, self).__init__(*a, **kw)

        Logger.__init__(self)
@@ -874,6 +901,8 @@ os.chdir(%(cwd)r)
 if env is not None:
    os.environ.clear()
    os.environ.update(env)
+else:
+    env = os.environ

 def is_exe(path):
    return os.path.isfile(path) and os.access(path, os.X_OK)
@@ -961,7 +990,7 @@ except Exception:
 %(func_src)s
 apply(%(func_name)s, %(func_args)r)

-os.execve(exe, argv, os.environ)
+os.execve(exe, argv, env)
 """ % locals()

        script = script.strip()
@@ -1302,7 +1331,7 @@ from ctypes import *; libc = CDLL('libc.so.6'); print(libc.getenv(%r))
        return misc.parse_ldd_output(data)

    def _get_fingerprint(self, remote):
-        cmd = '(openssl sha256 || sha256 || sha256sum) 2>/dev/null < '
+        cmd = '(sha256 || sha256sum || openssl sha256) 2>/dev/null < '
        cmd = cmd + sh_string(remote)
        data, status = self.run_to_end(cmd)

@@ -1617,7 +1646,6 @@ from ctypes import *; libc = CDLL('libc.so.6'); print(libc.getenv(%r))

        self.error('%r does not exist' % file_or_directory)

-
    def download(self, file_or_directory, local=None):
        """download(file_or_directory, local=None)

@@ -1631,7 +1659,10 @@ from ctypes import *; libc = CDLL('libc.so.6'); print(libc.getenv(%r))
        if not self.sftp:
            self.error("Cannot determine remote file type without SFTP")

-        if 0 == self.system('test -d ' + sh_string(file_or_directory)).wait():
+        with self.system('test -d ' + sh_string(file_or_directory)) as io:
+            is_dir = io.wait()
+
+        if 0 == is_dir:
            self.download_dir(file_or_directory, local)
        else:
            self.download_file(file_or_directory, local)
@@ -1992,7 +2023,7 @@ from ctypes import *; libc = CDLL('libc.so.6'); print(libc.getenv(%r))
        if value is not None:
            with open(path, 'w+') as f:
                f.write(value)
-        else:
+        elif os.path.exists(path):
            with open(path, 'r+') as f:
                return f.read()

@@ -732,37 +732,37 @@ class tube(Timeout, Logger):
    def sendafter(self, delim, data, timeout = default):
        """sendafter(delim, data, timeout = default) -> str

-        A combination of ``recvuntil(delim, timeout)`` and ``send(data)``.
+        A combination of ``recvuntil(delim, timeout=timeout)`` and ``send(data)``.
        """

-        res = self.recvuntil(delim, timeout)
+        res = self.recvuntil(delim, timeout=timeout)
        self.send(data)
        return res

    def sendlineafter(self, delim, data, timeout = default):
        """sendlineafter(delim, data, timeout = default) -> str

-        A combination of ``recvuntil(delim, timeout)`` and ``sendline(data)``."""
+        A combination of ``recvuntil(delim, timeout=timeout)`` and ``sendline(data)``."""

-        res = self.recvuntil(delim, timeout)
+        res = self.recvuntil(delim, timeout=timeout)
        self.sendline(data)
        return res

    def sendthen(self, delim, data, timeout = default):
        """sendthen(delim, data, timeout = default) -> str

-        A combination of ``send(data)`` and ``recvuntil(delim, timeout)``."""
+        A combination of ``send(data)`` and ``recvuntil(delim, timeout=timeout)``."""

        self.send(data)
-        return self.recvuntil(delim, timeout)
+        return self.recvuntil(delim, timeout=timeout)

    def sendlinethen(self, delim, data, timeout = default):
        """sendlinethen(delim, data, timeout = default) -> str

-        A combination of ``sendline(data)`` and ``recvuntil(delim, timeout)``."""
+        A combination of ``sendline(data)`` and ``recvuntil(delim, timeout=timeout)``."""

        self.send(data + self.newline)
-        return self.recvuntil(delim, timeout)
+        return self.recvuntil(delim, timeout=timeout)

    def interactive(self, prompt = term.text.bold_red('$') + ' '):
        """interactive(prompt = pwnlib.term.text.bold_red('$') + ' ')
@@ -574,9 +574,9 @@ def update_cyclic_pregenerated(size):
        cyclic_pregen += de_bruijn_gen.next()

 def hexdump_iter(fd, width=16, skip=True, hexii=False, begin=0, style=None,
-                 highlight=None, cyclic=False):
+                 highlight=None, cyclic=False, groupsize=4):
    r"""hexdump_iter(s, width = 16, skip = True, hexii = False, begin = 0,
-                    style = None, highlight = None, cyclic = False) -> str generator
+                    style = None, highlight = None, cyclic = False, groupsize=4) -> str generator

    Return a hexdump-dump of a string as a generator of lines.  Unless you have
    massive amounts of data you probably want to use :meth:`hexdump`.
@@ -584,6 +584,7 @@ def hexdump_iter(fd, width=16, skip=True, hexii=False, begin=0, style=None,
    Arguments:
        fd(file): File object to dump.  Use :meth:`StringIO.StringIO` or :meth:`hexdump` to dump a string.
        width(int): The number of characters per line
+        groupsize(int): The number of characters per group
        skip(bool): Set to True, if repeated lines should be replaced by a "*"
        hexii(bool): Set to True, if a hexii-dump should be returned instead of a hexdump.
        begin(int):  Offset of the first byte to print in the left column
@@ -601,7 +602,7 @@ def hexdump_iter(fd, width=16, skip=True, hexii=False, begin=0, style=None,
        >>> tmp.flush()
        >>> tmp.seek(4)
        >>> print '\n'.join(hexdump_iter(tmp))
-        00000000  48 45 4c 4c  4f 2c 20 57  4f 52 4c 44               │HELL│O, W│ORLD││
+        00000000  48 45 4c 4c  4f 2c 20 57  4f 52 4c 44               │HELL│O, W│ORLD│
        0000000c

        >>> t = tube()
@@ -613,6 +614,9 @@ def hexdump_iter(fd, width=16, skip=True, hexii=False, begin=0, style=None,
    style     = style or {}
    highlight = highlight or []

+    if groupsize < 1:
+        groupsize = width
+
    for b in highlight:
        if isinstance(b, str):
            b = ord(b)
@@ -625,15 +629,10 @@ def hexdump_iter(fd, width=16, skip=True, hexii=False, begin=0, style=None,
    lines       = []
    last_unique = ''
    byte_width  = len('00 ')
-    column_sep  = '  '
-    line_fmt    = '%%(offset)08x  %%(hexbytes)-%is │%%(printable)s│' % (len(column_sep)+(width*byte_width))
    spacer      = ' '
    marker      = (style.get('marker') or (lambda s:s))('│')

-    if hexii:
-        column_sep = ''
-        line_fmt   = '%%(offset)08x  %%(hexbytes)-%is│' % (len(column_sep)+(width*byte_width))
-    else:
+    if not hexii:
        def style_byte(b):
            hbyte = '%02x' % ord(b)
            abyte = b if isprint(b) else '·'
@@ -696,31 +695,38 @@ def hexdump_iter(fd, width=16, skip=True, hexii=False, begin=0, style=None,
        # Generate contents for line
        hexbytes = ''
        printable = ''
+        color_chars = 0
+        abyte = abyte_previous = ''
        for i, b in enumerate(chunk):
            if not hexii:
+                abyte_previous = abyte
                hbyte, abyte = cache[ord(b)]
+                color_chars += len(hbyte) - 2
            else:
                hbyte, abyte = _hexiichar(b), ''

-            if i % 4 == 3 and i < width - 1:
+            if (i + 1) % groupsize == 0 and i < width - 1:
                hbyte += spacer
-                abyte += marker
+                abyte_previous += abyte
+                abyte = marker

            hexbytes += hbyte + ' '
+            printable += abyte_previous
+
+        if abyte != marker:
            printable += abyte

-        if i + 1 < width:
-            delta = width - i - 1
+        dividers_per_line = (width // groupsize)
+        if width % groupsize == 0:
+            dividers_per_line -= 1

-            # How many hex-bytes would we have printed?
-            count = byte_width * delta
-
-            # How many dividers do we need to fill out the line?
-            dividers_per_line = (width // 4) - (1 if width % 4 == 0 else 0)
-            dividers_printed = (i // 4) + (1 if i % 4 == 3 else 0)
-            count += dividers_per_line - dividers_printed
-
-            hexbytes += ' ' * count
+        if hexii:
+            line_fmt = '%%(offset)08x  %%(hexbytes)-%is│' % (width*byte_width)
+        else:
+            line_fmt = '%%(offset)08x  %%(hexbytes)-%is │%%(printable)s│' % (
+                 (width * byte_width)
+                + color_chars
+                + dividers_per_line )

        line = line_fmt % {'offset': offset, 'hexbytes': hexbytes, 'printable': printable}
        yield line
@@ -729,15 +735,16 @@ def hexdump_iter(fd, width=16, skip=True, hexii=False, begin=0, style=None,
    yield line

 def hexdump(s, width=16, skip=True, hexii=False, begin=0,
-            style=None, highlight=None, cyclic=False):
+            style=None, highlight=None, cyclic=False, groupsize=4):
    r"""hexdump(s, width = 16, skip = True, hexii = False, begin = 0,
-               style = None, highlight = None, cyclic = False) -> str generator
+               style = None, highlight = None, cyclic = False, groupsize=4) -> str generator

    Return a hexdump-dump of a string.

    Arguments:
        s(str): The data to hexdump.
        width(int): The number of characters per line
+        groupsize(int): The number of characters per group
        skip(bool): Set to True, if repeated lines should be replaced by a "*"
        hexii(bool): Set to True, if a hexii-dump should be returned instead of a hexdump.
        begin(int):  Offset of the first byte to print in the left column
@@ -760,7 +767,7 @@ def hexdump(s, width=16, skip=True, hexii=False, begin=0,
        00000020

        >>> print hexdump('A'*32, width=8)
-        00000000  41 41 41 41  41 41 41 41   │AAAA│AAAA│
+        00000000  41 41 41 41  41 41 41 41  │AAAA│AAAA│
        *
        00000020

@@ -880,7 +887,7 @@ def hexdump(s, width=16, skip=True, hexii=False, begin=0,
        00000010
        >>> print hexdump('A'*16, width=12)
        00000000  41 41 41 41  41 41 41 41  41 41 41 41  │AAAA│AAAA│AAAA│
-        0000000c  41 41 41 41                            │AAAA││
+        0000000c  41 41 41 41                            │AAAA│
        00000010
        >>> print hexdump('A'*16, width=13)
        00000000  41 41 41 41  41 41 41 41  41 41 41 41  41  │AAAA│AAAA│AAAA│A│
@@ -894,6 +901,15 @@ def hexdump(s, width=16, skip=True, hexii=False, begin=0,
        00000000  41 41 41 41  41 41 41 41  41 41 41 41  41 41 41  │AAAA│AAAA│AAAA│AAA│
        0000000f  41                                               │A│
        00000010
+
+        >>> print hexdump('A'*24, width=16, groupsize=8)
+        00000000  41 41 41 41 41 41 41 41  41 41 41 41 41 41 41 41  │AAAAAAAA│AAAAAAAA│
+        00000010  41 41 41 41 41 41 41 41                           │AAAAAAAA│
+        00000018
+        >>> print hexdump('A'*24, width=16, groupsize=-1)
+        00000000  41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41  │AAAAAAAAAAAAAAAA│
+        00000010  41 41 41 41 41 41 41 41                          │AAAAAAAA│
+        00000018
    """
    s = packing.flat(s)
    return '\n'.join(hexdump_iter(StringIO.StringIO(s),
@@ -903,7 +919,8 @@ def hexdump(s, width=16, skip=True, hexii=False, begin=0,
                                  begin,
                                  style,
                                  highlight,
-                                  cyclic))
+                                  cyclic,
+                                  groupsize))

 def negate(value, width = None):
    """
@@ -190,6 +190,8 @@ def run_in_new_terminal(command, terminal = None, args = None):
          variable), ``x-terminal-emulator`` is used.
        - If tmux is detected (by the presence of the ``$TMUX`` environment
          variable), a new pane will be opened.
+        - If GNU Screen is detected (by the presence of the ``$STY`` environment
+          variable), a new screen will be opened.

    Arguments:
        command (str): The command to run.
@@ -219,6 +221,9 @@ def run_in_new_terminal(command, terminal = None, args = None):
        elif 'TMUX' in os.environ and which('tmux'):
            terminal = 'tmux'
            args     = ['splitw']
+        elif 'STY' in os.environ and which('screen'):
+            terminal = 'screen'
+            args     = ['-t','pwntools-gdb','bash','-c']

    if not terminal:
        log.error('Could not find a terminal binary to use. Set context.terminal to your terminal.')
@@ -479,43 +479,121 @@ def make_unpacker(word_size = None, endianness = None, sign = None, **kwargs):
    else:
        return lambda number: unpack(number, word_size, endianness, sign)

+def _fit(pieces, preprocessor, packer, filler):
+    # Pulls bytes from `filler` and adds them to `pad` until it ends in `key`.
+    # Returns the index of `key` in `pad`.
+    pad = []
+    def fill(key):
+        key = list(key)
+        while len(pad) < len(key) or pad[-len(key):] != key:
+            pad.append(filler.next())
+        return len(pad) - len(key)

+    # Key conversion:
+    # - convert str/unicode keys to offsets
+    # - convert large int (no null-bytes in a machine word) keys to offsets
+    pieces_ = dict()
+    large_key = 2**(context.word_size-8)
+    for k, v in pieces.items():
+        if isinstance(k, (int, long)):
+            if k >= large_key:
+                k = fill(pack(k))
+        elif isinstance(k, unicode):
+            k = fill(k.encode('utf8'))
+        elif isinstance(k, bytearray):
+            k = fill(str(k))
+        elif isinstance(k, str):
+            k = fill(k)
+        else:
+            raise TypeError("flat(): offset must be of type int or str, but got '%s'" % type(k))
+        if k in pieces_:
+            raise ValueError("flag(): multiple values at offset %d" % k)
+        pieces_[k] = v
+    pieces = pieces_

-def _flat(args, preprocessor, packer):
+    # We must "roll back" `filler` so each recursive call to `_flat` gets it in
+    # the right position
+    filler = iters.chain(pad, filler)
+
+    # Build output
+    out = ''
+    for k, v in sorted(pieces.items()):
+        if k < len(out):
+            raise ValueError("flat(): data at offset %d overlaps with previous data which ends at offset %d" % (k, len(out)))
+
+        # Fill up to offset
+        while len(out) < k:
+            out += filler.next()
+
+        # Recursively flatten data
+        out += _flat([v], preprocessor, packer, filler)
+
+    return filler, out
+
+def _flat(args, preprocessor, packer, filler):
    out = []
    for arg in args:

-        if not isinstance(arg, (list, tuple)):
+        if not isinstance(arg, (list, tuple, dict)):
            arg_ = preprocessor(arg)
            if arg_ != None:
                arg = arg_

        if hasattr(arg, '__flat__'):
-            out.append(arg.__flat__())
+            val = arg.__flat__()
        elif isinstance(arg, (list, tuple)):
-            out.append(_flat(arg, preprocessor, packer))
+            val = _flat(arg, preprocessor, packer, filler)
+        elif isinstance(arg, dict):
+            filler, val = _fit(arg, preprocessor, packer, filler)
        elif isinstance(arg, str):
-            out.append(arg)
+            val = arg
        elif isinstance(arg, unicode):
-            out.append(arg.encode('utf8'))
+            val = arg.encode('utf8')
        elif isinstance(arg, (int, long)):
-            out.append(packer(arg))
+            val = packer(arg)
        elif isinstance(arg, bytearray):
-            out.append(str(arg))
+            val = str(arg)
        else:
            raise ValueError("flat(): Flat does not support values of type %s" % type(arg))
+
+        out.append(val)
+
+        # Advance `filler` for "non-recursive" values
+        if not isinstance(arg, (list, tuple, dict)):
+            for _ in xrange(len(val)):
+                filler.next()
+
    return ''.join(out)

@LocalContext
 def flat(*args, **kwargs):
-    """flat(*args, preprocessor = None, word_size = None, endianness = None, sign = None)
+    """flat(*args, preprocessor = None, length = None, filler = de_bruijn(),
+     word_size = None, endianness = None, sign = None) -> str

    Flattens the arguments into a string.

-    This function takes an arbitrary number of arbitrarily nested lists and
-    tuples. It will then find every string and number inside those and flatten
-    them out. Strings are inserted directly while numbers are packed using the
-    :func:`pack` function.
+    This function takes an arbitrary number of arbitrarily nested lists, tuples
+    and dictionaries.  It will then find every string and number inside those
+    and flatten them out.  Strings are inserted directly while numbers are
+    packed using the :func:`pack` function.  Unicode strings are UTF-8 encoded.
+
+    Dictionary keys give offsets at which to place the corresponding values
+    (which are recursively flattened).  Offsets are relative to where the
+    flattened dictionary occurs in the output (i.e. `{0: 'foo'}` is equivalent
+    to `'foo'`).  Offsets can be integers, unicode strings or regular strings.
+    Integer offsets >= `2**(word_size-8)` are converted to a string using
+    `:func:pack`.  Unicode strings are UTF-8 encoded.  After these conversions
+    offsets are either integers or strings.  In the latter case, the offset will
+    be the lowest index at which the string occurs in `filler`.  See examples
+    below.
+
+    Space between pieces of data is filled out using the iterable `filler`.  The
+    `n`'th byte in the output will be byte at index ``n % len(iterable)`` byte
+    in `filler` if it has finite length or the byte at index `n` otherwise.
+
+    If `length` is given, the output will be padded with bytes from `filler` to
+    be this size.  If the output is longer than `length`, a :py:exc:`ValueError`
+    exception is raised.

    The three kwargs `word_size`, `endianness` and `sign` will default to using
    values in :mod:`pwnlib.context` if not specified as an argument.
@@ -524,8 +602,10 @@ def flat(*args, **kwargs):
      args: Values to flatten
      preprocessor (function): Gets called on every element to optionally
         transform the element before flattening. If :const:`None` is
-         returned, then the original value is uded.
-      word_size (int): Word size of the converted integer (in bits).
+         returned, then the original value is used.
+      length: The length of the output.
+      filler: Iterable to use for padding.
+      word_size (int): Word size of the converted integer.
      endianness (str): Endianness of the converted integer ("little"/"big").
      sign (str): Signedness of the converted integer (False/True)

@@ -534,21 +614,48 @@ def flat(*args, **kwargs):
      '\\x01\\x00testABABABABABAB'
      >>> flat([1, [2, 3]], preprocessor = lambda x: str(x+1))
      '234'
-"""
+      >>> flat({12: 0x41414141,
+      ...       24: 'Hello',
+      ...      })
+      'aaaabaaacaaaAAAAeaaafaaaHello'
+      >>> flat({'caaa': ''})
+      'aaaabaaa'
+      >>> flat({12: 'XXXX'}, filler = 'AB', length = 20)
+      'ABABABABABABXXXXABAB'
+      >>> flat({ 8: [0x41414141, 0x42424242],
+      ...       20: 'CCCC'})
+      'aaaabaaaAAAABBBBeaaaCCCC'
+      >>> flat({ 0x61616162: 'X'})
+      'aaaaX'
+      >>> flat({4: {0: 'X', 4: 'Y'}})
+      'aaaaXaaaY'
+
+    """
+    # HACK: To avoid circular imports we need to delay the import of `cyclic`
+    from pwnlib.util import cyclic

    preprocessor = kwargs.pop('preprocessor', lambda x: None)
-    word_size    = kwargs.pop('word_size', None)
+    filler       = kwargs.pop('filler', cyclic.de_bruijn())
+    length       = kwargs.pop('length', None)

    if kwargs != {}:
        raise TypeError("flat() does not support argument %r" % kwargs.popitem()[0])

-    return _flat(args, preprocessor, make_packer(word_size))
+    filler = iters.cycle(filler)
+    out = _flat(args, preprocessor, make_packer(), filler)

+    if length:
+        if len(out) > length:
+            raise ValueError("flat(): Arguments does not fit within `length` (= %d) bytes" % length)
+        out += ''.join(filler.next() for _ in xrange(length - len(out)))

-@LocalContext
-def fit(pieces=None, **kwargs):
-    """fit(pieces, filler = de_bruijn(), length = None, preprocessor = None) -> str
+    return out

+def fit(*args, **kwargs):
+    """Legacy alias for `:func:flat`"""
+    return flat(*args, **kwargs)
+
+"""
    Generates a string from a dictionary mapping offsets to data to place at
    that offset.

@@ -582,97 +689,8 @@ def fit(pieces=None, **kwargs):
      sign (str): Signedness of the converted integer (False/True)

    Examples:
-      >>> fit({12: 0x41414141,
-      ...      24: 'Hello',
-      ...     })
-      'aaaabaaacaaaAAAAeaaafaaaHello'
-      >>> fit({'caaa': ''})
-      'aaaabaaa'
-      >>> fit({12: 'XXXX'}, filler = 'AB', length = 20)
-      'ABABABABABABXXXXABAB'
-      >>> fit({ 8: [0x41414141, 0x42424242],
-      ...      20: 'CCCC'})
-      'aaaabaaaAAAABBBBeaaaCCCC'
-      >>> fit({ 0x61616162: 'X'})
-      'aaaaX'

    """
-    # HACK: To avoid circular imports we need to delay the import of `cyclic`
-    from pwnlib.util import cyclic
-
-    filler       = kwargs.pop('filler', cyclic.de_bruijn())
-    length       = kwargs.pop('length', None)
-    preprocessor = kwargs.pop('preprocessor', lambda x: None)
-
-    if kwargs != {}:
-        raise TypeError("fit() does not support argument %r" % kwargs.popitem()[0])
-
-    packer = make_packer()
-    filler = iters.cycle(filler)
-    out = ''
-
-    if not length and not pieces:
-        return ''
-
-    if not pieces:
-        return ''.join(filler.next() for f in range(length))
-
-    def fill(out, value):
-        while value not in out:
-            out += filler.next()
-        return out, out.index(value)
-
-    # convert str keys to offsets
-    # convert large int keys to offsets
-    pieces_ = dict()
-    for k, v in pieces.items():
-        if isinstance(k, (int, long)):
-            # cyclic() generally starts with 'aaaa'
-            if k >= 0x61616161:
-                out, k = fill(out, pack(k))
-        elif isinstance(k, str):
-            out, k = fill(out, k)
-        else:
-            raise TypeError("fit(): offset must be of type int or str, but got '%s'" % type(k))
-        pieces_[k] = v
-    pieces = pieces_
-
-    # convert values to their flattened forms
-    for k,v in pieces.items():
-        pieces[k] = _flat([v], preprocessor, packer)
-
-    # if we were provided a length, make sure everything fits
-    last = max(pieces)
-    if length and last and (last + len(pieces[last]) > length):
-        raise ValueError("fit(): Pieces do not fit within `length` (= %d) bytes" % length)
-
-    # insert data into output
-    out = list(out)
-    l = 0
-    for k, v in sorted(pieces.items()):
-        if k < l:
-            raise ValueError("fit(): data at offset %d overlaps with previous data which ends at offset %d" % (k, l))
-        while len(out) < k:
-            out.append(filler.next())
-        v = _flat([v], preprocessor, packer)
-        l = k + len(v)
-
-        # consume the filler for each byte of actual data
-        for i in range(len(out), l):
-            filler.next()
-
-        out[k:l] = v
-
-    # truncate/pad output
-    if length:
-        if l > length:
-            raise ValueError("fit(): Pieces does not fit within `length` (= %d) bytes" % length)
-        while len(out) < length:
-            out.append(filler.next())
-    else:
-        out = out[:l]
-
-    return ''.join(out)

 def signed(integer):
    return unpack(pack(integer), signed=True)
@@ -99,9 +99,9 @@ def name(pid):
        Name of process as listed in ``/proc/<pid>/status``.

    Example:
-        >>> pid = pidof('init')[0]
-        >>> name(pid) == 'init'
-        True
+        >>> p = process('cat')
+        >>> name(p.pid)
+        'cat'
    """
    return psutil.Process(pid).name()

@@ -245,6 +245,8 @@ def status(pid):
    try:
        with open('/proc/%d/status' % pid) as fd:
            for line in fd:
+                if ':' not in line:
+                    continue
                i = line.index(':')
                key = line[:i]
                val = line[i + 2:-1] # initial :\t and trailing \n
@@ -1,6 +1,6 @@
 _const_codes = [
    'POP_TOP','ROT_TWO','ROT_THREE','ROT_FOUR','DUP_TOP',
-    'BUILD_LIST','BUILD_MAP','BUILD_TUPLE',
+    'BUILD_LIST','BUILD_MAP','BUILD_TUPLE','BUILD_SET',
    'LOAD_CONST','RETURN_VALUE','STORE_SUBSCR', 'STORE_MAP'
    ]

@@ -1 +1 @@
-__version__ = '3.11.0'
+__version__ = '3.13.0'
@@ -2,6 +2,7 @@
 import glob
 import os
 import platform
+import subprocess
 import sys
 import traceback
 from distutils.command.install import INSTALL_SCHEMES
@@ -45,7 +46,7 @@ for filename in glob.glob('pwnlib/commandline/*'):
 install_requires     = ['paramiko>=1.15.2',
                        'mako>=1.0.0',
                        'pyelftools>=0.2.4',
-                        'capstone',
+                        'capstone>=3.0.5rc2', # See Gallopsled/pwntools#971, Gallopsled/pwntools#1160
                        'ropgadget>=5.3',
                        'pyserial>=2.7',
                        'requests>=2.0',
@@ -54,10 +55,10 @@ install_requires     = ['paramiko>=1.15.2',
                        'pygments>=2.0',
                        'pysocks',
                        'python-dateutil',
-                        'pypandoc',
                        'packaging',
                        'psutil>=3.3.0',
-                        'intervaltree',
+                        'intervaltree<3.0', # See Gallopsled/pwntools#1238
+                        'sortedcontainers<2.0', # See Gallopsled/pwntools#1154
                        'unicorn']

 # Check that the user has installed the Python development headers
@@ -70,23 +71,16 @@ if not os.path.exists(PythonH):
 # Convert README.md to reStructuredText for PyPI
 long_description = ''
 try:
-    import pypandoc
-    try:
-        pypandoc.get_pandoc_path()
-    except OSError:
-        pypandoc.download_pandoc()
-    long_description = pypandoc.convert_file('README.md', 'rst')
-except ImportError:
-    pass
+    long_description = subprocess.check_output(['pandoc', 'README.md', '--to=rst'])
 except Exception as e:
    print >>sys.stderr, "Failed to convert README.md through pandoc, proceeding anyway"
    traceback.print_exc()

-
 setup(
    name                 = 'pwntools',
+    python_requires      = '~=2.7',
    packages             = find_packages(),
-    version              = '3.11.0',
+    version              = '3.13.0',
    data_files           = [('',
                             glob.glob('*.md') + glob.glob('*.txt')),
                            ],
@@ -11,7 +11,7 @@ local_deb_extract()

 install_deb()
 {
-    version=${2:-zesty}
+    version=${2:-artful}
    package=$1
    echo "Installing $package"
    INDEX="http://packages.ubuntu.com/en/$version/amd64/$package/download"
@@ -184,7 +184,7 @@ setup_osx()
 }

 if [[ "$USER" == "travis" ]]; then
-    setup_travis
+#   setup_travis
    setup_android_emulator
 elif [[ "$USER" == "shippable" ]]; then
    sudo apt-get update
Author	SHA1	Message	Date
Arusekk	33524d74a5	Release 3.13.0	2019-11-05 18:39:55 +01:00
Arusekk	d8a2359100	Merge branch 'stable' into beta	2019-11-05 10:40:48 +01:00
xtrm0	215b2e6920	Reduce cache filename length on rop.py (fix #1180 ) (#1204 ) * Reduce cache filename length on rop.py (fix #1180) * ROP cache elfs sorting	2019-11-04 17:11:16 +01:00
Arusekk	6f510a3270	Merge branch 'stable' into beta	2019-10-30 15:49:06 +01:00
ipersons	ff2f4f5bf2	Fix Nested SSH Connectors (#1175 ) * Handle layered proxying in ssh_connecter * better error handling * Fix ssh problems	2019-10-30 15:46:22 +01:00
Arusekk	c0056b71d7	Merge branch 'stable' into beta	2019-10-22 20:51:03 +02:00
Oshawk	92e01a7dca	Fix 'break' syscall (#1355 ) * Fix 'break' syscall Fixed the 'break' syscall by renaming the method to break_ so as not to collide with the keyword. * Bypass pypa/pip#6275	2019-10-22 20:48:45 +02:00
Arusekk	e3690538e7	Merge branch 'stable' into beta	2019-09-22 15:22:02 +02:00
WGH	ecd6d235cd	Fix TypeError: find_module() takes exactly 3 arguments (2 given) (#1342 ) pwnlib.shellcraft installs its custom import finder (LazyImporter) into sys.meta_path. According to PEP 302, finder is expected to have prototype finder.find_module(fullname, path=None) Note that the second argument is optional. pwnlib's LazyImporter has second argument required, which leads to compatibility problems, surfacing when trying to use pwnlib with other third-party modules: >>> import pwnlib >>> import faker Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/tmp/.venv/lib/python2.7/site-packages/faker/__init__.py", line 2, in <module> from faker.factory import Factory # noqa F401 File "/tmp/.venv/lib/python2.7/site-packages/faker/factory.py", line 12, in <module> from faker.config import AVAILABLE_LOCALES, DEFAULT_LOCALE, PROVIDERS File "/tmp/.venv/lib/python2.7/site-packages/faker/config.py", line 15, in <module> AVAILABLE_LOCALES = find_available_locales(PROVIDERS) File "/tmp/.venv/lib/python2.7/site-packages/faker/utils/loading.py", line 45, in find_available_locales provider_module = import_module(provider_path) File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module __import__(name) File "/tmp/.venv/lib/python2.7/site-packages/faker/providers/internet/__init__.py", line 4, in <module> from text_unidecode import unidecode File "/tmp/.venv/lib/python2.7/site-packages/text_unidecode/__init__.py", line 6, in <module> _replaces = pkgutil.get_data(__name__, 'data.bin').decode('utf8').split('\x00') File "/usr/lib64/python2.7/pkgutil.py", line 576, in get_data loader = get_loader(package) File "/usr/lib64/python2.7/pkgutil.py", line 462, in get_loader return find_loader(fullname) File "/usr/lib64/python2.7/pkgutil.py", line 473, in find_loader loader = importer.find_module(fullname) TypeError: find_module() takes exactly 3 arguments (2 given)	2019-09-22 15:17:21 +02:00
Arusekk	cebd26d85f	Merge branch 'stable' into beta	2019-08-28 11:29:54 +02:00
lockshaw	5d2798750d	Fix timeout parameter passing in sendlineafter and other similar functions (#1277 ) * Fix parameter order of sendlineafter * Fix the same bug for the other functions in tube.py * Update the docstrings	2019-08-27 10:32:24 +02:00
Zach Riggle	b3694ab6cb	Merge pull request #1292 from xairy/stable Provide correct arch name to gdb for sparc64	2019-05-07 17:35:10 -05:00
Andrey Konovalov	94bb5fde0f	Provide correct arch name to gdb for sparc64 "sparc64" is not correct gdb arch name, set it to "sparc:v9" instead.	2019-04-20 15:27:35 +02:00
Zach Riggle	547905fae4	Merge pull request #1264 from foens/stable Fix small issues in thumb shellcode	2019-02-02 03:01:03 -06:00
Kasper Føns	dec0e1d71f	add doctest	2019-02-01 16:35:17 +01:00
Zach Riggle	0d56697583	Merge branch 'stable' into stable	2019-02-01 08:53:30 -06:00
Zach Riggle	c03d08b1c4	Merge pull request #1260 from Arusekk/travis-xenial-1260 Switch Travis CI to 16.04 (xenial)	2019-02-01 08:53:14 -06:00
Kasper Føns	aeee34f62b	Fix erroneous comment	2019-02-01 15:26:49 +01:00
Kasper Føns	3f1dc93245	Fix wrong import of mov instruction	2019-02-01 15:26:28 +01:00
Arusekk	20e9fc2f53	Fix typo	2019-01-31 11:53:38 +01:00
Arusekk	c54ad5647d	Spawning a process to test its name	2019-01-31 11:47:59 +01:00
Arusekk	010c17dd6f	Re-apply #282 changes	2019-01-31 11:14:25 +01:00
Arusekk	d5e02f27ac	util.proc portability	2019-01-31 10:53:27 +01:00
Arusekk	9609374e97	Fix punctuation in disassemblies	2019-01-31 09:52:48 +01:00
Zach Riggle	3ecd151f2b	Merge branch 'stable' into travis-xenial-1260	2019-01-28 16:47:39 -06:00
Zach Riggle	bf10bd70f8	Merge pull request #1252 from Gallopsled/zachriggle-patch-1 Fixes args exception with "=="	2019-01-28 16:47:26 -06:00
Zach Riggle	00bbef79bd	Merge branch 'stable' into travis-xenial-1260	2019-01-28 16:47:14 -06:00
Zach Riggle	afa7753f87	Merge branch 'stable' into zachriggle-patch-1	2019-01-28 15:24:43 -06:00
Arusekk	39dd2c3fe3	Fixed race condition (#1258 ) ssh_channel.close() being called from ssh_channel.recvall()	2019-01-28 15:24:30 -06:00
Zach Riggle	59fe841fe5	Specify Python version and cache directories	2019-01-28 14:43:59 -06:00
Arusekk	c9fb770507	Switch Travis CI to 16.04 (xenial) Ubuntu 16.04 finally includes `binutils-mips-linux-gnu`, which is required. So there is no need to download packages manually anymore. This can help supporting python 3.	2019-01-26 10:17:44 +01:00
Zach Riggle	19eed571a5	Fixes args exception with "==" Fixes #1249	2019-01-20 22:47:35 -06:00
Kyle Zeng	0dc0a9acc0	check qemu_path correctly and give out a reasonable error msg (#1243 )	2019-01-07 14:42:24 -06:00
Zach Riggle	774ed3b85e	Do not use intervaltree 3.xx (#1242 ) In the future, we may want to update Pwntools to use the newer version. In the meantime, pin to 2.xx	2018-12-28 23:19:54 -06:00
Zach Riggle	b8c826e057	Merge branch 'stable' into beta	2018-10-17 13:15:07 -05:00
joesixpack666	292b81af17	close ssh_channel properly in download() (#1208 ) (#1209 ) * close ssh_channel properly in download() (#1208) * Update ssh.py	2018-10-17 13:14:20 -05:00
Zach Riggle	c627115b85	Merge tag '3.12.1' into beta	2018-09-17 17:18:24 -05:00
Zach Riggle	90056fbeca	Version bump to 3.12.1	2018-09-17 17:11:45 -05:00
Zach Riggle	60df8107c0	corefile: Update NT_XXX type names for pyelftools==0.25 (#1198 ) * corefile: Update NT_XXX type names for pyelftools==0.25 * travis: Do not use Sphinx version 1.8.0, it fails with recursive errors * corefile: Fix stack detection and Mapping.__contains__ logic * corefile: Better detection of AT_EXECFN when NT_AUXV does not have it * corefile: Avoid trying to parse corrupt stacks * Fix doctests for new warning * log: Fix warning_once caching	2018-09-17 17:04:40 -05:00
Ricky Zhou	7467402ecf	Work around renamed ENUM_P_TYPE variables in elftools 0.25 and up.	2018-09-17 14:07:24 -05:00
Zach Riggle	7dc76c745f	Merge branch 'stable' into beta	2018-07-02 16:57:53 -05:00
Zach Riggle	80e12c7031	Switch FTP servers to ftp.ubuntu.com Fixes Gallopsled/pwntools#1172	2018-07-02 16:57:16 -05:00
pwn	f4159bdad4	check if ":" exists before index(':') (#1159 ) * check if ":" exists before index(':') * Update proc.py	2018-06-04 11:13:03 -05:00
Zach Riggle	c09091055f	Merge remote-tracking branch 'upstream/stable' into beta	2018-05-31 15:23:13 -05:00
Zach Riggle	26e6bce81a	Fix broken versions of package dependencies (#1162 ) Fixes Gallopsled/pwntools#971 Fixes Gallopsled/pwntools#1154 Fixes Gallopsled/pwntools#1160	2018-05-31 15:22:45 -05:00
Inndy	6af09645e7	Prevent crash from not existed cache file in ssh.checksec() (#1150 ) * Prevent crash from not existed cache file in ssh.checksec() * Use path.exists instead of exception handling * Fix space * Update ssh.py	2018-05-30 14:35:23 -05:00
Bruce Chen	146348f514	fix wrong gadget selection in setRegisters() (#1118 ) (#1156 ) * fix wrong gadget selection in setRegisters() (#1118) * Fix detection logic * Update rop.py * add test cases for filtering out the bad instructions in rop.py	2018-05-30 13:31:33 -05:00
Michael Rodler	fe6619b336	fixed called to qemu.ld_prefix passing env dict as target (#1152 )	2018-05-25 16:21:57 -05:00
Zach Riggle	eeedb2c10b	Merge remote-tracking branch 'upstream/stable' into beta	2018-05-25 14:38:34 -05:00
Bruce Chen	5ae13e8c8c	switch to trusty build in travis ci and fix some test cases (#1155 )	2018-05-25 14:38:06 -05:00
Zach Riggle	2a4a633e23	Merge remote-tracking branch 'upstream/stable' into beta	2018-04-23 16:21:56 -05:00
Alexey Vishnyakov	67473560c7	Check libc- as well (#1131 )	2018-04-02 14:03:38 -05:00
Tyler Lambert	1cb0502157	Replace string format char (#1125 ) * replace string format char * add parentheses	2018-03-19 16:26:03 -05:00
Zach Riggle	b53ec5e3ce	Merge remote-tracking branch 'upstream/stable' into beta	2018-02-28 15:22:16 -06:00
Zach Riggle	9604f7847e	Fix tests relying on broken autocontrib version (#1121 )	2018-02-28 15:09:35 -06:00
Mathias Svensson	1e6f207956	Release 3.13.0beta0	2018-02-22 16:50:59 +01:00
Mathias Svensson	c888a8cbe0	Release 3.12.0	2018-02-22 16:42:25 +01:00
Mathias Svensson	9f9d884790	Merge branch 'beta' into stable	2018-02-22 13:54:37 +01:00
Mathias Svensson	39bc5808ea	Update CHANGELOG from dev	2018-02-22 13:54:19 +01:00
Zach Riggle	b507bd7ed9	Merge branch 'beta' into dev	2018-02-20 14:50:12 -06:00
Zach Riggle	31f7c61c1f	Merge remote-tracking branch 'stable' into beta	2018-02-20 14:49:38 -06:00
Zach Riggle	2a73356c78	Fix test breakage caused by Sphinx deprecation (#1113 ) * Fix test breakage caused by Sphinx deprecation * Fix missing import in RST Closes #1112	2018-02-20 14:49:09 -06:00
Zach Riggle	758a9549c7	Update CHANGELOG	2018-02-20 12:30:57 -06:00
Zach Riggle	2255b7f546	Update asm.py Fixes #1108	2018-02-19 12:08:00 -06:00
Andriamilanto T	cd8f8d3f8d	Removed useragentstring.com as user agents source, fixed BeautifulSoup usages and fetched new UAs (#1107 )	2018-02-13 17:33:40 -06:00
david942j	25c84daa8a	Fix usage of syscall open in shellcraft templates (#1106 )	2018-02-09 12:53:00 -06:00
Arusekk	8a3a577394	Add BUILD_SET to const opcodes (#1103 )	2018-02-07 12:43:37 -06:00
eyalitki	90f23df80b	Implemented the dump() feature in the DynELF. (#1104 ) * Implemented the dump() feature in the DynELF. * Simplified the return value and made read-only pages optional	2018-02-06 15:14:29 -06:00
Zach Riggle	a208251d5f	Fix typo (#1099 )	2018-01-26 17:40:34 -08:00
Zach Riggle	aadf40980d	Enable setting context.os via context.binary (#1101 ) In order to support this, we detect the interpreter used via PT_INTERP Currently, Linux is the default and Android is detected based on the path	2018-01-25 15:32:35 -08:00
Zach Riggle	14b2f29d17	Merge branch 'beta' into dev	2018-01-25 10:12:02 -08:00
Zach Riggle	68085012f1	Merge branch 'stable' into beta	2018-01-25 10:11:33 -08:00
Zach Riggle	5fdc08451d	Workaround broken "pidof" on older Android emulators	2018-01-25 10:10:56 -08:00
Zach Riggle	f3b5a14fb5	Don't accidentally debug init, which causes a kernel panic	2018-01-25 10:10:56 -08:00
Zach Riggle	63dfed74c6	Do not throw an exception if the Corefile cannot be deleted after copy	2018-01-25 10:10:56 -08:00
Dimos Poupos	c7c5019b01	Improved hexdump output (#1094 ) * improved hexdump output & groupsize feature * doctest fix * Parameters rearranged * Update fiddling.py	2018-01-25 09:41:32 -08:00
Morten Brøns-Pedersen	3ff283cbef	unify `flat` and `fit` (#1096 )	2018-01-25 09:38:25 -08:00
Kristoffer Søholm	9859f54a21	Remove pypandoc as a dependency (#1098 ) The project is pretty much unmaintained, and it is much more sane to assume that the packager has pandoc installed.	2018-01-25 09:36:01 -08:00
Christophe GRENIER	3ff7a9e940	Fix for very old openssl: try sha256 and sha256sum before "openssl sha256" (#1068 ) Check for sha256 and sha256sum before trying openssl sha256 as "openssl sha256" doesn't return an error code in some old versions of openssl (OpenSSL 0.9.8g 19 Oct 2007 at least) that doesn't support sha256.	2018-01-23 16:58:56 -08:00
Zach Riggle	7860eecf02	Prevent Python3 installation via "pip3 install pwntools" (#1093 ) Pwntools does not work on Python3, so that it installs successfully is misleading. More information on 'python_requires' can be found here: https://packaging.python.org/tutorials/distributing-packages/#python-requires Specifically, it requires pip 9.0 or better -- but this is the 'official' way to do this. Fixes Gallopsled/pwntools#1092	2018-01-04 16:42:50 -06:00
Mathias Svensson	5c4def05b2	Start working on 3.13.0dev	2018-01-03 08:23:46 +01:00
Mathias Svensson	e50b978398	Release 3.12.0beta0	2018-01-03 08:21:05 +01:00
Zach Riggle	35789287d8	Update changelog for 3.11 and 3.12	2018-01-02 12:53:30 -06:00
Zach Riggle	7a1ab829eb	Expose pwnlib.tubes.server.server in the toplevel	2018-01-02 12:47:43 -06:00
Zach Riggle	899baec704	Android bootldr extraction, gdb fixes (#1085 ) * Add sysroot= argument to gdb.debug, expose via 'pwn debug' Also sets a sane default sysroot for Android debugging * Add missing newline for "set deatch-on-fork off" Introduced in `e89a9b66ac` * Add support for extracting Android boot.img * Fix incorrect variable in 'pwn debug' * Ensure "file /path/to/file" is the first command	2017-12-21 15:18:38 -06:00
Disconnect3d	932cb57bf9	Better error message when gdbserver fails on LD_PRELOAD (#1083 ) When one passes a `LD_PRELOAD` which cannot be preloaded the `process_created` string might end up looking like this: ``` "ERROR: ld.so: object '/bin/bash' from LD_PRELOAD cannot be preloaded (cannot dynamically load executable): ignored.\n" ``` When we end up in such situation, pwndbg shown unintuitive error: ``` $ python -c 'from pwn import ; gdb.debug("/bin/ls", env={"LD_PRELOAD": "/bin/bash"})' [+] Starting local process '/usr/bin/gdbserver': pid 20691 Traceback (most recent call last): File "<string>", line 1, in <module> File "pwnlib/context/__init__.py", line 1349, in setter return function(a, *kw) File "pwnlib/gdb.py", line 397, in debug port = _gdbserver_port(gdbserver, ssh) File "pwnlib/gdb.py", line 234, in _gdbserver_port gdbserver.pid = int(process_created.split()[-1], 0) ValueError: invalid literal for int() with base 0: 'ignored.' ``` After this patch it will show: ``` $ python -c 'from pwn import ; gdb.debug("/bin/ls", env={"LD_PRELOAD": "/bin/bash"})' [+] Starting local process '/usr/bin/gdbserver': pid 20863 Traceback (most recent call last): File "<string>", line 1, in <module> File "pwnlib/context/__init__.py", line 1349, in setter return function(a, kw) File "pwnlib/gdb.py", line 402, in debug port = _gdbserver_port(gdbserver, ssh) File "pwnlib/gdb.py", line 236, in _gdbserver_port 'Failed to spawn process under gdbserver. gdbserver error message: %s' % process_created ValueError: Failed to spawn process under gdbserver. gdbserver error message: ERROR: ld.so: object '/bin/bash' from LD_PRELOAD cannot be preloaded (cannot dynamically load executable): ignored. [] Stopped process '/bin/ls' (pid 20863) ```	2017-12-12 14:22:43 -06:00
Zach Riggle	6a61617c73	Fix emulation of fault_addr (#1066 ) As @bennofs noticed, our detection logic was incorrect: "This is not a good test if I understand the issue correctly. From my tests, it seems that fault_addr was only null before this patch if it would've been >= 2**47. This is because on 64 bit, that's where kernel memory is mapped so we get "protection fault" instead of "segfault". A protection fault does not have the same siginfo as a normal segfault: si_code will be SI_KERNEL = 0x80 and there won't be a fault_addr (see http://elixir.free-electrons.com/linux/v4.14-rc8/source/kernel/signal.c#L1052)"	2017-12-11 17:48:25 -06:00
Devin Neal	ab8879afa3	Add server module (#1067 ) * add server module * modify example * fix travis errors * add queuing and blocking, fix requests * add server.rst, change accepter to daemon * fix import in server.rst * added server entry in sockets.rst * fix import in sockets.rst	2017-12-11 17:27:51 -06:00
bannsec	515f678f2d	Adding GNU Screen Support (#1076 ) * Adding GNU Screen support * Adding doc Fixes Gallopsled/pwntools#1070	2017-12-07 15:25:52 -06:00
bannsec	c3afdd681d	Adding non-default gdb handling (#1074 ) Fixes Gallopsled/pwntools#1071	2017-12-07 15:25:27 -06:00
Owlz	5d537a6189	Fix typo on Gadget.addr -> Gadget.address	2017-12-04 14:30:41 -06:00
Zach Riggle	9ba966ddba	Switch server to ftp.ubuntu.org Currently ftp.debian.org is down on port 21	2017-12-04 14:30:41 -06:00
Zach Riggle	6ea5722f5b	Add a comment to dup shellcodes	2017-11-10 17:00:25 -06:00
Zach Riggle	4e24f665b8	Use hex() for some ROP values (#1050 ) * Use hex() for some ROP values * Call str() on everything else * Fix one more doctest	2017-11-01 16:15:25 -05:00
Zach Riggle	34cedc96d8	Remove old / unused lines in ABI	2017-11-01 16:14:40 -05:00
Zach Riggle	e8065440c2	Use badbytes for ROP.search_iter	2017-11-01 16:14:40 -05:00
Zach Riggle	de34906b4c	Better handling for Corefiles with unsupported architectures	2017-11-01 16:14:40 -05:00
Zach Riggle	e89a9b66ac	Handle debugging under Android better with 'pwn debug' (#1053 ) * Handle debugging under Android better with 'pwn debug' Previously, when using --process or --pid, we would terminate the gdbserver instance when pwntools exited. Now we wait for the user to tell us to exit. * Handle launching Android processes by filename * Use the --multi option with gdbserver; disable detach-on-fork * Intentionally break doctest to make sure its running * Revert "Intentionally break doctest to make sure its running" This reverts commit `00578ad177`.	2017-10-30 14:17:47 -05:00
cpupower	64acece61b	Copy env using .copy method to preserve order (OrderedDict) (#1061 ) * Copy env in process._validate using .copy method to keep the order in case of OrderedDict * Use supplied env object to preserve order in case of OrderedDict	2017-10-30 14:12:45 -05:00
Mathias Svensson	bb9004c298	Begin working on 3.12.0	2017-10-25 12:23:09 +02:00