* feat(libcdb): use local debuginfod client cache before hitting servers (#2623)
If gdb / eu-debuginfod-find / abrt has already downloaded the matching
debuginfo for our libc's build-id (cached at $DEBUGINFOD_CACHE_PATH or
$XDG_CACHE_HOME/debuginfod_client/ or ~/.cache/debuginfod_client/),
unstrip_libc() now reuses it directly instead of doing another network
round-trip via debuginfod servers. Validated by ELF magic so non-ELF
junk in the cache is ignored.
Closes#2623
* Update CHANGELOG.md for #2726
* Check our own cache before looking at debuginfod-client
---------
Co-authored-by: Chris (ChrisJr404) <[email protected]>
Co-authored-by: Peace-Maker <[email protected]>
* Support paginated libc.rip API requests
libc.rip used to truncate the result set but supports pagination now. Fetch all results reliably now by requesting all pages.
Fixes#2651
* Update CHANGELOG
The external debuginfod servers are really slow or won't respond at all like debuginfod.ubuntu.com.
The caching in CI doesn't help since we only cache it for 3 month and the query fails afterwards most of the time. We cache the negative response too which doesn't help if the service is flakey itself :D
The nginx docs say that you have to specify the $request_uri manually when a variable is used in the `proxy_pass` value.
Since we just put `/` at the end, all requests were rewritten to `/` instead of the intended e.g. `/api/find` endpoints.
https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass
This broke in #2718 but didn't manifest right away since the responses were still cached in CI. Now after the cache expired the fetch failed.
* feat(libcdb): add extra_mirrors arg + PWN_EXTRA_LIBC_MIRRORS env to download_libraries (#2666)
When the libc.rip URL points at a deb.debian.org / archive.ubuntu.com /
security.debian.org URL whose package has been pulled (e.g. older
vulnerable libc versions like 'GLIBC 2.36-9+deb12u6'), download_libraries
now also tries the same /pool/... path against any user-supplied mirror
prefix(es). Mirrors can be passed via the new keyword argument
extra_mirrors=... (str or list) or the comma/whitespace-separated
PWN_EXTRA_LIBC_MIRRORS environment variable.
Closes#2666
* Update CHANGELOG.md for #2725
* Rename environment variable to `PWNLIB_EXTRA_LIBC_MIRRORS`
All our other user-facing environment variables are `PWNLIB_` instead of `PWN_`.
* Move `import re` to toplevel
* Add type hints to touched functions
* Add example usage for download_libraries with extra_mirrors parameter
* Fix envvar in CHANGELOG
---------
Co-authored-by: Chris (ChrisJr404) <[email protected]>
Co-authored-by: Peace-Maker <[email protected]>
When trying multiple debuginfod servers, the first one to fail would save the failed attempt and block the remaining servers to be tried.
Only save that we couldn't find the debuginfo if we tried all the configured servers.
* Add Ubuntu and Debian debuginfod servers
Most CTF challenges are hosted on Ubuntu or Debian [citation needed], so check their debuginfod servers before trying the federating one as a fallback.
This is required since the elfutils.org server seems down for maintenance occasionally.
* Update CHANGELOG
* Fix installation on Python 3.5 and lower on stable
The unix-ar package requires Python 3.6. Only install it on Python 3.6 then and use the Python 2 fallback for Python <= 3.5 too.
* Update CHANGELOG
* Drop Python 2.7 support / Require Python 3.10
Only test on Python 3 and bump minimal required python version to 3.10.
* Update CHANGELOG
* Allow to install on Python 3.4
* State intend to not artifially raise the required Python version
* Fix sphinx warnings in docstrings
* Update sphinx for Python 3.13 support
Could not import extension sphinx.builders.epub3 (exception: No module named 'imghdr')
`imghdr` was removed in Python 3.13.
* Add local libc database provider for libcdb
* Remove unnecessary assert
* Add docstring for local_libcdb
* Suppress warning output while `context.libdb` sets default
* Testing the local system's libc first
* Set falsely `context.lcoal_libcdb` to turn off local libc-database integration
* Fix docstring
* Make path check in validator
* Fix doctests
* Add CHANGELOG
* Fix gdb.debug doctests
* Accept bytes in ssh.which
But still return a `str` return for backwards-compatibility.
* CI: Make rpyc visible in gdb's python
* Fix not running all doctests
They appear to have to be a free standing block of `>>>` with an empty line above.
* Remove unnecessary text type check in ssh.which
* Match against local system libc first in libcdb
Don't do any requests if the libc currently in use on the system
running the exploit matches already. This is a small short circuit
optimization when the remote target uses the same libc as the
local one.
This looks at the libc loaded by the local shell binary. This appears
more dynamic than hardcoding library paths.
Refs #983
* Update CHANGELOG
* Handle missing SHELL envvar
* Fix hash lookup
* Retry failed lookups after one week in libcdb
The libc databases might be updated to include the searched version,
so a request that failed once might work in the future.
Refs #983
* Update CHANGELOG
* Add ELF.set_runpath()
Shells out to the `patchelf` tool to patch the ELF's RUNPATH. This lets the dynamic loader look for needed shared libraries in the given path first before the system libraries when running the binary.
* Add ELF.set_interpreter()
Shells out to the `patchelf` tool to patch the ELF's PT_INTERP segment. This allows to change the ld.so used when running the binary.
* Add convenience wrapper to set runpath & interpreter
A helper function to patch the ELF such that it uses the dynamic loader and other libraries in the given folder.
* Add method to download libraries matching a libc
Download the matching libraries for the given libc binary and cache them in a local directory using `libcdb.download_libraries()`. The libraries are looked up using libc.rip and fetched from the official package repositories if available.
Only .deb and .pkg.tar.* packages are currently supported (Debian/Ubuntu, Arch).
* Add --libc argument to pwnup template
This generates code into the template which allows you to run the binary using the given libc.
The foreign libc is used by default, but you can choose to run the binary against your system's local libc using the `LOCAL_LIBC` command line argument when executing the exploit script.
* Work around Python 2 tarfile not supporting xz
* Fix crash when libc wasn't found on libc.rip
* Update README
* Use subprocess.check_output instead of pwnlib.process
* Special case Python 2 instead of 3
* Only catch Exceptions instead of everything
Co-authored-by: Arusekk <[email protected]>
* Check launchpad.net for Ubuntu libcs
This mimics the way io12/pwninit obtains the ld.so.
If the download from libc.rip fails, try launchpad.net.
* Hide comment in template when --quiet
* Please confused pylint in PY2 context
Co-authored-by: Arusekk <[email protected]>
---------
Co-authored-by: Arusekk <[email protected]>
* Fix libcdb.unstrip_libc tests
The old debugsymbols seem to have been pruned.
Update hashes to recent Ubuntu 22.04 libcs.
* Switch wget test to httpbingo.org
httpbin.org is very slow to respond.
Cherry-picked from 1998ff0920
* Fix libcdb.unstrip_libc tests
The old debugsymbols seem to have been pruned.
Update hashes to recent Ubuntu 22.04 libcs.
* Switch wget test to httpbingo.org
httpbin.org is very slow to respond.
* libcdb: Add option to search by function offsets
When you're able to leak addresses of the libc library, use `libcdb.search_by_symbol_offsets()` to find and download the matching libc library from https://libc.rip.
If there are multiple matches, the user is prompted to select one interactively. The selection can be saved in the code for future executions of the script.
Fixes#1867
* libcdb: Handle multiple results when looking up by hash
Sometimes the same library appears to be indexed multiple times (see 0b52d2e713d243f0f65d808fcd3fbe372bb3cd32). Handle that situation by selecting the first in the list, since they should all be identical given the same hash value.
* Update CHANGELOG
* Fix off-by-one when pre-selecting a libc
* Do .json() only once
---------
Co-authored-by: Arusekk <[email protected]>
* Switch to federated debuginfod service
elfutils.org offers a federating proxy server which forwards the requests to all other debuginfod servers of the different linux distributions.
https://sourceware.org/elfutils/Debuginfod.html
Use that instead by default to increase our successrate and potentially include new servers automatically once the distro sets one up.
* Use DEBUGINFOD_URLS envvar for server selection
The `DEBUGINFOD_URLS` environment variable is what you use for your normal debuginfod setup to tell it where to look for debug symbols.
Parse it as well and prepend it to the list of servers to try to allow for dynamic adjusting of the target servers.
* Fix CI after Groovy Gorilla went away for libc unstrip test
Build elfutils 0.181 from source since we can't use builds
from a newer ubuntu version anymore.
* Install python wheels in CI
* libcdb: Load debug info and unstrip libc binary
The debug info is obtained from a debuginfod server maintained by systemtap.org. elfutils is used to unstrip the binary and merge the debug info back.
Try to unstrip every libc binary downloaded by the libcdb module by default. #1818
* Install elfutils on CI
* Update CHANGELOG.md
* Add more tests
* Fix negative test
* Raise an exception when eu-unstrip fails
This will hopefully get logged during doctests.
* Install elfutils 0.181 from groovy
Use APT Pinning to select the newer elfutils version from groovy instead of the buggy 0.176 version available in focal.
* Fix missing import of requests
pylint?
Co-authored-by: Arusekk <[email protected]>
* Defer import of requests and urllib
This saves 100ms-150ms of startup time on "from pwn import *"
* Update CHANGELOG
* More deferred imports to save ~20ms
* Try libc.rip for libc lookup
As of #1171 the libcdb repository isn't seeing any updates anymore. When looking up a libc by a hash or build_id try the libc database maintained by @niklasb as well.
The database uses a REST api documented at https://github.com/niklasb/libc-database/blob/master/searchengine/api.yml
* Catch exceptions properly
* Update CHANGELOG
* Add doctests
Co-authored-by: Arusekk <[email protected]>
Rogdham