Client objects now carry long-lived curl handles for outgoing
connections. This makes it more efficient for multiple sequential
queries, because the TCP connections and/or TLS state info are kept
around awhile, avoiding O(100ms) setup latencies. debuginfod is
adjusted to take advantage of this for federation. Other clients
should gradually do this too, perhaps including elfutils itself (in
the libdwfl->debuginfod_client hooks).
A large gdb session with 117 debuginfo downloads was observed to run
twice as fast (45s vs. 1m30s wall-clock time), just in nuking this
extra setup latency. This was tested via a debuginfod intermediary:
it should be even faster once gdb reuses its own debuginfod_client.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
One place in the description referred to the parameter for
.deb/.ddeb files as "-D" as opposed to the correct "-U".
Signed-off-by: Andreas Ziegler <andreas.ziegler@fau.de>
A couple of closely related pieces of work allow more early warning
about low storage/memory conditions:
- New prometheus metrics to track filesystem freespace, and more
details about some errors.
- Frequent checking of $TMPDIR freespace, to trigger fdcache
emergency flushes.
- Switch to floating point prometheus metrics, to communicate
fractions - and short time intervals - accurately.
- Fix startup-time pthread-creation error handling.
Testing is smoke-test-level only as it is hard to create
free-space-limited $TMPDIRs. Locally tested against tiny through
medium tmpfs filesystems, with or without sqlite db also there. Shows
a pleasant stream of diagnostics and metrics during shortage but
generally does not fail outright. However, catching an actual
libstdc++- or kernel-level OOM is beyond our ken.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Add DEBUGINFOD_SONAME macro to API for use by those of libdebuginfod
clients that would like to dlopen the library in the same way as
__libdwfl_debuginfod_init does.
Signed-off-by: Dmitry V. Levin <ldv@altlinux.org>
On very large servers, it's desirable to be able to interrupt a rescan
or groom cycle. SIGUSR[12] now do that. (Unfortunately, this is not
practically testable in the testsuite, since these cycles are so fast
on that small dataset.) We also expose more internal progress count
about the grooming pass, so the administrator can assess possible need
to interrupt.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
When a file couldn't be retrieved because of an bad HTTPS certificate
find-debuginfod currently says:
Server query failed: No such file or directory
With this patch it will say:
Server query failed: Connection refused
Signed-off-by: Mark Wielaard <mark@klomp.org>
Make it possible to build just the debuginfod client or to create a
dummy libdebuginfod that doesn't link against libcurl. The dummy library
can be used for bootstrapping. For testing purposes you can also build
debuginfod against the dummy libdebuginfod but then the debuginfod
server will not be able to do delegation.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Document and sanity check the format of the header string form that can
be passed to debuginfod_add_http_header. It should contain precisely
one colon, which cannot be the first or last character. And the function
should only be used to add optional headers, not replace any existing
standard ones. Anything else isn't supported.
Signed-off-by: Mark Wielaard <mark@klomp.org>
When starting debuginfod with --port=0 it would start using a random port
(possibly different for ipv4 and ipv6). This seems to be an accidental
and not very useful functionality. Just produce an error when started
with --port=0.
https://sourceware.org/bugzilla/show_bug.cgi?id=25728
Signed-off-by: Mark Wielaard <mark@klomp.org>
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Extend the debuginfod-find command line interface to permit
/file/names instead of only buildid hexadecimal strings.
v2: move code into debuginfod-find; client API remains buildid only.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Extend the debuginfod client API with a function to stuff outgoing
headers into libcurl http transfers. Use this from debuginfod so
federated trees of debuginfod/httpd can trace back to to the
originating client for administrative purposes. Docs & test included.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Programs are sometimes compiled with source path names containing
noise like /./ or // or /foo/../, and these survive into DWARF. This
code allows either raw or canonicalized pathnames in the webapi, by
letting the client pass things verbatim, and letting the server
store/accept both raw and canonicalized path names for source files.
Tests included & docs updated.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Reported-by: Eli Schwartz <eschwartz@archlinux.org>
Tested-by: Eli Schwartz <eschwartz@archlinux.org>
This function lets a client know, during or after a progressfn
callback, what the url of the winning outgoing download is/was.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Add a pair of functions to associate a void* parameter with a client
object. Requested by GDB team as a way to pass file names and such
user-interface data through to a progressfn callback.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
PR25502: debuginfod client should default to XDG cache
Location of client cache now defaults to $XDG_CACHE_HOME/debuginfod_client.
If XDG_CACHE_HOME is not set then fallback to $HOME/.cache/debuginfod_client.
Also maintain backwards compatibility with the previous default path-
if $HOME/.debuginfod_client_cache exists, use that instead of the new
defaults.
Signed-off-by: Aaron Merey <amerey@redhat.com>
It was reported that libarchive (bsdtar) at least as far back as rhel7
(3.1.2) can natively process RPM files, so there's no need to mediate
those accesses through rpm2cpio. There's no noteworthy performance or
testing impact.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Signed-off-by: Mark Wielaard <mark@klomp.org>
Introduce new option --fdcache-prefetch to accelerate repeated
queries from the same debuginfo archive.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
I've copied the type name for the progress function from the manual and
noticed that it didn't compile...
Signed-off-by: Konrad Kleine <kkleine@redhat.com>
Add a '-Z EXT[=CMD]' option to debuginfod, which lets it scan any given
extension and run CMD on it to unwrap distro archives. For example,
for arch-linux pacman files, -Z '.tar.zst=zstdcat' lets debuginfod
grok debug and source content in split-debuginfo files.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
An earlier variant of the default progress function could write to any
file. Which is still in the documentation. But the actual implementation
just uses stderr. Fix the documentation to match.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Add a facility to service webapi and dwz/altdebug requests that
resolve to archives via a $TMPDIR file cache. This permits
instantaneous dwz resolution during -debuginfo rpm scanning, and also
instantanous duplicate webapi requests. The cache is limited both in
number of entries and in storage space. Heuristics provide
serviceable defaults.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
We switch from a thread per supplied PATH, with a semaphore based
concurrency control, to a fixed number of worker threads collecting
the result of a plain directory traversal being put into a work queue.
This allows maximal continuous concurrency, even if the PATH
directories are dramatically differently sized. There is no more need
to use concurrency-motivated subdirectory wildcards for PATH entries:
just a single top level directory will work fast. doc & tests incl.
Use just one timeout using CURLOPT_LOW_SPEED_TIME (default 90 seconds)
and CURLOPT_LOW_SPEED_LIMIT (100K).
Signed-off-by: Mark Wielaard <mark@klomp.org>
Add support for scanning .deb / .ddeb files, enabled with a new
command line option "-U". Using a synthetic .deb/.ddeb from a Ubuntu
18 machine, extend the debuginfod testsuite with some .deb processing,
if the dpkg-deb binary is installed.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
This facility allows a default progress-printing function to be
installed if the given environment variable is set. Some larger usage
experience (systemtap fetching kernels) indicates the default timeout
is too short, so forked it into a connection timeout (default short)
and a transfer timeout (default unlimited).
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Add a mandatory debuginfod_begin()/_end() call pair to manage a client
object that represents persistent but non-global state. From libdwfl,
dlopen the debuginfod.so client library early on. This hopefully
makes sure that the code (and the libcurl.so dependency) is loaded
before the program goes into multi-threaded mode.
Signed-off-by: Mark Wielaard <mark@klomp.org>
This webapi extensions allows admins to hook up debuginfod to a
prometheus-compatible monitoring system for general situational
statistics. The metrics are simple enough that local curl requests
can give a user a sense of what's going on. The metrics are
documented as unstable with respect to future versions.
In order to support file/rpm archives that are organized via symlink
trees, add an "-L" option to debuginfod, meaning about the same as for
find(1) or ls(1): to traverse rather than ignore symlinks.
For interactive clients such as gdb, interruptibility is important for
usability during longer downloads. This patchset adds a
download-progress callback function to the debuginfod client library,
with which a caller app can interrupt a download as well as be
notified of its quantitative progress.
Add the server to the debuginfod/ subdirectory. This is a highly
multithreaded c++11 program (still buildable on rhel7's gcc 4.8,
which is only partly c++11 compliant). Includes an initial suite
of tests, man pages, and a sample systemd service.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Signed-off-by: Aaron Merey <amerey@redhat.com>
Introduce the debuginfod/ subdirectory, containing the client for a
new debuginfo-over-http service, in shared-library and command-line
forms. Two functions in libdwfl make calls into the client library to
fetch elf/dwarf files by buildid, as a fallback. Instead of normal
dynamic linking (thus pulling in a variety of curl dependencies),
the libdwfl hooks use dlopen/dlsym. Server & tests coming in patch 2.
Signed-off-by: Aaron Merey <amerey@redhat.com>
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
It is already possible to select the symbol table to print by name,
using --symbols=SECTION. This allows printing the dynamic symbol table
with --symbols=.dynsym. binutils readelf allows printing just the
dynamic symbol table by type using --dyn-sym. Add the same option
and document it. Also add a testcase to show --symbols=.dynsym and
--dyn-sym produce the same output.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Now that we have manpages lets also package them. Rename COPYING to
COPYING-GFDL to make it not clash with the top-level COPYING file.
Also fix up the spec file so it can be used to create a srpm again.
Add eu-stack to the file list.
Signed-off-by: Mark Wielaard <mark@klomp.org>
- Added doc/README
- Updated doc/ChangeLog
- Added a eu-readelf manpage based on the one from binutils.
- Added a brand new manpage for eu-elfclassify the new utility.
- Add some new files in the doc directory and sync makefile with upstream.
- Reenable the compilation of doc directory.
- Disable sgml file building
- Build man pages the automake way
Since V1
- Put man pages in the proper directories.
- Added copy of Gnu Free Documentation License
- Modified eu-readelf.1 man page to match the supported options.
Signed-off-by: Ben Woodard <woodard@redhat.com>
Signed-off-by: Mark Wielaard <mark@klomp.org>
* Change name from "Red Hat elfutils" to "elfutils".
* Update license of standalone tools and test from GPLv2 to GPLv3+.
* Change license of libraries from GPLv2+exception to GPLv2/LGPLv3+.
* Add Developer Certificate of Origin based contributor policy.
top-level:
- COPYING: Upgraded from GPLv2 to GPLv3.
- CONTRIBUTING, COPYING-GPLv2, COPYING-LGPLv3: New files.
- NEWS: Added note about new contribution and license policy.
- Makefile.am: Updated to GPLv3, added new files to EXTRA_DIST.
- configure.ac: Update to GPLv3, changed AC_INIT name to 'elfutils'.
backends, lib, libasm, libcpu, libdw, libdwfl, libebl, libelf:
- All files updated to GPLv2/LGPLv3+. Except some very small files
(<5 lines) which didn't have any headers at all before, the linker
.maps files and the libcpu/defs files which only contain data and
libelf/elf.h which comes from glibc and is under LGPLv2+.
config:
- elfutils.spec.in: Add new License: headers and new %doc files.
- Update all license headers to GPLv2/LGPLv3+ for files used by libs.
src, tests:
- All files updated to GPLv3+. Except for the test bz2 data files, the
linker maps and script files and some very small files (<5 lines)
that don't have any headers.
Signed-off-by: Richard Fontana <rfontana@redhat.com>
Signed-off-by: Mark Wielaard <mjw@redhat.com>
lizhenlin