PR27701 implemented curl handle reuse in debuginfod_client objects,
but with an unexpected bug. Server responses returning an error
"latched" because the curl_easy handles for error cases weren't all
systematically removed from the curl multi handle. This prevented
their proper re-addition the next time.
This version of the code simplfies matters by making only the curl
curl_multi handle long-lived. This turns out to be enough, because it
can maintain a pool of long-lived http/https connections and related
data, and lend them out to short-lived curl_easy handles. This mode
handles errors or hung downloads even better, because the easy handles
don't undergo complex state transitions between reuse.
A new test case confirms this correction via the federating debuginfod
instance (cleaning caches between subtests to make sure http* is being
used and reused).
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
New function in system.h that returns true if a string has a given
prefix, false otherwise. Use it in place of strncmp.
Signed-off-by: Martin Liška <mliska@suse.cz>
Add debuginfod_config_cache for reading and writing to cache
configuration files, make use of the function within
debuginfod_clean_cache and debuginfod_query_server.
In debuginfod_query_server, create 000-permission file on failed
queries. Before querying each BUILDID, if corresponding 000 file
detected, compare its stat mtime with parameter from
.cache/cache_miss_s. If mtime is fresher, then return ENOENT and
exit; otherwise unlink the 000 file and proceed to a new query.
tests: add test in run-debuginfod-find.sh
test if the 000 file is created on failed query; if querying the
same failed BUILDID, whether the query should proceed without
going through server; set the cache_miss_s to 0 and query the same
buildid, and this time should go through the server.
Signed-off-by: Alice Zhang <alizhang@redhat.com>
With PR25365, we accidentally lost the ability to rmdir client-cache
directories corresponding to buildids. Bring this back, with some
attention to a possible race between a client doing cleanup and
another client doing lookups at the same time.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Client objects now carry long-lived curl handles for outgoing
connections. This makes it more efficient for multiple sequential
queries, because the TCP connections and/or TLS state info are kept
around awhile, avoiding O(100ms) setup latencies. debuginfod is
adjusted to take advantage of this for federation. Other clients
should gradually do this too, perhaps including elfutils itself (in
the libdwfl->debuginfod_client hooks).
A large gdb session with 117 debuginfo downloads was observed to run
twice as fast (45s vs. 1m30s wall-clock time), just in nuking this
extra setup latency. This was tested via a debuginfod intermediary:
it should be even faster once gdb reuses its own debuginfod_client.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
On very large servers, each database-stat counting pass can take tens
of minutes (!), and doing it twice per groom pass does not seriously
improve data quality. Just do it once, after stale data removal &
basic sqlite vacuum.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
In order to assist problem diagnosis / monitoring, use this
gnu-flavoured pthread function to set purpose names to the various
child threads debuginfod starts. libmicrohttpd already sets this for
its threads.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Clang warns about this:
../../debuginfod/debuginfod-client.c:899:28: error: implicit conversion from 'long' to 'double' changes value from 9223372036854775807 to 9223372036854775808 [-Werror,-Wimplicit-int-float-conversion]
pa = (dl > LONG_MAX ? LONG_MAX : (long)dl);
~ ^~~~~~~~
/usr/lib64/clang/10.0.1/include/limits.h:47:19: note: expanded from macro 'LONG_MAX'
^~~~~~~~~~~~
<built-in>:38:22: note: expanded from here
^~~~~~~~~~~~~~~~~~~~
Modified for jakub's observation about LONG_MAX overflow.
Signed-off-by: Timm Bäder <tbaeder@redhat.com>
dpkg-deb has been reported to be fragile when running under
debuginfod, whereas bsdtar (libarchive) is happy with all these
flavors of files. Switch to a bsdtar based pipeline, now
equipped with an escaped glob pattern that adapts to a variety
of interior data.tar* compression formats.
No testsuite impact. .ipk format tested with some random openwrt and
kino-extension binaries found on the net. Some of these are built
with out buildid, and hardly any with debuginfo, but whatever, bsdtar
and elfutils extract whatever info is there.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Signed-off-by: Dorinda Bassey <dorindabassey@gmail.com>
A couple of closely related pieces of work allow more early warning
about low storage/memory conditions:
- New prometheus metrics to track filesystem freespace, and more
details about some errors.
- Frequent checking of $TMPDIR freespace, to trigger fdcache
emergency flushes.
- Switch to floating point prometheus metrics, to communicate
fractions - and short time intervals - accurately.
- Fix startup-time pthread-creation error handling.
Testing is smoke-test-level only as it is hard to create
free-space-limited $TMPDIRs. Locally tested against tiny through
medium tmpfs filesystems, with or without sqlite db also there. Shows
a pleasant stream of diagnostics and metrics during shortage but
generally does not fail outright. However, catching an actual
libstdc++- or kernel-level OOM is beyond our ken.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Start using a second sqlite3 database connection for webapi query
servicing. This allows much better concurrency when long-running
grooming operations are in progress.
No testsuite impact. Grooming times are too short to try to hit with
concurrent requests. OTOH the existing tests did show some
interesting regressions that needed fixing, like needing not to
dual-wield db and dbq when doing rpm-dwz-related lookups from during
scanning, and the way in which corrupted databases are reported.
These needed some automated invocations of gdb on the running
debuginfod binaries that just failed their testing, for in-situ
debugging.
Hand-tested for function on a huge 20GB index file. Allowed webapi
queries to be run throughout random points of the grooming process,
including especially the long count(*) report loops before & after.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Move subdirectory parts of the top level .gitignore into appropriate
subdirectories. This would be consistent with ChangeLog files,
currently one has to update the top level ChangeLog file when
the top level .gitignore file is changed in a way that affects
a specific subdirectory only.
Signed-off-by: Dmitry V. Levin <ldv@altlinux.org>
This would allow to switch from "libdebuginfod-" VERSION ".so"
to DEBUGINFOD_SONAME in __libdwfl_debuginfod_init, and to remove the
fall back to dlopen of "libdebuginfod.so" which would no longer be needed.
Signed-off-by: Dmitry V. Levin <ldv@altlinux.org>
Add DEBUGINFOD_SONAME macro to API for use by those of libdebuginfod
clients that would like to dlopen the library in the same way as
__libdwfl_debuginfod_init does.
Signed-off-by: Dmitry V. Levin <ldv@altlinux.org>
The "-" character is not allowed in a metric label_name, whoops,
so use "_" for one of the new sqlite metrics.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
On Mark's request, use a monotonic clock for metrics/reports related
to time interval measurement. gettimeofday can jump a bit, which
could distort metrics. Tests unaffected.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Add metrics for tracking sqlite3 error counts and query performance.
The former looks like a new sibling of the "error_count" family, and
is tested by dd-corrupting a live database file then triggering some
debuginfod activity.
error_count{sqlite3="file is not a database"} 1
The latter looks like _count/_sum pairs for each type of sqlite
prepared-statement used in the code, and is grep smoke-tested. They
should assist a sysadmin in tuning db storage. This example shows a
6.4 ms/operation cost:
sqlite3_milliseconds_count{step-done="rpm-file-intern"} 318
sqlite3_milliseconds_sum{reset="rpm-file-intern"} 2033
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
This doesn't change any functionality, but simply shows something a little
user friendlier when accessing the server "by hand" (in a browser).
Signed-off-by: Mark Wielaard <mark@klomp.org>
Improve monitoring of debuginfod instances by tracking thread_busy
status for the threads responding to http requests. While these are
usually short-lived, longer archive-uncompress operations can take
long enough time to show up on top/uptime. This should also assist
noticing abusive clients and guide scaling of the service.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
debuginfod-find -v enables a progressfn that prints the Progress every
time the callback is called. For slow transfers or big downloads this
can be really verbose (hundreds a times a second). Slow it down a bit,
so it only prints the progress at most 5 times a second.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Use defined constants for permission values. Also add fallback
definitions for them in system.h, to allow for compatibility with
systems that don't provide these macros.
Include system.h in all tests/ files that required it.
Signed-off-by: Érico Rolim <erico.erc@gmail.com>
Signed-off-by: Mark Wielaard <mark@klomp.org>
Uncomment the <libintl.h> include, since textdomain() and
bindtextdomain() functions provided by it are being used.
Signed-off-by: Érico Rolim <erico.erc@gmail.com>
- Make configure.ac test for fts and obstack availability;
- Add fts and obstack ldflags to all files that need them;
- Add missing argp ldflags to programs in debuginfod/.
Signed-off-by: Érico Rolim <erico.erc@gmail.com>
Added new metrics for scanning that allow estimation of its reading
bandwidth. Accelerated responsivity to SIGINT shutdown during
archive-scanning phase, which previously insisted on completely
processing the current archive. Noted in systemd service file that in
the worst case, it might still take a long time. Accelerated
traversals by moving regex -I/-X handling to apply to file names only
(as always documented), so directory traversal metrics are accurate
regardless of their name.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
... and add new metrics about progress of traversal and groom
processes. Correct one control flow abnormality that could
prematurely end a scanner thread and might have accounted for
the inconsistent test results from the previous patch.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
On very large servers, it's desirable to be able to interrupt a rescan
or groom cycle. SIGUSR[12] now do that. (Unfortunately, this is not
practically testable in the testsuite, since these cycles are so fast
on that small dataset.) We also expose more internal progress count
about the grooming pass, so the administrator can assess possible need
to interrupt.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
debuginfod now knows to handle a case where a buildid search is
satisfiable from more than one source (e.g., archive location), but
some of them are invalid. New exception catching beneath the sqlite
scanning loop ensures all possible matches are scanned in case of
errors.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
When a file couldn't be retrieved because of an bad HTTPS certificate
find-debuginfod currently says:
Server query failed: No such file or directory
With this patch it will say:
Server query failed: Connection refused
Signed-off-by: Mark Wielaard <mark@klomp.org>
Add an error_count{} family of metrics for each libc/libarchive/http
exception instance created during operation. Add a family of fdcache*
metrics for tracking fdcache operations and status. Test via a
injecting a permission-000 empty nothing.rpm in the testsuite.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
During a recent from-scratch reindexing of the rpm/deb corpus at
debuginfod.elfutils.org, we found the fdcache chewed up an abnormal
amount of $TMPDIR space. This was due to internal .dwz lookups, which
triggered fdcache prefetching as for a webapi query, but there was not
a timely fdcache eviction pass to clean it up again. Rather than add
that pass, it's better to suppress the prefetching completely, as an
internal .dwz search will only ever need that file, not any others
from the same archive.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Since PR25548, we let debuginfod answer /buildid/HEX/source/PATH
queries with both canonicalized and raw PATHs. It canonicalizes
incoming paths, but still stored the raw paths in the database too.
This near-dupe storage is not needed, since the queries would always
find the canonicalized version too, so stop doing that. This saves
database space/time.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
The BUILD_STATIC build, as used by the coverage setup, was broken for
debuginfod because the DUMMY_LIBDEBUGINFOD case was mixed up. It should
include libcurl_LIBS when NOT doing a dummy build.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Check scheme instead of effective url so that user may abbreviate
DEBUGINFOD_URL. Add one test for scheme free http url.
Notice that libcurl does not provide an almighty scheme free url
support, /path/to/something without FILE:// can not be recognized
in most circumstances, therefore for the neatness of our code
structure, DEBUGINFOD_ URL of scheme "FILE" must be input as URI.
Signed-off-by: Alice Zhang <alizhang@redhat.com>
Make it possible to build just the debuginfod client or to create a
dummy libdebuginfod that doesn't link against libcurl. The dummy library
can be used for bootstrapping. For testing purposes you can also build
debuginfod against the dummy libdebuginfod but then the debuginfod
server will not be able to do delegation.
Signed-off-by: Mark Wielaard <mark@klomp.org>
To make our code build with -Werror as well as against older libmicrohttpd,
we must conditionalize the data type (int vs. enum) returned by callbacks
and some mhd functions.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
When allocating handle_data we should check for out of memory failures.
Also when the allocation has succeeded make sure we always clean up by
going to out1 on any future errors. So move the curl_multi_init call
earlier, because that goes to out0 on failure.
Signed-off-by: Mark Wielaard <mark@klomp.org>
When is debuginfod_query_server is given an hexadecimal string as
build-id build_id_len will be zero. We were checking the size of
the build_id_bytes destination string instead of the string length
of build_id input string. Make sure the input string is not too
big or strcpy might overwrite then end of the build_id_bytes array.
Signed-off-by: Mark Wielaard <mark@klomp.org>
We need to make sure that we can always place a zero terminator at
the end of suffix when we are copying the filename. So add one more
char to the suffix array. And make sure that we can always add an
extra escape character when we need to escape the current character.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Although we check for and/or create the interval_path right before,
there is still a possibility that the fopen call fails. Handle that
as if the file is unreadable.
Signed-off-by: Mark Wielaard <mark@klomp.org>
zhanghaibo