When the SSL handshake cannot proceed because the server is slow to
respond, poll() with a timeout instead of spinning.
MFC after: 1 week
PR: 298962
Event: Oslo Hackathon 2026
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D60975
When reading an FTP or HTTP response, error out if we read 64 kB before
hitting a newline. Otherwise a runaway or malicious server could have
us spinning for quite a while allocating more and more memory before we
gave up or crashed.
MFC after: 1 week
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D60007
* Reorganize the connection loop to make it a little more readable
* Start the timeout clock earlier
* Correctly calculate the poll timeout before calling poll()
* Don't leak the socket on failure
Fixes: 848f360c8f ("libfetch: Apply timeout to connection attempts")
Fixes: b02e02958d ("libfetch: Fix handling of connection failures")
MFC after: 3 days
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D58512
After commit 848f360c8f, if one tries to connect to a closed port,
fetch reports "Operation now in progress", which is rather confusing.
Return a more useful error message, restoring the old behaviour.
Fixes: 848f360c8f ("libfetch: Apply timeout to connection attempts")
Reviewed by: des
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D58481
Reduce the amount of copying we do when performing buffered reads.
MFC after: 1 week
Reviewed by: op
Differential Revision: https://reviews.freebsd.org/D58113
Mark the socket non-blocking before connecting and poll for completion,
applying fetchTimeout if set.
MFC after: 1 week
Reviewed by: op
Differential Revision: https://reviews.freebsd.org/D57909
Previously, we would read FTP control connection messages and HTTP
reponse headers one character at a time. Now, we read as much as will
fit in our buffer and look for a newline. If there is data left over,
it will be reused by the next fetch_getln() call. This also requires
the addition of a fetch_bufread() which takes the buffer into account,
otherwise the start of the HTTP response body will be stuck in the
buffer after we read the last line of the header.
This should noticeably improve HTTP performance, especially for small
transfers.
MFC after: 1 week
Reviewed by: op
Differential Revision: https://reviews.freebsd.org/D57907
* Make fetch_ssl_read() and fetch_ssl_write() behave more like read(2)
and write(2), and drop fetch_socket_read() in favor of read(2).
* Don't request POLLERR, it's implied.
* Don't needlessly set errno, it's relatively costly.
* Always check for EAGAIN from writev(2), otherwise we will abort on a
short write instead of proceeding to poll(2).
* Always check for EAGAIN from poll(2) even though it can't happen on
FreeBSD; POSIX says it can, and it might in the future.
* Rewrite fetch_read() and fetch_writev() to be more similar to each
other. The main difference is that a partial read is treated as
success while a partial write is treated as failure.
PR: 296316
MFC after: 1 week
Reviewed by: op
Differential Revision: https://reviews.freebsd.org/D57906
Previously, we would read FTP control connection messages and HTTP
reponse headers one character at a time. Now, we read as much as will
fit in our buffer and look for a newline. If there is data left over,
it will be reused by the next fetch_getln() call. This also requires
the addition of a fetch_bufread() which takes the buffer into account,
otherwise the start of the HTTP response body will be stuck in the
buffer after we read the last line of the header.
This should noticeably improve HTTP performance, especially for small
transfers.
MFC after: 1 week
* Make fetch_ssl_read() and fetch_ssl_write() behave more like read(2)
and write(2), and drop fetch_socket_read() in favor of read(2).
* Don't request POLLERR, it's implied.
* Don't needlessly set errno, it's relatively costly.
* Always check for EAGAIN from writev(2), otherwise we will abort on a
short write instead of proceeding to poll(2).
* Always check for EAGAIN from poll(2) even though it can't happen on
FreeBSD; POSIX says it can, and it might in the future.
* Rewrite fetch_read() and fetch_writev() to be more similar to each
other. The main difference is that a partial read is treated as
success while a partial write is treated as failure.
PR: 296316
MFC after: 1 week
If socket() fails because the address family or protocol is unsupported,
just continue with the next address.
MFC after: 1 week
Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D55407
This fixes an issue where the first address that DNS returns is blocked
by a packet filter, so we hang for a while, then the user hits Ctrl-C,
interrupting connect(2), whereupon we move on to the next address, get
a connection, request the file, and return to fetch(1), which sees that
SIGINT was caught and bails.
Note that we make no attempt to enforce fetchTimeout in the connection
phase, and never have. It's feasible, but non-trivial, so we'll leave
it as an exercise for future us.
PR: 293312
MFC after: 1 week
Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D55406
* Provide a wrapper for the common if (verbose) fetch_info(...) idiom.
* Replace remaining instances of fprintf(stderr, ...) with fetch_info().
* Fix a few style nits.
MFC after: 1 week
Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D55405
* Drop the ssl_meth member, there is no reason to hang on to it.
* Replace deprecated SSLv23_client_method() with TLS_client_method().
* Check the return value from SSL_CTX_new().
MFC after: 1 week
PR: 292903
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D55098
This defaults to an error in clang HEAD, use a char-by-char
initializer instead.
Reviewed by: emaste, jhb
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D52532
With the recent inclusion of the FD_CLOFORK and FD_RESOLVE_BENEATH flags,
we must avoid clearing them when setting only FD_CLOEXEC.
Signed-off-by: Ricardo Branco <[email protected]>
Reviewed by: kib, markj
MFC after: 1 month
Pull Request: https://github.com/freebsd/freebsd-src/pull/1766
Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.
Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/
Sponsored by: Netflix
Before certctl(8), there was no system trust store, and libfetch
relied on the CA certificate bundle from the ca_root_nss port to
verify peers.
We now have a system trust store and a reliable mechanism for
manipulating it (to explicitly add, remove, or revoke certificates),
but if ca_root_nss is installed, libfetch will still prefer that to
the system trust store.
With this change, unless explicitly overridden, libfetch will rely on
OpenSSL to pick up the default system trust store.
PR: 256902
MFC after: 3 days
Reviewed by: kevans
Differential Revision: https://reviews.freebsd.org/D42059
This change removes pre-OpenSSL 1.1 supporting code and removes/adjusted
preprocessor conditionals which were tautilogically true as FreeBSD main
has shipped with OpenSSL 1.1+ for some time.
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D40711
As of OpenSSL 1.1 SSL_library_init() and SSL_load_error_strings() are
deprecated. There are replacement initialization functions but they do
not need to be called: "As of version 1.1.0 OpenSSL will automatically
allocate all resources that it needs so no explicit initialisation is
required."
Wrap both calls in an OPENSSL_VERSION_NUMBER block.
PR: 271615
Reviewed by: Pierre Pronchery <[email protected]>
Event: Kitchener-Waterloo Hackathon 202305
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40265
With the change to return EAI_ADDRFAMILY from getaddrinfo(), fetch
would print "Unknown resolver error" for that error. Add that error
and its string to libfetch's table, using an #ifdef just in case.
Correct error strings for EAI_NODATA (although it is currently unused)
and EAI_NONAME. Should maybe rework the code to use gai_strerror(3),
but that doesn't map directly, and the current strings are shortened.
Reviewed in https://reviews.freebsd.org/D37139 with related changes.
Reviewed by: bz
MFC after: 1 month
The length passed to strncpy is the length of the source string, not
the destination buffer. This triggers a non-fatal warning in GCC 12.
Hoewver, the code is also odd. It is really just a memcpy of the
string without its nul terminator. For that use case, memcpy is
clearer.
Reviewed by: imp, emaste
Differential Revision: https://reviews.freebsd.org/D36824
Literal references to /usr/local exist in a large number of files in
the FreeBSD base system. Many are in contributed software, in configuration
files, or in the documentation, but 19 uses have been identified in C
source files or headers outside the contrib and sys/contrib directories.
This commit makes it possible to set _PATH_LOCALBASE in paths.h to use
a different prefix for locally installed software.
In order to avoid changes to openssh source files, LOCALBASE is passed to
the build via Makefiles under src/secure. While _PATH_LOCALBASE could have
been used here, there is precedent in the construction of the path used to
a xauth program which depends on the LOCALBASE value passed on the compiler
command line to select a non-default directory.
This could be changed in a later commit to make the openssh build
consistently use _PATH_LOCALBASE. It is considered out-of-scope for this
commit.
Reviewed by: imp
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D26942
In the successful case, sockshost is not freed prior to return.
The failure case can now be hit after fetch_reopen(), which was not true
before. Thus, we need to make sure to clean up all of the conn resources
which will also close sd. For all of the points prior to fetch_reopen(), we
continue to just close sd.
CID: 1419598, 1419616
fetch_socks5_getenv will allocate memory for the host (or set it to NULL) in
all cases through the function; the caller is responsible for freeing it if
we end up allocating.
While I'm here, I've eliminated a label that just jumps to the next line...
This commit separates out port parsing and validation from grabbing the host
from the env var. The only related bit really is that we need to be more
specific with the delimiter in the IPv6 case.
This change adds SOCKS5 support to the library fetch(3) and updates the man
page.
Details: Within the fetch_connect() function, fetch(3) checks if the
SOCKS5_PROXY environment variable is set. If so, it connects to this host
rather than the end-host. It then initializes the SOCKS5 connection in
accordance with RFC 1928 and returns the resulting conn_t (file descriptor)
for usage by the regular FTP/HTTP handlers.
Design Decision: This change defaults all DNS resolutions through the proxy
by sending all IPs as hostnames. Going forward, another feature might be to
create another environmental variable to toggle resolutions through the
proxy or not..
One may set the SOCKS5_PROXY environment variable in any of the formats:
SOCKS5_PROXY=proxy.example.com
SOCKS5_PROXY=proxy.example.com:1080
SOCKS5_PROXY=192.0.2.0
SOCKS5_PROXY=198.51.100.0:1080
SOCKS5_PROXY=[2001:db8::1]
SOCKS5_PROXY=[2001:db8::2]:1080
Then perform a request with fetch(1).
(note by kevans)
I've since been informed that Void Linux/xbps has a fork of libfetch that
also implements SOCKS5. I may compare/contrast the two in the mid-to-near
future.
Submitted by: Farhan Khan <farhan farhan codes>
Differential Revision: https://reviews.freebsd.org/D18908
value of $HOME and always use the home directory from the passwd
database, unless $HOME was unset, in which case it would use (null).
While there, clean up handling of netrcfd and add debugging aids.
MFC after: 3 weeks
Mainly focus on files that use BSD 2-Clause license, however the tool I
was using mis-identified many licenses so this was mostly a manual - error
prone - task.
The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.
the host argument (e.g. "www.freebsd.org:443"), the service pointer,
which is supposed to point to the port or service part, instead points
to the separator, causing getaddrinfo() to fail.
Note that I have not been able to trigger this bug with fetch(1), nor
do I believe it is possible, as libfetch always parses the host:port
specification itself. I discovered it when I copied fetch_resolve()
into an unrelated project.
MFC after: 3 days
a separator between host and port, and using strchr() to search for it.
Rewrite fetch_resolve() so it handles bracketed literals correctly, and
remove similar code elsewhere to avoid passing unbracketed literals to
fetch_resolve(). Remove #ifdef INET6 so we still parse IP literals
correctly even if we do not have the ability to connect to them.
While there, fix an off-by-one error which caused HTTP 400 errors to be
misinterpreted as redirects.
PR: 217723
MFC after: 1 week
Reported by: bapt, bz, cem, ngie
Dag-Erling Smørgrav