libretro/RetroArch
1
0 Fork 0
mirror of https://github.com/libretro/RetroArch.git synced 2026-11-08 02:08:22 -10:00
Code Issues Packages Projects Releases 1 Wiki Activity

net_http: drain socket per update; fix close-delimited bodies, response leaks and cache races (#19323)

Browse Source 
* net_http: drain the socket per update, fix close-delimited bodies, free unretrieved response state, and lock the DNS cache

Four defects found while auditing tasks/task_http.c, which is a thin
state machine over net_http_update() and owns no I/O of its own.
Measurements below are from samples/tasks/http/http_transfer_stress_test.c
(added separately), driving this file against a loopback server with
SO_RCVBUF pinned to 64KiB.

1. One recv() per update capped throughput.

socket_receive_all_nonblocking() is a single recv() despite the name,
and net_http_update() issued exactly one per call.
task_http_iterate_transfer() calls net_http_update() once per
task-queue tick, so transfer time was

    (body_size / bytes_per_recv) * tick_period

with bytes_per_recv bounded by the kernel receive buffer.  The tick
period is ~1ms threaded (the retro_sleep(1) in
task_http_iterate_transfer) and one frame -- 16.7ms at 60Hz --
unthreaded.  Measured: 4MiB needed 45 body ticks with 46 recv() calls,
i.e. one recv per tick.  At 60Hz that is 0.8s of pure scheduling
latency for a transfer worth ~0.02s of I/O; 16MiB measured 6.3s
against 0.09s for a plain drain loop.

net_http_update() now loops until the socket reports EAGAIN.  Two
bounds keep a fast or hostile peer from holding the calling thread
(the video thread in unthreaded builds): NET_HTTP_DRAIN_BUDGET caps
bytes per call so a saturated link yields rather than stalling a
frame, and NET_HTTP_DRAIN_MAX_ITERS caps syscalls so a peer dribbling
single bytes cannot spin us.  Exceeding either just returns false and
resumes next tick, as before.  Same syscall count, 45 ticks -> 1.

2. The receive window collapsed before each buffer growth.

For the doubling-growth body types the window handed to recv() is
buflen - pos, which decays toward zero just before each realloc --
measured at 16 bytes with 3.6MB still outstanding.  Grow early against
NET_HTTP_MIN_RECV_WINDOW instead of waiting for pos to reach buflen.
Minimum observed window is now ~48KiB.

3. Connection-close-delimited bodies were discarded entirely.

bodytype defaults to T_FULL and response->len is only ever assigned by
the Content-Length parse or the chunked decoder, so it stays 0 for
T_FULL.  The shared tail of net_http_receive_body() then ran

    if (response->pos > response->len) return false;

which the first body byte trips, failing the transfer with status -1
and dropping the payload.  The only T_FULL path that worked was a
genuinely empty body.  Give T_FULL its own branch that accumulates and
lets the newlen < 0 case at the top of the function terminate it.
Measured: 4MiB EOF-framed body went from 0 bytes returned to complete.

4. net_http_delete() freed no response state, despite "Cleans up all
memory".

Neither response.data nor response.headers was freed, and the doc
comment on net_http_data() claiming the buffer "is freed by
net_http_delete()" was false.  Any path that did not retrieve both
accessors leaked -- notably task_http.c's cancel and OOM branches,
which free the body but never call net_http_headers_ex(), so every
cancelled download leaked its header list.  ASan over the test suite
reported 21113 bytes in 73 allocations, all response header
string_lists.

Track ownership explicitly: the handle owns both until the caller
retrieves them, at which point only the flag flips (the pointers stay
put, so repeated accessor calls remain idempotent -- task_http.c calls
net_http_data() twice on the error path).  net_http_delete() frees
whatever was never claimed.  ASan now reports no leaks.

5. Two data races on the process-global caches.

net_http_connect() called net_http_dns_cache_find() with no lock.  That
is not a read-only lookup: it calls net_http_dns_cache_remove_expired(),
which unlinks entries, freeaddrinfo()s their addrinfo and free()s the
entry, and it also joins resolver threads and bumps timestamps.  One
download could therefore free a cache entry another was walking under
the lock -- a use-after-free of the entry and its addrinfo, not a
benign race.  Take the lock, and NULL-check the result: the entry can
expire between net_http_new_socket() and here, so the dereference was
reachable rather than the "big bug" the old comment assumed.

Separately, net_http_update() wrote conn->in_use unlocked while
net_http_conn_pool_remove_expired() read it under the pool lock and
fed the fd to select().  Take the pool lock for that write.

TSan reported six races across four sites with six concurrent
transfers; it is now clean.

Not addressed here: the lazy `if (!dns_cache_lock) dns_cache_lock =
slock_new();` in net_http_new_socket() is itself unsynchronised, so a
first-use race can leave two threads holding different mutexes.  It
does not reproduce once the locks exist, and fixing it properly wants
an explicit net_http_init()/net_http_deinit() pair called before any
task thread starts -- which is also where the DNS cache and connection
pool should be torn down, since neither has any teardown path today.

Verified with ASan, UBSan, LSan and TSan over the new suite; all clean,
57/57 checks passing.

* samples/tasks/http: add transfer stress and URL dedup regression tests

Two additions alongside the existing http_method_match_test.c.

http_transfer_stress_test.c drives the real net_http.c against a
loopback server and asserts on what task_http.c would observe.  It
covers framing correctness (Content-Length, chunked across chunk sizes
1..65536, and connection-close-delimited), the same again with the
server dribbling a few bytes per write so every header line, chunk
header and chunk boundary is torn across receive calls, per-tick
transfer cost, receive-window floor, failure teardown (truncated
bodies, four malformed head shapes) and six concurrent transfers for
TSan to look at the process-global DNS cache and connection pool.

Two details make the tick-cost measurement mean something.

It interposes socket() to pin SO_RCVBUF, because per-update byte yield
otherwise tracks host /proc/sys/net tuning and autoscaling, and recv()
to count calls and record the requested window.  The server side uses
read(), so only net_socket.c traffic is recorded.  Both are plain
strong definitions resolved through RTLD_NEXT -- no LD_PRELOAD needed,
since net_http.c and net_socket.c link into the binary.

It also sleeps the task-queue tick period between updates.  Without
that the client polls far faster than the peer can fill the socket,
every recv() takes a sliver, and the number reports the server's write
rate rather than the structural per-tick cap -- it swung between 263
and 93207 B/tick across runs.  Sleeping the period lets the receive
buffer saturate between calls, which is the regime RetroArch actually
runs in and the only one where ticks-per-body is stable.  Ticks-per-body
is cadence-independent once saturated, so measuring at the 1ms threaded
cadence keeps the test fast while the 60Hz figure is the same count
scaled by 16.7ms.  Setup ticks (DNS/connect/send) are counted
separately so resolver latency does not pollute throughput.

http_url_dedup_test.c is pure logic with no dependencies, following the
verbatim-oracle convention of http_method_match_test.c and
archive_name_safety_test.c.  task_http.c stores the request URL in
connection_url[NAME_MAX_LENGTH] via strlcpy but task_http_finder()
compares that truncated copy against the raw candidate, so once a URL
reaches NAME_MAX_LENGTH the two can never match and the "concurrent
download of the same file is not allowed" guard silently stops
guarding.  Playlist thumbnail URLs trigger it routinely, and the
consequence is not just wasted bandwidth: both tasks reach their
completion callback, and cb_http_task_download_pl_thumbnail() ends in
filestream_write_file() on an identical path.

The failure is one-directional -- truncation cannot conflate two
distinct URLs, because the candidate is never truncated to match -- so
the test asserts both that a long URL fails to dedup itself today and
that distinct long URLs stay distinct, the latter guarding against a
future fix that hashes the key to a fixed width and introduces the
false positive this bug does not have.  It pins the divergence at
exactly NAME_MAX_LENGTH and builds with -DNAME_MAX_LENGTH=128 for the
small-path platforms.

This test is red until connection_url is made to hold the full URL; it
documents the live defect.

Build:
  make -f Makefile.stress check [SANITIZER=address|thread|undefined]
  cc -Wall -std=gnu99 -g -O0 -o http_url_dedup_test http_url_dedup_test.c

---------

Co-authored-by: libretroadmin <>
This commit is contained in:
LibretroAdminandlibretroadmin authored and GitHub committed 2026-08-02 07:25:03 +02:00
1 parent 296b0080e3
commit f9ecd73148
4 files changed
+1458 -35
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

libretro-common/net/net_http.c
+196 -35
View File
@@ -51,6 +51,17 @@
* the Content-Length header. */
#define NET_HTTP_MAX_CONTENT_LENGTH ((size_t)256 * 1024 * 1024)
/* Receive-window floor. Below this a recv() costs a syscall and a
* round trip to move almost nothing; see the drain loop in
* net_http_update(). */
#define NET_HTTP_MIN_RECV_WINDOW (32 * 1024)
/* Per-call drain bounds. The budget keeps a saturated link from
* stalling a frame; the iteration cap keeps a peer that dribbles
* single bytes from spinning us. */
#define NET_HTTP_DRAIN_BUDGET ((size_t)4 * 1024 * 1024)
#define NET_HTTP_DRAIN_MAX_ITERS 256
enum response_part
{
P_HEADER_TOP = 0,
@@ -91,6 +102,16 @@ static slock_t *conn_pool_lock = NULL;
typedef struct response
{
/* Ownership of data/headers transfers to the caller when it
* retrieves them via net_http_data() / net_http_headers_ex().
* Until then this handle owns them and net_http_delete() frees
* them. Previously net_http_delete() freed neither, so any path
* that did not retrieve both leaked -- which is every cancelled
* download (task_http.c's cancel branch never calls the headers
* accessor) and, before the accessors were reached, every
* transport failure. */
bool owns_data;
bool owns_headers;
char *data;
struct string_list *headers;
size_t pos;
@@ -857,6 +878,8 @@ struct http_t *net_http_new(struct http_connection_t *conn)
state->request.port = conn->port;
state->response.status = -1;
state->response.owns_data = true;
state->response.owns_headers = true;
state->response.buflen = 64 * 1024; /* Start with larger buffer to reduce reallocations */
state->response.data = (char*)malloc(state->response.buflen);
state->response.headers = string_list_new();
@@ -1010,9 +1033,33 @@ static bool net_http_connect(struct http_t *state)
{
struct addrinfo *addr = NULL, *next_addr = NULL;
struct conn_pool_entry *conn = state->conn;
struct dns_cache_entry *dns_entry = net_http_dns_cache_find(state->request.domain, state->request.port);
/* we just used/added this in _new_socket above, if it's not there it's a big bug */
struct dns_cache_entry *dns_entry;
/* net_http_dns_cache_find() is not a read-only lookup: it calls
* net_http_dns_cache_remove_expired(), which unlinks entries,
* freeaddrinfo()s their addrinfo and free()s the entry, and it
* joins resolver threads and bumps timestamps. Calling it here
* without the lock (as this function used to) let one download
* free a cache entry while another was walking the same list under
* the lock -- a use-after-free of the entry and its addrinfo, not
* merely a benign race. ThreadSanitizer flags it as soon as two
* transfers overlap. */
LOCK_DNS_CACHE();
dns_entry = net_http_dns_cache_find(state->request.domain,
state->request.port);
/* Normally populated by net_http_new_socket() just above, but the
* entry can expire between the two calls, so this is not the
* "big bug" the old comment claimed -- it is reachable, and
* dereferencing NULL here crashed. */
if (!dns_entry)
{
UNLOCK_DNS_CACHE();
net_http_log_transport_state(state, "connect_missing_dns_entry", -1);
state->err = true;
return false;
}
addr = dns_entry->addr;
UNLOCK_DNS_CACHE();
#ifndef HAVE_SSL
if (state->ssl)
@@ -1589,6 +1636,20 @@ static bool net_http_receive_body(struct http_t *state, ssize_t newlen)
response->pos = response->len + leftover;
response->part = P_BODY_CHUNKLEN;
}
else if (response->bodytype == T_FULL)
{
/* Body is delimited by the peer closing the connection, so
* there is no expected length to compare against. response->len
* stays 0 for T_FULL (it is only ever assigned by the
* Content-Length parse or the chunked decoder), which is why the
* shared "pos > len" check below cannot be used here: the first
* body byte would trip it and the whole transfer would be
* discarded with status -1. Just accumulate; the terminal
* condition is the newlen < 0 branch at the top of this
* function, which sets P_DONE and shrinks the buffer. */
response->pos += newlen;
response->len = response->pos;
}
else
{
response->pos += newlen;
@@ -1765,43 +1826,123 @@ bool net_http_update(struct http_t *state, size_t* progress, size_t* total)
response = (struct response*)&state->response;
/* Drain the socket, rather than taking a single bite out of it.
*
* socket_receive_all_nonblocking() is one recv() despite the name,
* and this function used to issue exactly one per call. Since
* task_http_iterate_transfer() calls us once per task-queue tick,
* transfer time was
*
* (body_size / bytes_per_recv) * tick_period
*
* with bytes_per_recv capped by the kernel receive buffer. The
* tick period is ~1ms threaded and one frame (16.7ms at 60Hz)
* unthreaded, so a 4MiB download over a 64KiB receive buffer cost
* ~45 ticks -- around 0.8s of pure scheduling latency unthreaded,
* against ~0.02s of actual transfer. Nothing was slow except the
* number of round trips.
*
* Looping until the socket reports EAGAIN collapses that to one
* tick in the common case. Two bounds keep a fast or hostile peer
* from holding the calling thread (which is the video thread in
* unthreaded builds):
*
* - NET_HTTP_DRAIN_BUDGET caps bytes moved per call, so a
* saturated link yields rather than stalling a frame;
* - NET_HTTP_DRAIN_MAX_ITERS caps syscalls per call, so a peer
* dribbling one byte at a time cannot spin us.
*
* Exceeding either bound just returns false and we resume on the
* next tick, which is the pre-existing behaviour. */
{
size_t drained = 0;
int iters = 0;
for (;;)
{
size_t window;
/* Keep a floor under the receive window. For the
* doubling-growth body types (T_CHUNK, T_FULL) the window is
* buflen - pos, which decays toward zero just before each
* realloc; measured as low as 16 bytes with megabytes still
* outstanding. Those reads are round trips that move almost
* nothing. Grow early instead of waiting for pos to reach
* buflen. */
window = response->buflen - response->pos;
if ( window < NET_HTTP_MIN_RECV_WINDOW
&& response->part != P_DONE)
{
char *tmp;
size_t want = response->buflen * 2;
if (want < response->pos + NET_HTTP_MIN_RECV_WINDOW)
want = response->pos + NET_HTTP_MIN_RECV_WINDOW;
if (!(tmp = (char*)realloc(response->data, want)))
{
state->err = true;
break;
}
response->data = tmp;
response->buflen = want;
window = response->buflen - response->pos;
}
#ifdef HAVE_SSL
if (state->ssl && state->conn->ssl_ctx)
_len = ssl_socket_receive_all_nonblocking(state->conn->ssl_ctx, &state->err,
(uint8_t*)response->data + response->pos,
response->buflen - response->pos);
else
if (state->ssl && state->conn->ssl_ctx)
_len = ssl_socket_receive_all_nonblocking(
state->conn->ssl_ctx, &state->err,
(uint8_t*)response->data + response->pos, window);
else
#endif
_len = socket_receive_all_nonblocking(state->conn->fd, &state->err,
(uint8_t*)response->data + response->pos,
response->buflen - response->pos);
_len = socket_receive_all_nonblocking(state->conn->fd,
&state->err,
(uint8_t*)response->data + response->pos, window);
if (response->part < P_BODY)
{
if (_len < 0 || state->err)
{
net_http_log_transport_state(state, "receive_header_failed", _len);
net_http_conn_pool_remove(state->conn);
state->conn = NULL;
state->err = true;
response->part = P_DONE;
response->status = -1;
return true;
}
_len = net_http_receive_header(state, _len);
}
if (response->part < P_BODY)
{
if (_len < 0 || state->err)
{
net_http_log_transport_state(state,
"receive_header_failed", _len);
net_http_conn_pool_remove(state->conn);
state->conn = NULL;
state->err = true;
response->part = P_DONE;
response->status = -1;
return true;
}
_len = net_http_receive_header(state, _len);
}
if (response->part >= P_BODY && response->part < P_DONE)
{
if (!net_http_receive_body(state, _len))
{
net_http_log_transport_state(state, "receive_body_failed", _len);
net_http_conn_pool_remove(state->conn);
state->conn = NULL;
state->err = true;
response->part = P_DONE;
response->status = -1;
return true;
if (response->part >= P_BODY && response->part < P_DONE)
{
if (!net_http_receive_body(state, _len))
{
net_http_log_transport_state(state,
"receive_body_failed", _len);
net_http_conn_pool_remove(state->conn);
state->conn = NULL;
state->err = true;
response->part = P_DONE;
response->status = -1;
return true;
}
}
if (response->part == P_DONE || state->err)
break;
/* _len == 0 is EAGAIN: the socket is drained for now.
* _len < 0 past the header stage is a close, which the body
* parser above has already turned into P_DONE for T_FULL and
* into an error otherwise; either way we are finished here. */
if (_len <= 0)
break;
drained += (size_t)_len;
if ( drained >= NET_HTTP_DRAIN_BUDGET
|| ++iters >= NET_HTTP_DRAIN_MAX_ITERS)
break;
}
}
@@ -1830,7 +1971,14 @@ bool net_http_update(struct http_t *state, size_t* progress, size_t* total)
}
if (state->conn)
{
/* net_http_conn_pool_remove_expired() reads in_use under the
* pool lock and feeds the fd to select(); writing it unlocked
* raced with that walk. */
LOCK_POOL();
state->conn->in_use = false;
UNLOCK_POOL();
}
state->conn = NULL;
if (response->status >= 300 && response->status < 400)
@@ -1878,6 +2026,7 @@ struct string_list *net_http_headers_ex(struct http_t *state, bool accept_err)
return NULL;
if (!accept_err && state->err)
return NULL;
state->response.owns_headers = false;
return state->response.headers;
}
@@ -1910,6 +2059,11 @@ uint8_t* net_http_data(struct http_t *state, size_t* len, bool accept_err)
if (len)
*len = state->response.len;
/* Ownership moves to the caller. The pointer is deliberately left
* in place so repeated calls stay idempotent; only the ownership
* flag changes, and net_http_delete() consults that. */
state->response.owns_data = false;
return (uint8_t*)state->response.data;
}
@@ -1925,6 +2079,13 @@ void net_http_delete(struct http_t *state)
if (state->conn)
net_http_conn_pool_remove(state->conn);
/* Free whatever the caller never took ownership of. Without this
* the doc comment on this function ("Cleans up all memory") was
* simply false for the response side. */
if (state->response.owns_data && state->response.data)
free(state->response.data);
if (state->response.owns_headers && state->response.headers)
string_list_free(state->response.headers);
if (state->request.domain)
free(state->request.domain);
if (state->request.path)
samples/tasks/http/Makefile.stress
+61
View File
@@ -0,0 +1,61 @@
# Transport-level stress/regression harness for net_http.c as driven
# by tasks/task_http.c. Links the real libretro-common networking
# sources so the test exercises production code, not a model of it.
#
# make -f Makefile.stress check
# make -f Makefile.stress check SANITIZER=address
# make -f Makefile.stress check SANITIZER=thread
# make -f Makefile.stress check SANITIZER=undefined
TARGET := http_transfer_stress_test
CORE_DIR := ../../..
LIBRETRO_COMM_DIR := $(CORE_DIR)/libretro-common
SOURCES := \
http_transfer_stress_test.c \
$(LIBRETRO_COMM_DIR)/net/net_http.c \
$(LIBRETRO_COMM_DIR)/net/net_compat.c \
$(LIBRETRO_COMM_DIR)/net/net_socket.c \
$(LIBRETRO_COMM_DIR)/compat/compat_strl.c \
$(LIBRETRO_COMM_DIR)/compat/compat_strcasestr.c \
$(LIBRETRO_COMM_DIR)/compat/compat_posix_string.c \
$(LIBRETRO_COMM_DIR)/compat/fopen_utf8.c \
$(LIBRETRO_COMM_DIR)/string/stdstring.c \
$(LIBRETRO_COMM_DIR)/lists/string_list.c \
$(LIBRETRO_COMM_DIR)/file/file_path.c \
$(LIBRETRO_COMM_DIR)/file/file_path_io.c \
$(LIBRETRO_COMM_DIR)/file/retro_dirent.c \
$(LIBRETRO_COMM_DIR)/streams/file_stream.c \
$(LIBRETRO_COMM_DIR)/vfs/vfs_implementation.c \
$(LIBRETRO_COMM_DIR)/encodings/encoding_utf.c \
$(LIBRETRO_COMM_DIR)/features/features_cpu.c \
$(LIBRETRO_COMM_DIR)/rthreads/rthreads.c \
$(LIBRETRO_COMM_DIR)/time/rtime.c
OBJS := $(SOURCES:.c=.o)
CFLAGS += -Wall -std=gnu99 -g -O1 -fno-omit-frame-pointer \
-I$(LIBRETRO_COMM_DIR)/include -DHAVE_THREADS
LDFLAGS +=
LIBS += -lpthread -ldl -lm
ifneq ($(SANITIZER),)
CFLAGS := -fsanitize=$(SANITIZER) $(CFLAGS)
LDFLAGS := -fsanitize=$(SANITIZER) $(LDFLAGS)
endif
all: $(TARGET)
%.o: %.c
$(CC) -c -o $@ $< $(CFLAGS)
$(TARGET): $(OBJS)
$(CC) -o $@ $^ $(LDFLAGS) $(LIBS)
check: $(TARGET)
./$(TARGET)
clean:
rm -f $(TARGET) $(OBJS)
.PHONY: all check clean
samples/tasks/http/http_transfer_stress_test.c
+813
View File
@@ -0,0 +1,813 @@
/* Copyright (C) 2010-2026 The RetroArch team
*
* ---------------------------------------------------------------------------------------
* The following license statement only applies to this file (http_transfer_stress_test.c).
* ---------------------------------------------------------------------------------------
*
* Permission is hereby granted, free of charge,
* to any person obtaining a copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation the rights to
* use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
* and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
* INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
/* Transport-level regression tests for libretro-common/net/net_http.c
* as driven by tasks/task_http.c.
*
* task_http.c is a thin state machine over net_http_update(); it owns
* no I/O of its own. Every property that matters to it -- does the
* body arrive intact, how many task-queue ticks does a transfer cost,
* does a failure tear down cleanly -- is a property of net_http.c
* reached through that one call. So this test drives the real
* net_http.c against a real loopback server and asserts on what
* task_http.c would observe.
*
* WHAT THIS PINS DOWN
*
* 1. Framing correctness. Content-Length, chunked (across a spread of
* chunk sizes), and connection-close-delimited bodies must all
* reconstruct byte-for-byte. The dribble mode re-runs them with the
* server writing a few bytes at a time, so every header line, chunk
* header and chunk boundary is torn across receive calls.
*
* 2. Tick cost. This is the throughput guard, and it is the reason the
* test bothers with sockets at all.
*
* net_http_update() issues exactly one recv() per call --
* socket_receive_all_nonblocking() is a single recv() despite the
* name. task_http_iterate_transfer() calls net_http_update() once
* per task-queue tick. Transfer time is therefore
*
* updates_required x tick_period
*
* and updates_required is body_size / (bytes drained per recv),
* which is capped by the kernel receive buffer. The tick period is
* ~1ms threaded (the retro_sleep(1) in task_http_iterate_transfer)
* and one frame -- 16.7ms at 60Hz -- unthreaded. A 16 MiB download
* over a 64 KiB receive buffer needs ~370 updates, which is ~0.4s
* threaded and ~6.2s unthreaded, against ~0.09s for a plain drain
* loop. Nothing is slow here except the number of round trips.
*
* ASSERT_BYTES_PER_UPDATE below is the floor. It fails today, on
* purpose: this test is written to hold once net_http_update()
* drains the socket rather than taking a single bite from it.
*
* 3. Receive window. For chunked and EOF-delimited bodies the buffer
* grows by doubling and the window handed to recv() is
* buflen - pos, so it decays toward zero just before each doubling.
* Those near-empty recv() calls are pure round-trip overhead.
* ASSERT_MIN_WINDOW is the floor, measured only while enough body
* remains that a full-size read was possible.
*
* 4. Failure teardown. Truncated bodies, malformed status lines and
* absurd Content-Length values must all end in a clean error with
* no leak -- which is what makes this worth running under
* ASan/LSan/UBSan.
*
* 5. Concurrency. net_http.c keeps a process-global DNS cache and
* connection pool. The concurrent test drives several transfers at
* once so TSan can see the shared-state accesses, including the
* lazy `if (!dns_cache_lock) dns_cache_lock = slock_new();` in
* net_http_new_socket() -- an unsynchronised first-use
* initialisation of the very lock that is supposed to serialise
* that cache.
*
* DETERMINISM
*
* Tick cost depends on the socket receive buffer, which is host-tuned
* and autoscaled. The test therefore interposes socket() and pins
* SO_RCVBUF to TEST_RCVBUF on every stream socket it creates, so the
* per-update byte yield does not drift with /proc/sys/net tuning. It
* also interposes recv() to count calls and record the requested
* window; the server side uses read(), so only net_socket.c's traffic
* is recorded. Both interpositions are plain strong definitions
* resolved through RTLD_NEXT -- no LD_PRELOAD needed, since net_http.c
* and net_socket.c are linked into this binary.
*
* Build and run via the sample Makefile:
* make -f Makefile.stress check
* make -f Makefile.stress check SANITIZER=address
* make -f Makefile.stress check SANITIZER=thread
* make -f Makefile.stress check SANITIZER=undefined
*/
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <dlfcn.h>
#include <errno.h>
#include <pthread.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <net/net_http.h>
#include <net/net_compat.h>
/* Pinned so per-update yield does not track host tuning. */
#define TEST_RCVBUF (64 * 1024)
/* Throughput floor. A transfer must move at least this many bytes per
* net_http_update() call, averaged over the body. One recv() per
* update against a TEST_RCVBUF socket yields well under this. */
#define ASSERT_BYTES_PER_UPDATE (256 * 1024)
/* Receive-window floor, applied only while at least this much body is
* still outstanding -- the genuinely final read is allowed to be
* short. */
#define ASSERT_MIN_WINDOW 4096
/* Tick period used for the throughput measurement: the threaded
* cadence set by the retro_sleep(1) in task_http_iterate_transfer().
* Ticks-per-body is cadence-independent once the receive buffer
* saturates, so the 60Hz unthreaded figure is this same tick count
* scaled by 16.7ms -- measuring at 1ms just keeps the test fast. */
#define TEST_TICK_US 1000
#define BODY_LARGE (4 * 1024 * 1024)
static int failures;
static int checks;
#define CHECK(cond, ...) \
do { \
checks++; \
if (!(cond)) \
{ \
printf(" FAIL: "); printf(__VA_ARGS__); printf("\n"); \
failures++; \
} \
} while (0)
/* ================================================================= */
/* recv()/socket() interposition */
/* ================================================================= */
static ssize_t (*real_recv)(int, void*, size_t, int);
static int (*real_socket)(int, int, int);
/* Recording is enabled only around the client drive loop, and the
* server thread uses read() rather than recv(), so no cross-thread
* contention on these counters arises in the single-transfer tests.
* The concurrent test leaves recording off for exactly that reason --
* it is there for TSan to look at net_http.c's globals, not to
* measure. */
static int g_record;
static long g_recv_calls;
static size_t g_min_window;
static size_t g_window_at_min_outstanding;
static size_t g_outstanding; /* body bytes still expected */
static void record_reset(size_t body)
{
g_recv_calls = 0;
g_min_window = (size_t)-1;
g_window_at_min_outstanding = 0;
g_outstanding = body;
g_record = 1;
}
static void record_stop(void)
{
g_record = 0;
}
ssize_t recv(int fd, void *buf, size_t len, int flags)
{
ssize_t r;
if (!real_recv)
real_recv = (ssize_t(*)(int, void*, size_t, int))dlsym(RTLD_NEXT, "recv");
if (g_record)
{
g_recv_calls++;
/* Only judge the window while a full-size read was actually
* possible; the tail of a transfer legitimately asks for less. */
if (g_outstanding > ASSERT_MIN_WINDOW && len < g_min_window)
{
g_min_window = len;
g_window_at_min_outstanding = g_outstanding;
}
}
r = real_recv(fd, buf, len, flags);
if (g_record && r > 0)
g_outstanding = (g_outstanding > (size_t)r) ? g_outstanding - (size_t)r : 0;
return r;
}
int socket(int domain, int type, int protocol)
{
int fd;
if (!real_socket)
real_socket = (int(*)(int, int, int))dlsym(RTLD_NEXT, "socket");
fd = real_socket(domain, type, protocol);
if (fd >= 0 && type == SOCK_STREAM
&& (domain == AF_INET || domain == AF_INET6))
{
int sz = TEST_RCVBUF;
setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &sz, sizeof(sz));
}
return fd;
}
/* ================================================================= */
/* Loopback server */
/* ================================================================= */
enum framing
{
FRAME_LEN = 0, /* Content-Length */
FRAME_CHUNKED, /* Transfer-Encoding: chunked */
FRAME_EOF /* neither; body ends at close */
};
struct srv_spec
{
size_t body;
size_t chunk; /* FRAME_CHUNKED only */
size_t dribble; /* bytes per write; 0 = write greedily */
enum framing frame;
int truncate; /* close early, mid-body */
const char *raw_head; /* override the status line + headers */
int listen_fd;
int port;
};
static const unsigned char *g_pattern;
static size_t g_pattern_len;
/* A non-uniform payload, so a de-chunker that drops or duplicates a
* span is caught rather than masked by a run of identical bytes. */
static void pattern_init(size_t n)
{
size_t i;
unsigned char *p = (unsigned char*)malloc(n);
unsigned int s = 0x1234567u;
if (!p)
abort();
for (i = 0; i < n; i++)
{
s = s * 1103515245u + 12345u;
p[i] = (unsigned char)(s >> 16);
}
g_pattern = p;
g_pattern_len = n;
}
/* Write with optional dribbling, so boundaries land mid-buffer. */
static int srv_write(int fd, const void *buf, size_t len, size_t dribble)
{
const char *p = (const char*)buf;
size_t sent = 0;
while (sent < len)
{
size_t want = len - sent;
ssize_t w;
if (dribble && want > dribble)
want = dribble;
w = send(fd, p + sent, want, MSG_NOSIGNAL);
if (w <= 0)
return 0;
sent += (size_t)w;
}
return 1;
}
static void *server_thread(void *arg)
{
struct srv_spec *sp = (struct srv_spec*)arg;
int cs = accept(sp->listen_fd, NULL, NULL);
char req[4096];
char head[512];
size_t got = 0;
size_t sent = 0;
size_t stop;
int n;
if (cs < 0)
return NULL;
/* read(), not recv(), so the interposer records only client I/O. */
while (got < sizeof(req) - 1)
{
ssize_t r = read(cs, req + got, sizeof(req) - 1 - got);
if (r <= 0)
break;
got += (size_t)r;
req[got] = '\0';
if (strstr(req, "\r\n\r\n"))
break;
}
if (sp->raw_head)
n = snprintf(head, sizeof(head), "%s", sp->raw_head);
else if (sp->frame == FRAME_LEN)
n = snprintf(head, sizeof(head),
"HTTP/1.1 200 OK\r\nContent-Length: %lu\r\n\r\n",
(unsigned long)sp->body);
else if (sp->frame == FRAME_CHUNKED)
n = snprintf(head, sizeof(head),
"HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n");
else
n = snprintf(head, sizeof(head), "HTTP/1.1 200 OK\r\n\r\n");
if (!srv_write(cs, head, (size_t)n, sp->dribble))
goto done;
/* Truncated runs stop a little past halfway -- far enough in that
* the client has committed to a body, short enough that it cannot
* be mistaken for a complete one. */
stop = sp->truncate ? (sp->body / 2 + 7) : sp->body;
if (sp->frame == FRAME_CHUNKED)
{
while (sent < stop)
{
char cl[32];
size_t c = stop - sent;
int cn;
if (sp->chunk && c > sp->chunk)
c = sp->chunk;
cn = snprintf(cl, sizeof(cl), "%lx\r\n", (unsigned long)c);
if (!srv_write(cs, cl, (size_t)cn, sp->dribble))
goto done;
if (!srv_write(cs, g_pattern + sent, c, sp->dribble))
goto done;
if (!srv_write(cs, "\r\n", 2, sp->dribble))
goto done;
sent += c;
}
if (!sp->truncate)
srv_write(cs, "0\r\n\r\n", 5, sp->dribble);
}
else
srv_write(cs, g_pattern, stop, sp->dribble);
done:
shutdown(cs, SHUT_RDWR);
close(cs);
return NULL;
}
static int srv_start(struct srv_spec *sp, pthread_t *th)
{
struct sockaddr_in sa;
socklen_t sl = sizeof(sa);
int one = 1;
sp->listen_fd = real_socket ? real_socket(AF_INET, SOCK_STREAM, 0)
: socket(AF_INET, SOCK_STREAM, 0);
if (sp->listen_fd < 0)
return 0;
setsockopt(sp->listen_fd, SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one));
memset(&sa, 0, sizeof(sa));
sa.sin_family = AF_INET;
sa.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
sa.sin_port = 0;
if (bind(sp->listen_fd, (struct sockaddr*)&sa, sizeof(sa)) < 0)
return 0;
if (listen(sp->listen_fd, 4) < 0)
return 0;
if (getsockname(sp->listen_fd, (struct sockaddr*)&sa, &sl) < 0)
return 0;
sp->port = ntohs(sa.sin_port);
return pthread_create(th, NULL, server_thread, sp) == 0;
}
/* ================================================================= */
/* Client driver */
/* ================================================================= */
struct xfer_result
{
char *data;
size_t len;
long updates; /* every net_http_update() call */
long setup_updates; /* calls before the first body byte landed */
long body_updates; /* calls from the first body byte onward */
int status;
int err;
};
/* Drive one transfer the way task_http_transfer_handler() does: build
* the connection, then call net_http_update() once per tick until it
* reports done. */
static int run_transfer(int port, struct xfer_result *out, unsigned tick_us)
{
char url[128];
struct http_connection_t *conn;
struct http_t *h;
size_t pos = 0, tot = 0;
/* Generous ceiling; only trips if the state machine wedges. */
const long tick_cap = 40000000L;
memset(out, 0, sizeof(*out));
snprintf(url, sizeof(url), "http://127.0.0.1:%d/payload", port);
if (!(conn = net_http_connection_new(url, "GET", NULL)))
return 0;
net_http_connection_iterate(conn);
if (!net_http_connection_done(conn))
{
net_http_connection_free(conn);
return 0;
}
if (!(h = net_http_new(conn)))
{
net_http_connection_free(conn);
return 0;
}
net_http_connection_free(conn);
/* Separate the two phases. Before the first body byte lands,
* net_http_update() is polling for DNS resolution (which runs on
* its own thread and is only observed at tick granularity), then
* for connect, then sending the request -- none of which is
* throughput. Only the calls from the first body byte onward
* measure how many task-queue ticks the payload costs. */
while (!net_http_update(h, &pos, &tot))
{
out->updates++;
if (pos > 0)
out->body_updates++;
/* Model the real task-queue cadence. Without this the client
* polls far faster than the peer can fill the socket, so each
* recv() takes a sliver and the measurement reports the
* server's write rate rather than the structural per-tick cap.
* Sleeping the tick period lets the receive buffer saturate
* between calls, which is the regime RetroArch actually runs
* in and the only one where ticks-per-body is stable. */
if (tick_us)
usleep(tick_us);
if (out->updates > tick_cap)
{
printf(" (transfer wedged after %ld updates)\n", out->updates);
break;
}
}
out->setup_updates = out->updates - out->body_updates;
out->err = net_http_error(h) ? 1 : 0;
out->status = net_http_status(h);
out->data = (char*)net_http_data(h, &out->len, true);
net_http_delete(h);
return 1;
}
/* ================================================================= */
/* Tests */
/* ================================================================= */
static const char *frame_name(enum framing f)
{
if (f == FRAME_LEN) return "content-length";
if (f == FRAME_CHUNKED) return "chunked";
return "eof";
}
/* Body must reconstruct byte-for-byte under every framing, at every
* chunk size, torn or not. */
static void test_framing(enum framing f, size_t body, size_t chunk,
size_t dribble)
{
struct srv_spec sp;
pthread_t th;
struct xfer_result r;
memset(&sp, 0, sizeof(sp));
sp.body = body;
sp.chunk = chunk;
sp.frame = f;
sp.dribble = dribble;
printf(" framing=%-14s body=%-8lu chunk=%-6lu dribble=%lu\n",
frame_name(f), (unsigned long)body, (unsigned long)chunk,
(unsigned long)dribble);
if (!srv_start(&sp, &th))
{
printf(" SKIP: server start failed\n");
return;
}
record_reset(body);
if (!run_transfer(sp.port, &r, 0))
{
printf(" SKIP: client setup failed\n");
record_stop();
pthread_join(th, NULL);
close(sp.listen_fd);
return;
}
record_stop();
CHECK(r.len == body, "short body: got %lu of %lu",
(unsigned long)r.len, (unsigned long)body);
if (r.data && r.len == body)
CHECK(memcmp(r.data, g_pattern, body) == 0,
"body content mismatch");
CHECK(r.status == 200, "status %d, expected 200", r.status);
free(r.data);
pthread_join(th, NULL);
close(sp.listen_fd);
}
/* The throughput guard. Reports the achieved bytes-per-update and
* the smallest receive window offered while the body was still
* substantially outstanding. */
static void test_tick_cost(enum framing f, size_t chunk)
{
struct srv_spec sp;
pthread_t th;
struct xfer_result r;
double per_update;
long body_ticks;
memset(&sp, 0, sizeof(sp));
sp.body = BODY_LARGE;
sp.chunk = chunk;
sp.frame = f;
printf(" tick cost: framing=%s chunk=%lu\n",
frame_name(f), (unsigned long)chunk);
if (!srv_start(&sp, &th))
{
printf(" SKIP: server start failed\n");
return;
}
record_reset(sp.body);
if (!run_transfer(sp.port, &r, TEST_TICK_US))
{
printf(" SKIP: client setup failed\n");
record_stop();
pthread_join(th, NULL);
close(sp.listen_fd);
return;
}
record_stop();
/* A transfer that completes inside a single net_http_update() call
* exits the drive loop on that same call, so body_updates is 0.
* That is the best possible result, not a divide-by-zero: charge it
* one tick. */
body_ticks = r.body_updates ? r.body_updates : 1;
per_update = (double)r.len / (double)body_ticks;
printf(" %lu bytes in %ld body ticks (%.0f B/tick), "
"%ld setup ticks, %ld recv() calls, min window %lu B "
"(with %lu B outstanding)\n",
(unsigned long)r.len, body_ticks, per_update,
r.setup_updates, g_recv_calls,
g_min_window == (size_t)-1 ? 0UL : (unsigned long)g_min_window,
(unsigned long)g_window_at_min_outstanding);
CHECK(r.len == sp.body, "short body: got %lu of %lu",
(unsigned long)r.len, (unsigned long)sp.body);
/* At 60Hz a tick is a frame. Below this floor a core download is
* measured in seconds of wall clock spent waiting for ticks. */
CHECK(per_update >= (double)ASSERT_BYTES_PER_UPDATE,
"only %.0f bytes per net_http_update() (floor %d) -- "
"%ld body ticks needed; at 16.7ms/tick unthreaded that is "
"%.1fs for %luMiB",
per_update, ASSERT_BYTES_PER_UPDATE, body_ticks,
body_ticks * 0.0167, (unsigned long)(sp.body >> 20));
/* DNS resolution runs on its own thread but is only ever observed
* from net_http_update(), so its latency is quantised to the tick
* period -- a frame at 60Hz, whatever the resolver actually took. */
printf(" (setup phase: %ld ticks polling DNS/connect/send)\n",
r.setup_updates);
/* Doubling-growth framings decay the window to nothing just before
* each realloc; those reads are round trips that move almost no
* data. */
if (g_min_window != (size_t)-1)
CHECK(g_min_window >= ASSERT_MIN_WINDOW,
"receive window collapsed to %lu B while %lu B of body "
"remained (floor %d)",
(unsigned long)g_min_window,
(unsigned long)g_window_at_min_outstanding,
ASSERT_MIN_WINDOW);
free(r.data);
pthread_join(th, NULL);
close(sp.listen_fd);
}
/* A server that promises N bytes and delivers fewer must surface an
* error rather than a silently short body. */
static void test_truncated(void)
{
struct srv_spec sp;
pthread_t th;
struct xfer_result r;
memset(&sp, 0, sizeof(sp));
sp.body = 512 * 1024;
sp.frame = FRAME_LEN;
sp.truncate = 1;
printf(" truncated content-length body\n");
if (!srv_start(&sp, &th))
{
printf(" SKIP: server start failed\n");
return;
}
if (!run_transfer(sp.port, &r, 0))
{
printf(" SKIP: client setup failed\n");
pthread_join(th, NULL);
close(sp.listen_fd);
return;
}
CHECK(r.err || r.len != sp.body,
"truncated transfer reported success with a full-length body");
free(r.data);
pthread_join(th, NULL);
close(sp.listen_fd);
}
/* Malformed head lines must terminate the transfer cleanly rather
* than being parsed into junk state. Run these under ASan/UBSan --
* that is where the value is. */
static void test_malformed_head(const char *label, const char *head)
{
struct srv_spec sp;
pthread_t th;
struct xfer_result r;
memset(&sp, 0, sizeof(sp));
sp.body = 4096;
sp.frame = FRAME_EOF;
sp.raw_head = head;
printf(" malformed head: %s\n", label);
if (!srv_start(&sp, &th))
{
printf(" SKIP: server start failed\n");
return;
}
if (!run_transfer(sp.port, &r, 0))
{
printf(" SKIP: client setup failed\n");
pthread_join(th, NULL);
close(sp.listen_fd);
return;
}
CHECK(r.err || r.status <= 0 || r.len == 0,
"malformed head accepted: status=%d len=%lu",
r.status, (unsigned long)r.len);
free(r.data);
pthread_join(th, NULL);
close(sp.listen_fd);
}
/* Several transfers in flight at once, so TSan can watch the
* process-global DNS cache and connection pool -- including the
* unsynchronised lazy creation of the locks meant to protect them. */
#define CONCURRENT_N 6
struct conc_arg
{
int port;
size_t body;
int ok;
};
static void *conc_thread(void *a)
{
struct conc_arg *ca = (struct conc_arg*)a;
struct xfer_result r;
if (run_transfer(ca->port, &r, 0))
{
ca->ok = (r.len == ca->body
&& r.data
&& memcmp(r.data, g_pattern, ca->body) == 0);
free(r.data);
}
return NULL;
}
static void test_concurrent(void)
{
struct srv_spec sp[CONCURRENT_N];
pthread_t srv[CONCURRENT_N];
pthread_t cli[CONCURRENT_N];
struct conc_arg ca[CONCURRENT_N];
size_t body = 256 * 1024;
int i, started = 0;
printf(" %d concurrent transfers\n", CONCURRENT_N);
for (i = 0; i < CONCURRENT_N; i++)
{
memset(&sp[i], 0, sizeof(sp[i]));
sp[i].body = body;
sp[i].frame = FRAME_LEN;
if (!srv_start(&sp[i], &srv[i]))
break;
ca[i].port = sp[i].port;
ca[i].body = body;
ca[i].ok = 0;
if (pthread_create(&cli[i], NULL, conc_thread, &ca[i]) != 0)
break;
started++;
}
for (i = 0; i < started; i++)
{
pthread_join(cli[i], NULL);
pthread_join(srv[i], NULL);
close(sp[i].listen_fd);
}
CHECK(started == CONCURRENT_N,
"only started %d of %d concurrent transfers", started,
CONCURRENT_N);
for (i = 0; i < started; i++)
CHECK(ca[i].ok, "concurrent transfer %d did not reconstruct", i);
}
/* ================================================================= */
int main(void)
{
size_t chunk_sizes[] = { 1, 3, 511, 1024, 8192, 65536 };
size_t i;
real_recv = (ssize_t(*)(int, void*, size_t, int))dlsym(RTLD_NEXT, "recv");
real_socket = (int(*)(int, int, int))dlsym(RTLD_NEXT, "socket");
if (!real_recv || !real_socket)
{
printf("SKIP: cannot resolve libc recv/socket via RTLD_NEXT\n");
return 0;
}
network_init();
pattern_init(BODY_LARGE);
printf("net_http transfer stress tests "
"(SO_RCVBUF pinned to %d B)\n\n", TEST_RCVBUF);
printf("[framing]\n");
test_framing(FRAME_LEN, 256 * 1024, 0, 0);
test_framing(FRAME_EOF, 256 * 1024, 0, 0);
for (i = 0; i < sizeof(chunk_sizes) / sizeof(chunk_sizes[0]); i++)
test_framing(FRAME_CHUNKED, 256 * 1024, chunk_sizes[i], 0);
printf("\n[framing, torn across reads]\n");
test_framing(FRAME_LEN, 32 * 1024, 0, 7);
test_framing(FRAME_EOF, 32 * 1024, 0, 3);
test_framing(FRAME_CHUNKED, 32 * 1024, 1, 1);
test_framing(FRAME_CHUNKED, 32 * 1024, 511, 5);
printf("\n[tick cost]\n");
test_tick_cost(FRAME_LEN, 0);
test_tick_cost(FRAME_CHUNKED, 8192);
test_tick_cost(FRAME_EOF, 0);
printf("\n[failure teardown]\n");
test_truncated();
test_malformed_head("not HTTP at all",
"GARBAGE\r\n\r\n");
test_malformed_head("non-digit status",
"HTTP/1.1 2X0 OK\r\n\r\n");
test_malformed_head("absurd content-length",
"HTTP/1.1 200 OK\r\nContent-Length: 99999999999999999999\r\n\r\n");
test_malformed_head("content-length and chunked together",
"HTTP/1.1 200 OK\r\nContent-Length: 4096\r\n"
"Transfer-Encoding: chunked\r\n\r\n");
printf("\n[concurrency]\n");
test_concurrent();
free((void*)g_pattern);
printf("\n%s (%d check%s, %d failure%s)\n",
failures ? "FAILED" : "PASSED",
checks, checks == 1 ? "" : "s",
failures, failures == 1 ? "" : "s");
return failures ? 1 : 0;
}
samples/tasks/http/http_url_dedup_test.c
+388
View File
@@ -0,0 +1,388 @@
/* Copyright (C) 2010-2026 The RetroArch team
*
* ---------------------------------------------------------------------------------------
* The following license statement only applies to this file (http_url_dedup_test.c).
* ---------------------------------------------------------------------------------------
*
* Permission is hereby granted, free of charge,
* to any person obtaining a copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation the rights to
* use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
* and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
* INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
/* Regression test for the duplicate-download suppression key in
* tasks/task_http.c and tasks/task_http_emscripten.c.
*
* Both files store the request URL in a fixed-size member:
*
* char connection_url[NAME_MAX_LENGTH];
*
* populated with strlcpy(), and task_http_finder() compares that
* stored copy against the raw candidate URL to decide whether a GET
* is already in flight:
*
* return string_is_equal(http->connection_url, (const char*)user_data);
*
* NAME_MAX_LENGTH is 256 on most targets and 128 on the small-path
* platforms (see libretro-common/include/retro_miscellaneous.h).
*
* The asymmetry is the bug: the *stored* side is truncated by
* strlcpy, the *candidate* side is not. Once a URL reaches
* NAME_MAX_LENGTH bytes the two can never compare equal, so
* task_http_finder() returns false for every candidate and the
* concurrency guard in task_push_http_transfer_generic() -- the one
* whose comment reads "Concurrent download of the same file is not
* allowed" -- silently stops guarding anything.
*
* Note what this is *not*: truncation cannot cause a false positive.
* Two distinct long URLs sharing a prefix do not get conflated,
* because the candidate is never truncated to match. The failure is
* one-directional, which is why it has gone unnoticed -- nothing
* breaks loudly, downloads just quietly stop being deduplicated.
*
* The consequence is not merely wasted bandwidth. Two tasks for the
* same URL both reach their completion callback, and callbacks such
* as cb_http_task_download_pl_thumbnail() in
* tasks/task_pl_thumbnail_download.c end in
*
* filestream_write_file(transf->path, data->data, data->len)
*
* against an identical path. Two writers, one file, no ordering:
* a torn or interleaved image on disk. Playlist thumbnail URLs are
* the natural trigger, since they concatenate a long CDN base path
* with a percent-encoded game title and routinely exceed 256 bytes.
*
* The finder predicate is static to task_http.c and not exposed in a
* header, so this test keeps a behavioural copy as the oracle -- the
* same convention http_method_match_test.c and
* archive_name_safety_test.c use. If task_http.c changes how the key
* is derived, update the copy here to match.
*
* Build standalone:
* cc -Wall -std=gnu99 -g -O0 -o http_url_dedup_test http_url_dedup_test.c
* ./http_url_dedup_test
*
* Also worth running under ASan/LSan, which covers the heap-owned
* replacement key's allocation and teardown:
* make clean all SANITIZER=address && ./http_url_dedup_test
*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
/* Mirrors retro_miscellaneous.h. Override on the command line to
* exercise the small-path platforms: -DNAME_MAX_LENGTH=128 */
#ifndef NAME_MAX_LENGTH
#define NAME_MAX_LENGTH 256
#endif
static int failures;
#define CHECK(cond, ...) \
do { \
if (!(cond)) \
{ \
printf(" FAIL: "); printf(__VA_ARGS__); printf("\n"); \
failures++; \
} \
} while (0)
/* Behavioural copy of compat/strl.c::strlcpy -- truncating, always
* NUL-terminating, returning the length of the source. */
static size_t t_strlcpy(char *dest, const char *source, size_t size)
{
size_t src_size = 0;
const char *psrc = source;
if (size)
{
size_t n = size;
while (--n && (*dest++ = *source++))
;
*dest = '\0';
}
while (*psrc++)
src_size++;
return src_size;
}
/* --------------------------------------------------------------- */
/* Oracle A: the current key -- fixed buffer, silently truncating. */
/* --------------------------------------------------------------- */
struct key_fixed
{
char connection_url[NAME_MAX_LENGTH];
};
static void key_fixed_set(struct key_fixed *k, const char *url)
{
k->connection_url[0] = '\0';
t_strlcpy(k->connection_url, url, sizeof(k->connection_url));
}
static int key_fixed_matches(const struct key_fixed *k, const char *url)
{
/* task_http_finder(): stored (truncated) vs candidate (raw). */
return strcmp(k->connection_url, url) == 0;
}
/* --------------------------------------------------------------- */
/* Oracle B: the replacement -- heap-owned, full-length. */
/* */
/* A NULL key means "no key", which matches nothing. An allocation */
/* failure therefore degrades to "admit the download" rather than */
/* to a silent drop, which is the safe direction: a redundant */
/* download costs bandwidth, a dropped one strands the caller's */
/* completion callback forever. */
/* --------------------------------------------------------------- */
struct key_owned
{
char *connection_url;
};
static int key_owned_set(struct key_owned *k, const char *url)
{
size_t n = strlen(url) + 1;
k->connection_url = (char*)malloc(n);
if (!k->connection_url)
return 0;
memcpy(k->connection_url, url, n);
return 1;
}
static void key_owned_free(struct key_owned *k)
{
free(k->connection_url);
k->connection_url = NULL;
}
static int key_owned_matches(const struct key_owned *k, const char *url)
{
if (!k->connection_url || !url)
return 0;
return strcmp(k->connection_url, url) == 0;
}
/* --------------------------------------------------------------- */
/* Fixtures */
/* --------------------------------------------------------------- */
/* Build a URL shaped like the ones RetroArch actually requests for
* playlist thumbnails: a long stable CDN prefix, `pad` bytes of
* filler, then a distinguishing tail. */
static char *make_thumb_url(size_t pad, const char *tail)
{
static const char base[] =
"http://thumbnails.libretro.com/"
"Sony%20-%20PlayStation/Named_Snaps/";
size_t blen = sizeof(base) - 1;
size_t tlen = strlen(tail);
char *url = (char*)malloc(blen + pad + tlen + 1);
if (!url)
return NULL;
memcpy(url, base, blen);
memset(url + blen, 'A', pad);
memcpy(url + blen + pad, tail, tlen + 1);
return url;
}
static char *dup_str(const char *s)
{
size_t n = strlen(s) + 1;
char *d = (char*)malloc(n);
if (d)
memcpy(d, s, n);
return d;
}
/* --------------------------------------------------------------- */
/* The core defect: a long URL fails to match itself, so the
* concurrency guard never fires and duplicate downloads are
* admitted. */
static void test_long_url_fails_to_dedup_itself(void)
{
char *url = make_thumb_url(NAME_MAX_LENGTH, "Castlevania%20-%20SOTN.png");
char *same;
struct key_fixed ka;
struct key_owned kb;
printf("test_long_url_fails_to_dedup_itself\n");
if (!url)
{
printf(" SKIP: out of memory\n");
return;
}
if (!(same = dup_str(url)))
{
printf(" SKIP: out of memory\n");
free(url);
return;
}
CHECK(strlen(url) >= NAME_MAX_LENGTH,
"fixture must exceed the key buffer (len=%u, buf=%u)",
(unsigned)strlen(url), (unsigned)NAME_MAX_LENGTH);
/* Pre-fix: the guard silently stops guarding. */
key_fixed_set(&ka, url);
CHECK(key_fixed_matches(&ka, same) == 1,
"fixed-size key fails to match its own URL -- the "
"\"concurrent download of the same file\" guard is disabled, "
"so two tasks race to filestream_write_file() the same path");
/* Post-fix: the guard works at any length. */
if (key_owned_set(&kb, url))
{
CHECK(key_owned_matches(&kb, same) == 1,
"owned key must match its own URL regardless of length");
key_owned_free(&kb);
}
free(url);
free(same);
}
/* The failure is one-directional: distinct long URLs must still be
* distinct. Both keys are expected to pass this -- it is here so a
* future "fix" that normalises or hashes the key to a fixed width
* cannot quietly introduce the false positive this bug does not
* have. */
static void test_distinct_long_urls_stay_distinct(void)
{
size_t pad = NAME_MAX_LENGTH;
char *url_a = make_thumb_url(pad, "Castlevania%20-%20SOTN.png");
char *url_b = make_thumb_url(pad, "Silent%20Hill.png");
struct key_fixed ka;
struct key_owned kb;
printf("test_distinct_long_urls_stay_distinct\n");
if (!url_a || !url_b)
{
printf(" SKIP: out of memory\n");
free(url_a);
free(url_b);
return;
}
CHECK(strcmp(url_a, url_b) != 0, "fixture URLs must differ");
key_fixed_set(&ka, url_a);
CHECK(key_fixed_matches(&ka, url_b) == 0,
"distinct URLs must not be conflated (fixed key)");
if (key_owned_set(&kb, url_a))
{
CHECK(key_owned_matches(&kb, url_b) == 0,
"distinct URLs must not be conflated (owned key)");
key_owned_free(&kb);
}
free(url_a);
free(url_b);
}
/* URLs comfortably inside the buffer must behave identically under
* both keys -- the replacement must not perturb the common case. */
static void test_short_urls_unaffected(void)
{
static const char a[] = "http://buildbot.libretro.com/nightly/.index";
static const char b[] = "http://buildbot.libretro.com/nightly/.index2";
struct key_fixed ka;
struct key_owned kb;
printf("test_short_urls_unaffected\n");
key_fixed_set(&ka, a);
CHECK(key_fixed_matches(&ka, a) == 1, "short URL must self-match (fixed)");
CHECK(key_fixed_matches(&ka, b) == 0, "short distinct URLs must differ (fixed)");
if (key_owned_set(&kb, a))
{
CHECK(key_owned_matches(&kb, a) == 1, "short URL must self-match (owned)");
CHECK(key_owned_matches(&kb, b) == 0, "short distinct URLs must differ (owned)");
key_owned_free(&kb);
}
}
/* Walk the URL length across the buffer boundary and pin down
* exactly where the fixed key stops working. Self-match must hold
* for every length under the owned key; under the fixed key it is
* expected to hold for len < NAME_MAX_LENGTH and fail from
* NAME_MAX_LENGTH onward. Asserting the precise boundary means a
* change to NAME_MAX_LENGTH, or a partial fix, surfaces here rather
* than silently shifting. */
static void test_self_match_boundary(void)
{
size_t len;
size_t first_fail = 0;
printf("test_self_match_boundary\n");
for (len = NAME_MAX_LENGTH - 4; len <= NAME_MAX_LENGTH + 4; len++)
{
struct key_fixed ka;
struct key_owned kb;
char *u = (char*)malloc(len + 1);
char *v;
if (!u)
continue;
memset(u, 'u', len);
u[len] = '\0';
if (!(v = dup_str(u)))
{
free(u);
continue;
}
key_fixed_set(&ka, u);
if (!key_fixed_matches(&ka, v) && !first_fail)
first_fail = len;
if (key_owned_set(&kb, u))
{
CHECK(key_owned_matches(&kb, v) == 1,
"owned key must self-match at len=%u", (unsigned)len);
key_owned_free(&kb);
}
free(u);
free(v);
}
CHECK(first_fail == NAME_MAX_LENGTH,
"fixed key expected to first fail self-match at len=%u, got %u",
(unsigned)NAME_MAX_LENGTH, (unsigned)first_fail);
}
int main(void)
{
printf("task_http URL dedup key regression tests "
"(NAME_MAX_LENGTH=%u)\n\n", (unsigned)NAME_MAX_LENGTH);
test_short_urls_unaffected();
test_distinct_long_urls_stay_distinct();
test_long_url_fails_to_dedup_itself();
test_self_match_boundary();
printf("\n%s (%d failure%s)\n",
failures ? "FAILED" : "PASSED",
failures, failures == 1 ? "" : "s");
return failures ? 1 : 0;
}