Harden the gateway: optional token for /stats and /metrics, per-address cap, flood close, sandboxed unit (#33)

* Gate /stats and /metrics behind an optional token and harden the gateway

- TR_METRICS_TOKEN: when set, /stats and /metrics answer 404 unless the
  request sends it as a bearer token. /healthz and /readyz stay public.
  The token never shows in Debug output or errors.
- TR_MAX_CLIENTS_PER_IP caps connections per address (IPv6 per /64), off
  by default. TR_CLIENT_IP_HEADER takes the address from the edge's
  header. The first connection logs whether x-forwarded-for was there,
  never its value, so the cap can be checked behind Freestyle's edge.
- A client that keeps sending after a budget's worth of RateLimited
  replies is closed with 1008 instead of being answered forever.
- cli stats sends TR_METRICS_TOKEN when set.
- vm-setup.sh: optional /etc/tr-gateway/env, stop timeout, memory and
  task caps, and systemd sandboxing (security score 8.3 to 3.1).

* Let the site's server URL come from a repository variable

deploy-site.yml passes the TR_SERVER_URL repository variable to
build-web.sh when it's set, so moving the site to the gateway's custom
domain is a variable and a redeploy, not a code change.

* Accept the bearer scheme in any case

HTTP auth scheme names are case-insensitive, so "bearer <token>" now
works too. The token itself is still compared exactly.

.github/workflows/deploy-site.yml

@@ -50,7 +50,14 @@ jobs:
           echo "$SHA256  $RUNNER_TEMP/binaryen.tar.gz" | sha256sum -c -
           tar xzf "$RUNNER_TEMP/binaryen.tar.gz" -C "$RUNNER_TEMP"
           echo "$RUNNER_TEMP/binaryen-$BINARYEN/bin" >> "$GITHUB_PATH"
-      - run: scripts/build-web.sh
+      # The repository variable TR_SERVER_URL, when set, picks the server the site connects to,
+      # for example after the gateway moves to a custom domain. Unset, build-web.sh's default holds.
+      - name: Build the browser app
+        env:
+          SERVER_URL: ${{ vars.TR_SERVER_URL }}
+        run: |
+          if [ -n "$SERVER_URL" ]; then export TR_SERVER_URL="$SERVER_URL"; fi
+          scripts/build-web.sh
       - uses: actions/upload-artifact@v7
         with:
           name: web-app

deploy/freestyle/README.md

@@ -6,6 +6,8 @@ The Terminal data server runs on a Freestyle VM at **https://terminal-gateway.st
 TR_SERVER_URL=wss://terminal-gateway.style.dev cargo run --release
 ```
 
+The desktop reads `TR_SERVER_URL` at startup, and the site's deploy workflow reads the repository variable of the same name (`server/README.md`), so moving to another domain changes no code.
+
 ## What runs where
 
 - One VM, slug `terminal-gateway`, from the `freestyle/ubuntu-sm` snapshot: 2 vCPU, 4 GiB of memory, 16 GB of disk.
@@ -36,7 +38,7 @@ A deploy restarts the gateway, which drops every client connection. Clients reco
 curl -fsS https://terminal-gateway.style.dev/healthz   # the process is up
 curl -fsS https://terminal-gateway.style.dev/readyz    # upstream connected, every venue's mids in
 cli() { cargo run -q --release -p tr-gateway --example cli -- --url wss://terminal-gateway.style.dev "$@"; }
-cli stats
+TR_METRICS_TOKEN=... cli stats   # the token only when the gateway has one
 cli watch --quotes BTC --book BTC --trades BTC --bars BTC:1m --secs 30
 ```
 
@@ -46,6 +48,17 @@ On the VM, `systemctl status tr-gateway` and `journalctl -u tr-gateway` show the
 
 2 vCPU and 4 GiB running around the clock come to about $80 a month of Freestyle usage above the free allowance.
 
-## `/stats` and `/metrics` are public
+## Settings and the operator token
 
-Anyone can read `/stats` (JSON) and `/metrics` (Prometheus text) through the public URL right now. They show counts, budgets, latency and the upstream subscriptions, which reveal the markets clients are watching. They carry no client addresses or user data. Micky hasn't decided whether to restrict them, so they stay as they are for now. Restricting them later means serving them on a second port the edge doesn't publish, or checking a token.
+`/stats` (JSON) and `/metrics` (Prometheus text) show counts, budgets, latency and the upstream subscriptions, which reveal the markets clients are watching. They carry no client addresses or user data. With `TR_METRICS_TOKEN` set they answer 404 to anyone without the token, and `/healthz` and `/readyz` stay public.
+
+Settings for the service go in `/etc/tr-gateway/env` on the VM, one `NAME=value` a line, readable only by root. `vm-setup.sh` creates the directory but never writes or prints the file, so a deploy keeps it. To create a token on the VM without it crossing the network or a terminal:
+
+```sh
+npx -y [email protected] vm exec terminal-gateway --linux-user root -- bash -c \
+  'umask 077; f=/etc/tr-gateway/env; grep -q ^TR_METRICS_TOKEN= "$f" 2>/dev/null || echo "TR_METRICS_TOKEN=$(openssl rand -hex 32)" >> "$f"; systemctl restart tr-gateway'
+```
+
+Reading it back for `cli stats` or a scraper prints it, so do that off stream. Without the token, `cli stats` says the server answered 404.
+
+The unit also caps the service at 3 GiB of memory (reclaiming hard past 2 GiB) and 512 tasks, and runs it with most of systemd's sandboxing on: no capabilities, a read-only system, no devices, no kernel tunables or modules, and only IP, Unix and netlink sockets. `systemd-analyze security` rates it 3.1 (OK), down from 8.3 (exposed) before.

deploy/freestyle/vm-setup.sh

@@ -35,6 +35,10 @@ mv /usr/local/bin/tr-gateway.new /usr/local/bin/tr-gateway
 install -m 0755 target/release/examples/cli /usr/local/bin/tr-cli
 git rev-parse HEAD > /opt/terminal/DEPLOYED_COMMIT
 
+# Settings and secrets (TR_METRICS_TOKEN and the like) go in /etc/tr-gateway/env, which only
+# root reads. This script never writes or prints that file; see README.md.
+install -d -m 0700 -o root -g root /etc/tr-gateway
+
 cat > /etc/systemd/system/tr-gateway.service <<'UNIT'
 [Unit]
 Description=Terminal data server (tr-gateway)
@@ -47,14 +51,38 @@ StartLimitIntervalSec=0
 User=trgw
 Environment=TR_GATEWAY_BIND=0.0.0.0:8080
 Environment=RUST_LOG=info
+# Optional; read by systemd as root before it drops to trgw. Overrides the lines above.
+EnvironmentFile=-/etc/tr-gateway/env
 ExecStart=/usr/local/bin/tr-gateway
 Restart=always
 RestartSec=2
+# SIGTERM closes every session and upstream connection first; that takes a second or two.
+TimeoutStopSec=15
 LimitNOFILE=65536
+# The gateway uses tens of MiB. Past MemoryHigh the kernel reclaims hard; past MemoryMax it's
+# killed and restarted, before the VM itself runs out.
+MemoryHigh=2G
+MemoryMax=3G
+TasksMax=512
+UMask=0077
 NoNewPrivileges=true
 ProtectSystem=strict
 ProtectHome=true
 PrivateTmp=true
+PrivateDevices=true
+ProtectKernelTunables=true
+ProtectKernelModules=true
+ProtectKernelLogs=true
+ProtectControlGroups=true
+ProtectClock=true
+ProtectHostname=true
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
+RestrictNamespaces=true
+RestrictRealtime=true
+RestrictSUIDSGID=true
+LockPersonality=true
+CapabilityBoundingSet=
+SystemCallArchitectures=native
 
 [Install]
 WantedBy=multi-user.target

scripts/build-web.sh

@@ -15,7 +15,9 @@
 # The app gets crypto markets from the Terminal data server at TR_SERVER_URL, which is built
 # in. It defaults to the production server; set it to another ws:// or wss:// URL, or to
 # nothing (TR_SERVER_URL=) for no server. A page can also ask for a server on the same machine
-# with ?server=ws://localhost:8787/ws, or for none with ?server=off.
+# with ?server=ws://localhost:8787/ws, or for none with ?server=off. The deploy workflow passes
+# the repository variable TR_SERVER_URL when it's set, so moving the site to another server
+# needs no code change.
 #
 # Needs the wasm32-unknown-unknown target (rustup target add wasm32-unknown-unknown),
 # wasm-bindgen-cli at the version pinned in Cargo.lock, and wasm-opt from binaryen.

server/README.md

@@ -8,7 +8,7 @@ The Terminal data server. For now it carries Hyperliquid's public API (see "Inte
 cargo run --release -p tr-gateway
 ```
 
-It listens on `localhost:8787`. Clients connect to `ws://localhost:8787/ws`. `/healthz` says the process is up, `/readyz` says the upstream is connected and every venue's mids have arrived, `/metrics` is Prometheus text and `/stats` is JSON.
+It listens on `localhost:8787`. Clients connect to `ws://localhost:8787/ws`. `/healthz` says the process is up, `/readyz` says the upstream is connected and every venue's mids have arrived, `/metrics` is Prometheus text and `/stats` is JSON. With `TR_METRICS_TOKEN` set, `/metrics` and `/stats` answer 404 unless the request sends `Authorization: Bearer <token>`; the health checks stay open.
 
 The desktop app uses it for crypto markets and Hyperliquid HIP-3 perps (`xyz:AAPL`) when `TR_SERVER_URL` is set. Perps have no other source, so without a server they don't load:
 
@@ -16,7 +16,7 @@ The desktop app uses it for crypto markets and Hyperliquid HIP-3 perps (`xyz:AAP
 TR_SERVER_URL=ws://localhost:8787 cargo run --release
 ```
 
-The browser build uses it too. `scripts/build-web.sh` builds in `wss://terminal-gateway.style.dev/ws`, or `TR_SERVER_URL` when it is set (empty for none). To try a local gateway with a built page, open the page with `?server=ws://localhost:8787/ws`.
+The browser build uses it too. `scripts/build-web.sh` builds in `wss://terminal-gateway.style.dev/ws`, or `TR_SERVER_URL` when it is set (empty for none). The site's deploy workflow passes the repository variable `TR_SERVER_URL` when it is set, so pointing the site at another server is `gh variable set TR_SERVER_URL --body wss://<host>/ws` and a redeploy. To try a local gateway with a built page, open the page with `?server=ws://localhost:8787/ws`.
 
 The gateway serves plain HTTP and websockets. For `wss://` on 443, put a TLS-terminating edge in front of it and bind to an address the edge can reach, for example `TR_GATEWAY_BIND=0.0.0.0:8080`. Clients accept `wss://` URLs. The gateway pings each client every 20 s so idle connections survive proxies. The hosted one runs on Freestyle; see `deploy/freestyle/README.md`.
 
@@ -50,6 +50,15 @@ Environment variables, also listed by `tr-gateway --help`:
 | `TR_REST_WEIGHT_PER_MIN` | 800 | `info` weight a minute, at most 1100 |
 | `TR_LINGER_SECS` | 20 | How long an unwatched market keeps its upstream streams |
 | `TR_MAX_CLIENTS` | 1000 | Client connections |
+| `TR_MAX_CLIENTS_PER_IP` | 0 (no cap) | Client connections from one address, or one IPv6 /64. Over it, the upgrade gets 429 |
+| `TR_CLIENT_IP_HEADER` | none | Header the edge puts the client address in, for example `x-forwarded-for`; its last entry counts. Unset, the peer address counts |
+| `TR_METRICS_TOKEN` | none | Bearer token `/metrics` and `/stats` require. Unset, both are public |
+
+## Limits on clients
+
+Each connection may hold 64 subscriptions, 512 quote keys and 8 requests in flight, and send 200 messages in 10 s. Messages are at most 64 KiB. A message over the rate gets `RateLimited` and is dropped; a client that sends another 200 of those in 10 s is closed with code 1008. A connection that hasn't sent Hello in 10 s, or sent nothing at all (not even a pong) in 90 s, is closed. `TR_MAX_CLIENTS` caps connections in total and `TR_MAX_CLIENTS_PER_IP` per address.
+
+A per-address cap needs the real client address. Behind an edge that terminates TLS, every connection comes from the edge, so set `TR_CLIENT_IP_HEADER` to the header the edge adds, and only if the edge overwrites or appends to it (otherwise clients could pick their own address). The gateway logs once, for its first client connection, whether that header (or `x-forwarded-for` when none is set) was there, without its value, so you can check before turning the cap on.
 
 ## Hyperliquid's limits
 

server/examples/cli.rs

@@ -8,7 +8,7 @@
 //!           stream topics, print about one line a second per topic, then latency stats
 //!   load    --clients N [TOPICS] [--secs N]
 //!           N connections with the same subscriptions; prints totals and latency
-//!   stats   print the server's /stats
+//!   stats   print the server's /stats, sending TR_METRICS_TOKEN as a bearer token when set
 //!
 //! TOPICS: --quotes K,K,..  --book K  --trades K  --bars K:TF (TF: 1m 5m 15m 30m 1h 4h 1d 1w 1M)
 //! ```
@@ -42,7 +42,7 @@ usage: cli [--url ws://localhost:8787] [--json] COMMAND
   watch [TOPICS] [--secs N]           stream topics, then print latency stats
   load --clients N [TOPICS] [--secs N]
                                       N connections with the same subscriptions
-  stats                               print the server's /stats
+  stats                               print the server's /stats (token: TR_METRICS_TOKEN)
 TOPICS: --quotes K,K,..  --book K  --trades K  --bars K:TF (1m 5m 15m 30m 1h 4h 1d 1w 1M)";
 
 fn usage() -> ! {
@@ -245,7 +245,15 @@ fn http_base(ws: &str) -> String {
 }
 
 async fn print_stats(url: &str) -> Result<()> {
-    let body = reqwest::get(format!("{}/stats", http_base(url))).await?.text().await?;
+    let mut req = reqwest::Client::new().get(format!("{}/stats", http_base(url)));
+    if let Ok(token) = std::env::var("TR_METRICS_TOKEN") {
+        req = req.bearer_auth(token.trim());
+    }
+    let resp = req.send().await?;
+    if resp.status() == reqwest::StatusCode::NOT_FOUND {
+        bail!("/stats: 404; the server may want TR_METRICS_TOKEN");
+    }
+    let body = resp.error_for_status()?.text().await?;
     let v: serde_json::Value = serde_json::from_str(&body)?;
     let u = &v["upstream"];
     println!(

server/src/access.rs

@@ -0,0 +1,199 @@
+//! Who may use what: the optional token in front of `/stats` and `/metrics`, and the cap on
+//! connections from one client address.
+//!
+//! Client addresses only ever live in memory as map keys. They are never logged or reported.
+
+use std::collections::HashMap;
+use std::fmt;
+use std::net::{IpAddr, Ipv6Addr, SocketAddr};
+use std::sync::{Arc, Mutex};
+
+use axum::http::{HeaderMap, HeaderName, header};
+
+/// A secret from the environment. Its `Debug` never shows the value.
+#[derive(Clone, PartialEq, Eq)]
+pub struct Secret(Arc<str>);
+
+impl Secret {
+    pub fn new(s: &str) -> Self {
+        Self(s.into())
+    }
+
+    /// Whether `given` is the secret, in time that doesn't depend on where they differ.
+    pub fn matches(&self, given: &str) -> bool {
+        let (a, b) = (self.0.as_bytes(), given.as_bytes());
+        a.len() == b.len() && a.iter().zip(b).fold(0u8, |d, (x, y)| d | (x ^ y)) == 0
+    }
+}
+
+impl fmt::Debug for Secret {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        f.write_str("Secret(..)")
+    }
+}
+
+/// Whether a request may read `/stats` and `/metrics`: always without a token, and with one
+/// only when it sends `Authorization: Bearer <token>`. The scheme's case doesn't matter (RFC
+/// 9110 §11.1); the token's does.
+pub fn operator(token: Option<&Secret>, headers: &HeaderMap) -> bool {
+    let Some(token) = token else { return true };
+    headers
+        .get(header::AUTHORIZATION)
+        .and_then(|v| v.to_str().ok())
+        .and_then(|v| v.trim().split_once(' '))
+        .filter(|(scheme, _)| scheme.eq_ignore_ascii_case("bearer"))
+        .is_some_and(|(_, given)| token.matches(given.trim()))
+}
+
+/// The client's address: the last entry of `header` when it is set and parses (the entry the
+/// edge in front of the gateway added), otherwise the peer the connection came from.
+pub fn client_ip(peer: SocketAddr, header: Option<&HeaderName>, headers: &HeaderMap) -> IpAddr {
+    header
+        .and_then(|h| headers.get_all(h).iter().next_back())
+        .and_then(|v| v.to_str().ok())
+        .and_then(|v| v.rsplit(',').next())
+        .and_then(|v| v.trim().parse::<IpAddr>().ok())
+        .unwrap_or(peer.ip())
+        .to_canonical()
+}
+
+/// One IPv6 host usually has a whole /64, so connections count against their /64.
+fn bucket(ip: IpAddr) -> IpAddr {
+    match ip {
+        IpAddr::V6(v6) => {
+            let s = v6.segments();
+            IpAddr::V6(Ipv6Addr::new(s[0], s[1], s[2], s[3], 0, 0, 0, 0))
+        }
+        v4 => v4,
+    }
+}
+
+/// Connections open per client address, capped at `max` (0 turns the cap off).
+#[derive(Clone, Default)]
+pub struct PerIp {
+    max: usize,
+    open: Arc<Mutex<HashMap<IpAddr, usize>>>,
+}
+
+impl PerIp {
+    pub fn new(max: usize) -> Self {
+        Self { max, open: Arc::default() }
+    }
+
+    /// Count a connection from `ip` until the slot drops, or `None` when `ip` is at the cap.
+    pub fn admit(&self, ip: IpAddr) -> Option<IpSlot> {
+        let key = bucket(ip);
+        if self.max > 0 {
+            let mut open = self.open.lock().expect("per-ip lock");
+            let n = open.entry(key).or_insert(0);
+            if *n >= self.max {
+                return None;
+            }
+            *n += 1;
+        }
+        Some(IpSlot { key, owner: self.clone() })
+    }
+
+    /// Addresses with a connection open now.
+    pub fn addresses(&self) -> usize {
+        self.open.lock().expect("per-ip lock").len()
+    }
+}
+
+/// One admitted connection.
+pub struct IpSlot {
+    key: IpAddr,
+    owner: PerIp,
+}
+
+impl Drop for IpSlot {
+    fn drop(&mut self) {
+        if self.owner.max == 0 {
+            return;
+        }
+        let mut open = self.owner.open.lock().expect("per-ip lock");
+        if let Some(n) = open.get_mut(&self.key) {
+            *n -= 1;
+            if *n == 0 {
+                open.remove(&self.key);
+            }
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn headers(pairs: &[(&str, &str)]) -> HeaderMap {
+        let mut h = HeaderMap::new();
+        for (k, v) in pairs {
+            h.append(HeaderName::from_bytes(k.as_bytes()).unwrap(), v.parse().unwrap());
+        }
+        h
+    }
+
+    #[test]
+    fn the_token_must_match_exactly() {
+        let t = Secret::new("s3cret");
+        assert!(operator(None, &headers(&[])), "no token configured: open");
+        assert!(!operator(Some(&t), &headers(&[])));
+        assert!(!operator(Some(&t), &headers(&[("authorization", "Bearer s3cre")])));
+        assert!(!operator(Some(&t), &headers(&[("authorization", "Bearer s3cret2")])));
+        assert!(!operator(Some(&t), &headers(&[("authorization", "Basic s3cret")])));
+        assert!(!operator(Some(&t), &headers(&[("authorization", "s3cret")])));
+        assert!(operator(Some(&t), &headers(&[("authorization", "Bearer s3cret")])));
+        assert!(operator(Some(&t), &headers(&[("authorization", "bearer s3cret")])), "the scheme ignores case");
+        assert!(operator(Some(&t), &headers(&[("authorization", "BEARER  s3cret")])));
+        assert!(!operator(Some(&t), &headers(&[("authorization", "Bearer S3CRET")])), "the token doesn't");
+        assert!(!operator(Some(&t), &headers(&[("authorization", "Bearers3cret")])));
+        assert_eq!(format!("{t:?}"), "Secret(..)", "Debug never shows the value");
+    }
+
+    #[test]
+    fn the_client_address_comes_from_the_trusted_header_when_set() {
+        let peer: SocketAddr = "10.0.0.1:5000".parse().unwrap();
+        let xff = HeaderName::from_static("x-forwarded-for");
+        let h = headers(&[("x-forwarded-for", "203.0.113.9, 198.51.100.7")]);
+        assert_eq!(client_ip(peer, None, &h), peer.ip(), "no header configured: the peer");
+        assert_eq!(client_ip(peer, Some(&xff), &h).to_string(), "198.51.100.7", "the entry the edge added");
+        let two = headers(&[("x-forwarded-for", "203.0.113.9"), ("x-forwarded-for", "198.51.100.8")]);
+        assert_eq!(client_ip(peer, Some(&xff), &two).to_string(), "198.51.100.8");
+        assert_eq!(client_ip(peer, Some(&xff), &headers(&[("x-forwarded-for", "junk")])), peer.ip());
+        assert_eq!(client_ip(peer, Some(&xff), &headers(&[])), peer.ip());
+        let mapped = headers(&[("x-forwarded-for", "::ffff:198.51.100.7")]);
+        assert_eq!(client_ip(peer, Some(&xff), &mapped).to_string(), "198.51.100.7");
+    }
+
+    #[test]
+    fn connections_per_address_are_capped_and_released() {
+        let p = PerIp::new(2);
+        let a: IpAddr = "198.51.100.7".parse().unwrap();
+        let b: IpAddr = "198.51.100.8".parse().unwrap();
+        let s1 = p.admit(a).unwrap();
+        let _s2 = p.admit(a).unwrap();
+        assert!(p.admit(a).is_none(), "a third from one address is refused");
+        let _other = p.admit(b).expect("another address has its own allowance");
+        drop(s1);
+        let _s3 = p.admit(a).expect("a closed connection frees its slot");
+        assert_eq!(p.addresses(), 2);
+        drop((_s2, _s3, _other));
+        assert_eq!(p.addresses(), 0, "nothing is kept for addresses that left");
+    }
+
+    #[test]
+    fn ipv6_counts_per_64() {
+        let p = PerIp::new(1);
+        let _s = p.admit("2001:db8:1:2::1".parse().unwrap()).unwrap();
+        assert!(p.admit("2001:db8:1:2:ffff::9".parse().unwrap()).is_none(), "same /64");
+        assert!(p.admit("2001:db8:1:3::1".parse().unwrap()).is_some(), "another /64");
+    }
+
+    #[test]
+    fn zero_turns_the_cap_off() {
+        let p = PerIp::new(0);
+        let a: IpAddr = "198.51.100.7".parse().unwrap();
+        let slots: Vec<_> = (0..100).map(|_| p.admit(a).unwrap()).collect();
+        assert_eq!((slots.len(), p.addresses()), (100, 0));
+    }
+}

server/src/config.rs

@@ -4,7 +4,9 @@ use std::net::{Ipv4Addr, SocketAddr, ToSocketAddrs};
 use std::time::Duration;
 
 use anyhow::{Context, Result};
+use axum::http::HeaderName;
 
+use crate::access::Secret;
 use crate::hl::HlConfig;
 use crate::session::SessionConfig;
 use crate::store::HubConfig;
@@ -17,6 +19,12 @@ pub struct Config {
     pub hub: HubConfig,
     pub session: SessionConfig,
     pub max_clients: usize,
+    /// Client connections from one address (an IPv6 /64), or 0 for no cap.
+    pub max_clients_per_ip: usize,
+    /// The header the edge puts the client's address in. Without it, the peer's address counts.
+    pub client_ip_header: Option<HeaderName>,
+    /// When set, `/stats` and `/metrics` answer only requests that carry it as a bearer token.
+    pub metrics_token: Option<Secret>,
 }
 
 impl Default for Config {
@@ -27,6 +35,9 @@ impl Default for Config {
             hub: HubConfig::default(),
             session: SessionConfig::default(),
             max_clients: 1000,
+            max_clients_per_ip: 0,
+            client_ip_header: None,
+            metrics_token: None,
         }
     }
 }
@@ -79,6 +90,15 @@ impl Config {
         if let Some(n) = num("TR_MAX_CLIENTS")? {
             c.max_clients = n;
         }
+        if let Some(n) = num("TR_MAX_CLIENTS_PER_IP")? {
+            c.max_clients_per_ip = n;
+        }
+        if let Some(h) = var("TR_CLIENT_IP_HEADER") {
+            let name = HeaderName::try_from(h.as_str()).with_context(|| format!("TR_CLIENT_IP_HEADER={h}"))?;
+            c.client_ip_header = Some(name);
+        }
+        // Never echoed back, not even in an error.
+        c.metrics_token = var("TR_METRICS_TOKEN").map(|t| Secret::new(&t));
         Ok(c)
     }
 }
@@ -87,6 +107,7 @@ pub const HELP: &str = "\
 tr-gateway: the Terminal data server (interim: Hyperliquid's public feed)
 
 Serves websocket clients at /ws, plus /healthz, /readyz, /metrics and /stats.
+With TR_METRICS_TOKEN set, /metrics and /stats answer 404 without that bearer token.
 Plain HTTP only: put a TLS-terminating edge in front of it for wss:// on 443.
 
 Environment:
@@ -98,6 +119,10 @@ Environment:
   TR_REST_WEIGHT_PER_MIN        info endpoint weight a minute, at most 1100 (default 800)
   TR_LINGER_SECS                keep an unwatched market's streams this long (default 20)
   TR_MAX_CLIENTS                client connections (default 1000)
+  TR_MAX_CLIENTS_PER_IP         client connections from one address or IPv6 /64 (default 0: no cap)
+  TR_CLIENT_IP_HEADER           header holding the client address, e.g. x-forwarded-for (default:
+                                none, the peer address counts)
+  TR_METRICS_TOKEN              bearer token for /metrics and /stats (default: none, both public)
   TR_HL_WS_URL, TR_HL_INFO_URL  upstream endpoints (default Hyperliquid mainnet)
   RUST_LOG                      log filter (default info)
 ";

server/src/http.rs

@@ -1,20 +1,23 @@
 //! The gateway's HTTP side: the client websocket at `/ws`, and `/healthz`, `/readyz`,
-//! `/metrics` (Prometheus text) and `/stats` (JSON) for operators.
+//! `/metrics` (Prometheus text) and `/stats` (JSON) for operators. With an operator token set,
+//! those two answer 404 to requests without it.
 
 use std::collections::BTreeMap;
 use std::fmt::Write as _;
+use std::net::{Ipv4Addr, SocketAddr};
 use std::sync::Arc;
-use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
 use std::time::Instant;
 
-use axum::Router;
-use axum::extract::{State, WebSocketUpgrade};
-use axum::http::{StatusCode, header};
+use axum::extract::{ConnectInfo, State, WebSocketUpgrade};
+use axum::http::{HeaderMap, HeaderName, StatusCode, header};
 use axum::response::{IntoResponse, Response};
 use axum::routing::get;
+use axum::{Extension, Router};
 use serde::Serialize;
 use tokio::sync::watch;
 
+use crate::access::{self, PerIp, Secret};
 use crate::metrics::{Metrics, Quantiles, TopicKind};
 use crate::provider::UpstreamStats;
 use crate::session::{self, SessionConfig};
@@ -29,6 +32,25 @@ pub struct AppState {
     closing: watch::Receiver<bool>,
     next_session: Arc<AtomicU64>,
     started: Instant,
+    access: Arc<Access>,
+}
+
+/// Who may use the gateway. Open by default: no token, no per-address cap.
+#[derive(Default)]
+pub struct Access {
+    /// Required, as a bearer token, by `/stats` and `/metrics`.
+    pub metrics_token: Option<Secret>,
+    pub per_ip: PerIp,
+    /// The header the edge puts the client's address in.
+    pub client_ip_header: Option<HeaderName>,
+    /// Whether the first client connection has been looked at (see [`note_forwarding`]).
+    noted: AtomicBool,
+}
+
+impl Access {
+    pub fn new(metrics_token: Option<Secret>, max_per_ip: usize, client_ip_header: Option<HeaderName>) -> Self {
+        Self { metrics_token, per_ip: PerIp::new(max_per_ip), client_ip_header, noted: AtomicBool::new(false) }
+    }
 }
 
 impl AppState {
@@ -40,8 +62,14 @@ impl AppState {
             closing,
             next_session: Arc::new(AtomicU64::new(1)),
             started: Instant::now(),
+            access: Arc::default(),
         }
     }
+
+    pub fn with_access(mut self, access: Access) -> Self {
+        self.access = Arc::new(access);
+        self
+    }
 }
 
 pub fn router(state: AppState) -> Router {
@@ -54,14 +82,44 @@ pub fn router(state: AppState) -> Router {
         .with_state(state)
 }
 
-async fn ws(State(st): State<AppState>, upgrade: WebSocketUpgrade) -> Response {
+async fn ws(
+    State(st): State<AppState>,
+    peer: Option<Extension<ConnectInfo<SocketAddr>>>,
+    headers: HeaderMap,
+    upgrade: WebSocketUpgrade,
+) -> Response {
     if st.hub.metrics.clients.load(Ordering::Relaxed) >= st.max_clients as i64 {
         return (StatusCode::SERVICE_UNAVAILABLE, "too many clients\n").into_response();
     }
+    let a = &st.access;
+    note_forwarding(a, &headers);
+    let peer = peer.map_or(SocketAddr::from((Ipv4Addr::UNSPECIFIED, 0)), |Extension(ConnectInfo(p))| p);
+    let Some(slot) = a.per_ip.admit(access::client_ip(peer, a.client_ip_header.as_ref(), &headers)) else {
+        tracing::debug!("refused a client: too many connections from its address");
+        return (StatusCode::TOO_MANY_REQUESTS, "too many connections from your address\n").into_response();
+    };
     let id = st.next_session.fetch_add(1, Ordering::Relaxed);
-    upgrade
-        .max_message_size(64 * 1024)
-        .on_upgrade(move |socket| session::run(socket, st.hub, st.sessions, id, st.closing))
+    upgrade.max_message_size(64 * 1024).on_upgrade(move |socket| async move {
+        session::run(socket, st.hub, st.sessions, id, st.closing).await;
+        drop(slot);
+    })
+}
+
+/// Log once whether client connections arrive with the client-address header, so an operator
+/// can tell whether `TR_CLIENT_IP_HEADER` and a per-address cap will work behind their edge.
+/// Only whether the header is there, never what it holds.
+fn note_forwarding(a: &Access, headers: &HeaderMap) {
+    if a.noted.swap(true, Ordering::Relaxed) {
+        return;
+    }
+    let name = a.client_ip_header.clone().unwrap_or(HeaderName::from_static("x-forwarded-for"));
+    let seen = if headers.contains_key(&name) { "carries" } else { "has no" };
+    tracing::info!("the first client connection {seen} a {name} header");
+}
+
+/// `/stats` and `/metrics` as if they weren't there, unless the request carries the token.
+fn hidden(st: &AppState, headers: &HeaderMap) -> Option<Response> {
+    (!access::operator(st.access.metrics_token.as_ref(), headers)).then(|| StatusCode::NOT_FOUND.into_response())
 }
 
 async fn readyz(State(st): State<AppState>) -> Response {
@@ -127,11 +185,17 @@ pub fn stats(st: &AppState) -> Stats {
     }
 }
 
-async fn stats_json(State(st): State<AppState>) -> Response {
+async fn stats_json(State(st): State<AppState>, headers: HeaderMap) -> Response {
+    if let Some(r) = hidden(&st, &headers) {
+        return r;
+    }
     axum::Json(stats(&st)).into_response()
 }
 
-async fn metrics(State(st): State<AppState>) -> Response {
+async fn metrics(State(st): State<AppState>, headers: HeaderMap) -> Response {
+    if let Some(r) = hidden(&st, &headers) {
+        return r;
+    }
     let s = stats(&st);
     let mut o = String::new();
     let mut put = |name: &str, kind: &str, help: &str, rows: &[(String, f64)]| {

server/src/lib.rs

@@ -3,6 +3,7 @@
 //! For now it carries Hyperliquid's public feed ([`hl::HlPublicProvider`]): one shared set of
 //! upstream connections, fanned out to every client over the protocol in `tr-proto`.
 
+pub mod access;
 pub mod config;
 pub mod hl;
 pub mod http;
@@ -75,7 +76,8 @@ pub async fn start(cfg: Config) -> Result<Gateway> {
     let hub = Hub::new(provider, cfg.hub.clone(), metrics);
     hub.start();
     let (closing, closing_rx) = watch::channel(false);
-    let state = http::AppState::new(hub, cfg.session.clone(), cfg.max_clients, closing_rx);
+    let access = http::Access::new(cfg.metrics_token.clone(), cfg.max_clients_per_ip, cfg.client_ip_header.clone());
+    let state = http::AppState::new(hub, cfg.session.clone(), cfg.max_clients, closing_rx).with_access(access);
     let listener = tokio::net::TcpListener::bind(cfg.bind).await.context("binding the listen address")?;
     let addr = listener.local_addr()?;
     let (stop, stopped) = oneshot::channel::<()>();
@@ -84,7 +86,10 @@ pub async fn start(cfg: Config) -> Result<Gateway> {
         let shutdown = async {
             let _ = stopped.await;
         };
-        if let Err(e) = axum::serve(listener, app).with_graceful_shutdown(shutdown).await {
+        if let Err(e) = axum::serve(listener, app.into_make_service_with_connect_info::<SocketAddr>())
+            .with_graceful_shutdown(shutdown)
+            .await
+        {
             tracing::error!("server: {e}");
         }
     });

server/src/session.rs

@@ -265,6 +265,9 @@ async fn serve(
     let mut writer_done = false;
     let mut rpcs = tokio::task::JoinSet::new();
     let rate = crate::limit::WindowLimiter::new(cfg.max_messages_per_10s, Duration::from_secs(10));
+    // Messages over the budget, each answered with RateLimited. Another budget's worth of those
+    // in 10 s and the client is flooding, not bursting: it's closed.
+    let strikes = crate::limit::WindowLimiter::new(cfg.max_messages_per_10s, Duration::from_secs(10));
 
     loop {
         tokio::select! {
@@ -278,6 +281,14 @@ async fn serve(
                     continue;
                 };
                 if rate.try_acquire(1).is_err() {
+                    if strikes.try_acquire(1).is_err() {
+                        tracing::debug!("session {session}: still flooding after RateLimited; closing");
+                        let m = format!("over {} messages in 10 s, again and again; closing", cfg.max_messages_per_10s);
+                        link.send(&ServerMsg::Error { code: ErrorCode::RateLimited, message: m }, None).await;
+                        let close = Message::Close(Some(CloseFrame { code: 1008, reason: "rate limited".into() }));
+                        let _ = tokio::time::timeout(cfg.slow_consumer, link.out.send(Out { msg: close, recv: None })).await;
+                        break;
+                    }
                     let m = format!("more than {} messages in 10 s; this one was ignored", cfg.max_messages_per_10s);
                     link.send(&ServerMsg::Error { code: ErrorCode::RateLimited, message: m }, None).await;
                     continue;

server/tests/hardening.rs

@@ -0,0 +1,165 @@
+//! The gateway's defences against clients: the operator token on `/stats` and `/metrics`, the
+//! cap on connections from one address, and closing clients that keep flooding it.
+//!
+//! None of this needs market data, so the upstream is a port nothing listens on.
+
+use std::net::{Ipv4Addr, SocketAddr};
+use std::time::Duration;
+
+use futures::{SinkExt, StreamExt};
+use tokio_tungstenite::tungstenite::Message;
+use tokio_tungstenite::tungstenite::client::IntoClientRequest;
+use tr_gateway::Config;
+use tr_proto::{ClientInfo, ClientMsg, Encoding, PROTOCOL_VERSION, ServerMsg};
+
+type Ws = tokio_tungstenite::WebSocketStream<tokio_tungstenite::MaybeTlsStream<tokio::net::TcpStream>>;
+
+const WAIT: Duration = Duration::from_secs(5);
+
+async fn gateway(tweak: impl FnOnce(&mut Config)) -> tr_gateway::Gateway {
+    let mut cfg = Config { bind: SocketAddr::from((Ipv4Addr::LOCALHOST, 0)), ..Default::default() };
+    // Nothing listens on the discard port: the upstream stays down, which these tests don't mind.
+    cfg.hl.pool.ws_url = "ws://127.0.0.1:9/ws".into();
+    cfg.hl.info_url = "http://127.0.0.1:9/info".into();
+    tweak(&mut cfg);
+    tr_gateway::start(cfg).await.unwrap()
+}
+
+async fn get(gw: &tr_gateway::Gateway, path: &str, auth: Option<&str>) -> u16 {
+    let mut req = reqwest::Client::new().get(format!("http://{}{path}", gw.addr));
+    if let Some(a) = auth {
+        req = req.header("authorization", a);
+    }
+    req.send().await.expect("the gateway answers").status().as_u16()
+}
+
+/// Open a websocket, optionally claiming a client address in `x-forwarded-for`. `Err` carries
+/// the HTTP status of a refused upgrade.
+async fn connect(gw: &tr_gateway::Gateway, forwarded: Option<&str>) -> Result<Ws, u16> {
+    let mut req = format!("ws://{}/ws", gw.addr).into_client_request().unwrap();
+    if let Some(f) = forwarded {
+        req.headers_mut().insert("x-forwarded-for", f.parse().unwrap());
+    }
+    match tokio_tungstenite::connect_async(req).await {
+        Ok((ws, _)) => Ok(ws),
+        Err(tokio_tungstenite::tungstenite::Error::Http(r)) => Err(r.status().as_u16()),
+        Err(e) => panic!("connect: {e}"),
+    }
+}
+
+fn text(m: &ClientMsg) -> Message {
+    Message::text(serde_json::to_string(m).unwrap())
+}
+
+async fn hello(ws: &mut Ws) {
+    let info = ClientInfo { name: "hardening-test".into(), version: "0".into() };
+    let m = ClientMsg::Hello {
+        version: PROTOCOL_VERSION,
+        encoding: Encoding::Json,
+        client: info,
+        token: None,
+        resume: None,
+    };
+    ws.send(text(&m)).await.unwrap();
+    let welcome = tokio::time::timeout(WAIT, ws.next()).await.expect("a welcome").unwrap().unwrap();
+    let w: ServerMsg = serde_json::from_str(welcome.to_text().unwrap()).unwrap();
+    assert!(matches!(w, ServerMsg::Welcome { .. }), "{w:?}");
+}
+
+/// Wait until the gateway has `n` clients, so a closed socket's slot is free again.
+async fn clients(gw: &tr_gateway::Gateway, n: i64) {
+    let t0 = tokio::time::Instant::now();
+    while gw.hub().metrics.clients.load(std::sync::atomic::Ordering::Relaxed) != n {
+        assert!(t0.elapsed() < WAIT, "never got to {n} clients");
+        tokio::time::sleep(Duration::from_millis(20)).await;
+    }
+}
+
+#[tokio::test]
+async fn stats_and_metrics_stay_public_without_a_token() {
+    let gw = gateway(|_| {}).await;
+    for path in ["/healthz", "/stats", "/metrics"] {
+        assert_eq!(get(&gw, path, None).await, 200, "{path}");
+    }
+    gw.shutdown().await;
+}
+
+#[tokio::test]
+async fn a_token_hides_stats_and_metrics_from_everyone_without_it() {
+    let gw = gateway(|c| c.metrics_token = Some(tr_gateway::access::Secret::new("t0ken"))).await;
+    for path in ["/stats", "/metrics"] {
+        assert_eq!(get(&gw, path, None).await, 404, "{path} without a token");
+        assert_eq!(get(&gw, path, Some("Bearer wrong")).await, 404, "{path} with a wrong token");
+        assert_eq!(get(&gw, path, Some("t0ken")).await, 404, "{path} without the Bearer scheme");
+        assert_eq!(get(&gw, path, Some("Bearer t0ken")).await, 200, "{path} with the token");
+        assert_eq!(get(&gw, path, Some("bearer t0ken")).await, 200, "{path}: the scheme ignores case");
+    }
+    assert_eq!(get(&gw, "/healthz", None).await, 200, "health checks need no token");
+    let mut ws = connect(&gw, None).await.expect("clients need no token");
+    hello(&mut ws).await;
+    gw.shutdown().await;
+}
+
+#[tokio::test]
+async fn one_address_gets_a_capped_number_of_connections() {
+    let gw = gateway(|c| c.max_clients_per_ip = 2).await;
+    let mut a = connect(&gw, None).await.unwrap();
+    let mut b = connect(&gw, None).await.unwrap();
+    hello(&mut a).await;
+    hello(&mut b).await;
+    assert_eq!(connect(&gw, None).await.err(), Some(429), "a third from one address is refused");
+    // Without a trusted header configured, a claimed address changes nothing.
+    assert_eq!(connect(&gw, Some("198.51.100.7")).await.err(), Some(429));
+    a.close(None).await.unwrap();
+    drop(a);
+    clients(&gw, 1).await;
+    let mut c = connect(&gw, None).await.expect("a closed connection frees its slot");
+    hello(&mut c).await;
+    gw.shutdown().await;
+}
+
+#[tokio::test]
+async fn behind_an_edge_the_cap_counts_the_forwarded_address() {
+    let gw = gateway(|c| {
+        c.max_clients_per_ip = 1;
+        c.client_ip_header = Some("x-forwarded-for".parse().unwrap());
+    })
+    .await;
+    let _a = connect(&gw, Some("198.51.100.7")).await.unwrap();
+    // The edge appends the address it saw; only that last entry counts.
+    assert_eq!(connect(&gw, Some("203.0.113.1, 198.51.100.7")).await.err(), Some(429));
+    let _b = connect(&gw, Some("198.51.100.7, 198.51.100.8")).await.expect("another address");
+    let _c = connect(&gw, Some("198.51.100.9")).await.expect("another address");
+    gw.shutdown().await;
+}
+
+/// A client over the message budget gets `RateLimited` for each extra message. One that keeps
+/// going after a budget's worth of those is closed, rather than answered forever.
+#[tokio::test]
+async fn clients_that_keep_flooding_are_closed() {
+    let gw = gateway(|c| c.session.max_messages_per_10s = 5).await;
+    let mut ws = connect(&gw, None).await.unwrap();
+    hello(&mut ws).await;
+    // Five within the budget, five answered with RateLimited, and the one that closes it. No
+    // more: a server closing on unread data resets the connection, losing what it sent.
+    for t in 0..11 {
+        ws.send(text(&ClientMsg::Ping { t })).await.unwrap();
+    }
+    let mut limited = 0;
+    let closed = tokio::time::timeout(WAIT, async {
+        while let Some(m) = ws.next().await {
+            match m {
+                Ok(Message::Text(t)) if t.contains("RateLimited") => limited += 1,
+                Ok(Message::Close(f)) => return f.map(|f| u16::from(f.code)),
+                Ok(_) => {}
+                Err(e) => panic!("{e}"),
+            }
+        }
+        None
+    })
+    .await;
+    assert!(closed.is_ok(), "still open after 11 messages against a budget of 5 ({limited} RateLimited)");
+    assert_eq!(limited, 6, "five warnings, then the reason for closing");
+    assert_eq!(closed.unwrap(), Some(1008), "closed as a policy violation");
+    gw.shutdown().await;
+}