37 Commits

Author SHA1 Message Date
Julian Bez
eafa69a592 chore(devex): add markdownlint to CI (#38899) 2025-10-02 06:52:17 +00:00
Paweł Szczur
caf34c5d80 feat: add active livestream subs + increase incoming events queue (#36624) 2025-08-14 07:16:43 -07:00
Paweł Szczur
6a964438b4 feat: extra metrics to livestream, bump to go1.24.6 (#36414) 2025-08-08 21:26:41 +00:00
Paweł Szczur
98da70dc32 chore: livestream update go version to v1.24.4 + deps (#34145) 2025-06-26 10:23:19 +02:00
Paweł Szczur
43a6c710fe feat(livestream): proper cors header allow origins (#31896) 2025-05-05 11:56:55 +00:00
Paweł Szczur
ed897bebec bug: remove request id middleware (#31884) 2025-05-03 14:04:53 -04:00
Paweł Szczur
8f9c29365a chore: yet another cleanup of livestream. (#31752) 2025-04-30 12:12:21 +02:00
Paweł Szczur
2110e681cd feat: livestream fast JSON parsing (#31707) 2025-04-29 15:47:40 +02:00
Paweł Szczur
1a6847563b feat: livestream parallel parsing (#31696) 2025-04-29 13:01:05 +02:00
Paweł Szczur
af1f6dcd21 feat: upgrade kafka-go, give some more buffers to reading, remove sentry (#31692) 2025-04-29 10:05:18 +00:00
Paweł Szczur
e70cdf2e71 chore: fixes and updates to livesteam (#31050)
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2025-04-10 13:16:25 +02:00
Paweł Szczur
e268860092 chore: bump to go 1.24 (#28721) 2025-02-14 13:41:27 +01:00
Paweł Szczur
6df952de44 chore: livestream proper 401 status code (#28026) 2025-01-29 13:54:07 +01:00
Paweł Szczur
b2da7c847b chore: do not use path in metric labels if response code is 404 not found. (#28020) 2025-01-29 11:20:28 +00:00
Paweł Szczur
c66641b11a fix: livestream: disable compression in promhttp handler (#27999) 2025-01-28 23:02:33 +00:00
Paweł Szczur
a0945c093b feat: add metrics to livestream server (#27953) 2025-01-28 13:04:34 +00:00
Paweł Szczur
bb6f6c80a4 fix(livestream): proper JWT library version (#27924) 2025-01-27 09:29:31 -08:00
Paweł Szczur
b64e60185c fix: livestream upgrade actions for a build (#27900) 2025-01-27 11:21:38 +01:00
Paweł Szczur
dd562753fb fix: no more crashes on empty message. (#27899) 2025-01-27 08:49:52 +00:00
Paweł Szczur
b264195e4a chore: upgrade Django to 4.2.17 and Go to 1.23 (#27682)
Co-authored-by: Paul D'Ambra <paul@posthog.com>
2025-01-20 17:23:17 +01:00
dependabot[bot]
1379f65980 build(deps): bump golang.org/x/crypto from 0.23.0 to 0.31.0 in /livestream (#26855)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: James Greenhill <fuziontech@gmail.com>
2024-12-12 22:32:23 +00:00
James Greenhill
32cca0c287 chore: do not return 404 if no stats are available on /stats endpoint (#25462) 2024-10-08 22:32:21 +00:00
James Greenhill
5d036bb48c chore: use the wrapper token if it exists (#25381) 2024-10-03 16:45:44 -07:00
James Greenhill
eba5bb5515 chore: add debugging and logging when no valid token is found (#25380) 2024-10-03 16:13:18 -07:00
James Greenhill
1d85df2b4c chore: set defaults for message payload and remove custom unmarshaller (#25377) 2024-10-03 15:41:24 -07:00
James Greenhill
33d28034bb chore: make the timestamp field optional (looks like it's coming in with a variety of types) (#25368) 2024-10-03 09:31:50 -07:00
James Greenhill
190ecf82c5 fix: Fix TTL to be time.Second based (#24383) 2024-08-14 15:58:36 -07:00
James Greenhill
904c75050e fix: docker was not building for live events (#24382) 2024-08-14 15:14:43 -07:00
James Greenhill
8cecfbfdad feat: add universal 60 sec event counter to livestream (#24380) 2024-08-14 14:28:07 -07:00
James Greenhill
fc8953cc6f feat: livestream service tests (#24209) 2024-08-06 15:08:34 -07:00
Frank Hamand
c34a1b229f feat: remove postgres token fetch from livestream (#23089)
Co-authored-by: James Greenhill <fuziontech@gmail.com>
2024-07-10 11:58:52 -07:00
Michael Matloka
a46a8e1a27 chore(livestream): Ignore invalid IP address errors (#23118) 2024-06-20 13:30:53 +00:00
James Greenhill
37f7c8ec9a chore: add sentry to liveevents (#23057)
* chore: add sentry to liveevents

* Use `isProd` for `Debug`

* Add a bunch of `CaptureException` calls

* Properly bubble `getPGConn()` error

* Fix `ParseIP` error handling

* Remove unused `personFromDistinctId()`

---------

Co-authored-by: Michael Matloka <dev@twixes.com>
Co-authored-by: Michael Matloka <michal@matloka.com>
2024-06-19 21:23:55 +00:00
Frank Hamand
be5148915f feat: add api_token to livestream jwt claims (#23082)
* feat: add api_token to livestream jwt claim

this will allow us to drop the postgres dependency soon

* add continuous deployment for livestream
2024-06-19 14:25:55 +02:00
Frank Hamand
e650a759a0 fix: add ca-certs to livestream image (#23078) 2024-06-19 10:15:41 +01:00
Julian Bez
96f60f80db fix(insights): Fix interval for total value queries (#23037) 2024-06-19 09:11:38 +00:00
James Greenhill
59eaa99c14 chore: move livestream to posthog monorepo (#23044)
* 🔥 initial commit

* update readme

* Update README.md

* Update README.md

* deploy scripts

* very basic consumer setup

* add some configs and docker-compose

* formatting for testing

* add tailscale

* flip from dev to prod flag

* set default to be not prod

* default for group_id

* tailscale up

* update gitignore

* basic geolocation

* remove unused localServer

* document mmdb

* just make configs an example

* drop raw print

* add a start script (downloads the mmdb)

* add readme and update configs.example

* ts working

* if in start

* update start script

* fix start

* fix start

* fix more

* add sql endpoints for tokenId and Person lookups

* work towards filter

* sub channel

* fix subChan

* hardcode team2 token

* add cors

* only allow get and head

* add atomicbool

* add channel to kafka

* add logs

* verbose logs

* make array

* drop sub ptrs

* more logs

* helps to loop

* drop some logigng

* move sub branch

* logging

* drop log

* hog

* Deal with numeric distinct ids later

* logs

* api_key

* send 1/1000

* remove log

* remove more logs

* change response payload

* set timestamp if needed

* fill in person_id if team_id is set

* require teamid, convert to token

* clean up subs on disconnect

* log

* check for token in another place

* clean up subs on disconnect

* drop modulo and log

* fix no assign

* don't reuse db conn for now

* drop a log

* add back commented out log

* Don't block on send to client channel

* add geo bool

* only geo events

* use wrapper ip

* don't require team in geo mode

* add an endpoint and stats keeper for teams

* remove stats keeper

* start stats keeper

* wire it up

* change the shape of the response

* omit empty error

* omit empty on the stats as well

* enable logging on back pressure

* add jwt endpoint for testing

* support multiple event types

* Get Auth Setup

* jwt team is float so turn that into int

* logs

* add auth for stats endpoint

* remove tailscale and use autoTLS on public endpoints

* default to :443 for auto tls

* remove un-needed endpoints and handlers

* Use compression because... a lot of data (#9)

* add dockerfile and CI/CD (#10)

* add dockerfile and CI/CD

* Use ubuntu not alpine

couldn't build in alpine :'(

* Add MMDB download to Dockerfile (#11)

* Use clearer name for MMDB

* Don't connect to Kafka over SSL in dev

* Fix JWT token in example config

* Add postgres.url to example config

* Add expected scope

* Fix const syntax

* Put scope validation where claims are known

* Fix audience validation

* moves

* ignore livestream for ci

* main -> master

* move GA to root

* docker lint fix

* fix typo

* fixes for docker builds

* test docker build

* livestream build docker

* dang

* Update .github/workflows/livestream-docker-image.yml

Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>

* Update .github/workflows/livestream-docker-image.yml

Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>

* don't build posthog container when PR is pushed for rust or livestream

* Update .github/workflows/livestream-docker-image.yml

Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>

* add a lot of paths-ignore

* Update .github/workflows/livestream-docker-image.yml

Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>

* Dorny filters are handling most of what I was trying to do

* remove tailscale to speed up builds

* maybe?

* push container to github.com/posthog/postog

* don't build container on PR

* remove more filters because dorny

---------

Co-authored-by: Brett Hoerner <brett@bretthoerner.com>
Co-authored-by: Zach Waterfield <zlwaterfield@gmail.com>
Co-authored-by: Frank Hamand <frankhamand@gmail.com>
Co-authored-by: Michael Matloka <michal@matloka.com>
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
2024-06-18 15:38:53 +00:00