32 Commits

Author SHA1 Message Date
Ben White
b20fd9371f fix(node): Don't keep the messages array in the closure (#39775) 2025-10-16 15:47:51 +02:00
Paweł Ledwoń
7e911d2621 chore: remove blobby v1 (#37625)
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
2025-09-10 11:02:37 +02:00
David Newell
16711c3e29 chore: replace error tracking in plugin server (#31654) 2025-04-29 09:19:26 +02:00
Ben White
fc58b35d57 chore: Add retries to some node tests (#29725) 2025-03-10 16:13:38 +01:00
Ben White
5821ba14ec feat: Improve plugin server startup (#29425) 2025-03-07 11:43:02 +01:00
Ben White
d234167f80 feat(cdp): Mini purge of old ingesters (#29341) 2025-03-03 15:28:31 +01:00
Ben White
dc77ad8b01 chore(cdp): Removed job and schedules (#29352) 2025-03-03 13:31:54 +01:00
Ben White
547ad4c4bd fix: Make v2 ingestion the default (#29236) 2025-02-26 11:32:21 +00:00
Meikel Ratz
d4dbac3663 feat(cdp): remove legacy kafka based consumer (#27760)
Co-authored-by: Ben White <ben@posthog.com>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
2025-01-23 14:09:32 +01:00
Ben White
080bef59fa chore: Remove unused env vars (#27596) 2025-01-17 10:15:14 +01:00
Ben White
b20d6f86ba feat(cdp): Pre-cyclotron refactor (#24681) 2024-09-04 10:44:34 +02:00
Oliver Browne
9734a40c96 feat: cyclotron (#24228)
Co-authored-by: Brett Hoerner <brett@bretthoerner.com>
Co-authored-by: Ben White <ben@posthog.com>
2024-08-21 12:24:56 -06:00
Oliver Browne
7207da93e9 feat: Inline plugins (plugins without running VM2) (#23443)
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Ben White <ben@posthog.com>
2024-07-24 14:49:31 +03:00
Tiina Turban
60ba76d9c4 feat: Part 1 of ability to spin up different pipelines (#23601) 2024-07-17 14:11:49 +02:00
Ben White
784a393404 fix: Simplify plugin server tests (#18358) 2023-11-03 10:40:10 +01:00
Ben White
ffdda1d392 feat: Realtime playback for new ingestion flow (#15627) 2023-06-23 12:39:07 +02:00
Harry Waye
a54a0773eb chore: remove stalenss code (#15476)
I don't see it used anywhere.
2023-05-10 17:12:41 +00:00
Harry Waye
7ba6fa7148 chore(plugin-server): remove piscina workers (#15327)
* chore(plugin-server): remove piscina workers

Using Piscina workers introduces complexity that would rather be
avoided. It does offer the ability to scale work across multiple CPUs,
but we can achieve this via starting multiple processes instead. It may
also provide some protection from deadlocking the worker process, which
I believe Piscina will handle by killing worker processes and
respawning, but we have K8s liveness checks that will also handle this.

This should simplify 1. prom metrics exporting, and 2. using
node-rdkafka.

* remove piscina from package.json

* use createWorker

* wip

* wip

* wip

* wip

* fix export test

* wip

* wip

* fix server stop tests

* wip

* mock process.exit everywhere

* fix health server tests

* Remove collectMetrics

* wip
2023-05-03 14:42:16 +00:00
Yakko Majuri
c47a73165a feat(plugin-server): use graphile-worker crontab (#12242)
* yeet references to redlock

* rename jobs/ to graphile-worker/

* feat(plugin-server): use graphile-worker crontab

* remove debugging

* yeet redlock dependency

* remove legacy test

* Update comment

* Update plugin-server/src/main/pluginsServer.ts

Co-authored-by: Harry Waye <harry@posthog.com>

* address review, update tests

* fix old tests

* testing, testing

* maybe fix sigterm

Co-authored-by: Harry Waye <harry@posthog.com>
2022-10-18 11:44:41 -03:00
Yakko Majuri
53b527dbbe refactor(graphile-worker): update terminology, clearer capabilities approach for setup (#12203)
* rename legacy references to queue to more appropriate worker terminology

* rename startJobsConsumer -> startGraphileWorker, no-op refactor

* add back enqueue success and failure metrics

* fix mock import

* fix test for good
2022-10-12 10:24:22 -03:00
Yakko Majuri
a34228c49f refactor: yeet job queues scaffolding in favor of only graphile worker (#12178)
* refactor: rename graphile queue to graphile worker

* refactor: rename job-queues/ to jobs/

* refactor: move graphile-worker to top level jobs/ dir

* refactor: remove references to jobQueueManager

* remove promise from startJobQueueConsumer

* remove job-queue-manager.ts

* remove non-test references to JobQueueBase

* make fs-queue independent from JobQueueBase

* rename FsQueue to MockGraphileWorker

* add missing pauseConsumer method to MockGraphileWorker

* rename fs-queue.ts --> mock-graphile-worker.ts

* delete job-queue-base.ts

* get rid of JobQueue type

* rename graphileQueue --> graphileWorker

* rename JobQueueConsumerControl --> JobsConsumerControl

* remove unused jobs test

* rename startJobQueueConsumer --> startJobsConsumer

* fix tests job imports

* rename jobQueueManager --> graphileWorker in tests

* remove JobQueueManager tests

* fix import

* handle metrics and retries on graphileWorker.enqueue

* minor fix

* Delete buffer.ts

* Revert "Delete buffer.ts"

This reverts commit 40f1761d3139c58d7d6510ee19c5b1253040e209.

* add initial test scaffolding

* bring back relevant worker control promises

* fix existing tests

* add tests for graphile worker

* fix exportEvents retries test

* update e2e buffer test
2022-10-11 15:40:34 -03:00
Yakko Majuri
4bce5dfa8a feat: (bring back) buffer 3.0 again (#10896)
* Revert "Revert "feat: (bring back) buffer 3.0  (#10874)" (#10883)"

This reverts commit e203bc7cfa.

* reduce graphile load
2022-07-20 12:16:13 +00:00
Yakko Majuri
e203bc7cfa Revert "feat: (bring back) buffer 3.0 (#10874)" (#10883)
This reverts commit 3e772b8614.
2022-07-19 17:50:06 +00:00
Yakko Majuri
3e772b8614 feat: (bring back) buffer 3.0 (#10874)
* Revert "Revert "feat: buffer 3.0 (graphile) (#10735)" (#10802)"

This reverts commit ca8c4d0271.

* add metrics and error tracking
2022-07-19 16:34:07 +00:00
Yakko Majuri
ca8c4d0271 Revert "feat: buffer 3.0 (graphile) (#10735)" (#10802)
This reverts commit 9a2a9046cb.
2022-07-14 18:24:58 +00:00
Yakko Majuri
9a2a9046cb feat: buffer 3.0 (graphile) (#10735)
* feat: buffer 3.0 (graphile)

* fixes

* test

* address review

* add test for buffer processAt
2022-07-13 11:32:00 +00:00
Karl-Aksel Puulmann
6c637397fc refactor(plugin-server): extract capabilities for VMs and plugin server (#9714)
* Extract calculation method from vm/lazy.ts

* Add tests for capabilities.ts

* Move capabilities.ts under vm which it's more closely tied to

* Add (blank) capabilities object to hub

This will be used to check whether to start certain jobs

* Don't start plugin scheduled tasks unless have the capabilities

* scheduledTasks => pluginScheduledTasks in capabilities

* Cancel all jobs on schedule exit

* Remove capabilities code from services/schedule.ts

* scheduleControl => pluginScheduleControl

* Flip conditional

* Do less manual cleanup

* Add test for stopping node-schedule at exit

* Move cancelAllScheduledJobs

* Test capabilities check in server.test.ts

* Capabilities check for jobs

* Improve comment

* Rename method

* Rename jobs => processJobs

* Fix failed import
2022-05-11 09:01:54 +03:00
Yakko Majuri
77a3790abf chore(plugin-server-tests): refactor tests to use a top-level describe (#8955) 2022-03-10 11:18:24 +00:00
Yakko Majuri
003f0682c4 Plugin server health endpoint (#7983)
* show last event ingested timestamp in system status

* wip plugin server healthcheck endpoint

* plugin server health endpoint

* fix readme prettier issue

* update tests

* move server.listen

* fixes, tests

* longer staleness default

* change endpoint to _health
2022-01-26 09:30:01 -03:00
James Greenhill
434e379f9a Add 'plugin-server/' from commit '01a99a4e26b0b11f068a7073d6b94e53a7214d33'
git-subtree-dir: plugin-server
git-subtree-mainline: 776b056b6d
git-subtree-split: 01a99a4e26
2021-10-28 14:59:19 -07:00
James Greenhill
145937a435 Revert "Monorepo with updated history" 2021-10-28 14:55:17 -07:00
James Greenhill
65512ae16f Pack up plugin-server for moving 2021-10-12 15:45:42 -07:00