* cleanup: remove unused team arg from registerLastStep
* cleanup: rename promises to ackPromises to make it more clear thats what they are
* cleanup(plugin-server): make waitForAck explicit/required
* add Kafka produce/ack metrics
* Clarify Kafka produce metric/labels
It's also an optimization as we also already do not schedule the job if
the plugin is disabled. It just means that if loads of jobs are
scheduled to graphile before the plugin was disabled then it will take a
while to get through.
We want to be able to make plugins non-global, but we don't want to
break existing plugins. This commit adds a test to ensure that plugins
continue to work when they are made non-global.
Previously they would have been disabled by the `is_global` check.
TODO: what will be the impact of making this change.
* test(plugin-server): add test for attachments
We didn't have one before, now we do!
* test: add tests for plugin secrets
Adds basic test for using configurations in plugins. This is a first
step towards getting lazy loading of plugins in safely:
https://github.com/PostHog/posthog/pull/15704
* feat(jobs): don't execute queued jobs for disabled configs
* add functional test for disabled plugins
---------
Co-authored-by: Harry Waye <harry@posthog.com>
There is a race condition in these tests where the consumer isn't
consuming in time to pick up bad messages, so we ensure that we set the
offsets to the earliest messages.
* chore(plugin-server): use librdkafka producer everywhere
We say some 10x improvements in the throughput for session recordings.
Hopefully there will be more improvements here as well, although it's a
little less clear cut.
I don't try to provide any improvements in guarantees around message
production here.
* we still need to enable snappy for kafkajs
* chore(plugin-server): remove piscina workers
Using Piscina workers introduces complexity that would rather be
avoided. It does offer the ability to scale work across multiple CPUs,
but we can achieve this via starting multiple processes instead. It may
also provide some protection from deadlocking the worker process, which
I believe Piscina will handle by killing worker processes and
respawning, but we have K8s liveness checks that will also handle this.
This should simplify 1. prom metrics exporting, and 2. using
node-rdkafka.
* remove piscina from package.json
* use createWorker
* wip
* wip
* wip
* wip
* fix export test
* wip
* wip
* fix server stop tests
* wip
* mock process.exit everywhere
* fix health server tests
* Remove collectMetrics
* wip
This also adds a test to ensure we are capturing usage of $set/$set_once
at the top level of the event, as posthog-js uses this method.
This was initiated by the issue mentioned
[here](https://github.com/PostHog/posthog-js/issues/615).
* chore(recordings): use cooperative-sticky rebalance strategy
This should make rebalances and lag during deploys a little less
painful. I'm setting this as the globally used strategy, when we e.g.
want to use another strategy for a specific consumer group, we can make
this configurable.
* disable rebalance_callback
* use node-rdkafka-acosom fork instead, for cooperative support
