We want to be able to make plugins non-global, but we don't want to
break existing plugins. This commit adds a test to ensure that plugins
continue to work when they are made non-global.
Previously they would have been disabled by the `is_global` check.
TODO: what will be the impact of making this change.
* test(plugin-server): add test for attachments
We didn't have one before, now we do!
* test: add tests for plugin secrets
Adds basic test for using configurations in plugins. This is a first
step towards getting lazy loading of plugins in safely:
https://github.com/PostHog/posthog/pull/15704
* chore(plugin-server): remove piscina workers
Using Piscina workers introduces complexity that would rather be
avoided. It does offer the ability to scale work across multiple CPUs,
but we can achieve this via starting multiple processes instead. It may
also provide some protection from deadlocking the worker process, which
I believe Piscina will handle by killing worker processes and
respawning, but we have K8s liveness checks that will also handle this.
This should simplify 1. prom metrics exporting, and 2. using
node-rdkafka.
* remove piscina from package.json
* use createWorker
* wip
* wip
* wip
* wip
* fix export test
* wip
* wip
* fix server stop tests
* wip
* mock process.exit everywhere
* fix health server tests
* Remove collectMetrics
* wip
* test(plugin-server): use librdkafka for functional tests
While trying to port the session recordings to use node-librdkafka I
found it useful to first implement it in the functional tests.
* use obj destructuring to make calls more self explanatory
* chore(session-recordings): separate topics for events as recordings
WIP
* fix tests
* Use simpler consumer for session recordings
* wip
* still batch things by batchSize
* add tests, improve comments
* rename topic var
* push performance_events to session recordings topic also
* Add completely separate consumer for session-recordings
* wip
* use session_id for partition key
* fix test
* handle team_id/token null
* wip
* fix tests
* wip
* use kafka_topic var in logs
* use logger
* fix test
* Fix $performance_event topic usage
* fix tests
* fix check for null/undefined
* Update posthog/api/capture.py
Co-authored-by: Tomás Farías Santana <tomas@tomasfarias.dev>
* Add test for kafka error handling
* Remove falsy teamId check
* fix statsd error
* kick ci
* Use existing getTeamByToken
* remove partition key from recordings
* Make sure producer is connected !
* fix session id kafka key test
* add back throws!
* set producer on each test
* skip flaky test
* add flush error logs
* wait for persons to be ingested
* fix skip
Co-authored-by: Tomás Farías Santana <tomas@tomasfarias.dev>
* feat(person-on-events): add option to delay all events
This change implements the option outlined in
https://github.com/PostHog/product-internal/pull/405
Here I do not try to do any large structural changes to the code, I'll
leave that for later although it does mean the code has a few loose
couplings between pipeline steps that probably should be strongly
coupled. I've tried to comment these to try to make it clear about the
couplings.
I've also added a workflow to run the functional tests against both
configurations, which we can remove once we're happy with the new
implementation.
Things of note:
1. We can't enable this for all users yet, not without the live events
view and not without verifying that the buffer size is sufficiently
large. We can however enable this for the test team and verify that
it functions as expected.
2. I have not handled the case mentioned in the above PR regarding
guarding against processing the delayed events before all events in
the delay window have been processed.
wip
test(person-on-events): add currently failing test for person on events
This test doesn't work with the previous behaviour of the
person-on-events implementation, but should pass with the new delay all
events behaviour.
* add test for KafkaJSError behaviour
* add comment re delay
* add test for create_alias
* chore: increase exports timeout
It seems to fail in CI, but only for the delayed events enabled tests.
I'm not sure why, but I'm guessing it's because the events are further
delayed by the new implementation.
* chore: fix test
* add test for ordering of person properties
* use ubuntu-latest-8-cores runner
* add tests for plugin processEvent
* chore: ensure plugin processEvent isn't run multiple times
* expand on person properties ordering test
* wip
* wip
* add additional test
* change fullyProcessEvent to onlyUpdatePersonIdAssociations
* update test
* add test to ensure person properties do not propagate backwards in time
* simplicfy person property tests
* weaken guarantee in test
* chore: make sure we don't update properties on the first parse
We should only be updating person_id and asociated distinct_ids on first
parse.
* add tests for dropping events
* increase export timeout
* increase historical exports timeout
* increase default waitForExpect interval to 1 second
* chore(plugin-server): split functional tests into feature based files
This is intended to make it more obvious what we are testing, and to try
and identify the major themes of the plugin-server functionality.
As a by product it should make things more parallelizable for jest as
the tests in different files will be isolated, runnable in separate
workers.
* use random api token, avoid db constraints
* make tests silent
* format
* chore: set number of jest workers
These tests should be pretty light given they just hit other APIs and
don't do much themselves. Memory could be an issue on constrained
environments. We shall see.
