* everything except plugin server and sync_available_features
* sync_available_features_done, some plugin_server done?
* and a tiny bit more
* linting
* try to fix some tests
* more test fixes/
* clean up typos
* weed wacking bugs
* more test shenanigans
* fix plugin server
* actually fix plugin server test?
* still fixing tests
* another attempt
* some pr feedback
* small fix
* fix database query accessor
* fix functional tests
* fix tests
* Update query snapshots
* Update query snapshots
* Update query snapshots
* update some comments and fxn names
* fix plugin server test
* Update query snapshots
* Update query snapshots
* Update query snapshots
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* split out InternalPerson and Person types
* disable old backfill code
* add lazy person env var setting
* lazily create person/distinct_id rows
* remove backfill.ts
* drop properties after person lookup
* remove leftover backfill bit
* fix: for empty group props
We were failing if the props were empty.
* refactor: make property definitions db dependencies explicit
We were relying on implicit details setup via `resetTestDatabase` which
makes it hard to reason about what's going on. It also uses hard coded
ideas which limits our ability to isolate tests properly, which would
help both for writing tests with confidence that they are correct, and
for adding for example parallelism.
* add test
This removes the timekeeper library and uses jest fake timers instead.
This also creates the hub once and reuses it for all tests, which is
faster than creating a new hub for each test.
* add support for token field in kafka message
* formPipelineEvent
* rename pipeline files according to new order
* wip team_id and anonymize ips
* conditional handlers and tests
* some plugin server fixes
* fix capture bug
* fix
* more fixes
* fix capture tests
* pipeline update
* fix + investigate database resets
* fix import order
* testing and typing updates
* add test for capture endpoint
* testing
* python typing
* plugin server test
* functional test
* fix test
* another fix
* make sure no team ids clash in tests
* fix
* add more metrics and logs
* cache nulls
* updates
* add more metrics
* chore(plugins-server): use Kafka to buffer app jobs requests
To remove the dependency on the Graphile Worker database on things that
may be requesting app job runs we push the jobs to a Kafka topic.
* chore: use KAFKA_JOBS instead of string literal `'jobs'`
* chore: rename startJobsBufferConsumer -> startJobsConsumer
* avoid checking eventId
* fix lint
* fix producer wrapper tests
* fix retries test
* handle offset sync
* wip
* wip
* remove exports
* do better
* use Producer not wrapper
* reset db
* mock once
* Add test for raising to the consumer
* Update plugin-server/tests/main/ingestion-queues/run-async-handlers-event-pipeline.test.ts
Co-authored-by: Yakko Majuri <38760734+yakkomajuri@users.noreply.github.com>
* and in the darkness bind them
* fix tests
* don't forget the name update!
* rename DependencyError to DependencyUnavailable
* separate dlq
* update comment
Co-authored-by: Yakko Majuri <38760734+yakkomajuri@users.noreply.github.com>
* chore(plugin-server): use DELETE instead of TRUNCATE
Truncate seems a little slow. Other options to consider:
1. PostgreSQL fsync settings in tests
2. using tmpfs for "persistence"
3. use transaction/rollback: not totally sure we'd be able to do this
in our tests but may be worth a try.
* wip
* wip
* set fsync=off
* Delete all tables in current schema
* don't bother with fsync=off
* tests(plugin-server): run same tests for single and multi process modes
Previously we were running different tests, now we run the same.
* exclude async on ingestion capability
* fix typing
* do not test for onSnapshot
* fix autocapture test
* refactor(e2e): make e2e tests 4x faster
I'm a little scared to make changes to the pluginsServer without having
some way to test that is close to production. I started changing some of
the internals in https://github.com/PostHog/posthog/pull/12191 but
started to see test failures for internals that I'm not confident in
handling without having some higher level tests testing the
pluginsServer functionality as a whole, so I started looking at the e2e
tests which are very slow and look like they don't have great coverage.
Rather than restarting the pluginsServer each time, we can use the
feature of partitioning by teamId to run multiple tests against the same
server.
To do this I've added a few helper functions for pulling out of
ClickHouse which enforce that you include a filter on teamId. It's a
little more verbose but also hopefully serves as good documentation for
how e.g. plugins work and are loaded.
I've also avoided reaching into the internals of the pluginsServer as
much as possible, only doing so to reload the plugins. Ideally we'd also
handle the reload via the PubSub which would allow us to, for example,
run the e2e tests against a separate pluginsServer process thereby
getting more confidence, but that's a stretch and maybe something we can
do without. This would however make it easy, for instance to:
1. make big refactors without fear of breaking things
2. swap out, e.g. the anonymous consumer to use instead something
written in bash or maybe something not quite as absurd.
* enable buffer, remove random teamId
* site.ts
* Update snapshots
* fix file in test archive
* Update snapshots
* more renames
* plugin source file updated at
* fix test
* last renames
* add null
* blank
* remove what we don't need
* get test to pass
* use new posthog-js
* new version
Co-authored-by: mariusandra <mariusandra@users.noreply.github.com>
* Add a is_system flag to activity logs
* Allow writing activity logs from within plugin-server
* Make changes object non-required
* Render system user information
* Log when export finishes or fails in plugin activity log
* Update activityLogLogic.plugin.test.tsx
* support adding custom files
* compile web.ts
* add injected code into /decide
* transpile better
* add id and payload stub to injected code
* skip backend if web only
* more web source
* add payload to default demo
* values list and inline for
* revert
* simpler query
* reload plugin if source changed
* pass on config that has "web":true
* add a bootloader for scripts larger than 1kb
* rename payload to config
* access posthog also in injected script
* cleanup
* pass meta object to inject
* split web_js code
* List
* add a web token for plugin configs
* test for web.ts
* use WEB_APP_INJECTION env to enable/disable injection
* fix types
* remove instance setting
* add inject_web_apps on team
* track installed web apps on team
* refresh from db
* enabling web apps makes an extra query
* null true
* clean imports
* null check
* fix migration
* fix noise in tests
* test web app injection
* update Team without going through a signal loop
* update plugin test
* update plugin test
* typo
* Update snapshots
* typing
* inject only via external scripts
* enable injection in the app
* update posthog-js
* blank source files when updating
* url dataclass
* comment
* add and remove files
* not users
* only show button to add files if there are files to add
* update in bulk
* TYPE_CHECKING
* abstract transpilation just a bit
* feedback and bug
* we're down under
* errors if failed
* raise on 404
* clean up more
* refactor
* add updated_at to list
* no need to check code equality
* remove australia ip override
* test
* remove support for random files
Co-authored-by: mariusandra <mariusandra@users.noreply.github.com>
* Experimental tracing support for plugin server
* Add tag to postgresTransaction
* Track event pipeline steps as separate spans
* Track kafka queueMessage?
* Tracing for processEvent, onEvent, onSnapshot
* plugin.runTask
* Move sentry code
* Make tracing rate configurable
* SiteUrlManager
* Update site_url
* Update imports
* Use SiteUrlManager with webhooks
* Remove most siteUrl usecases from preIngestionEvent
* Run onEvent even if action lookup fails
* Add tests
* Re-add some null guards
* Add missing test file
* Missing await
* Added setting for customising the properties used for the "display name" for any given Person
* Reworded the setting
* Formatting fixes
* Removed transiition on Tag component that was interfering with sorting animation
* Changed field type to string array
* Reverted hook based approach based on PR feedback for using global state from Kea
* Reverted personsLogic addition
* Reverted test case
* Formatting
* Removed dead code
* Fixed null check for Teams model
* Fixed array format for sql command
* fix migration
* Reverted default on the backend and moved it to the frontend for now
* Formatting
* Updated test snapshots
* Skipped migration tests
* Corrected migrations
* Fixed import order
* test
* test
* test
* untest
Co-authored-by: Ben White <ben@posthog.com>
Co-authored-by: Marius Andra <marius.andra@gmail.com>
* add description to action
* update tests
* no limit to description and use pageheader
* add ee_action
* add description as global relationship:
* fix tests
* revert to using descriptions
* revert to using descriptions
* allow description to be null
* fix tests
* persist mode
* fix e2e
* Address review points, clean up some margins
* Clean up the way editing state is determined
* Restore default description value and improve typing
* Fix typing better
* Update sql.ts
Co-authored-by: Michael Matloka <dev@twixes.com>
