* Swapped to use KAFKA_HOSTS everywhere
* Fixed up type of kafka config options and setup separate kafka hosts for blob consumer
* allow session recordings to have its own kafka security protocol
* remove slash commands from this pr
* syntax must be obeyed
* Update UI snapshots for `chromium` (2)
* Update UI snapshots for `chromium` (2)
* fix
* Update query snapshots
* no empty strings in kafka hosts
* fix snapshot
* fix test
---------
Co-authored-by: Paul D'Ambra <paul@posthog.com>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* chore: use pnpm to manage dependencies
* Fix CI errors
* Don't report Docker image size for external PRs
* Fix pnpm-lock.yaml formatting
* Fix module versions
* Ignore pnpm-lock.yaml
* Upgrade Cypress action for pnpm support
* Set up node and pnpm before Cypress
* Fix typescript issues
* Include patches directory in Dockerfile
* Fix Jest tests in CI
* Update lockfile
* Update lockfile
* Clean up Dockerfile
* Update pnpm-lock.yaml to reflect current package.json files
* remove yarn-error.log from .gitignore
* formatting
* update data exploration readme
* type jest.config.ts
* fix @react-hook issues for jest
* fix react-syntax-highlighter issues for jest
* fix jest issues from query-selector-shadow-dom
* fix transform ignore patterns and undo previous fixes
* add missing storybook peer dependencies
* fix nullish coalescing operator for storybook
* reorder storybook plugins
* update editor-update-tsd warning to new npm script
* use legacy ssl for chromatic / node 18 compatibility
* use pnpm for visual regression testing workflow
* use node 16 for chromatic
* add @babel/plugin-proposal-nullish-coalescing-operator as direct dependency
* try fix for plugin-server
* cleanup
* fix comment and warning
* update more comments
* update playwright dockerfile
* update plugin source types
* conditional image size reporting
* revert react-native instructions
* less restrictive pnpm verions
* use ref component name in line with style guide
Co-authored-by: Jacob Gillespie <jacobwgillespie@gmail.com>
It looks like the plugin-server isn't shutting down cleanly, from
looking at the logs. They abruptly stop.
We have a trap to pick kill the yarn command on EXIT, however, yarn v1
doesn't propagate SIGTERM to subprocesses, hence node never recieves it.
Separately it looks like the shutdown ends up being called multiple
times which results in a force shutdown. I'm not entirely sure what is
going on here but I'll leave that to another PR.
* chore(dev): use network mode host for docker-compose services
This removes the need to add kafka to /etc/hosts.
As far as I can tell this should be fine for peoples local dev except
they will be required to reset and re-migrate ClickHouse tables as they
will be trying to pull from `kafka` instead of `localhost`.
* remove ports from redis
* Update a few more references
We were for instance calling trap at a point where it wouldn't get
called, and giving special status to some processes to run in the
foreground.
Instead we:
1. wait for any process exit
2. use it's exit code for the calling process
3. kill background processes on EXIT
* use plugin-server in same repo
* remove plugins dir
* update docker files
* clean up docker files:
* visual separation
* run plugin server test stack
* build and push test image
* revert push image
* fix default yarn start
* another fix
* fix some more
* add plugin-server-prod script
* make executable
* update scripts:
* revert prod script
* fixes
* fix tests
* clean
* move build to docker
* build plugin-server in prod dockerfile
* use different entrypoint for e2e test
* fix bugs with ts-node-dev
* restore yarn.lock
* lost a few changes in the rebase
* fix dockerfile
Co-authored-by: James Greenhill <fuziontech@gmail.com>
* only load events table from the URL on allow list of URLs
* add return type to function
* pass scene URL into events table
* use urlpattern to check if scene is loaded in events table
* don't need to check if still on the same page as when loaded because urltoaction is now specific
* which means we don't need url pattern matching at all
* add scene url to events scene export's logic props
* use a different action name for each run or retries can never complete as the action cannot be saved
* do not poll results if moved away from sceneUrl
* remove tabs from person/action event tables
* so cypress can run locally
* break the dependency of running one test first so the other can pass
* move check into logic and add an e2e test
Co-authored-by: Marius Andra <marius.andra@gmail.com>
* Add setting for handing off process_event_ee to plugin server
* Add StatsD settings to KEYS
* bin/plugin-server → start-plugin-server & docker-plugin-server
* Simplify to only add docker-plugin-server
* Bring back original comment
* Turn down verbosity of plugin server install
* Remove redundant if
* Fix comment
* Remove lone newline
* Roll back unsafe script changes
* Simplify dockerized plugins
* Add some depends_on
* Clarify HAND_OFF_INGESTION env var
* Use posthog-plugin-server 1.0.0-alpha.1
* Enhance bin/plugin-server and rm bin/docker-plugin-server
* Move around PLUGIN_SERVER_INGESTION_HANDOFF ifs
* Use posthog-plugin-server@1.0.0-alpha.2
* Support kafka+ssl:// in plugin-server
* Produce to topic events_ingestion_handoff for plugin server
* Use posthog-plugin-server@1.0.0-alpha.3
* Don't import Kafka topics in FOSS
* Use @posthog/plugin-server
* Update yarn.lock
* Add commands for external ClickHouse setup/teardown
* Actually delete test CH teardown command
* ClickhouseTestRunner.setup_test_environment() in setup_test_clickhouse
* Rework test setup script to work with Postgres too
* Restore master plugins dir for merge
* Unset PLUGIN_SERVER_INGESTION_HANDOFF in docker-compose.ch.yml
* Fix unimportant typo
* Build log_event data dict only once
* Make it clear in bin/plugin-server help that it's bin
* Space space
* Add relevant settings to KEYS in bin/plugins-server
* Log all EE events to events_handoff Kafka topic for plugin server
* Clean up settings
* Fix FOSS
* Don't introduce KAFKA_EVENTS_HANDOFF
* Add cosmetic newline
* Add DEBUG WAL print()
* nodejs worker concurrency on heroku
* upgrade plugin server
* upgrade server
* fix another bug with 0.3.2
* plugin server with resilient redis
* plugin server with more resilient workers
* new plugin server
* update
* plugin progress
* blah
* add posthog config for plugins
* test gitignore
* new functionality for plugins
* support local plugin paths
* also ignore symlinks
* add positional argument
* fixes
* small fixes
* config polish
* config passed to posthog plugin
* ooooooops
* symlink fix
* cleanse dir before loading
* add cache to plugins
* pickle the goods
* unlink symlink
* pass full config
* unlink even if link points to nothing
* fix fix
* return none if value is empty
* plugin model
* plugins scene
* add config schema to plugins
* install plugins
* save descriptions
* show descriptions
* edit plugin
* save plugin config
* plugin modal
* uninstall plugins
* UX cleanup
* add "required" to plugin config
* open plugin modal after install
* split to subcomponents
* install custom plugins
* rework backend for model plugins
* Plugins on models
* simple reload pubsub
* fix apps not installed
* fix master/main issue
* fix reload command
* use the github api to get the default branch
* init plugins only if not running migrate/makemigrations
* store plugins zip archives in postgres
* tag plugins to specific versions
* save plugins in pluginConfig
* update pluginConfigs instead of adding new rows, remove from redux on uninstall
* remove debug
* run plugins from db by team
* reload when deleting
* remove debug
* smarter handling of dynamic plugins, support local plugins again
* improve typings, add some nicer warnings
* yarn lock file after merge
* squash migrations and add "locked" field to plugins
* error if folder not found in zip
* unregister plugins
* skip plugin init in test mode
* basic plugin test
* avoid mutating the same prop hash
* add pip tools to requirements.txt
* fix mypy, fix manage.py script error
* avoid plugins with mypy
* mypy fix
* abstract redis into plugin and add team_id to reload
* refactor and start work on syncing with posthog.json
* start testing plugin loading from json
* test plugin deletion
* test for syncing plugins from config
* complete and then test local json plugin sync
* test converting back and forward between an local and http path
* remove global plugin config from plugins array in posthog.json
* rename configSchema --> config_schema
* fix migration after merge
* rename from_cli to from_json
* mypy
* import pip after plugin loaded
* show error details
* raise exceptions visible to the frontend
* sync plugins on load
* access control to updating plugins from the web
* access control
* remove posthog.json from git
* test config schema from json
* if you can install via the web, you can also configure
* remove separate view access
* title as "Plugins" instead of "Installed Plugins" if we can't install ourselves and don't see the repository
* add self.team to plugin base class
* add instance_init method
* refactor into files
* sync global plugin config from json
* make global plugins work, add test
* global plugins in interface, make them take precedence over local plugins
* add comments to plugin base class
* reload/reset plugins before each test
* add error field to plugins
* add many plugin zips
* add many plugin zips, fix imports
* store errors on plugin object and test them
* fix types
* add null to error
* can be with any team ID in the test
* save problems running plugins in the plugin_config model
* try to create redis connection pool only once
* throw if no redis
* mypy
* get instance inside heartbeat and not top level
* try caching pubsub
* try pip install with -q
* install pip externally
* remove uuid and typing, now in stdlib
* more verbosity
* add pip back
* catch exceptions
* quiet and no input for pip
* check plugin reload every 10sec on new task
* fix type errors
* fix requirements error message
* use repository.json
* only load and reload plugins on workers
* rename task
* support local js plugins via py-mini-racer
* load js plugins from zip files
* extract jsplugin class and convert to syntax that uses global functions instead of initializing a class
* process events via grpc
* process events with the "posthog-plugins" queue to enable plugins
* remove old native python & mini racer plugin code
* default to false
* change env vars
* fix test
* remove grpc tools
* skip plugins in migrate.py scripts
* fix migration
* change output of settings debug banner to STDERR
* start posthog plugin server with worker
* try to fix python 3.7 test
* add fallback for the optional argument
* annoying CI test debug
* try to finally fix python 3.7 test
* here we go again
* move plugins under instance
* move plugins npm start into its own folder
* more console.log debugginf
* and again
* move plugins to separate script
* more prints
* fix test error
* docker config
* small fixes
* dckerfile fix
* reload plugins via pubsub, upgrade version
* plugins that support team setup code
* sync if made changes from config
* move plugins in menu
* require node 14 in heroku for better plugin support (namely ?. support)
* bump node version in dockerfiles
* update node versions for github actions
* update the concurrency for heroku workers
* update the concurrency for heroku workers (add link)
* Fix migrations after merge
* add ignore_result to process_event tasks
* fix: docker-preview run in parallel bug
* change order of commands
* remove separate plugins server conf script
* clarify intent
* revert castaway change
* add context to plugins/sync.py
* change everything to ValidationError
* delegate destroy to super
* no request to repository url if can't install
* make the if cleaner
* add clarifying line
* add clarifying line
* fix url field type
* rename get_redis_instance to get_client, move to posthog.redis
* remove duplicate validation
* flip if around
* simplify api logic
* simplify plugin_config api, fix global_plugin error
* remove unnecessary field rename
* mypy
* Plugins UI (#2090)
* base UI
* more UI
* load plugin image if available
* toggle enabled plugin
* plugin cards for available
* custom plugins
* change plugin configuration to drawer
* asks for confirmation when enabling or disabling a plugin
* loading state
* separation of concerns, leave new styles for separate PR
* general improvements
* remove button when installation is not available
* preemptively avoid merge conflict with #2114
* move papercups widget & hide bottom bar when drawer is open
* allow clicking the entire plugin card
* address all feedback
* move plugins under "project" menu
* Hide "configure" from globally enabled plugins
Co-authored-by: Marius Andra <marius.andra@gmail.com>
* add plugins opt in toggle to project/plugins
* choose pipeline based on team setting
* add "beta"
* plugin opt-in opt-out pages
* adjust install button
* remove tasks that are never called, remove PLUGINS_ENABLED global key
* fix responsive card display
* fix typo and drawer width
* skeleton fixes
* typo
* use "posthog-plugin-server" npm package
* "posthog-plugin-server" doc
* require the plugin server to be online before enabling plugins
* remove a few needeless "?." cases
* add hint for config_schema
* add hint for errors
* show plugin errors
* stop clicks if clicking on error
* show plugin errors
* loading indicators
* reload plugins when opting in/out
* nicer beta tag
* add frontend type
* fix mypy error
* fix test
* disable plugins if MULTI_TENANCY
* upgrade plugin-server version
* save event with plugin error
* upgrade plugin server
* squashed & optimized migrations
* remove unused import
* updates opt-in copy & hides tech details for cloud version
* fix cypress tests
* compare with None
* change plugins url and add redirect
* remove ellipsis
* use code snippet in plugin errors
* change github regex
* fix loading flickering on installing plugins
* add comment to plugin archive
* fix python style
* remove pip-tools (relic from the python plugin era)
* hard pin plugin server version
* remove copying of posthog.json from dev dockerfile (breaks if file doesn't exist, copied later anyway)
* update lockfile
Co-authored-by: James Greenhill <fuziontech@gmail.com>
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
