* Upgrade dependencies
* Fix middleware error
Among
https://docs.djangoproject.com/en/4.2/releases/4.0/#features-removed-in-4-0
* Upgrade psycopg
We need to be on >= 3.1.8
Locally there is an additional problem that somehow psycopg2
seemingly overshadows psycopg, making it appear that 3.1 works.
Had to install pip install "psycopg[binary,pool]==3.1.2" to
recreate the problem.
* Go to Django 4.1 because of problems with psycopg3
We use custom SQL that somehow doesn't get formatted in the right way
using server or client side cursors.
* Update query snapshots
* Update query snapshots
* Update query snapshots
* Update query snapshots
* Switch TaggedItem tests to assert ValidationError
Because full_clean validates since Django 4.1, see
https://docs.djangoproject.com/en/4.2/releases/4.1/#validation-of-constraints
* Remove type: ignore comments
Come up as
error: Unused "type: ignore" comment
* Update query snapshots
* Figure out psycopg problem and try Django 4.2 again
* Update query snapshots
* Fix other IN errors
* Fix getting status
* Fix psycopg3 issues
* Fix psycopg issues
* Update query snapshots
* Update query snapshots
* Update query snapshots
* Update query snapshots
* Update deps
* Update query snapshots
* Update query snapshots
* Update query snapshots
* Update query snapshots
* Fix more tests
* Adjust baseline
* Remove sqlcommenter (should be PostgresQL only anyways)
* Fix file
* Update query snapshots
* Update query snapshots
* Update query snapshots
* Fix queries
* Fix query
* Revert
* Update requirements.in
* Remove restore-virtualenv
Because not maintained anymore
* Revert "Remove restore-virtualenv"
This reverts commit c2a7ef8a1e293cc6bfd9e213f564a812a7831063.
* mypy
* Adjust num queries
* Adjust num queries
* Adjust num queries
* Update query snapshots
* Add to updated_fields
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* feat(batch-exports): Add HTTP Batch Export destination
To possibly be reused in the future, but for now it only submits
payloads in the PostHog /batch format.
* add geoip_disable, don't toJSONString elements_chain, and mark some HTTP status codes as non-retryable
* Add heartbeating to HTTP batch export
* Update query snapshots
* Update query snapshots
* fix: Re-use client session
* refactor: Rename last_uploaded_part_timestamp to last_uploaded_timestamp
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Tomás Farías Santana <tomas@tomasfarias.dev>
* Upgrade pydantic and all related
* Upgrade mypy
* Add mypy-baseline
To update baseline when you fix something (only then!) use:
[mypy cmd] | mypy-baseline sync
* just code for migrations
* use all timezones because update would cause common_timezones to be less inclusive
* install new dependencies
* add comment
* revert
* restore
* fix type
* chore(batch-exports): add snowflake export workflow
This workflow uses Snowflake internal stages to load data from
ClickHouse into a Snowflake table. We maintain the existing events table
schema as used in the existing Snowflake App.
Something I haven't done yet is:
1. made sure e.g. we get the `elements` and `person_set` etc. data into
Snowflake.
2. the additional frontend to enable configuring the Snowflake
connection.
* remove unsed var
* include excluded events in test
* feat(batch_exports): add backend API and S3 temporal workflow
This adds the backend API for batch exports, which will handle reverse
ETL exports to e.g. S3, Snowflake etc.
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* docs(capture): add spec for capture endpoints
This adds an OpenAPI specification for the capture endpoints. This is
useful for documenting the API, and further can be used e.g. in tests to
validate that both clients and the API are behaving as expected.
* feat: Add Temporal to the dev and hobby stacks
* disable elastic for hobby because of resources
* checkpoint
* update requirements
* worker is up, but without the sandbox
* ensure temporal does not depend on elastic
* Feedbacked
* pip-compile dev
* mypy fixes
* add a bit of colorful logging
* add django temporal worker to the mix
* checkpoint for dev-full docker
* Working on docker-full, but checkpointing for now
* add migration bits for full
* chore(recordings): add command to generate session recording events
The intention of this is to be able to generate somewhat realistic
session recording events to test the ingestion pipeline performance,
such that we don't need to rely on pushing to production.
This command just outputs for stdout a single JSON encoded line per
event, which can then e.g. be piped into kafkacat or
kafka-console-producer, or put to a file to then be used by vegeta to
perform load testing on the capture endpoint.
* Python 3.10
Performance gains go brrr
* Add missing SAML deps
* Add missing dep to dockerfile
* Update mypy to 0.981 for 3.10.7 compatibility
Needed this bug to be fixed: https://github.com/python/mypy/issues/13627
This also incidentally fixed the mypy bug in csv_exporter.py
* bump to 3.10.10
so that we upgrade urllib3 which sentry latest needs
Problem
running pip install -r ./requirements.txt worked fine
but running pip install -r ./requirements.txt && pip install -r ./requirements-dev.txt gave an error message
Installing collected packages: urllib3
Attempting uninstall: urllib3
Found existing installation: urllib3 1.26.13
Uninstalling urllib3-1.26.13:
Successfully uninstalled urllib3-1.26.13
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
sentry-sdk 1.11.1 requires urllib3>=1.26.11; python_version >= "3.6", but you have urllib3 1.26.5 which is incompatible.
Changes
Upgraded the requests library and regenerated the requirements files
* Revert "chore(tests): add pytest-randomly plugin to randomize test run order (#12307)"
This reverts commit 313de83cb8.
* Update snapshots
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
