## Problem We want to detect when a "new" url is seen, create a collection out of it, and (later PR) send a notification in the weekly digest about this new URL I went back and forth on some decisions, but heres where they stand rn: 1. "New URLS": havent been seen in 90 days. Made this decision because I was worried about the performance of checking all URLs in perpetuity -- also, seems semi reasonable, if a URL isnt seen for 3 months then is seen again, thats still interesting? 2. Pattern matching: I was worried about params as part of URLs, so we do some regex stuff to try and group URLs together into one playlist when the only difference is a id/uuid/hash as part of a url (ie, /settings/2 will be grouped with settings/3) 3. LIMITs / short circuits: again seemed a semi-sensible tradeoff on exact accuracy vs performance ## Changes 1. defined new synthetic playlist source `NewUrlsSyntheticPlaylistSource() ` 2. This playlist source can actually return multiple playlists (one per new URL) 3. 2 caches -- normalized_url -> count (for list display), and normalized_url -> list of session ids (for populating the collection when a user clicks into it) 4. when user clicks on a collection to watch, use URL hash to lookup session IDs for that collection in the cache 4. do lots of URL normalizing/pattern matching / logic to ensure we're grouping URLs together, and only showing URLS first seen in the last 14 days (and not otherwise within the last 90 days) 5. limit this to 20 playlists/urls 3. gate all this behind a flag ## How did you test this code? had claude help write lots of date-checking and url-pattern-matching tests test shows me locally running a helper (to clear cache, offscreen) before refreshing, see a ~3-4 second cold load time [Screen Recording 2025-11-04 at 11.22.01 AM.mov <span class="graphite__hidden">(uploaded via Graphite)</span> <img class="graphite__hidden" src="https://app.graphite.dev/user-attachments/thumbnails/f8803d5d-c241-4c8b-91f4-ee00b0a06394.mov" />](https://app.graphite.dev/user-attachments/video/f8803d5d-c241-4c8b-91f4-ee00b0a06394.mov) <!-- Docs reminder: If this change requires updated docs, please do that! Engineers are the primary people responsible for their documentation. 🙌 --> 👉 _Stay up-to-date with [PostHog coding conventions](https://posthog.com/docs/contribute/coding-conventions) for a smoother review._ ## Changelog: (features only) Is this feature complete? <!-- Optional, but helpful for our content team! --> <!-- Yes if this is okay to go in the changelog. No if it's still hidden behind a feature flag, or part of a feature that's not complete yet, etc. -->
Products
Each product in PostHog is a vertical slice: it contains its backend (Django app), frontend (React/TypeScript), and optionally shared code. This structure ensures product features are self-contained and can evolve independently.
The entire product folder (products/<product_name>/) is treated as a Turborepo package.
Backend and frontend are sub-parts of that package.
This is the (future) home for all PostHog products (RFC).
Folder structure
products/
__init__.py
<product_name>/ # Turborepo package boundary
__init__.py # allows imports like products.<product>.backend.*
backend/ # Django app
__init__.py # marks backend as Python package/Django app
models.py
migrations/
api.py
serializers.py
tests/ # backend tests live here
frontend/ # frontend app
components/
pages/
tests/ # frontend tests live here
shared/ # optional: cross-cutting code for both backend & frontend
package.json # defines the product package in Turborepo
manifest.tsx # describes the product's features
Backend conventions
-
Each
backend/folder is a real Django app. -
Register it in
INSTALLED_APPSviaAppConfig:# products/feature_flags/backend/apps.py from django.apps import AppConfig class FeatureFlagsConfig(AppConfig): name = "products.feature_flags.backend" label = "feature_flags" verbose_name = "Feature Flags" -
✅ Always use the real Python path for imports:
from products.feature_flags.backend.models import FeatureFlag -
✅ For relations, use string app labels:
class Experiment(models.Model): feature_flag = models.ForeignKey( "feature_flags.FeatureFlag", on_delete=models.CASCADE, ) -
❌ Do not import models from
posthog.modelsor create re-exports likeproducts.feature_flags.models.
This avoids circular imports and keeps migrations/app labels stable.
Frontend conventions
- Each
frontend/directory contains the frontend app for the product. - It lives under the same package as the backend.
- Backend and frontend tooling can be independent (
requirements.txtvs.package.json) but remain in the same Turborepo package. - Tests for frontend code live inside
frontend/tests/.
Shared code
If backend and frontend need shared schemas, validators, or constants, put them in a shared/ directory under the product.
Keep shared code minimal to avoid tight coupling.
Product requirements
- Each high level product should have its own folder.
- Please keep the top level folders
under_scorecased, as dashes make it hard to import files in some languages (e.g. Python).
- Please keep the top level folders
- Each product has a few required files / folders:
manifest.tsx- describes the product's features. All manifest files are combined intofrontend/src/products.tsxon build.package.json- describes the frontend dependencies. Ideally they should all bepeerDependenciesof whatever is infrontend/package.json__init__.py- allows imports likeproducts.<product>.backend.*(only if backend exists)backend/__init__.py- marks the backend directory as a Python package/Django app (only if backend exists).frontend/- React frontend code. We run prettier/eslint only on files in thefrontendfolder on commit.backend/- Python backend code. It's treated as a separate django app.
Adding a new product
- Create a new folder
products/your_product_name, keep it underscore-cased. - Create a
manifest.tsxfile- Describe the product's frontend
scenes,routes,urls, file system types, and project tree (navbar) items. - All manifest files are combined into a single
frontend/src/products.tsxfile on build. - NOTE: we don't copy imports into
products.tsx. If you add new icons, update the imports manually infrontend/src/products.tsx. It only needs to be done once. - NOTE: if you want to add a link to the old pre-project-tree navbar, do so manually in
frontend/src/layout/navigation-3000/navigationLogic.tsx
- Describe the product's frontend
- Create a
package.jsonfile:- Keep the package name as
@posthog/products-your-product-name. Include@posthog/products-in the name. - Update the global
frontend/package.json: add your new npm package underdependencies. - If your scenes are linked up with the right paths, things should just work.
- Each scene can either export a React component as its default export, or define a
export const scene: SceneExport = { logic, component }object to export both a logic and a component. This way the logic stays mounted when you move away from the page. This is useful if you don't want to reload everything each time the scene is loaded.
- Keep the package name as
- Create
__init__.pyandbackend/__init__.pyfiles if your product has python backend code.__init__.pyallows imports likeproducts.<name>.backend.*backend/__init__.pymarks the backend directory as a Python package / Django app.- Register the backend as a Django app with an
AppConfigthat setslabel = "<name>"(notproducts.<name>). - Modify
posthog/settings/web.pyand add your new product underPRODUCTS_APPS. - Modify
tach.tomland add a new block for your product. We usetachto track cross-dependencies between python apps. - Modify
posthog/api/__init__.pyand add your API routes as you normally would (e.g.import products.early_access_features.backend.api as early_access_feature) - NOTE: we will automate some of these steps in the future, but for now, please do them manually.
Adding or moving backend models and migrations
- Create or move your backend models under the product's
backend/folder. - Use direct imports from the product location (e.g.,
from products.experiments.backend.models import Experiment) - Use string-based foreign key references to avoid circular imports (e.g.,
models.ForeignKey("posthog.Team", on_delete=models.CASCADE)) - Create a
products/your_product_name/backend/migrationsfolder. - Run
python manage.py makemigrations your_product_name -n initial_migration - If this is a brand-new model, you're done.
- If you're moving a model from the old
posthog/models/folder, there are more things to do:- Make sure the model's
Metaclass hasdb_table = 'old_table_name'set along withmanaged = True. - Run
python manage.py makemigrations posthog -n remove_old_product_name - The generated migrations will want to
DROP TABLEyour old model, andCREATE TABLEthe new one. This is not what we want. - Instead, we want to run
migrations.SeparateDatabaseAndStatein both migrations. - Follow the example in
posthog/migrations/0548_migrate_early_access_features.pyandproducts/early_access_features/migrations/0001_initial_migration.py. - Move all operations into
state_operations = []and keep thedatabase_operations = []empty in both migrations. - Run and test this a few times before merging. Data loss is irreversible.
- Make sure the model's
TODO
- A story for Python testing - run tests automatically, only test apps that changed, etc