Data sources: list all 24, their licences and how to remove one

.github/ISSUE_TEMPLATE/data_source.md

@@ -0,0 +1,15 @@
+---
+name: Data source
+about: Ask for a public source to be removed from AIHawk's job database
+title: "Data source: "
+---
+
+**Do not paste personal data.** No e-mail address, phone number or sign-in. If you need to be contacted, say so and we will reply on this issue. If you would rather not write in public, use the private report in SECURITY.md.
+
+**Which source** (its name in docs/data-sources.md, such as `riley`, and its address):
+
+**Who is asking** (your name or handle, and the repository or dataset you own or maintain, so we can check it):
+
+**What to remove**: the whole source, or only some postings (name the rule or the addresses):
+
+**Why** (optional): for example, the licence has changed, or the data was published by mistake.

NOTICE

@@ -12,3 +12,4 @@ The published job database
   Is merged from the public sources named in pipeline/build.py, each with its home and licence as far as
   it states one; each release's manifest.json lists them, with the rows each gave or the error that left
   it out that day, and every row names the sources that carried it. Each posting belongs to its employer and links back to the original.
+  [docs/data-sources.md](docs/data-sources.md) lists each source with its licence and how to ask for its removal.

README.md

@@ -74,6 +74,6 @@ Models go through OrcaRouter: `model` (default `deepseek/deepseek-v4.1-flash`) w
 - **Uninstall:** `pipx uninstall aihawk` (or `uv tool uninstall aihawk`), then delete `~/.aihawk`, which holds your CV, key and sign-ins.
 - **The search:** the plan is `filters.plan` in `~/.aihawk/aihawk.yaml`; restart `aihawk` after editing it.
 
-More: the [wiki](https://github.com/feder-cr/Jobs_Applier_AI_Agent_AIHawk/wiki), [CONTRIBUTING.md](CONTRIBUTING.md), [SECURITY.md](SECURITY.md), [CHANGELOG.md](CHANGELOG.md). The job database merges public sources, whose licences and posting links are in [NOTICE](NOTICE); each posting belongs to its employer. The report's shape is adapted from [career-ops](https://github.com/santifer/career-ops) (MIT).
+More: the [wiki](https://github.com/feder-cr/Jobs_Applier_AI_Agent_AIHawk/wiki), [CONTRIBUTING.md](CONTRIBUTING.md), [SECURITY.md](SECURITY.md), [CHANGELOG.md](CHANGELOG.md). The job database merges public sources, whose licences and posting links are in [NOTICE](NOTICE) and [docs/data-sources.md](docs/data-sources.md); each posting belongs to its employer. The report's shape is adapted from [career-ops](https://github.com/santifer/career-ops) (MIT).
 
 [MIT](LICENSE). The former Dots project is at [feder-cr/invisible_dots](https://github.com/feder-cr/invisible_dots).

docs/data-sources.md

@@ -0,0 +1,43 @@
+# Data sources
+
+AIHawk's job database merges 24 public sources. `pipeline/build.py` reads each one every day, and the release keeps the postings that survive the merge. Each posting names the sources that carried it.
+
+The licence column says what the source's own page or dataset card states. "Not stated" means no licence was found on that page or in the repository's files. Nothing here is a legal opinion. Postings belong to their employers, and the link in each row goes to the page this table relies on.
+
+The repository does not state how many postings each source gives. The total in [README.md](../README.md) is for the whole database.
+
+| Source | What it is | Where the data comes from | Licence or terms | Page relied on |
+|---|---|---|---|---|
+| `jobhive` | Job postings in one Parquet file | `storage.stapply.ai/jobhive/v1/all.parquet`; code home `kalil0321/ats-scrapers` | Not stated for the data. The repository's code is MIT. | [ats-scrapers](https://github.com/kalil0321/ats-scrapers) |
+| `riley` | Job postings from Riley-Dorrington's job-board aggregator | Release file `all_jobs.json.gz` of `Riley-Dorrington/job-board-aggregator` | Not stated for the release file. The repository's code is MIT, and its `data/` folder says CC BY-NC 4.0 for its curated company datasets. | [job-board-aggregator](https://github.com/Riley-Dorrington/job-board-aggregator) |
+| `rolebase` | Job postings as one JSON Lines file per board | `chandranshB/roleBase`, `data/jobs/*.jsonl` | AGPL-3.0 (the repository's stated licence) | [roleBase](https://github.com/chandranshB/roleBase) |
+| `scholarweave` | Jobs-Dataset-v2, active postings joined with companies | Hugging Face `scholarweave/Jobs-Dataset-v2` | Apache-2.0 | [dataset card](https://huggingface.co/datasets/scholarweave/Jobs-Dataset-v2) |
+| `headcount-pay` | Pay-transparency job postings | Hugging Face `HeadcountLabs/pay-transparency-jobs` | ODbL 1.0 | [dataset card](https://huggingface.co/datasets/HeadcountLabs/pay-transparency-jobs) |
+| `headcount-tech` | Tech job postings | Hugging Face `HeadcountLabs/tech-jobs` | ODbL 1.0 | [dataset card](https://huggingface.co/datasets/HeadcountLabs/tech-jobs) |
+| `headcount-remote` | Remote job postings | Hugging Face `HeadcountLabs/remote-jobs` | ODbL 1.0 | [dataset card](https://huggingface.co/datasets/HeadcountLabs/remote-jobs) |
+| `headcount-internships` | Internship postings | Hugging Face `HeadcountLabs/internships` | ODbL 1.0 | [dataset card](https://huggingface.co/datasets/HeadcountLabs/internships) |
+| `headcount-healthcare` | Healthcare job postings | Hugging Face `HeadcountLabs/healthcare-jobs` | ODbL 1.0 | [dataset card](https://huggingface.co/datasets/HeadcountLabs/healthcare-jobs) |
+| `eu-tech-jobs` | EU tech job postings, latest snapshot | Hugging Face `Aramente/eu-tech-jobs` | CC-BY-4.0 | [dataset card](https://huggingface.co/datasets/Aramente/eu-tech-jobs) |
+| `minmaxxxer` | Job postings with salaries | Hugging Face `minmaxxxer/job-postings-with-salaries` | Not stated. The card says "research-use" and gives no terms. It also says job descriptions belong to their employers. | [dataset card](https://huggingface.co/datasets/minmaxxxer/job-postings-with-salaries) |
+| `lookin4jobs` | Open job data in one JSON file | `lookin4jobs/open-jobs-data`, `data/jobs.json` | MIT (the repository's licence). The README says postings belong to their employers. | [open-jobs-data](https://github.com/lookin4jobs/open-jobs-data) |
+| `conorscode` | Open job data in one JSON file | `ConorsCode/open-jobs-data`, `data/jobs.json` | MIT (the repository's licence). The README says postings belong to their employers. | [open-jobs-data](https://github.com/ConorsCode/open-jobs-data) |
+| `fecklesspleb` | New jobs from ATS boards, in CSV | `fecklesspleb/ats-new-jobs-feed`, `data/jobs.csv` | MIT | [ats-new-jobs-feed](https://github.com/fecklesspleb/ats-new-jobs-feed) |
+| `simplify-new-grad` | New-grad job listings | `SimplifyJobs/New-Grad-Positions`, `.github/scripts/listings.json` | Not stated. No LICENSE file was found. | [New-Grad-Positions](https://github.com/SimplifyJobs/New-Grad-Positions) |
+| `simplify-internships` | Summer 2027 internship listings | `SimplifyJobs/Summer2027-Internships`, `.github/scripts/listings.json` | Not stated. No LICENSE file was found. | [Summer2027-Internships](https://github.com/SimplifyJobs/Summer2027-Internships) |
+| `vanshb03-new-grad` | New-grad job listings for 2027 | `vanshb03/New-Grad-2027`, `.github/scripts/listings.json` | MIT (`LICENSE.txt`) | [New-Grad-2027](https://github.com/vanshb03/New-Grad-2027) |
+| `vanshb03-internships` | Summer 2026 internship listings | `vanshb03/Summer2026-Internships`, `.github/scripts/listings.json` | MIT (`LICENSE.txt`) | [Summer2026-Internships](https://github.com/vanshb03/Summer2026-Internships) |
+| `simplify-trabaho` | Job listings in `data/listings.json` | `yanicells/SimplifyTrabaho` | MIT for the code. The README says the dataset is CC0 1.0 (public domain). | [SimplifyTrabaho](https://github.com/yanicells/SimplifyTrabaho) |
+| `headcount-engineering-interns` | Engineering internship postings, in CSV | `HeadcountLabs/engineering-internships`, `data/jobs.csv` | ODbL 1.0 for the job data (the repository's `LICENSE`). Job descriptions belong to their employers. | [engineering-internships](https://github.com/HeadcountLabs/engineering-internships) |
+| `headcount-software-interns` | Software engineering internship postings, in CSV | `HeadcountLabs/software-engineering-internships`, `data/jobs.csv` | ODbL 1.0 for the list data. Headcount Labs' own values are under DbCL 1.0. Job descriptions belong to their employers. | [software-engineering-internships](https://github.com/HeadcountLabs/software-engineering-internships) |
+| `applyguy` | 2027 new-grad jobs | `ApplyGuy/2027-New-Grad-Jobs`, `data/new-grad-jobs.json` | Not stated. No LICENSE file was found. | [2027-New-Grad-Jobs](https://github.com/ApplyGuy/2027-New-Grad-Jobs) |
+| `fde-jobs` | Forward-deployed engineer job list, in CSV | `stephanelx88/forward-deployed-engineer-jobs`, `data/fde-jobs.csv` | CC BY 4.0 | [forward-deployed-engineer-jobs](https://github.com/stephanelx88/forward-deployed-engineer-jobs) |
+| `remote-india` | Remote jobs open to India, in one JSON file | `raghavendra81422-boop/remote-jobs-open-to-india`, `data.json` | MIT (the repository's `LICENSE` file) | [LICENSE](https://raw.githubusercontent.com/raghavendra81422-boop/remote-jobs-open-to-india/HEAD/LICENSE) |
+
+## Removing a source
+
+A source's owner can ask for it to be removed. There are two ways.
+
+- **In public:** open an issue with the [data source template](../.github/ISSUE_TEMPLATE/data_source.md). Say which source it is, who you are, and what to remove.
+- **In private:** if you would rather not write in public, use the private report in [SECURITY.md](../SECURITY.md).
+
+What happens next: the source is removed from `SOURCES` in `pipeline/build.py` and from this table. The next daily build in [`.github/workflows/jobs-database.yml`](../.github/workflows/jobs-database.yml) does not read it. Releases already published keep their postings until the publish step drops them. It keeps the last three dated releases.

pipeline/publish.py

@@ -83,7 +83,7 @@ def main() -> None:
 
     tag = "jobs-" + manifest["built_at"][:19].replace(":", "").replace("-", "").replace("T", "-")
     names = [manifest["index"]["name"], *(item["name"] for item in manifest["files"])]
-    notes = f"{manifest['rows_out']} job postings. See {LATEST} for the current database."
+    notes = f"{manifest['rows_out']} job postings. See {LATEST} for the current database. Sources and licences: https://github.com/{arguments.repo}/blob/main/docs/data-sources.md"
     publish_files(arguments.repo, tag, arguments.out, names, notes)
 
     manifest["base_url"] = f"https://github.com/{arguments.repo}/releases/download/{tag}"

tests/test_data_sources.py

@@ -0,0 +1,32 @@
+"""The data sources doc lists exactly the sources the pipeline reads: no more, no fewer."""
+
+from __future__ import annotations
+
+import importlib.util
+import re
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+
+
+def build_sources() -> set[str]:
+    spec = importlib.util.spec_from_file_location("build", ROOT / "pipeline" / "build.py")
+    module = importlib.util.module_from_spec(spec)
+    sys.modules["build"] = module
+    spec.loader.exec_module(module)
+    return {source.name for source in module.SOURCES}
+
+
+def documented_sources() -> set[str]:
+    text = (ROOT / "docs" / "data-sources.md").read_text(encoding="utf-8")
+    return set(re.findall(r"^\| `([^`]+)` \|", text, flags=re.MULTILINE))
+
+
+def test_every_source_the_build_reads_is_documented() -> None:
+    assert build_sources() - documented_sources() == set()
+
+
+def test_every_documented_source_is_one_the_build_reads() -> None:
+    assert documented_sources() - build_sources() == set()
+    assert build_sources() and documented_sources()