{ "cells": [ { "cell_type": "markdown", "id": "dd809840", "metadata": {}, "source": [ "# Mapping CORDIS Horizon Europe projects to SDG targets\n", "\n", "**Registration v9, sha256 `72187842d621e55dedb6c6b366356131141cd47acf3324ab26bc7f06293c341c`, ratified by hash before any pipeline code was written.**\n", "\n", "This notebook reads committed artefacts and recomputes nothing. Every figure and\n", "number below comes from a file whose hash is on the cordis-sdg chain, and each\n", "cell prints the hash of what it read. Nothing here is typed from memory.\n", "\n", "**The finding this notebook is built around is a ceiling, not a score.** Of the\n", "95 (project, target) pairs the held-out reference contains, **53 have no lexical\n", "or crosswalk evidence in the project's own text at all**. No threshold reaches\n", "them. Recall of 0.221 is therefore *half of what this method could ever reach*,\n", "and the gap between 0.442 and 1.0 is the cost of the registration's rule that a\n", "mapping decision is made by rules a reader can run by hand and never by a model.\n", "\n", "The dependencies here are the notebook's alone. **The pipeline, the vocabulary\n", "build, the crosswalk verifier, the guard and the metrics are pure standard\n", "library**, and none of them needs anything in `requirements-notebook.txt`." ] }, { "cell_type": "code", "execution_count": 1, "id": "4727963d", "metadata": { "execution": { "iopub.execute_input": "2026-09-12T14:00:16.478484Z", "iopub.status.busy": "2026-09-12T14:00:16.477988Z", "iopub.status.idle": "2026-09-12T14:00:17.665969Z", "shell.execute_reply": "2026-09-12T14:00:17.664585Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "data/pipeline/evaluation-100-metrics.json babd37415581ce82ecb09c4501052ce811e834e57dad7c6735389ce321dd8a2e\n", "data/corpus/figures.json 03d79580081ca40bf0d1d25b5279a57880ddf09d750490553593387126d92f6f\n", "data/pipeline/parameters-v1.json 4a15d80f9040981d27a0977bef00251d384dedbb18e2b7460bb55d6d9f8cf834\n", "\n", "registration 72187842d621e55dedb6c6b366356131141cd47acf3324ab26bc7f06293c341c\n", "pipeline commit 0246412b44f722567feaa2e6c1b6e695de86c923\n", "parameters frozen at seq 207 values 3808845b553dfda751bf4a3e5c87bd17f32b6ca8c195a0c54321ed25e03e09e1\n", "mapping table 181d378a3078014193f3ae4f684b00b375ed62f3d24b2b34c060f6457b4a00d3\n" ] } ], "source": [ "import io\n", "import hashlib, json, pathlib, csv\n", "import matplotlib\n", "%config InlineBackend.figure_formats = ['svg']\n", "# DETERMINISM, because everything else here is checked by hash and a figure\n", "# that changes on every run cannot be. Matplotlib's SVG writer embeds a\n", "# and derives element ids from a per-process random salt, so two\n", "# runs of this notebook produced six figures that differed in 30 lines each\n", "# and in nothing anyone could see. Both are pinned here.\n", "matplotlib.rcParams['svg.hashsalt'] = 'cordis-sdg'\n", "%config InlineBackend.print_figure_kwargs = {'metadata': {'Date': None}}\n", "SVG = dict(format='svg', bbox_inches='tight', metadata={'Date': None})\n", "\n", "def save(fig, name):\n", " \"\"\"Write a figure so the bytes on disk are the bytes in the commit.\n", "\n", " savefig to a PATH opens it in text mode, so on Windows every newline\n", " becomes CRLF and sha256 of the file a reader checks out does not equal\n", " the hash published for it — 359 carriage returns in a 9 KB figure. Git\n", " normalises on the way in, so the repository was right and only the\n", " published hash was uncheckable on half the platforms that might check\n", " it. CLAUDE.md requires newline='' on every write in this repository for\n", " exactly this reason; a figure is a write. Counsel raised the consequence\n", " at cordis-sdg seq 269.\n", "\n", " Rendering to a buffer skips the text-mode translation altogether, so\n", " these bytes are LF on every platform.\n", " \"\"\"\n", " buf = io.BytesIO()\n", " fig.savefig(buf, **SVG)\n", " (FIGDIR / name).write_bytes(buf.getvalue())\n", " return name\n", "import matplotlib.pyplot as plt\n", "\n", "ROOT = pathlib.Path.cwd()\n", "if not (ROOT / \"data\").is_dir():\n", " ROOT = ROOT.parent # executed from notebooks/\n", "FIGDIR = ROOT / \"figures\"\n", "FIGDIR.mkdir(exist_ok=True)\n", "\n", "def read(rel):\n", " \"\"\"Read a committed artefact and print the hash of the bytes just read.\n", "\n", " Not the hash recorded somewhere about this file: the hash of what this\n", " cell opened. That is the difference between provenance and a caption.\"\"\"\n", " p = ROOT / rel\n", " b = p.read_bytes()\n", " print(\"%-44s %s\" % (rel, hashlib.sha256(b).hexdigest()))\n", " return json.loads(b.decode(\"utf-8\"))\n", "\n", "metrics = read(\"data/pipeline/evaluation-100-metrics.json\")\n", "figures = read(\"data/corpus/figures.json\")\n", "params = read(\"data/pipeline/parameters-v1.json\")\n", "print()\n", "print(\"registration \", metrics[\"registration_sha256\"])\n", "print(\"pipeline commit \", figures[\"pipeline_commit\"])\n", "print(\"parameters frozen at seq\", params[\"_frozen\"][\"freeze_seq\"], \"values\", metrics[\"parameters_values_sha256\"])\n", "print(\"mapping table \", figures[\"mapping_table_sha256\"])" ] }, { "cell_type": "markdown", "id": "c0531ad1", "metadata": {}, "source": [ "## 1. Method\n", "\n", "**The mapping decision is made by rules a reader can run by hand.** A language\n", "model writes the explanation of a decision already made; it never makes one.\n", "Nothing in the scoring path calls a model.\n", "\n", "Two load-bearing artefacts, both built before any project text was scored:\n", "\n", "| | |\n", "|---|---|\n", "| **Crosswalk** | EuroSciVoc category → SDG target, every row with a one-line justification citing the target's own text, reviewed before use |\n", "| **Vocabulary** | key terms extracted from target labels by a committed written rule, plus a curated synonym list, a stop-list and a negation rule |\n", "\n", "**Neither was derived from project text.** The crosswalk was built from the\n", "category vocabulary on one side and the taxonomy's target labels on the other;\n", "the only project data consulted was the list of distinct category paths, which\n", "is the table's own left-hand side. That is what makes the held-out set worth\n", "anything, and it is checkable rather than asserted: `verify_crosswalk.py` reads\n", "the same two sources and nothing else.\n", "\n", "**Validation was registered before it was run.** One seeded draw of 150\n", "projects, split 50 development and 100 evaluation. Parameters are tuned on the\n", "50 and frozen by the owner's word before the 100 is scored. The evaluation\n", "labels are sealed *and unread* until the frozen pipeline has run once on the 100\n", "and its output hash is posted." ] }, { "cell_type": "markdown", "id": "b0506934", "metadata": {}, "source": [ "## 2. Data preparation\n", "\n", "**One dataset, four distributions, because no single one carries the fields the\n", "registration matches on.** The projects extract has the objective, title and\n", "keywords; the deliverables extract has per-deliverable descriptions; the\n", "editorial description lives only in the reports JSON, in four narrative fields.\n", "All four are hashed in `data/manifest.json`.\n", "\n", "**The snapshot is not one upstream moment and does not claim to be.** The\n", "projects extract was last modified upstream eight days after the other three,\n", "and the four were retrieved at three local moments.\n", "\n", "**A standard CSV reader gets `project.csv` wrong.** CORDIS under-escapes any\n", "field whose content *begins* with a quotation mark, writing two quote characters\n", "where three are needed. Measured across every field: **994 rows (4.24%) have at\n", "least one such field; a standard reader breaks 193 of them outright and silently\n", "alters 801 more**, dropping the character. Every read of that file goes through\n", "`scripts/read_projects.py`, which splits on the literal separator instead.\n", "\n", "**Labellers and the pipeline read the same text, gaps included.** Both work from\n", "the hand-off artefact, where every field is present on every record and an\n", "absent field carries the literal string `(absent)` and is listed in that\n", "record's `absent_fields`. A labeller who cannot tell \"this project has no\n", "editorial description\" from \"this field was not included\" reads the same gap two\n", "ways on two different projects." ] }, { "cell_type": "markdown", "id": "67dc35c3", "metadata": {}, "source": [ "## 3. Mapping logic\n", "\n", "Evidence for a (project, target) pair comes from six independent channels: the\n", "five text fields, and the reviewed crosswalk. Each match is recorded as the\n", "exact phrase found with its source field.\n", "\n", "The weights below were tuned on the development 50 and frozen. Two properties\n", "are worth naming because they are choices rather than arithmetic:\n", "\n", "- **A single reviewed crosswalk row reaches the threshold on its own; a single\n", " lexical phrase never does.** One reviewed discipline-level claim is evidence;\n", " one word is not.\n", "- **A *contributing* crosswalk row counts at most once per target.** Two rows\n", " each saying \"a route to the target, not its subject\" would otherwise sum to\n", " exactly one *direct* row, which is arithmetic standing in for a reviewed\n", " judgement." ] }, { "cell_type": "code", "execution_count": 2, "id": "a5557b23", "metadata": { "execution": { "iopub.execute_input": "2026-09-12T14:00:17.668959Z", "iopub.status.busy": "2026-09-12T14:00:17.668630Z", "iopub.status.idle": "2026-09-12T14:00:17.678348Z", "shell.execute_reply": "2026-09-12T14:00:17.677013Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "parameters version v2 — frozen at cordis-sdg seq 207\n", "\n", " crosswalk_direct 3.0\n", " crosswalk_contributing 1.5\n", " synonym_subject 2.0\n", " synonym_domain 1.5\n", " synonym_policy 1.5\n", " synonym_acronym 1.5\n", " key_term_multi_word 1.0\n", " key_term_single_word 0.5\n", " key_term_not_discriminating 0.25\n", "\n", " threshold 3.0\n", " requires discriminating True\n", " cap on targets per project 5\n", " band high score >= 6.0 and >= 3 independent sources\n", " band medium score >= 4.5 and >= 2 independent sources\n", " band low score >= 3.0 and >= 1 independent sources\n" ] } ], "source": [ "w = params[\"evidence_weights\"]\n", "print(\"parameters version\", params[\"version\"], \"— frozen at cordis-sdg seq\", params[\"_frozen\"][\"freeze_seq\"])\n", "print()\n", "for k, v in w.items():\n", " if not k.startswith(\"_\"):\n", " print(\" %-30s %s\" % (k, v))\n", "a = params[\"assignment\"]\n", "print()\n", "print(\" threshold %s\" % a[\"threshold\"])\n", "print(\" requires discriminating %s\" % a[\"requires_discriminating_evidence\"])\n", "print(\" cap on targets per project %s\" % a[\"max_targets_per_project\"])\n", "for band in (\"high\", \"medium\", \"low\"):\n", " b = params[\"confidence_bands\"][band]\n", " print(\" band %-8s score >= %-4s and >= %s independent sources\"\n", " % (band, b[\"min_score\"], b[\"min_independent_sources\"]))" ] }, { "cell_type": "markdown", "id": "fbd405c0", "metadata": {}, "source": [ "## 4. Results on the development and evaluation sets\n", "\n", "**The development 50 is reported and is not the headline.** It is what the\n", "parameters were tuned against, so it cannot measure the method.\n", "\n", "Two labellers worked the development set, each set uploaded and hashed before\n", "either was revealed. Their agreement is the scale against which the pipeline's\n", "precision should be read: **two careful labellers do not fully agree about what\n", "a project is about.**" ] }, { "cell_type": "code", "execution_count": 3, "id": "35a40e63", "metadata": { "execution": { "iopub.execute_input": "2026-09-12T14:00:17.681224Z", "iopub.status.busy": "2026-09-12T14:00:17.680849Z", "iopub.status.idle": "2026-09-12T14:00:17.690397Z", "shell.execute_reply": "2026-09-12T14:00:17.688751Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "DEVELOPMENT 50 — inter-rater agreement, Hermes Cordis (identity ph-9c11aec45467c4d6) against Claude Chat (counsel), identity ph-b50aa7262165a51b\n", " both assigned 52 only first 14 only second 35 neither 8349\n", " target level agreement 0.9942 Cohen's kappa 0.677 Jaccard 0.515\n", " goal level agreement 0.9741 Cohen's kappa 0.806\n", "\n", "EVALUATION 100 — pipeline against Hermes Cordis (identity ph-9c11aec45467c4d6)\n", " reference 95 pairs over 100 projects, 31 projects with no target\n", " target level tp 21 fp 26 fn 74 precision 0.447 recall 0.221 F1 0.296\n", " goal level tp 29 fp 42 fn 50 precision 0.408 recall 0.367 F1 0.387\n" ] } ], "source": [ "proof = metrics[\"self_proof_against_seq_129\"]\n", "t, g = proof[\"target_level\"], proof[\"goal_level_secondary\"]\n", "print(\"DEVELOPMENT 50 — inter-rater agreement, %s against %s\" % (proof[\"first_labeller\"], proof[\"second_labeller\"]))\n", "print(\" both assigned %-4d only first %-4d only second %-4d neither %d\"\n", " % (t[\"both_assigned\"], t[\"only_first\"], t[\"only_second\"], t[\"neither\"]))\n", "print(\" target level agreement %.4f Cohen's kappa %.3f Jaccard %.3f\"\n", " % (t[\"agreement_rate\"], t[\"cohens_kappa\"], t[\"jaccard\"]))\n", "print(\" goal level agreement %.4f Cohen's kappa %.3f\"\n", " % (g[\"agreement_rate\"], g[\"cohens_kappa\"]))\n", "print()\n", "ev, eg = metrics[\"target_level\"], metrics[\"goal_level_secondary\"]\n", "ref = metrics[\"reference\"]\n", "print(\"EVALUATION 100 — pipeline against %s\" % ref[\"labeller\"])\n", "print(\" reference %d pairs over %d projects, %d projects with no target\"\n", " % (ref[\"pairs\"], ref[\"projects\"], ref[\"projects_with_no_target\"]))\n", "for name, m in ((\"target level\", ev), (\"goal level\", eg)):\n", " print(\" %-13s tp %-3d fp %-3d fn %-3d precision %.3f recall %.3f F1 %.3f\"\n", " % (name, m[\"true_positives\"], m[\"false_positives\"], m[\"false_negatives\"],\n", " m[\"precision\"], m[\"recall\"], m[\"f1\"]))" ] }, { "cell_type": "markdown", "id": "9bf29b49", "metadata": {}, "source": [ "### Two of the registration's four evaluation figures do not exist\n", "\n", "The registration provided for **20 of the 100 to be hand-labelled by the owner**\n", "as a human anchor. **They were never made.** That removes the evaluation set's\n", "inter-rater agreement entirely — one labeller cannot disagree with himself — and\n", "it removes the anchor. No substitute is offered for either: comparing the\n", "labeller to the pipeline would be the pipeline's own accuracy reported a second\n", "time under a name that implies two humans.\n", "\n", "**No human labelled anything in this study.** Every label on both sets was made\n", "by a language-model seat." ] }, { "cell_type": "markdown", "id": "e4d6b545", "metadata": {}, "source": [ "## 5. The recall ceiling — the finding\n", "\n", "A recall number invites the reading that tuning would fix it. **Here it would\n", "not**, and that is measurable rather than arguable: for each pair the reference\n", "contains, ask whether the pipeline found *any* evidence for that target in that\n", "project at all, at any score. A pair with none is not recoverable by lowering a\n", "threshold, reweighting, or any other tuning move." ] }, { "cell_type": "code", "execution_count": 4, "id": "74968179", "metadata": { "execution": { "iopub.execute_input": "2026-09-12T14:00:17.693895Z", "iopub.status.busy": "2026-09-12T14:00:17.693484Z", "iopub.status.idle": "2026-09-12T14:00:18.104434Z", "shell.execute_reply": "2026-09-12T14:00:18.103395Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "reference pairs 95\n", " with some evidence 42\n", " with NO evidence at any threshold 53\n", " ceiling 0.4421\n", " recall achieved 0.2211\n", " achieved as a share of the ceiling 50.0%\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "wrote figures/recall-ceiling.svg\n" ] }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.11.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "rc = metrics[\"recall_ceiling\"]\n", "print(\"reference pairs %d\" % rc[\"reference_pairs\"])\n", "print(\" with some evidence %d\" % rc[\"with_some_evidence\"])\n", "print(\" with NO evidence at any threshold %d\" % rc[\"with_no_evidence_at_any_threshold\"])\n", "print(\" ceiling %.4f\" % rc[\"ceiling\"])\n", "print(\" recall achieved %.4f\" % ev[\"recall\"])\n", "print(\" achieved as a share of the ceiling %.1f%%\" % (100 * ev[\"recall\"] / rc[\"ceiling\"]))\n", "\n", "fig, ax = plt.subplots(figsize=(9, 3.2))\n", "found = ev[\"true_positives\"]\n", "reachable_missed = rc[\"with_some_evidence\"] - found\n", "unreachable = rc[\"with_no_evidence_at_any_threshold\"]\n", "parts = [(found, \"#2d6a4f\", \"found (%d)\" % found),\n", " (reachable_missed, \"#95d5b2\", \"had evidence, not assigned (%d)\" % reachable_missed),\n", " (unreachable, \"#d8d8d8\", \"no evidence at any threshold (%d)\" % unreachable)]\n", "left = 0\n", "for value, colour, label in parts:\n", " ax.barh([0], [value], left=left, color=colour, edgecolor=\"white\", label=label)\n", " left += value\n", "ax.set_yticks([])\n", "ax.set_xlim(0, rc[\"reference_pairs\"])\n", "ax.set_xlabel(\"(project, target) pairs in the held-out reference\")\n", "ax.set_title(\"Recall is bounded by the method: %d of %d pairs are unreachable at any threshold\"\n", " % (unreachable, rc[\"reference_pairs\"]), fontsize=11)\n", "ax.legend(loc=\"upper center\", bbox_to_anchor=(0.5, -0.35), ncol=3, frameon=False, fontsize=9)\n", "for spine in (\"top\", \"right\", \"left\"):\n", " ax.spines[spine].set_visible(False)\n", "fig.tight_layout()\n", "save(fig, \"recall-ceiling.svg\")\n", "print(\"\\nwrote figures/recall-ceiling.svg\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "621ecbc3", "metadata": {}, "source": [ "**The held-out set behaves like the set the method was tuned on.** The\n", "development ceiling is 0.446 over the union of two label sets and the evaluation\n", "ceiling is 0.442 over one labeller. Those are not like for like and the\n", "closeness is evidence about the method's reach, not a controlled comparison.\n", "\n", "**The failure is silence, not noise.** Of the 64 projects where the pipeline and\n", "the reference disagree, the great majority are cases where the pipeline found\n", "nothing the labeller found, rather than cases where it invented something." ] }, { "cell_type": "code", "execution_count": 5, "id": "8c2cd7bc", "metadata": { "execution": { "iopub.execute_input": "2026-09-12T14:00:18.109171Z", "iopub.status.busy": "2026-09-12T14:00:18.108481Z", "iopub.status.idle": "2026-09-12T14:00:18.495686Z", "shell.execute_reply": "2026-09-12T14:00:18.493900Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "wrote figures/disagreement-shape.svg\n" ] }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.11.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "shape = figures[\"where_the_mapping_is_uncertain\"][\"disagreement_shape_on_the_evaluation_100\"]\n", "labels = [\"recall only\\n(found nothing\\nthe labeller found)\",\n", " \"both directions\",\n", " \"precision only\\n(found something\\nthe labeller did not)\"]\n", "values = [shape[\"recall_only\"], shape[\"both_directions\"], shape[\"precision_only\"]]\n", "fig, ax = plt.subplots(figsize=(7.5, 3.4))\n", "bars = ax.bar(labels, values, color=[\"#1d3557\", \"#457b9d\", \"#a8dadc\"])\n", "ax.bar_label(bars, padding=2)\n", "ax.set_ylabel(\"projects\")\n", "ax.set_title(\"How the %d disagreeing projects disagree\" % shape[\"projects_disagreeing\"], fontsize=11)\n", "for spine in (\"top\", \"right\"):\n", " ax.spines[spine].set_visible(False)\n", "fig.tight_layout()\n", "save(fig, \"disagreement-shape.svg\")\n", "print(\"wrote figures/disagreement-shape.svg\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "1581e003", "metadata": {}, "source": [ "## 6. The whole snapshot\n", "\n", "The frozen pipeline was run over all **23,451** projects to build the submission\n", "table. The counts below are from that run." ] }, { "cell_type": "code", "execution_count": 6, "id": "fce15242", "metadata": { "execution": { "iopub.execute_input": "2026-09-12T14:00:18.500792Z", "iopub.status.busy": "2026-09-12T14:00:18.500187Z", "iopub.status.idle": "2026-09-12T14:00:18.513742Z", "shell.execute_reply": "2026-09-12T14:00:18.511958Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "projects 23451\n", " with a target 9394 (40.1%)\n", " goal level only 6882 (29.3%)\n", " nothing at all 7175 (30.6%)\n", "rows in the table 28833\n", "\n", "targets in the taxonomy 169\n", "targets never assigned 33\n", "goals never assigned none — all 17 are reached\n" ] } ], "source": [ "shape = figures[\"corpus_shape\"]\n", "total = figures[\"coverage\"][\"projects_in_table\"]\n", "print(\"projects %d\" % total)\n", "print(\" with a target %5d (%.1f%%)\" % (shape[\"projects_with_a_target\"], 100*shape[\"projects_with_a_target\"]/total))\n", "print(\" goal level only %5d (%.1f%%)\" % (shape[\"projects_goal_level_only\"], 100*shape[\"projects_goal_level_only\"]/total))\n", "print(\" nothing at all %5d (%.1f%%)\" % (shape[\"projects_unassigned\"], 100*shape[\"projects_unassigned\"]/total))\n", "print(\"rows in the table %d\" % shape[\"rows\"])\n", "print()\n", "print(\"targets in the taxonomy %d\" % figures[\"least_linked_targets\"][\"targets_in_taxonomy\"])\n", "print(\"targets never assigned %d\" % len(figures[\"least_linked_targets\"][\"targets_never_assigned\"]))\n", "print(\"goals never assigned %s\" % (figures[\"goals_never_assigned\"] or \"none — all 17 are reached\"))" ] }, { "cell_type": "code", "execution_count": 7, "id": "ef77e907", "metadata": { "execution": { "iopub.execute_input": "2026-09-12T14:00:18.516925Z", "iopub.status.busy": "2026-09-12T14:00:18.516487Z", "iopub.status.idle": "2026-09-12T14:00:19.223723Z", "shell.execute_reply": "2026-09-12T14:00:19.222617Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "wrote figures/goals-by-project-count.svg\n" ] }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.11.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "goals = sorted(figures[\"goals_by_project_count\"], key=lambda d: -d[\"projects\"])\n", "fig, ax = plt.subplots(figsize=(9, 5))\n", "ax.barh([\"SDG %s\" % d[\"goal_id\"] for d in goals][::-1],\n", " [d[\"projects\"] for d in goals][::-1], color=\"#457b9d\")\n", "ax.set_xlabel(\"projects with at least one target in this goal\")\n", "ax.set_title(\"Which SDGs the corpus reaches, and which it barely does\", fontsize=11)\n", "for spine in (\"top\", \"right\"):\n", " ax.spines[spine].set_visible(False)\n", "fig.tight_layout()\n", "save(fig, \"goals-by-project-count.svg\")\n", "print(\"wrote figures/goals-by-project-count.svg\")\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 8, "id": "715e891a", "metadata": { "execution": { "iopub.execute_input": "2026-09-12T14:00:19.226682Z", "iopub.status.busy": "2026-09-12T14:00:19.226428Z", "iopub.status.idle": "2026-09-12T14:00:19.638625Z", "shell.execute_reply": "2026-09-12T14:00:19.637188Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "wrote figures/most-linked-targets.svg\n" ] }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.11.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "top = figures[\"most_linked_targets\"][:15]\n", "fig, ax = plt.subplots(figsize=(9, 5))\n", "ax.barh([d[\"key\"] for d in top][::-1], [d[\"count\"] for d in top][::-1], color=\"#2a9d8f\")\n", "ax.set_xlabel(\"(project, target) assignments\")\n", "ax.set_ylabel(\"SDG target\")\n", "ax.set_title(\"The fifteen most frequently assigned targets\", fontsize=11)\n", "for spine in (\"top\", \"right\"):\n", " ax.spines[spine].set_visible(False)\n", "fig.tight_layout()\n", "save(fig, \"most-linked-targets.svg\")\n", "print(\"wrote figures/most-linked-targets.svg\")\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 9, "id": "3f254e11", "metadata": { "execution": { "iopub.execute_input": "2026-09-12T14:00:19.640960Z", "iopub.status.busy": "2026-09-12T14:00:19.640735Z", "iopub.status.idle": "2026-09-12T14:00:19.990059Z", "shell.execute_reply": "2026-09-12T14:00:19.987941Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "wrote figures/relevance-by-field.svg\n", " humanities 1882 projects, 539 with a target, 28.6%\n", " natural sciences 13296 projects, 5093 with a target, 38.3%\n", " engineering and technology 6579 projects, 3224 with a target, 49.0%\n", " social sciences 5340 projects, 3007 with a target, 56.3%\n", " medical and health sciences 5045 projects, 2897 with a target, 57.4%\n", " agricultural sciences 1390 projects, 1134 with a target, 81.6%\n" ] }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.11.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "fields = [d for d in figures[\"by_scientific_field\"] if d[\"key\"] != \"(absent)\"]\n", "fields.sort(key=lambda d: d[\"rate\"])\n", "fig, ax = plt.subplots(figsize=(9, 4))\n", "bars = ax.barh([d[\"key\"] for d in fields], [100*d[\"rate\"] for d in fields], color=\"#e76f51\")\n", "ax.bar_label(bars, fmt=\"%.1f%%\", padding=2, fontsize=9)\n", "ax.set_xlabel(\"% of projects in this field carrying at least one target\")\n", "ax.set_xlim(0, 100)\n", "ax.set_title(\"SDG relevance varies by scientific field\", fontsize=11)\n", "for spine in (\"top\", \"right\"):\n", " ax.spines[spine].set_visible(False)\n", "fig.tight_layout()\n", "save(fig, \"relevance-by-field.svg\")\n", "print(\"wrote figures/relevance-by-field.svg\")\n", "for d in fields:\n", " print(\" %-32s %5d projects, %5d with a target, %.1f%%\"\n", " % (d[\"key\"], d[\"projects\"], d[\"with_a_target\"], 100*d[\"rate\"]))\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 10, "id": "0dc33166", "metadata": { "execution": { "iopub.execute_input": "2026-09-12T14:00:19.993854Z", "iopub.status.busy": "2026-09-12T14:00:19.993284Z", "iopub.status.idle": "2026-09-12T14:00:20.224518Z", "shell.execute_reply": "2026-09-12T14:00:20.223594Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "wrote figures/confidence-bands.svg\n", "{'high': 2625, 'medium': 2923, 'low': 9228}\n" ] }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.11.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "bands = {d[\"key\"]: d[\"count\"] for d in figures[\"confidence_bands\"]}\n", "order = [b for b in (\"high\", \"medium\", \"low\") if b in bands]\n", "fig, ax = plt.subplots(figsize=(6.5, 3.2))\n", "bars = ax.bar(order, [bands[b] for b in order], color=[\"#2d6a4f\", \"#74c69d\", \"#d8f3dc\"][:len(order)],\n", " edgecolor=\"#40916c\")\n", "ax.bar_label(bars, padding=2)\n", "ax.set_ylabel(\"target assignments\")\n", "ax.set_title(\"Most assignments are in the weakest band\", fontsize=11)\n", "for spine in (\"top\", \"right\"):\n", " ax.spines[spine].set_visible(False)\n", "fig.tight_layout()\n", "save(fig, \"confidence-bands.svg\")\n", "print(\"wrote figures/confidence-bands.svg\")\n", "print({k: bands[k] for k in order})\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "cbe7ea3e", "metadata": {}, "source": [ "## 7. Limitations\n", "\n", "**The evaluation reference is one automated labeller working blind.** The owner's\n", "20 hand-labels were never made, so there is no inter-rater figure on the\n", "evaluation set and no human anchor, and the pre-registered adjudication rule\n", "makes that labeller's set the sole reference for all 100 rather than for 80.\n", "\n", "**No human labelled anything in this study.**\n", "\n", "**Recall is bounded at 0.442 by the method**, not by its settings. 53 of 95\n", "reference pairs have no evidence at any threshold.\n", "\n", "**30.6% of the corpus receives no assignment at all** — 7,175 of 23,451\n", "projects — and 33 of the 169 targets are never assigned anywhere.\n", "\n", "**Most assignments sit in the weakest confidence band.**\n", "\n", "**Four fifths of the EuroSciVoc vocabulary is deliberately unmapped**, because no\n", "defensible one-line justification exists against any target's text. Inventing one\n", "would produce exactly the superficial mapping that a quality criterion penalises.\n", "\n", "**The explanation pass is parked.** No model credential is set in this\n", "environment and the registration forbids one in the repository or the room, so\n", "the pass records its prompts with their hashes and writes nothing that pretends\n", "to be a model's work. The `explanation` and `prompt_sha256` columns of the\n", "mapping table say so on every row rather than being blank.\n", "\n", "**There is no acceptance threshold.** This is a report of what the method does,\n", "not a gate it passed." ] }, { "cell_type": "code", "execution_count": 11, "id": "76bcd2c3", "metadata": { "execution": { "iopub.execute_input": "2026-09-12T14:00:20.227053Z", "iopub.status.busy": "2026-09-12T14:00:20.226814Z", "iopub.status.idle": "2026-09-12T14:00:20.298874Z", "shell.execute_reply": "2026-09-12T14:00:20.297827Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "figures written, and committed as SVG for the prototype:\n", " confidence-bands.svg 24.8 KB 7d55aa9745854158\n", " disagreement-shape.svg 31.0 KB d9c6a91a11481c32\n", " goals-by-project-count.svg 43.5 KB 7db6eae48b966e1f\n", " most-linked-targets.svg 40.9 KB 8f2caf29f882ebf6\n", " recall-ceiling.svg 35.7 KB 2877c3ac441ffbb0\n", " relevance-by-field.svg 42.6 KB 46baad33a8d40a4c\n" ] } ], "source": [ "print(\"figures written, and committed as SVG for the prototype:\")\n", "for p in sorted(FIGDIR.glob(\"*.svg\")):\n", " b = p.read_bytes()\n", " print(\" %-32s %6.1f KB %s\" % (p.name, len(b)/1024, hashlib.sha256(b).hexdigest()[:16]))" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.14.7" } }, "nbformat": 4, "nbformat_minor": 5 }