CORDIS - SDGs: Evidence-Based Mapping with Human Review

Evidence-based CORDIS to SDG mapping with local AI, deterministic validation and human review

Evidence-based CORDIS to SDG mapping with local AI, deterministic validation and human review

Overview

This solution develops a transparent and auditable method for proposing mappings between CORDIS research projects and the Sustainable Development Goals (SDGs).

The central design principle is that AI-generated mappings are not treated as final classifications. The system separates model inference, deterministic technical validation and human review, while preserving the source evidence used for each proposal.

The prototype is publicly available at:

https://cordis-sdg-prototype.streamlit.app/

Data and sampling

The source corpus is a pinned CORDIS Horizon projects snapshot containing 23,613 project records.

A stratified sample of 80 projects was selected across Horizon Europe programme parts using a deterministic procedure based on a fixed seed and SHA-256 ordering.

The evaluation structure consists of:

• 20 pilot projects
• 60 disjoint holdout projects

The pilot and holdout cohorts are non-overlapping and are derived from the same pinned source corpus.

The deployed prototype currently exposes the 20-project pilot snapshot. The 60-project holdout was evaluated separately and is not automatically incorporated into the public application.

SDG taxonomy

The solution uses the official SDG vocabulary exposed through EU Vocabularies.

The initial SDG scheme index was not sufficient for reliable mapping because it did not contain all concept descriptions required for hierarchical retrieval. A complete taxonomy snapshot was therefore materialised by dereferencing the official SDG concept URIs and merging the resulting RDF data.

The resulting taxonomy contains:

• 812 concepts
• 17 top-level SDG goals
• 816 parent relationships
• 18 concepts with multiple parents

The taxonomy snapshot is pinned and verified using SHA-256.

Local AI inference

Inference is performed locally using:

qwen2.5:3b-instruct-q4_K_M

The model is run through Ollama with deterministic generation settings, including temperature 0 and a fixed seed.

No human reference annotations are read during model inference.

The inference process uses two stages.

Goal pass

The model first evaluates the project objective against the 17 top-level SDG goals.

For every proposed link, the model must return:

• an allowed SDG URI
• a verbatim evidence excerpt from the project objective
• a contribution mechanism
• a rationale
• a qualitative confidence label
• an uncertainty statement

If the model cannot substantively justify any proposal, it may abstain.

Target pass

Only technically valid goal proposals are expanded to more specific SDG targets.

The second pass restricts the candidate space to the selected goals and their related concepts. This limits unconstrained classification and preserves the hierarchical structure of the SDG taxonomy.

If the goal evidence fails validation, the raw proposal is preserved for review but no target expansion is performed.

Deterministic technical validation

Every model proposal is subjected to deterministic checks before it is considered technically valid.

These checks include:

• closed-schema compliance
• valid SDG concept URI
• candidate-set membership
• verbatim evidence verification against the original CORDIS objective
• technical validation of the proposal structure

The main project states are:

• validated_proposals_pending_review — at least one proposal passed technical validation and still requires human review
• rejected_proposals_pending_review — generated proposals failed technical validation but are preserved for audit and review
• model_abstained_pending_review — the model returned no proposal and explicitly abstained
• model_error_retryable — inference failed because of a recoverable technical error such as a timeout

Technical validation does not confirm substantive SDG contribution.

Human review

All model outputs remain pending human review.

A technically valid proposal means only that the URI, schema and quoted evidence passed the automated checks. It does not mean that the proposed SDG mapping is substantively correct.

This distinction is deliberately exposed in both the database and the public application.

Pilot results

The deployed pilot contains 20 projects.

The current pilot snapshot contains:

• 20 projects processed successfully
• 14 projects with technically valid proposals pending review
• 6 projects with rejected proposals pending review
• 0 model abstentions
• 20 unique project IDs
• 20 proposal records

The pilot database has been checked for internal consistency:

• no project records without proposals
• no orphan proposal records
• all projects and proposals belong to the same recorded run
• the database snapshot matches the SHA-256 value stored in the deployment manifest

Holdout evaluation

A separate 60-project holdout cohort was evaluated using the same pinned corpus, taxonomy and inference method.

The holdout is fully disjoint from the 20-project pilot.

All 60 expected holdout projects were processed successfully and verified against the predefined cohort.

The final technical states are:

• 50 projects with validated_proposals_pending_review
• 10 projects with rejected_proposals_pending_review
• 0 projects with model_error_retryable
• 0 unexpected states

Final integrity checks confirmed:

• 60 unique holdout project IDs
• no missing expected projects
• no unexpected projects
• no overlap with the pilot cohort
• no orphan project records
• no orphan proposal records

These results describe technical validation status only. They do not constitute human-confirmed SDG mappings or ground truth.

Comparison with the single-human-reviewer, AI-assisted reference set showed limited substantive agreement. Exact concept agreement was observed for 4 of 60 holdout projects (6.7%), while a further 8 projects (13.3%) matched at the level of the same top-level SDG but selected a different concept. Overall, 12 of 60 projects (20.0%) agreed at least at SDG level.

Technical validation did not predict stronger agreement with the human reference. Among the 50 technically validated projects, 9 (18.0%) matched at least at SDG level, compared with 3 of 10 technically rejected projects (30.0%). Given the small rejected subset and the fact that the reference set was produced by a single human reviewer with AI assistance, these figures should be interpreted as agreement measurements rather than accuracy estimates.

This result highlights an important limitation of the current validation layer: deterministic checks establish structural validity, provenance and evidence correspondence, but do not by themselves establish semantic correctness of the selected SDG concept.

Provenance and reproducibility

The workflow records hashes for the main artefacts used by each run.

For the deployed pilot:

• the source CORDIS corpus hash is recorded
• the SDG taxonomy hash is recorded
• the system prompt hash is recorded
• the retrieval and method fingerprint is recorded
• the deployment subset hash is recorded
• the deployed SQLite database hash is recorded

The deployed SDG taxonomy and SQLite database have been independently recomputed and confirmed to match the hashes stored in the corresponding manifests.

The public application uses a static snapshot and performs no live model inference.

Interactive prototype

The Streamlit application allows users to inspect:

• run status
• project-level technical state
• proposed SDG concepts
• qualitative confidence
• model rationale
• uncertainty
• the exact source evidence from the CORDIS project objective

This makes individual mappings inspectable rather than presenting them as opaque predictions.

Limitations

This prototype has several important limitations.

The pilot and holdout samples are small and should not be interpreted as representative estimates for the complete CORDIS population.

The stratified sampling procedure is designed to support error discovery and coverage across programme parts, not to produce unbiased corpus-wide prevalence estimates.

Qualitative confidence labels are not calibrated probabilities.

A technically valid proposal may still be substantively incorrect.

The local model adapter was introduced after AI-assisted human holdout review. Consequently, the holdout must not be described as a fully independent untouched benchmark. This chronology is recorded explicitly in the run metadata and should be considered when interpreting evaluation results.

No model-generated mapping is treated as ground truth.

The holdout analysis also revealed possible concept-attractor behaviour. A small number of SDG concepts were selected repeatedly across semantically diverse projects despite limited agreement with the human reference. For example, SDG 13.3 was selected for 10 projects but agreed at least at top-level SDG level in only 2 cases; SDG 16.7 was selected 9 times with one same-SDG-or-exact agreement; SDG 8.7 was selected 6 times with no same-SDG agreement; and SDG 17.13 was selected 4 times with no same-SDG agreement.

The model distribution was not globally more concentrated than the human-reference distribution. Its five most frequent concepts accounted for 58.3% of holdout assignments, compared with 51.5% in the human reference, while overall concentration remained similar: HHI was 0.0844 for the model and 0.0978 for the human reference, with normalized entropy of 0.8683 and 0.8643 respectively.

The difference lies mainly in the shape of the distributions. The human reference was strongly dominated by SDG 9.5, which represented 26.5% of mapped concepts, whereas the model repeatedly selected several concepts — particularly SDG 13.3, SDG 16.7, SDG 8.7, SDG 17.16 and SDG 17.13 — and substantially under-selected SDG 9.5.

Taken together, these results are consistent with possible selection imbalance or concept-attractor behaviour in the model-selection stage. Because the comparison relies on a single-human-reviewer, AI-assisted reference rather than ground truth, this is diagnostic evidence rather than proof of model bias or concept-level error rates.

Design objective

The purpose of the solution is not to maximise the number of SDG assignments.

It is to provide a reproducible and reviewable mapping process in which every proposed link can be traced back to the original project evidence, technical failures are visible, uncertainty is preserved, and human judgement remains the final authority.