# Pipeline Workflow ## Research Publication and Metrics Reporting Pipeline This guide describes the operational sequence for refreshing the **Research Publication and Metrics Reporting Pipeline**. It assumes that the software has already been installed and configured as described in `INSTALLATION.md`. The workflow was developed and validated at the University of Kent using exports from Kent Academic Repository (KAR), institutional REF Unit of Assessment (UoA) information, OpenAlex and Scopus. Institutions adapting the workflow will need to map their own repository and organisational data to the expected inputs. ## Workflow summary The production workflow consists of eight stages: 1. `run\_impact\_enrichment.py` 2. `merge\_and\_expand\_uoa.py --merge-only` 3. `merge\_impact\_data.py` 4. `run\_metrics\_pipeline.py` 5. `build\_uoa\_reports.py` 6. `build\_school\_splits.py` 7. `build\_school\_reports.py` 8. `build\_university\_report\_dates.py` Stages 1 and 4 make external API calls and can take several hours for a large institutional dataset. Stages 2, 3, 5, 6, 7 and 8 are local processing/report-generation stages. ## Before starting a refresh ### 1\. Update repository output exports Replace the previous repository output exports in: ```text input/kar\_exports/outputs/ ``` with the current exports. Do not leave old and new versions of the same exports together in the directory, as the scripts load all supported input files that they find. The University of Kent implementation uses separate KAR output exports for organisational areas. ### 2\. Update repository author exports Replace the previous author exports in: ```text input/kar\_exports/authors/ ``` with the current versions. ### 3\. Update the staff/UoA lookup if required Replace or update the institutional lookup in: ```text input/staff\_uoa/ ``` when a newer version is available. ### 4\. Confirm API configuration The private `.env` file in the project root should contain: ```text OPENALEX\_API\_KEY=your\_openalex\_api\_key OPENALEX\_EMAIL=your\_contact\_email SCOPUS\_API\_KEY=your\_scopus\_api\_key ``` Do not publish the real `.env` file. ### 5\. Decide whether the run is a fresh metrics snapshot For a completely fresh metrics snapshot, previous API caches and processing state may need to be cleared or archived. This is particularly relevant because citation counts, citation percentiles, FWCI and other external metrics can change between reporting periods. Do not delete caches from a run that may need to be resumed. ### 6\. Archive previous generated outputs if required Generated files under `output/` and `input/impact\_data/` can be archived before a fresh run if previous reporting snapshots need to be retained. The public software package does not include generated institutional outputs. \--- # Stage 1 — OpenAlex impact enrichment ## Command From the project root: ```cmd python scripts\\run\_impact\_enrichment.py ``` ## Inputs Primarily: ```text input/kar\_exports/outputs/ ``` ## External service **OpenAlex** This is an API-heavy stage and may take several hours. ## Processing The script: * discovers the current repository output exports * extracts DOI and organisational information * queries OpenAlex * derives collaboration information * retrieves SDG information * retrieves cited-journal information * retrieves citing-journal information * maintains processing state for long-running work ## Outputs ```text input/impact\_data/REPORT\_publications.csv input/impact\_data/REPORT\_funding.csv ``` Processing information is stored under: ```text cache/impact\_tracking/ ``` This can include: ```text processed\_dois.csv failed\_dois.csv TEMP\_publications.csv TEMP\_funding.csv ``` ## QA after Stage 1 Confirm that both final report files exist. Useful checks include: * publication row count * unique DOI count * funding row count * number of processed DOIs * failed DOI count * whether the final DOI count is consistent with the processing state Failed OpenAlex matches should not automatically be treated as repository metadata errors. During validation, failed matches included: * malformed DOI strings * repository-specific identifiers * records not covered by OpenAlex * metadata mismatches * external request failures Where obvious DOI formatting errors are identified, they should preferably be corrected in the source repository so that the correction flows through future refreshes. \--- # Stage 2 — Merge repository data and expand by UoA ## Command ```cmd python scripts\\merge\_and\_expand\_uoa.py --merge-only ``` The `--merge-only` option is the current production mode. The public version does not contain the superseded OpenAlex/Scopus enrichment pathway that was previously present in this script. ## Inputs ```text input/kar\_exports/outputs/ input/kar\_exports/authors/ input/staff\_uoa/ ``` ## External services None. ## Processing The script: * combines repository output exports * combines and deduplicates author information * uses institutional email addresses to match authors to UoAs * associates outputs with the relevant UoA(s) * expands outputs associated with multiple UoAs * retains outputs for which no UoA match is available ## Output ```text output/stage3\_uoa\_expanded.csv ``` ## QA after Stage 2 Check: * number of source output rows * number of deduplicated author rows * number of UoA-expanded rows * number of blank-UoA rows * number of UoAs represented Outputs can legitimately appear more than once after UoA expansion. The KDR version avoids the redundant `.1` duplicate columns found in earlier production outputs. \--- # Stage 3 — Merge impact enrichment ## Command ```cmd python scripts\\merge\_impact\_data.py ``` ## Inputs ```text output/stage3\_uoa\_expanded.csv input/impact\_data/REPORT\_publications.csv ``` ## External services None. ## Processing The Stage 3 UoA-expanded dataset is left-joined to the impact-enrichment information. The merge is designed to retain every Stage 3 row even where no external enrichment is available. ## Output ```text output/stage4\_impact\_merged.csv ``` ## QA after Stage 3 Confirm that: * the Stage 4 row count equals the Stage 3 row count * expected impact fields have been added * no unexpected duplicate `.1` columns have appeared \--- # Stage 4 — OpenAlex and Scopus metrics enrichment ## Command ```cmd python scripts\\run\_metrics\_pipeline.py --input-file output\\stage4\_impact\_merged.csv ``` ## Input ```text output/stage4\_impact\_merged.csv ``` ## External services * OpenAlex * Scopus This is an API-heavy stage and may take several hours. ## Processing The script retrieves and derives article-level and journal-level metrics including: ### OpenAlex * citation count * field-weighted citation impact (FWCI) * citation percentile * top 1% indicator * top 10% indicator * citations by year * references cited * OpenAlex record link ### Scopus * citation count * document type * Scopus EID * Scopus record link * journal CiteScore * CiteScore year * SNIP * SJR The script checkpoints the final output during long runs and maintains separate OpenAlex and Scopus caches. Successful cached OpenAlex responses are reused, while previously unsuccessful OpenAlex lookups are retried on subsequent runs so that a temporary API or lookup failure does not become a permanent non-match. ## Outputs ```text output/final\_metrics\_report.csv output/final\_metrics\_report.xlsx output/uoa\_splits/ ``` ## QA after Stage 4 Confirm: * final row count equals the Stage 4 input row count * expected OpenAlex and Scopus metric columns are present * no unexpected `.1` columns are present * citation-year fields cover the intended reporting period * UoA split files have been generated It is also useful to examine external-source coverage. OpenAlex and Scopus have different coverage. A record found in one service but not the other should not automatically be interpreted as a repository metadata problem. ### Validation example In the corrected release validation dataset, 7,685 unique KAR records contained a DOI-bearing identifier. At publication level, using KAR ID as the unit of analysis: 7,474 matched OpenAlex (97.3%) 6,695 matched Scopus (87.1%) 6,559 matched both 915 matched OpenAlex only 136 matched Scopus only 75 matched neither These figures describe successful external-source retrieval for this institutional validation dataset and should not be interpreted as general measures of OpenAlex or Scopus database coverage. Post-validation QA demonstrated that an unsuccessful DOI lookup can reflect API or identifier-resolution behaviour as well as genuine non-coverage. \--- # Stage 5 — Build UoA reports ## Command ```cmd python scripts\\build\_uoa\_reports.py ``` ## Input ```text output/uoa\_splits/ ``` ## External services None. The international collaboration map uses Natural Earth geographic data accessed by the reporting code. ## Processing For each UoA split, the script creates a report package containing formatted data, summary statistics and visualisations. Depending on available data, outputs can include: * publications by year * open access status * international collaboration * international collaboration map * FWCI distribution * citation distribution * citation percentile summaries * subject areas * cited journals * citing journals * reciprocal journals * funders * document types * internal collaborations ## Output ```text output/uoa\_reports/ ``` Each UoA receives its own report directory. ## QA after Stage 5 Check that: * the number of report folders corresponds to the number of UoA split groups * Excel workbooks have been created * charts have been created * a small selection of reports opens correctly Not every UoA will necessarily contain every chart because some visualisations require suitable underlying data. \--- # Stage 6 — Build School splits ## Command ```cmd python scripts\\build\_school\_splits.py ``` ## Input ```text output/final\_metrics\_report.csv ``` ## External services None. ## Processing The script interprets the institutional organisational-unit information and creates a separate dataset for each School-level grouping. School reporting is non-exclusive: where an output is associated with multiple Schools, the record can appear in multiple School datasets. ## Outputs ```text output/school\_splits/final\_metrics\_school\_.csv output/school\_splits/school\_split\_summary.csv ``` ## QA after Stage 6 Check: * the expected School groups are present * `school\_split\_summary.csv` contains one row per generated School split * the total number of split rows can exceed the institutional dataset because School membership is non-exclusive \--- # Stage 7 — Build School reports ## Command ```cmd python scripts\\build\_school\_reports.py ``` ## Input ```text output/school\_splits/ ``` ## External services None. As with UoA reporting, the international collaboration map uses Natural Earth geographic data. ## Processing The script generates School-level reporting packages using the same broad reporting approach as the UoA reports. ## Output ```text output/school\_reports/ ``` ## QA after Stage 7 Check: * the number of School report folders corresponds to the School split files * Excel workbooks are present * charts have been generated * selected workbooks and charts open correctly \--- # Stage 8 — University report and Power BI dataset ## Command ```cmd python scripts\\build\_university\_report\_dates.py ``` ## Input Normally: ```text output/final\_metrics\_report.csv ``` ## External services None. ## Processing The script: * creates the University-level summary report * creates University-level charts * normalises the available repository dates * creates a Power BI-ready dataset The date priority is: 1. Date Published 2. Date Published Online 3. Date Accepted 4. Date Submitted The Power BI dataset includes: ```text Best\_Date\_Source Best\_Date\_Raw Best\_Date Best\_Month\_Start Best\_Year Best\_Month Best\_Date\_Precision ``` ## Outputs Under: ```text output/university\_report/ ``` including: ```text powerbi\_dataset.csv summary.xlsx ``` and University-level visualisations. ## QA after Stage 8 Confirm: * `powerbi\_dataset.csv` exists * its row count matches `final\_metrics\_report.csv` * the expected `Best\_Date...` fields are present * no unexpected duplicate `.1` columns are present * University summary and chart outputs have been generated The Power BI-ready CSV is the final data output of the Python pipeline. Configuration of the interactive **Research Publication and Metrics Dashboard** is documented separately in `POWER\_BI.md`. \--- # Validation results for this release The complete eight-stage workflow was validated end-to-end using a fresh University of Kent dataset. Key counts from the validation run were: ```text KAR output rows loaded at Stage 2: 10,044 Author rows after deduplication: 14,898 Stage 3 output/UoA rows: 10,403 Stage 4 merged rows: 10,403 Final metrics rows: 10,403 Power BI dataset rows: 10,403 DOI-bearing final rows: 8,127 ``` Reporting outputs included: ```text UoA groups: 27 UoA report folders: 27 UoA Excel workbooks: 54 UoA charts: 369 School splits: 15 School report folders: 15 School Excel workbooks: 30 School charts: 208 ``` These counts describe the validation dataset and should not be treated as fixed expectations for future refreshes. # Recommended refresh practice For routine use: 1. archive any previous reporting snapshot that needs to be retained 2. replace the repository output and author exports 3. update the UoA lookup where required 4. confirm `.env` configuration 5. decide whether external API caches should be reused or reset 6. run each stage in order 7. perform QA checks before progressing past the major enrichment stages 8. refresh the Power BI report from the newly generated `powerbi\_dataset.csv` Because OpenAlex and Scopus metrics change over time, a reporting refresh intended to provide a current metrics snapshot may require fresh API retrieval rather than relying entirely on older cached metric values. # Failure handling and data quality External enrichment failures should be reviewed separately from repository data quality. A failed OpenAlex or Scopus match can reflect: * a malformed DOI * a repository-specific identifier * a publication outside the source's coverage * a metadata mismatch * an API timeout or other external-service issue Where a repository DOI is clearly malformed, correcting the source repository is preferable to permanently adding institution-specific corrections to the enrichment code. Where the repository metadata are considered correct, they should not be changed solely to force a match in an external service. # Stopping and resuming The API-heavy stages maintain processing state and/or caches to support long-running work. If a run is interrupted, inspect the relevant cache, temporary files and output before restarting. Avoid deleting processing state until it is clear that a completely fresh run is required. For a new reporting period, retaining old caches may reduce API use but can also retain older metric values. The appropriate approach depends on whether the goal is reproducibility of a previous snapshot or creation of a current metrics snapshot.