# Research Publication and Metrics Reporting Pipeline ## KDR metadata **Title:** Research Publication and Metrics Reporting Pipeline **Creator:** Ian Badger **Institution:** University of Kent **Division:** Open Research Team, Library Services, Student Life, University of Kent **Contact:** researchsupport@kent.ac.uk **ORCiD:** 0009-0000-0124-2501 **Source of funding:** University of Kent **Licence:** CC BY 4.0 **Development period:** 27 March 2026 – 25 August 2026 **Related works / systems:** * Kent Academic Repository (KAR), based on EPrints * OpenAlex * Scopus ## Overview The Research Publication and Metrics Reporting Pipeline is a research software workflow developed at the University of Kent to transform institutional repository data into enriched research metrics and reporting outputs. The pipeline starts with exports from Kent Academic Repository (KAR) and associated institutional data. These records are enriched using OpenAlex and Scopus and then processed into reporting datasets for University-level, School-level and REF Unit of Assessment (UoA) analysis. The workflow produces the data used to generate the **Research Publication and Metrics Reports** and a Power BI-ready dataset used by the **Research Publication and Metrics Dashboard**. The pipeline has been developed as an adaptable and reproducible model rather than as a direct, institution-independent reproduction of the University of Kent reporting environment. The KAR exports, organisational structures and reporting requirements are institution-specific. Other institutions adopting the workflow would therefore need to map their own repository and organisational data to the expected inputs and adapt the reporting configuration. ## Why was this pipeline developed? The pipeline was developed to provide a repeatable way of combining institutional repository information with external research information sources to support research reporting and research impact analysis. KAR provides the institutional record of research outputs, including publication information, organisational affiliation and open access information. OpenAlex and Scopus provide complementary publication, citation, bibliographic and journal-level information. The workflow was developed to bring these sources together into a common dataset and to support reporting at several organisational levels without maintaining separate data-processing pipelines for each reporting view. The resulting workflow supports both institutional-level analysis and more detailed reporting organised by School and REF Unit of Assessment. ## Data sources and inputs ### Kent Academic Repository KAR provides the primary institutional publication records. The pipeline uses exports containing publication information and author information. The KAR data are institution-specific and are not included in this KDR software deposit. ### Institutional author and UoA information Additional institutional files provide author-level information and the mapping between institutional email addresses and REF Units of Assessment. These files are used to associate outputs with the relevant UoA(s). ### OpenAlex OpenAlex is used for publication enrichment and bibliometric information, including publication and subject information, collaborating institutions and countries, international collaboration, SDG classifications, cited and citing journals, citation counts, field-weighted citation impact, citation percentiles, top 1% and top 10% indicators, references cited and citations by year. OpenAlex access requires a user-supplied API key and a contact email configured through environment variables. ### Scopus Scopus is used as a complementary bibliometric source for citation counts, document type, Scopus identifier, journal CiteScore, CiteScore year, journal SNIP and journal SJR. Scopus is a licensed external service. Scopus data are used by the workflow but are not redistributed as part of this KDR software package. ## Workflow The current production workflow consists of eight stages. ### 1\. OpenAlex enrichment `run\_impact\_enrichment.py` KAR output exports are processed and matched to OpenAlex records using DOI information where available. The script produces: * `REPORT\_publications.csv` * `REPORT\_funding.csv` The stage also maintains processing and failure information to support long-running and resumable enrichment. ### 2\. UoA expansion `merge\_and\_expand\_uoa.py --merge-only` KAR outputs are combined with author information and the institutional staff-to-UoA lookup. Outputs associated with multiple UoAs are expanded so that an output can appear in each relevant UoA. Outputs without a matched UoA are retained with a blank UoA rather than being discarded. The current KDR version retains the merge functionality used in production and does not include the superseded OpenAlex/Scopus enrichment pathway that was previously present in this script. ### 3\. Impact merge `merge\_impact\_data.py` The UoA-expanded dataset is combined with the OpenAlex impact-enrichment output to create: * `stage4\_impact\_merged.csv` This stage preserves the complete Stage 2 record set and adds the available impact and enrichment fields. ### 4\. Metrics enrichment `run\_metrics\_pipeline.py` The Stage 4 dataset is enriched using both OpenAlex and Scopus. The resulting dataset is: * `final\_metrics\_report.csv` The script also creates: * an Excel companion file * UoA-specific CSV and Excel splits ### 5\. UoA reporting `build\_uoa\_reports.py` The UoA split datasets are transformed into report packages containing summary statistics, formatted Excel workbooks, charts, collaboration analyses, subject, funder, journal and citation summaries. ### 6\. School splitting `build\_school\_splits.py` The final metrics dataset is split by School using the institutional organisational structure. School reporting is non-exclusive. Where an output is associated with more than one School, it can appear in more than one School split. ### 7\. School reporting `build\_school\_reports.py` The School splits are transformed into School-level report packages containing summary statistics, formatted Excel workbooks, charts, collaboration analyses, subject, funder, journal and citation summaries. ### 8\. University and Power BI preparation `build\_university\_report\_dates.py` The final University-level report is generated and a Power BI-ready dataset is created: * `powerbi\_dataset.csv` This stage also creates standardised date fields for reporting and Power BI use, including: * `Best\_Date\_Source` * `Best\_Date\_Raw` * `Best\_Date` * `Best\_Month\_Start` * `Best\_Year` * `Best\_Month` * `Best\_Date\_Precision` The preferred date sequence is: 1. Date Published 2. Date Published Online 3. Date Accepted 4. Date Submitted ## Research Publication and Metrics Reports The reporting stages produce report packages organised by: * REF Unit of Assessment * School * University The report outputs include publication and open access information from KAR, bibliometric information from OpenAlex and Scopus, collaboration analysis, subject information, funder information and summary visualisations. The School reporting view is non-exclusive. School totals should therefore not be summed to derive an institutional total. Similarly, outputs can be associated with multiple UoAs and may therefore appear in more than one UoA report. ## Research Publication and Metrics Dashboard The final University stage creates `powerbi\_dataset.csv`, which is used as the data source for the **Research Publication and Metrics Dashboard** in Microsoft Power BI. The Python pipeline does not automatically recreate the complete Power BI report layout. The Power BI implementation requires a separate configuration step, including importing the generated dataset, configuring fields and relationships as required, creating slicers and visualisations, configuring report pages, and publishing and sharing the completed report. Power BI configuration is documented separately in `POWER\_BI.md`. ## Reproducibility and adaptation The workflow is designed to provide a reproducible and adaptable model for institutional research reporting. The deposited code does not contain the University of Kent's institutional data, API credentials or private environment configuration. Users adopting the workflow will need: * a repository export containing the fields required by the pipeline * author information suitable for institutional matching * an organisational/UoA lookup where relevant * access to OpenAlex * access to Scopus where Scopus metrics are required * a Python environment with the dependencies listed in `requirements.txt` The `.env.example` file documents the required environment variables: ```text OPENALEX\_API\_KEY= OPENALEX\_EMAIL= SCOPUS\_API\_KEY= ``` The real `.env` file containing private credentials should not be deposited. ## External-source coverage and limitations OpenAlex and Scopus provide complementary but non-identical coverage. External-source matching is also dependent on the availability and responsiveness of the external APIs and on the quality and form of identifiers supplied by the repository. In the corrected University of Kent validation run, 7,685 unique KAR records contained a DOI-bearing identifier. At publication level, using KAR ID as the unit of analysis: 7,474 (97.3%) matched OpenAlex 6,695 (87.1%) matched Scopus 6,559 matched both sources 915 matched OpenAlex only 136 matched Scopus only 75 matched neither source These figures should be interpreted as a validation of the pipeline's ability to retrieve records from the two external services for this institutional dataset, rather than as a general measure of database coverage. During post-validation QA, some apparent OpenAlex non-matches were found to result from unsuccessful API lookups that had previously been retained in the OpenAlex cache. The metrics-stage cache handling was subsequently corrected so that successful cached OpenAlex responses continue to be reused, while previously unsuccessful lookups are retried on subsequent runs. Further manual review of apparent non-matches also identified malformed or non-standard repository identifiers and cases where an OpenAlex work existed but DOI-based retrieval did not behave consistently. Consequently, failure to match a record in one external service should not automatically be interpreted as either repository metadata error or absence of the publication from the external source.DOI quality and enrichment review During validation of this release, failed OpenAlex DOI matches were reviewed against the KAR records. Some failures were caused by obvious DOI formatting problems in the source metadata, including spaces, punctuation and additional URL or text fragments. Fifteen records were corrected in KAR during the review. Other failures represented legitimate repository-specific identifiers, publications not covered by OpenAlex, or cases where OpenAlex and the repository metadata did not match sufficiently for enrichment. These records were not altered merely to obtain an OpenAlex match. The final pipeline therefore treats external enrichment as an additional information layer rather than as a validation authority for the underlying repository record. ## Validation The release candidate was tested end-to-end using a fresh University of Kent dataset. The validation run processed: * 10,044 KAR output rows * 14,898 author rows after author deduplication * 10,403 output/UoA rows after UoA expansion * 10,403 final metrics rows * 8,127 DOI-bearing final records The complete workflow successfully produced: * OpenAlex-enriched publication and funding outputs * UoA-expanded data * impact-merged data * OpenAlex and Scopus metric outputs * UoA report packages * School report packages * a University report * a Power BI-ready dataset The final Power BI dataset contained 10,403 rows and retained the standardised date fields described above. During validation, the pipeline also demonstrated that the reporting stages could process UoA and School outputs without reproducing the redundant `.1` columns present in earlier production outputs. The scripts were tested using live University of Kent repository data and live OpenAlex and Scopus services during development and validation. ## Known limitations ### Institution-specific repository structure The current pipeline was developed around KAR/EPrints exports and University of Kent organisational structures. Other repositories will require configuration and, potentially, code changes. ### External API dependence OpenAlex and Scopus are external services. Coverage, response times, API availability and metadata can change independently of the pipeline. ### DOI dependence DOI matching is an important part of the external enrichment process. Records without a DOI, records with malformed DOI strings and repository-specific identifiers may not receive external enrichment. ### Scopus licensing Scopus is a licensed service. Users must have appropriate access to use the Scopus stages of the workflow. ### Power BI configuration The Python pipeline creates a Power BI-ready dataset but does not reproduce the complete visual report automatically. The final dashboard configuration is institution-specific. ### School and UoA counting School and UoA reporting is not necessarily mutually exclusive. Outputs associated with more than one organisational unit can appear in multiple reporting groups. ## Package contents The KDR software package contains: * `run\_impact\_enrichment.py` * `merge\_and\_expand\_uoa.py` * `merge\_impact\_data.py` * `run\_metrics\_pipeline.py` * `build\_uoa\_reports.py` * `build\_school\_splits.py` * `build\_school\_reports.py` * `build\_university\_report\_dates.py` * `.env.example` * `requirements.txt` * `INSTALLATION.md` * `PIPELINE\_WORKFLOW.md` * `POWER\_BI.md` * `KNOWN\_LIMITATIONS.md` No institutional input data, generated research metrics datasets, API credentials or private configuration files are included in the deposit. ## Intended users The workflow is intended for: * institutional research teams * research support and research analytics teams * institutional repository managers * scholarly communications teams * libraries and research offices * institutions wishing to combine repository metadata with open and licensed research information sources Although developed and tested at the University of Kent, the workflow is intended as an adaptable model rather than a directly transferable institutional reporting system. ## Versioning This release represents the version of the workflow used for the associated Research Publication and Metrics project and its accompanying dissemination materials. Future releases may update external-source handling, DOI validation, failure reporting, date parsing, reporting configuration or Power BI implementation.