# Known Limitations ## Research Publication and Metrics Reporting Pipeline This document describes known methodological, technical and operational limitations of the **Research Publication and Metrics Reporting Pipeline**. The pipeline was validated end-to-end using live University of Kent repository data, OpenAlex and Scopus. The limitations below do not prevent the workflow from operating successfully, but they should be considered when interpreting outputs or adapting the software for another institution. ## Institution-specific inputs The workflow was developed around exports from Kent Academic Repository (KAR), which is based on EPrints, together with University of Kent organisational and REF Unit of Assessment (UoA) data. The scripts therefore assume the presence of particular repository fields and institutional structures. Other institutions should expect to map or adapt: * repository field names * author information * organisational-unit structures * School or Faculty structures * UoA or equivalent assessment classifications * reporting labels * Power BI configuration The deposited software is intended as an adaptable model rather than a directly transferable institutional reporting system. ## Institutional data are not included The KDR package does not include the University of Kent source data used to run or validate the pipeline. This includes: * KAR publication exports * KAR author exports * staff/UoA lookup data * generated metrics datasets * generated School and UoA reports * private API credentials Users therefore cannot reproduce the University of Kent reporting dataset directly from the deposited software alone. Reproducibility in this context refers to the documented processing workflow and its adaptation to suitable institutional inputs. ## External-source coverage and matching OpenAlex and Scopus do not provide identical coverage, and successful enrichment also depends on identifier matching and external API behaviour. In the corrected validation dataset, 7,685 unique KAR records contained a DOI-bearing identifier. At publication level, using KAR ID as the unit of analysis: 7,474 (97.3%) matched OpenAlex 6,695 (87.1%) matched Scopus 6,559 matched both 915 matched OpenAlex only 136 matched Scopus only 75 matched neither These figures describe successful retrieval for this institutional validation dataset rather than general database coverage. Manual investigation of apparent non-matches demonstrated that a failed DOI lookup does not necessarily mean that a publication is absent from an external source. Cases included publications subsequently retrievable by DOI, records present in OpenAlex where DOI-based retrieval behaved inconsistently, and malformed or non-standard identifiers in repository metadata. External-source non-matches should therefore be treated as enrichment or matching outcomes requiring interpretation rather than automatically as evidence of repository error or database non-coverage. ## External services can be unavailable or slow The API-dependent stages rely on services outside the control of the pipeline. During validation, OpenAlex requests occasionally returned network read timeouts or gateway/query timeouts. The workflow continued processing, but external-service responsiveness can increase runtime or leave some records without enrichment. During post-validation QA, it was identified that unsuccessful OpenAlex lookups stored as null cache entries could be reused on subsequent metrics runs, preventing those records from being retried. The metrics-stage cache handling was corrected so that successful OpenAlex responses remain cached while unsuccessful lookups are retried on subsequent runs. OpenAlex and Scopus may change: * API behaviour * metadata * coverage * rate limits * authentication requirements * service availability Future changes to either service may require corresponding changes to the pipeline. ## Long-running API stages The following stages can take several hours for a large institutional dataset: ```text run\_impact\_enrichment.py run\_metrics\_pipeline.py ``` Runtime depends on: * number of records * number of DOI-bearing records * API response times * rate limiting * additional cited/citing-work requests * whether caches are reused * local/network performance For long-running work, the computer should ideally remain awake and connected to the network. The scripts maintain processing state and/or caches, but uninterrupted execution is preferable for a completely fresh metrics snapshot. ## Metrics are time-dependent Citation counts, FWCI, citation percentiles, journal metrics and related indicators can change over time. A pipeline run therefore represents a **snapshot at the time of retrieval**, not a permanently reproducible metric value. Reusing old caches can reduce API calls but may also preserve older metric values. Users should decide whether a refresh is intended to: * reproduce a previous processing state, or * retrieve a current metrics snapshot before deciding whether caches should be retained. ## DOI matching and DOI quality DOI matching is important to external enrichment. Records can fail to match because: * no DOI is available * the DOI contains a source-data formatting error * the identifier is repository-specific * the publication is not covered by the external service * repository and external metadata differ * an external API request fails During validation, failed OpenAlex matches were manually reviewed. Fifteen records contained obvious DOI metadata problems and were corrected in KAR. Other non-matches included legitimate institutional identifiers and records for which the repository metadata were not considered erroneous. The workflow therefore does not assume that an external non-match means the repository record should be changed. ## DOI normalisation is deliberately conservative The current DOI-cleaning logic handles common DOI prefixes but does not attempt to repair every malformed DOI automatically. Examples of source-data issues observed during validation included: * embedded spaces * trailing punctuation * encoded characters * appended URL fragments * additional descriptive text Aggressive automatic repair could incorrectly alter valid or unusual identifiers. The preferred approach is therefore to review obvious data-quality problems and correct them in the source repository where appropriate. A future version could provide more detailed DOI validation or flagging without automatically rewriting the source identifier. ## Failed DOI reporting is basic Stage 1 records unsuccessful DOI processing in: ```text cache/impact\_tracking/failed\_dois.csv ``` In the current release this is a simple diagnostic file. Known limitations include: * the file is written without a formal header row * KAR ID is not included * publication title is not included * failures are not automatically classified into likely data-quality issues, coverage gaps and technical failures During validation, a separate QA workbook was created by matching failed DOI values back to the repository exports. A future version could generate a richer failed-DOI QA report automatically, including KAR ID, title, DOI, failure reason and a review category. ## Stage 4 does not create a dedicated missing-metrics report `run\_metrics\_pipeline.py` leaves unavailable OpenAlex or Scopus metrics blank. The current release does not automatically create a separate report showing: * OpenAlex-only matches * Scopus-only matches * records matched by both * records matched by neither These comparisons were performed during validation and were useful for understanding source coverage. A future version could generate this coverage report automatically without changing the underlying metrics calculations. ## Date parsing warning in Stage 4 During validation, `run\_metrics\_pipeline.py` produced a pandas warning when some ISO-formatted dates were parsed while `dayfirst=True` was specified. The warning did not stop processing. The validation run: * identified the intended reporting years as 2021–2026 * completed all 10,403 rows * generated the expected citation-window values * successfully produced the final reporting and Power BI datasets The current release retains the fully tested code rather than introducing a last-minute parsing change after end-to-end validation. A future version could make the date parsing more explicit and remove the warning. ## Records without DOI enrichment Repository records without a DOI are retained by the institutional workflow, but they cannot receive DOI-based OpenAlex or Scopus enrichment. This means metric coverage will always be lower than the total number of repository records where the dataset contains outputs without DOI identifiers. The absence of external metrics should therefore not be interpreted as the absence of an institutional research output. ## Scopus requires licensed access Scopus is not an open dependency. Users must have: * appropriate Scopus access * a valid API key * permission to use the relevant API services under their institutional arrangements The KDR package does not redistribute Scopus data or credentials. Institutions without Scopus access would need to adapt the metrics stage and downstream reporting accordingly. ## OpenAlex and Scopus metrics are not interchangeable Where both services provide citation information, their values may differ. Differences can result from: * source coverage * record matching * document indexing * citation-source coverage * metadata differences * update timing The workflow retains both sources rather than assuming that one should reproduce the other. The validation exercise identified differences in successful retrieval and metric availability between OpenAlex and Scopus. Post-validation investigation also demonstrated that apparent source-specific matches can reflect API timing, identifier resolution and matching behaviour as well as underlying database coverage. Direct comparison of the two sources therefore requires careful distinction between database coverage and retrieval or matching outcomes. ## School reporting is non-exclusive An output can be associated with more than one School. The School-splitting stage therefore duplicates a record into each relevant School reporting dataset where appropriate. As a result: * School totals should not be summed to calculate the institutional total * the total number of rows across School splits can exceed the number of rows in the institutional dataset This is intentional behaviour. ## UoA reporting is non-exclusive Outputs can also be associated with more than one REF Unit of Assessment. The UoA expansion stage creates one row per relevant output/UoA association. Outputs for which no UoA can be matched are retained with a blank UoA rather than discarded. As with Schools, UoA totals should not automatically be summed to calculate a University total. ## UoA matching depends on institutional email data The University of Kent workflow associates authors with UoAs using institutional email addresses and a staff/UoA lookup. Records may remain without a UoA where: * no institutional author email is available * the email is absent from the lookup * the organisational data are incomplete * the output cannot be associated with a current lookup record Other institutions may require a different matching strategy. ## Power BI is not fully generated by the Python pipeline The Python workflow creates: ```text output/university\_report/powerbi\_dataset.csv ``` but does not automatically generate the complete **Research Publication and Metrics Dashboard**. The Power BI presentation layer includes institution-specific: * report pages * slicers * visualisations * visual interactions * display labels * publishing configuration * SharePoint presentation These elements require separate configuration. See `POWER\_BI.md`. ## Power BI depends on schema stability Power BI visuals and transformations can depend directly on column names. Renaming or removing fields in the Python outputs can therefore break the dashboard even where the underlying data remain valid. Column-name changes should be treated as potentially breaking changes and tested against the Power BI report. During the KDR clean-up, six redundant `.1` duplicate columns were removed after they were verified to contain values identical to their corresponding primary fields. ## Short School labels are presentation-specific The University of Kent Power BI dashboard uses shortened School labels because full institutional names did not fit comfortably within the dashboard's compact slicers. These short labels are a presentation choice rather than a universal organisational model. Institutions adapting the dashboard should create display labels appropriate to their own structure. ## Geographic visualisations require external geographic data The UoA, School and University reporting scripts can create international collaboration maps using Natural Earth geographic data accessed at runtime. These particular visualisations therefore depend on: * network availability * continued availability of the external geographic dataset * compatibility with GeoPandas Failure to retrieve the geographic data may affect map creation without necessarily invalidating the underlying reporting dataset. ## Report visual availability depends on data Not every School or UoA will necessarily produce every possible chart. Some visualisations are only generated where appropriate underlying data are available. A difference in the number of charts between reporting groups is therefore not automatically an error. ## Generated reports are snapshots The generated Excel reports, charts and Power BI dataset reflect: * the repository exports used for that run * the institutional lookup data used for that run * external metadata available at the time * external metric values available at the time For longitudinal or audit purposes, institutions may wish to archive dated reporting snapshots before running a subsequent refresh. ## Tested release versus future enhancements The scripts included in this release were tested end-to-end as a complete workflow before deposit. Several possible improvements were identified during validation, including: * richer failed-DOI reporting * additional DOI validation * automated OpenAlex/Scopus coverage reporting * more explicit date parsing These have been documented rather than added after the final validation run. This preserves a clear relationship between the deposited release and the code that completed the successful end-to-end test. Future versions may incorporate these enhancements with appropriate testing.