# WEEKLY_WORKFLOW.md # Weekly Operational Workflow ## Purpose The OpenAlex Publication Discovery and Repository Enrichment Toolkit supports the weekly identification of new University research outputs and the maintenance of existing repository metadata. The toolkit compares newly identified OpenAlex records against a current export of the institutional repository and classifies each record into one of four categories: * New publication * Existing publication with additional metadata available * Existing publication (no action required) * Manual review The toolkit does **not** make changes directly to the repository. Repository staff remain responsible for reviewing and importing records. --- # Before you begin Ensure the following are available: * An OpenAlex API key configured in the toolkit (recommended) * The current version of the OpenAlex Publication Discovery and Repository Enrichment Toolkit. * A current export of the institutional repository. * Python and any required software dependencies. * Internet access for querying the OpenAlex API. --- # Step 1 – Export the repository Run an administrative search in the institutional repository against the **Live Archive**. Export the results as: **Multiline CSV (Staff)** Save the export as: `kar_export.csv` Place this file in the same folder as the toolkit. Normally only the **Live Archive** should be exported. Repository queues such as "Under Review" should only be included if there is a specific operational reason for doing so. --- # Step 2 – Run the toolkit Run: `openalex_repository_toolkit.py` The toolkit will: * query OpenAlex for newly published institutional outputs * compare those outputs with the repository export * identify duplicates using DOI and title matching * identify existing repository records that may be enriched * generate RIS files for new publications * produce reports for repository staff --- # Step 3 – Review the output The toolkit creates a structured set of output folders. ## new Contains RIS files and supporting JSON for publications that do not already exist in the repository. These records should be reviewed before import. --- ## books Contains books identified by OpenAlex. These may require different processing from journal articles depending on local repository workflows. --- ## book_chapters Contains book chapters identified by OpenAlex. --- ## datasets Contains research datasets identified by OpenAlex. These are exported separately because institutions may manage research datasets through a dedicated data repository rather than an institutional publications repository. --- ## enrichment Contains reports identifying existing repository records that may benefit from additional metadata available through OpenAlex. Typical metadata additions include: * DOI * Volume * Issue * Page numbers * Publication information Repository staff should review these suggested updates before making changes. --- ## duplicates Contains records already represented in the repository where no additional metadata has been identified. No further action is normally required. --- ## manual_review Contains records requiring staff judgement. Examples include: * publication lifecycle events (such as preprint and published article) * conflicting DOI information * ambiguous title matches --- ## raw_openalex Contains copies of the OpenAlex records retrieved during the run. These files provide an audit trail and can assist with troubleshooting. --- # Log files ## openalex_to_kar.log A detailed record of the toolkit run. Includes: * OpenAlex queries * matching decisions * routing decisions * summary statistics --- ## last_openalex_results.json Stores the OpenAlex records retrieved during the current run. Useful when investigating individual records. --- ## no_doi_log.txt Lists OpenAlex records where no DOI was available. The toolkit attempts title matching where possible, but these records may require additional review. --- ## last_run.txt Stores the date of the previous successful run. The toolkit automatically applies a short overlap period to minimise the risk of missing newly indexed publications. --- # Recommended processing order Repository staff are recommended to work through the outputs in the following order: 1. Review the **manual_review** folder. 2. Review the **enrichment** report and update existing repository records where appropriate. 3. Review RIS files in the **new**, **books**, **book_chapters** and **datasets** folders. 4. Import appropriate RIS files into the repository. 5. Archive the outputs from the completed run. --- # Expected results A typical weekly run will produce a combination of: * genuinely new publications * existing records requiring metadata enrichment * duplicate records requiring no action * a small number of manual review cases This is expected behaviour and reflects the different stages of publication and metadata availability across scholarly communication. --- # Good practice * Keep each weekly run together in its own archive folder. * Retain the repository export used for that run. * Retain the toolkit log and reports for auditing purposes. * Record the toolkit version used for each run. * Review enrichment suggestions before applying changes to repository records. Archiving each run provides a complete audit trail and allows decisions to be revisited if required. --- # Known workflow considerations The toolkit compares OpenAlex records with a snapshot of the repository rather than the live repository itself. As a result: * records added to the repository after the export will not be reflected until the next run * publication metadata may improve over time as publishers update Crossref, DataCite or other metadata sources * some records without DOIs may require title matching and manual review These are normal characteristics of the workflow rather than software errors.