# Installation

## Research Publication and Metrics Reporting Pipeline

This guide describes the installation and project structure used for the **Research Publication and Metrics Reporting Pipeline**.

The pipeline was validated end-to-end on Windows using **Python 3.13.7**. Earlier Python versions have not been formally tested as part of this release.

The deposited package does not include University of Kent institutional data, generated reports, API credentials or a private `.env` file.

## Requirements

The workflow requires:

- Python 3
- access to a command-line environment such as Windows Command Prompt or PowerShell
- an OpenAlex API key
- a contact email address for OpenAlex API requests
- a Scopus API key and appropriate institutional/licensed access where Scopus enrichment is required
- repository and institutional input files in the expected structure

The Python dependencies are listed in `requirements.txt`:

```text
pandas
requests
python-dotenv
openpyxl
xlrd
matplotlib
geopandas
```

`xlrd` is included because the pipeline supports legacy `.xls` input files as well as `.xlsx` and `.csv` files.

## Recommended project structure

The scripts derive the project root from their own location. The recommended structure is therefore:

```text
Research Publication and Metrics/
├── .env
├── .env.example
├── requirements.txt
├── scripts/
│   ├── run_impact_enrichment.py
│   ├── merge_and_expand_uoa.py
│   ├── merge_impact_data.py
│   ├── run_metrics_pipeline.py
│   ├── build_uoa_reports.py
│   ├── build_school_splits.py
│   ├── build_school_reports.py
│   └── build_university_report_dates.py
├── input/
│   ├── kar_exports/
│   │   ├── outputs/
│   │   └── authors/
│   ├── staff_uoa/
│   └── impact_data/
├── output/
└── cache/
```

The `output` and `cache` directories can be created by the workflow as required.

The real `.env` file is private and should **not** be included in a public deposit.

## 1. Install Python

Install a current supported version of Python 3.

The release was validated using:

```text
Python 3.13.7
```

To check the installed version:

```cmd
python --version
```

## 2. Install the Python dependencies

Open Command Prompt or PowerShell in the project root — the directory containing `requirements.txt` — and run:

```cmd
python -m pip install -r requirements.txt
```

If the dependencies are already installed, this command will normally leave the existing compatible installations in place.

### Optional dependency check

The following command was used during validation to confirm that all required packages could be imported:

```cmd
python -c "import pandas, requests, dotenv, openpyxl, xlrd, matplotlib, geopandas; print('All dependencies OK')"
```

A successful check returns:

```text
All dependencies OK
```

If a package is reported as missing, install the missing dependency before running the pipeline.

## 3. Configure API credentials

Copy `.env.example` to a new file named:

```text
.env
```

Keep `.env` in the **project root**, not in the `scripts` directory.

The file should contain:

```text
OPENALEX_API_KEY=your_openalex_api_key
OPENALEX_EMAIL=your_contact_email
SCOPUS_API_KEY=your_scopus_api_key
```

Do not add quotation marks unless required by the value itself.

Do not commit, publish or deposit the real `.env` file.

The public `.env.example` should contain blank values only:

```text
OPENALEX_API_KEY=
OPENALEX_EMAIL=
SCOPUS_API_KEY=
```

## 4. Prepare the input directories

### Repository output exports

Place repository publication/output exports in:

```text
input/kar_exports/outputs/
```

The University of Kent implementation uses KAR/EPrints exports. The scripts support `.csv`, `.xlsx` and `.xls` input formats where applicable.

The output exports are expected to contain a `KAR ID` field and the repository metadata required by the reporting workflow, including DOI and publication information where available.

### Repository author exports

Place author exports in:

```text
input/kar_exports/authors/
```

The University of Kent implementation uses these files to associate KAR records with Kent authors and institutional email addresses.

### Staff/UoA lookup

Place the institutional REF Unit of Assessment lookup in:

```text
input/staff_uoa/
```

The current workflow identifies the email and Unit of Assessment fields from the lookup and uses institutional email addresses to associate outputs with UoAs.

Institutions adapting the pipeline will need to provide an equivalent organisational or reporting lookup appropriate to their own structure.

### Impact data

The directory:

```text
input/impact_data/
```

is used for the OpenAlex enrichment outputs:

```text
REPORT_publications.csv
REPORT_funding.csv
```

These are generated by `run_impact_enrichment.py` and are subsequently used by the impact-merge stage.

## 5. Confirm the folder structure

Before running the pipeline, it can be useful to confirm that the required directories and private `.env` file are present.

From the project root:

```cmd
python -c "from pathlib import Path; p=Path('.'); print('Scripts:',(p/'scripts').exists()); print('Input:',(p/'input').exists()); print('KAR outputs:',(p/'input'/'kar_exports'/'outputs').exists()); print('Authors:',(p/'input'/'kar_exports'/'authors').exists()); print('UoA:',(p/'input'/'staff_uoa').exists()); print('Impact data:',(p/'input'/'impact_data').exists()); print('.env:',(p/'.env').exists())"
```

All values should normally return `True`.

## 6. Run scripts from the project root

The recommended approach is to open the command line in the project root and invoke scripts through the `scripts` directory.

For example:

```cmd
python scripts\run_impact_enrichment.py
```

If the command line is already inside the `scripts` directory, omit the additional `scripts\` prefix:

```cmd
python run_impact_enrichment.py
```

The scripts use their own filesystem location to identify the project root, so project-relative input, output and configuration paths are not dependent on a fixed `C:\...` installation location.

## 7. API-heavy stages

Two stages make substantial external API requests:

```text
run_impact_enrichment.py
run_metrics_pipeline.py
```

Runtime depends on the number of repository records, external API response times and the amount of enrichment requested.

For a large institutional dataset these stages can take several hours.

Long-running API stages should be allowed to complete without the computer entering sleep mode where possible.

The workflow uses cache/checkpoint information to support long-running processing, but an uninterrupted run is preferable when generating a completely fresh metrics snapshot.

## 8. Generated directories and files

During execution, the pipeline creates and uses files under:

```text
cache/
output/
input/impact_data/
```

These may include:

- OpenAlex processing state
- API caches
- failed DOI information
- intermediate Stage 3 and Stage 4 datasets
- the final metrics dataset
- UoA splits
- School splits
- UoA and School report packages
- University-level reports
- the Power BI-ready dataset

Generated institutional data are not part of the public KDR software package.

## 9. Running the complete pipeline

Installation only prepares the environment.

The eight-stage execution order, expected intermediate files and reporting outputs are documented separately in:

```text
PIPELINE_WORKFLOW.md
```

The final Power BI configuration is documented in:

```text
POWER_BI.md
```

Known methodological and technical limitations are documented in:

```text
KNOWN_LIMITATIONS.md
```

## Security and configuration notes

- Never include the real `.env` file in a public repository or KDR deposit.
- Never hard-code API keys into the Python scripts.
- Use `.env.example` only as a configuration template.
- Institutional repository exports may contain personal or organisational information and should be handled according to local data-management requirements.
- Scopus access is subject to the user's institutional licence and API permissions.

## Validation environment

The installation and folder structure described above were used for the successful end-to-end validation of this release.

Validation environment:

```text
Operating system: Windows
Python: 3.13.7
Configuration: project-relative filesystem structure
OpenAlex: live API
Scopus: live API
```

The validation run successfully progressed from repository exports through OpenAlex and Scopus enrichment, UoA and School reporting, University-level reporting and creation of the Power BI-ready dataset.