Rethinking Machine Learning in Clinical Transcriptomics: A Comprehensive Benchmarking Study

This repository serves as a reference for the preprocessing, training, and evaluation protocols, with scripts enabling fair and reproducible benchmarking.

Here we evaluated the task of patient stratification of cancer subtypes using transcriptomic data (RNAseq and Microarray) and different models (Random Forest (RF), Logistic Regression (LR), SVM, XGBoost, MLP, Sparse NN, VAE_CL, MLP_mixup, GNN, Spectral NN). Further, we applied strategies for mitigating the curse of dimensionality, including dimensionality reduction techniques and sample generation.

Abstract

Clinical transcriptomic datasets are increasingly used to train machine learning models, yet they are often treated uniformly despite substantial biological differences. Focusing on cancer subtype stratification, we introduce the concept of the complexity spiral: the tendency to prioritize increasingly sophisticated architectures over data-centric rigor. Using seven bulk transcriptomic cohorts, we benchmarked preprocessing and modeling strategies across the full pipeline, including normalization, batch correction, dimensionality reduction, and synthetic patient generation. We found that inadequate preprocessing and evaluation choices can artificially inflate performance, while greater model complexity rarely compensates for poor data quality. Our results highlight that it is essential to attend to the nature and limitations of the data, focusing on patient heterogeneity, label type, and high feature-to-sample ratios. In particular, we introduce a novel classification of biological label categories (molecular, histological, or clinical) that strongly influences task difficulty and achievable performance, with molecular labels consistently yielding higher accuracy than clinical outcomes. Further, deep learning models tend to overfit under low-sample regimes, while simpler tree-based methods, such as Random Forest and XGBoost, achieve more stable generalization across splits and provide robust baselines, often matching or outperforming more advanced approaches. Finally, we propose a decalogue of good practices and a ready-to-use code framework that, if adopted, will support researchers in developing translationally relevant and clinically reliable models.

Datasets

Breast Cancer Metastasis (microarray).
Breast Cancer Subtype and binarized version (RNASeq).
Multiple Myeloma (RNASeq).
Prostate Gleason (RNASeq).
Lower Grade Glioma (RNASeq).
Lung Histological (RNASeq, with and without ComBat batch correction).
Kidney Subtype (RNASeq, with and without ComBat batch correction).

Data preprocessed from source is available in data/raw. This is the input for the models. In data/preprocessed files are generated if not present by run.py and correspond to the data ready for the model (e.g., zscore preprocessed).

Installation & Environment Setup

Option 1: Docker (Recommended)

All required dependencies are containerized in a Docker image. Navigate to the Dockerfile location and build:

cd Dockers/Pytorch_cuda12_R4_4/
docker build -t phenotypic-prediction:latest .
docker run --gpus all -it -v /path/to/repo:/wdir phenotypic-prediction:latest

The Docker image includes:

Python packages: PyTorch 2.5.1 (CUDA 12.4), scikit-learn, pandas, xgboost, Lightning, torch-geometric, shap, and others
R packages: Bioconductor tools (limma, DESeq2, NetActivity, sva, etc.)
System dependencies: CUDA 12.4, cuDNN9, and development libraries

Option 2: Local Installation

If you prefer local installation, install the required packages from the Dockerfile. And install R packages as specified in the Dockerfile.

Quick Start

Here are some example commands to get started:

# Run a simple RF experiment on Breast Cancer data
python3 -u run.py -d BreastCancer_S -n zscore -m RF -p None -s 42 -Nf 5

# Run with synthetic data generation (VAE)
python3 -u run.py -d BreastCancer_S -n zscore -m MLP -p VAE -Mf 1.0 -s 42 -Nf 5

# Run transfer learning
python3 -u run_transfer_learning.py -d BreastCancer_S -n zscore -p None -s 42 -Nf 5

Project Structure

data/raw/: Raw datasets before preprocessing
data/preprocessed/: Preprocessed datasets ready for model training
code/models.py: Model implementations (sklearn and PyTorch Lightning)
code/utils.py: Training and grid search utility functions
code/preprocessing/: R scripts for dataset download and preprocessing
results/: Output folder where trained models and evaluation metrics are saved
- Each experiment creates a folder with the naming convention: n_<normalization>_p_<preprocessing>_s_<seed>_N_<folds>/
- Contains: scores.csv (evaluation metrics), model_weights.pth (trained model), model_config.json (hyperparameters)
logs/: Training logs and output files from experiments
Dockers/: Docker configuration for reproducible environment

How to run an experiment

python3 -u run.py -d <dataset> -n <normalization> -m <model> -p <preprocessing> -Mf <Mf> -s <seed> -Nf <number_folds> -Vs

Options:

<dataset>: BreastCancer_M, BreastCancer_S, BreastCancer_S_binarized, MultipleMyeloma, Prostate_Gleason, LowerGradeGlioma, Lung_Histological, Lung_Histological_No_ComBat, Kidney_Subtype, Kidney_Subtype_No_ComBat
<normalization>: none, zscore, minmax
<model>: Logistic_Regression, Random_Forest, SVM, XGBoost, MLP, SPARSE_NN, VAE_CL, SPARSE_VAE_CL, GNN, MLP_mixup, Spectral_NN
<preprocessing>: Preprocessing steps can be combined using underscore (_) separator. Available steps:
- Single methods: {'None', 'VAE', 'WGAN-GP', 'NetActivity', 'DEG'}
- Spectral embeddings: Pattern matching r'SPECTRAL-(?:SM|LM)-\d+-\d+$' (e.g., SPECTRAL-SM-20-50)
- Combined (e.g., VAE_NetActivity, WGAN-GP_DEG)
<Mf>: {1.0, 5.0, 10.0}. Multiplicative factor for synthetic data generation. Optional, only needed when using VAE or WGAN-GP preprocessing.
<seed>: int. Seed for reproducibility.
<number_folds>: int. Number of folds for cross-validation.
-Vs: Optional flag. When generating synthetic data, run validation analysis on synthetic data.
-gs / --gridsearch_result_file: Optional path to an existing grid-search results CSV. If provided, run.py skips the grid-search stage and loads that file instead.

How to run transfer learning

This script trains an MLP on PAN-CANCER data (excluding patients from the target dataset) and applies transfer learning to solve the target classification problem.

python3 -u run_transfer_learning.py -d <dataset> -n <normalization> -p <preprocessing> -s <seed> -Nf <number_folds>

Options:

<dataset>: BreastCancer_M, BreastCancer_S, BreastCancer_S_binarized, MultipleMyeloma, Prostate_Gleason, LowerGradeGlioma, Lung_Histological, Lung_Histological_No_ComBat, Kidney_Subtype, Kidney_Subtype_No_ComBat
<normalization>: none, zscore, minmax
<preprocessing>: {'None', 'NetActivity', 'DEG'}
<seed>: int. Seed for reproducibility.
<number_folds>: int. Number of folds for cross-validation.

Output Example

After running an experiment, the results folder will have the following structure:

results/
├── rnaseq/                          # Data type (rnaseq or microarray)
│   └── BreastCancer_S/              # Dataset name
│       ├── SVM/                     # Model name
│       │   ├── GridSearchs/         # Grid search results
│       │   │   └── n_zscore_p_None_s_42_N_10.csv
│       │   └── Best_models/         # Trained models
│       │       └── n_zscore_p_None_s_42_N_10/
│       │           ├── scores.csv           # Evaluation metrics (see below)
│       │           ├── model_weights.pth    # PyTorch model weights
│       │           └── model_config.json    # Hyperparameter configuration
│       ├── MLP/
│       │   ├── GridSearchs/
│       │   └── Best_models/
└── microarray/
    └── BreastCancer_M/
        └── ...

scores.csv columns (evaluation metrics)

For binary classification:

fold: Cross-validation fold number
time(min): Training time in minutes
train_roc_auc: Training ROC-AUC score
val_roc_auc: Validation ROC-AUC score
test_roc_auc: Test ROC-AUC score
test_accuracy: Test accuracy
test_balanced_acc: Test balanced accuracy
test_aupr: Area under precision-recall curve
test_balanced_f1: F1-score (weighted)

For multiclass classification (OVR = One-vs-Rest, OVO = One-vs-One):

fold: Cross-validation fold number
time(min): Training time in minutes
train_roc_auc_ovr: Training ROC-AUC (OVR)
val_roc_auc_ovr: Validation ROC-AUC (OVR)
test_roc_auc_ovr: Test ROC-AUC (OVR)
test_roc_auc_ovo: Test ROC-AUC (OVO)
test_accuracy: Test accuracy
test_balanced_acc: Test balanced accuracy
test_aupr_weighted: Weighted area under precision-recall curve
test_balanced_f1: F1-score (weighted)

Data preprocessing

The scripts under code/preprocessing/ are executed independently of run.py. They should be run from the repository root so that their relative output paths resolve correctly.

cd /path/to/Phenotypic_prediction
Rscript code/preprocessing/<script_name>.R

run.py expects the processed inputs to exist under data/raw/. In general, the workflow is:

Run the dataset-specific preprocessing script from the repository root.
Verify that the expected files were created under data/raw/.
Run python3 -u run.py ... or python3 -u run_transfer_learning.py ....

Dataset-specific preprocessing scripts

BreastCancer_S and BreastCancer_S_binarized: Run Rscript code/preprocessing/breast_subtypes_tcga_download.R This creates TCGA-BRCA/ in the current working directory with BRCA counts, TPMs, and metadata files. For run.py, place TCGA-BRCA_tpm_unstrand.csv and TCGA-BRCA_col_data.csv under data/raw/rnaseq/BreastCancer_S/. BreastCancer_S_binarized uses the same raw files as BreastCancer_S.
LowerGradeGlioma: Run Rscript code/preprocessing/lgg_download.R This creates TCGA-LGG/ in the current working directory with counts, TPMs, and metadata files. For run.py, place TCGA-LGG_tpm_unstrand.csv and TCGA-LGG_col_data.csv under data/raw/rnaseq/LowerGradeGlioma/.
Lung_Histological and Lung_Histological_No_ComBat: Run Rscript code/preprocessing/lung_download.R By default the script sets apply_combat <- FALSE and writes the no-ComBat dataset to data/raw/rnaseq/Lung_Histological_No_ComBat/. To generate the ComBat-corrected version, edit the script and set apply_combat <- TRUE; it will then write to data/raw/rnaseq/Lung_Histological/.
Kidney_Subtype and Kidney_Subtype_No_ComBat: Run Rscript code/preprocessing/kidney_donwload.R By default the script sets apply_combat <- FALSE and writes the no-ComBat dataset to data/raw/rnaseq/Kidney_Subtype_No_ComBat/. To generate the ComBat-corrected version, edit the script and set apply_combat <- TRUE; it will then write to data/raw/rnaseq/Kidney_Subtype/.
MultipleMyeloma: Run Rscript code/preprocessing/multiplemyeloma_preprocess.R This script expects the original CoMMpass files to already be available in ./MultipleMyeloma_RAW/ relative to the working directory. It generates filtered count and metadata files inside ./MultipleMyeloma_RAW/.
Transfer learning PAN-CANCER reference: Run Rscript code/preprocessing/pan_cancer_download.R This writes the PAN-CANCER reference dataset to data/raw/rnaseq/PAN-CANCER/.

Notes on outputs and working directory

The TCGA multi-project scripts (lung_download.R, kidney_donwload.R, pan_cancer_download.R, breast_m_external.R) write directly into data/raw/... and should be run from the repository root.
multiplemyeloma_preprocess.R, breast_subtypes_tcga_download.R, and lgg_download.R write to relative folders created from the current working directory. For reproducibility, run them from the repository root and then move or integrate the generated files into the expected data/raw/ layout if needed.
For BreastCancer_S and LowerGradeGlioma, a minimal setup after running the R script is:

mkdir -p data/raw/rnaseq/BreastCancer_S data/raw/rnaseq/LowerGradeGlioma
cp TCGA-BRCA/TCGA-BRCA_tpm_unstrand.csv TCGA-BRCA/TCGA-BRCA_col_data.csv data/raw/rnaseq/BreastCancer_S/
cp TCGA-LGG/TCGA-LGG_tpm_unstrand.csv TCGA-LGG/TCGA-LGG_col_data.csv data/raw/rnaseq/LowerGradeGlioma/

After raw data is present, run.py creates derived files in data/preprocessed/ automatically when they are missing.

Contributing: Adding New Models

For Sklearn-based models (Logistic_Regression, Random_Forest, SVM, XGBoost)

Add a function in code/models.py:

def YourModel(**kwargs):
    return YourSklearnModel(random_state=42, **kwargs)

Then add configuration in run.py:

'YourModel': {
    'param_grid': {'param1': [1, 2], 'param2': [0.1, 0.5]},
    'model_utils': {
        'grid_search_type': 'gridsearch_sklearn',
        'N_folds_to_use': 5,
        'train_type': 'train_sklearn'
    }
}

For PyTorch Lightning models (MLP, VAE_CL, GNN, Spectral_NN, SPARSE_NN)

Create a class in code/models.py inheriting from pl.LightningModule:

import pytorch_lightning as pl

class YourModel(pl.LightningModule):
    def __init__(self, input_dim, output_dim, **kwargs):
        super().__init__()
        # Define layers
        self.model = nn.Sequential(...)
        self.loss_fn = nn.CrossEntropyLoss()
    
    def forward(self, x):
        return self.model(x)
    
    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=0.001)
    
    def training_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self(x)
        loss = self.loss_fn(y_hat, y)
        return loss
    
    def validation_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self(x)
        loss = self.loss_fn(y_hat, y)
        self.log('val_loss', loss)
    
    def test_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self(x)
        # Compute and log test metrics
        loss = self.loss_fn(y_hat, y)
        self.log('test_loss', loss)

Then add configuration in run.py:

'YourModel': {
    'param_grid': {'learning_rate': [0.001, 0.01], 'hidden_dim': [64, 128]},
    'model_utils': {
        'grid_search_type': 'gridsearch_nn',
        'N_folds_to_use': 5,
        'train_type': 'train_nn'
    }
}

Key Implementation Details

Grid search function: Must match grid_search_type name in code/utils.py
Training function: Must match train_type name and save scores.csv with metrics from the "Output Example" section
N_folds_to_use: Typically smaller than final cross-validation folds (5-10) to speed up grid search

See code/models.py for implementation examples of each model type.

Contributing: Adding New Datasets

To add a new dataset, complete these three steps: place raw files in the expected folder, register the dataset in run.py, and add dataset-specific preprocessing in code/data_utils.py.

1) Place raw files under `data/raw/`

Create:

data/raw/<data_type>/<DatasetName>/

Where:

<data_type> is microarray or rnaseq
<DatasetName> is the exact dataset name you will pass with -d/--dataset in run.py

At minimum, include:

Expression matrix file (genes in rows, samples in columns)
Metadata/labels file

2) Register the dataset in `run.py`

In the available_datasets dictionary, add an entry like:

'YourDataset': {
    'data_type': 'rnaseq',  # or 'microarray'
    'files_names': {
        'data': 'your_expression_file.csv',
        'labels': 'your_labels_file.csv'
    },
    'normalizations': ['none', 'zscore', 'minmax']
},

3) Add preprocessing logic in `code/data_utils.py`

Inside preprocess_dataset(...), add a new branch for your dataset:

elif dataset == 'YourDataset':
    data_matrix = pd.read_csv(os.path.join(raw_path, data_file_name), index_col=0)
    metadata = pd.read_csv(os.path.join(raw_path, labels_file_name), index_col=0)

    # Build labels table
    df_labels = metadata[['barcode', 'class_name']].copy()
    label_map = {'ClassA': 0, 'ClassB': 1}

    df_labels.to_csv(os.path.join(preprocessed_folder, 'Labels_df.tsv'), index=False, sep='\t')
    df_labels.set_index('barcode', inplace=True)

    labels_data = df_labels['class_name'].map(label_map).rename('label').to_frame()

    with open(os.path.join(preprocessed_folder, 'name2label.json'), 'w') as f:
        json.dump(label_map, f)

    # Keep only labeled samples and remove all-zero genes
    raw_data_matrix = data_matrix[df_labels.index.tolist()]
    raw_data_matrix = raw_data_matrix.loc[~(raw_data_matrix == 0).all(axis=1)]

    df_zscore = normalize_dataframe(raw_data_matrix, normalization)

Expected outputs created automatically under data/preprocessed/<data_type>/<DatasetName>/<normalization>/:

normalized_gene_expression.csv
labels.csv (index = sample IDs, one integer column named label)

Notes

Automatic download currently exists only for BreastCancer_M. If needed, also extend download_dataset(...) in code/data_utils.py.
If you plan to use -p DEG with an RNA-seq dataset, ensure the expected raw count-format file exists for apply_DE_analysis(...).

Data Availability

Data availability is not uniform across datasets. There are three different cases in this repository:

1. Datasets that `run.py` can download automatically

BreastCancer_M run.py calls the Python download utilities in code/data_utils.py to fetch the microarray expression matrix and labels, then stores them under data/raw/microarray/BreastCancer_M/.

2. Datasets that require running `code/preprocessing/*.R` first

These datasets are not downloaded by run.py directly. They require the corresponding preprocessing script to be run manually before training:

BreastCancer_S and BreastCancer_S_binarized: Rscript code/preprocessing/breast_subtypes_tcga_download.R
LowerGradeGlioma: Rscript code/preprocessing/lgg_download.R
Lung_Histological and Lung_Histological_No_ComBat: Rscript code/preprocessing/lung_download.R
Kidney_Subtype and Kidney_Subtype_No_ComBat: Rscript code/preprocessing/kidney_donwload.R
PAN-CANCER reference data used by run_transfer_learning.py: Rscript code/preprocessing/pan_cancer_download.R

These scripts write their outputs into dataset-specific folders, typically under data/raw/, and those files are then consumed by run.py or run_transfer_learning.py.

3. Datasets that require external raw files to already exist

MultipleMyeloma Rscript code/preprocessing/multiplemyeloma_preprocess.R does not fetch the raw CoMMpass source files by itself. It expects the original raw files to already be present in ./MultipleMyeloma_RAW/ relative to the working directory, and then generates filtered count and metadata files from them.

In all cases, once the expected raw inputs are available, derived normalized files are created under data/preprocessed/ as needed when run.py or run_transfer_learning.py is executed.

Troubleshooting

CUDA/GPU Issues

Out of memory: Reduce batch size in model hyperparameters or use CPU mode (modify trainer config)
Device not found: Ensure NVIDIA drivers and CUDA 12.4 are installed. Run nvidia-smi to verify

Python/Package Issues

Import errors: Verify all packages installed: python3 -c "import torch, pytorch_lightning, sklearn"
Missing package: Install manually: pip install package_name

R/Data Preprocessing Issues

R package installation fails: Rebuild Docker image to ensure all Bioconductor packages install
Memory issues: Large datasets may require 16GB+ RAM for preprocessing

Training Issues

Model divergence: Try lower learning rates (0.0001) or different activation functions
Grid search too slow: Reduce N_folds_to_use in model config
Results not saved: Verify write permissions in results/ folder

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
Dockers/Pytorch_cuda12_R4_4		Dockers/Pytorch_cuda12_R4_4
code		code
data/complementary		data/complementary
.gitignore		.gitignore
README.md		README.md
run.py		run.py
run_transfer_learning.py		run_transfer_learning.py

Uh oh!

Folders and files

Latest commit

History

Repository files navigation