Object Detection: A Comparative Study on an Ecology Dataset (Australian Ferns)

Author: Ariel Halperin

Detectors compared

• YOLOv8m (anchor-free, CNN, COCO-pretrained)
• RetinaNet with a MiT-b0 backbone (anchor-based, transformer encoder, ImageNet-pretrained)

Problem

Detect three Australian fern species in iNaturalist photos:

• Adiantum aethiopicum (common maidenhair)
• Asplenium bulbiferum (hen and chicken fern)
• Blechnum nudum (fishbone water fern)

Two detectors were trained for the same task: an anchor-free CNN against a hybrid CNN+transformer. The dataset was manually annotated, and supplemented by morphologically similar hard negatives. The notebooks work through about 150 hyperparameter combinations across optimiser, learning-rate scheduling, augmentation, and backbone-unfreeze depth. GradCAM and EigenCAM, both explainable-AI techniques, show which image features inform each model's predictions. These views guide dedicated hard-negative mining rounds. Validation tuning picks each model's test-set deployment thresholds.

GradCAM heatmaps at three FPN levels (P3, P4, P5) on a hard adiantum example. Pre-mining attention (top) is diffuse. Post-mining (bottom), it tightens onto the ground-truth (GT) box and the plant structure.

Performance

Detector	val mAP@.5:.95	test macro F1	deploy conf / iou_nms
YOLOv8m	0.290	0.504	0.15 / 0.20
RetinaNet + MiT-b0	0.305	0.594	0.50 / 0.20

Each model's deployment thresholds were chosen by an F1 sweep on val, before any test-set evaluation. The train/val/test splits are stratified by per-class annotation density.

Mining improved true positive (TP) detection and feature mapping. Out-of-sample false positives (FP) also reduced, though not to zero. After mining, both detectors still fire on 18 of the 54 held-out probe images (YOLOv8m: 25 boxes, RetinaNet+MiT-b0: 19 boxes).

Sample test-set predictions, picked across image-density buckets. GT in blue (left). YOLOv8m (centre) and RetinaNet+MiT-b0 (right) in amber.

Off-diagonals are near zero across both models: when either detector fires, it gets the species right. Most error sits in the "missed" column.

How the project is organised

Each notebook runs as a self-contained experiment. Shared helpers are inlined into each notebook's setup cells, so the procedure stays visible at the call site. Sweep results are written to tuner_logs/ and tensorboard_logs/, and later notebooks read those frozen numbers rather than re-running upstream work.

Hard negatives were curated by hand. They cover Australian species that share morphology and backgrounds with the target ferns: cycads, grasstrees, and sedges. The pool excludes anything that risked containing unlabelled ferns. A held-out subset is set aside before training; it's used afterwards to check each model's out-of-sample FP rate.

Dataset retrieval, splits, and experiments are all automated and reproducible from the notebooks here.

Data

Path	Contents
csv_files/*.csv	iNaturalist exports per species (positive classes + _hard_negatives_filtered.csv, _unused_negatives.csv held out for OOS probe; hardneg_yolo8m/ is the per-species hard-neg pool used in mining).
cvat_exports/	Raw CVAT exports from the annotation editing pass.
dataset/fern_dataset_large_fixed.json	Definitive post-CVAT COCO JSON: 600 images, 3 classes. All notebooks load from here.
dataset/bucketed_splits_large.json	Fixed train/val/test splits stratified by count buckets.
dataset/deprecated/	Earlier intermediate annotation files retained for history.

The image files are too large for git. iNaturalist CSVs and the downloader scripts are provided to rebuild them locally.

Notebook order

Run from the project venv (see Reproducibility). Outputs are saved in the committed notebook cells.

Dataset prep:

1. downloader.ipynb — fetch iNaturalist photos from the species CSVs.
2. annotation.ipynb — train and apply a YOLOxl auto-annotator; export COCO JSON for CVAT.
3. redownload_large.ipynb — re-fetch all images at 1024 px after a resolution issue, and remap the bounding-box coords.
4. prepare_splits_large_imgs.ipynb — EDA, area/count bucketing, fixed train/val/test splits, augmentation previews. prepare_splits.ipynb is the earlier 500 px version, kept for reference.

YOLOv8m experiments:

5. yolo8m_head_and_grid.ipynb — head pretrain + Pt. 1 grid search.
6. yolo8m_stage2_cosine_and_bn_unfreeze.ipynb — Pt. 2 cosine decay + BN unfreeze.
7. yolo8m_stage2_5_shallow.ipynb — AdamW shallow-unfreeze ablation.
8. yolo8m_negative_mining.ipynb — Pt. 3 hard-negative mining.
9. yolo8m_deployment_fixup_run.ipynb — extra body retrain at label_smoothing=0, after catching a train pipeline issue on the label smoothing sweep.
10. yolo8m_deployment_sweep.ipynb — confidence-threshold F1 sweep on val, frozen for test.

Pt. 1 grid search: best val_AP per cell across the constant-LR runs. AdamW grid is 16 cells (LR by WD). Each SGD grid is 12 cells (LR by WD at fixed momentum). Best cell per grid outlined.

RetinaNet + MiT-b0 experiments:

11. retnet_mit_setup_and_heads.ipynb — model definition + head pretrain grid.
12. retnet_mit_grid_search.ipynb — body fine-tune grid via keras_tuner.
13. retnet_mit_cams_and_negatives.ipynb — GradCAM + EigenCAM interpretability, then hard-neg retrain.
14. retnet_mit_deployment_sweep.ipynb — confidence-threshold sweep, frozen for test.

Joint:

15. global_evaluation.ipynb — head-to-head on the held-out test set and the out-of-sample hard negative set.

Reusable modules

Each core function is defined inline for notebook readability, as well as in a repo-root module for reuse and iteration.

File	Purpose
visualisation.py	Annotated batch previews, GT/pred overlays, CAM galleries.
utilities.py	Loading helpers, area/count bucketing, IoU + COCO mapping.
eval_helpers.py	sweep_val, report_anchors, and image loading for the deployment sweeps.
model_configs.py	YOLOv8m build factory and hyperparameter slots.
retnet_mit_configs.py	RetinaNet + MiT-b0 build factory and head/body recipes.
backbones.py	MiT-b0 preset loader (works around a keras_cv subclass quirk in MiTBackbone.from_preset).
eigen_grad_cam.py	EigenCAM and GradCAM implementations for the FPN levels of the RetinaNet model.
downloader.py	iNaturalist download helpers reused across notebooks.

Reproducibility

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
jupyter notebook

• TensorFlow 2.20, Keras 3.13, keras_cv 0.9, keras_hub 0.26, keras_tuner 1.4.8. Full version pins are in requirements.txt.
• Trained .keras checkpoints are not in git. Re-running the relevant sweep cell rebuilds the model and reproduces the result. Full training curves in tensorboard_logs/, as well as JSON metrics for all grid sweeps and tuning rounds in tuner_logs/.
• All experiments except the initial annotator trains were run on the UNE Carmack HPC (NVIDIA L40, 48GB VRAM).

Evidence: TensorBoard

• Raw event files: tensorboard_logs/yolo8m/, tensorboard_logs/retnet_mit_head/, tensorboard_logs/retnet_mit_body/.
• Screenshots from the TensorBoard UI: tensorboard_screenshots/.

Launch locally with:

tensorboard --logdir tensorboard_logs

Repo contents

What's in the repo:

• Notebooks
• Annotation JSONs and CVAT exports
• iNaturalist CSVs
• Training history JSONs under tuner_logs/
• TensorBoard event files
• Reusable Python modules
• This README and .gitignore

What's left out, for size:

• Model weights (*.keras, *.h5)
• Image datasets (fern_images_*/, coco_veg/, and raw image dirs under dataset/)

The iNaturalist CSVs and downloader scripts rebuild the image sets. Re-running each sweep cell rebuilds the checkpoints as tested (within seed variation).