This repository contains code for the experiments shown in the paper "Metric-Guided Synthetic Image Data Rendering for Deep Learning compatible with Agentic AI".
Please cite as follows:
Developing deep learning computer vision solutions for specific scientific applications often requires collecting and annotating a large dataset from scratch in a laborious, expensive and error-prone process. Synthetic data generation through 3D modelling and scene rendering not only promises to simplify this process but also drastically increases the accuracy of annotations by generating them programmatically. However, minimising the domain gap between real images and 3D rendered synthetic images visually is subjective and lacks systematic quantitative guidance. Here, we propose to use the difference in image naturalness measured by the normalised gradients to guide the gradual improvement of the rendered scene. We demonstrate that this metric correlates not only with improved visual perception of the scene, but also with a higher zero-shot performance of an object detection model aimed at object detection in photographs of virological plaque assays. Finally, we turn procedural data rendering into a skill of agentic AI.
Please set up the environment using conda or pip. We recommend creating a new environment for this project. Navigate to the root directory of this project and run:
conda create -f stardist_environment.yml
conda activate stardist2
pip install -e .
For some systems, the following line may be needed:
CC=gcc-<GCC VERSION> CXX=g++-<GCC VERSION> pip install -e .
Use the Dockerfile in data_generation with the respective python code and Blender file.
Run data_generation/post_processing.py on all segmentation masks by specifying the path to the masks folder.
Please use the scripts under examples/2D_hydra/ and scripts/ in combination with an appropriate config file from configs/ to run the code. Please change the config with appropriate data, output and model weight paths.
Since HSD and HSD-WBR our branched architectures share plenty of common functionality with SD, a slight switch is needed to ensure the correct methods are being referred to. Please comment out one of the two lines below from stardist/models/__init__.py to use either a non-branched or branched(hydra) architecture.
from .model2d import Config2D, StarDist2D, StarDistData2D
from .model2d_hydra import Config2D, StarDist2D, StarDistData2D