SILICA: Repurposing Diffusion Priors for Joint Glass Segmentation and Depth Estimation

Tarun R, Anuj Verma, Laksh Nanwani, Sourav Garg, Madhava Krishna
Robotics Research Center (RRC), IIIT Hyderabad, India

🛠️ Setup

1. Clone the repository.

   git clone https://github.com/rtarun1/Silica

2. Install the required dependencies.

   Using uv:

   curl -LsSf https://astral.sh/uv/install.sh | sh
   cd Silica
   uv sync

   Using pip:

   cd Silica
   python3 -m venv .venv
   source .venv/bin/activate
   pip install -e .

🚀 Run inference

bash infer.sh

🏋️ Training

1. Depth training dataset preparation

• Hypersim:
  • Download this script into your $PATH_TO_RAW_HYPERSIM_DATA directory for data downloading.
  • Run the following command to download the data:

    cd $PATH_TO_RAW_HYPERSIM_DATA
    
    # Download the tone-mapped images
    python ./download.py --contains scene_cam_ --contains final_preview --contains tonemap.jpg --silent
    
    # Download the depth maps
    python ./download.py --contains scene_cam_ --contains geometry_hdf5 --contains depth_meters --silent
    

  • Download the split file from here and put it in the $PATH_TO_RAW_HYPERSIM_DATA directory.
  • Process the data with the command: bash utils/process_hypersim.sh.

2. Glass Segmentation training dataset preparation

• Mirage18k dataset is available here. You can download the dataset using the following command:

  git lfs install
  git clone https://huggingface.co/datasets/rtarun1/mirage18k

• For Trans10k, GSDS, GDD, 3DRef, and GWD please follow the instructions provided in their respective repositories to download the datasets.

3. Train the model:

   bash train.sh

📊 Evaluation

bash eval.sh

🤖 ROS2 Humble inference

1. Clone the repository.

   git clone https://github.com/rtarun1/Silica
   
   # or add submodule to your existing workspace
   # git submodule add https://github.com/rtarun1/Silica silica

2. Install the required dependencies.

   Using pip:

   cd Silica
   python3 -m pip install -e .

3. Build the ROS2 workspace:

   cd <your_ros2_workspace>
   colcon build --symlink-install --packages-select silica_ros

4. Source the workspace:

   source <your_ros2_workspace>/install/setup.bash

5. Run the Silica demo launch file:

   ros2 launch silica_ros silica.launch.py

Note

Make sure that you have a camera and that the camera is publishing RGB images. You will also need (aligned) depth images to get the glass-aware point-cloud.

The topics can be set in the config file config/params.yaml.

🔗 Citation

If you find our work useful in your research, please consider citing our paper:

@misc{r2026silicarepurposingdiffusionpriors,
      title={SILICA: Repurposing Diffusion Priors for Joint Glass Segmentation and Depth Estimation}, 
      author={Tarun R and Anuj Verma and Laksh Nanwani and Sourav Garg and K. Madhava Krishna},
      year={2026},
      eprint={2607.24249},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.24249}, 
}

👏 Acknowledgement

We thank the authors of Marigold, Lotus and Stable Diffusion for their valuable research and for open-sourcing the code and pretrained models that made this work possible.