STAI — Automated Visual Inspection & Defect Removal

End-to-end automated visual inspection system for flat manufactured parts (ceramic tiles, PCBs, etc.). A conveyor delivers parts past a camera, a CNN-based anomaly detector classifies each part as OK or DEFECT, and an ABB CRB 15000 (GoFa) cobot picks defective parts into a reject tray. The full pipeline runs end-to-end in ABB RobotStudio simulation and is designed to swap to a physical robot with only a config change.

Built as the Project 1 deliverable for COM0403 — Special Topics in AI.

---

Highlights

• PatchCore anomaly detection — feature-memory-bank approach, returns both an image-level score and a pixel-level heatmap (no per-class labels needed at training time).
• FastAPI backend with /inspect, /api/config, /api/stats, and a /ws/dashboard WebSocket stream.
• Next.js 15 dashboard — live feed, event log, robot status, stats charts, threshold tuning.
• Arduino + DroidCam trigger — HC-SR04 ultrasonic sensor fires a phone-camera capture, frame is POSTed to backend.
• RAPID socket server — backend talks to RobotStudio over TCP using a tiny line protocol (DEFECT,x,y,z).
• One-command Windows launch via start.ps1 (auto-creates venvs, installs deps, opens backend / frontend / physical in separate terminals).

Architecture

   ┌──────────────┐    TRIGGER     ┌────────────┐    POST /inspect    ┌──────────────┐
   │  HC-SR04 +   ├───────────────▶│ Physical   ├────────────────────▶│   Backend    │
   │  Arduino     │                │ trigger.py │   (image bytes)     │  (FastAPI)   │
   └──────────────┘                └────────────┘                     └──────┬───────┘
                                                                             │
                            ┌────────────────────────────────────────────────┤
                            │                                                │
                            ▼                                                ▼
                    ┌──────────────┐                                ┌─────────────────┐
                    │   PatchCore  │   image_score, heatmap         │   RobotStudio   │
                    │  Inference   │◀──────────────────────────────▶│  RAPID Socket   │
                    │  (PyTorch)   │   DEFECT,x,y,z                 │     Server      │
                    └──────────────┘                                └─────────────────┘
                            │
                            ▼
                    ┌──────────────┐    WebSocket /ws/dashboard     ┌─────────────────┐
                    │   SQLite     │───────────────────────────────▶│   Next.js UI    │
                    │  event log   │                                │   (dashboard)   │
                    └──────────────┘                                └─────────────────┘

Repo layout

stai/
├── backend/        FastAPI app — REST + WebSocket + ML bridge + Robot TCP client
├── ml/             PatchCore training, dataset, exported models, heatmap outputs
├── frontend/       Next.js 15 + Tailwind + shadcn/ui dashboard
├── physical/       Arduino sketch + Python trigger/capture/post pipeline
├── robotstudio/    RAPID modules (MainModule / SocketServer / PickPlace)
├── docs/           API + socket protocol contracts, model card, demo guide
├── docs.md         Master design brief
├── QUICKSTART.md   5-minute setup walkthrough (Windows)
└── start.ps1       One-shot launcher

Quickstart

The fastest path is the Windows one-shot launcher — see QUICKSTART.md for full hardware/software prerequisites.

git clone https://github.com/kocaemre/stai.git
cd stai
powershell -ep Bypass -File .\start.ps1

start.ps1 creates Python venvs and installs Node deps on first run, then launches backend + frontend + physical trigger in separate windows. RobotStudio is launched separately (start the simulation before running start.ps1).

Useful flags:

• -NoPhysical — skip the Arduino/DroidCam window (run without hardware)
• -Setup — only install dependencies, don't launch

Manual (platform-agnostic)

# Backend
cd backend
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

# Frontend (in another shell)
cd frontend
cp .env.local.example .env.local
npm install
npm run dev

• Swagger UI: http://localhost:8000/docs
• Dashboard: http://localhost:3000

Documentation

Doc	Audience
docs.md	Full design brief — system overview, decisions, deadlines
QUICKSTART.md	Step-by-step setup on Windows
docs/API.md	REST + WebSocket contract (for frontend + physical integration)
docs/PROTOCOL.md	Socket protocol between backend ↔ RobotStudio
docs/MODEL_CARD.md	Model inputs, outputs, thresholds
docs/DEFECT_FLOW.md	End-to-end defect path through the system
docs/DEMO.md	Demo script and screenshots
backend/README.md	Backend dev guide
physical/README.md	Hardware wiring + capture pipeline
robotstudio/README.md	RAPID modules and station setup
ml/README.md	Training, dataset prep, model export

Tech stack

• Backend: Python 3.11+, FastAPI, SQLAlchemy (SQLite), PyTorch
• ML: PatchCore (feature memory bank), torchvision backbone, NumPy
• Frontend: Next.js 15, Tailwind, shadcn/ui, Recharts
• Physical: Arduino (HC-SR04), Python (pyserial, opencv), DroidCam
• Robot: ABB RobotStudio + RAPID, plain TCP socket

Team

• Backend / system integration: Emre Koca (@kocaemre)
• Frontend / dashboard: Bora Kaan (@BoraKaanx)
• ML / dataset / model: İlhan Berk Güven (@iberkguven), Ahmet Furkan (@Ahmet-furkan01)
• Robot / RobotStudio (RAPID): Hakan (@hakantnc)
• Physical input (Arduino + camera): Semi Kağan (@Semikgn)

Contributions, issues, and PRs welcome.

License

MIT — see file for full text.