Extract-it! OCR — Chrome extension

Extract text from paused video frames using region-based OCR (Tesseract).

Project Summary

• Purpose: Pause any video on the web, click the overlay, drag a region and extract text from the current frame.
• Tech: Chrome Extension (Manifest V3), Tesseract.js, canvas-based cropping, chrome.tabs.captureVisibleTab for screenshots.

Files of interest

• Extension root: chrome-extension/
  • background.js — service worker handling screenshot capture
  • content.js — content script that injects overlay, selection UI and performs OCR
  • manifest.json — extension manifest (ensure tesseract.min.js is listed before content.js)
  • styles.css — overlay and text-layer styles
  • tesseract.min.js — Tesseract.js bundle used by the extension
• System design diagram (Mermaid): Extract-It! templates demo
  • Extract-It!/templates-demo/ocr_extension_design.mermaid

System Design (high level)

• Content script monitors video elements; when a video is paused it injects a small overlay button.
• Clicking the overlay initializes Tesseract (worker if available) and enters selection mode.
• The extension asks the background service worker to call chrome.tabs.captureVisibleTab and returns a screenshot image.
• Content script crops the screenshot to the selected region, draws it to a canvas and sends the canvas to Tesseract for recognition.
• Recognized text is displayed in an overlay layer and optionally stored locally.

Below is the system design diagram (Mermaid). GitHub supports rendering Mermaid blocks in READMEs — if your repo displays Markdown, this will render interactively.

graph LR
  USER[👤 User] -->|1. Pauses Video| VIDEO[🎬 Video]
  VIDEO -->|2. Shows Button| BTN[🔘 'E' Button]
    
  BTN -->|3. Click & Drag| SELECT[📐 Select Region]
    
  SELECT -->|4. Capture| BG[📸 Background\nScreenshot]
    
  BG -->|5. Process| OCR[🔍 Tesseract.js\nOCR Engine]
    
  OCR -->|6. Extract Text| RESULT[📄 Text Result]
    
  RESULT -->|7. Display| DISPLAY[💬 Show on Video]
  RESULT -->|8. Copy| CLIP[📋 Clipboard]
  RESULT -->|9. Save| STORAGE[💾 Local Storage\nHistory]
    
  STORAGE -->|Ctrl+Shift+E| EXPORT[📥 Export .txt]
    
  classDef userClass fill:#4CAF50,stroke:#2E7D32,color:#fff
  classDef processClass fill:#2196F3,stroke:#1565C0,color:#fff
  classDef outputClass fill:#FF9800,stroke:#E65100,color:#fff
    
  class USER,VIDEO userClass
  class BTN,SELECT,BG,OCR processClass
  class RESULT,DISPLAY,CLIP,STORAGE,EXPORT outputClass

Loading

Interactive explanation:

• 1. Pauses Video — user pauses any HTML5 video element; content.js listens for pause events and injects the overlay.
• 2. Shows Button — overlay E button appended to the video's parent container (createOverlay() in content.js).
• 3. Click & Drag — clicking the overlay calls initWorker() then startSelection() to draw a selection rectangle.
• 4. Capture — the extension asks the background service worker to capture the visible tab (chrome.tabs.captureVisibleTab) which returns a screenshot image.
• 5. Process — the content script crops the screenshot to the selected region (canvas) and sends it to Tesseract.js. The code supports both worker-based (createWorker) and global Tesseract.recognize fallbacks.
• 6. Extract Text — Tesseract returns recognized text.
• 7. Display — the recognized text is shown as a translucent overlay above the video (#ocr-text-layer).
• 8. Copy — UI can copy text to clipboard (not implemented by default; see content.js hooks).
• 9. Save — recognized text can be stored to chrome.storage.local as history; Ctrl+Shift+E in the diagram denotes an export shortcut to download history as .txt.

Files and mappings:

• content.js: overlay UI, selection, OCR call, display logic.
• background.js: TAKE_SCREENSHOT message handler that calls chrome.tabs.captureVisibleTab.
• manifest.json: make sure tesseract.min.js is listed before content.js in content_scripts so the library is available to the content script.

If you want the mermaid source instead of the rendered block, see Extract-It!/templates-demo/ocr_extension_design.mermaid.

Quick install (developer)

1. Open Chrome and go to chrome://extensions/.
2. Enable Developer mode (top-right).
3. Click Load unpacked and select the chrome-extension folder inside this workspace.
4. Reload the extension after any code changes.

Usage

1. Open a page with a video (YouTube, etc.).
2. Pause the video — the overlay button (E) should appear on the video container.
3. Click the overlay, then drag to select a region on the page.
4. Release the mouse to run OCR. The extracted text will appear as a translucent overlay above the video.

---

Live demo (playable on the repository page)

A demo video is included in this repository and is showcased below for convenience:

• File: Extract-It!/templates-demo/Extract-It! OCR Demo Recording.mp4

Troubleshooting (common console errors and fixes)

• TypeError: worker.load is not a function — occurs when window.Tesseract exposes the older global API (no createWorker) or when worker creation previously failed. Fixes:

  • Ensure tesseract.min.js is loaded before content.js in manifest.json.
  • The content script includes a fallback to use Tesseract.recognize if no worker API exists.

• DataCloneError: Failed to execute 'postMessage' on 'Worker': () => {} could not be cloned. — caused by passing a non-cloneable function (e.g. a logger callback) into createWorker options. Fix: don't pass functions into worker options.

• Permissions policy violation: unload is not allowed in this document. — informational from host page scripts (YouTube). Not caused by the extension.

• net::ERR_ABORTED 403 (Forbidden) — network/YouTube resource access issue (expired/forbidden video URL). Not directly related to the extension.

• CORS errors for third-party ad requests — server-side; cannot be fixed by the extension.

If you see persistent errors after a change, open the extension's background and content script consoles (right-click the extension icon → Inspect background/service worker; open DevTools for the page for content script logs) and paste stack traces.

Development notes

• getUsageCount and local storage: content.js checks for an existing getUsageCount function and sets overlay.title to show a local extract count. If you want to persist and increment counts inside the extension, implement helpers using chrome.storage.local:

// Example helpers (in content script or injected module):
function getUsageCount(cb) {
  chrome.storage.local.get({ usageCount: 0 }, res => cb(res.usageCount || 0));
}
function incrementUsageCount() {
  chrome.storage.local.get({ usageCount: 0 }, res => {
    chrome.storage.local.set({ usageCount: (res.usageCount || 0) + 1 });
  });
}

• If you want to always use the simpler global API instead of worker-based Tesseract, the content.js already contains the recognition fallback — you can remove the worker path to simplify and avoid worker-related issues.

Testing checklist

• Load unpacked extension and verify no errors appear in the background/service worker console.
• Pause a video and ensure the overlay shows up.
• Click overlay and perform a selection; verify OCR result is correct and readable.
• Test on multiple pages (YouTube, Vimeo) and under different DPR (devicePixelRatio) settings.

Notes & Limitations

• OCR quality depends on video resolution, font size, contrast, and Tesseract language model.
• Using Tesseract in the browser is CPU-heavy — expect delays for large regions or on low-powered devices.
• Some pages (cross-origin iframes, protected video elements) may restrict screenshots or access.

Next steps (suggestions)

• Add a small UI to configure language selection, DPI scaling and OCR accuracy presets.
• Persist user preferences and usage count using chrome.storage.local and expose a basic history view.
• Add tests and a tiny harness for automated smoke tests using Puppeteer to pause a test video and verify OCR result.

---

If you want, I can also:

• add the example chrome.storage.local helpers into content.js now,
• create a small demo GIF from a sample mp4 using ffmpeg commands,
• or embed the mermaid diagram into this README as an SVG (needs rendering).

Tell me which of these you'd like next.