Skip to content

SPEC.md

Latest commit

 

History

History
342 lines (254 loc) · 22.4 KB

SPEC.md

File metadata and controls

342 lines (254 loc) · 22.4 KB
Raw

Voice Notes — Showcase App Spec

A polished AI voice-notes app. Records audio, transcribes it, runs an LLM pass for summary + task extraction, gates power features behind a paywall, and reports errors. No application backend — all user data (notes, audio, tasks) lives locally on device. The only server-side code is a thin set of Expo Router API routes that proxy AI provider calls so secret keys never ship in the JS bundle. They're stateless and store nothing.

The build is split into independent stages so each one can be shipped on its own.

1. Product overview

Tagline: Speak it. We'll write it down, summarize it, and pull out your tasks.

Core flows

  1. Sign in with Clerk (email + Apple/Google OAuth via the new native sign-in components).
  2. Tap record → speak → tap stop.
  3. App transcribes the audio via ElevenLabs Scribe.
  4. App runs an LLM pass to produce: title, 3-bullet summary, tasks (checklist), and tags.
  5. Note is saved locally and listed on the home screen.
  6. Free tier: 3 minutes/day, 1 summary/day. Pro (RevenueCat): unlimited minutes, smart summaries, export to Markdown / share sheet.
  7. Errors and slow transcriptions are reported to Sentry with user + session context.

Out of scope (intentionally): stateful backend (no DB, no user-data persistence on the server), multi-device sync of audio, collaboration, web app.

2. Tech stack

Layer Choice Notes
App framework Expo SDK 55 + Expo Router (already scaffolded) File-based routing, typed routes
Language TypeScript, React 19, RN 0.83
Auth Clerk (@clerk/clerk-expo) New native sign-in components, OAuth (Apple, Google)
Audio recording expo-audio Replaces deprecated expo-av
Server (key proxy) Expo Router API routes + EAS Hosting Stateless +api.ts handlers that proxy AI provider calls. No DB, no user data stored. Same repo, same deploy.
Transcription ElevenLabs Scribe (/v1/speech-to-text) Default: proxied through /api/transcribe. If their SDK ships a safe public-key flow, swap to direct device call (TBD — see open questions).
LLM (summary + tasks) OpenAI via the Vercel AI SDK (ai + @ai-sdk/openai) Runs server-side in /api/summarize+api.ts. generateObject against a Zod schema for structured output. Default model gpt-4o-mini.
Local DB expo-sqlite + a thin Drizzle ORM layer Typed queries, migrations
Object storage expo-file-system Audio blobs in app sandbox
Paywall RevenueCat (react-native-purchases + react-native-purchases-ui) Sandbox entitlement: pro
Observability Sentry (@sentry/react-native) Errors, performance traces around record→transcribe→summarize
Styling StyleSheet.create (RN built-in) Plus a small theme.ts for tokens (colors, spacing, type)
State React Query for async, zustand for UI state Keep it small
Icons expo-symbols (SF Symbols on iOS) Already a dep

Configuration & secrets: all keys live in a top-level .env file (standard Expo workflow). Two clear tiers:

  1. Public, client-side keys — EXPO_PUBLIC_* prefix. Inlined into the JS bundle at build time. Used by SDKs that are designed for in-bundle distribution (Clerk publishable key, RevenueCat SDK keys, Sentry DSN).
  2. Private, server-only keys — no prefix. Read by Expo Router API routes via process.env.* at request time. Never reach the device. Used for OpenAI, and (default) ElevenLabs.

.env is gitignored; .env.example ships in the repo with empty placeholders. EAS Hosting picks up the same env names from project secrets at deploy time.

Var Used by Where it runs
EXPO_PUBLIC_CLERK_PUBLISHABLE_KEY Clerk (Stage 2) Client
EXPO_PUBLIC_REVENUECAT_IOS_KEY / _ANDROID_KEY RevenueCat (Stage 7) Client (RC public SDK keys)
EXPO_PUBLIC_SENTRY_DSN Sentry (Stage 8) Client
EXPO_PUBLIC_API_BASE_URL Client → API routes Client (points at EAS Hosting URL)
OPENAI_API_KEY OpenAI / AI SDK (Stage 6) Server only (/api/summarize)
ELEVENLABS_API_KEY ElevenLabs (Stage 5, default) Server only (/api/transcribe)
CLERK_SECRET_KEY API-route auth check Server only (@clerk/backend)
SENTRY_AUTH_TOKEN Source-map upload at build time Build-time only

API-route auth: every API route requires a valid Clerk session token. The client sends Authorization: Bearer <token> (from useAuth().getToken()); the route verifies it with @clerk/backend before calling the upstream provider. Stops randos on the internet from burning the OpenAI key.

3. Repository layout (target)

src/
  app/                       # expo-router
    (auth)/                  # unauthenticated stack
      sign-in.tsx
      sign-up.tsx
    (app)/                   # authenticated tabs
      _layout.tsx            # tab bar
      index.tsx              # Notes list
      record.tsx             # Record screen (modal-style)
      settings.tsx
      note/[id].tsx          # Note detail
    api/                     # Expo Router API routes (server-only)
      _utils/
        auth.ts              # verify Clerk Bearer token via @clerk/backend
      transcribe+api.ts      # POST audio → ElevenLabs Scribe
      summarize+api.ts       # POST transcript → OpenAI via AI SDK
    _layout.tsx              # ClerkProvider, RC, Sentry, QueryClient
  components/
    ui/                      # buttons, cards, sheets
    record/                  # waveform, mic button, timer
    notes/                   # NoteCard, SummaryView, TaskList
    paywall/                 # PaywallSheet wrapper
  features/
    auth/                    # clerk hooks/wrappers
    audio/                   # recorder service
    transcription/           # client wrapper that POSTs audio to /api/transcribe
    summarize/               # client wrapper that POSTs transcript to /api/summarize + shared zod schemas
    notes/                   # db schema, queries, sync
    entitlements/            # RC hooks, gating helpers
    telemetry/               # sentry init + spans
  db/
    schema.ts                # drizzle schema
    client.ts                # sqlite + drizzle
    migrations/
  constants/
  hooks/

4. Data model

// db/schema.ts (Drizzle)
notes {
  id: text (uuid) pk
  user_id: text          // Clerk user id
  title: text
  created_at: int (ms)
  updated_at: int (ms)
  duration_ms: int
  audio_uri: text        // file:// in sandbox
  transcript: text       // raw STT output
  summary: text | null   // markdown
  tags: text             // json array
  status: 'recorded' | 'transcribing' | 'summarizing' | 'ready' | 'error'
}
tasks {
  id: text pk
  note_id: text fk
  text: text
  done: int (0|1)
  order: int
}

Local only. Scoped by user_id so switching Clerk accounts gives you a clean list without wiping the DB.

5. Stages

Each stage is a shippable unit. The app should run end-to-end at the close of every stage.

Stage 0 — Baseline & house-keeping

Goal: clean slate to build on.

  • Strip the Expo template (explore.tsx, sample components).
  • Drop the existing global.css — styling is plain StyleSheet.create.
  • Add a src/constants/theme.ts with color, spacing, radius, and type tokens for both color schemes.
  • Add zustand, @tanstack/react-query, drizzle-orm, expo-sqlite, zod.
  • Convert app.json → app.config.ts. Only EXPO_PUBLIC_* keys reach the client (publishable Clerk key, RC SDK keys, Sentry DSN, EXPO_PUBLIC_API_BASE_URL). Server-only keys (OPENAI_API_KEY, ELEVENLABS_API_KEY, CLERK_SECRET_KEY) are read by API routes via process.env at request time — never forwarded onto extra, never reachable from the device.
  • Enable Expo Router server output: set web.output: "server" (or experiments.server: true per the SDK 55 docs) in app.config.ts so API routes are emitted at build time.
  • Add a stub app/api/health+api.ts returning { ok: true } so we can verify the route runtime works on day one.
  • Commit .env.example with the full key list from §2; add .env and .env.local to .gitignore (Expo already does this by default — verify).
  • Define features/ folder structure (empty stubs).
  • Replace README with a project overview.

Done when: bun start boots a blank tab layout with two empty tabs (Notes, Settings) styled from theme.ts via StyleSheet.

Stage 1 — Navigation shell & design system

Goal: the visual skeleton for the rest of the app.

  • Three-tab layout (Notes / Record / Settings) with a center FAB-style record tab.
  • Empty states for Notes ("Your first recording is one tap away").
  • Reusable primitives: Button, Card, Sheet, Pressable with haptics — each owns its own StyleSheet, pulls values from theme.ts.
  • Light + dark mode via useColorScheme, with a useTheme() hook returning the active token set.
  • Typography scale, spacing tokens, brand color all live in theme.ts; no inline magic numbers in components.

Done when: the app is "demo-pretty" — you could screenshot it for the App Store with placeholder data.

Stage 2 — Auth with Clerk

Goal: real sign-in/sign-up, with Apple + Google OAuth, using Clerk's new native sign-in components.

  • Install @clerk/clerk-expo, wire ClerkProvider in root _layout.tsx with tokenCache from expo-secure-store.
  • (auth) route group: sign-in.tsx and sign-up.tsx using the new <SignIn /> / <SignUp /> native flows.
  • Apple OAuth via expo-apple-authentication, Google via expo-auth-session.
  • Protect (app) group with a redirect if !isSignedIn, use Expo Route guard
  • Settings screen: show user avatar, email, sign-out button using useUser / useAuth.
  • Persist a user_id foreign key on every DB row going forward.

Done when: sign in with Apple → land on empty Notes tab; sign out → kick back to sign-in.

Stage 3 — Local DB & notes CRUD

Goal: notes live somewhere and persist across launches.

  • expo-sqlite + Drizzle, schema from §4, generated migrations checked in.
  • features/notes/ exposes useNotes(), useNote(id), createNote(), deleteNote(), updateNote() via React Query with optimistic updates.
  • Notes list with swipe-to-delete and pull-to-refresh.
  • Note detail screen renders title, transcript (placeholder for now), and a tasks checklist.
  • Seed script for dev (5 fake notes) behind __DEV__.

Done when: you can tap "+ New (mock)" in Settings, see it appear instantly, kill the app, relaunch, and it's still there.

Stage 4 — Audio recording

Goal: capture audio reliably and store it in the sandbox.

  • expo-audio recorder with permissions request flow, including a friendly explainer screen.
  • Record screen: big mic button, live duration, animated waveform (Reanimated 4 worklet driven by metering).
  • Pause/resume, cancel, save → writes to ${FileSystem.documentDirectory}recordings/<uuid>.m4a.
  • On save, insert a notes row with status: 'recorded', audio_uri, duration_ms.
  • Inline player on note detail screen.

Done when: record 10s, see it as a row on Notes with playable audio.

Stage 5 — Transcription with ElevenLabs

Goal: raw audio → text, with a state machine the user can see, secret keys staying on the server.

Pre-flight check: confirm whether ElevenLabs offers a safe public/scoped key for client-side STT (e.g. short-lived signed URLs or a browser-safe key like RC's). Two paths:

  • Default (private key): route the call through app/api/transcribe+api.ts. Client uploads multipart/form-data to /api/transcribe (with the user's Clerk Bearer token); the route verifies auth, then forwards to POST https://api.elevenlabs.io/v1/speech-to-text (Scribe) using process.env.ELEVENLABS_API_KEY. The route streams the response back so we don't buffer audio in memory twice. (See open question 6.)
  • If a safe public key exists: drop the proxy and call ElevenLabs directly from the device (latency win, one fewer hop). Keep the proxy code in a branch as the "production-shaped" reference.

Client work (same regardless of path):

  • features/transcription/client.ts: thin wrapper that takes a local audio URI, builds the multipart body, attaches the Clerk token, and POSTs to EXPO_PUBLIC_API_BASE_URL + '/api/transcribe' (or directly to ElevenLabs in the public-key variant).
  • After save in Stage 4, automatically transition the note status: 'recorded' → 'transcribing' → 'ready'.
  • On the note detail screen: skeleton transcript while pending, real text when done, retry button on error.
  • Pretty transcript view: paragraphs split on long pauses, timestamps in a side gutter (Scribe returns word-level timing).
  • Free-tier minute counter (used in Stage 7's gating).

Done when: record → 5–10s later transcript appears in the note. Disconnect network mid-flight and the error → retry path works. Inspecting the JS bundle shows no ElevenLabs key.

Stage 6 — AI summary & task extraction

Goal: the moment the app stops being a tape recorder. OpenAI key never leaves the server.

  • Install ai + @ai-sdk/openai + zod.
  • Shared schema (features/summarize/schema.ts): Zod NoteSummarySchema = { title, summary: string[], tasks: { text: string }[], tags: string[] }. Imported by both the API route and the client so types match end to end.
  • Server (app/api/summarize+api.ts):
    • POST handler. Verifies Clerk Bearer token via _utils/auth.ts.
    • Body: { transcript: string }.
    • Calls generateObject({ model: openai('gpt-4o-mini'), schema: NoteSummarySchema, system, prompt: transcript }) using process.env.OPENAI_API_KEY.
    • Returns the validated object as JSON. AI SDK handles the structured-output coercion, so no hand-rolled JSON parsing on either side.
    • Streaming variant (behind ?stream=1): streamObject + result.toTextStreamResponse() for a live-bullet UX.
  • Client (features/summarize/client.ts): tiny fetch wrapper that POSTs to /api/summarize, attaches the Clerk token, parses with NoteSummarySchema.parse() for defense in depth.
  • Trigger automatically on status === 'ready' after transcription, OR manually on a free account via a "Summarize" button (gated by Stage 7).
  • Note detail layout becomes: header (title + tags) → summary bullets → tasks checklist (toggleable, persisted) → expandable raw transcript.
  • Re-run summary action (with a confirmation if it would burn a free-tier credit).

Done when: new recording auto-produces a clean title, 3 bullets, and a checked-off-able task list within ~10s of stopping. Streaming variant works behind the flag.

Stage 7 — Paywall with RevenueCat

Goal: monetize the smart features, with a real sandbox flow.

  • Configure RC dashboard: products pro_monthly, pro_yearly, entitlement pro, paywall configured in RC's no-code editor.
  • react-native-purchases init with Clerk userId as the RC app user id (so entitlement follows the account).
  • react-native-purchases-ui <RevenueCatUI.Paywall /> presented as a modal sheet.
  • Gating helpers: useIsPro(), <ProGate feature="summary">…</ProGate>.
  • Free tier limits: 3 min/day recording, 1 summary/day, no export. Counters live in SQLite, reset at local midnight.
  • "Upgrade to Pro" entry points: Settings, Record screen when the cap is hit, Summary button when daily summary used.
  • Restore purchases in Settings.

Done when: sandbox purchase flips useIsPro() to true, removes the limits, and persists across app restart and account sign-out/sign-in.

Stage 8 — Observability with Sentry

Goal: production-grade error + performance reporting.

  • @sentry/react-native with the Expo plugin, source maps uploaded via EAS hook (client).
  • @sentry/node (or the Expo Router server integration if available) inside app/api/_utils/sentry.ts so the API routes report too (server).
  • Set user context from Clerk (id, hashed email — no PII sent in plaintext) on both sides; on the server the user is set per-request from the verified Clerk token.
  • Wrap the navigator with the Sentry Expo Router instrumentation.
  • Custom spans across the wire:
    • Client: record.save, transcribe.client, summarize.client.
    • Server: transcribe.server (forwarding to ElevenLabs) and summarize.server (calling OpenAI via the AI SDK), with attributes (duration_ms, bytes, model, provider).
  • Propagate sentry-trace / baggage headers from client → API route so a single trace covers the full request.
  • Crash button in Settings (dev-only) to demo a stack trace; matching /api/_debug/boom+api.ts (gated to __DEV__) to demo a server-side crash.
  • Profiling enabled at 10% sample on the client.

Done when: force a crash → it shows up in Sentry within seconds with user, route, and the right span breadcrumbs. The summarize and transcribe server spans connect to their client spans via the trace header, so a single timeline shows the full request.

Stage 9 — Polish, export, share

Goal: the App-Store-quality wrap.

  • Pro: export note as Markdown to Files, share sheet (transcript + summary + tasks).
  • Search across notes (SQLite FTS5 if it fits the time budget, otherwise LIKE).
  • Tag filter chips.
  • Re-record / append to existing note.
  • Onboarding carousel on first launch.
  • App icon, splash, store screenshots (the Expo screenshot-optimization skill can drive this).

Done when: the app feels like something you'd pay for.

6. Cross-cutting requirements

  • Privacy copy on first record: explain audio leaves the device (ElevenLabs + OpenAI, via our own proxy), with links.
  • Offline behavior: recording always works offline; transcription/summary queue and run on reconnect.
  • Error UX: every async surface has loading / empty / error states. No silent failures.
  • Accessibility: Dynamic Type, VoiceOver labels on the mic button and waveform, ≥44pt tap targets.
  • API-route hygiene: every route verifies a Clerk session, sets a request-scoped Sentry user, and rate-limits per userId (in-memory token bucket — fine for a demo). No request body is logged.
  • Testing posture: smoke bun start + iOS sim happy path at the end of every stage; for stages 5/6, also curl the API route directly to prove auth rejects unsigned requests. No formal unit tests in scope unless time permits.

7. Suggested build order & dependencies

Stage 0 → 1 → 2 (Clerk) → 3 → 4 → 5 (ElevenLabs) → 6 (OpenAI / AI SDK) → 7 (RevenueCat) → 8 (Sentry) → 9

Stages 5 and 6 can swap; everything else is hard-ordered (paywall gating depends on the features it gates existing first; Sentry server spans depend on the API routes existing).

8. Open questions to resolve before Stage 2

  1. Default OpenAI model for Stage 6 — gpt-4o-mini for cost vs. gpt-4o / gpt-4.1 for output quality?
  2. App bundle id and Apple/Google OAuth client setup — needed before Clerk OAuth can be tested on a real device.
  3. RevenueCat product SKUs and the exact paywall design (template vs. custom).
  4. Sentry org/project + DSN. Server SDK choice (@sentry/node vs. an Expo-Router-specific integration if one ships in time).
  5. Will there be any cross-device sync, even a token one (e.g. Pro entitlement only)? If yes, Clerk public metadata is enough; if no, drop the word "sync" from the marketing copy.
  6. ElevenLabs key model: confirm whether they offer a safe public/scoped key for client-side STT. If yes, Stage 5 can call directly from the device; if no, default to the /api/transcribe proxy path.
  7. API hosting: EAS Hosting (preferred — same repo, same deploy story) vs. a separate Vercel/Cloudflare deploy if EAS Hosting can't host Expo Router API routes. Affects the EXPO_PUBLIC_API_BASE_URL value and the deploy story at the start of Stage 5.