CamTalk/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

CamTalk is a multimodal real-time AI visual dialogue assistant. Users interact via camera and microphone — the app captures visual scenes and voice input, sends them to AI services, and responds with both text and speech. The project is currently in the design-document phase; source code is being built incrementally.

Design docs (Chinese): docs/ contains the full architecture, API contracts, user stories, cost control strategies, and technology selection rationale. Read these before implementing any feature.

Architecture

Three-layer system:

1. Browser Client (React 18 + TypeScript, Vite) — media capture, edge preprocessing (VAD via @ricky0123/vad-web, keyframe detection via ONNX Runtime Web), UI rendering. Core hook: useVisionSession().
2. Go Gateway (gorilla/websocket, Redis, Viper, Zap) — WebSocket server, session management, model routing, AI orchestration, rate limiting. One goroutine per WebSocket connection.
3. Cloud AI Services — GPT-4o (LLM), Deepgram (STT), OpenAI TTS. Accessed only through the Go gateway, never directly from the browser.

Key pattern: LLM text chunks and TTS audio are streamed in parallel to the client to minimize perceived latency.

Storage: Cold/hot separation — Redis for real-time session state, PostgreSQL for conversation history and usage stats (deferred past MVP). Repository interface pattern (HistoryRepository, UsageRepository) with in-memory MVP implementations.

Tech Stack

Layer	Tech
Frontend	React 18, TypeScript, Vite, ONNX Runtime Web, @ricky0123/vad-web
Backend	Go, gorilla/websocket, Redis, Viper, Zap
LLM	GPT-4o (primary), Claude Sonnet (backup)
STT	Deepgram (primary), FunASR (self-hosted backup)
TTS	OpenAI TTS (primary), Edge TTS (free alternative)
Model routing	GPT-4o-mini for lightweight classification

Build & Run Commands

# Frontend
cd frontend && npm install
npm run dev          # Vite dev server
npm run build        # Production build
npm run lint         # ESLint
npm run test         # Vitest

# Backend
cd backend && go mod download
go run ./cmd/server          # Start gateway on :8080
go build -o bin/camtalk ./cmd/server
go test ./...                # Run all tests
go test -run TestName ./path # Run single test
go vet ./...                 # Static analysis

Infrastructure: Redis required for session state. PostgreSQL optional for MVP (in-memory fallback).

WebSocket Protocol

Endpoint: ws://localhost:8080/ws

All messages are JSON text frames with {type, request_id?, timestamp?} envelope. See docs/AI 视觉对话助手/项目实现/接口文档.md for the full contract.

Client → Server: query (image Base64 + audio Base64), config, interrupt, ping Server → Client: connected, stt_result, llm_chunk, llm_done, tts_audio, error, pong

Heartbeat: Client pings every 30s. Server disconnects after 60s of silence. Reconnection: Exponential backoff with jitter — 1s, 2s, 4s, 8s… max 30s.

REST API (Auxiliary)

• GET /api/health — health check (version, uptime, active sessions)
• POST /api/sessions — create session (optional, MVP auto-creates on WS connect)
• DELETE /api/sessions/{id} — destroy session

Error Codes

INVALID_MESSAGE, SESSION_NOT_FOUND, RATE_LIMITED, IMAGE_TOO_LARGE, AUDIO_TOO_SHORT, LLM_TIMEOUT, LLM_ERROR, STT_ERROR, TTS_ERROR, INTERNAL_ERROR

Frontend Component Structure

Component	Responsibility
CameraManager	Camera stream capture
MicManager	Microphone audio capture
EdgeProcessor	VAD + keyframe detection (ONNX Runtime)
WebSocketManager	WS connection lifecycle
ChatPanel	Message display
VideoPreview	Camera feed display

Backend Module Structure

Module	Responsibility
WebSocket Hub	Connection management, broadcast/direct push
Session Manager	Session state, conversation history (Redis + TTL)
Model Router	Select AI model per request (rule engine + cost threshold)
AI Orchestrator	Parallel/sequential AI calls with context timeout
Rate Limiter	Per-user token bucket rate limiting

Coding Conventions

• Go: Follow standard Go conventions. Use context.Context for cancellation/timeout in all AI calls. Use sync.RWMutex for concurrent map access. Struct tags use json:"snake_case".
• TypeScript: Strict mode. Interfaces for all data models. WebSocket message types as discriminated unions (type field).
• Commit messages: Use conventional commits format: feat:, fix:, docs:, refactor:, test:, chore:
• No auto-push: Do not push to remote unless explicitly asked.
• Docs-first: When implementing a feature, update the relevant interface doc in docs/ if the implementation diverges from the spec.