XEngineers/CamTalk
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request 'fix: 修复 deploy 工作流缺少 Node.js 导致 checkout 失败的问题' #72

Merged
huanghaosheng merged 61 commits from develop into main 2026-06-14 14:37:27 +08:00
Conversation 0 Commits 61 Files Changed 52 +4332 -819
52 changed files with 4332 additions and 819 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.env.example
View File

@@ -11,7 +11,7 @@ CAMTALK_AI_TTS_API_KEY=
# CAMTALK_AI_LLM_MODEL=gpt-4o
# CAMTALK_AI_LLM_ENDPOINT=https://api.openai.com/v1
# CAMTALK_AI_LLM_TIMEOUT=10
# CAMTALK_AI_STT_ENDPOINT=wss://api.deepgram.com/v1/listen
# CAMTALK_AI_STT_ENDPOINT=https://api.xiaomimimo.com/v1
# CAMTALK_AI_TTS_ENDPOINT=https://api.openai.com/v1
# CAMTALK_AI_TTS_VOICE=alloy
# CAMTALK_AI_TTS_SPEED=1.0

63
.gitea/workflows/deploy.yml Normal file
View File

@@ -0,0 +1,63 @@
name: Deploy
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
# ---- PR 时：验证 Docker 镜像可构建 ----
verify:
if: github.event_name == 'pull_request'
runs-on: aliyun
steps:
- name: Setup Node.js
run: |
sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories
apk add --no-cache nodejs
- name: Checkout
uses: "http://8.161.227.145:3000/huanghaosheng/checkout@releases/v4"
- name: Verify Frontend Build
run: docker build -t camtalk-frontend-test ./frontend
- name: Verify Backend Build
run: docker build -t camtalk-backend-test ./backend
- name: Cleanup
run: |
docker rmi camtalk-frontend-test camtalk-backend-test || true
# ---- push main 时：构建并部署 ----
deploy:
if: github.event_name == 'push'
runs-on: aliyun
steps:
- name: Setup Node.js
run: |
sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories
apk add --no-cache nodejs
- name: Checkout
uses: "http://8.161.227.145:3000/huanghaosheng/checkout@releases/v4"
- name: Build and Deploy
run: |
chmod +x deploy.sh
./deploy.sh build
./deploy.sh restart
- name: Health Check
run: |
for i in $(seq 1 10); do
if curl -sf http://localhost/api/health > /dev/null 2>&1; then
echo "服务启动成功"
exit 0
fi
echo "等待服务启动... ($i/10)"
sleep 3
done
echo "服务启动超时"
exit 1

36
CLAUDE.md
View File

@@ -12,24 +12,23 @@ CamTalk 是一款多模态实时 AI 视觉对话助手。用户通过摄像头
三层系统：
1. **浏览器客户端**（React 18 + TypeScript, Vite）—— 媒体采集、边缘预处理（VAD 通过 `@ricky0123/vad-web`、关键帧检测通过 ONNX Runtime Web）、UI 渲染。核心 Hook：`useVisionSession()`
2. **Go 网关**（gorilla/websocket, Redis, Viper, Zap）—— WebSocket 服务器、会话管理、模型路由、AI 编排、速率限制。每个 WebSocket 连接一个 goroutine。
3. **云端 AI 服务** —— GPT-4o（LLM）、Deepgram（STT）、OpenAI TTS。仅通过 Go 网关访问，浏览器不直连。
1. **浏览器客户端**（React 18 + TypeScript, Vite）—— 媒体采集、边缘预处理（VAD 通过 `@ricky0123/vad-web`、关键帧检测通过 Canvas 像素比较）、UI 渲染。核心 Hook：`useVisionSession()`
2. **Go 网关**（Gin, gorilla/websocket, Viper, Zap）—— WebSocket 服务器、会话管理、AI 编排。每个 WebSocket 连接一个 goroutine。
3. **云端 AI 服务** —— 通过 OpenAI 兼容接口可灵活切换。默认：GPT-4o（LLM）、Deepgram（STT）、OpenAI TTS。仅通过 Go 网关访问，浏览器不直连。
**关键模式**：LLM 文本流和 TTS 音频流并行推送给客户端，以最小化感知延迟。
**存储**：冷热分离 —— Redis 存实时会话状态，PostgreSQL 存对话历史和用量统计（MVP 后引入）。Repository 接口模式（`HistoryRepository`、`UsageRepository`），MVP 用内存实现。
**存储**：MVP 阶段使用进程内存（`MemoryManager`），Redis 实现已就绪可通过配置切换，PostgreSQL 为规划中。Repository 接口模式（`HistoryRepository`、`UsageRepository`），MVP 用内存实现。
## 技术栈
| 层级 | 技术 |
|------|------|
| 前端 | React 18, TypeScript, Vite, ONNX Runtime Web, @ricky0123/vad-web |
| 后端 | Go, gorilla/websocket, Redis, Viper, Zap |
| LLM | GPT-4o（主）, Claude Sonnet（备） |
| STT | Deepgram（主）, FunASR（自部署备选） |
| TTS | OpenAI TTS（主）, Edge TTS（免费替代） |
| 模型路由 | GPT-4o-mini 用于轻量分类 |
| 前端 | React 18, TypeScript, Vite, @ricky0123/vad-web |
| 后端 | Go, Gin, gorilla/websocket, Viper, Zap |
| LLM | GPT-4o（默认，通过 OpenAI 兼容接口可切换） |
| STT | Deepgram（默认） / MiMo ASR |
| TTS | OpenAI TTS（默认） / MiMo TTS |
## 构建与运行命令
@@ -50,7 +49,7 @@ go test -run TestName ./path # 运行单个测试
go vet ./... # 静态分析
```
基础设施：Redis 为会话状态必需。PostgreSQL 为 MVP 可选（内存回退）。
基础设施：MVP 使用进程内存管理会话状态。Redis 已实现可通过配置切换，PostgreSQL 为规划中。
## WebSocket 协议
@@ -80,7 +79,7 @@ go vet ./... # 静态分析
|------|------|
| `CameraManager` | 摄像头流采集 |
| `MicManager` | 麦克风音频采集 |
| `EdgeProcessor` | VAD + 关键帧检测（ONNX Runtime） |
| `EdgeProcessor` | VAD + 关键帧检测（Canvas 像素比较） |
| `WebSocketManager` | WebSocket 连接生命周期管理 |
| `ChatPanel` | 消息展示 |
| `VideoPreview` | 摄像头画面预览 |
@@ -89,11 +88,14 @@ go vet ./... # 静态分析
| 模块 | 职责 |
|------|------|
| WebSocket Hub | 连接管理、广播/定向推送 |
| Session Manager | 会话状态、对话历史（Redis + TTL） |
| Model Router | 按请求选择 AI 模型（规则引擎 + 成本阈值） |
| AI Orchestrator | 并行/串行 AI 调用编排，context 超时控制 |
| Rate Limiter | 按用户的令牌桶速率限制 |
| WebSocket Handler | 连接管理、单播消息推送 |
| Session Manager | 会话状态、对话历史（Memory/Redis，30 分钟 TTL） |
| AI Orchestrator | STT→LLM→TTS 流式并行管道编排 |
| AI Service Layer | AI 服务抽象层（STT/LLM/TTS 多 provider） |
| REST API | 健康检查、会话管理（Gin 路由） |
| Models | 数据模型定义 |
| Model Router | 按请求选择 AI 模型（规划中） |
| Rate Limiter | 按用户的令牌桶速率限制（规划中） |
## 编码规范

135
README.md
View File

@@ -1,2 +1,137 @@
# CamTalk
多模态实时 AI 视觉对话助手。用户通过摄像头和麦克风与 AI 交互，AI 理解视觉场景和语音输入后，以文字和语音形式给出自然回应。
## 架构
三层系统，前端做轻量预处理，后端做智能编排，云端 AI 服务按需调用：
```
浏览器客户端 Go 网关 :8080 云端 AI 服务
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ 媒体采集 │ │ WebSocket Handler│ │ STT（语音识别） │
│ VAD 语音检测 │ WebSocket│ Session Manager │ HTTP │ LLM（多模态推理） │
│ 关键帧检测 │ ◄──────► │ AI Orchestrator │ ◄──────► │ TTS（语音合成） │
│ UI 渲染 │ │ REST API │ │ │
└─────────────────┘ └─────────────────┘ └─────────────────┘
```
**关键模式**：LLM 文本流和 TTS 音频流并行推送，用户先看到文字、紧接着听到语音，感知延迟 < 0.5 秒。
## 技术栈
| 层级 | 技术 |
|------|------|
| 前端 | React 18, TypeScript, Vite, @ricky0123/vad-web |
| 后端 | Go, Gin, gorilla/websocket, Viper, Zap |
| STT | Deepgram（默认） / MiMo ASR |
| LLM | GPT-4o（默认，通过 OpenAI 兼容接口可切换） |
| TTS | OpenAI TTS（默认） / MiMo TTS |
## 项目结构
```
CamTalk/
├── frontend/ # 浏览器客户端
│ └── src/
│ ├── components/ # UI 组件
│ │ ├── CameraManager/ # 摄像头流采集
│ │ ├── MicManager/ # 麦克风音频采集
│ │ ├── EdgeProcessor/ # VAD + 关键帧检测
│ │ ├── WebSocketManager/ # WS 连接管理
│ │ ├── ChatPanel/ # 消息展示
│ │ ├── VideoPreview/ # 摄像头画面预览
│ │ ├── ConfigPanel/ # 配置面板
│ │ └── Toast/ # 通知提示
│ ├── hooks/ # 自定义 Hooks
│ │ ├── useVisionSession.ts # 核心会话 Hook
│ │ └── useObservationMode.ts # 观察模式
│ ├── lib/ # 工具库
│ │ ├── websocket.ts # WebSocket 连接管理
│ │ ├── audio.ts # 音频编码
│ │ ├── ttsPlayer.ts # TTS 播放器
│ │ └── sampling.ts # 采样策略
│ └── types/ # TypeScript 类型定义
├── backend/ # Go 网关
│ ├── cmd/server/ # 入口
│ └── internal/
│ ├── ai/ # AI 服务抽象层
│ │ ├── llm/ # LLM 服务（OpenAI 兼容）
│ │ ├── stt/ # STT 服务（Deepgram/MiMo）
│ │ └── tts/ # TTS 服务（OpenAI/MiMo）
│ ├── orchestrator/ # AI 编排器（STT→LLM→TTS 管道）
│ ├── session/ # 会话管理（Memory/Redis）
│ ├── ws/ # WebSocket Handler
│ ├── api/ # REST API
│ ├── config/ # 配置管理
│ ├── models/ # 数据模型
│ ├── errors/ # 错误码
│ └── logger/ # 日志
├── docs/ # 设计文档
└── CLAUDE.md # Claude Code 指引
```
## 快速开始
### 前置条件
- Node.js >= 18
- Go >= 1.24
### 前端
```bash
cd frontend
npm install
npm run dev # Vite 开发服务器 http://localhost:5173
```
### 后端
```bash
cd backend
go mod download
go run ./cmd/server # 启动网关 :8080
```
### 配置
后端配置文件位于 `backend/config.yaml`，支持环境变量覆盖（前缀 `CAMTALK_`）。
```bash
# 最小启动（需要至少一个 AI 服务的 API Key）
cd backend
CAMTALK_AI_LLM_API_KEY=sk-xxx \
CAMTALK_AI_STT_API_KEY=xxx \
go run ./cmd/server
```
配置优先级：环境变量 > `config.{env}.yaml` > `config.yaml` > `.env`
## WebSocket 协议
连接地址：`ws://localhost:8080/ws`
所有消息为 JSON 文本帧，统一信封格式 `{type, request_id?, timestamp?}`。
**客户端 → 服务端**：`query`、`config`、`interrupt`、`ping`
**服务端 → 客户端**：`connected`、`stt_result`、`llm_chunk`、`llm_done`、`tts_audio`、`error`、`pong`
完整协议见 [docs/03-接口文档.md](docs/03-接口文档.md)。
## 文档
| 文档 | 内容 |
|------|------|
| [01-项目概述](docs/01-项目概述.md) | 项目目标与核心挑战 |
| [02-系统架构](docs/02-系统架构.md) | 三层架构、技术栈、部署方案 |
| [03-接口文档](docs/03-接口文档.md) | WebSocket 协议、REST API、配置管理 |
| [04-技术选型](docs/04-技术选型.md) | AI 服务栈、持久化层、前端边缘处理选型 |
| [05-用户故事](docs/05-用户故事.md) | 用户场景与优先级 |
| [06-语音交互](docs/06-语音交互.md) | VAD → STT → LLM → TTS 全链路 |
| [07-视觉理解](docs/07-视觉理解.md) | 帧采样、关键帧检测、多模态输入 |
| [08-成本控制](docs/08-成本控制.md) | 采样策略、端云协同、模型分级 |
## License
[MIT](LICENSE) © XEngineers

1
backend/.gitignore vendored
View File

@@ -3,7 +3,6 @@
bin/
# 环境配置
.env
config.dev.yaml
config.prod.yaml

27
backend/Dockerfile Normal file
View File

@@ -0,0 +1,27 @@
# ---- 构建阶段 ----
FROM golang:1.26-alpine AS builder
WORKDIR /app
# 先复制依赖清单，利用 Docker 缓存层
COPY go.mod go.sum ./
RUN go mod download
# 复制源码并构建
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o /camtalk ./cmd/server
# ---- 运行阶段 ----
FROM alpine:3.20
RUN apk add --no-cache ca-certificates tzdata
WORKDIR /app
# 复制二进制和配置
COPY --from=builder /camtalk .
COPY config.yaml .
EXPOSE 8080
ENTRYPOINT ["./camtalk"]

60
backend/cmd/server/main.go
View File

@@ -5,6 +5,7 @@ import (
"errors"
"net/http"
"os/signal"
"strings"
"syscall"
"time"
@@ -21,6 +22,10 @@ import (
"github.com/hhs/camtalk/internal/ws"
)
// Version 通过构建时 -ldflags 注入，如：
// go build -ldflags "-X main.Version=v1.0.0" ./cmd/server
var Version string
var startTime = time.Now()
func main() {
@@ -42,16 +47,47 @@ func main() {
// 初始化 Session Manager（MVP 默认内存实现）
var sessionMgr session.Manager
// TODO: 当 Redis 配置非空时切换为 RedisManager
sessionMgr = session.NewMemoryManager(30*time.Minute, 20)
sessionMgr = session.NewMemoryManager(
time.Duration(cfg.Session.TTL)*time.Minute,
cfg.Session.MaxHistory,
)
defer sessionMgr.(*session.MemoryManager).Stop()
// 初始化 AI 服务
sttService := stt.NewDeepgramService(cfg.AI.STT.APIKey, cfg.AI.STT.Model, cfg.AI.STT.Endpoint, logger.Log)
llmService := llm.NewOpenAIService(cfg.AI.LLM.APIKey, cfg.AI.LLM.Model, cfg.AI.LLM.Endpoint, cfg.AI.LLM.Timeout, logger.Log)
ttsService := tts.NewOpenAIService(cfg.AI.TTS.APIKey, cfg.AI.TTS.Model, cfg.AI.TTS.Voice, cfg.AI.TTS.Endpoint, cfg.AI.TTS.Speed, cfg.AI.TTS.Timeout, logger.Log)
logger.Log.Infow("initializing AI services",
"stt.provider", cfg.AI.STT.Provider,
"stt.model", cfg.AI.STT.Model,
"llm.provider", cfg.AI.LLM.Provider,
"llm.model", cfg.AI.LLM.Model,
"tts.provider", cfg.AI.TTS.Provider,
"tts.model", cfg.AI.TTS.Model,
"tts.voice", cfg.AI.TTS.Voice,
)
var sttService stt.Service
switch strings.ToLower(cfg.AI.STT.Provider) {
case "mimo", "xiaomi":
sttService = stt.NewMiMoService(cfg.AI.STT.APIKey, cfg.AI.STT.Model, cfg.AI.STT.Endpoint, cfg.AI.STT.Timeout, logger.Log)
logger.Log.Infow("STT service initialized", "provider", "mimo", "model", cfg.AI.STT.Model, "endpoint", cfg.AI.STT.Endpoint)
default:
sttService = stt.NewDeepgramService(cfg.AI.STT.APIKey, cfg.AI.STT.Model, cfg.AI.STT.Endpoint, cfg.AI.STT.Timeout, logger.Log)
logger.Log.Infow("STT service initialized", "provider", "deepgram", "model", cfg.AI.STT.Model)
}
llmService := llm.NewOpenAIService(cfg.AI.LLM.APIKey, cfg.AI.LLM.Model, cfg.AI.LLM.Endpoint, cfg.AI.LLM.Timeout, cfg.AI.LLM.HTTPClientTimeout, logger.Log)
logger.Log.Infow("LLM service initialized", "provider", cfg.AI.LLM.Provider, "model", cfg.AI.LLM.Model, "endpoint", cfg.AI.LLM.Endpoint, "timeout", cfg.AI.LLM.Timeout)
var ttsService tts.Service
switch strings.ToLower(cfg.AI.TTS.Provider) {
case "mimo", "xiaomi":
ttsService = tts.NewMiMoService(cfg.AI.TTS.APIKey, cfg.AI.TTS.Model, cfg.AI.TTS.Voice, cfg.AI.TTS.Endpoint, cfg.AI.TTS.Timeout, cfg.AI.TTS.HTTPClientTimeout, logger.Log)
logger.Log.Infow("TTS service initialized", "provider", "mimo", "model", cfg.AI.TTS.Model, "voice", cfg.AI.TTS.Voice, "endpoint", cfg.AI.TTS.Endpoint)
default:
ttsService = tts.NewOpenAIService(cfg.AI.TTS.APIKey, cfg.AI.TTS.Model, cfg.AI.TTS.Voice, cfg.AI.TTS.Endpoint, cfg.AI.TTS.Speed, cfg.AI.TTS.Timeout, cfg.AI.TTS.HTTPClientTimeout, logger.Log)
logger.Log.Infow("TTS service initialized", "provider", "openai", "model", cfg.AI.TTS.Model, "voice", cfg.AI.TTS.Voice, "speed", cfg.AI.TTS.Speed)
}
// 初始化 Orchestrator
orch := orchestrator.New(sttService, llmService, ttsService, sessionMgr, cfg.AI.LLM.Model)
orch := orchestrator.New(sttService, llmService, ttsService, sessionMgr, cfg)
// Gin 模式
if cfg.App.Env == "prod" {
@@ -64,7 +100,7 @@ func main() {
// REST API
apiGroup := r.Group("/api")
{
apiGroup.GET("/health", healthHandler(sessionMgr))
apiGroup.GET("/health", healthHandler(sessionMgr, cfg))
}
// Session REST 端点
@@ -72,7 +108,7 @@ func main() {
sessionHandler.RegisterRoutes(apiGroup)
// WebSocket
r.GET("/ws", ws.ServeWS(sessionMgr, orch))
r.GET("/ws", ws.ServeWS(sessionMgr, orch, cfg))
// HTTP Server
srv := &http.Server{
@@ -96,7 +132,7 @@ func main() {
<-ctx.Done()
logger.Log.Info("shutting down...")
shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
shutdownCtx, cancel := context.WithTimeout(context.Background(), time.Duration(cfg.Server.ShutdownTimeout)*time.Second)
defer cancel()
if err := srv.Shutdown(shutdownCtx); err != nil {
@@ -106,11 +142,15 @@ func main() {
}
// healthHandler 健康检查。
func healthHandler(sessionMgr session.Manager) gin.HandlerFunc {
func healthHandler(sessionMgr session.Manager, cfg *config.Config) gin.HandlerFunc {
return func(c *gin.Context) {
version := Version
if version == "" {
version = cfg.App.Version
}
c.JSON(200, gin.H{
"status": "ok",
"version": "0.1.0",
"version": version,
"uptime_seconds": int(time.Since(startTime).Seconds()),
"active_sessions": sessionMgr.ActiveCount(),
})

15
backend/config.yaml
View File

@@ -15,20 +15,23 @@ redis:
ai:
stt:
provider: Xiaomi MiMo
model: mimo-v2.5
endpoint: "https://token-plan-cn.xiaomimimo.com/v1"
provider: mimo
model: mimo-v2.5-asr
endpoint: "https://api.xiaomimimo.com/v1"
api_key: "sk-c3jhv58rr5djhxw398w2rrij5tfpnpdgxqq1bojagshzviah"
llm:
provider: dashscope
model: qwen3-vl-plus
endpoint: "https://dashscope.aliyuncs.com/compatible-mode/v1"
api_key: "sk-ws-H.REHELLY.C4s3.MEUCIQCRee37XWEKp2szaxVLFDtR1rxNNsf372zMvCR0Xl6UvQIgZgvhRTvaa1FmhbCQJgaHu4Jny29AQkn01-3hX9CWBOg"
timeout: 30
tts:
provider: Xiaomi MiMo
model: mimo-v2.5
voice: alloy
provider: mimo
model: mimo-v2.5-tts
voice: mimo_default
speed: 1.0
endpoint: "https://token-plan-cn.xiaomimimo.com/v1"
api_key: "tp-c9e7scwfx94qvqyhpnahnw8uaiya01za2qzvg4xe24rp3xiv"
timeout: 5
storage:

20
backend/internal/ai/llm/openai.go
View File

@@ -26,24 +26,23 @@ type OpenAIService struct {
}
// NewOpenAIService 创建 OpenAI LLM 服务。
func NewOpenAIService(apiKey, model, endpoint string, timeoutSec int, logger *zap.SugaredLogger) *OpenAIService {
if model == "" {
model = "gpt-4o"
}
if endpoint == "" {
endpoint = "https://api.openai.com/v1"
}
// model、endpoint 由 config 层保证非空。
func NewOpenAIService(apiKey, model, endpoint string, timeoutSec, httpClientTimeoutSec int, logger *zap.SugaredLogger) *OpenAIService {
timeout := time.Duration(timeoutSec) * time.Second
if timeout <= 0 {
timeout = 10 * time.Second
}
httpClientTimeout := time.Duration(httpClientTimeoutSec) * time.Second
if httpClientTimeout <= 0 {
httpClientTimeout = 60 * time.Second
}
return &OpenAIService{
apiKey: apiKey,
model: model,
endpoint: endpoint,
timeout: timeout,
logger: logger,
client: &http.Client{Timeout: 60 * time.Second}, // HTTP client timeout > LLM timeout
client: &http.Client{Timeout: httpClientTimeout},
}
}
@@ -62,7 +61,7 @@ type chatMessage struct {
type contentPart struct {
Type string `json:"type"`
Text string `json:"text,omitempty"`
Text string `json:"text"`
ImageURL *imageURL `json:"image_url,omitempty"`
}
@@ -101,6 +100,9 @@ func (o *OpenAIService) ChatStream(ctx context.Context, req Request) (<-chan Chu
if err != nil {
return nil, fmt.Errorf("llm: marshal request: %w", err)
}
if err != nil {
return nil, fmt.Errorf("llm: marshal request: %w", err)
}
// 创建带超时的 context
ctx, cancel := context.WithTimeout(ctx, o.timeout)

12
backend/internal/ai/llm/openai_test.go
View File

@@ -53,7 +53,7 @@ func TestOpenAIService_ChatStream_Success(t *testing.T) {
})
defer srv.Close()
svc := NewOpenAIService("test-key", "gpt-4o", srv.URL, 10, zap.NewNop().Sugar())
svc := NewOpenAIService("test-key", "gpt-4o", srv.URL, 10, 60, zap.NewNop().Sugar())
ch, err := svc.ChatStream(context.Background(), Request{
Text: "这是什么？",
@@ -99,7 +99,7 @@ func TestOpenAIService_ChatStream_WithImage(t *testing.T) {
})
defer srv.Close()
svc := NewOpenAIService("test-key", "gpt-4o", srv.URL, 10, zap.NewNop().Sugar())
svc := NewOpenAIService("test-key", "gpt-4o", srv.URL, 10, 60, zap.NewNop().Sugar())
ch, err := svc.ChatStream(context.Background(), Request{
Image: []byte("fake-jpeg-data"),
@@ -123,7 +123,7 @@ func TestOpenAIService_ChatStream_WithHistory(t *testing.T) {
})
defer srv.Close()
svc := NewOpenAIService("test-key", "gpt-4o", srv.URL, 10, zap.NewNop().Sugar())
svc := NewOpenAIService("test-key", "gpt-4o", srv.URL, 10, 60, zap.NewNop().Sugar())
ch, err := svc.ChatStream(context.Background(), Request{
Text: "继续",
@@ -149,7 +149,7 @@ func TestOpenAIService_ChatStream_APIError(t *testing.T) {
})
defer srv.Close()
svc := NewOpenAIService("bad-key", "gpt-4o", srv.URL, 10, zap.NewNop().Sugar())
svc := NewOpenAIService("bad-key", "gpt-4o", srv.URL, 10, 60, zap.NewNop().Sugar())
_, err := svc.ChatStream(context.Background(), Request{
Text: "test",
@@ -172,7 +172,7 @@ func TestOpenAIService_ChatStream_Timeout(t *testing.T) {
})
defer srv.Close()
svc := NewOpenAIService("test-key", "gpt-4o", srv.URL, 1, zap.NewNop().Sugar()) // 1s timeout
svc := NewOpenAIService("test-key", "gpt-4o", srv.URL, 1, 60, zap.NewNop().Sugar()) // 1s timeout
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
@@ -204,7 +204,7 @@ func TestOpenAIService_ChatStream_UsageInResponse(t *testing.T) {
})
defer srv.Close()
svc := NewOpenAIService("test-key", "gpt-4o", srv.URL, 10, zap.NewNop().Sugar())
svc := NewOpenAIService("test-key", "gpt-4o", srv.URL, 10, 60, zap.NewNop().Sugar())
ch, err := svc.ChatStream(context.Background(), Request{Text: "test"})
if err != nil {

17
backend/internal/ai/stt/deepgram.go
View File

@@ -18,21 +18,22 @@ type DeepgramService struct {
apiKey string
model string
endpoint string
timeout time.Duration
logger *zap.SugaredLogger
}
// NewDeepgramService 创建 Deepgram STT 服务。
func NewDeepgramService(apiKey, model, endpoint string, logger *zap.SugaredLogger) *DeepgramService {
if model == "" {
model = "nova-2"
}
if endpoint == "" {
endpoint = "wss://api.deepgram.com/v1/listen"
// model、endpoint 由 config 层保证非空，timeoutSec 为 0 时默认 5 秒。
func NewDeepgramService(apiKey, model, endpoint string, timeoutSec int, logger *zap.SugaredLogger) *DeepgramService {
timeout := time.Duration(timeoutSec) * time.Second
if timeout <= 0 {
timeout = 5 * time.Second
}
return &DeepgramService{
apiKey: apiKey,
model: model,
endpoint: endpoint,
timeout: timeout,
logger: logger,
}
}
@@ -57,8 +58,8 @@ func (d *DeepgramService) Recognize(ctx context.Context, audio []byte, opts Opti
// 构建 WebSocket URL，附带查询参数
wsURL := d.buildURL(opts)
// 5 秒总超时
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
// 总超时
ctx, cancel := context.WithTimeout(ctx, d.timeout)
defer cancel()
// 建立 WebSocket 连接

191
backend/internal/ai/stt/deepgram_test.go
View File

@@ -1,191 +0,0 @@
package stt
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"github.com/gorilla/websocket"
"go.uber.org/zap"
)
var upgrader = websocket.Upgrader{
CheckOrigin: func(r *http.Request) bool { return true },
}
// newMockDeepgram 创建模拟 Deepgram WebSocket 服务。
// 返回 httptest.Server 和对应的 ws:// URL。
func newMockDeepgram(t *testing.T, handler func(conn *websocket.Conn)) *httptest.Server {
t.Helper()
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
conn, err := upgrader.Upgrade(w, r, nil)
if err != nil {
t.Logf("upgrade error: %v", err)
return
}
defer conn.Close()
handler(conn)
}))
return srv
}
// wsToWss 将 http:// 转换为 ws://。
func wsToWss(httpURL string) string {
return "ws" + strings.TrimPrefix(httpURL, "http")
}
func TestDeepgramService_Recognize_Success(t *testing.T) {
srv := newMockDeepgram(t, func(conn *websocket.Conn) {
// 读取音频数据
_, _, err := conn.ReadMessage()
if err != nil {
t.Errorf("read audio: %v", err)
return
}
// 发送中间结果（非 final）
intermediate := deepgramResponse{
IsFinal: false,
}
intermediate.Channel.Alternatives = []struct {
Transcript string `json:"transcript"`
Confidence float64 `json:"confidence"`
}{{Transcript: "你好", Confidence: 0.9}}
data, _ := json.Marshal(intermediate)
_ = conn.WriteMessage(websocket.TextMessage, data)
// 发送最终结果
final := deepgramResponse{
IsFinal: true,
}
final.Channel.Alternatives = []struct {
Transcript string `json:"transcript"`
Confidence float64 `json:"confidence"`
}{{Transcript: "你好世界", Confidence: 0.95}}
data, _ = json.Marshal(final)
_ = conn.WriteMessage(websocket.TextMessage, data)
// 等待客户端关闭
_, _, _ = conn.ReadMessage()
})
defer srv.Close()
svc := NewDeepgramService("test-key", "", wsToWss(srv.URL)+"/v1/listen", zap.NewNop().Sugar())
text, err := svc.Recognize(context.Background(), []byte("fake-pcm-audio"), Options{
Encoding: "pcm_s16le",
SampleRate: 16000,
Language: "zh-CN",
})
if err != nil {
t.Fatalf("Recognize() error: %v", err)
}
if text != "你好世界" {
t.Errorf("Recognize() = %q, want %q", text, "你好世界")
}
}
func TestDeepgramService_Recognize_EmptyAudio(t *testing.T) {
svc := NewDeepgramService("test-key", "", "ws://localhost", zap.NewNop().Sugar())
_, err := svc.Recognize(context.Background(), nil, Options{})
if err == nil {
t.Fatal("Recognize() with empty audio should return error")
}
}
func TestDeepgramService_Recognize_ConnectError(t *testing.T) {
svc := NewDeepgramService("test-key", "", "ws://localhost:1", zap.NewNop().Sugar())
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
_, err := svc.Recognize(ctx, []byte("audio"), Options{})
if err == nil {
t.Fatal("Recognize() with bad endpoint should return error")
}
}
func TestDeepgramService_Recognize_Timeout(t *testing.T) {
// 模拟一个永不响应的服务端
srv := newMockDeepgram(t, func(conn *websocket.Conn) {
// 读取音频但不发送任何结果，让客户端超时
_, _, _ = conn.ReadMessage()
time.Sleep(10 * time.Second)
})
defer srv.Close()
svc := NewDeepgramService("test-key", "", wsToWss(srv.URL)+"/v1/listen", zap.NewNop().Sugar())
ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
defer cancel()
_, err := svc.Recognize(ctx, []byte("audio"), Options{})
if err == nil {
t.Fatal("Recognize() should timeout")
}
}
func TestDeepgramService_Recognize_MultipleFinals(t *testing.T) {
srv := newMockDeepgram(t, func(conn *websocket.Conn) {
_, _, _ = conn.ReadMessage()
// 发送多个 final 结果（多句话场景）
for _, text := range []string{"你好", "世界"} {
resp := deepgramResponse{IsFinal: true}
resp.Channel.Alternatives = []struct {
Transcript string `json:"transcript"`
Confidence float64 `json:"confidence"`
}{{Transcript: text, Confidence: 0.9}}
data, _ := json.Marshal(resp)
_ = conn.WriteMessage(websocket.TextMessage, data)
}
_, _, _ = conn.ReadMessage()
})
defer srv.Close()
svc := NewDeepgramService("test-key", "", wsToWss(srv.URL)+"/v1/listen", zap.NewNop().Sugar())
text, err := svc.Recognize(context.Background(), []byte("audio"), Options{})
if err != nil {
t.Fatalf("Recognize() error: %v", err)
}
if text != "你好世界" {
t.Errorf("Recognize() = %q, want %q", text, "你好世界")
}
}
func TestDeepgramService_buildURL(t *testing.T) {
svc := NewDeepgramService("key", "", "wss://api.deepgram.com/v1/listen", zap.NewNop().Sugar())
tests := []struct {
name string
opts Options
want []string // URL 中应包含的参数
}{
{
name: "defaults",
opts: Options{},
want: []string{"encoding=pcm_s16le", "sample_rate=16000", "language=zh-CN"},
},
{
name: "custom",
opts: Options{Encoding: "wav", SampleRate: 44100, Language: "en"},
want: []string{"encoding=wav", "sample_rate=44100", "language=en"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
u := svc.buildURL(tt.opts)
for _, param := range tt.want {
if !strings.Contains(u, param) {
t.Errorf("buildURL() = %q, should contain %q", u, param)
}
}
})
}
}

233
backend/internal/ai/stt/mimo.go Normal file
View File

@@ -0,0 +1,233 @@
package stt
import (
"bytes"
"context"
"encoding/base64"
"encoding/binary"
"encoding/json"
"fmt"
"io"
"net/http"
"strings"
"time"
"go.uber.org/zap"
)
// MiMoService 基于 Xiaomi MiMo ASR HTTP API 的语音识别实现。
// 接口兼容 OpenAI chat/completions 格式，音频仅支持 mp3/wav。
type MiMoService struct {
apiKey string
model string
endpoint string
timeout time.Duration
logger *zap.SugaredLogger
}
// NewMiMoService 创建 MiMo STT 服务。
// model、endpoint 由 config 层保证非空，timeoutSec 为 0 时默认 10 秒。
func NewMiMoService(apiKey, model, endpoint string, timeoutSec int, logger *zap.SugaredLogger) *MiMoService {
timeout := time.Duration(timeoutSec) * time.Second
if timeout <= 0 {
timeout = 10 * time.Second
}
return &MiMoService{
apiKey: apiKey,
model: model,
endpoint: endpoint,
timeout: timeout,
logger: logger,
}
}
// mimoRequest MiMo ASR 请求体。
type mimoRequest struct {
Model string `json:"model"`
Messages []mimoMessage `json:"messages"`
ASROptions *mimoASROptions `json:"asr_options,omitempty"`
}
type mimoMessage struct {
Role string `json:"role"`
Content []mimoContent `json:"content"`
}
type mimoContent struct {
Type string `json:"type"`
InputAudio *mimoAudioIn `json:"input_audio,omitempty"`
}
type mimoAudioIn struct {
Data string `json:"data"` // data URL: data:{mime};base64,{data}
}
type mimoASROptions struct {
Language string `json:"language"`
}
// mimoResponse MiMo ASR 非流式响应。
type mimoResponse struct {
Choices []struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
} `json:"choices"`
}
// Recognize 实现 stt.Service。将音频发送到 MiMo ASR API，返回识别文本。
func (m *MiMoService) Recognize(ctx context.Context, audio []byte, opts Options) (string, error) {
if len(audio) == 0 {
return "", fmt.Errorf("stt: empty audio")
}
// MiMo 仅支持 mp3/wav，若输入为原始 PCM 则封装为 WAV
audioData := audio
mimeType := "audio/wav"
if !isWAV(audio) && !isMP3(audio) {
wav, err := pcmToWAV(audio, opts.SampleRate, 1)
if err != nil {
return "", fmt.Errorf("stt: pcm to wav: %w", err)
}
audioData = wav
} else if isMP3(audio) {
mimeType = "audio/mpeg"
}
b64 := base64.StdEncoding.EncodeToString(audioData)
dataURL := fmt.Sprintf("data:%s;base64,%s", mimeType, b64)
// 映射语言代码
language := mapLanguage(opts.Language)
reqBody := mimoRequest{
Model: m.model,
Messages: []mimoMessage{
{
Role: "user",
Content: []mimoContent{
{
Type: "input_audio",
InputAudio: &mimoAudioIn{
Data: dataURL,
},
},
},
},
},
}
if language != "" {
reqBody.ASROptions = &mimoASROptions{Language: language}
}
body, err := json.Marshal(reqBody)
if err != nil {
return "", fmt.Errorf("stt: marshal request: %w", err)
}
url := strings.TrimRight(m.endpoint, "/") + "/chat/completions"
ctx, cancel := context.WithTimeout(ctx, m.timeout)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(body))
if err != nil {
return "", fmt.Errorf("stt: create request: %w", err)
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "Bearer "+m.apiKey)
resp, err := http.DefaultClient.Do(req)
if err != nil {
return "", fmt.Errorf("stt: request mimo: %w", err)
}
defer resp.Body.Close()
respBody, err := io.ReadAll(resp.Body)
if err != nil {
return "", fmt.Errorf("stt: read response: %w", err)
}
if resp.StatusCode != http.StatusOK {
return "", fmt.Errorf("stt: mimo returned %d: %s", resp.StatusCode, string(respBody))
}
var mResp mimoResponse
if err := json.Unmarshal(respBody, &mResp); err != nil {
return "", fmt.Errorf("stt: unmarshal response: %w", err)
}
if len(mResp.Choices) == 0 {
return "", fmt.Errorf("stt: mimo returned empty choices")
}
text := strings.TrimSpace(mResp.Choices[0].Message.Content)
return text, nil
}
// mapLanguage 将标准语言代码映射为 MiMo 支持的值（auto/zh/en）。
func mapLanguage(lang string) string {
switch {
case lang == "":
return "auto"
case strings.HasPrefix(lang, "zh"):
return "zh"
case strings.HasPrefix(lang, "en"):
return "en"
default:
return "auto"
}
}
// isWAV 检查数据是否为 WAV 格式（RIFF 头）。
func isWAV(data []byte) bool {
return len(data) > 4 && string(data[:4]) == "RIFF"
}
// isMP3 检查数据是否为 MP3 格式（ID3 标签或帧同步字）。
func isMP3(data []byte) bool {
if len(data) > 3 && string(data[:3]) == "ID3" {
return true
}
// 帧同步字：0xFF 0xFB/0xF3/0xF2
return len(data) > 2 && data[0] == 0xFF && (data[1]&0xE0) == 0xE0
}
// pcmToWAV 将原始 PCM 数据封装为 WAV 文件。
func pcmToWAV(pcm []byte, sampleRate, channels int) ([]byte, error) {
if sampleRate == 0 {
sampleRate = 16000
}
if channels == 0 {
channels = 1
}
bitsPerSample := 16
byteRate := sampleRate * channels * bitsPerSample / 8
blockAlign := channels * bitsPerSample / 8
dataSize := len(pcm)
var buf bytes.Buffer
// RIFF header
buf.WriteString("RIFF")
binary.Write(&buf, binary.LittleEndian, uint32(36+dataSize))
buf.WriteString("WAVE")
// fmt 子块
buf.WriteString("fmt ")
binary.Write(&buf, binary.LittleEndian, uint32(16)) // 子块大小
binary.Write(&buf, binary.LittleEndian, uint16(1)) // PCM 格式
binary.Write(&buf, binary.LittleEndian, uint16(channels)) // 通道数
binary.Write(&buf, binary.LittleEndian, uint32(sampleRate)) // 采样率
binary.Write(&buf, binary.LittleEndian, uint32(byteRate)) // 字节率
binary.Write(&buf, binary.LittleEndian, uint16(blockAlign)) // 块对齐
binary.Write(&buf, binary.LittleEndian, uint16(bitsPerSample)) // 每样本位数
// data 子块
buf.WriteString("data")
binary.Write(&buf, binary.LittleEndian, uint32(dataSize))
buf.Write(pcm)
return buf.Bytes(), nil
}

255
backend/internal/ai/stt/mimo_test.go Normal file
View File

@@ -0,0 +1,255 @@
package stt
import (
"context"
"encoding/base64"
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"testing"
"go.uber.org/zap"
)
func newTestMiMoService(handler http.HandlerFunc) (*MiMoService, *httptest.Server) {
srv := httptest.NewServer(handler)
s := NewMiMoService("test-key", "mimo-v2.5-asr", srv.URL, 0, zap.NewNop().Sugar())
return s, srv
}
func TestMiMoService_Recognize_Success(t *testing.T) {
s, srv := newTestMiMoService(func(w http.ResponseWriter, r *http.Request) {
// 验证请求
if r.Header.Get("Authorization") != "Bearer test-key" {
t.Errorf("expected Authorization Bearer test-key, got %s", r.Header.Get("Authorization"))
}
if r.URL.Path != "/chat/completions" {
t.Errorf("expected path /chat/completions, got %s", r.URL.Path)
}
var req mimoRequest
body, _ := io.ReadAll(r.Body)
if err := json.Unmarshal(body, &req); err != nil {
t.Fatalf("unmarshal request: %v", err)
}
if req.Model != "mimo-v2.5-asr" {
t.Errorf("expected model mimo-v2.5-asr, got %s", req.Model)
}
if len(req.Messages) == 0 || req.Messages[0].Role != "user" {
t.Error("expected user message")
}
if req.ASROptions == nil || req.ASROptions.Language != "zh" {
t.Errorf("expected language zh, got %v", req.ASROptions)
}
resp := mimoResponse{
Choices: []struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
}{
{Message: struct {
Content string `json:"content"`
}{Content: "你好世界"}},
},
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(resp)
})
defer srv.Close()
// 发送一个简单的有效 WAV（44 字节头 + 少量 PCM）
wav := makeValidWAV([]byte{0x00, 0x00, 0x00, 0x00})
text, err := s.Recognize(context.Background(), wav, Options{Language: "zh-CN"})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if text != "你好世界" {
t.Errorf("expected '你好世界', got '%s'", text)
}
}
func TestMiMoService_Recognize_EmptyAudio(t *testing.T) {
s, srv := newTestMiMoService(func(w http.ResponseWriter, r *http.Request) {})
defer srv.Close()
_, err := s.Recognize(context.Background(), nil, Options{})
if err == nil {
t.Fatal("expected error for empty audio")
}
}
func TestMiMoService_Recognize_ServerError(t *testing.T) {
s, srv := newTestMiMoService(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
w.Write([]byte("internal error"))
})
defer srv.Close()
wav := makeValidWAV([]byte{0x00, 0x00})
_, err := s.Recognize(context.Background(), wav, Options{})
if err == nil {
t.Fatal("expected error for 500 response")
}
}
func TestMiMoService_Recognize_EmptyChoices(t *testing.T) {
s, srv := newTestMiMoService(func(w http.ResponseWriter, r *http.Request) {
resp := mimoResponse{Choices: nil}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(resp)
})
defer srv.Close()
wav := makeValidWAV([]byte{0x00, 0x00})
_, err := s.Recognize(context.Background(), wav, Options{})
if err == nil {
t.Fatal("expected error for empty choices")
}
}
func TestMiMoService_Recognize_PCMAutoWrap(t *testing.T) {
// 测试原始 PCM 数据自动封装为 WAV
s, srv := newTestMiMoService(func(w http.ResponseWriter, r *http.Request) {
var req mimoRequest
body, _ := io.ReadAll(r.Body)
if err := json.Unmarshal(body, &req); err != nil {
t.Fatalf("unmarshal request: %v", err)
}
// 验证 data URL 格式
if len(req.Messages) == 0 || len(req.Messages[0].Content) == 0 {
t.Fatal("empty message content")
}
dataURL := req.Messages[0].Content[0].InputAudio.Data
if len(dataURL) < 22 || dataURL[:14] != "data:audio/wav" {
t.Errorf("expected wav data URL, got prefix: %s", dataURL[:min(len(dataURL), 30)])
}
// 验证 base64 可解码
b64Part := dataURL[22:] // skip "data:audio/wav;base64,"
decoded, err := base64.StdEncoding.DecodeString(b64Part)
if err != nil {
t.Fatalf("base64 decode failed: %v", err)
}
// 应该是有效 WAV（RIFF 头）
if len(decoded) < 44 || string(decoded[:4]) != "RIFF" {
t.Error("decoded data is not a valid WAV")
}
resp := mimoResponse{
Choices: []struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
}{
{Message: struct {
Content string `json:"content"`
}{Content: "test"}},
},
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(resp)
})
defer srv.Close()
// 发送原始 PCM（非 WAV/MP3）
pcm := []byte{0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07}
text, err := s.Recognize(context.Background(), pcm, Options{SampleRate: 16000})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if text != "test" {
t.Errorf("expected 'test', got '%s'", text)
}
}
func TestMapLanguage(t *testing.T) {
tests := []struct {
input string
want string
}{
{"", "auto"},
{"zh-CN", "zh"},
{"zh", "zh"},
{"en-US", "en"},
{"en", "en"},
{"ja", "auto"},
}
for _, tt := range tests {
got := mapLanguage(tt.input)
if got != tt.want {
t.Errorf("mapLanguage(%q) = %q, want %q", tt.input, got, tt.want)
}
}
}
func TestIsWAV(t *testing.T) {
if !isWAV([]byte("RIFF....")) {
t.Error("expected true for RIFF header")
}
if isWAV([]byte("ID3...")) {
t.Error("expected false for ID3 header")
}
if isWAV([]byte{0x00}) {
t.Error("expected false for short data")
}
}
func TestIsMP3(t *testing.T) {
if !isMP3([]byte("ID3\x03")) {
t.Error("expected true for ID3 header")
}
if !isMP3([]byte{0xFF, 0xFB, 0x00}) {
t.Error("expected true for MP3 sync word")
}
if isMP3([]byte("RIFF")) {
t.Error("expected false for RIFF header")
}
}
func makeValidWAV(pcm []byte) []byte {
// 构造一个最小有效 WAV
wav := make([]byte, 44+len(pcm))
copy(wav[:4], "RIFF")
// little-endian size = 36 + len(pcm)
size := uint32(36 + len(pcm))
wav[4] = byte(size)
wav[5] = byte(size >> 8)
wav[6] = byte(size >> 16)
wav[7] = byte(size >> 24)
copy(wav[8:12], "WAVE")
copy(wav[12:16], "fmt ")
// fmt chunk size = 16
wav[16] = 16
// PCM format = 1
wav[20] = 1
// channels = 1
wav[22] = 1
// sample rate = 16000
wav[24] = 0x80
wav[25] = 0x3E
// byte rate = 32000
wav[28] = 0x00
wav[29] = 0x7D
// block align = 2
wav[32] = 2
// bits per sample = 16
wav[34] = 16
copy(wav[36:40], "data")
dSize := uint32(len(pcm))
wav[40] = byte(dSize)
wav[41] = byte(dSize >> 8)
wav[42] = byte(dSize >> 16)
wav[43] = byte(dSize >> 24)
copy(wav[44:], pcm)
return wav
}
func min(a, b int) int {
if a < b {
return a
}
return b
}

208
backend/internal/ai/tts/mimo.go Normal file
View File

@@ -0,0 +1,208 @@
package tts
import (
"bytes"
"context"
"encoding/base64"
"encoding/json"
"fmt"
"io"
"net/http"
"strings"
"time"
"go.uber.org/zap"
)
// MiMoService 基于 Xiaomi MiMo TTS API 的语音合成实现。
// 接口兼容 OpenAI chat/completions 格式，通过 messages 传递待合成文本与风格指令。
type MiMoService struct {
apiKey string
model string
voice string
endpoint string
timeout time.Duration
logger *zap.SugaredLogger
client *http.Client
}
// NewMiMoService 创建 MiMo TTS 服务。
// model、voice、endpoint 由 config 层保证非空。
func NewMiMoService(apiKey, model, voice, endpoint string, timeoutSec, httpClientTimeoutSec int, logger *zap.SugaredLogger) *MiMoService {
timeout := time.Duration(timeoutSec) * time.Second
if timeout <= 0 {
timeout = 5 * time.Second
}
httpClientTimeout := time.Duration(httpClientTimeoutSec) * time.Second
if httpClientTimeout <= 0 {
httpClientTimeout = 30 * time.Second
}
return &MiMoService{
apiKey: apiKey,
model: model,
voice: voice,
endpoint: endpoint,
timeout: timeout,
logger: logger,
client: &http.Client{Timeout: httpClientTimeout},
}
}
// mimoTTSRequest MiMo TTS API 请求体。
type mimoTTSRequest struct {
Model string `json:"model"`
Messages []mimoTTSMessage `json:"messages"`
Audio mimoTTSAudio `json:"audio"`
Stream bool `json:"stream"`
}
// mimoTTSMessage MiMo TTS 消息。
type mimoTTSMessage struct {
Role string `json:"role"` // "user"（风格指令）| "assistant"（待合成文本）
Content string `json:"content"`
}
// mimoTTSAudio MiMo TTS 音频配置。
type mimoTTSAudio struct {
Format string `json:"format"` // "mp3" | "wav" | "pcm16"
Voice string `json:"voice"` // 预置音色 ID
}
// mimoTTSResponse MiMo TTS 非流式响应。
type mimoTTSResponse struct {
Choices []struct {
Message struct {
Audio struct {
Data string `json:"data"` // base64 编码的音频数据
} `json:"audio"`
} `json:"message"`
} `json:"choices"`
}
// mimoTTSStreamResponse MiMo TTS 流式响应。
type mimoTTSStreamResponse struct {
Choices []struct {
Delta struct {
Audio struct {
Data string `json:"data"` // base64 编码的音频数据片段
} `json:"audio"`
} `json:"delta"`
} `json:"choices"`
}
// SynthesizeStream 实现 tts.Service。从 textStream 读取句子，逐句调用 MiMo TTS API。
func (m *MiMoService) SynthesizeStream(ctx context.Context, textStream <-chan string, opts Options) (<-chan Chunk, error) {
voice := opts.Voice
if voice == "" {
voice = m.voice
}
ch := make(chan Chunk, 4)
go func() {
defer close(ch)
for text := range textStream {
if text == "" {
continue
}
audio, err := m.synthesize(ctx, text, voice)
if err != nil {
m.logger.Warnw("mimo tts: synthesize failed", "error", err, "text", text)
// 静默跳过，不中断整个流
continue
}
select {
case ch <- Chunk{Audio: audio, IsLast: true, Final: false}:
case <-ctx.Done():
return
}
}
// textStream 关闭，发送 Final 标记
select {
case ch <- Chunk{Audio: nil, IsLast: false, Final: true}:
case <-ctx.Done():
}
}()
return ch, nil
}
// synthesize 调用 MiMo TTS API 合成单个句子。
// 使用非流式调用，返回完整音频数据（base64 解码后）。
func (m *MiMoService) synthesize(ctx context.Context, text, voice string) ([]byte, error) {
// 单句超时
ctx, cancel := context.WithTimeout(ctx, m.timeout)
defer cancel()
// 构建 MiMo TTS 请求：文本放在 assistant 消息中
reqBody := mimoTTSRequest{
Model: m.model,
Messages: []mimoTTSMessage{
{
Role: "assistant",
Content: text,
},
},
Audio: mimoTTSAudio{
Format: "mp3",
Voice: voice,
},
Stream: false,
}
payload, err := json.Marshal(reqBody)
if err != nil {
return nil, fmt.Errorf("mimo tts: marshal request: %w", err)
}
url := strings.TrimRight(m.endpoint, "/") + "/chat/completions"
req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(payload))
if err != nil {
return nil, fmt.Errorf("mimo tts: create request: %w", err)
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("api-key", m.apiKey)
resp, err := m.client.Do(req)
if err != nil {
return nil, fmt.Errorf("mimo tts: send request: %w", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
errBody, _ := io.ReadAll(resp.Body)
return nil, fmt.Errorf("mimo tts: api error (status %d): %s", resp.StatusCode, string(errBody))
}
// 非流式响应：解析 JSON，提取 base64 音频数据
respBody, err := io.ReadAll(resp.Body)
if err != nil {
return nil, fmt.Errorf("mimo tts: read response: %w", err)
}
var ttsResp mimoTTSResponse
if err := json.Unmarshal(respBody, &ttsResp); err != nil {
return nil, fmt.Errorf("mimo tts: unmarshal response: %w", err)
}
if len(ttsResp.Choices) == 0 {
return nil, fmt.Errorf("mimo tts: empty choices in response")
}
audioData := ttsResp.Choices[0].Message.Audio.Data
if audioData == "" {
return nil, fmt.Errorf("mimo tts: empty audio data in response")
}
// base64 解码音频数据
audio, err := base64.StdEncoding.DecodeString(audioData)
if err != nil {
return nil, fmt.Errorf("mimo tts: decode audio base64: %w", err)
}
return audio, nil
}

414
backend/internal/ai/tts/mimo_test.go Normal file
View File

@@ -0,0 +1,414 @@
package tts
import (
"context"
"encoding/base64"
"encoding/json"
"fmt"
"io"
"net/http"
"net/http/httptest"
"strings"
"sync/atomic"
"testing"
"time"
"go.uber.org/zap"
)
// mockMiMoTTSServer 创建模拟 MiMo TTS API 的 HTTP 服务器。
func mockMiMoTTSServer(t *testing.T, handler http.HandlerFunc) *httptest.Server {
t.Helper()
return httptest.NewServer(handler)
}
// buildMiMoTTSResponse 构造 MiMo TTS 非流式响应 JSON。
func buildMiMoTTSResponse(audioData string) []byte {
resp := mimoTTSResponse{
Choices: []struct {
Message struct {
Audio struct {
Data string `json:"data"`
} `json:"audio"`
} `json:"message"`
}{
{
Message: struct {
Audio struct {
Data string `json:"data"`
} `json:"audio"`
}{
Audio: struct {
Data string `json:"data"`
}{Data: audioData},
},
},
},
}
data, _ := json.Marshal(resp)
return data
}
func TestMiMoService_SynthesizeStream_Success(t *testing.T) {
var callCount int32
srv := mockMiMoTTSServer(t, func(w http.ResponseWriter, r *http.Request) {
atomic.AddInt32(&callCount, 1)
if r.Method != http.MethodPost {
t.Errorf("method = %s, want POST", r.Method)
}
if !strings.Contains(r.URL.Path, "/chat/completions") {
t.Errorf("path = %s, should contain /chat/completions", r.URL.Path)
}
// 验证 api-key 认证头
apiKey := r.Header.Get("api-key")
if apiKey != "test-key" {
t.Errorf("api-key = %q, want %q", apiKey, "test-key")
}
// 验证请求体
body, _ := io.ReadAll(r.Body)
var req mimoTTSRequest
if err := json.Unmarshal(body, &req); err != nil {
t.Errorf("unmarshal request: %v", err)
}
if req.Model != "mimo-v2.5-tts" {
t.Errorf("model = %q, want %q", req.Model, "mimo-v2.5-tts")
}
if len(req.Messages) != 1 || req.Messages[0].Role != "assistant" {
t.Errorf("expected 1 assistant message, got %d messages", len(req.Messages))
}
if req.Audio.Voice != "冰糖" {
t.Errorf("voice = %q, want %q", req.Audio.Voice, "冰糖")
}
if req.Audio.Format != "mp3" {
t.Errorf("format = %q, want %q", req.Audio.Format, "mp3")
}
// 返回假音频数据（base64 编码）
audioB64 := base64.StdEncoding.EncodeToString([]byte("fake-mp3-data"))
w.Header().Set("Content-Type", "application/json")
w.Write(buildMiMoTTSResponse(audioB64))
})
defer srv.Close()
svc := NewMiMoService("test-key", "mimo-v2.5-tts", "冰糖", srv.URL, 5, 30, zap.NewNop().Sugar())
textStream := sendSentences("你好", "世界", "！")
ch, err := svc.SynthesizeStream(context.Background(), textStream, Options{
Voice: "冰糖", OutputFmt: "mp3", SampleRate: 24000,
})
if err != nil {
t.Fatalf("SynthesizeStream() error: %v", err)
}
var chunks []Chunk
for c := range ch {
chunks = append(chunks, c)
}
// 应该有 3 个音频 chunk + 1 个 Final 标记
if len(chunks) != 4 {
t.Fatalf("got %d chunks, want 4", len(chunks))
}
// 验证前 3 个有音频数据，IsLast 为 true（每句结束）
for i := 0; i < 3; i++ {
if string(chunks[i].Audio) != "fake-mp3-data" {
t.Errorf("chunk[%d].Audio = %q, want %q", i, string(chunks[i].Audio), "fake-mp3-data")
}
if !chunks[i].IsLast {
t.Errorf("chunk[%d].IsLast should be true (sentence end)", i)
}
if chunks[i].Final {
t.Errorf("chunk[%d].Final should be false", i)
}
}
// 验证最后一个是 Final（整轮结束）
if !chunks[3].Final {
t.Error("last chunk should be Final")
}
if chunks[3].Audio != nil {
t.Error("last chunk Audio should be nil")
}
// 验证调用了 3 次 API（3 个句子）
if atomic.LoadInt32(&callCount) != 3 {
t.Errorf("API called %d times, want 3", callCount)
}
}
func TestMiMoService_SynthesizeStream_APIError(t *testing.T) {
srv := mockMiMoTTSServer(t, func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
fmt.Fprintf(w, "internal error")
})
defer srv.Close()
svc := NewMiMoService("test-key", "mimo-v2.5-tts", "冰糖", srv.URL, 5, 30, zap.NewNop().Sugar())
textStream := sendSentences("你好")
ch, err := svc.SynthesizeStream(context.Background(), textStream, Options{})
if err != nil {
t.Fatalf("SynthesizeStream() error: %v", err)
}
// 应该只有一个 Final chunk（音频被跳过）
var chunks []Chunk
for c := range ch {
chunks = append(chunks, c)
}
if len(chunks) != 1 {
t.Fatalf("got %d chunks, want 1 (Final only)", len(chunks))
}
if !chunks[0].Final {
t.Error("chunk should be Final")
}
}
func TestMiMoService_SynthesizeStream_Timeout(t *testing.T) {
srv := mockMiMoTTSServer(t, func(w http.ResponseWriter, r *http.Request) {
time.Sleep(3 * time.Second)
audioB64 := base64.StdEncoding.EncodeToString([]byte("late-mp3"))
w.Header().Set("Content-Type", "application/json")
w.Write(buildMiMoTTSResponse(audioB64))
})
defer srv.Close()
// 1 秒超时
svc := NewMiMoService("test-key", "mimo-v2.5-tts", "冰糖", srv.URL, 1, 30, zap.NewNop().Sugar())
textStream := sendSentences("很长的句子")
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
ch, err := svc.SynthesizeStream(ctx, textStream, Options{})
if err != nil {
t.Fatalf("SynthesizeStream() error: %v", err)
}
var chunks []Chunk
for c := range ch {
chunks = append(chunks, c)
}
// 超时后音频被跳过，只有 Final
if len(chunks) != 1 {
t.Fatalf("got %d chunks, want 1", len(chunks))
}
if !chunks[0].Final {
t.Error("chunk should be Final")
}
}
func TestMiMoService_SynthesizeStream_EmptyText(t *testing.T) {
var callCount int32
srv := mockMiMoTTSServer(t, func(w http.ResponseWriter, r *http.Request) {
atomic.AddInt32(&callCount, 1)
audioB64 := base64.StdEncoding.EncodeToString([]byte("mp3"))
w.Header().Set("Content-Type", "application/json")
w.Write(buildMiMoTTSResponse(audioB64))
})
defer srv.Close()
svc := NewMiMoService("test-key", "mimo-v2.5-tts", "冰糖", srv.URL, 5, 30, zap.NewNop().Sugar())
// 空句子应该被跳过
textStream := sendSentences("", "你好", "")
ch, err := svc.SynthesizeStream(context.Background(), textStream, Options{})
if err != nil {
t.Fatalf("SynthesizeStream() error: %v", err)
}
var chunks []Chunk
for c := range ch {
chunks = append(chunks, c)
}
// 只有 "你好" 应该被合成
if atomic.LoadInt32(&callCount) != 1 {
t.Errorf("API called %d times, want 1", callCount)
}
// 1 个音频（IsLast: true）+ 1 个 Final
if len(chunks) != 2 {
t.Fatalf("got %d chunks, want 2", len(chunks))
}
}
func TestMiMoService_SynthesizeStream_ContextCancelled(t *testing.T) {
srv := mockMiMoTTSServer(t, func(w http.ResponseWriter, r *http.Request) {
audioB64 := base64.StdEncoding.EncodeToString([]byte("mp3"))
w.Header().Set("Content-Type", "application/json")
w.Write(buildMiMoTTSResponse(audioB64))
})
defer srv.Close()
svc := NewMiMoService("test-key", "mimo-v2.5-tts", "冰糖", srv.URL, 5, 30, zap.NewNop().Sugar())
textStream := make(chan string, 3)
textStream <- "第一句"
textStream <- "第二句"
textStream <- "第三句"
close(textStream)
ctx, cancel := context.WithCancel(context.Background())
// 立即取消
cancel()
ch, err := svc.SynthesizeStream(ctx, textStream, Options{})
if err != nil {
t.Fatalf("SynthesizeStream() error: %v", err)
}
// 消费 channel，应该很快结束
var count int
for range ch {
count++
}
// 可能收到 0 个或 1 个 chunk，取决于时序
t.Logf("received %d chunks after context cancel", count)
}
func TestMiMoService_SynthesizeStream_PartialFailure(t *testing.T) {
var callCount int32
srv := mockMiMoTTSServer(t, func(w http.ResponseWriter, r *http.Request) {
n := atomic.AddInt32(&callCount, 1)
if n == 2 {
// 第二个句子失败
w.WriteHeader(http.StatusInternalServerError)
fmt.Fprintf(w, "error")
return
}
audioB64 := base64.StdEncoding.EncodeToString([]byte(fmt.Sprintf("mp3-%d", n)))
w.Header().Set("Content-Type", "application/json")
w.Write(buildMiMoTTSResponse(audioB64))
})
defer srv.Close()
svc := NewMiMoService("test-key", "mimo-v2.5-tts", "冰糖", srv.URL, 5, 30, zap.NewNop().Sugar())
textStream := sendSentences("第一句", "第二句", "第三句")
ch, err := svc.SynthesizeStream(context.Background(), textStream, Options{})
if err != nil {
t.Fatalf("SynthesizeStream() error: %v", err)
}
var chunks []Chunk
for c := range ch {
chunks = append(chunks, c)
}
// 2 个成功音频（IsLast: true）+ 1 个 Final（第二句被跳过）
if len(chunks) != 3 {
t.Fatalf("got %d chunks, want 3", len(chunks))
}
if !chunks[0].IsLast {
t.Error("first audio chunk should be IsLast")
}
if !chunks[1].IsLast {
t.Error("second audio chunk should be IsLast")
}
if !chunks[len(chunks)-1].Final {
t.Error("last chunk should be Final")
}
}
func TestMiMoService_SynthesizeStream_CustomVoice(t *testing.T) {
srv := mockMiMoTTSServer(t, func(w http.ResponseWriter, r *http.Request) {
body, _ := io.ReadAll(r.Body)
var req mimoTTSRequest
if err := json.Unmarshal(body, &req); err != nil {
t.Errorf("unmarshal request: %v", err)
}
if req.Audio.Voice != "茉莉" {
t.Errorf("voice = %q, want %q", req.Audio.Voice, "茉莉")
}
audioB64 := base64.StdEncoding.EncodeToString([]byte("mp3"))
w.Header().Set("Content-Type", "application/json")
w.Write(buildMiMoTTSResponse(audioB64))
})
defer srv.Close()
svc := NewMiMoService("test-key", "mimo-v2.5-tts", "冰糖", srv.URL, 5, 30, zap.NewNop().Sugar())
textStream := sendSentences("你好")
ch, err := svc.SynthesizeStream(context.Background(), textStream, Options{Voice: "茉莉"})
if err != nil {
t.Fatalf("SynthesizeStream() error: %v", err)
}
for range ch {
}
}
func TestMiMoService_SynthesizeStream_DefaultVoice(t *testing.T) {
srv := mockMiMoTTSServer(t, func(w http.ResponseWriter, r *http.Request) {
body, _ := io.ReadAll(r.Body)
var req mimoTTSRequest
if err := json.Unmarshal(body, &req); err != nil {
t.Errorf("unmarshal request: %v", err)
}
// 未指定 voice 时应使用默认 "冰糖"
if req.Audio.Voice != "冰糖" {
t.Errorf("voice = %q, want %q (default)", req.Audio.Voice, "冰糖")
}
audioB64 := base64.StdEncoding.EncodeToString([]byte("mp3"))
w.Header().Set("Content-Type", "application/json")
w.Write(buildMiMoTTSResponse(audioB64))
})
defer srv.Close()
// 不指定 voice
svc := NewMiMoService("test-key", "mimo-v2.5-tts", "冰糖", srv.URL, 5, 30, zap.NewNop().Sugar())
textStream := sendSentences("你好")
ch, err := svc.SynthesizeStream(context.Background(), textStream, Options{})
if err != nil {
t.Fatalf("SynthesizeStream() error: %v", err)
}
for range ch {
}
}
func TestMiMoService_SynthesizeStream_EmptyAudioData(t *testing.T) {
srv := mockMiMoTTSServer(t, func(w http.ResponseWriter, r *http.Request) {
// 返回空音频数据
w.Header().Set("Content-Type", "application/json")
w.Write(buildMiMoTTSResponse(""))
})
defer srv.Close()
svc := NewMiMoService("test-key", "mimo-v2.5-tts", "冰糖", srv.URL, 5, 30, zap.NewNop().Sugar())
textStream := sendSentences("你好")
ch, err := svc.SynthesizeStream(context.Background(), textStream, Options{})
if err != nil {
t.Fatalf("SynthesizeStream() error: %v", err)
}
var chunks []Chunk
for c := range ch {
chunks = append(chunks, c)
}
// 空音频数据导致错误，句子被跳过，只有 Final
if len(chunks) != 1 {
t.Fatalf("got %d chunks, want 1", len(chunks))
}
if !chunks[0].Final {
t.Error("chunk should be Final")
}
}

24
backend/internal/ai/tts/openai.go
View File

@@ -25,23 +25,19 @@ type OpenAIService struct {
}
// NewOpenAIService 创建 OpenAI TTS 服务。
func NewOpenAIService(apiKey, model, voice, endpoint string, speed float64, timeoutSec int, logger *zap.SugaredLogger) *OpenAIService {
if model == "" {
model = "tts-1"
}
if voice == "" {
voice = "alloy"
}
// model、voice、endpoint 由 config 层保证非空。
func NewOpenAIService(apiKey, model, voice, endpoint string, speed float64, timeoutSec, httpClientTimeoutSec int, logger *zap.SugaredLogger) *OpenAIService {
if speed <= 0 {
speed = 1.0
}
if endpoint == "" {
endpoint = "https://api.openai.com/v1"
}
timeout := time.Duration(timeoutSec) * time.Second
if timeout <= 0 {
timeout = 5 * time.Second
}
httpClientTimeout := time.Duration(httpClientTimeoutSec) * time.Second
if httpClientTimeout <= 0 {
httpClientTimeout = 30 * time.Second
}
return &OpenAIService{
apiKey: apiKey,
model: model,
@@ -50,7 +46,7 @@ func NewOpenAIService(apiKey, model, voice, endpoint string, speed float64, time
endpoint: endpoint,
timeout: timeout,
logger: logger,
client: &http.Client{Timeout: 30 * time.Second},
client: &http.Client{Timeout: httpClientTimeout},
}
}
@@ -91,15 +87,15 @@ func (o *OpenAIService) SynthesizeStream(ctx context.Context, textStream <-chan
}
select {
case ch <- Chunk{Audio: audio, IsLast: false}:
case ch <- Chunk{Audio: audio, IsLast: true, Final: false}:
case <-ctx.Done():
return
}
}
// textStream 关闭，发送 IsLast 标记
// textStream 关闭，发送 Final 标记
select {
case ch <- Chunk{Audio: nil, IsLast: true}:
case ch <- Chunk{Audio: nil, IsLast: false, Final: true}:
case <-ctx.Done():
}
}()

59
backend/internal/ai/tts/openai_test.go
View File

@@ -58,7 +58,7 @@ func TestOpenAIService_SynthesizeStream_Success(t *testing.T) {
})
defer srv.Close()
svc := NewOpenAIService("test-key", "", "alloy", srv.URL, 1.0, 5, zap.NewNop().Sugar())
svc := NewOpenAIService("test-key", "tts-1", "alloy", srv.URL, 1.0, 5, 30, zap.NewNop().Sugar())
textStream := sendSentences("你好", "世界", "！")
@@ -74,24 +74,27 @@ func TestOpenAIService_SynthesizeStream_Success(t *testing.T) {
chunks = append(chunks, c)
}
// 应该有 3 个音频 chunk + 1 个 IsLast 标记
// 应该有 3 个音频 chunk + 1 个 Final 标记
if len(chunks) != 4 {
t.Fatalf("got %d chunks, want 4", len(chunks))
}
// 验证前 3 个有音频数据
// 验证前 3 个有音频数据，IsLast 为 true（每句结束）
for i := 0; i < 3; i++ {
if string(chunks[i].Audio) != "fake-mp3-data" {
t.Errorf("chunk[%d].Audio = %q, want %q", i, string(chunks[i].Audio), "fake-mp3-data")
}
if chunks[i].IsLast {
t.Errorf("chunk[%d].IsLast should be false", i)
if !chunks[i].IsLast {
t.Errorf("chunk[%d].IsLast should be true (sentence end)", i)
}
if chunks[i].Final {
t.Errorf("chunk[%d].Final should be false", i)
}
}
// 验证最后一个是 IsLast
if !chunks[3].IsLast {
t.Error("last chunk should be IsLast")
// 验证最后一个是 Final（整轮结束）
if !chunks[3].Final {
t.Error("last chunk should be Final")
}
if chunks[3].Audio != nil {
t.Error("last chunk Audio should be nil")
@@ -110,7 +113,7 @@ func TestOpenAIService_SynthesizeStream_APIError(t *testing.T) {
})
defer srv.Close()
svc := NewOpenAIService("test-key", "", "alloy", srv.URL, 1.0, 5, zap.NewNop().Sugar())
svc := NewOpenAIService("test-key", "tts-1", "alloy", srv.URL, 1.0, 5, 30, zap.NewNop().Sugar())
textStream := sendSentences("你好")
@@ -119,17 +122,17 @@ func TestOpenAIService_SynthesizeStream_APIError(t *testing.T) {
t.Fatalf("SynthesizeStream() error: %v", err)
}
// 应该只有一个 IsLast chunk（音频被跳过）
// 应该只有一个 Final chunk（音频被跳过）
var chunks []Chunk
for c := range ch {
chunks = append(chunks, c)
}
if len(chunks) != 1 {
t.Fatalf("got %d chunks, want 1 (IsLast only)", len(chunks))
t.Fatalf("got %d chunks, want 1 (Final only)", len(chunks))
}
if !chunks[0].IsLast {
t.Error("chunk should be IsLast")
if !chunks[0].Final {
t.Error("chunk should be Final")
}
}
@@ -142,7 +145,7 @@ func TestOpenAIService_SynthesizeStream_Timeout(t *testing.T) {
defer srv.Close()
// 1 秒超时
svc := NewOpenAIService("test-key", "", "alloy", srv.URL, 1.0, 1, zap.NewNop().Sugar())
svc := NewOpenAIService("test-key", "tts-1", "alloy", srv.URL, 1.0, 1, 30, zap.NewNop().Sugar())
textStream := sendSentences("很长的句子")
@@ -159,12 +162,12 @@ func TestOpenAIService_SynthesizeStream_Timeout(t *testing.T) {
chunks = append(chunks, c)
}
// 超时后音频被跳过，只有 IsLast
// 超时后音频被跳过，只有 Final
if len(chunks) != 1 {
t.Fatalf("got %d chunks, want 1", len(chunks))
}
if !chunks[0].IsLast {
t.Error("chunk should be IsLast")
if !chunks[0].Final {
t.Error("chunk should be Final")
}
}
@@ -177,7 +180,7 @@ func TestOpenAIService_SynthesizeStream_EmptyText(t *testing.T) {
})
defer srv.Close()
svc := NewOpenAIService("test-key", "", "alloy", srv.URL, 1.0, 5, zap.NewNop().Sugar())
svc := NewOpenAIService("test-key", "tts-1", "alloy", srv.URL, 1.0, 5, 30, zap.NewNop().Sugar())
// 空句子应该被跳过
textStream := sendSentences("", "你好", "")
@@ -197,7 +200,7 @@ func TestOpenAIService_SynthesizeStream_EmptyText(t *testing.T) {
t.Errorf("API called %d times, want 1", callCount)
}
// 1 个音频 + 1 个 IsLast
// 1 个音频（IsLast: true）+ 1 个 Final
if len(chunks) != 2 {
t.Fatalf("got %d chunks, want 2", len(chunks))
}
@@ -210,7 +213,7 @@ func TestOpenAIService_SynthesizeStream_ContextCancelled(t *testing.T) {
})
defer srv.Close()
svc := NewOpenAIService("test-key", "", "alloy", srv.URL, 1.0, 5, zap.NewNop().Sugar())
svc := NewOpenAIService("test-key", "tts-1", "alloy", srv.URL, 1.0, 5, 30, zap.NewNop().Sugar())
// 发送多个句子，但在第一个后取消
textStream := make(chan string, 3)
@@ -252,7 +255,7 @@ func TestOpenAIService_SynthesizeStream_PartialFailure(t *testing.T) {
})
defer srv.Close()
svc := NewOpenAIService("test-key", "", "alloy", srv.URL, 1.0, 5, zap.NewNop().Sugar())
svc := NewOpenAIService("test-key", "tts-1", "alloy", srv.URL, 1.0, 5, 30, zap.NewNop().Sugar())
textStream := sendSentences("第一句", "第二句", "第三句")
@@ -266,12 +269,18 @@ func TestOpenAIService_SynthesizeStream_PartialFailure(t *testing.T) {
chunks = append(chunks, c)
}
// 2 个成功音频 + 1 个 IsLast（第二句被跳过）
// 2 个成功音频（IsLast: true）+ 1 个 Final（第二句被跳过）
if len(chunks) != 3 {
t.Fatalf("got %d chunks, want 3", len(chunks))
}
if !chunks[len(chunks)-1].IsLast {
t.Error("last chunk should be IsLast")
if !chunks[0].IsLast {
t.Error("first audio chunk should be IsLast")
}
if !chunks[1].IsLast {
t.Error("second audio chunk should be IsLast")
}
if !chunks[len(chunks)-1].Final {
t.Error("last chunk should be Final")
}
}
@@ -286,7 +295,7 @@ func TestOpenAIService_SynthesizeStream_CustomVoice(t *testing.T) {
})
defer srv.Close()
svc := NewOpenAIService("test-key", "", "alloy", srv.URL, 1.0, 5, zap.NewNop().Sugar())
svc := NewOpenAIService("test-key", "tts-1", "alloy", srv.URL, 1.0, 5, 30, zap.NewNop().Sugar())
textStream := sendSentences("你好")

3
backend/internal/ai/tts/tts.go
View File

@@ -21,5 +21,6 @@ type Options struct {
// Chunk 一个音频片段。
type Chunk struct {
Audio []byte // MP3 音频数据（未 Base64 编码）
IsLast bool // 是否为最后一片
IsLast bool // 当前句子是否为最后一片（每句结束时为 true）
Final bool // 整轮 TTS 是否结束（所有句子合成完毕后为 true，此时 Audio 为 nil）
}

86
backend/internal/config/config.go
View File

@@ -11,12 +11,19 @@ import (
// Config 应用配置。
type Config struct {
App AppConfig `mapstructure:"app"`
Server ServerConfig `mapstructure:"server"`
Redis RedisConfig `mapstructure:"redis"`
AI AIConfig `mapstructure:"ai"`
Storage StorageConfig `mapstructure:"storage"`
Log LogConfig `mapstructure:"log"`
App AppConfig `mapstructure:"app"`
Server ServerConfig `mapstructure:"server"`
Session SessionConfig `mapstructure:"session"`
Redis RedisConfig `mapstructure:"redis"`
AI AIConfig `mapstructure:"ai"`
Storage StorageConfig `mapstructure:"storage"`
Log LogConfig `mapstructure:"log"`
}
// SessionConfig 会话管理配置。
type SessionConfig struct {
TTL int `mapstructure:"ttl"` // 会话过期时间（分钟）
MaxHistory int `mapstructure:"max_history"` // 对话历史上限（条）
}
type AppConfig struct {
@@ -25,10 +32,14 @@ type AppConfig struct {
}
type ServerConfig struct {
Host string `mapstructure:"host"`
Port int `mapstructure:"port"`
ReadTimeout int `mapstructure:"read_timeout"`
WriteTimeout int `mapstructure:"write_timeout"`
Host string `mapstructure:"host"`
Port int `mapstructure:"port"`
ReadTimeout int `mapstructure:"read_timeout"`
WriteTimeout int `mapstructure:"write_timeout"`
HeartbeatInterval int `mapstructure:"heartbeat_interval"` // 心跳检查间隔（秒）
HeartbeatTimeout int `mapstructure:"heartbeat_timeout"` // 心跳超时（秒）
ShutdownTimeout int `mapstructure:"shutdown_timeout"` // 优雅关闭超时（秒）
AllowedOrigins []string `mapstructure:"allowed_origins"` // CORS 允许的来源，空表示允许所有
}
// Addr 返回 host:port 地址。
@@ -49,28 +60,34 @@ type AIConfig struct {
}
type STTConfig struct {
Provider string `mapstructure:"provider"`
APIKey string `mapstructure:"api_key"`
Model string `mapstructure:"model"`
Endpoint string `mapstructure:"endpoint"`
Provider string `mapstructure:"provider"`
APIKey string `mapstructure:"api_key"`
Model string `mapstructure:"model"`
Endpoint string `mapstructure:"endpoint"`
Timeout int `mapstructure:"timeout"` // STT 超时（秒）
HTTPClientTimeout int `mapstructure:"http_client_timeout"` // HTTP 客户端超时（秒）
}
type LLMConfig struct {
Provider string `mapstructure:"provider"`
APIKey string `mapstructure:"api_key"`
Model string `mapstructure:"model"`
Endpoint string `mapstructure:"endpoint"`
Timeout int `mapstructure:"timeout"`
Provider string `mapstructure:"provider"`
APIKey string `mapstructure:"api_key"`
Model string `mapstructure:"model"`
Endpoint string `mapstructure:"endpoint"`
Timeout int `mapstructure:"timeout"`
HTTPClientTimeout int `mapstructure:"http_client_timeout"` // HTTP 客户端超时（秒）
}
type TTSConfig struct {
Provider string `mapstructure:"provider"`
APIKey string `mapstructure:"api_key"`
Model string `mapstructure:"model"`
Voice string `mapstructure:"voice"`
Speed float64 `mapstructure:"speed"`
Endpoint string `mapstructure:"endpoint"`
Timeout int `mapstructure:"timeout"`
Provider string `mapstructure:"provider"`
APIKey string `mapstructure:"api_key"`
Model string `mapstructure:"model"`
Voice string `mapstructure:"voice"`
Speed float64 `mapstructure:"speed"`
Endpoint string `mapstructure:"endpoint"`
Timeout int `mapstructure:"timeout"`
HTTPClientTimeout int `mapstructure:"http_client_timeout"` // HTTP 客户端超时（秒）
OutputFormat string `mapstructure:"output_format"` // 输出格式：mp3/wav
SampleRate int `mapstructure:"sample_rate"` // 输出采样率
}
type StorageConfig struct {
@@ -91,28 +108,42 @@ func Load() (*Config, error) {
v.AddConfigPath(".")
v.AddConfigPath("./config")
v.AddConfigPath("./backend")
v.AddConfigPath("..") // 兼容从 backend/cmd/ 启动
v.AddConfigPath("../..") // 兼容从 backend/cmd/server/ 启动
// 默认值
v.SetDefault("app.env", "dev")
v.SetDefault("app.version", "dev")
v.SetDefault("server.host", "0.0.0.0")
v.SetDefault("server.port", 8080)
v.SetDefault("server.read_timeout", 30)
v.SetDefault("server.write_timeout", 30)
v.SetDefault("server.heartbeat_interval", 30)
v.SetDefault("server.heartbeat_timeout", 60)
v.SetDefault("server.shutdown_timeout", 10)
v.SetDefault("session.ttl", 30)
v.SetDefault("session.max_history", 20)
v.SetDefault("redis.addr", "localhost:6379")
v.SetDefault("redis.db", 0)
v.SetDefault("ai.stt.provider", "deepgram")
v.SetDefault("ai.stt.model", "nova-2")
v.SetDefault("ai.stt.endpoint", "wss://api.deepgram.com/v1/listen")
v.SetDefault("ai.stt.timeout", 5)
v.SetDefault("ai.stt.http_client_timeout", 30)
v.SetDefault("ai.llm.provider", "openai")
v.SetDefault("ai.llm.model", "gpt-4o")
v.SetDefault("ai.llm.endpoint", "https://api.openai.com/v1")
v.SetDefault("ai.llm.timeout", 10)
v.SetDefault("ai.llm.http_client_timeout", 60)
v.SetDefault("ai.tts.provider", "openai")
v.SetDefault("ai.tts.model", "tts-1")
v.SetDefault("ai.tts.voice", "alloy")
v.SetDefault("ai.tts.voice", "mimo_default")
v.SetDefault("ai.tts.speed", 1.0)
v.SetDefault("ai.tts.endpoint", "https://api.openai.com/v1")
v.SetDefault("ai.tts.timeout", 5)
v.SetDefault("ai.tts.http_client_timeout", 30)
v.SetDefault("ai.tts.output_format", "mp3")
v.SetDefault("ai.tts.sample_rate", 24000)
v.SetDefault("storage.driver", "memory")
v.SetDefault("log.level", "info")
v.SetDefault("log.format", "console")
@@ -134,6 +165,7 @@ func Load() (*Config, error) {
// 按优先级尝试：当前目录、上级目录（兼容从 backend/ 或项目根目录启动）
_ = godotenv.Load()
_ = godotenv.Load("../.env")
_ = godotenv.Load("../../.env") // 兼容从 backend/cmd/server/ 启动
// 环境变量覆盖
v.SetEnvPrefix("CAMTALK")

4
backend/internal/models/models.go
View File

@@ -55,6 +55,7 @@ type WsQuery struct {
RequestID string `json:"request_id"`
Image string `json:"image"` // base64
Audio string `json:"audio"` // base64
Text string `json:"text"` // 用户手动输入的文本（有值时跳过 STT）
MimeType string `json:"mime_type"` // 默认 "audio/pcm"
}
@@ -111,7 +112,8 @@ type WsTTSAudio struct {
RequestID string `json:"request_id"`
Audio string `json:"audio"` // base64
MimeType string `json:"mime_type"` // "audio/mp3" 或 "audio/pcm"
IsLast bool `json:"is_last"`
IsLast bool `json:"is_last"` // 当前句子的音频是否完整（每句结束时为 true）
Final bool `json:"final"` // 整轮 TTS 是否结束（所有句子合成完毕后为 true）
}
// WsError 服务端 error 消息。

176
backend/internal/orchestrator/pipeline.go
View File

@@ -11,6 +11,7 @@ import (
"github.com/hhs/camtalk/internal/ai/llm"
"github.com/hhs/camtalk/internal/ai/stt"
"github.com/hhs/camtalk/internal/ai/tts"
"github.com/hhs/camtalk/internal/config"
"github.com/hhs/camtalk/internal/logger"
"github.com/hhs/camtalk/internal/models"
"github.com/hhs/camtalk/internal/session"
@@ -18,11 +19,15 @@ import (
// Pipeline 实现 Orchestrator 接口，管理 STT → LLM → TTS 流式管道。
type Pipeline struct {
sttService stt.Service
llmService llm.Service
ttsService tts.Service
sessionMgr session.Manager
model string // LLM 模型名，用于 llm_done 上报
sttService stt.Service
llmService llm.Service
ttsService tts.Service
sessionMgr session.Manager
model string // LLM 模型名，用于 llm_done 上报
ttsVoice string // TTS 音色
ttsSpeed float64 // TTS 语速
ttsOutputFmt string // TTS 输出格式
ttsSampleRate int // TTS 输出采样率
}
// New 创建 Pipeline 实例。
@@ -31,14 +36,18 @@ func New(
llmService llm.Service,
ttsService tts.Service,
sessionMgr session.Manager,
model string,
cfg *config.Config,
) *Pipeline {
return &Pipeline{
sttService: sttService,
llmService: llmService,
ttsService: ttsService,
sessionMgr: sessionMgr,
model: model,
sttService: sttService,
llmService: llmService,
ttsService: ttsService,
sessionMgr: sessionMgr,
model: cfg.AI.LLM.Model,
ttsVoice: cfg.AI.TTS.Voice,
ttsSpeed: cfg.AI.TTS.Speed,
ttsOutputFmt: cfg.AI.TTS.OutputFormat,
ttsSampleRate: cfg.AI.TTS.SampleRate,
}
}
@@ -53,22 +62,27 @@ func (p *Pipeline) ProcessQuery(
log := logger.Log
startTime := time.Now()
// 解码音频数据
audio, err := base64.StdEncoding.DecodeString(req.Audio)
if err != nil {
log.Errorw("音频解码失败", "error", err)
sender.SendError(models.WsError{
Type: "error",
RequestID: req.RequestID,
Code: "INVALID_MESSAGE",
Message: "音频数据解码失败",
})
return err
// 解码音频数据（文本输入模式可跳过）
var audio []byte
if req.Text == "" && req.Audio != "" {
var err error
audio, err = base64.StdEncoding.DecodeString(req.Audio)
if err != nil {
log.Errorw("音频解码失败", "error", err)
sender.SendError(models.WsError{
Type: "error",
RequestID: req.RequestID,
Code: "INVALID_MESSAGE",
Message: "音频数据解码失败",
})
return err
}
}
// 解码图片数据（可选）
var image []byte
if req.Image != "" {
var err error
image, err = base64.StdEncoding.DecodeString(req.Image)
if err != nil {
log.Errorw("图片解码失败", "error", err)
@@ -101,45 +115,64 @@ func (p *Pipeline) ProcessQuery(
return err
}
// Step 1: STT 语音识别
log.Infow("开始语音识别", "request_id", req.RequestID)
sttResult, err := p.sttService.Recognize(ctx, audio, stt.Options{
Encoding: "pcm_s16le",
SampleRate: 16000,
Language: sess.Config.Language,
})
if err != nil {
log.Errorw("语音识别失败", "error", err)
sender.SendError(models.WsError{
Type: "error",
RequestID: req.RequestID,
Code: "STT_ERROR",
Message: "语音识别失败",
})
return err
}
// Step 1: 获取用户文本（语音识别或直接使用输入文本）
var userText string
if req.Text != "" {
// 文本输入模式：跳过 STT，直接使用用户输入的文本
log.Infow("使用文本输入", "request_id", req.RequestID, "text", req.Text)
userText = req.Text
// 发送 STT 结果
if err := sender.SendSTTResult(models.WsSTTResult{
Type: "stt_result",
RequestID: req.RequestID,
Text: sttResult,
IsFinal: true,
}); err != nil {
log.Errorw("发送 STT 结果失败", "error", err)
// 发送 stt_result 以保持前端消息流一致性
if err := sender.SendSTTResult(models.WsSTTResult{
Type: "stt_result",
RequestID: req.RequestID,
Text: userText,
IsFinal: true,
}); err != nil {
log.Errorw("发送 STT 结果失败", "error", err)
}
} else {
// 语音模式：执行 STT 语音识别
log.Infow("开始语音识别", "request_id", req.RequestID)
sttResult, err := p.sttService.Recognize(ctx, audio, stt.Options{
Encoding: "pcm_s16le",
SampleRate: 16000,
Language: sess.Config.Language,
})
if err != nil {
log.Errorw("语音识别失败", "error", err)
sender.SendError(models.WsError{
Type: "error",
RequestID: req.RequestID,
Code: "STT_ERROR",
Message: "语音识别失败",
})
return err
}
userText = sttResult
// 发送 STT 结果
if err := sender.SendSTTResult(models.WsSTTResult{
Type: "stt_result",
RequestID: req.RequestID,
Text: userText,
IsFinal: true,
}); err != nil {
log.Errorw("发送 STT 结果失败", "error", err)
}
}
// 追加用户消息到历史
p.sessionMgr.AppendMessage(ctx, sessionID, models.Message{
Role: "user",
Content: sttResult,
Content: userText,
})
// Step 2+3: LLM 流式推理 + TTS 并行合成
log.Infow("开始 LLM 推理", "request_id", req.RequestID)
llmReq := llm.Request{
Image: image,
Text: sttResult,
Text: userText,
History: history,
Language: sess.Config.Language,
}
@@ -166,11 +199,12 @@ func (p *Pipeline) ProcessQuery(
var ttsErr error
// goroutine 1: 消费 LLM token + 句子切分
var tokenUsage *llm.TokenUsage
wg.Add(1)
go func() {
defer wg.Done()
defer close(sentenceCh)
fullText = p.consumeLLMStream(ctx, llmStream, req.RequestID, sender, splitter)
fullText, tokenUsage = p.consumeLLMStream(ctx, llmStream, req.RequestID, sender, splitter)
}()
// goroutine 2: TTS 合成（如果启用）
@@ -205,13 +239,25 @@ func (p *Pipeline) ProcessQuery(
// 发送 llm_done
latency := time.Since(startTime).Milliseconds()
if err := sender.SendLLMDone(models.WsLLMDone{
done := models.WsLLMDone{
Type: "llm_done",
RequestID: req.RequestID,
FullText: fullText,
Model: p.model,
LatencyMs: latency,
}); err != nil {
}
if tokenUsage != nil {
done.TokensUsed = struct {
Prompt int `json:"prompt"`
Completion int `json:"completion"`
Total int `json:"total"`
}{
Prompt: tokenUsage.Prompt,
Completion: tokenUsage.Completion,
Total: tokenUsage.Total,
}
}
if err := sender.SendLLMDone(done); err != nil {
log.Errorw("发送 llm_done 失败", "error", err)
}
@@ -225,33 +271,36 @@ func (p *Pipeline) ProcessQuery(
}
// consumeLLMStream 消费 LLM 流式输出，发送 llm_chunk 并进行句子切分。
// 返回完整文本和 token 用量。
func (p *Pipeline) consumeLLMStream(
ctx context.Context,
stream <-chan llm.Chunk,
requestID string,
sender Sender,
splitter *Splitter,
) string {
) (string, *llm.TokenUsage) {
log := logger.Log
var fullText strings.Builder
var tokenUsage *llm.TokenUsage
for chunk := range stream {
// 检查上下文是否已取消
select {
case <-ctx.Done():
log.Infow("LLM 流被中断", "request_id", requestID)
return fullText.String()
return fullText.String(), tokenUsage
default:
}
if chunk.Done {
// 流结束
// 流结束，记录 token 用量
if chunk.TokensUsed != nil {
tokenUsage = chunk.TokensUsed
log.Infow("LLM 用量统计",
"request_id", requestID,
"prompt_tokens", chunk.TokensUsed.Prompt,
"completion_tokens", chunk.TokensUsed.Completion,
"total_tokens", chunk.TokensUsed.Total,
"prompt_tokens", tokenUsage.Prompt,
"completion_tokens", tokenUsage.Completion,
"total_tokens", tokenUsage.Total,
)
}
break
@@ -277,7 +326,7 @@ func (p *Pipeline) consumeLLMStream(
// 刷新切分器中的剩余文本
splitter.Flush()
return fullText.String()
return fullText.String(), tokenUsage
}
// synthesizeTTS 从句子 channel 读取文本，进行 TTS 合成并发送音频。
@@ -290,10 +339,10 @@ func (p *Pipeline) synthesizeTTS(
log := logger.Log
ttsStream, err := p.ttsService.SynthesizeStream(ctx, sentenceCh, tts.Options{
Voice: "alloy",
Speed: 1.0,
OutputFmt: "mp3",
SampleRate: 24000,
Voice: p.ttsVoice,
Speed: p.ttsSpeed,
OutputFmt: p.ttsOutputFmt,
SampleRate: p.ttsSampleRate,
})
if err != nil {
log.Errorw("TTS 合成启动失败", "error", err)
@@ -319,6 +368,7 @@ func (p *Pipeline) synthesizeTTS(
Audio: audioBase64,
MimeType: "audio/mp3",
IsLast: chunk.IsLast,
Final: chunk.Final,
}); err != nil {
log.Errorw("发送 tts_audio 失败", "error", err)
}

57
backend/internal/orchestrator/pipeline_test.go
View File

@@ -13,6 +13,7 @@ import (
"github.com/hhs/camtalk/internal/ai/llm"
"github.com/hhs/camtalk/internal/ai/stt"
"github.com/hhs/camtalk/internal/ai/tts"
"github.com/hhs/camtalk/internal/config"
"github.com/hhs/camtalk/internal/logger"
"github.com/hhs/camtalk/internal/models"
)
@@ -254,7 +255,12 @@ func TestProcessQuery_Success(t *testing.T) {
mockSender.On("SendTTSAudio", mock.Anything).Return(nil)
// 创建 Pipeline
pipeline := New(mockSTT, mockLLM, mockTTS, mockSession, "gpt-4o")
pipeline := New(mockSTT, mockLLM, mockTTS, mockSession, &config.Config{
AI: config.AIConfig{
LLM: config.LLMConfig{Model: "gpt-4o"},
TTS: config.TTSConfig{Voice: "alloy", Speed: 1.0, OutputFormat: "mp3", SampleRate: 24000},
},
})
// 执行
ctx := context.Background()
@@ -305,7 +311,12 @@ func TestProcessQuery_STTError(t *testing.T) {
mockSender.On("SendError", mock.Anything).Return(nil)
pipeline := New(mockSTT, mockLLM, mockTTS, mockSession, "gpt-4o")
pipeline := New(mockSTT, mockLLM, mockTTS, mockSession, &config.Config{
AI: config.AIConfig{
LLM: config.LLMConfig{Model: "gpt-4o"},
TTS: config.TTSConfig{Voice: "alloy", Speed: 1.0, OutputFormat: "mp3", SampleRate: 24000},
},
})
ctx := context.Background()
err := pipeline.ProcessQuery(ctx, "session-123", req, nil, mockSender)
@@ -357,7 +368,12 @@ func TestProcessQuery_LLMError(t *testing.T) {
mockSender.On("SendError", mock.Anything).Return(nil)
pipeline := New(mockSTT, mockLLM, mockTTS, mockSession, "gpt-4o")
pipeline := New(mockSTT, mockLLM, mockTTS, mockSession, &config.Config{
AI: config.AIConfig{
LLM: config.LLMConfig{Model: "gpt-4o"},
TTS: config.TTSConfig{Voice: "alloy", Speed: 1.0, OutputFormat: "mp3", SampleRate: 24000},
},
})
ctx := context.Background()
err := pipeline.ProcessQuery(ctx, "session-123", req, nil, mockSender)
@@ -415,7 +431,12 @@ func TestProcessQuery_TTSError(t *testing.T) {
mockTTS.On("SynthesizeStream", mock.Anything, mock.Anything, mock.Anything).
Return(nil, errors.New("TTS service unavailable"))
pipeline := New(mockSTT, mockLLM, mockTTS, mockSession, "gpt-4o")
pipeline := New(mockSTT, mockLLM, mockTTS, mockSession, &config.Config{
AI: config.AIConfig{
LLM: config.LLMConfig{Model: "gpt-4o"},
TTS: config.TTSConfig{Voice: "alloy", Speed: 1.0, OutputFormat: "mp3", SampleRate: 24000},
},
})
ctx := context.Background()
err := pipeline.ProcessQuery(ctx, "session-123", req, nil, mockSender)
@@ -485,7 +506,12 @@ func TestProcessQuery_ContextCancelled(t *testing.T) {
}()
mockTTS.On("SynthesizeStream", mock.Anything, mock.Anything, mock.Anything).Return((<-chan tts.Chunk)(ttsCh), nil)
pipeline := New(mockSTT, mockLLM, mockTTS, mockSession, "gpt-4o")
pipeline := New(mockSTT, mockLLM, mockTTS, mockSession, &config.Config{
AI: config.AIConfig{
LLM: config.LLMConfig{Model: "gpt-4o"},
TTS: config.TTSConfig{Voice: "alloy", Speed: 1.0, OutputFormat: "mp3", SampleRate: 24000},
},
})
// 创建可取消的上下文
ctx, cancel := context.WithCancel(context.Background())
@@ -545,7 +571,12 @@ func TestProcessQuery_DisabledTTS(t *testing.T) {
mockSender.On("SendLLMChunk", mock.Anything).Return(nil)
mockSender.On("SendLLMDone", mock.Anything).Return(nil)
pipeline := New(mockSTT, mockLLM, mockTTS, mockSession, "gpt-4o")
pipeline := New(mockSTT, mockLLM, mockTTS, mockSession, &config.Config{
AI: config.AIConfig{
LLM: config.LLMConfig{Model: "gpt-4o"},
TTS: config.TTSConfig{Voice: "alloy", Speed: 1.0, OutputFormat: "mp3", SampleRate: 24000},
},
})
ctx := context.Background()
err := pipeline.ProcessQuery(ctx, "session-123", req, nil, mockSender)
@@ -617,7 +648,12 @@ func TestProcessQuery_InvalidAudio(t *testing.T) {
mockSender.On("SendError", mock.Anything).Return(nil)
pipeline := New(mockSTT, mockLLM, mockTTS, mockSession, "gpt-4o")
pipeline := New(mockSTT, mockLLM, mockTTS, mockSession, &config.Config{
AI: config.AIConfig{
LLM: config.LLMConfig{Model: "gpt-4o"},
TTS: config.TTSConfig{Voice: "alloy", Speed: 1.0, OutputFormat: "mp3", SampleRate: 24000},
},
})
ctx := context.Background()
err := pipeline.ProcessQuery(ctx, "session-123", req, nil, mockSender)
@@ -650,7 +686,12 @@ func TestProcessQuery_SessionNotFound(t *testing.T) {
mockSender.On("SendError", mock.Anything).Return(nil)
pipeline := New(mockSTT, mockLLM, mockTTS, mockSession, "gpt-4o")
pipeline := New(mockSTT, mockLLM, mockTTS, mockSession, &config.Config{
AI: config.AIConfig{
LLM: config.LLMConfig{Model: "gpt-4o"},
TTS: config.TTSConfig{Voice: "alloy", Speed: 1.0, OutputFormat: "mp3", SampleRate: 24000},
},
})
ctx := context.Background()
err := pipeline.ProcessQuery(ctx, "session-123", req, nil, mockSender)

43
backend/internal/ws/handler.go
View File

@@ -10,6 +10,7 @@ import (
"github.com/gin-gonic/gin"
"github.com/gorilla/websocket"
"github.com/hhs/camtalk/internal/config"
"github.com/hhs/camtalk/internal/errors"
"github.com/hhs/camtalk/internal/logger"
"github.com/hhs/camtalk/internal/models"
@@ -17,8 +18,23 @@ import (
"github.com/hhs/camtalk/internal/session"
)
var upgrader = websocket.Upgrader{
CheckOrigin: func(r *http.Request) bool { return true }, // 开发阶段允许所有来源
// newUpgrader 根据配置创建 WebSocket upgrader。
func newUpgrader(cfg *config.Config) websocket.Upgrader {
allowedOrigins := cfg.Server.AllowedOrigins
return websocket.Upgrader{
CheckOrigin: func(r *http.Request) bool {
if len(allowedOrigins) == 0 {
return true // 未配置则允许所有来源（开发模式）
}
origin := r.Header.Get("Origin")
for _, o := range allowedOrigins {
if o == origin || o == "*" {
return true
}
}
return false
},
}
}
// Client 代表一个 WebSocket 客户端连接。
@@ -75,13 +91,21 @@ func (w *WSClient) SendError(err models.WsError) error {
}
// ServeWS 处理 WebSocket 升级请求。
func ServeWS(sessionMgr session.Manager, orch orchestrator.Orchestrator) gin.HandlerFunc {
func ServeWS(sessionMgr session.Manager, orch orchestrator.Orchestrator, cfg *config.Config) gin.HandlerFunc {
upgrader := newUpgrader(cfg)
heartbeatInterval := time.Duration(cfg.Server.HeartbeatInterval) * time.Second
heartbeatTimeout := time.Duration(cfg.Server.HeartbeatTimeout) * time.Second
version := cfg.App.Version
maxHistory := cfg.Session.MaxHistory
return func(c *gin.Context) {
serveWS(c, sessionMgr, orch)
serveWS(c, sessionMgr, orch, upgrader, heartbeatInterval, heartbeatTimeout, version, maxHistory)
}
}
func serveWS(c *gin.Context, sessionMgr session.Manager, orch orchestrator.Orchestrator) {
func serveWS(c *gin.Context, sessionMgr session.Manager, orch orchestrator.Orchestrator,
upgrader websocket.Upgrader, heartbeatInterval, heartbeatTimeout time.Duration, version string, maxHistory int) {
conn, err := upgrader.Upgrade(c.Writer, c.Request, nil)
if err != nil {
logger.Log.Errorw("websocket upgrade failed", "error", err)
@@ -108,7 +132,7 @@ func serveWS(c *gin.Context, sessionMgr session.Manager, orch orchestrator.Orche
_ = client.SendJSON(models.WsConnected{
Type: "connected",
SessionID: sessionID,
ServerVersion: "0.1.0",
ServerVersion: version,
})
logger.Log.Infow("client connected", "session", sessionID)
@@ -122,12 +146,12 @@ func serveWS(c *gin.Context, sessionMgr session.Manager, orch orchestrator.Orche
// 启动心跳检查 goroutine
done := make(chan struct{})
go func() {
ticker := time.NewTicker(30 * time.Second)
ticker := time.NewTicker(heartbeatInterval)
defer ticker.Stop()
for {
select {
case <-ticker.C:
if time.Since(lastPong) > 60*time.Second {
if time.Since(lastPong) > heartbeatTimeout {
logger.Log.Warnw("heartbeat timeout", "session", sessionID)
conn.Close()
return
@@ -159,6 +183,7 @@ func serveWS(c *gin.Context, sessionMgr session.Manager, orch orchestrator.Orche
switch envelope.Type {
case "ping":
lastPong = time.Now() // 刷新心跳计时器
_ = client.SendJSON(models.WsPong{Type: "pong"})
case "query":
@@ -180,7 +205,7 @@ func serveWS(c *gin.Context, sessionMgr session.Manager, orch orchestrator.Orche
}
// 获取对话历史
history, _ := client.sessionMgr.GetHistory(context.Background(), sessionID, 20)
history, _ := client.sessionMgr.GetHistory(context.Background(), sessionID, maxHistory)
// 创建可取消的 context
ctx, cancel := context.WithCancel(context.Background())

10
backend/internal/ws/handler_test.go
View File

@@ -13,6 +13,7 @@ import (
"github.com/stretchr/testify/require"
"context"
"github.com/hhs/camtalk/internal/config"
"github.com/hhs/camtalk/internal/logger"
"github.com/hhs/camtalk/internal/models"
"github.com/hhs/camtalk/internal/orchestrator"
@@ -138,7 +139,12 @@ func setupTestServer(t *testing.T, orch orchestrator.Orchestrator) (*httptest.Se
t.Cleanup(func() { sessionMgr.Stop() })
r := gin.New()
r.GET("/ws", ServeWS(sessionMgr, orch))
cfg := &config.Config{
App: config.AppConfig{Version: "test"},
Server: config.ServerConfig{HeartbeatInterval: 30, HeartbeatTimeout: 60},
Session: config.SessionConfig{MaxHistory: 20},
}
r.GET("/ws", ServeWS(sessionMgr, orch, cfg))
srv := httptest.NewServer(r)
@@ -181,7 +187,7 @@ func TestWS_Connected(t *testing.T) {
msg := readJSON(t, conn)
assert.Equal(t, "connected", msg["type"])
assert.NotEmpty(t, msg["session_id"])
assert.Equal(t, "0.1.0", msg["server_version"])
assert.Equal(t, "test", msg["server_version"])
}
// TestWS_PingPong 验证 ping/pong 心跳。

91
deploy.sh Executable file
View File

@@ -0,0 +1,91 @@
#!/usr/bin/env bash
set -euo pipefail
PROJECT_DIR="$(cd "$(dirname "$0")" && pwd)"
cd "$PROJECT_DIR"
# 颜色输出
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'
info() { echo -e "${GREEN}[INFO]${NC} $*"; }
warn() { echo -e "${YELLOW}[WARN]${NC} $*"; }
error() { echo -e "${RED}[ERROR]${NC} $*" >&2; }
# 确保 .env 存在
ensure_env() {
if [ ! -f .env ]; then
if [ -f .env.example ]; then
warn ".env 文件不存在，从 .env.example 复制模板"
cp .env.example .env
warn "请编辑 .env 填入实际的 API Key，然后重新运行"
exit 1
else
error ".env.example 也不存在，请手动创建 .env"
exit 1
fi
fi
}
cmd_build() {
info "构建 Docker 镜像..."
docker compose build
info "构建完成"
}
cmd_up() {
ensure_env
info "启动服务..."
docker compose up -d
info "服务已启动"
info "前端: http://localhost"
info "健康检查: http://localhost/api/health"
}
cmd_down() {
info "停止服务..."
docker compose down
info "服务已停止"
}
cmd_restart() {
info "重启服务..."
cmd_down
cmd_up
}
cmd_logs() {
docker compose logs -f "${@}"
}
cmd_status() {
docker compose ps
}
usage() {
cat <<EOF
CamTalk 部署脚本
用法: $0 <命令>
命令:
build 构建 Docker 镜像
up 启动服务（后台运行）
down 停止服务
restart 重启服务
logs 查看日志（可加服务名，如: $0 logs backend）
status 查看服务状态
EOF
}
case "${1:-}" in
build) cmd_build ;;
up) cmd_up ;;
down) cmd_down ;;
restart) cmd_restart ;;
logs) shift; cmd_logs "$@" ;;
status) cmd_status ;;
*) usage; exit 1 ;;
esac

30
docker-compose.yml Normal file
View File

@@ -0,0 +1,30 @@
services:
frontend:
build:
context: ./frontend
dockerfile: Dockerfile
container_name: camtalk-frontend
ports:
- "80:80"
depends_on:
- backend
networks:
- camtalk-net
restart: unless-stopped
backend:
build:
context: ./backend
dockerfile: Dockerfile
container_name: camtalk-backend
env_file:
- .env
environment:
- APP_ENV=production
networks:
- camtalk-net
restart: unless-stopped
networks:
camtalk-net:
driver: bridge

113
docs/02-系统架构.md
View File

@@ -9,7 +9,7 @@
| 层级 | 职责 | 关键约束 |
|------|------|---------|
| **客户端（浏览器）** | 媒体采集、边缘预处理、UI 渲染 | 浏览器资源有限，模型需轻量 |
| **Go 网关** | 会话管理、模型路由、AI 服务编排 | 高并发、低延迟、状态管理 |
| **Go 网关** | 会话管理、AI 服务编排、流式管道 | 高并发、低延迟、状态管理 |
| **AI 服务** | LLM 推理、语音识别、语音合成 | 按量计费，需控制调用频率 |
> 为什么要单独加一层 Go 网关，而不是让前端直连 AI API？1）API Key 安全性；2）统一的速率限制和成本管控；3）多模型路由逻辑集中在一处便于维护。
@@ -22,7 +22,7 @@
|------|------|---------|
| 框架 | React 18 + TypeScript | 组件化开发，类型安全，生态成熟 |
| 构建 | Vite | 开发热更新快，构建产物小 |
| 实时通信 | WebSocket（原生 API） | 浏览器原生支持，无需额外依赖 |
| 实时通信 | WebSocket（原生 API） + 自封装连接管理 | 浏览器原生支持，封装心跳/重连/消息分发 |
| 边缘推理 | ONNX Runtime Web | 浏览器端跑轻量模型（VAD、关键帧检测） |
| 语音检测 | @ricky0123/vad-web | 基于 WebRTC VAD，纯前端零延迟 |
| 媒体采集 | MediaDevices API | 浏览器原生摄像头/麦克风访问 |
@@ -32,22 +32,22 @@
| 技术 | 选型 | 选择理由 |
|------|------|---------|
| 语言 | Go | 高并发 goroutine 模型，适合长连接管理 |
| HTTP 框架 | Gin | 高性能 HTTP 路由，中间件生态成熟 |
| WebSocket | gorilla/websocket | Go 生态最成熟的 WebSocket 库 |
| 会话存储 | Redis | 高速 KV 存储，适合会话状态和上下文缓存 |
| 持久化存储 | PostgreSQL | 对话历史、用量统计、用户偏好（MVP 阶段可选） |
| 配置管理 | Viper | 支持 YAML + 环境变量覆盖，详见 `03-接口文档.md` 第六章 |
| 会话存储 | Redis（规划中） / Memory（MVP 默认） | 高速 KV 存储，MVP 阶段使用进程内存，可通过配置切换到 Redis |
| 持久化存储 | PostgreSQL（规划中） | 对话历史、用量统计、用户偏好（MVP 阶段未实现） |
| 配置管理 | Viper + godotenv | 支持 YAML + .env + 环境变量覆盖，详见 `03-接口文档.md` 第六章 |
| 日志 | Zap | 高性能结构化日志 |
### AI 服务
| 能力 | 主选方案 | 备选方案 | 选型考量 |
|------|---------|---------|---------|
| 多模态 LLM | GPT-4o | Claude Sonnet | 视觉理解能力强，API 成熟 |
| 语音识别 STT | Deepgram | FunASR 自部署 | 流式识别延迟低（<500ms） |
| 语音合成 TTS | OpenAI TTS | Edge TTS（免费） | 音质自然，支持流式 |
| 轻量分类 | GPT-4o-mini | Haiku | 模型路由时的复杂度判断 |
| 多模态 LLM | GPT-4o（默认） | 通义千问等 OpenAI 兼容模型 | 通过 OpenAI 兼容接口，可灵活切换 |
| 语音识别 STT | Deepgram（默认） | MiMo ASR（小米） | 支持多 provider 切换 |
| 语音合成 TTS | OpenAI TTS（默认） | MiMo TTS（小米） | 支持多 provider 切换 |
> 不必绑定单一厂商。Go 网关的模型路由层统一封装不同 AI 服务的调用接口，按场景动态切换。
> 不必绑定单一厂商。Go 网关的 AI 服务层统一封装不同服务商的调用接口，通过配置切换 provider。
## 核心交互流程
@@ -78,52 +78,35 @@ Browser Go Gateway STT LLM TTS
| 模块 | 职责 | 关键实现 |
|------|------|---------|
| WebSocket Hub | 管理所有客户端连接，广播/定向推送 | goroutine per connection |
| Session Manager | 维护用户会话状态、对话历史 | Redis Hash + List，30 分钟 TTL（详见 `03-接口文档.md` 第五章） |
| Model Router | 根据请求类型选择 AI 模型 | 规则引擎 + 成本阈值 |
| AI Orchestrator | 编排多路 AI 调用（并行/串行） | context 取消 + 超时控制 |
| Rate Limiter | 防止单用户过度消耗 API 额度 | 令牌桶算法 |
| WebSocket Handler | 管理客户端连接生命周期，单播消息推送 | goroutine per connection |
| Session Manager | 维护用户会话状态、对话历史 | Memory（MVP 默认）/ Redis（可切换），30 分钟 TTL（详见 `03-接口文档.md` 第五章） |
| AI Orchestrator | 编排 STT→LLM→TTS 流式并行管道 | context 取消 + 超时控制 + 句子切分 |
| AI Service Layer | AI 服务抽象层（STT/LLM/TTS） | 多 provider 支持（Deepgram/MiMo/OpenAI 等） |
| REST API | 健康检查、会话管理端点 | Gin 路由 |
| Error Handler | 统一错误码定义与发送 | 错误码枚举 |
| Logger | 日志初始化封装 | Zap 结构化日志 |
| Models | 数据模型定义 | WebSocket 消息、会话、配置等 |
| Model Router | 根据请求类型选择 AI 模型（规划中） | 规则引擎 + 成本阈值 |
| Rate Limiter | 防止单用户过度消耗 API 额度（规划中） | 令牌桶算法 |
AI Orchestrator 核心代码（句子级流式并行）：
AI Orchestrator 核心接口（`internal/orchestrator/orchestrator.go`）：
```go
func (o *Orchestrator) ProcessQuery(ctx context.Context, client MessageSender, req *QueryRequest) {
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
// Step 1: STT — 识别用户语音（串行）
text, err := o.stt.Recognize(ctx, req.Audio, STTOptions{...})
if err != nil {
client.SendError(req.RequestID, "STT_ERROR", err.Error())
return
}
client.SendSTTResult(req.RequestID, text, true)
// Step 2: LLM 流式输出 + 句子切分（并行）
llmStream, _ := o.llm.ChatStream(ctx, LLMRequest{Image: req.Image, Text: text, ...})
sentenceCh := make(chan string, 4)
go func() {
defer close(sentenceCh)
var buf strings.Builder
for chunk := range llmStream {
client.SendLLMChunk(req.RequestID, chunk.Delta) // 逐 token 推送文字
buf.WriteString(chunk.Delta)
if isSentenceEnd(chunk.Delta) { // 按 。！？\n 切分
sentenceCh <- buf.String()
buf.Reset()
}
}
if buf.Len() > 0 { sentenceCh <- buf.String() }
}()
// Step 3: TTS 并行消费句子流
ttsStream, _ := o.tts.SynthesizeStream(ctx, sentenceCh, TTSOptions{...})
for chunk := range ttsStream {
client.SendTTSAudio(req.RequestID, chunk.Audio, chunk.IsLast)
}
// Orchestrator AI 编排器接口。
type Orchestrator interface {
ProcessQuery(ctx context.Context, sessionID string, req models.WsQuery,
history []models.Message, sender Sender) error
}
```
Pipeline 实现（`internal/orchestrator/pipeline.go`）流程：
1. Base64 解码音频/图片
2. 调用 `stt.Recognize()` → 发送 `stt_result`
3. 调用 `llm.ChatStream()` 获取流式输出，goroutine 消费 token → 发送 `llm_chunk` + 句子切分
4. 另一 goroutine 从句子 channel 读取 → 调用 `tts.SynthesizeStream()` → 发送 `tts_audio`
5. 流结束 → 发送 `llm_done`
6. TTS 失败静默跳过，STT/LLM 失败发送对应 error 消息
> **关键优化**：LLM 文本流和 TTS 音频流**并行推送**——客户端先逐 token 展示文字，同时 TTS 逐句子合成并推送音频，用户感知延迟大幅降低。详细的 AI 服务层接口和编排策略见 `03-接口文档.md` 第三、四章。
## 前端组件
@@ -132,17 +115,19 @@ func (o *Orchestrator) ProcessQuery(ctx context.Context, client MessageSender, r
|------|------|
| CameraManager | 摄像头流采集 |
| MicManager | 麦克风音频采集 |
| EdgeProcessor | VAD + 关键帧检测（ONNX Runtime） |
| EdgeProcessor | VAD + 关键帧检测（Canvas 像素比较） |
| WebSocketManager | WS 连接生命周期管理 |
| ChatPanel | 消息展示 |
| VideoPreview | 摄像头画面预览 |
| ConfigPanel | 右侧抽屉式配置面板（主题、TTS 开关、detail level、语言） |
| Toast | 轻量通知提示（3 秒自动消失） |
核心 Hook：`useVisionSession()` 封装一次完整的视觉对话会话（摄像头、VAD、WebSocket、消息状态）。
```typescript
function useVisionSession() {
const [messages, setMessages] = useState<Message[]>([]);
const wsRef = useWebSocket("ws://localhost:8080/ws");
const wsRef = useWebSocket(`${window.location.protocol === "https:" ? "wss:" : "ws:"}//${window.location.host}/ws`);
const videoRef = useRef<HTMLVideoElement>(null);
const { captureFrame } = useCamera(videoRef);
@@ -173,7 +158,7 @@ function useVisionSession() {
| 阶段 | 存储方案 | 持久化内容 | 理由 |
|------|---------|-----------|------|
| MVP | Redis only | 无 | 快速验证核心功能，重启丢数据可接受 |
| MVP | Memory（进程内） | 无 | 快速验证核心功能，重启丢数据可接受。Redis 实现已就绪，可通过 `storage.driver` 配置切换 |
| 上线 | Redis + PostgreSQL | 对话历史、用户偏好、用量统计 | 用户需要查看历史，运营需要成本数据 |
| 规模化 | Redis + PG + 对象存储 | 图像帧、音频片段归档 | 大文件不适合存关系库 |
@@ -262,24 +247,8 @@ server {
> WebSocket 是长连接，Nginx 必须配置 `Upgrade` 和 `Connection` 头。`proxy_read_timeout` 需要覆盖心跳间隔（客户端 30s ping），否则 Nginx 会主动断开空闲连接。
### 开发环境（Vite proxy）
### 开发环境
开发时前端（Vite :5173）和后端（Gin :8080）不同端口，用 Vite 内置代理解决跨域：
开发时前端（Vite :5173）和后端（Gin :8080）不同端口。前端 WebSocket 地址基于 `window.location.host` 动态构建，通过 Vite `server.proxy` 转发到后端，无需硬编码端口。
```typescript
// frontend/vite.config.ts
export default defineConfig({
plugins: [react()],
server: {
proxy: {
"/api": "http://localhost:8080",
"/ws": {
target: "ws://localhost:8080",
ws: true,
},
},
},
});
```
前端代码中 WebSocket 地址改为相对路径 `ws://localhost:5173/ws`，Vite 自动代理到后端。部署时 Nginx 同理，前端无需区分开发/生产地址。
`vite.config.ts` 中配置了 `/ws`（WebSocket）和 `/api`（REST）的代理，目标为 `http://localhost:8080`。

109
docs/03-接口文档.md
View File

@@ -41,19 +41,22 @@ interface WsMessage {
#### `query` — 发起一次视觉对话
用户说完话后，客户端同时发送当前图像帧和语音片段：
用户说完话后，客户端同时发送当前图像帧和语音片段。也支持文本输入模式（手动输入文字时跳过语音识别）：
```typescript
interface QueryMessage {
type: "query";
request_id: string; // 客户端生成的 UUID
image: string; // Base64 编码的 JPEG 图像（不含 data: 前缀）
audio: string; // Base64 编码的音频片段（PCM 16kHz）
audio: string; // Base64 编码的音频片段（PCM 16kHz），文本输入时为空字符串
text?: string; // 用户手动输入的文本（有值时跳过 STT，直接使用此文本）
mime_type?: string; // 音频格式，默认 "audio/pcm"
}
```
> 为什么图像和音频放在同一条消息里？因为 VAD 检测到用户说完话时，需要同时捕获"此刻的画面"和"说的话"，拆成两条消息会增加时序同步的复杂度。
>
> **文本输入模式**：当用户关闭麦克风后，可通过对话框手动输入文字。此时 `text` 字段携带用户输入，`audio` 为空字符串，服务端跳过 STT 直接使用 `text` 进行 LLM 推理。
#### `config` — 更新会话配置
@@ -73,7 +76,7 @@ interface ConfigMessage {
```typescript
interface InterruptMessage {
type: "interrupt";
request_id?: string; // 可选，指定打断哪次请求
request_id?: string; // 可选，当前实现不使用此字段，服务端始终取消当前活跃请求
}
```
@@ -143,16 +146,22 @@ interface TTSAudioMessage {
type: "tts_audio";
request_id: string;
audio: string; // Base64 编码的音频片段
mime_type: string; // "audio/mpeg"
is_last: boolean; // 是否为最后一片
mime_type: string; // "audio/mp3"
is_last: boolean; // 当前句子的音频是否完整（每句结束时为 true）
final: boolean; // 整轮 TTS 是否结束（所有句子合成完毕后为 true）
}
```
**字段语义**：
- `is_last`: 每个句子合成完毕后为 `true`，前端收到此信号即可将该句子加入播放队列。每句 TTS 音频由一次独立的 API 调用生成，对应一个 `tts_audio` 消息。
- `final`: 所有句子合成完毕后为 `true`（此时 `audio` 为空字符串），用于前端判断本轮 TTS 已全部到齐。
**音频格式规范**（前端播放依赖此约定）：
| 属性 | 值 | 说明 |
|------|------|------|
| 编码 | `audio/mpeg`（MP3） | 浏览器 `<audio>` 原生支持，OpenAI TTS 默认输出 |
| 编码 | `audio/mp3`（MP3） | 浏览器 `<audio>` 原生支持，OpenAI TTS 默认输出 |
| 采样率 | 24kHz | OpenAI TTS 默认 |
| 声道 | 单声道 | 语音不需要立体声 |
| 传输 | Base64 编码的 MP3 片段 | 每个 `tts_audio` 消息携带一个句子的音频 |
@@ -162,31 +171,37 @@ interface TTSAudioMessage {
**前端播放实现要点**：
1. **排队播放**：收到 `tts_audio` 时，将 Base64 解码为 Blob URL 并加入播放队列。第一片到达即开始播放，后续片段在 `onended` 回调中自动衔接。
2. **错误容错**：单个片段播放失败时跳过，继续播放队列中下一个，不中断整个回复。
1. **排队播放**：收到 `is_last: true` 时，将该句子的音频片段拼接为 Blob URL 并加入播放队列。第一句到达即开始播放，后续句子在 `onended` 回调中自动衔接。
2. **错误容错**：单个句子播放失败时跳过，继续播放队列中下一个，不中断整个回复。
3. **打断清理**：收到 `interrupt` 消息或用户触发打断时，清空播放队列并释放所有 Blob URL。
4. **类型锁定**：`mime_type` 字段固定为 `"audio/mpeg"`，前端解码时直接使用，无需运行时判断。
4. **类型锁定**：`mime_type` 字段固定为 `"audio/mp3"`，前端解码时直接使用，无需运行时判断。
```typescript
// 前端播放器伪代码
class AudioPlayer {
private queue: string[] = []; // Blob URL 队列
private sentenceChunks: string[] = []; // 当前句子的音频片段缓冲
private queue: string[] = []; // 已就绪的句子 Blob URL 队列
enqueue(base64: string) {
const url = decodeBase64Audio(base64, "audio/mpeg");
this.queue.push(url);
if (this.queue.length === 1) this.playNext(); // 第一片到了就开始播
enqueue(base64: string, isLast: boolean) {
this.sentenceChunks.push(base64);
if (isLast) {
// 当前句子音频完整，拼接并加入播放队列
const url = decodeBase64Audio(this.sentenceChunks.join(""), "audio/mp3");
this.sentenceChunks = [];
this.queue.push(url);
if (this.queue.length === 1) this.playNext(); // 第一句到了就开始播
}
}
private playNext() {
if (this.queue.length === 0) return;
const audio = new Audio(this.queue[0]);
audio.onended = () => { URL.revokeObjectURL(this.queue.shift()!); this.playNext(); };
audio.onerror = () => { URL.revokeObjectURL(this.queue.shift()!); this.playNext(); };
const audio = new Audio(this.queue.shift()!);
audio.onended = () => { URL.revokeObjectURL(audio.src); this.playNext(); };
audio.onerror = () => { URL.revokeObjectURL(audio.src); this.playNext(); };
audio.play();
}
clear() { this.queue.forEach(url => URL.revokeObjectURL(url)); this.queue = []; }
clear() { this.queue.forEach(url => URL.revokeObjectURL(url)); this.queue = []; this.sentenceChunks = []; }
}
```
@@ -211,13 +226,13 @@ interface PongMessage {
### 消息流时序
一次完整交互：
**语音模式**（麦克风开启）：
```
Client Server
| |
|-- query {image, audio} ------>|
|<-- stt_result {text} ---------|
|<-- stt_result {text} ---------| (语音识别)
| |
|<-- llm_chunk {delta: "这"} ---| (LLM 流式输出)
|<-- llm_chunk {delta: "是一"} -|
@@ -225,7 +240,25 @@ Client Server
|<-- llm_done {full_text} ------|
| |
|<-- tts_audio {audio} ---------| (TTS 音频流)
|<-- tts_audio {is_last: true} -|
|<-- tts_audio {is_last: true} -| (句子完成)
|<-- tts_audio {final: true} ---| (TTS 全部结束)
```
**文本输入模式**（麦克风关闭，手动输入文字）：
```
Client Server
| |
|-- query {image, text} ------->| (跳过 STT)
|<-- stt_result {text} ---------| (回显用户文本)
| |
|<-- llm_chunk {delta: "好的"} -| (LLM 流式输出)
|<-- llm_chunk {delta: "，我"} -|
|<-- llm_done {full_text} ------|
| |
|<-- tts_audio {audio} ---------| (TTS 音频流)
|<-- tts_audio {is_last: true} -| (句子完成)
|<-- tts_audio {final: true} ---| (TTS 全部结束)
```
---
@@ -293,7 +326,7 @@ DELETE /api/sessions/{session_id}
## 三、AI 服务层接口
Go 网关内部与外部 AI 服务（Deepgram STT、GPT-4o、OpenAI TTS）的调用契约。前后端联调时，后端需实现这些接口。
Go 网关内部与外部 AI 服务（STT、LLM、TTS）的调用契约。默认配置为 Deepgram STT、GPT-4o LLM、OpenAI TTS，但通过 OpenAI 兼容接口可灵活切换到其他服务商（如 MiMo ASR、通义千问等）。前后端联调时，后端需实现这些接口。
### STT 服务接口
@@ -370,7 +403,7 @@ user: [图片 + 用户语音文本]
```
- 超时：10 秒，超时返回 `LLM_TIMEOUT` 错误
- 模型选择：默认 `gpt-4o`，由 Model Router 按需切换
- 模型选择：默认 `gpt-4o`，可通过配置切换到其他 OpenAI 兼容模型
### TTS 服务接口
@@ -644,7 +677,7 @@ backend/config.dev.yaml # 开发环境（go run 时使用）
backend/config.prod.yaml # 生产环境
```
Viper 加载顺序：先读 `config.yaml`，再根据 `APP_ENV` 环境变量尝试读 `config.{env}.yaml` 覆盖，最后所有环境变量自动覆盖对应字段。
Viper 加载顺序：先读 `config.yaml`，再根据 `APP_ENV` 环境变量尝试读 `config.{env}.yaml` 覆盖，最后所有环境变量自动覆盖对应字段。此外，代码还通过 `godotenv` 加载 `.env` 文件（优先级最低，仅用于本地开发环境）。
### Go 配置结构体
@@ -686,6 +719,7 @@ type AIConfig struct {
type STTConfig struct {
Provider string `mapstructure:"provider"` // "deepgram"
APIKey string `mapstructure:"api_key"`
Model string `mapstructure:"model"` // 默认 "nova-2"
Endpoint string `mapstructure:"endpoint"` // 默认 "wss://api.deepgram.com/v1/listen"
}
@@ -700,6 +734,7 @@ type LLMConfig struct {
type TTSConfig struct {
Provider string `mapstructure:"provider"` // "openai"
APIKey string `mapstructure:"api_key"`
Model string `mapstructure:"model"` // 默认 "tts-1"
Voice string `mapstructure:"voice"` // 默认 "alloy"
Speed float64 `mapstructure:"speed"` // 默认 1.0
Endpoint string `mapstructure:"endpoint"` // 默认 "https://api.openai.com/v1"
@@ -738,6 +773,7 @@ redis:
ai:
stt:
provider: deepgram
model: nova-2
endpoint: "wss://api.deepgram.com/v1/listen"
llm:
provider: openai
@@ -746,6 +782,7 @@ ai:
timeout: 10
tts:
provider: openai
model: tts-1
voice: alloy
speed: 1.0
endpoint: "https://api.openai.com/v1"
@@ -791,8 +828,9 @@ func Load() (*Config, error) {
// 1. 读默认配置文件
v.SetConfigName("config")
v.SetConfigType("yaml")
v.AddConfigPath("./config") // go run 时
v.AddConfigPath(".") // 二进制运行时
v.AddConfigPath(".")
v.AddConfigPath("./config")
v.AddConfigPath("./backend")
if err := v.ReadInConfig(); err != nil {
if _, ok := err.(viper.ConfigFileNotFoundError); !ok {
return nil, fmt.Errorf("read config: %w", err)
@@ -800,7 +838,7 @@ func Load() (*Config, error) {
}
// 2. 按环境覆盖
env := os.Getenv("CAMTALK_APP_ENV")
env := os.Getenv("APP_ENV")
if env == "" {
env = "dev"
}
@@ -886,6 +924,7 @@ type QueryRequest struct {
RequestID string `json:"request_id"`
Image []byte `json:"-"` // Base64 解码后
Audio []byte `json:"-"` // Base64 解码后
Text string `json:"text"` // 用户手动输入的文本（有值时跳过 STT）
MimeType string `json:"mime_type"`
}
@@ -1030,7 +1069,7 @@ func NewApp(cfg *Config) *App {
## 十、连接管理
**心跳机制**：客户端每 30 秒发送 `ping`，服务端回复 `pong`。超过 60 秒无 `ping`，服务端判定连接断开并清理会话资源。
**心跳机制**：客户端每 30 秒发送应用层 `{type: "ping"}` 消息，服务端回复 `{type: "pong"}` 并刷新心跳计时器。超过 60 秒无 `ping`，服务端判定连接断开并清理会话资源。
**重连策略**（指数退避 + 抖动）：
@@ -1049,18 +1088,8 @@ function reconnect(attempt: number) {
**生产环境**：Nginx 将 `/`（前端）、`/api/*`（REST）、`/ws`（WebSocket）统一反代到同一域名，详见 `02-系统架构.md` 部署架构章节。
**开发环境**：Vite 内置代理，前端 :5173 的 `/api` 和 `/ws` 请求代理到后端 :8080：
**开发环境**：前端 WebSocket 地址基于 `window.location.host` 动态构建（相对路径），通过 Vite `server.proxy` 转发到后端 `http://localhost:8080`。REST API（`/api`）同理通过 Vite 代理转发。
```typescript
// frontend/vite.config.ts
server: {
proxy: {
"/api": "http://localhost:8080",
"/ws": { target: "ws://localhost:8080", ws: true },
},
},
```
**Go 后端 WebSocket CheckOrigin**：生产环境 Nginx 同源，`CheckOrigin` 可保持默认（拒绝跨域）。开发环境由 Vite proxy 转发，不存在跨域。因此后端无需配置 CORS 中间件，`CheckOrigin` 保持 gorilla/websocket 默认值即可。
**Go 后端 WebSocket CheckOrigin**：生产环境 Nginx 同源，`CheckOrigin` 可保持默认（拒绝跨域）。开发环境通过 Vite proxy 转发，前后端同源，无需额外配置 `CheckOrigin`。
> 如果未来需要支持第三方客户端直连（如移动端），再按需添加 CORS 中间件和 `CheckOrigin` 白名单。

46
docs/04-技术选型.md
View File

@@ -4,20 +4,58 @@
本文档记录项目中各项技术的**选型过程、替代方案对比和决策理由**。技术选型没有"绝对正确"，只有"更适合"。
**定位**：持久化部分是拓展选型，不阻塞 MVP（MVP 用 Redis 即可）。前端边缘处理部分是 MVP 阶段就需要确定的技术栈。
**定位**：持久化部分是拓展选型，不阻塞 MVP（MVP 用内存存储即可）。前端边缘处理部分是 MVP 阶段就需要确定的技术栈。AI 服务栈（STT/LLM/TTS）已确定默认选型，可通过配置灵活切换。
```
技术选型
├── 持久化层 → 数据库选型: PostgreSQL
├── AI 服务栈
│ ├── STT: Deepgram（默认） / MiMo ASR
│ ├── LLM: GPT-4o（默认） / 通义千问等 OpenAI 兼容模型
│ └── TTS: OpenAI TTS（默认） / MiMo TTS
├── 持久化层 → 数据库选型: PostgreSQL（规划中，MVP 阶段使用内存存储）
└── 前端边缘处理层
├── 边缘推理: ONNX Runtime Web
├── 边缘推理: ONNX Runtime Web（规划中，MVP 使用 Canvas 像素比较）
├── 语音检测: @ricky0123/vad-web
└── 媒体采集: MediaDevices API
```
---
## 一、持久化层选型
## 一、AI 服务栈选型
### STT（语音识别）
| 方案 | 延迟 | 成本 | 特点 |
|------|------|------|------|
| **Deepgram**（默认） | <500ms | 按分钟计费 | 流式识别，延迟极低，WebSocket 接口 |
| **MiMo ASR**（小米） | ~1s | 按量计费 | 国产替代，兼容 OpenAI chat/completions 格式，HTTP 非流式 |
| Whisper API | 1-3s | 按分钟计费 | 准确率高，支持多语言 |
| FunASR | <500ms | 自部署免费 | 阿里开源，中文优化 |
当前默认使用 Deepgram nova-2，可通过 `ai.stt.provider` 配置切换到 MiMo ASR。
### LLM（多模态大模型）
| 方案 | 成本 | 特点 |
|------|------|------|
| **GPT-4o**（默认） | $2.5/1M tokens | 视觉理解能力强，API 成熟，流式推理 |
| 通义千问 qwen3-vl-plus | 按量计费 | 阿里云，通过 OpenAI 兼容接口调用 |
| Claude Sonnet | $3/1M tokens | Anthropic，长上下文能力强 |
代码通过 OpenAI 兼容接口调用，可灵活切换到任何兼容服务商。配置 `ai.llm.provider`、`ai.llm.model`、`ai.llm.endpoint` 即可。
### TTS（语音合成）
| 方案 | 成本 | 特点 |
|------|------|------|
| **OpenAI TTS**（默认） | $15/1M 字符 | 音质自然，支持流式，默认模型 tts-1，语音 alloy |
| MiMo TTS（小米） | 按量计费 | 国产替代，通过配置切换 |
当前默认使用 OpenAI TTS（tts-1, alloy），可通过 `ai.tts.provider` 配置切换。
---
## 二、持久化层选型（规划中，MVP 阶段使用内存存储）
### 数据特征分析

37
docs/06-语音交互.md
View File

@@ -27,8 +27,11 @@ const vad = await MicVAD.new({
// audio: Float32Array，送入 STT
sendToSTT(audio);
},
positiveSpeechThreshold: 0.5, // 检测灵敏度
minSpeechDuration: 250 // 最短语音时长 ms
positiveSpeechThreshold: 0.5, // 检测灵敏度
negativeSpeechThreshold: 0.35, // 结束灵敏度
minSpeechMs: 250, // 最短语音时长 ms
redemptionMs: 300, // 语音结束确认时间 ms
preSpeechPadMs: 300, // 语音前填充 ms
});
vad.start();
@@ -39,34 +42,26 @@ vad.start();
| 方案 | 延迟 | 成本 | 特点 |
|------|------|------|------|
| Whisper API | 1-3s | 按分钟计费 | 准确率高，支持多语言 |
| **Deepgram** | <500ms | 按分钟计费 | 流式识别，延迟极低 |
| **Deepgram**（默认） | <500ms | 按分钟计费 | 流式识别，延迟极低 |
| **MiMo ASR**（小米） | ~1s | 按量计费 | 国产替代，兼容 OpenAI 格式，HTTP 非流式 |
| 浏览器原生 | ~1s | 免费 | 中文效果一般 |
| FunASR | <500ms | 自部署免费 | 阿里开源，中文优化 |
流式 STT 是低延迟的关键——不必等用户说完，边说边识别：
当前实现为**一次性语音识别**（非流式）：前端 VAD 检测到用户说完后，将完整音频片段发送到后端，后端调用 `stt.Recognize()` 一次性返回识别结果。流式 STT 为未来优化方向。
```typescript
// Deepgram 流式识别示例
const ws = new WebSocket("wss://api.deepgram.com/v1/listen", {
headers: { Authorization: `Token ${API_KEY}` }
});
ws.onmessage = (event) => {
const { transcript, is_final } = JSON.parse(event.data).channel.alternatives[0];
if (is_final) {
onFinalTranscript(transcript); // 一句完整语音，送入 LLM
}
};
```
音频编码格式：前端 `audio.ts` 将 Float32Array 转为 Int16 PCM（16kHz, pcm_s16le）再编码为 Base64。
## 环节三：TTS（文字转语音）
流式 TTS：检测 LLM 输出中的句子边界，每检测到一句就立即送入 TTS 合成并播放，不必等全部生成完。
句子切分规则：按中文标点（`。！？`）、英文标点（`. ! ?`）和换行符切分。
当前实现参数：Voice `"alloy"`、Speed `1.0`、OutputFmt `"mp3"`、SampleRate `24000`。
方案选择：
- **OpenAI TTS**：音质好，延迟中等，按字符计费
- **Edge TTS**：微软免费方案，音质不错，延迟略高
- **Fish Speech / CosyVoice**：开源方案，支持声音克隆，可自部署
- **OpenAI TTS**（默认）：音质好，延迟中等，按字符计费，模型 tts-1
- **MiMo TTS**（小米）：国产替代，通过配置切换
- **Edge TTS**（规划中）：微软免费方案，音质不错，延迟略高
## 延迟优化要点

28
docs/07-视觉理解.md
View File

@@ -15,18 +15,30 @@
| 事件驱动采样 | 用户主动触发（如拍照按钮） | 精确提问场景 |
| **混合策略** | 低频定时 + 高频事件触发 | **通用推荐方案** |
关键帧检测核心逻辑：
关键帧检测核心逻辑（TypeScript 实现，`EdgeProcessor/index.tsx`）：
```python
import numpy as np
```typescript
// 降低分辨率到 160x120 做检测，兼顾速度与精度
const DETECT_WIDTH = 160;
const DETECT_HEIGHT = 120;
def is_keyframe(prev_frame, curr_frame, threshold=30):
"""通过帧间像素差异判断是否为关键帧"""
diff = np.mean(np.abs(prev_frame.astype(int) - curr_frame.astype(int)))
return diff > threshold
function calcSimilarity(prev: ImageData, curr: ImageData): number {
const pixelCount = prev.width * prev.height;
let diffSum = 0;
// 只比较 RGB 三通道，跳过 Alpha
for (let i = 0; i < prev.data.length; i += 4) {
diffSum += Math.abs(prev.data[i] - curr.data[i])
+ Math.abs(prev.data[i+1] - curr.data[i+1])
+ Math.abs(prev.data[i+2] - curr.data[i+2]);
}
const avgDiff = diffSum / (pixelCount * 3);
return 1 - avgDiff / 255; // 相似度：1 = 完全相同，0 = 完全不同
}
```
> 实际开发中，先降低分辨率（如 320x240）做关键帧检测，再对命中帧保留原始分辨率送入 LLM，兼顾速度与精度。
阈值说明：
- 对话模式：`similarity > 0.9` 时跳过（视为重复帧）
- 观察模式：`similarity < 0.85` 时触发变化回调
## 图像编码与多模态输入

39
docs/08-成本控制.md
View File

@@ -27,15 +27,12 @@
| 本地预筛选 | 中 | 高 | 用轻量模型判断"是否值得问 LLM" |
```typescript
// 混合策略：定时低频 + 事件高频
const NORMAL_INTERVAL = 5000; // 正常 5 秒一帧
// 混合策略：定时低频 + 事件高频（sampling.ts）
const IDLE_INTERVAL = 5000; // 空闲 5 秒一帧
const ACTIVE_INTERVAL = 1000; // 用户说话时 1 秒一帧
let isUserSpeaking = false;
setInterval(() => {
captureAndSend(isUserSpeaking ? "low" : "high");
}, isUserSpeaking ? ACTIVE_INTERVAL : NORMAL_INTERVAL);
// SamplingController 根据 VAD 状态切换采样间隔
// detail_level 通过 session config 静态配置，不随说话状态动态变化
```
## 策略二：端云协同——把计算推到边缘
@@ -43,11 +40,11 @@ setInterval(() => {
不是所有计算都需要上云。可前置到客户端的计算：
- **VAD 语音检测**：浏览器端完成，减少无效音频上传（节省 ~70% 带宽）
- **人脸/物体检测**：用 ONNX Runtime 跑轻量模型（如 YOLOv8-nano ~6MB，推理 ~30ms），只在检测到新物体时触发 LLM
- **重复画面过滤**：计算帧间相似度，相似度 > 90% 直接跳过
- **敏感内容过滤**：NSFW 检测前置，避免无效 API 调用
- **人脸/物体检测**（规划中）：用 ONNX Runtime 跑轻量模型（如 YOLOv8-nano ~6MB，推理 ~30ms），只在检测到新物体时触发 LLM。当前 MVP 使用 Canvas 像素比较做关键帧检测
- **重复画面过滤**：计算帧间相似度，对话模式 similarity > 0.9 跳过，观察模式 similarity < 0.85 触发
- **敏感内容过滤**（规划中）：NSFW 检测前置，避免无效 API 调用
## 策略三：模型分级——用对模型做对事
## 策略三：模型分级——用对模型做对事（规划中）
不是每个问题都需要最贵的模型：
@@ -58,20 +55,10 @@ setInterval(() => {
└── 代码/推理 → o1 ($15/1M tokens)
```
```typescript
async function routeQuery(image: string, question: string) {
const complexity = await classifyComplexity(question);
const modelMap = {
simple: "gpt-4o-mini", // "这是什么？"
moderate: "gpt-4o", // "分析这张图"
complex: "o1" // "推理/规划"
};
return callLLM(modelMap[complexity], image, question);
}
```
> 当前 MVP 阶段使用单一模型（默认 GPT-4o），模型分级路由为未来优化方向。通过配置 `ai.llm.model` 可手动切换模型。
## 策略四：缓存与复用
## 策略四：缓存与复用（规划中）
- **语义缓存**：相似问题直接返回缓存结果（如反复问"这是什么"）
- **上下文复用**：连续对话中，未变化的图像不必重复发送
- **Prompt 压缩**：精简 system prompt，减少每轮的固定 token 开销
- **语义缓存**（规划中）：相似问题直接返回缓存结果（如反复问"这是什么"）
- **上下文复用**：连续对话中，未变化的图像不必重复发送（已通过重复画面过滤实现）
- **对话历史裁剪**：前端按 `MAX_HISTORY_ROUNDS = 10` 裁剪，后端按 `defaultHistorySize = 20` 裁剪，限制每轮的固定 token 开销

5
docs/10-功能创意.md Normal file
View File

@@ -0,0 +1,5 @@
1.视频录制
2.对话翻译
3.对话总结
4.手动对话功能
5.视频框大小可调整，可最小化然后拖动

4
frontend/.gitignore vendored
View File

@@ -12,6 +12,10 @@ dist
dist-ssr
*.local
# VAD 静态资源（从 node_modules 自动复制）
public/silero_vad_*.onnx
public/vad.worklet.bundle.min.js
# Editor directories and files
.vscode/*
!.vscode/extensions.json

11
frontend/.mcp.json Normal file
View File

@@ -0,0 +1,11 @@
{
"mcpServers": {
"shadcn": {
"command": "npx",
"args": [
"shadcn@latest",
"mcp"
]
}
}
}

25
frontend/Dockerfile Normal file
View File

@@ -0,0 +1,25 @@
# ---- 构建阶段 ----
FROM node:22-alpine AS builder
WORKDIR /app
# 先复制依赖清单，利用 Docker 缓存层
COPY package.json package-lock.json ./
RUN npm ci
# 复制源码并构建
COPY . .
RUN npm run build
# ---- 运行阶段 ----
FROM nginx:stable-alpine
# 复制 Nginx 配置
COPY nginx.conf /etc/nginx/conf.d/default.conf
# 复制构建产物
COPY --from=builder /app/dist /usr/share/nginx/html
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

5
frontend/index.html
View File

@@ -1,10 +1,11 @@
<!doctype html>
<html lang="en">
<html lang="zh-CN">
<head>
<meta charset="UTF-8" />
<link rel="icon" type="image/svg+xml" href="/favicon.svg" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>frontend</title>
<meta name="description" content="CamTalk — 多模态实时 AI 视觉对话助手，通过摄像头和麦克风与 AI 自然交互" />
<title>CamTalk — AI 视觉对话助手</title>
</head>
<body>
<div id="root"></div>

46
frontend/nginx.conf Normal file
View File

@@ -0,0 +1,46 @@
server {
listen 80;
server_name _;
root /usr/share/nginx/html;
index index.html;
# 前端静态资源
location / {
try_files $uri $uri/ /index.html;
}
# REST API 反代
location /api/ {
proxy_pass http://backend:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
# WebSocket 反代
location /ws {
proxy_pass http://backend:8080;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 86400s;
proxy_send_timeout 86400s;
}
# 静态资源缓存
location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot)$ {
expires 7d;
add_header Cache-Control "public, immutable";
}
# Gzip 压缩
gzip on;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
gzip_min_length 256;
}

188
frontend/skills/redesign-skill.md Normal file
View File

@@ -0,0 +1,188 @@
# Redesign Skill - Anti-Slop Frontend Framework
> Source: https://github.com/Leonxlnx/taste-skill/tree/main/skills/redesign-skill
> Purpose: Audit and upgrade existing frontend code to eliminate generic AI-generated UI patterns.
## How This Works
When applied to an existing project, follow this sequence:
1. **Scan** — Read the codebase. Identify the framework, styling method (Tailwind, vanilla CSS, styled-components, etc.), and current design patterns.
2. **Diagnose** — Run through the audit below. List every generic pattern, weak point, and missing state you find.
3. **Fix** — Apply targeted upgrades working with the existing stack. Do not rewrite from scratch. Improve what's there.
---
## Design Audit
### Typography
Check for these problems and fix them:
- **Browser default fonts or Inter everywhere.** Replace with a font that has character. Good options: `Geist`, `Outfit`, `Cabinet Grotesk`, `Satoshi`. For editorial/creative projects, pair a serif header with a sans-serif body.
- **Headlines lack presence.** Increase size for display text, tighten letter-spacing, reduce line-height. Headlines should feel heavy and intentional.
- **Body text too wide.** Limit paragraph width to roughly 65 characters. Increase line-height for readability.
- **Only Regular (400) and Bold (700) weights used.** Introduce Medium (500) and SemiBold (600) for more subtle hierarchy.
- **Numbers in proportional font.** Use a monospace font or enable tabular figures (`font-variant-numeric: tabular-nums`) for data-heavy interfaces.
- **Missing letter-spacing adjustments.** Use negative tracking for large headers, positive tracking for small caps or labels.
- **All-caps subheaders everywhere.** Try lowercase italics, sentence case, or small-caps instead.
- **Orphaned words.** Single words sitting alone on the last line. Fix with `text-wrap: balance` or `text-wrap: pretty`.
### Color and Surfaces
- **Pure `#000000` background.** Replace with off-black, dark charcoal, or tinted dark (`#0a0a0a`, `#121212`, or a dark navy).
- **Oversaturated accent colors.** Keep saturation below 80%. Desaturate accents so they blend with neutrals instead of screaming.
- **More than one accent color.** Pick one. Remove the rest. Consistency beats variety.
- **Mixing warm and cool grays.** Stick to one gray family. Tint all grays with a consistent hue (warm or cool, not both).
- **Purple/blue "AI gradient" aesthetic.** This is the most common AI design fingerprint. Replace with neutral bases and a single, considered accent.
- **Generic `box-shadow`.** Tint shadows to match the background hue. Use colored shadows (e.g., dark blue shadow on a blue background) instead of pure black at low opacity.
- **Flat design with zero texture.** Add subtle noise, grain, or micro-patterns to backgrounds. Pure flat vectors feel sterile.
- **Perfectly even gradients.** Break the uniformity with radial gradients, noise overlays, or mesh gradients instead of standard linear 45-degree fades.
- **Inconsistent lighting direction.** Audit all shadows to ensure they suggest a single, consistent light source.
- **Random dark sections in a light mode page (or vice versa).** A single dark-background section breaking an otherwise light page looks like a copy-paste accident. Either commit to a full dark mode or keep a consistent background tone throughout. If contrast is needed, use a slightly darker shade of the same palette — not a sudden jump to `#111` in the middle of a cream page.
- **Empty, flat sections with no visual depth.** Sections that are just text on a plain background feel unfinished. Add high-quality background imagery (blurred, overlaid, or masked), subtle patterns, or ambient gradients. Use reliable placeholder sources like `https://picsum.photos/seed/{name}/1920/1080` when real assets are not available. Experiment with background images behind hero sections, feature blocks, or CTAs — even a subtle full-width photo at low opacity adds presence.
### Layout
- **Everything centered and symmetrical.** Break symmetry with offset margins, mixed aspect ratios, or left-aligned headers over centered content.
- **Three equal card columns as feature row.** This is the most generic AI layout. Replace with a 2-column zig-zag, asymmetric grid, horizontal scroll, or masonry layout.
- **Using `height: 100vh` for full-screen sections.** Replace with `min-height: 100dvh` to prevent layout jumping on mobile browsers (iOS Safari viewport bug).
- **Complex flexbox percentage math.** Replace with CSS Grid for reliable multi-column structures.
- **No max-width container.** Add a container constraint (around 1200-1440px) with auto margins so content doesn't stretch edge-to-edge on wide screens.
- **Cards of equal height forced by flexbox.** Allow variable heights or use masonry when content varies in length.
- **Uniform border-radius on everything.** Vary the radius: tighter on inner elements, softer on containers.
- **No overlap or depth.** Elements sit flat next to each other. Use negative margins to create layering and visual depth.
- **Symmetrical vertical padding.** Top and bottom padding are always identical. Adjust optically — bottom padding often needs to be slightly larger.
- **Dashboard always has a left sidebar.** Try top navigation, a floating command menu, or a collapsible panel instead.
- **Missing whitespace.** Double the spacing. Let the design breathe. Dense layouts work for data dashboards, not for marketing pages.
- **Buttons not bottom-aligned in card groups.** When cards have different content lengths, CTAs end up at random heights. Pin buttons to the bottom of each card so they form a clean horizontal line regardless of content above.
- **Feature lists starting at different vertical positions.** In pricing tables or comparison cards, the list of features should start at the same Y position across all columns. Use consistent spacing above the list or fixed-height title/price blocks.
- **Inconsistent vertical rhythm in side-by-side elements.** When placing cards, columns, or panels next to each other, align shared elements (titles, descriptions, prices, buttons) across all items. Misaligned baselines make the layout look broken.
- **Mathematical alignment that looks optically wrong.** Centering by the math doesn't always look centered to the eye. Icons next to text, play buttons in circles, or text in buttons often need 1-2px optical adjustments to feel right.
### Interactivity and States
- **No hover states on buttons.** Add background shift, slight scale, or translate on hover.
- **No active/pressed feedback.** Add a subtle `scale(0.98)` or `translateY(1px)` on press to simulate a physical click.
- **Instant transitions with zero duration.** Add smooth transitions (200-300ms) to all interactive elements.
- **Missing focus ring.** Ensure visible focus indicators for keyboard navigation. This is an accessibility requirement, not optional.
- **No loading states.** Replace generic circular spinners with skeleton loaders that match the layout shape.
- **No empty states.** An empty dashboard showing nothing is a missed opportunity. Design a composed "getting started" view.
- **No error states.** Add clear, inline error messages for forms. Do not use `window.alert()`.
- **Dead links.** Buttons that link to `#`. Either link to real destinations or visually disable them.
- **No indication of current page in navigation.** Style the active nav link differently so users know where they are.
- **Scroll jumping.** Anchor clicks jump instantly. Add `scroll-behavior: smooth`.
- **Animations using `top`, `left`, `width`, `height`.** Switch to `transform` and `opacity` for GPU-accelerated, smooth animation.
### Content
- **Generic names like "John Doe" or "Jane Smith".** Use diverse, realistic-sounding names.
- **Fake round numbers like `99.99%`, `50%`, `$100.00`.** Use organic, messy data: `47.2%`, `$99.00`, `+1 (312) 847-1928`.
- **Placeholder company names like "Acme Corp", "Nexus", "SmartFlow".** Invent contextual, believable brand names.
- **AI copywriting cliches.** Never use "Elevate", "Seamless", "Unleash", "Next-Gen", "Game-changer", "Delve", "Tapestry", or "In the world of...". Write plain, specific language.
- **Exclamation marks in success messages.** Remove them. Be confident, not loud.
- **"Oops!" error messages.** Be direct: "Connection failed. Please try again."
- **Passive voice.** Use active voice: "We couldn't save your changes" instead of "Mistakes were made."
- **All blog post dates identical.** Randomize dates to appear real.
- **Same avatar image for multiple users.** Use unique assets for every distinct person.
- **Lorem Ipsum.** Never use placeholder latin text. Write real draft copy.
- **Title Case On Every Header.** Use sentence case instead.
### Component Patterns
- **Generic card look (border + shadow + white background).** Remove the border, or use only background color, or use only spacing. Cards should exist only when elevation communicates hierarchy.
- **Always one filled button + one ghost button.** Add text links or tertiary styles to reduce visual noise.
- **Pill-shaped "New" and "Beta" badges.** Try square badges, flags, or plain text labels.
- **Accordion FAQ sections.** Use a side-by-side list, searchable help, or inline progressive disclosure.
- **3-card carousel testimonials with dots.** Replace with a masonry wall, embedded social posts, or a single rotating quote.
- **Pricing table with 3 towers.** Highlight the recommended tier with color and emphasis, not just extra height.
- **Modals for everything.** Use inline editing, slide-over panels, or expandable sections instead of popups for simple actions.
- **Avatar circles exclusively.** Try squircles or rounded squares for a less generic look.
- **Light/dark toggle always a sun/moon switch.** Use a dropdown, system preference detection, or integrate it into settings.
- **Footer link farm with 4 columns.** Simplify. Focus on main navigational paths and legally required links.
### Iconography
- **Lucide or Feather icons exclusively.** These are the "default" AI icon choice. Use Phosphor, Heroicons, or a custom set for differentiation.
- **Rocketship for "Launch", shield for "Security".** Replace cliche metaphors with less obvious icons (bolt, fingerprint, spark, vault).
- **Inconsistent stroke widths across icons.** Audit all icons and standardize to one stroke weight.
- **Missing favicon.** Always include a branded favicon.
- **Stock "diverse team" photos.** Use real team photos, candid shots, or a consistent illustration style instead of uncanny stock imagery.
### Code Quality
- **Div soup.** Use semantic HTML: `<header>`, `<main>`, `<article>`, `<section>`, `<footer>`.
- **Inline styles mixed with CSS classes.** Move all styling to the project's styling system.
- **Hardcoded pixel widths.** Use relative units (`%`, `rem`, `em`, `max-width`) for flexible layouts.
- **Missing alt text on images.** Describe image content for screen readers. Never leave `alt=""` or `alt="image"` on meaningful images.
- **Arbitrary z-index values like `9999`.** Establish a clean z-index scale in the theme/variables.
- **Commented-out dead code.** Remove all debug artifacts before shipping.
- **Import hallucinations.** Check that every import actually exists in `package.json` or the project dependencies.
- **Missing meta tags.** Add proper `<title>`, `<description>`, `<og:image>`, and social sharing meta tags.
### Strategic Omissions (What AI Typically Forgets)
- **No legal links.** Add privacy policy and terms of service links in the footer.
- **No "back" navigation.** Dead ends in user flows. Every page needs a way back.
- **No custom 404 page.** Design a helpful, branded "page not found" experience.
- **No form validation.** Add client-side validation for emails, required fields, and format checks.
- **No "skip to content" link.** Essential for keyboard users. Add a hidden skip-link.
- **No cookie consent.** If required by jurisdiction, add a compliant consent banner.
---
## Upgrade Techniques
When upgrading a project, pull from these high-impact techniques to replace generic patterns:
### Typography Upgrades
- **Variable font animation.** Interpolate weight or width on scroll or hover for text that feels alive.
- **Outlined-to-fill transitions.** Text starts as a stroke outline and fills with color on scroll entry or interaction.
- **Text mask reveals.** Large typography acting as a window to video or animated imagery behind it.
### Layout Upgrades
- **Broken grid / asymmetry.** Elements that deliberately ignore column structure — overlapping, bleeding off-screen, or offset with calculated randomness.
- **Whitespace maximization.** Aggressive use of negative space to force focus on a single element.
- **Parallax card stacks.** Sections that stick and physically stack over each other during scroll.
- **Split-screen scroll.** Two halves of the screen sliding in opposite directions.
### Motion Upgrades
- **Smooth scroll with inertia.** Decouple scrolling from browser defaults for a heavier, cinematic feel.
- **Staggered entry.** Elements cascade in with slight delays, combining Y-axis translation with opacity fade. Never mount everything at once.
- **Spring physics.** Replace linear easing with spring-based motion for a natural, weighty feel on all interactive elements.
- **Scroll-driven reveals.** Content entering through expanding masks, wipes, or draw-on SVG paths tied to scroll progress.
### Surface Upgrades
- **True glassmorphism.** Go beyond `backdrop-filter: blur`. Add a 1px inner border and a subtle inner shadow to simulate edge refraction.
- **Spotlight borders.** Card borders that illuminate dynamically under the cursor.
- **Grain and noise overlays.** A fixed, pointer-events-none overlay with subtle noise to break digital flatness.
- **Colored, tinted shadows.** Shadows that carry the hue of the background rather than using generic black.
---
## Fix Priority
Apply changes in this order for maximum visual impact with minimum risk:
1. **Font swap** — biggest instant improvement, lowest risk
2. **Color palette cleanup** — remove clashing or oversaturated colors
3. **Hover and active states** — makes the interface feel alive
4. **Layout and spacing** — proper grid, max-width, consistent padding
5. **Replace generic components** — swap cliche patterns for modern alternatives
6. **Add loading, empty, and error states** — makes it feel finished
7. **Polish typography scale and spacing** — the premium final touch
---
## Rules
- Work with the existing tech stack. Do not migrate frameworks or styling libraries.
- Do not break existing functionality. Test after every change.
- Before importing any new library, check the project's dependency file first.
- If the project uses Tailwind, check the version (v3 vs v4) before modifying config.
- If the project has no framework, use vanilla CSS.
- Keep changes reviewable and focused. Small, targeted improvements over big rewrites.

98
frontend/skills/soft-skill.md Normal file
View File

@@ -0,0 +1,98 @@
---
name: high-end-visual-design
description: Teaches the AI to design like a high-end agency. Defines the exact fonts, spacing, shadows, card structures, and animations that make a website feel expensive. Blocks all the common defaults that make AI designs look cheap or generic.
---
# Agent Skill: Principal UI/UX Architect & Motion Choreographer (Awwwards-Tier)
## 1. Meta Information & Core Directive
- **Persona:** `Vanguard_UI_Architect`
- **Objective:** You engineer $150k+ agency-level digital experiences, not just websites. Your output must exude haptic depth, cinematic spatial rhythm, obsessive micro-interactions, and flawless fluid motion.
- **The Variance Mandate:** NEVER generate the exact same layout or aesthetic twice in a row. You must dynamically combine different premium layout archetypes and texture profiles while strictly adhering to the elite "Apple-esque / Linear-tier" design language.
## 2. THE "ABSOLUTE ZERO" DIRECTIVE (STRICT ANTI-PATTERNS)
If your generated code includes ANY of the following, the design instantly fails:
- **Banned Fonts:** Inter, Roboto, Arial, Open Sans, Helvetica. (Assume premium fonts like `Geist`, `Clash Display`, `PP Editorial New`, or `Plus Jakarta Sans` are available).
- **Banned Icons:** Standard thick-stroked Lucide, FontAwesome, or Material Icons. Use only ultra-light, precise lines (e.g., Phosphor Light, Remix Line).
- **Banned Borders & Shadows:** Generic 1px solid gray borders. Harsh, dark drop shadows (`shadow-md`, `rgba(0,0,0,0.3)`).
- **Banned Layouts:** Edge-to-edge sticky navbars glued to the top. Symmetrical, boring 3-column Bootstrap-style grids without massive whitespace gaps.
- **Banned Motion:** Standard `linear` or `ease-in-out` transitions. Instant state changes without interpolation.
## 3. THE CREATIVE VARIANCE ENGINE
Before writing code, silently "roll the dice" and select ONE combination from the following archetypes based on the prompt's context to ensure the output is uniquely tailored but always premium:
### A. Vibe & Texture Archetypes (Pick 1)
1. **Ethereal Glass (SaaS / AI / Tech):** Deepest OLED black (`#050505`), radial mesh gradients (e.g., subtle glowing purple/emerald orbs) in the background. Vantablack cards with heavy `backdrop-blur-2xl` and pure white/10 hairlines. Wide geometric Grotesk typography.
2. **Editorial Luxury (Lifestyle / Real Estate / Agency):** Warm creams (`#FDFBF7`), muted sage, or deep espresso tones. High-contrast Variable Serif fonts for massive headings. Subtle CSS noise/film-grain overlay (`opacity-[0.03]`) for a physical paper feel.
3. **Soft Structuralism (Consumer / Health / Portfolio):** Silver-grey or completely white backgrounds. Massive bold Grotesk typography. Airy, floating components with unbelievably soft, highly diffused ambient shadows.
### B. Layout Archetypes (Pick 1)
1. **The Asymmetrical Bento:** A masonry-like CSS Grid of varying card sizes (e.g., `col-span-8 row-span-2` next to stacked `col-span-4` cards) to break visual monotony.
- **Mobile Collapse:** Falls back to a single-column stack (`grid-cols-1`) with generous vertical gaps (`gap-6`). All `col-span` overrides reset to `col-span-1`.
2. **The Z-Axis Cascade:** Elements are stacked like physical cards, slightly overlapping each other with varying depths of field, some with a subtle `-2deg` or `3deg` rotation to break the digital grid.
- **Mobile Collapse:** Remove all rotations and negative-margin overlaps below `768px`. Stack vertically with standard spacing. Overlapping elements cause touch-target conflicts on mobile.
3. **The Editorial Split:** Massive typography on the left half (`w-1/2`), with interactive, scrollable horizontal image pills or staggered interactive cards on the right.
- **Mobile Collapse:** Converts to a full-width vertical stack (`w-full`). Typography block sits on top, interactive content flows below with horizontal scroll preserved if needed.
**Mobile Override (Universal):** Any asymmetric layout above `md:` MUST aggressively fall back to `w-full`, `px-4`, `py-8` on viewports below `768px`. Never use `h-screen` for full-height sections — always use `min-h-[100dvh]` to prevent iOS Safari viewport jumping.
## 4. HAPTIC MICRO-AESTHETICS (COMPONENT MASTERY)
### A. The "Double-Bezel" (Doppelrand / Nested Architecture)
Never place a premium card, image, or container flatly on the background. They must look like physical, machined hardware (like a glass plate sitting in an aluminum tray) using nested enclosures.
- **Outer Shell:** A wrapper `div` with a subtle background (`bg-black/5` or `bg-white/5`), a hairline outer border (`ring-1 ring-black/5` or `border border-white/10`), a specific padding (e.g., `p-1.5` or `p-2`), and a large outer radius (`rounded-[2rem]`).
- **Inner Core:** The actual content container inside the shell. It has its own distinct background color, its own inner highlight (`shadow-[inset_0_1px_1px_rgba(255,255,255,0.15)]`), and a mathematically calculated smaller radius (e.g., `rounded-[calc(2rem-0.375rem)]`) for concentric curves.
### B. Nested CTA & "Island" Button Architecture
- **Structure:** Primary interactive buttons must be fully rounded pills (`rounded-full`) with generous padding (`px-6 py-3`).
- **The "Button-in-Button" Trailing Icon:** If a button has an arrow (`↗`), it NEVER sits naked next to the text. It must be nested inside its own distinct circular wrapper (e.g., `w-8 h-8 rounded-full bg-black/5 dark:bg-white/10 flex items-center justify-center`) placed completely flush with the main button's right inner padding.
### C. Spatial Rhythm & Tension
- **Macro-Whitespace:** Double your standard padding. Use `py-24` to `py-40` for sections. Allow the design to breathe heavily.
- **Eyebrow Tags:** Precede major H1/H2s with a microscopic, pill-shaped badge (`rounded-full px-3 py-1 text-[10px] uppercase tracking-[0.2em] font-medium`).
## 5. MOTION CHOREOGRAPHY (FLUID DYNAMICS)
Never use default transitions. All motion must simulate real-world mass and spring physics. Use custom cubic-beziers (e.g., `transition-all duration-700 ease-[cubic-bezier(0.32,0.72,0,1)]`).
### A. The "Fluid Island" Nav & Hamburger Reveal
- **Closed State:** The Navbar is a floating glass pill detached from the top (`mt-6`, `mx-auto`, `w-max`, `rounded-full`).
- **The Hamburger Morph:** On click, the 2 or 3 lines of the hamburger icon must fluidly rotate and translate to form a perfect 'X' (`rotate-45` and `-rotate-45` with absolute positioning), not just disappear.
- **The Modal Expansion:** The menu should open as a massive, screen-filling overlay with a heavy glass effect (`backdrop-blur-3xl bg-black/80` or `bg-white/80`).
- **Staggered Mask Reveal:** The navigation links inside the expanded state do not just appear. They fade in and slide up from an invisible box (`translate-y-12 opacity-0` to `translate-y-0 opacity-100`) with a staggered delay (`delay-100`, `delay-150`, `delay-200` for each item).
### B. Magnetic Button Hover Physics
- Use the `group` utility. On hover, do not just change the background color.
- Scale the entire button down slightly (`active:scale-[0.98]`) to simulate physical pressing.
- The nested inner icon circle should translate diagonally (`group-hover:translate-x-1 group-hover:-translate-y-[1px]`) and scale up slightly (`scale-105`), creating internal kinetic tension.
### C. Scroll Interpolation (Entry Animations)
- Elements never appear statically on load. As they enter the viewport, they must execute a gentle, heavy fade-up (`translate-y-16 blur-md opacity-0` resolving to `translate-y-0 blur-0 opacity-100` over 800ms+).
- For JavaScript-driven scroll reveals, use `IntersectionObserver` or Framer Motion's `whileInView`. Never use `window.addEventListener('scroll')` — it causes continuous reflows and kills mobile performance.
## 6. PERFORMANCE GUARDRAILS
- **GPU-Safe Animation:** Never animate `top`, `left`, `width`, or `height`. Animate exclusively via `transform` and `opacity`. Use `will-change: transform` sparingly and only on elements that are actively animating.
- **Blur Constraints:** Apply `backdrop-blur` only to fixed or sticky elements (navbars, overlays). Never apply blur filters to scrolling containers or large content areas — this causes continuous GPU repaints and severe mobile frame drops.
- **Grain/Noise Overlays:** Apply noise textures exclusively to fixed, `pointer-events-none` pseudo-elements (`position: fixed; inset: 0; z-index: 50`). Never attach them to scrolling containers.
- **Z-Index Discipline:** Do not use arbitrary `z-50` or `z-[9999]`. Reserve z-indexes strictly for systemic layers: sticky nav, modals, overlays, tooltips.
## 7. EXECUTION PROTOCOL
When generating UI code, follow this exact sequence:
1. **[SILENT THOUGHT]** Roll the Variance Engine (Section 3). Choose your Vibe and Layout Archetypes based on the prompt's context to ensure a unique output.
2. **[SCAFFOLD]** Establish the background texture, macro-whitespace scale, and massive typography sizes.
3. **[ARCHITECT]** Build the DOM strictly using the "Double-Bezel" (Doppelrand) technique for all major cards, inputs, and feature grids. Use exaggerated squircle radii (`rounded-[2rem]`).
4. **[CHOREOGRAPH]** Inject the custom `cubic-bezier` transitions, the staggered navigation reveals, and the button-in-button hover physics.
5. **[OUTPUT]** Deliver flawless, pixel-perfect React/Tailwind/HTML code. Do not include basic, generic fallbacks.
## 8. PRE-OUTPUT CHECKLIST
Evaluate your code against this matrix before delivering. This is the last filter.
- [ ] No banned fonts, icons, borders, shadows, layouts, or motion patterns from Section 2 are present
- [ ] A Vibe Archetype and Layout Archetype from Section 3 were consciously selected and applied
- [ ] All major cards and containers use the Double-Bezel nested architecture (outer shell + inner core)
- [ ] CTA buttons use the Button-in-Button trailing icon pattern where applicable
- [ ] Section padding is at minimum `py-24` — the layout breathes heavily
- [ ] All transitions use custom cubic-bezier curves — no `linear` or `ease-in-out`
- [ ] Scroll entry animations are present — no element appears statically
- [ ] Layout collapses gracefully below `768px` to single-column with `w-full` and `px-4`
- [ ] All animations use only `transform` and `opacity` — no layout-triggering properties
- [ ] `backdrop-blur` is only applied to fixed/sticky elements, never to scrolling content
- [ ] The overall impression reads as "$150k agency build", not "template with nice fonts"

1206
frontend/skills/taste-skill-main.md Normal file
View File

File diff suppressed because it is too large Load Diff

523
frontend/src/App.css
View File

@@ -1,33 +1,41 @@
/* ============================================================
CamTalk — Web 端应用样式
双栏布局：左侧视频面板 + 右侧聊天面板
设计方向：深色科技极简，Ethereal Glass 美学
字体：Inter 优先，系统字体栈兜底
颜色：冷灰体系，单一蓝色强调色
============================================================ */
:root {
--color-primary: #2563eb;
--color-primary-hover: #1d4ed8;
--color-bg: #0b0f1a;
--color-surface: #151b2b;
--color-surface-2: #1c2438;
--color-text: #e2e8f0;
--color-text-muted: #64748b;
--color-border: #1e293b;
--color-success: #22c55e;
--color-warning: #f59e0b;
--color-error: #ef4444;
--radius: 10px;
--radius-sm: 6px;
--color-primary: #3b82f6;
--color-primary-hover: #2563eb;
--color-bg: #0a0a0f;
--color-surface: #111118;
--color-surface-2: #1a1a24;
--color-surface-3: #22222e;
--color-text: #d4d4dc;
--color-text-muted: #5a5a6e;
--color-border: #1f1f2a;
--color-success: #34d399;
--color-warning: #fbbf24;
--color-error: #f87171;
--radius: 12px;
--radius-sm: 8px;
--transition-fast: 0.2s cubic-bezier(0.25, 0.46, 0.45, 0.94);
--transition-smooth: 0.4s cubic-bezier(0.16, 1, 0.3, 1);
}
[data-theme="light"] {
--color-primary: #2563eb;
--color-primary-hover: #1d4ed8;
--color-bg: #f1f5f9;
--color-bg: #f4f4f8;
--color-surface: #ffffff;
--color-surface-2: #f8fafc;
--color-text: #0f172a;
--color-text-muted: #64748b;
--color-border: #e2e8f0;
--color-surface-2: #f0f0f5;
--color-surface-3: #e8e8ee;
--color-text: #111118;
--color-text-muted: #6b6b80;
--color-border: #dddde5;
--color-success: #16a34a;
--color-warning: #d97706;
--color-error: #dc2626;
@@ -44,9 +52,12 @@ html, body, #root {
}
body {
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", sans-serif;
font-family: "Inter", -apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, sans-serif;
font-variant-numeric: tabular-nums;
background: var(--color-bg);
color: var(--color-text);
-webkit-font-smoothing: antialiased;
-moz-osx-font-smoothing: grayscale;
}
/* ---- App Shell ---- */
@@ -64,11 +75,14 @@ body {
display: flex;
align-items: center;
justify-content: space-between;
padding: 0 20px;
height: 48px;
padding: 0 24px;
height: 56px;
background: var(--color-surface);
border-bottom: 1px solid var(--color-border);
box-shadow: 0 1px 8px rgba(10, 10, 15, 0.3);
flex-shrink: 0;
position: relative;
z-index: 10;
}
.header__left {
@@ -78,14 +92,17 @@ body {
}
.header__title {
font-size: 1.2rem;
font-weight: 700;
font-size: 1.15rem;
font-weight: 600;
letter-spacing: -0.02em;
color: var(--color-text);
}
.header__subtitle {
font-size: 0.8rem;
font-size: 0.78rem;
font-weight: 400;
color: var(--color-text-muted);
letter-spacing: 0.01em;
}
.header__right {
@@ -95,30 +112,34 @@ body {
}
.header__stats {
font-size: 0.75rem;
font-size: 0.72rem;
color: var(--color-text-muted);
padding: 4px 10px;
padding: 4px 12px;
background: var(--color-surface-2);
border-radius: 12px;
border-radius: 20px;
font-weight: 500;
letter-spacing: 0.01em;
}
.badge {
font-size: 0.75rem;
padding: 3px 10px;
border-radius: 12px;
font-size: 0.72rem;
padding: 4px 12px;
border-radius: 20px;
font-weight: 500;
letter-spacing: 0.01em;
transition: all var(--transition-fast);
}
.badge--connected {
color: var(--color-success);
background: rgba(34, 197, 94, 0.1);
border: 1px solid rgba(34, 197, 94, 0.3);
background: rgba(52, 211, 153, 0.08);
border: 1px solid rgba(52, 211, 153, 0.2);
}
.badge--connecting {
color: var(--color-warning);
background: rgba(245, 158, 11, 0.1);
border: 1px solid rgba(245, 158, 11, 0.3);
background: rgba(251, 191, 36, 0.08);
border: 1px solid rgba(251, 191, 36, 0.2);
}
.badge--disconnected {
@@ -132,10 +153,13 @@ body {
border: none;
font-size: 1.1rem;
cursor: pointer;
padding: 6px;
padding: 8px;
border-radius: var(--radius-sm);
transition: background 0.15s;
transition: all var(--transition-fast);
color: var(--color-text-muted);
display: inline-flex;
align-items: center;
justify-content: center;
}
.btn-icon:hover {
@@ -143,6 +167,10 @@ body {
color: var(--color-text);
}
.btn-icon:active {
transform: scale(0.92);
}
.lang-group {
display: flex;
gap: 2px;
@@ -158,9 +186,9 @@ body {
font-size: 0.72rem;
font-weight: 600;
padding: 4px 10px;
border-radius: 4px;
border-radius: 6px;
cursor: pointer;
transition: all 0.15s;
transition: all var(--transition-fast);
}
.lang-btn:hover {
@@ -187,8 +215,8 @@ body {
flex-shrink: 0;
display: flex;
flex-direction: column;
padding: 16px;
gap: 12px;
padding: 20px;
gap: 14px;
}
.video-container {
@@ -198,6 +226,11 @@ body {
border-radius: var(--radius);
overflow: hidden;
border: 1px solid var(--color-border);
transition: border-color var(--transition-fast);
}
.video-container:hover {
border-color: var(--color-surface-3);
}
.video-container .video-preview {
@@ -210,7 +243,7 @@ body {
width: 100%;
height: 100%;
object-fit: cover;
background: #1a1a2e;
background: #12121a;
}
.video-placeholder {
@@ -220,29 +253,36 @@ body {
flex-direction: column;
align-items: center;
justify-content: center;
gap: 8px;
gap: 10px;
color: var(--color-text-muted);
font-size: 0.82rem;
font-weight: 400;
}
.video-placeholder__icon {
font-size: 2.4rem;
opacity: 0.2;
font-size: 2.2rem;
opacity: 0.15;
margin-bottom: 4px;
}
.video-placeholder__hint {
font-size: 0.72rem;
opacity: 0.6;
}
.video-indicator {
position: absolute;
bottom: 16px;
left: 16px;
background: rgba(0, 0, 0, 0.75);
backdrop-filter: blur(8px);
background: rgba(10, 10, 15, 0.8);
backdrop-filter: blur(12px);
-webkit-backdrop-filter: blur(12px);
color: var(--color-success);
padding: 6px 14px;
border-radius: var(--radius-sm);
font-size: 0.8rem;
font-size: 0.78rem;
font-weight: 500;
animation: pulse 1.5s ease-in-out infinite;
animation: pulse 1.5s cubic-bezier(0.25, 0.46, 0.45, 0.94) infinite;
}
.live-badge {
@@ -252,22 +292,23 @@ body {
display: flex;
align-items: center;
gap: 6px;
background: rgba(0, 0, 0, 0.75);
backdrop-filter: blur(8px);
background: rgba(10, 10, 15, 0.8);
backdrop-filter: blur(12px);
-webkit-backdrop-filter: blur(12px);
color: white;
padding: 4px 12px;
padding: 5px 12px;
border-radius: var(--radius-sm);
font-size: 0.75rem;
font-size: 0.72rem;
font-weight: 600;
letter-spacing: 0.5px;
letter-spacing: 0.04em;
}
.live-badge__dot {
width: 8px;
height: 8px;
width: 7px;
height: 7px;
border-radius: 50%;
background: var(--color-success);
animation: pulse 1.5s ease-in-out infinite;
animation: pulse 1.5s cubic-bezier(0.25, 0.46, 0.45, 0.94) infinite;
}
.video-indicator--loading { color: var(--color-warning); }
@@ -278,45 +319,49 @@ body {
position: absolute;
top: 12px;
right: 12px;
background: rgba(37, 99, 235, 0.85);
backdrop-filter: blur(8px);
background: rgba(59, 130, 246, 0.85);
backdrop-filter: blur(12px);
-webkit-backdrop-filter: blur(12px);
color: white;
padding: 3px 10px;
border-radius: var(--radius-sm);
font-size: 0.7rem;
font-size: 0.68rem;
font-weight: 700;
letter-spacing: 0.5px;
letter-spacing: 0.06em;
}
.observation-badge {
position: absolute;
top: 12px;
left: 12px;
background: rgba(34, 197, 94, 0.85);
backdrop-filter: blur(8px);
background: rgba(52, 211, 153, 0.85);
backdrop-filter: blur(12px);
-webkit-backdrop-filter: blur(12px);
color: white;
padding: 3px 10px;
border-radius: var(--radius-sm);
font-size: 0.7rem;
font-size: 0.68rem;
font-weight: 700;
animation: pulse 2s ease-in-out infinite;
letter-spacing: 0.04em;
animation: pulse 2s cubic-bezier(0.25, 0.46, 0.45, 0.94) infinite;
}
.video-overlay {
position: absolute;
inset: 0;
background: rgba(0, 0, 0, 0.35);
background: rgba(10, 10, 15, 0.4);
display: flex;
align-items: center;
justify-content: center;
backdrop-filter: blur(2px);
backdrop-filter: blur(3px);
-webkit-backdrop-filter: blur(3px);
}
.video-overlay__spinner {
width: 36px;
height: 36px;
border: 3px solid rgba(255, 255, 255, 0.2);
border-top-color: white;
border: 2.5px solid rgba(255, 255, 255, 0.15);
border-top-color: rgba(255, 255, 255, 0.8);
border-radius: 50%;
animation: spin 0.8s linear infinite;
}
@@ -327,53 +372,66 @@ body {
flex-shrink: 0;
display: flex;
justify-content: center;
padding: 4px 0;
padding: 6px 0;
}
.video-controls__row {
display: flex;
gap: 6px;
gap: 8px;
flex-wrap: wrap;
justify-content: center;
}
.btn--ctrl {
padding: 6px 12px;
padding: 7px 14px;
font-size: 0.78rem;
border-radius: var(--radius-sm);
border: 1px solid var(--color-border);
background: var(--color-surface-2);
color: var(--color-text-muted);
cursor: pointer;
transition: all 0.15s;
transition: all var(--transition-fast);
white-space: nowrap;
}
.btn--ctrl:hover {
background: var(--color-border);
background: var(--color-surface-3);
color: var(--color-text);
border-color: var(--color-surface-3);
}
.btn--ctrl:active {
transform: translateY(1px);
}
.btn--ctrl-on {
background: rgba(34, 197, 94, 0.15);
background: rgba(52, 211, 153, 0.1);
color: var(--color-success);
border-color: rgba(34, 197, 94, 0.3);
border-color: rgba(52, 211, 153, 0.25);
}
.btn--ctrl-on:hover {
background: rgba(52, 211, 153, 0.18);
}
.btn--ctrl-off {
background: rgba(239, 68, 68, 0.1);
background: rgba(248, 113, 113, 0.08);
color: var(--color-error);
border-color: rgba(239, 68, 68, 0.2);
border-color: rgba(248, 113, 113, 0.18);
}
.btn--ctrl-off:hover {
background: rgba(248, 113, 113, 0.15);
}
.btn--speaking {
animation: micPulse 0.8s ease-in-out infinite;
box-shadow: 0 0 12px rgba(34, 197, 94, 0.5);
animation: micPulse 0.8s cubic-bezier(0.25, 0.46, 0.45, 0.94) infinite;
box-shadow: 0 0 12px rgba(52, 211, 153, 0.4);
}
@keyframes micPulse {
0%, 100% { box-shadow: 0 0 8px rgba(34, 197, 94, 0.4); transform: scale(1); }
50% { box-shadow: 0 0 16px rgba(34, 197, 94, 0.8); transform: scale(1.05); }
0%, 100% { box-shadow: 0 0 8px rgba(52, 211, 153, 0.3); transform: scale(1); }
50% { box-shadow: 0 0 18px rgba(52, 211, 153, 0.7); transform: scale(1.04); }
}
/* ---- 右侧聊天面板 ---- */
@@ -391,21 +449,23 @@ body {
display: flex;
align-items: center;
justify-content: space-between;
padding: 12px 16px;
padding: 14px 24px;
font-weight: 600;
font-size: 0.85rem;
font-size: 0.82rem;
border-bottom: 1px solid var(--color-border);
flex-shrink: 0;
color: var(--color-text-muted);
letter-spacing: -0.01em;
}
.chat-panel-header__mode {
font-size: 0.7rem;
font-size: 0.68rem;
font-weight: 500;
color: var(--color-success);
background: rgba(34, 197, 94, 0.1);
padding: 2px 8px;
border-radius: 10px;
background: rgba(52, 211, 153, 0.08);
padding: 3px 10px;
border-radius: 12px;
letter-spacing: 0.02em;
}
.chat-panel-body {
@@ -419,33 +479,91 @@ body {
.chat-panel {
flex: 1;
overflow-y: auto;
padding: 12px 16px;
display: flex;
flex-direction: column;
gap: 10px;
min-height: 0;
}
.chat-panel__messages {
flex: 1;
overflow-y: auto;
padding: 16px 24px;
display: flex;
flex-direction: column;
gap: 12px;
}
.chat-panel__welcome {
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
flex: 1;
gap: 12px;
color: var(--color-text-muted);
font-size: 0.85rem;
padding: 40px 0;
}
.chat-panel__welcome-icon {
font-size: 2.4rem;
opacity: 0.12;
margin-bottom: 4px;
}
.chat-panel__welcome-hint {
font-size: 0.75rem;
opacity: 0.6;
max-width: 240px;
text-align: center;
line-height: 1.5;
}
.chat-panel--empty {
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
flex: 1;
gap: 12px;
color: var(--color-text-muted);
font-size: 0.85rem;
}
.chat-panel--empty-icon {
font-size: 2.4rem;
opacity: 0.12;
margin-bottom: 4px;
}
.chat-panel--empty-hint {
font-size: 0.75rem;
opacity: 0.6;
max-width: 240px;
text-align: center;
line-height: 1.5;
}
.chat-message {
padding: 10px 14px;
padding: 12px 16px;
border-radius: var(--radius-sm);
background: var(--color-surface-2);
max-width: 92%;
max-width: min(92%, 65ch);
transition: background var(--transition-fast);
}
.chat-message:hover {
background: var(--color-surface-3);
}
.chat-message--user {
border-left: 2px solid var(--color-primary);
align-self: flex-end;
background: rgba(37, 99, 235, 0.08);
background: rgba(59, 130, 246, 0.06);
}
.chat-message--user:hover {
background: rgba(59, 130, 246, 0.1);
}
.chat-message--assistant {
@@ -457,24 +575,77 @@ body {
}
.chat-message__role {
font-size: 0.65rem;
font-size: 0.62rem;
font-weight: 600;
text-transform: uppercase;
letter-spacing: 0.5px;
letter-spacing: 0.08em;
color: var(--color-text-muted);
margin-bottom: 3px;
margin-bottom: 4px;
}
.chat-message__content {
font-size: 0.82rem;
line-height: 1.55;
line-height: 1.65;
white-space: pre-wrap;
max-width: 65ch;
}
.chat-message__meta {
margin-top: 6px;
font-size: 0.65rem;
margin-top: 8px;
font-size: 0.62rem;
color: var(--color-text-muted);
font-weight: 400;
letter-spacing: 0.01em;
}
/* ---- Chat Input ---- */
.chat-input {
display: flex;
gap: 8px;
padding: 12px 24px 16px;
border-top: 1px solid var(--color-border);
background: var(--color-surface);
}
.chat-input__field {
flex: 1;
padding: 10px 14px;
border: 1px solid var(--color-border);
border-radius: var(--radius-sm);
background: var(--color-surface-2);
color: var(--color-text);
font-size: 0.82rem;
outline: none;
transition: border-color var(--transition-fast);
}
.chat-input__field::placeholder {
color: var(--color-text-muted);
}
.chat-input__field:focus {
border-color: var(--color-primary);
}
.chat-input__send {
padding: 10px 16px;
border: none;
border-radius: var(--radius-sm);
background: var(--color-primary);
color: white;
font-size: 0.9rem;
cursor: pointer;
transition: opacity var(--transition-fast);
}
.chat-input__send:hover:not(:disabled) {
opacity: 0.9;
}
.chat-input__send:disabled {
opacity: 0.4;
cursor: not-allowed;
}
/* ---- Streaming Cursor ---- */
@@ -483,22 +654,24 @@ body {
display: inline;
animation: blink 0.8s step-end infinite;
color: var(--color-success);
font-weight: 300;
}
/* ---- System Message ---- */
.system-message {
text-align: center;
padding: 8px 16px;
margin: 0 20px;
padding: 10px 16px;
margin: 0 24px;
border-radius: var(--radius-sm);
font-size: 0.8rem;
font-size: 0.78rem;
font-weight: 500;
}
.system-message--warning {
background: rgba(245, 158, 11, 0.1);
background: rgba(251, 191, 36, 0.08);
color: var(--color-warning);
border: 1px solid rgba(245, 158, 11, 0.2);
border: 1px solid rgba(251, 191, 36, 0.15);
}
/* ---- Drawer (Config Panel) ---- */
@@ -507,9 +680,10 @@ body {
position: fixed;
inset: 0;
z-index: 100;
background: rgba(0, 0, 0, 0.4);
backdrop-filter: blur(2px);
animation: fadeIn 0.2s ease;
background: rgba(10, 10, 15, 0.5);
backdrop-filter: blur(4px);
-webkit-backdrop-filter: blur(4px);
animation: fadeIn 0.25s var(--transition-smooth);
}
.drawer {
@@ -522,22 +696,23 @@ body {
border-left: 1px solid var(--color-border);
display: flex;
flex-direction: column;
animation: slideInRight 0.25s ease;
box-shadow: -8px 0 24px rgba(0, 0, 0, 0.3);
animation: slideInRight 0.35s var(--transition-smooth);
box-shadow: -8px 0 40px rgba(10, 10, 15, 0.5);
}
.drawer__header {
display: flex;
align-items: center;
justify-content: space-between;
padding: 14px 16px;
padding: 16px 20px;
border-bottom: 1px solid var(--color-border);
flex-shrink: 0;
}
.drawer__title {
font-weight: 600;
font-size: 0.9rem;
font-size: 0.88rem;
letter-spacing: -0.01em;
}
.drawer__close {
@@ -546,9 +721,12 @@ body {
color: var(--color-text-muted);
cursor: pointer;
font-size: 1rem;
padding: 4px 8px;
padding: 6px 8px;
border-radius: var(--radius-sm);
transition: all 0.15s;
transition: all var(--transition-fast);
display: inline-flex;
align-items: center;
justify-content: center;
}
.drawer__close:hover {
@@ -556,26 +734,30 @@ body {
color: var(--color-text);
}
.drawer__close:active {
transform: scale(0.92);
}
.drawer__body {
flex: 1;
overflow-y: auto;
padding: 16px;
padding: 20px;
}
/* ---- Config Group ---- */
.config-group {
margin-bottom: 20px;
margin-bottom: 24px;
}
.config-group__title {
font-size: 0.7rem;
font-size: 0.68rem;
font-weight: 600;
text-transform: uppercase;
letter-spacing: 0.8px;
letter-spacing: 0.1em;
color: var(--color-text-muted);
margin-bottom: 12px;
padding-bottom: 8px;
margin-bottom: 14px;
padding-bottom: 10px;
border-bottom: 1px solid var(--color-border);
}
@@ -583,16 +765,23 @@ body {
display: flex;
align-items: center;
justify-content: space-between;
padding: 10px 0;
padding: 10px 8px;
margin: 0 -8px;
font-size: 0.82rem;
color: var(--color-text);
cursor: pointer;
border-radius: var(--radius-sm);
transition: background var(--transition-fast);
}
.config-row:hover {
background: var(--color-surface-2);
}
.config-row__info {
display: flex;
flex-direction: column;
gap: 2px;
gap: 3px;
}
.config-row__label {
@@ -601,8 +790,9 @@ body {
}
.config-row__desc {
font-size: 0.72rem;
font-size: 0.7rem;
color: var(--color-text-muted);
line-height: 1.4;
}
.config-row select,
@@ -611,23 +801,36 @@ body {
color: var(--color-text);
border: 1px solid var(--color-border);
border-radius: var(--radius-sm);
padding: 5px 10px;
font-size: 0.8rem;
padding: 6px 10px;
font-size: 0.78rem;
cursor: pointer;
transition: all var(--transition-fast);
}
.config-row select:hover,
.config-row input[type="checkbox"]:hover {
border-color: var(--color-surface-3);
}
.config-row select:focus,
.config-row input[type="checkbox"]:focus {
outline: 2px solid rgba(59, 130, 246, 0.4);
outline-offset: 1px;
}
.config-row input[type="checkbox"] {
width: 18px;
height: 18px;
cursor: pointer;
accent-color: var(--color-primary);
}
/* ---- Toast ---- */
.toast-container {
position: fixed;
top: 68px;
right: 20px;
top: 72px;
right: 24px;
z-index: 1000;
display: flex;
flex-direction: column;
@@ -637,16 +840,30 @@ body {
.toast {
padding: 10px 16px;
border-radius: var(--radius-sm);
font-size: 0.82rem;
font-size: 0.8rem;
font-weight: 500;
cursor: pointer;
animation: slideIn 0.3s ease-out;
animation: slideIn 0.35s var(--transition-smooth);
max-width: 320px;
backdrop-filter: blur(8px);
backdrop-filter: blur(12px);
-webkit-backdrop-filter: blur(12px);
box-shadow: 0 8px 32px rgba(10, 10, 15, 0.4);
}
.toast--error { background: rgba(239, 68, 68, 0.9); color: white; }
.toast--warning { background: rgba(245, 158, 11, 0.9); color: #000; }
.toast--info { background: rgba(37, 99, 235, 0.9); color: white; }
.toast--error {
background: rgba(248, 113, 113, 0.92);
color: white;
}
.toast--warning {
background: rgba(251, 191, 36, 0.92);
color: #111;
}
.toast--info {
background: rgba(59, 130, 246, 0.92);
color: white;
}
/* ---- Buttons ---- */
@@ -654,18 +871,23 @@ body {
padding: 8px 20px;
border: none;
border-radius: var(--radius-sm);
font-size: 0.85rem;
font-size: 0.82rem;
font-weight: 600;
cursor: pointer;
transition: all 0.15s;
transition: all var(--transition-fast);
display: inline-flex;
align-items: center;
gap: 6px;
letter-spacing: 0.01em;
}
.btn:active {
transform: translateY(1px);
}
.btn--lg {
padding: 12px 32px;
font-size: 0.95rem;
font-size: 0.92rem;
border-radius: var(--radius);
}
@@ -685,43 +907,46 @@ body {
}
.btn--secondary:hover {
background: var(--color-border);
background: var(--color-surface-3);
border-color: var(--color-surface-3);
}
.btn--warning {
background: var(--color-warning);
color: #000;
color: #111;
}
.btn--warning:hover { opacity: 0.9; }
.btn--warning:hover {
opacity: 0.9;
}
.btn--danger {
background: rgba(239, 68, 68, 0.15);
background: rgba(248, 113, 113, 0.1);
color: var(--color-error);
border: 1px solid rgba(239, 68, 68, 0.3);
border: 1px solid rgba(248, 113, 113, 0.2);
}
.btn--danger:hover {
background: rgba(239, 68, 68, 0.25);
background: rgba(248, 113, 113, 0.18);
}
.btn--active {
background: var(--color-primary);
color: white;
box-shadow: 0 0 12px rgba(37, 99, 235, 0.4);
animation: pulseBtn 2s ease-in-out infinite;
box-shadow: 0 0 12px rgba(59, 130, 246, 0.35);
animation: pulseBtn 2s cubic-bezier(0.25, 0.46, 0.45, 0.94) infinite;
}
/* ---- Animations ---- */
@keyframes pulse {
0%, 100% { opacity: 1; }
50% { opacity: 0.6; }
50% { opacity: 0.55; }
}
@keyframes pulseBtn {
0%, 100% { box-shadow: 0 0 12px rgba(37, 99, 235, 0.4); }
50% { box-shadow: 0 0 20px rgba(37, 99, 235, 0.7); }
0%, 100% { box-shadow: 0 0 12px rgba(59, 130, 246, 0.35); }
50% { box-shadow: 0 0 22px rgba(59, 130, 246, 0.6); }
}
@keyframes blink {
@@ -734,7 +959,7 @@ body {
}
@keyframes slideIn {
from { opacity: 0; transform: translateX(20px); }
from { opacity: 0; transform: translateX(16px); }
to { opacity: 1; transform: translateX(0); }
}
@@ -751,7 +976,7 @@ body {
/* ---- Scrollbar ---- */
::-webkit-scrollbar {
width: 6px;
width: 4px;
}
::-webkit-scrollbar-track {
@@ -760,7 +985,7 @@ body {
::-webkit-scrollbar-thumb {
background: var(--color-border);
border-radius: 3px;
border-radius: 2px;
}
::-webkit-scrollbar-thumb:hover {

25
frontend/src/App.tsx
View File

@@ -58,6 +58,7 @@ function App() {
isMicOn,
toggleCamera,
toggleMic,
sendTextMessage,
} = useVisionSession();
const isConnected = connectionStatus === "connected";
@@ -108,8 +109,12 @@ function App() {
className="btn-icon"
onClick={() => setShowConfig((v) => !v)}
title="设置"
aria-label="设置"
>
⚙️
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="1.8" strokeLinecap="round" strokeLinejoin="round">
<circle cx="12" cy="12" r="3" />
<path d="M19.4 15a1.65 1.65 0 0 0 .33 1.82l.06.06a2 2 0 0 1-2.83 2.83l-.06-.06a1.65 1.65 0 0 0-1.82-.33 1.65 1.65 0 0 0-1 1.51V21a2 2 0 0 1-4 0v-.09A1.65 1.65 0 0 0 9 19.4a1.65 1.65 0 0 0-1.82.33l-.06.06a2 2 0 0 1-2.83-2.83l.06-.06A1.65 1.65 0 0 0 4.68 15a1.65 1.65 0 0 0-1.51-1H3a2 2 0 0 1 0-4h.09A1.65 1.65 0 0 0 4.6 9a1.65 1.65 0 0 0-.33-1.82l-.06-.06a2 2 0 0 1 2.83-2.83l.06.06A1.65 1.65 0 0 0 9 4.68a1.65 1.65 0 0 0 1-1.51V3a2 2 0 0 1 4 0v.09a1.65 1.65 0 0 0 1 1.51 1.65 1.65 0 0 0 1.82-.33l.06-.06a2 2 0 0 1 2.83 2.83l-.06.06A1.65 1.65 0 0 0 19.4 9a1.65 1.65 0 0 0 1.51 1H21a2 2 0 0 1 0 4h-.09a1.65 1.65 0 0 0-1.51 1z" />
</svg>
</button>
</div>
</header>
@@ -161,8 +166,21 @@ function App() {
)}
{!isConnected && !stream && (
<div className="video-placeholder">
<span className="video-placeholder__icon">📷</span>
<span>点击右侧按钮开始对话</span>
<svg className="video-placeholder__icon" width="48" height="48" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="1.2" strokeLinecap="round" strokeLinejoin="round">
<path d="M23 19a2 2 0 0 1-2 2H3a2 2 0 0 1-2-2V8a2 2 0 0 1 2-2h4l2-3h6l2 3h4a2 2 0 0 1 2 2z" />
<circle cx="12" cy="13" r="4" />
</svg>
<span>点击下方按钮开始对话</span>
</div>
)}
{isConnected && !isCameraOn && (
<div className="video-placeholder">
<svg className="video-placeholder__icon" width="48" height="48" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="1.2" strokeLinecap="round" strokeLinejoin="round">
<path d="M23 19a2 2 0 0 1-2 2H3a2 2 0 0 1-2-2V8a2 2 0 0 1 2-2h4l2-3h6l2 3h4a2 2 0 0 1 2 2z" />
<circle cx="12" cy="13" r="4" />
</svg>
<span>摄像头未开启</span>
<span className="video-placeholder__hint">可在右侧聊天框打字对话</span>
</div>
)}
</div>
@@ -226,6 +244,7 @@ function App() {
messages={messages}
currentReply={currentReply}
connectionStatus={connectionStatus}
onSendText={sendTextMessage}
/>
</div>
</div>

115
frontend/src/components/ChatPanel/index.tsx
View File

@@ -1,9 +1,9 @@
// ============================================================
// ChatPanel — 消息展示面板
// 职责：渲染对话消息列表、流式光标、元数据、自动滚动
// 职责：渲染对话消息列表、流式光标、元数据、自动滚动、文本输入
// ============================================================
import { useEffect, useRef } from "react";
import { useEffect, useRef, useState } from "react";
import type { ChatMessage } from "../../types";
import type { ConnectionStatus } from "../../lib/websocket";
@@ -11,12 +11,16 @@ interface ChatPanelProps {
messages: ChatMessage[];
currentReply?: string;
connectionStatus: ConnectionStatus;
onSendText?: (text: string) => void;
}
export function ChatPanel({ messages, currentReply, connectionStatus }: ChatPanelProps) {
export function ChatPanel({ messages, currentReply, connectionStatus, onSendText }: ChatPanelProps) {
const bottomRef = useRef<HTMLDivElement>(null);
const containerRef = useRef<HTMLDivElement>(null);
const isAutoScroll = useRef(true);
const [inputText, setInputText] = useState("");
const isConnected = connectionStatus === "connected";
// 用户上滚时暂停自动滚动，滚到底部时恢复
useEffect(() => {
@@ -39,52 +43,87 @@ export function ChatPanel({ messages, currentReply, connectionStatus }: ChatPane
}
}, [messages, currentReply]);
// 空状态
if (messages.length === 0 && !currentReply) {
if (connectionStatus !== "connected") {
return (
<div className="chat-panel chat-panel--empty">
<p>点击下方按钮开始对话</p>
</div>
);
}
// 提交文本消息
const handleSubmit = (e: React.FormEvent) => {
e.preventDefault();
if (!inputText.trim() || !onSendText) return;
onSendText(inputText);
setInputText("");
};
// 未连接时的空状态
if (!isConnected) {
return (
<div className="chat-panel chat-panel--empty">
<p>对着摄像头说话即可</p>
<span className="chat-panel--empty-icon">💬</span>
<p>点击下方按钮开始对话</p>
<span className="chat-panel--empty-hint">连接后可打字或语音与 AI 交互</span>
</div>
);
}
return (
<div className="chat-panel" ref={containerRef}>
{messages.map((msg, index) => (
<div key={index} className={`chat-message chat-message--${msg.role}`}>
<div className="chat-message__role">
{msg.role === "user" ? "你" : "AI"}
<div className="chat-panel">
<div className="chat-panel__messages" ref={containerRef}>
{/* 空状态提示 */}
{messages.length === 0 && !currentReply && (
<div className="chat-panel__welcome">
<span className="chat-panel__welcome-icon">💬</span>
<p>在下方输入文字开始对话</p>
<span className="chat-panel__welcome-hint">也可以开启麦克风用语音对话</span>
</div>
<div className="chat-message__content">{msg.content}</div>
{msg.role === "assistant" && msg.tokensUsed !== undefined && (
<div className="chat-message__meta">
{msg.tokensUsed} tokens
{msg.latencyMs !== undefined && ` · ${(msg.latencyMs / 1000).toFixed(1)}s`}
{msg.model && ` · ${msg.model}`}
)}
{messages.map((msg, index) => (
<div key={index} className={`chat-message chat-message--${msg.role}`}>
<div className="chat-message__role">
{msg.role === "user" ? "你" : "AI"}
</div>
)}
</div>
))}
{/* 流式回复（尚未完成） */}
{currentReply && (
<div className="chat-message chat-message--assistant chat-message--streaming">
<div className="chat-message__role">AI</div>
<div className="chat-message__content">
{currentReply}
<span className="cursor">▌</span>
<div className="chat-message__content">{msg.content}</div>
{msg.role === "assistant" && msg.tokensUsed !== undefined && (
<div className="chat-message__meta">
{msg.tokensUsed} tokens
{msg.latencyMs !== undefined && ` · ${(msg.latencyMs / 1000).toFixed(1)}s`}
{msg.model && ` · ${msg.model}`}
</div>
)}
</div>
</div>
)}
))}
<div ref={bottomRef} />
{/* 流式回复（尚未完成） */}
{currentReply && (
<div className="chat-message chat-message--assistant chat-message--streaming">
<div className="chat-message__role">AI</div>
<div className="chat-message__content">
{currentReply}
<span className="cursor">▌</span>
</div>
</div>
)}
<div ref={bottomRef} />
</div>
{/* 文本输入框 - 连接后始终显示 */}
{onSendText && (
<form className="chat-input" onSubmit={handleSubmit}>
<input
type="text"
className="chat-input__field"
placeholder="输入文字对话..."
value={inputText}
onChange={(e) => setInputText(e.target.value)}
/>
<button
type="submit"
className="chat-input__send"
disabled={!inputText.trim()}
title="发送"
>
➤
</button>
</form>
)}
</div>
);
}

2
frontend/src/components/EdgeProcessor/index.tsx
View File

@@ -48,6 +48,8 @@ export function useVAD(options?: VADOptions) {
getStream: () => Promise.resolve(stream),
startOnLoad: true,
model: "legacy",
// 指向 node_modules 中的 WASM 文件（由 Vite 中间件提供）
onnxWASMBasePath: "/node_modules/onnxruntime-web/dist/",
onSpeechStart: () => {
setIsSpeaking(true);

86
frontend/src/hooks/useVisionSession.ts
View File

@@ -158,6 +158,10 @@ export function useVisionSession() {
return;
}
// 停止上一轮的 TTS 播放，防止新旧音频重叠
ttsPlayerRef.current?.stop();
setIsAudioPlaying(false);
const frame = captureFrame();
if (!frame) {
console.warn("[Session] 无法捕获图像帧");
@@ -262,9 +266,7 @@ export function useVisionSession() {
case "tts_audio":
getTTSPlayer().enqueue(msg.audio, msg.mime_type, msg.is_last);
if (!msg.is_last) {
setIsAudioPlaying(true);
}
setIsAudioPlaying(true);
break;
case "error":
@@ -280,21 +282,26 @@ export function useVisionSession() {
/** 启动会话 */
const startSession = useCallback(async () => {
// 1. 获取摄像头和麦克风
await startCamera();
setIsCameraOn(true);
const micStream = await startMic();
if (!micStream) {
showToast("无法获取麦克风权限", "error");
return;
}
setIsMicOn(true);
// 2. 连接 WebSocket
// 1. 连接 WebSocket（必须）
connect();
// 3. 启动 VAD（传入麦克风 stream）
await startVAD(micStream);
// 2. 尝试获取摄像头（可选）
try {
await startCamera();
setIsCameraOn(true);
} catch {
console.warn("[Session] 无法获取摄像头权限，将以纯文本模式运行");
}
// 3. 尝试获取麦克风（可选）
const micStream = await startMic();
if (micStream) {
setIsMicOn(true);
// 4. 启动 VAD（仅在麦克风可用时）
await startVAD(micStream);
} else {
console.warn("[Session] 无法获取麦克风权限，将以文本输入模式运行");
}
}, [startCamera, startMic, connect, startVAD]);
/** 结束会话 */
@@ -332,13 +339,17 @@ export function useVisionSession() {
/** 麦克风开关 */
const toggleMic = useCallback(async () => {
if (isMicOn) {
await stopVAD();
stopMic();
setIsMicOn(false);
} else {
const micStream = await startMic();
setIsMicOn(!!micStream);
if (micStream) {
await startVAD(micStream);
setIsMicOn(true);
}
}
}, [isMicOn, startMic, stopMic]);
}, [isMicOn, startMic, stopMic, startVAD, stopVAD]);
/** 打断当前回复 */
const interrupt = useCallback(() => {
@@ -359,6 +370,44 @@ export function useVisionSession() {
setIsProcessing(false);
}, [send, currentReply]);
/** 发送文本消息（手动输入） */
const sendTextMessage = useCallback(
(text: string) => {
if (!text.trim() || isProcessingRef.current) return;
// 停止上一轮的 TTS 播放
ttsPlayerRef.current?.stop();
setIsAudioPlaying(false);
// 捕获当前摄像头画面
const frame = captureFrame();
const requestId = uuidv4();
send({
type: "query",
request_id: requestId,
image: frame ? dataUrlToBase64(frame) : "",
audio: "", // 文本输入无音频
text: text.trim(),
});
// 更新请求统计
setStats((prev) => ({ ...prev, queryCount: prev.queryCount + 1 }));
// 添加用户消息
setMessages((prev) => [
...prev,
{ role: "user", content: text.trim(), timestamp: Date.now() },
]);
// 记录到对话历史
historyRef.current.push({ role: "user", content: text.trim() });
setIsProcessing(true);
},
[captureFrame, send],
);
return {
messages,
currentReply,
@@ -383,5 +432,6 @@ export function useVisionSession() {
isMicOn,
toggleCamera,
toggleMic,
sendTextMessage,
};
}

84
frontend/src/lib/ttsPlayer.ts
View File

@@ -1,18 +1,21 @@
// ============================================================
// TTS Player — 语音播放器
// 职责：收集后端流式 tts_audio 片段，拼接后播放
// 格式：MVP 仅支持 audio/mp3，pcm 为 TODO
// 职责：接收后端流式 tts_audio 片段，按句子排队播放
// 设计：第一句到达即开始播放，后续句子在 onended 回调中自动衔接
// ============================================================
type OnEndCallback = () => void;
export class TTSPlayer {
private chunks: string[] = [];
/** 当前句子的音频片段缓冲（每句可能含多个 chunk） */
private sentenceChunks: string[] = [];
/** 已就绪的句子 Blob URL 播放队列 */
private queue: string[] = [];
private audio: HTMLAudioElement | null = null;
private _isPlaying = false;
private onEndCallback: OnEndCallback | null = null;
/** 注册播放完成回调 */
/** 注册播放完成回调（所有句子播完后触发） */
onEnd(cb: OnEndCallback): void {
this.onEndCallback = cb;
}
@@ -25,25 +28,40 @@ export class TTSPlayer {
/**
* 入队一个 TTS 音频片段
* @param base64 Base64 编码的音频数据
* @param mimeType 音频格式（"audio/mp3" 或 "audio/pcm"）
* @param isLast 是否为最后一个片段
* @param mimeType 音频格式（"audio/mp3"）
* @param isLast 当前句子是否合成完毕（每句结束时为 true）
*/
enqueue(base64: string, mimeType: string, isLast: boolean): void {
this.chunks.push(base64);
this.sentenceChunks.push(base64);
if (isLast) {
this.play(mimeType);
// 当前句子的音频已完整，拼接并加入播放队列
const combined = this.sentenceChunks.join("");
this.sentenceChunks = [];
const url = this.base64ToBlobUrl(combined, mimeType);
this.queue.push(url);
// 如果当前没有在播，立即开始播放
if (!this._isPlaying) {
this.playNext();
}
}
}
/** 停止播放并清空缓冲区 */
/** 停止播放并清空所有队列 */
stop(): void {
if (this.audio) {
this.audio.pause();
this.audio.removeAttribute("src");
this.audio = null;
}
this.chunks = [];
// 释放队列中的 Blob URL
for (const url of this.queue) {
URL.revokeObjectURL(url);
}
this.queue = [];
this.sentenceChunks = [];
this._isPlaying = false;
}
@@ -57,48 +75,48 @@ export class TTSPlayer {
this.audio?.play();
}
/** 拼接所有片段并播放 */
private play(mimeType: string): void {
if (this.chunks.length === 0) return;
// 拼接所有 Base64 片段
const combined = this.chunks.join("");
this.chunks = [];
// Base64 → Uint8Array → Blob
const binary = atob(combined);
const bytes = new Uint8Array(binary.length);
for (let i = 0; i < binary.length; i++) {
bytes[i] = binary.charCodeAt(i);
/** 播放队列中的下一个句子 */
private playNext(): void {
if (this.queue.length === 0) {
this._isPlaying = false;
this.onEndCallback?.();
return;
}
const blob = new Blob([bytes], { type: mimeType });
const url = URL.createObjectURL(blob);
// 播放
const url = this.queue.shift()!;
const audio = new Audio(url);
this.audio = audio;
this._isPlaying = true;
audio.onended = () => {
URL.revokeObjectURL(url);
this._isPlaying = false;
this.audio = null;
this.onEndCallback?.();
this.playNext();
};
audio.onerror = () => {
console.error("[TTS] 播放失败");
console.error("[TTS] 播放失败，跳过");
URL.revokeObjectURL(url);
this._isPlaying = false;
this.audio = null;
this.onEndCallback?.();
this.playNext();
};
audio.play().catch((err) => {
console.error("[TTS] play() 被拒绝:", err);
this._isPlaying = false;
URL.revokeObjectURL(url);
this.audio = null;
this.onEndCallback?.();
this.playNext();
});
}
/** Base64 字符串转 Blob URL */
private base64ToBlobUrl(base64: string, mimeType: string): string {
const binary = atob(base64);
const bytes = new Uint8Array(binary.length);
for (let i = 0; i < binary.length; i++) {
bytes[i] = binary.charCodeAt(i);
}
const blob = new Blob([bytes], { type: mimeType });
return URL.createObjectURL(blob);
}
}

5
frontend/src/lib/websocket.ts
View File

@@ -6,7 +6,10 @@
import type { ClientMessage, ServerMessage, WsMessage } from "../types";
const WS_URL = "ws://localhost:8080/ws";
// WebSocket 地址：优先使用环境变量，否则基于当前页面地址自动推导
const WS_URL =
import.meta.env.VITE_WS_URL ||
`${window.location.protocol === "https:" ? "wss:" : "ws:"}//${window.location.host}/ws`;
const PING_INTERVAL = 30_000; // 30 秒心跳
const MAX_RECONNECT_DELAY = 30_000; // 最大重连延迟 30 秒

6
frontend/src/types/index.ts
View File

@@ -46,7 +46,8 @@ export interface QueryMessage {
type: "query";
request_id: string;
image: string; // Base64 JPEG（不含 data: 前缀）
audio: string; // Base64 PCM 16kHz
audio: string; // Base64 PCM 16kHz（文本输入时为空字符串）
text?: string; // 用户手动输入的文本（有值时跳过 STT）
mime_type?: string; // 默认 "audio/pcm"
}
@@ -114,7 +115,8 @@ export interface TTSAudioMessage {
request_id: string;
audio: string; // Base64 音频片段
mime_type: string; // "audio/mp3" 或 "audio/pcm"
is_last: boolean;
is_last: boolean; // 当前句子的音频是否完整（每句结束时为 true）
final: boolean; // 整轮 TTS 是否结束
}
export interface ErrorMessage {

62
frontend/vite.config.ts
View File

@@ -1,7 +1,63 @@
import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react'
import fs from 'fs'
import path from 'path'
const ortDistDir = path.resolve(__dirname, 'node_modules/onnxruntime-web/dist')
const vadDistDir = path.resolve(__dirname, 'node_modules/@ricky0123/vad-web/dist')
const publicDir = path.resolve(__dirname, 'public')
// 需要复制到 public 的静态资源
const staticFiles = [
{ src: vadDistDir, name: 'silero_vad_legacy.onnx' },
{ src: vadDistDir, name: 'silero_vad_v5.onnx' },
{ src: vadDistDir, name: 'vad.worklet.bundle.min.js' },
]
// https://vite.dev/config/
export default defineConfig({
plugins: [react()],
})
server: {
proxy: {
'/ws': {
target: 'http://localhost:8080',
ws: true,
},
'/api': {
target: 'http://localhost:8080',
},
},
},
plugins: [
react(),
{
name: 'serve-vad-assets',
// 1. 启动时将 VAD 模型/工作线程复制到 public
configResolved() {
for (const { src, name } of staticFiles) {
const dest = path.join(publicDir, name)
if (!fs.existsSync(dest)) {
fs.copyFileSync(path.join(src, name), dest)
}
}
},
// 2. 中间件拦截所有 WASM 相关请求，从原始路径提供文件
configureServer(server) {
server.middlewares.use((req, res, next) => {
const url = req.url?.split('?')[0] ?? ''
// 匹配任意路径下的 ort-wasm 文件（包括 .vite/deps/ 和根路径）
const wasmMatch = url.match(/\/(ort-wasm-simd-threaded\.(mjs|wasm))$/)
if (wasmMatch) {
const fileName = wasmMatch[1]
const filePath = path.join(ortDistDir, fileName)
if (fs.existsSync(filePath)) {
res.setHeader('Content-Type', fileName.endsWith('.wasm') ? 'application/wasm' : 'application/javascript')
res.setHeader('Cache-Control', 'no-cache')
res.end(fs.readFileSync(filePath))
return
}
}
next()
})
},
},
],
})

11
功能创意.md Normal file
View File

@@ -0,0 +1,11 @@
1.视频录制
2.对话翻译
3.对话总结
4.手动对话功能
5.视频框大小可调整，可最小化然后拖动
6.增加对话情景选择功能，参考豆包（面试官，英语老师，辩论赛，同声翻译等）
修复bug:
对话语音重复
麦克风开关