2026-06-12 17:08:20 +08:00
|
|
|
|
# 接口文档
|
|
|
|
|
|
|
|
|
|
|
|
## 概述
|
|
|
|
|
|
|
|
|
|
|
|
前后端通信接口定义。以 WebSocket 承载实时对话,REST 端点支撑基础运维。**暂不实现持久化**,但通过 Repository 接口模式为后续扩展预留接入点。
|
|
|
|
|
|
|
|
|
|
|
|
**设计原则**:
|
|
|
|
|
|
- WebSocket 为主:所有对话数据走 WebSocket
|
|
|
|
|
|
- REST 为辅:仅用于健康检查、会话管理等低频操作
|
|
|
|
|
|
- 接口先行:先定义契约,再填充实现——前后端可并行开发
|
|
|
|
|
|
|
|
|
|
|
|
## 接口全景
|
|
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
浏览器 Go Gateway :8080
|
|
|
|
|
|
WebSocket Client <--> /ws (实时对话)
|
|
|
|
|
|
HTTP Client --> GET /api/health
|
|
|
|
|
|
HTTP Client <--> POST/DELETE /api/sessions
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
|
|
## 一、WebSocket 协议
|
|
|
|
|
|
|
|
|
|
|
|
连接地址:`ws://localhost:8080/ws`
|
|
|
|
|
|
|
|
|
|
|
|
### 消息格式约定
|
|
|
|
|
|
|
|
|
|
|
|
所有 WebSocket 消息均为 JSON 文本帧,统一结构:
|
|
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
|
interface WsMessage {
|
|
|
|
|
|
type: string; // 消息类型,必填
|
|
|
|
|
|
request_id?: string; // 可选,用于请求-响应关联
|
|
|
|
|
|
timestamp?: number; // 可选,毫秒时间戳
|
|
|
|
|
|
[key: string]: any; // 类型特定字段
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
### 客户端 → 服务端消息
|
|
|
|
|
|
|
|
|
|
|
|
#### `query` — 发起一次视觉对话
|
|
|
|
|
|
|
|
|
|
|
|
用户说完话后,客户端同时发送当前图像帧和语音片段:
|
|
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
|
interface QueryMessage {
|
|
|
|
|
|
type: "query";
|
|
|
|
|
|
request_id: string; // 客户端生成的 UUID
|
|
|
|
|
|
image: string; // Base64 编码的 JPEG 图像(不含 data: 前缀)
|
|
|
|
|
|
audio: string; // Base64 编码的音频片段(PCM 16kHz)
|
|
|
|
|
|
mime_type?: string; // 音频格式,默认 "audio/pcm"
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
> 为什么图像和音频放在同一条消息里?因为 VAD 检测到用户说完话时,需要同时捕获"此刻的画面"和"说的话",拆成两条消息会增加时序同步的复杂度。
|
|
|
|
|
|
|
|
|
|
|
|
#### `config` — 更新会话配置
|
|
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
|
interface ConfigMessage {
|
|
|
|
|
|
type: "config";
|
|
|
|
|
|
payload: {
|
|
|
|
|
|
tts_enabled?: boolean; // 是否开启语音合成,默认 true
|
|
|
|
|
|
detail_level?: "low" | "high"; // 图像精度,默认 "low"
|
|
|
|
|
|
language?: string; // 交互语言,默认 "zh-CN"
|
|
|
|
|
|
};
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
#### `interrupt` — 打断当前回复
|
|
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
|
interface InterruptMessage {
|
|
|
|
|
|
type: "interrupt";
|
|
|
|
|
|
request_id?: string; // 可选,指定打断哪次请求
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
#### `ping` — 心跳保活
|
|
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
|
interface PingMessage {
|
|
|
|
|
|
type: "ping";
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
### 服务端 → 客户端消息
|
|
|
|
|
|
|
|
|
|
|
|
#### `connected` — 连接建立确认
|
|
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
|
interface ConnectedMessage {
|
|
|
|
|
|
type: "connected";
|
|
|
|
|
|
session_id: string; // 服务端生成的会话 ID
|
|
|
|
|
|
server_version: string; // 服务端版本号,如 "0.1.0"
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
#### `stt_result` — 语音识别结果
|
|
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
|
interface STTResultMessage {
|
|
|
|
|
|
type: "stt_result";
|
|
|
|
|
|
request_id: string;
|
|
|
|
|
|
text: string; // 识别出的用户语音文本
|
|
|
|
|
|
is_final: boolean; // 是否为最终结果
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
#### `llm_chunk` — LLM 流式输出片段
|
|
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
|
interface LLMChunkMessage {
|
|
|
|
|
|
type: "llm_chunk";
|
|
|
|
|
|
request_id: string;
|
|
|
|
|
|
delta: string; // 本次增量文本
|
|
|
|
|
|
role: "assistant";
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
#### `llm_done` — LLM 输出完成
|
|
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
|
interface LLMDoneMessage {
|
|
|
|
|
|
type: "llm_done";
|
|
|
|
|
|
request_id: string;
|
|
|
|
|
|
full_text: string; // 完整回复文本
|
|
|
|
|
|
tokens_used: {
|
|
|
|
|
|
prompt: number;
|
|
|
|
|
|
completion: number;
|
|
|
|
|
|
total: number;
|
|
|
|
|
|
};
|
|
|
|
|
|
model: string; // 实际使用的模型名
|
|
|
|
|
|
latency_ms: number; // 端到端延迟(毫秒)
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
#### `tts_audio` — TTS 音频流片段
|
|
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
|
interface TTSAudioMessage {
|
|
|
|
|
|
type: "tts_audio";
|
|
|
|
|
|
request_id: string;
|
|
|
|
|
|
audio: string; // Base64 编码的音频片段
|
2026-06-13 13:18:17 +08:00
|
|
|
|
mime_type: string; // "audio/mpeg"
|
2026-06-12 17:08:20 +08:00
|
|
|
|
is_last: boolean; // 是否为最后一片
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
2026-06-13 13:18:17 +08:00
|
|
|
|
**音频格式规范**(前端播放依赖此约定):
|
|
|
|
|
|
|
|
|
|
|
|
| 属性 | 值 | 说明 |
|
|
|
|
|
|
|------|------|------|
|
|
|
|
|
|
| 编码 | `audio/mpeg`(MP3) | 浏览器 `<audio>` 原生支持,OpenAI TTS 默认输出 |
|
|
|
|
|
|
| 采样率 | 24kHz | OpenAI TTS 默认 |
|
|
|
|
|
|
| 声道 | 单声道 | 语音不需要立体声 |
|
|
|
|
|
|
| 传输 | Base64 编码的 MP3 片段 | 每个 `tts_audio` 消息携带一个句子的音频 |
|
|
|
|
|
|
| 切片粒度 | 按句子切分 | LLM 输出中按 `。!?\n` 等标点切分,每个句子独立合成 |
|
|
|
|
|
|
|
|
|
|
|
|
**流式播放时序**:`tts_audio` 消息按句子顺序到达,前端应按序排队播放,不要等全部到齐再播。
|
|
|
|
|
|
|
|
|
|
|
|
**前端播放实现要点**:
|
|
|
|
|
|
|
|
|
|
|
|
1. **排队播放**:收到 `tts_audio` 时,将 Base64 解码为 Blob URL 并加入播放队列。第一片到达即开始播放,后续片段在 `onended` 回调中自动衔接。
|
|
|
|
|
|
2. **错误容错**:单个片段播放失败时跳过,继续播放队列中下一个,不中断整个回复。
|
|
|
|
|
|
3. **打断清理**:收到 `interrupt` 消息或用户触发打断时,清空播放队列并释放所有 Blob URL。
|
|
|
|
|
|
4. **类型锁定**:`mime_type` 字段固定为 `"audio/mpeg"`,前端解码时直接使用,无需运行时判断。
|
|
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
|
// 前端播放器伪代码
|
|
|
|
|
|
class AudioPlayer {
|
|
|
|
|
|
private queue: string[] = []; // Blob URL 队列
|
|
|
|
|
|
|
|
|
|
|
|
enqueue(base64: string) {
|
|
|
|
|
|
const url = decodeBase64Audio(base64, "audio/mpeg");
|
|
|
|
|
|
this.queue.push(url);
|
|
|
|
|
|
if (this.queue.length === 1) this.playNext(); // 第一片到了就开始播
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
private playNext() {
|
|
|
|
|
|
if (this.queue.length === 0) return;
|
|
|
|
|
|
const audio = new Audio(this.queue[0]);
|
|
|
|
|
|
audio.onended = () => { URL.revokeObjectURL(this.queue.shift()!); this.playNext(); };
|
|
|
|
|
|
audio.onerror = () => { URL.revokeObjectURL(this.queue.shift()!); this.playNext(); };
|
|
|
|
|
|
audio.play();
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
clear() { this.queue.forEach(url => URL.revokeObjectURL(url)); this.queue = []; }
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
2026-06-12 17:08:20 +08:00
|
|
|
|
#### `error` — 错误通知
|
|
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
|
interface ErrorMessage {
|
|
|
|
|
|
type: "error";
|
|
|
|
|
|
request_id?: string;
|
|
|
|
|
|
code: string; // 错误码,见下方错误码表
|
|
|
|
|
|
message: string; // 人类可读的错误描述
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
#### `pong` — 心跳响应
|
|
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
|
interface PongMessage {
|
|
|
|
|
|
type: "pong";
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
### 消息流时序
|
|
|
|
|
|
|
|
|
|
|
|
一次完整交互:
|
|
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
Client Server
|
|
|
|
|
|
| |
|
|
|
|
|
|
|-- query {image, audio} ------>|
|
|
|
|
|
|
|<-- stt_result {text} ---------|
|
|
|
|
|
|
| |
|
|
|
|
|
|
|<-- llm_chunk {delta: "这"} ---| (LLM 流式输出)
|
|
|
|
|
|
|<-- llm_chunk {delta: "是一"} -|
|
|
|
|
|
|
|<-- llm_chunk {delta: "朵花"} -|
|
|
|
|
|
|
|<-- llm_done {full_text} ------|
|
|
|
|
|
|
| |
|
|
|
|
|
|
|<-- tts_audio {audio} ---------| (TTS 音频流)
|
|
|
|
|
|
|<-- tts_audio {is_last: true} -|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
|
|
## 二、REST API
|
|
|
|
|
|
|
|
|
|
|
|
### 健康检查
|
|
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
GET /api/health
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
响应:
|
|
|
|
|
|
|
|
|
|
|
|
```json
|
|
|
|
|
|
{
|
|
|
|
|
|
"status": "ok",
|
|
|
|
|
|
"version": "0.1.0",
|
|
|
|
|
|
"uptime_seconds": 3600,
|
|
|
|
|
|
"active_sessions": 42
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
### 创建会话(可选,MVP 自动创建)
|
|
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
POST /api/sessions
|
|
|
|
|
|
Content-Type: application/json
|
|
|
|
|
|
|
|
|
|
|
|
{
|
|
|
|
|
|
"user_id": "optional-user-id",
|
|
|
|
|
|
"config": {
|
|
|
|
|
|
"tts_enabled": true,
|
|
|
|
|
|
"detail_level": "low",
|
|
|
|
|
|
"language": "zh-CN"
|
|
|
|
|
|
}
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
响应:
|
|
|
|
|
|
|
|
|
|
|
|
```json
|
|
|
|
|
|
{
|
|
|
|
|
|
"session_id": "550e8400-e29b-41d4-a716-446655440000",
|
|
|
|
|
|
"created_at": "2026-06-12T15:41:00Z"
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
### 销毁会话
|
|
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
DELETE /api/sessions/{session_id}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
响应:`204 No Content`
|
|
|
|
|
|
|
|
|
|
|
|
### 预留端点(暂不实现)
|
|
|
|
|
|
|
|
|
|
|
|
| 端点 | 方法 | 用途 |
|
|
|
|
|
|
|------|------|------|
|
|
|
|
|
|
| `/api/sessions/{id}/messages` | GET | 查询对话历史 |
|
|
|
|
|
|
| `/api/usage` | GET | 查询用量统计 |
|
|
|
|
|
|
| `/api/users/{id}/preferences` | GET/PUT | 用户偏好管理 |
|
|
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
2026-06-13 13:18:17 +08:00
|
|
|
|
## 三、AI 服务层接口
|
|
|
|
|
|
|
|
|
|
|
|
Go 网关内部与外部 AI 服务(Deepgram STT、GPT-4o、OpenAI TTS)的调用契约。前后端联调时,后端需实现这些接口。
|
|
|
|
|
|
|
|
|
|
|
|
### STT 服务接口
|
|
|
|
|
|
|
|
|
|
|
|
语音识别:接收前端采集的音频,返回识别文本。
|
|
|
|
|
|
|
|
|
|
|
|
```go
|
|
|
|
|
|
// STTService 语音识别服务契约。
|
|
|
|
|
|
type STTService interface {
|
|
|
|
|
|
// Recognize 识别一段完整音频,返回最终文本。
|
|
|
|
|
|
Recognize(ctx context.Context, audio []byte, opts STTOptions) (string, error)
|
|
|
|
|
|
|
|
|
|
|
|
// RecognizeStream 流式识别(边说边识别,可选实现)。
|
|
|
|
|
|
// audioStream 持续接收音频片段,返回的 channel 持续输出中间结果。
|
|
|
|
|
|
RecognizeStream(ctx context.Context, audioStream <-chan []byte, opts STTOptions) (<-chan STTPartial, error)
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
// STTOptions 语音识别参数。
|
|
|
|
|
|
type STTOptions struct {
|
|
|
|
|
|
Encoding string // "pcm_s16le" — 前端 VAD 输出格式
|
|
|
|
|
|
SampleRate int // 16000 — 前端麦克风采样率
|
|
|
|
|
|
Language string // "zh-CN"
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
// STTPartial 流式识别的中间/最终结果。
|
|
|
|
|
|
type STTPartial struct {
|
|
|
|
|
|
Text string
|
|
|
|
|
|
IsFinal bool
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
**Deepgram 接入约定**:
|
|
|
|
|
|
- 连接方式:WebSocket `wss://api.deepgram.com/v1/listen`
|
|
|
|
|
|
- 音频格式:PCM 16-bit signed little-endian,16kHz 单声道(与前端 `MicManager` 输出一致)
|
|
|
|
|
|
- 返回格式:`channel.alternatives[0].transcript`,`is_final` 字段标识最终结果
|
|
|
|
|
|
- 超时:单次识别 5 秒超时
|
|
|
|
|
|
|
|
|
|
|
|
### LLM 服务接口
|
|
|
|
|
|
|
|
|
|
|
|
多模态推理:接收图像 + 文本 + 对话历史,流式返回回复。
|
|
|
|
|
|
|
|
|
|
|
|
```go
|
|
|
|
|
|
// LLMService 多模态大模型服务契约。
|
|
|
|
|
|
type LLMService interface {
|
|
|
|
|
|
// ChatStream 流式推理,返回增量文本的 channel。
|
|
|
|
|
|
// 调用方必须消费 channel 直到 Done=true,否则需 cancel ctx 以释放连接。
|
|
|
|
|
|
ChatStream(ctx context.Context, req LLMRequest) (<-chan LLMChunk, error)
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
// LLMRequest 推理请求。
|
|
|
|
|
|
type LLMRequest struct {
|
|
|
|
|
|
Image []byte // JPEG 图片(已从 Base64 解码)
|
|
|
|
|
|
Text string // 用户语音识别后的文本
|
|
|
|
|
|
History []Message // 最近 N 轮对话历史
|
|
|
|
|
|
Language string // "zh-CN"
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
// LLMChunk 流式推理的一个增量片段。
|
|
|
|
|
|
type LLMChunk struct {
|
|
|
|
|
|
Delta string // 增量文本
|
|
|
|
|
|
Done bool // 是否结束
|
|
|
|
|
|
TokensUsed *TokenUsage // 仅 Done=true 时有值
|
|
|
|
|
|
Model string // 实际使用的模型名
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
// TokenUsage 用量统计。
|
|
|
|
|
|
type TokenUsage struct {
|
|
|
|
|
|
Prompt int
|
|
|
|
|
|
Completion int
|
|
|
|
|
|
Total int
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
**OpenAI API 接入约定**:
|
|
|
|
|
|
- 端点:`POST https://api.openai.com/v1/chat/completions`
|
|
|
|
|
|
- 图片传入:`image_url` 字段使用 `data:image/jpeg;base64,...` 格式
|
|
|
|
|
|
- 流式响应:`stream: true`,通过 SSE 逐 chunk 返回
|
|
|
|
|
|
- Prompt 结构:
|
|
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
system: "你是一个视觉助手。用户通过摄像头看到一个场景,并用语音向你提问。
|
|
|
|
|
|
请用简洁自然的中文回答。如果涉及视觉描述,先说'我看到...'。"
|
|
|
|
|
|
user: [图片 + 用户语音文本]
|
|
|
|
|
|
(重复 History 中的历史消息)
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
- 超时:10 秒,超时返回 `LLM_TIMEOUT` 错误
|
|
|
|
|
|
- 模型选择:默认 `gpt-4o`,由 Model Router 按需切换
|
|
|
|
|
|
|
|
|
|
|
|
### TTS 服务接口
|
|
|
|
|
|
|
|
|
|
|
|
语音合成:接收文本流,输出音频 chunk 流。
|
|
|
|
|
|
|
|
|
|
|
|
```go
|
|
|
|
|
|
// TTSService 语音合成服务契约。
|
|
|
|
|
|
type TTSService interface {
|
|
|
|
|
|
// SynthesizeStream 流式合成。
|
|
|
|
|
|
// textStream 接收句子级文本(由 Orchestrator 的句子切分器产出),
|
|
|
|
|
|
// 返回的 channel 持续输出 MP3 音频 chunk。
|
|
|
|
|
|
SynthesizeStream(ctx context.Context, textStream <-chan string, opts TTSOptions) (<-chan TTSChunk, error)
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
// TTSOptions 合成参数。
|
|
|
|
|
|
type TTSOptions struct {
|
|
|
|
|
|
Voice string // "alloy" | "nova" | "shimmer" | ...
|
|
|
|
|
|
Speed float64 // 1.0 为正常语速
|
|
|
|
|
|
OutputFmt string // "mp3" — 固定使用 MP3,浏览器原生支持
|
|
|
|
|
|
SampleRate int // 24000
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
// TTSChunk 一个音频片段。
|
|
|
|
|
|
type TTSChunk struct {
|
|
|
|
|
|
Audio []byte // MP3 音频数据(未 Base64 编码,由发送层编码)
|
|
|
|
|
|
IsLast bool // 是否为最后一片
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
**OpenAI TTS 接入约定**:
|
|
|
|
|
|
- 端点:`POST https://api.openai.com/v1/audio/speech`
|
|
|
|
|
|
- 模型:`tts-1`(低延迟优先)或 `tts-1-hd`(高音质)
|
|
|
|
|
|
- 输出格式:`mp3`,24kHz
|
|
|
|
|
|
- 流式:使用 `response_format: "mp3"` 并读取 response body 流
|
|
|
|
|
|
- 超时:单个句子 5 秒超时
|
|
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
|
|
## 四、AI 编排器(Orchestrator)
|
|
|
|
|
|
|
|
|
|
|
|
### 编排策略:句子级流式并行
|
|
|
|
|
|
|
|
|
|
|
|
核心矛盾:LLM 流式输出逐 token,TTS 需要完整句子才能合成。解法:**句子切分器 + 管道并行**。
|
|
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
LLM 流式输出: "这" "是一" "朵红色" "的花。" "它看起" "来很美" "丽。"
|
|
|
|
|
|
↓
|
|
|
|
|
|
┌── 句子检测器(按 。!?\n 切分)──┐
|
|
|
|
|
|
↓ ↓
|
|
|
|
|
|
句子1: "这是一朵红色的花。" 句子2: "它看起来很美丽。"
|
|
|
|
|
|
↓ ↓
|
|
|
|
|
|
TTS 合成 TTS 合成
|
|
|
|
|
|
↓ ↓
|
|
|
|
|
|
音频 chunk → 推送前端 音频 chunk → 推送前端
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
**时序保证**:
|
|
|
|
|
|
- `llm_chunk` 消息一定先于对应句子的 `tts_audio` 到达客户端
|
|
|
|
|
|
- 用户先看到文字,紧接着听到语音(感知延迟 < 0.5 秒)
|
|
|
|
|
|
- 不必等 LLM 全部输出完才开始 TTS
|
|
|
|
|
|
|
|
|
|
|
|
### Orchestrator 接口
|
|
|
|
|
|
|
|
|
|
|
|
```go
|
|
|
|
|
|
// Orchestrator AI 编排器,协调 STT → LLM → TTS 全链路。
|
|
|
|
|
|
type Orchestrator struct {
|
|
|
|
|
|
stt STTService
|
|
|
|
|
|
llm LLMService
|
|
|
|
|
|
tts TTSService
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
// ProcessQuery 处理一次完整的视觉对话请求。
|
|
|
|
|
|
// 通过 client 向前端实时推送 stt_result、llm_chunk、llm_done、tts_audio 消息。
|
|
|
|
|
|
func (o *Orchestrator) ProcessQuery(ctx context.Context, client MessageSender, req *QueryRequest) {
|
|
|
|
|
|
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
|
|
|
|
|
|
defer cancel()
|
|
|
|
|
|
|
|
|
|
|
|
// Step 1: STT — 识别用户语音
|
|
|
|
|
|
text, err := o.stt.Recognize(ctx, req.Audio, STTOptions{
|
|
|
|
|
|
Encoding: "pcm_s16le", SampleRate: 16000, Language: "zh-CN",
|
|
|
|
|
|
})
|
|
|
|
|
|
if err != nil {
|
|
|
|
|
|
client.SendError(req.RequestID, "STT_ERROR", err.Error())
|
|
|
|
|
|
return
|
|
|
|
|
|
}
|
|
|
|
|
|
client.SendSTTResult(req.RequestID, text, true)
|
|
|
|
|
|
|
|
|
|
|
|
// Step 2: LLM 流式输出 + 句子切分
|
|
|
|
|
|
llmStream, _ := o.llm.ChatStream(ctx, LLMRequest{
|
|
|
|
|
|
Image: req.Image, Text: text, Language: "zh-CN",
|
|
|
|
|
|
})
|
|
|
|
|
|
|
|
|
|
|
|
sentenceCh := make(chan string, 4)
|
|
|
|
|
|
go func() {
|
|
|
|
|
|
defer close(sentenceCh)
|
|
|
|
|
|
var buf strings.Builder
|
|
|
|
|
|
var fullText strings.Builder
|
|
|
|
|
|
for chunk := range llmStream {
|
|
|
|
|
|
// 即时推送文字给客户端(逐 token 显示)
|
|
|
|
|
|
client.SendLLMChunk(req.RequestID, chunk.Delta)
|
|
|
|
|
|
fullText.WriteString(chunk.Delta)
|
|
|
|
|
|
buf.WriteString(chunk.Delta)
|
|
|
|
|
|
// 遇到句子边界就吐出
|
|
|
|
|
|
if isSentenceEnd(chunk.Delta) {
|
|
|
|
|
|
sentenceCh <- buf.String()
|
|
|
|
|
|
buf.Reset()
|
|
|
|
|
|
}
|
|
|
|
|
|
}
|
|
|
|
|
|
// 最后一段不足一句的也吐出
|
|
|
|
|
|
if buf.Len() > 0 {
|
|
|
|
|
|
sentenceCh <- buf.String()
|
|
|
|
|
|
}
|
|
|
|
|
|
// 推送 llm_done
|
|
|
|
|
|
client.SendLLMDone(req.RequestID, fullText.String(), chunk.TokensUsed, chunk.Model)
|
|
|
|
|
|
}()
|
|
|
|
|
|
|
|
|
|
|
|
// Step 3: TTS 并行消费句子流
|
|
|
|
|
|
ttsStream, _ := o.tts.SynthesizeStream(ctx, sentenceCh, TTSOptions{
|
|
|
|
|
|
Voice: "alloy", OutputFmt: "mp3", SampleRate: 24000,
|
|
|
|
|
|
})
|
|
|
|
|
|
for chunk := range ttsStream {
|
|
|
|
|
|
client.SendTTSAudio(req.RequestID, chunk.Audio, chunk.IsLast)
|
|
|
|
|
|
}
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
// isSentenceEnd 判断 delta 中是否包含句子结束标志。
|
|
|
|
|
|
func isSentenceEnd(delta string) bool {
|
|
|
|
|
|
return strings.ContainsAny(delta, "。!?\n.!?\n")
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
### 并发控制
|
|
|
|
|
|
|
|
|
|
|
|
- 每个 `ProcessQuery` 调用在独立 goroutine 中运行
|
|
|
|
|
|
- `context.WithTimeout` 确保 10 秒总超时
|
|
|
|
|
|
- `interrupt` 消息触发 `cancel()`,LLM/TTS 流式全部中断
|
|
|
|
|
|
- 同一 session 内同时只允许一个活跃请求,新请求自动取消上一个
|
|
|
|
|
|
|
|
|
|
|
|
### 错误处理与降级
|
|
|
|
|
|
|
|
|
|
|
|
| 故障点 | 处理策略 | 客户端表现 |
|
|
|
|
|
|
|--------|---------|-----------|
|
|
|
|
|
|
| STT 失败 | 发送 `STT_ERROR`,终止本次请求 | 回退到纯文本模式 |
|
|
|
|
|
|
| LLM 超时(>10s) | 发送 `LLM_TIMEOUT`,取消 TTS | 提示用户重试 |
|
|
|
|
|
|
| LLM 部分输出后失败 | 已推送的 `llm_chunk` 保留,发送 `error` 通知中断 | 显示已收到的部分文字 |
|
|
|
|
|
|
| TTS 失败 | 静默跳过,`llm_done` 正常发送 | 只有文字回复,无语音 |
|
|
|
|
|
|
| TTS 部分失败 | 已推送的音频保留,后续句子跳过 | 部分句子有语音 |
|
|
|
|
|
|
| interrupt 打断 | cancel context,清空所有流 | 前端清空播放队列 |
|
|
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
2026-06-13 13:24:25 +08:00
|
|
|
|
## 五、Session Manager
|
|
|
|
|
|
|
|
|
|
|
|
WebSocket Handler 和 AI Orchestrator 之间的会话管理层。负责维护会话生命周期、对话上下文和配置状态。
|
|
|
|
|
|
|
|
|
|
|
|
### Redis 数据结构
|
|
|
|
|
|
|
|
|
|
|
|
每个会话在 Redis 中占 2 个 key:
|
|
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
session:{id}:meta → Hash (会话元数据)
|
|
|
|
|
|
session:{id}:history → List (对话历史)
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
**Hash — `session:{id}:meta`**
|
|
|
|
|
|
|
|
|
|
|
|
| field | 类型 | 示例 | 说明 |
|
|
|
|
|
|
|-------|------|------|------|
|
|
|
|
|
|
| `session_id` | string | `"550e8400-..."` | 主键冗余 |
|
|
|
|
|
|
| `config.tts_enabled` | string | `"true"` | Redis Hash 值均为 string |
|
|
|
|
|
|
| `config.detail_level` | string | `"low"` | |
|
|
|
|
|
|
| `config.language` | string | `"zh-CN"` | |
|
|
|
|
|
|
| `created_at` | string | `"2026-06-13T10:00:00Z"` | RFC3339 |
|
|
|
|
|
|
| `last_active` | string | `"2026-06-13T10:05:30Z"` | 每次消息刷新 |
|
|
|
|
|
|
| `active_request_id` | string | `"uuid"` 或 `""` | 当前处理中的请求 ID,用于 interrupt |
|
|
|
|
|
|
|
|
|
|
|
|
**List — `session:{id}:history`**
|
|
|
|
|
|
|
|
|
|
|
|
每个元素是一条 JSON 序列化的 Message:
|
|
|
|
|
|
|
|
|
|
|
|
```json
|
|
|
|
|
|
{"role":"user","content":"这是什么花?"}
|
|
|
|
|
|
{"role":"assistant","content":"这是一朵红色的玫瑰。"}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
- `LPUSH` 新消息到左头(最新在前)
|
|
|
|
|
|
- `LRANGE 0 {limit-1}` 取最近 N 轮
|
|
|
|
|
|
- `LTRIM 0 {max-1}` 限制总条数(默认保留最近 20 条 = 10 轮对话)
|
|
|
|
|
|
|
|
|
|
|
|
### TTL 策略
|
|
|
|
|
|
|
|
|
|
|
|
| 场景 | TTL | 说明 |
|
|
|
|
|
|
|------|-----|------|
|
|
|
|
|
|
| 创建时 | 30 分钟 | `EXPIRE` 设置 |
|
|
|
|
|
|
| 每次收到消息 | 重置 30 分钟 | `EXPIRE` 刷新 |
|
|
|
|
|
|
| WebSocket 断开 | 不主动删 | 等自然过期,支持重连恢复 |
|
|
|
|
|
|
| 超过 30 分钟无活动 | 自动过期 | Redis 自动清理 meta + history |
|
|
|
|
|
|
| 显式销毁(REST API) | 立即 `DEL` | 两个 key 一起删 |
|
|
|
|
|
|
|
|
|
|
|
|
### 接口定义
|
|
|
|
|
|
|
|
|
|
|
|
```go
|
|
|
|
|
|
// SessionManager 会话管理器。
|
|
|
|
|
|
// WebSocket Handler 通过此接口操作会话,不直接接触 Redis。
|
|
|
|
|
|
type SessionManager interface {
|
|
|
|
|
|
// Create 创建新会话,返回 session ID。
|
|
|
|
|
|
Create(ctx context.Context, config models.SessionConfig) (string, error)
|
|
|
|
|
|
|
|
|
|
|
|
// Get 获取会话(含 config)。不存在返回 ErrSessionNotFound。
|
|
|
|
|
|
Get(ctx context.Context, sessionID string) (*models.Session, error)
|
|
|
|
|
|
|
|
|
|
|
|
// UpdateConfig 更新会话配置(config 消息触发)。
|
|
|
|
|
|
UpdateConfig(ctx context.Context, sessionID string, patch models.SessionConfigPatch) error
|
|
|
|
|
|
|
|
|
|
|
|
// GetHistory 获取最近 N 轮对话历史(供 Orchestrator 构建 LLM 上下文)。
|
|
|
|
|
|
GetHistory(ctx context.Context, sessionID string, limit int) ([]models.Message, error)
|
|
|
|
|
|
|
|
|
|
|
|
// AppendMessage 追加一条对话消息,同时刷新 TTL。
|
|
|
|
|
|
AppendMessage(ctx context.Context, sessionID string, msg models.Message) error
|
|
|
|
|
|
|
|
|
|
|
|
// SetActiveRequest 标记当前正在处理的请求 ID(interrupt 用)。
|
|
|
|
|
|
SetActiveRequest(ctx context.Context, sessionID string, requestID string) error
|
|
|
|
|
|
|
|
|
|
|
|
// ClearActiveRequest 清除活跃请求标记(请求完成或中断后)。
|
|
|
|
|
|
ClearActiveRequest(ctx context.Context, sessionID string) error
|
|
|
|
|
|
|
|
|
|
|
|
// Touch 刷新 TTL(心跳时调用)。
|
|
|
|
|
|
Touch(ctx context.Context, sessionID string) error
|
|
|
|
|
|
|
|
|
|
|
|
// Destroy 显式销毁会话(REST API DELETE 或连接断开清理)。
|
|
|
|
|
|
Destroy(ctx context.Context, sessionID string) error
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
### WebSocket Handler 集成
|
|
|
|
|
|
|
|
|
|
|
|
```go
|
|
|
|
|
|
// query 分支
|
|
|
|
|
|
case "query":
|
|
|
|
|
|
var msg models.WsQuery
|
|
|
|
|
|
json.Unmarshal(message, &msg)
|
|
|
|
|
|
|
|
|
|
|
|
sessionMgr.Touch(ctx, sessionID) // 刷新 TTL
|
|
|
|
|
|
sessionMgr.SetActiveRequest(ctx, sessionID, msg.RequestID) // 标记活跃请求
|
|
|
|
|
|
|
|
|
|
|
|
history, _ := sessionMgr.GetHistory(ctx, sessionID, 20) // 获取对话上下文
|
|
|
|
|
|
|
|
|
|
|
|
go orchestrator.ProcessQuery(ctx, client, &msg, history) // 异步编排
|
|
|
|
|
|
|
|
|
|
|
|
// interrupt 分支
|
|
|
|
|
|
case "interrupt":
|
|
|
|
|
|
reqID, _ := sessionMgr.GetActiveRequestID(ctx, sessionID)
|
|
|
|
|
|
if reqID != "" {
|
|
|
|
|
|
cancelFunc(reqID) // 取消对应 context
|
|
|
|
|
|
sessionMgr.ClearActiveRequest(ctx, sessionID)
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
// 连接断开
|
|
|
|
|
|
// 不调用 Destroy,让 session 自然过期(支持重连恢复)
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
### MVP 内存实现
|
|
|
|
|
|
|
|
|
|
|
|
联调阶段无 Redis 时,用同一接口的内存实现:
|
|
|
|
|
|
|
|
|
|
|
|
```go
|
|
|
|
|
|
type InMemorySessionManager struct {
|
|
|
|
|
|
mu sync.RWMutex
|
|
|
|
|
|
sessions map[string]*sessionEntry
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
type sessionEntry struct {
|
|
|
|
|
|
session models.Session
|
|
|
|
|
|
history []models.Message
|
|
|
|
|
|
activeReqID string
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
注入时根据配置切换:
|
|
|
|
|
|
|
|
|
|
|
|
```go
|
|
|
|
|
|
var sessionMgr SessionManager
|
|
|
|
|
|
if cfg.Redis.Addr != "" {
|
|
|
|
|
|
sessionMgr = NewRedisSessionManager(redisClient, 30*time.Minute, 20)
|
|
|
|
|
|
} else {
|
|
|
|
|
|
sessionMgr = NewInMemorySessionManager()
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
|
|
## 六、数据模型
|
2026-06-13 13:18:17 +08:00
|
|
|
|
|
|
|
|
|
|
> 注:Go 和 TypeScript 的数据模型定义见下方。AI 服务层的 Go 模型见上方"AI 服务层接口"章节。
|
2026-06-12 17:08:20 +08:00
|
|
|
|
|
|
|
|
|
|
### Go 后端模型
|
|
|
|
|
|
|
|
|
|
|
|
```go
|
|
|
|
|
|
// ---- 核心模型(MVP 实现)----
|
|
|
|
|
|
|
|
|
|
|
|
type Session struct {
|
|
|
|
|
|
ID string `json:"session_id"`
|
|
|
|
|
|
CreatedAt time.Time `json:"created_at"`
|
|
|
|
|
|
Config SessionConfig `json:"config"`
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
type SessionConfig struct {
|
|
|
|
|
|
TTSEnabled bool `json:"tts_enabled"`
|
|
|
|
|
|
DetailLevel string `json:"detail_level"` // "low" | "high"
|
|
|
|
|
|
Language string `json:"language"`
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
type QueryRequest struct {
|
|
|
|
|
|
RequestID string `json:"request_id"`
|
|
|
|
|
|
Image []byte `json:"-"` // Base64 解码后
|
|
|
|
|
|
Audio []byte `json:"-"` // Base64 解码后
|
|
|
|
|
|
MimeType string `json:"mime_type"`
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
type Message struct {
|
|
|
|
|
|
Role string `json:"role"` // "user" | "assistant"
|
|
|
|
|
|
Content string `json:"content"`
|
|
|
|
|
|
ImageURL string `json:"image_url,omitempty"`
|
|
|
|
|
|
TokensUsed int `json:"tokens_used,omitempty"`
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
### TypeScript 前端模型
|
|
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
|
interface Session {
|
|
|
|
|
|
sessionId: string;
|
|
|
|
|
|
createdAt: string;
|
|
|
|
|
|
config: SessionConfig;
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
interface SessionConfig {
|
|
|
|
|
|
ttsEnabled: boolean;
|
|
|
|
|
|
detailLevel: "low" | "high";
|
|
|
|
|
|
language: string;
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
interface ChatMessage {
|
|
|
|
|
|
role: "user" | "assistant";
|
|
|
|
|
|
content: string;
|
|
|
|
|
|
imageUrl?: string;
|
|
|
|
|
|
timestamp: number;
|
|
|
|
|
|
tokensUsed?: number;
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
// WebSocket 消息联合类型
|
|
|
|
|
|
type ServerMessage =
|
|
|
|
|
|
| ConnectedMessage
|
|
|
|
|
|
| STTResultMessage
|
|
|
|
|
|
| LLMChunkMessage
|
|
|
|
|
|
| LLMDoneMessage
|
|
|
|
|
|
| TTSAudioMessage
|
|
|
|
|
|
| ErrorMessage
|
|
|
|
|
|
| PongMessage;
|
|
|
|
|
|
|
|
|
|
|
|
type ClientMessage =
|
|
|
|
|
|
| QueryMessage
|
|
|
|
|
|
| ConfigMessage
|
|
|
|
|
|
| InterruptMessage
|
|
|
|
|
|
| PingMessage;
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
2026-06-13 13:24:25 +08:00
|
|
|
|
## 七、扩展接口设计
|
2026-06-12 17:08:20 +08:00
|
|
|
|
|
|
|
|
|
|
通过 Repository 接口隔离存储层,MVP 用内存实现,后续替换为数据库——业务逻辑零改动。
|
|
|
|
|
|
|
|
|
|
|
|
```go
|
|
|
|
|
|
// HistoryRepository — 对话历史存储契约
|
|
|
|
|
|
// MVP: 内存实现(session 内有效,断开即丢)
|
|
|
|
|
|
// 后续: PostgreSQL 实现
|
|
|
|
|
|
type HistoryRepository interface {
|
|
|
|
|
|
SaveMessage(ctx context.Context, sessionID string, msg Message) error
|
|
|
|
|
|
GetMessages(ctx context.Context, sessionID string, limit int) ([]Message, error)
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
// UsageRepository — 用量统计存储契约
|
|
|
|
|
|
// MVP: 内存计数器
|
|
|
|
|
|
// 后续: PostgreSQL 按天聚合
|
|
|
|
|
|
type UsageRepository interface {
|
|
|
|
|
|
RecordUsage(ctx context.Context, sessionID string, usage UsageRecord) error
|
|
|
|
|
|
GetDailyUsage(ctx context.Context, userID string, days int) ([]UsageDaily, error)
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
MVP 内存实现:
|
|
|
|
|
|
|
|
|
|
|
|
```go
|
|
|
|
|
|
type InMemoryHistory struct {
|
|
|
|
|
|
mu sync.RWMutex
|
|
|
|
|
|
sessions map[string][]Message
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
func (h *InMemoryHistory) SaveMessage(ctx context.Context, sessionID string, msg Message) error {
|
|
|
|
|
|
h.mu.Lock()
|
|
|
|
|
|
defer h.mu.Unlock()
|
|
|
|
|
|
h.sessions[sessionID] = append(h.sessions[sessionID], msg)
|
|
|
|
|
|
return nil
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
func (h *InMemoryHistory) GetMessages(ctx context.Context, sessionID string, limit int) ([]Message, error) {
|
|
|
|
|
|
h.mu.RLock()
|
|
|
|
|
|
defer h.mu.RUnlock()
|
|
|
|
|
|
msgs := h.sessions[sessionID]
|
|
|
|
|
|
if limit > 0 && len(msgs) > limit {
|
|
|
|
|
|
msgs = msgs[len(msgs)-limit:]
|
|
|
|
|
|
}
|
|
|
|
|
|
return msgs, nil
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
注入点(应用启动时根据配置选择实现):
|
|
|
|
|
|
|
|
|
|
|
|
```go
|
|
|
|
|
|
func NewApp(cfg *Config) *App {
|
|
|
|
|
|
var history HistoryRepository
|
|
|
|
|
|
var usage UsageRepository
|
|
|
|
|
|
|
|
|
|
|
|
switch cfg.Storage.Driver {
|
|
|
|
|
|
case "postgres":
|
|
|
|
|
|
pool, _ := pgxpool.New(ctx, cfg.Storage.DSN)
|
|
|
|
|
|
history = &PgHistory{pool: pool}
|
|
|
|
|
|
usage = &PgUsage{pool: pool}
|
|
|
|
|
|
default: // "memory" — MVP 默认
|
|
|
|
|
|
history = &InMemoryHistory{sessions: make(map[string][]Message)}
|
|
|
|
|
|
usage = &InMemoryUsage{}
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
return &App{
|
|
|
|
|
|
orchestrator: NewOrchestrator(cfg.AI, history, usage),
|
|
|
|
|
|
sessionMgr: NewSessionManager(cfg.Session, history),
|
|
|
|
|
|
}
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
> 依赖倒置原则——业务层依赖接口,不依赖具体实现。MVP 注入 `InMemoryHistory`,上线时一行代码换成 `PgHistory`。
|
|
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
2026-06-13 13:24:25 +08:00
|
|
|
|
## 八、错误码
|
2026-06-12 17:08:20 +08:00
|
|
|
|
|
|
|
|
|
|
| 错误码 | 含义 | 客户端处理建议 |
|
|
|
|
|
|
|--------|------|--------------|
|
|
|
|
|
|
| `INVALID_MESSAGE` | 消息格式不合法 | 检查 JSON 结构,不重试 |
|
|
|
|
|
|
| `SESSION_NOT_FOUND` | 会话不存在或已过期 | 重新建立 WebSocket 连接 |
|
|
|
|
|
|
| `RATE_LIMITED` | 请求频率超限 | 延迟后重试,提示用户稍等 |
|
|
|
|
|
|
| `IMAGE_TOO_LARGE` | 图像超过 4MB 限制 | 降低分辨率或压缩质量 |
|
|
|
|
|
|
| `AUDIO_TOO_SHORT` | 音频片段 < 250ms | 忽略,等待下次语音输入 |
|
|
|
|
|
|
| `LLM_TIMEOUT` | LLM 推理超时(>10s) | 提示用户重试 |
|
|
|
|
|
|
| `LLM_ERROR` | LLM 服务异常 | 提示用户重试,服务端记录日志 |
|
|
|
|
|
|
| `STT_ERROR` | 语音识别失败 | 回退到纯文本输入模式 |
|
|
|
|
|
|
| `TTS_ERROR` | 语音合成失败 | 静默回退到纯文本回复 |
|
|
|
|
|
|
| `INTERNAL_ERROR` | 服务端内部错误 | 提示用户重试 |
|
|
|
|
|
|
|
2026-06-13 13:24:25 +08:00
|
|
|
|
## 九、连接管理
|
2026-06-12 17:08:20 +08:00
|
|
|
|
|
|
|
|
|
|
**心跳机制**:客户端每 30 秒发送 `ping`,服务端回复 `pong`。超过 60 秒无 `ping`,服务端判定连接断开并清理会话资源。
|
|
|
|
|
|
|
|
|
|
|
|
**重连策略**(指数退避 + 抖动):
|
|
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
|
function reconnect(attempt: number) {
|
|
|
|
|
|
const delay = Math.min(1000 * Math.pow(2, attempt), 30000); // 最大 30s
|
|
|
|
|
|
const jitter = Math.random() * 1000;
|
|
|
|
|
|
setTimeout(() => connect(), delay + jitter);
|
|
|
|
|
|
}
|
|
|
|
|
|
// attempt: 0 → 1s, 1 → 2s, 2 → 4s, 3 → 8s, ... 最大 30s
|
|
|
|
|
|
```
|