接入隔离电脑画面与人工接管续跑

docs/agent-platform-implementation.md

@@ -83,3 +83,6 @@
 - 临时助手阶段：新增创建并分派任务的工具，助手没有单聊、不是永久团队成员，只能执行所属目标的当前阶段；固化隔离模式，禁止提权和跨任务使用。目标结束后退役，按原 Runner 进行持久清理重试；清理只删除助手执行数据，保留控制面产物。详见 `docs/task-helpers.md`。
 - Web、iOS、Android 已加入临时助手标识和持久成员选择过滤。iOS 27 项 Workspace 测试通过；Android 模型测试和 Debug Kotlin 编译通过。真实 Hermes Worker / Docker 的临时助手执行和退役清理验收已通过。
 - 临时助手最终验收：服务端类型检查与 759 项全量测试通过，5 项真实环境测试默认跳过；打包 Runner 的真实临时助手完成/清理验收通过；Web 4 项流程测试和临时助手标识截图通过。iOS 27 项 Workspace 测试、Android 31 项模型测试和 Debug Kotlin 编译通过。测试容器全部清理。其余镜像管理、电脑接管、共享环境和桌面交付项继续推进。
+
+- 电脑控制阶段：Web、macOS、iOS、Android 入口已接入只读画面与独立人工控制凭据；暂停模型、等待在途工具、同一电脑递增代次转交、交还后继续原任务。真实 Docker 终端点击/输入/文件证明和原任务续跑通过。详见 `docs/computer-control.md`。跨端原生视觉、镜像管理、共享环境和最终桌面交付仍待完成。
+- 电脑控制源码检查：类型检查和 Web 构建通过，762 项全量测试通过，5 项浏览器流程通过，iOS 28 项 Workspace 测试通过，Android 模型测试和 Kotlin 编译通过。窄屏电脑面板截图已核对。本轮验收容器全部清理，正式 Web 服务 PID 28292 未改变。

docs/computer-control.md

@@ -0,0 +1,29 @@
+# 电脑查看与人工接管
+
+Web 和 macOS 共用电脑面板；iOS、Android 使用相同的账号 API。只对当前会话成员和当前目标的临时助手显示电脑入口，旧服务不声明 `computerControl` 时不显示入口。模型没有人工接管工具，也不接收控制凭据。
+
+## 行为
+
+1. 打开面板先查看画面，不获得输入权。只读截图走独立通道，不排在长时间终端命令之后。
+2. 用户明确点击接管后，控制面签发限时控制凭据。Runner 暂停模型进程并等待已经提交的工具结束；这期间显示“正在等待当前操作结束”，输入保持禁用。
+3. 原电脑确认没有在途工具后，以递增的租约代次转交给人工；保留同一容器、桌面应用和工作区。已经启动的浏览器等后台应用继续运行。
+4. 输入必须携带本页面控制凭据、当前租约代次及最近画面 ID。坐标越界、旧画面、旧代次和其他页面的请求会被拒绝。重复操作 ID 不会再次输入。
+5. 交还说明与现有会话、工具结果一起传入新的模型段，保留原工作标识，继续原任务。交还是模型的续跑段，不是复用已终止的模型进程。
+6. 无任务时可直接打开电脑。结束该控制会话会停止电脑并保留持久工作区。
+
+## 权限与故障
+
+控制凭据仅由接管接口返回，不出现在状态接口；服务端保存其散列及加密副本，用于同一请求的安全重试。有效期为 30 秒，前台页面约每 8 秒续期。Runner 每 5 秒检查授权；到期、账号撤权、Agent 退役后停止所属工作，不自动恢复。移动端进入后台和 Web 页面隐藏后停止续期。再次操作需要新接管。
+
+每次操作仍经过账号、Agent 来源、Runner 能力、授权版本和环境归属检查。模型无法选择真实账号、Runner、环境 ID 或读取人工凭据。Runner 的控制能力为 `computer-control-v1`，不支持时明确拒绝，不回退宿主电脑。
+
+接管请求编号在响应不确定时保留，允许同一页面重试；其他页面等待原会话结束或到期。模型已提交但没有回执的工具，会在续跑上下文中注明结果不确定，不能假装未执行。停止未确认时保留资源占用，不能交给下一使用者。
+
+当前交互为定期 PNG 画面、点击、文字和常用按键；不宣称视频串流、音频或完整系统远程桌面。CUA 接受输入只能证明命令已送达，界面效果由下一帧确认。
+
+## 验证
+
+- 控制面测试：只读状态不返回凭据、其他账号/页面拒绝、令牌和授权版本校验、同一接管请求重试。
+- Runner 测试：查看不调用输入、代次/画面/坐标校验、输入去重、撤权停止、原电脑租约转交。
+- 真实 Docker + 打包 Runner + Hermes 模型循环：执行中接管、原任务转为等待、真实截图、点击打开终端、输入命令生成文件、错误令牌拒绝、交还后原任务完成。模型响应来自本地确定性测试服务，未调用付费模型。
+- Web 浏览器测试覆盖只读、接管、输入、交还和窄屏；iOS 28 项 Workspace 测试通过，包含授权/画面代次和撤权清理；Android 模型测试与 Debug Kotlin 编译通过。原生电脑面板的模拟器视觉验收还需完成，不能以编译替代。

docs/computer-provider.md

@@ -10,7 +10,7 @@
 - 仅挂载该环境的持久工作目录；不挂载 Docker socket、宿主 HOME、Hermes 配置或凭据。命令固定以 guest UID/GID 1000 执行，工作目录为 `/home/cua/workspace`。
 - 固定独立 IPC/cgroup 命名空间、禁止提权、收紧 capability，并设置 CPU、内存、swap 与进程数上限。每次执行前核对运行实例、镜像、归属、挂载与约束。
 - 默认 `--network none`，不发布任何端口。容器只有 loopback 网卡，不能直接访问宿主服务或外网。自己的主机名映射到 loopback，保证 XFCE/VNC 在无网络环境下正常启动。可通过专属进程通道启用受控公网代理，仍不增加外部网卡或端口。详见 `docs/computer-network.md`。
-- 桌面通过容器内的 CUA socket 操作；验收已读取真实截图。后续查看/接管接口仍需账号鉴权和独立的控制租约。
+- 桌面通过容器内的 CUA socket 操作；验收已读取真实截图。查看和人工接管已接入账号鉴权及独立控制租约，见 `docs/computer-control.md`。
 - 取消或无法确认的 guest 命令会停止所属私有容器，确认停止后才释放串行执行位置。仅终止 `docker exec` 客户端不能证明 guest 进程已停止。
 - 删除容器保留工作目录，重建后可继续读取工作文件。工作区快照、恢复、显式删除及共享桌面尚未实现。
 
@@ -18,7 +18,7 @@
 
 ## 参考与差异
 
-吸收 OpenMausBot `server/container-computer.ts` 的不可变镜像、每 Bot 环境、持久工作目录、能力检查和实际健康/画面验证；默认网络进一步收紧为无网络环境。后续吸收 rakazo 的屏幕租约和接管机制，在任务协调层加入持久 fencing 代次，先确认旧执行停止，再允许新控制者使用资源。
+吸收 OpenMausBot `server/container-computer.ts` 的不可变镜像、每 Bot 环境、持久工作目录、能力检查和实际健康/画面验证；默认网络进一步收紧为无网络环境。吸收 rakazo 的屏幕租约和接管机制，在任务协调层加入持久 fencing 代次，先确认旧执行停止，再允许新控制者使用资源。
 
 ## 验收状态
 

src/client/components/workspace/ComputerPanel.vue

@@ -0,0 +1,94 @@
+<script setup lang="ts">
+import {computed,onBeforeUnmount,onMounted,ref,watch} from 'vue'
+import {apiRequest} from '@/api/client'
+import {createUuid} from '@/utils/id'
+import type {WorkspaceAgent} from '@shared/workspace'
+import type {ComputerControlStatus,ComputerFrame,ComputerInput} from '@shared/computerControl'
+import type {JsonValue} from '@shared/types'
+const props=defineProps<{agents:WorkspaceAgent[]}>(),emit=defineEmits<{close:[]}>()
+const selected=ref(props.agents[0]?.id??''),dialog=ref<HTMLDialogElement>()
+const state=ref<ComputerControlStatus>({mode:'off'}),frame=ref<ComputerFrame>(),ticket=ref<{controlId:string;token:string}>()
+const error=ref(''),busy=ref(false),text=ref(''),notes=ref('')
+let takeRequestId:string|undefined
+let timer:ReturnType<typeof setTimeout>|undefined,closed=false,polling=false,lastRenew=0,generation=0
+const labels:Record<ComputerControlStatus['mode'],string>={off:'电脑未运行',agent:'Agent 正在操作',pausing:'正在等待当前操作结束',human:'人工控制中',resuming:'正在交还 Agent',error:'电脑需要检查'}
+const mine=computed(()=>!!ticket.value&&ticket.value.controlId===state.value.controlId)
+const controllable=computed(()=>mine.value&&state.value.mode==='human'&&frame.value?.generation===state.value.generation&&!busy.value)
+const endpoint=()=>`/api/app/agents/${encodeURIComponent(selected.value)}/computer`
+async function post(path:string,body:unknown){return apiRequest<any>(endpoint()+path,{method:'POST',body:body as JsonValue})}
+async function refresh(){
+  if(polling||busy.value||closed||!selected.value||document.hidden)return
+  polling=true;const current=++generation,base=endpoint()
+  try{
+    const status=await apiRequest<ComputerControlStatus>(base)
+    if(current!==generation||closed)return
+    state.value=status
+    if(ticket.value&&status.controlId!==ticket.value.controlId&&status.mode!=='pausing')ticket.value=undefined
+    if(mine.value&&Date.now()-lastRenew>8000){await post('/renew',ticket.value);lastRenew=Date.now()}
+    if(status.mode!=='off'){
+      const image=await apiRequest<ComputerFrame>(base+'/frame')
+      if(current===generation&&!closed)frame.value=image
+    }
+  }catch(cause){if(current===generation&&!closed){error.value=cause instanceof Error?cause.message:'无法读取电脑状态';if([401,403,410].includes(Number((cause as {status?:number}).status))){ticket.value=undefined;frame.value=undefined;state.value={mode:'off'}}}}
+  finally{polling=false}
+}
+async function run(action:()=>Promise<void>){
+  if(busy.value)return;generation++;busy.value=true;error.value=''
+  try{await action()}catch(cause){error.value=cause instanceof Error?cause.message:'电脑操作失败'}finally{busy.value=false;await refresh()}
+}
+function take(){void run(async()=>{takeRequestId??=createUuid();const value=await post('/take',{requestId:takeRequestId});ticket.value={controlId:value.controlId,token:value.token};state.value=value;lastRenew=0})}
+function send(action:ComputerInput){
+  if(!controllable.value||!frame.value||!ticket.value)return
+  const image=frame.value,credentials=ticket.value
+  void run(async()=>{await post('/input',{...credentials,requestId:createUuid(),generation:image.generation,frameId:image.id,action})})
+}
+function click(event:MouseEvent){
+  if(!controllable.value||!frame.value)return
+  const rect=(event.currentTarget as HTMLImageElement).getBoundingClientRect()
+  send({kind:'click',x:Math.min(frame.value.width-1,Math.max(0,Math.floor((event.clientX-rect.left)*frame.value.width/rect.width))),y:Math.min(frame.value.height-1,Math.max(0,Math.floor((event.clientY-rect.top)*frame.value.height/rect.height))),button:event.button===2?'right':'left',count:event.detail>1?2:1})
+}
+async function giveBack(){if(ticket.value){await post('/giveback',{...ticket.value,notes:notes.value});ticket.value=undefined;takeRequestId=undefined;notes.value=''}}
+async function close(){if(busy.value)return;await run(giveBack);if(!ticket.value){closed=true;dialog.value?.close();emit('close')}}
+async function cycle(){await refresh();if(!closed)timer=setTimeout(()=>void cycle(),1200)}
+watch(selected,()=>{generation++;takeRequestId=undefined;frame.value=undefined;ticket.value=undefined;state.value={mode:'off'};error.value='';void refresh()})
+onMounted(()=>{dialog.value?.showModal();void cycle()})
+onBeforeUnmount(()=>{closed=true;generation++;clearTimeout(timer)})
+</script>
+<template>
+  <Teleport to="body">
+    <dialog ref="dialog" class="computer-panel" aria-label="隔离电脑" @cancel.prevent="close">
+      <header>
+        <div><strong>隔离电脑</strong><small>{{ labels[state.mode] }}</small></div>
+        <select v-if="agents.length>1" v-model="selected" aria-label="选择电脑" :disabled="!!ticket"><option v-for="agent in agents" :key="agent.id" :value="agent.id">{{ agent.name }}</option></select>
+        <span v-else>{{ agents[0]?.name }}</span>
+        <button type="button" :disabled="busy" @click="close">{{ ticket ? '交还并关闭' : '关闭' }}</button>
+      </header>
+      <div class="computer-screen" :class="{controlling:controllable}">
+        <img v-if="frame" :src="`data:image/png;base64,${frame.data}`" alt="隔离电脑当前画面" draggable="false" @click="click" @contextmenu.prevent="click">
+        <p v-else>{{ state.mode==='off' ? '打开电脑后可查看桌面。' : '正在读取电脑画面…' }}</p>
+        <span v-if="frame&&!controllable" class="view-label">{{ state.mode==='off'?'上次画面':'仅查看' }}</span>
+      </div>
+      <p v-if="error" class="computer-error" role="alert">{{ error }}</p>
+      <footer>
+        <p v-if="!ticket">查看不会输入任何内容。接管后 Agent 会暂停，已有操作结束后才允许人工操作。</p>
+        <button v-if="!ticket&&state.mode!=='human'" type="button" :disabled="busy" @click="take">{{ state.mode==='off'?'打开并控制':'接管电脑' }}</button>
+        <template v-if="mine">
+          <div class="key-row"><button v-for="key in ['Return','Tab','Escape','BackSpace','Up','Down','Left','Right']" :key="key" type="button" :disabled="!controllable" @click="send({kind:'key',key})">{{ ({Return:'回车',Escape:'Esc',BackSpace:'退格',Up:'↑',Down:'↓',Left:'←',Right:'→'} as Record<string,string>)[key]||key }}</button></div>
+          <label>输入到电脑<textarea v-model="text" rows="2" maxlength="16000" placeholder="输入文字后点击发送" :disabled="!controllable" /></label>
+          <button type="button" :disabled="!controllable||!text" @click="send({kind:'text',text});text=''">输入文字</button>
+          <label>交还说明<textarea v-model="notes" rows="2" maxlength="4000" placeholder="告诉 Agent 你完成了什么，以及接下来需要做什么" /></label>
+          <button type="button" :disabled="busy||state.mode!=='human'" @click="run(giveBack)">{{ state.canResume ? '交还 Agent 并继续' : '结束控制' }}</button>
+          <small>控制只属于当前页面。页面断开后控制权会到期，任务不会自行恢复。</small>
+        </template>
+      </footer>
+    </dialog>
+  </Teleport>
+</template>
+<style scoped>
+.computer-panel{width:min(1120px,calc(100vw - 32px));max-height:calc(100dvh - 32px);padding:0;border:1px solid var(--line);border-radius:16px;background:var(--surface);color:var(--text-primary);overflow:auto}
+.computer-panel::backdrop{background:rgba(0,0,0,.45)}header{display:flex;align-items:center;gap:14px;padding:14px 18px;border-bottom:1px solid var(--line)}header div{display:grid;gap:4px;margin-right:auto}small{font-size:12px;color:var(--text-muted);line-height:1.5}
+button,select,textarea{font:inherit;border:1px solid var(--line);border-radius:8px;background:var(--surface-raised);color:var(--text-primary)}button,select{min-height:44px;padding:8px 12px}button{cursor:pointer}button:disabled{opacity:.5;cursor:default}button:focus-visible,select:focus-visible,textarea:focus-visible{outline:2px solid var(--accent);outline-offset:2px}
+.computer-screen{position:relative;display:flex;align-items:center;justify-content:center;min-height:220px;background:#14181e}.computer-screen img{max-width:100%;max-height:58vh;width:auto;height:auto;user-select:none}.controlling img{cursor:crosshair}.computer-screen p{color:#d8dee7}.view-label{position:absolute;top:10px;left:12px;padding:4px 8px;border-radius:6px;background:rgba(0,0,0,.65);color:white;font-size:12px;pointer-events:none}
+footer{padding:16px 18px;display:grid;gap:12px}footer p{margin:0;font-size:13px;color:var(--text-secondary);line-height:1.6}label{display:grid;gap:6px;font-size:13px}textarea{padding:10px;min-width:0;resize:vertical}.key-row{display:flex;gap:6px;flex-wrap:wrap}.computer-error{color:var(--danger);padding:0 18px;font-size:13px}
+@media(max-width:600px){.computer-panel{width:100vw;max-height:100dvh;border-radius:0}header{flex-wrap:wrap;padding:12px}footer{padding:12px}.computer-screen img{max-height:48vh}}
+</style>

src/client/views/ConversationsView.vue

@@ -6,6 +6,7 @@ import { useRoute, useRouter } from 'vue-router'
 import WorkspaceShell from '@/components/app/WorkspaceShell.vue'
 import RemoteAgentPicker from '@/components/workspace/RemoteAgentPicker.vue'
 import ConversationList from '@/components/workspace/ConversationList.vue'
+import ComputerPanel from '@/components/workspace/ComputerPanel.vue'
 import TaskPlan from '@/components/workspace/TaskPlan.vue'
 import type { AgentAssignment } from '@shared/agentTasks'
 import type { WorkspaceTask } from '@shared/workspace'
@@ -184,6 +185,8 @@ let cursor = 0,
   pendingFingerprint = ''
 const selected = computed(() => (typeof route.params.id === 'string' ? route.params.id : undefined))
 const selectedTask = computed(() => typeof route.query.taskId === 'string' ? route.query.taskId : '')
+const computerOpen=ref(false)
+const computerAgents=computed(()=>agents.value.filter(agent=>agent.execution==='computer'&&!agent.archived&&(active.value?.memberIds.includes(agent.id)||(!!agent.temporaryGoalId&&agent.temporaryGoalId===selectedTask.value))))
 const members = computed(() => agents.value.filter((a) => active.value?.memberIds.includes(a.id)))
 const isAgentDialog = computed(() => dialog.value === 'agent' || dialog.value === 'editAgent')
 const body = (v: unknown) => v as JsonValue
@@ -666,6 +669,7 @@ onBeforeUnmount(() => {
         <template #header-leading><button v-if="active" class="workspace-list-back icon-button" aria-label="返回 Bot 列表" @click="router.push('/conversations')"><AppIcon name="chevron-left" /></button></template>
         <template #header-actions>
           <div v-if="active" class="header-actions">
+            <button v-if="computerAgents.length" type="button" @click="computerOpen=true">电脑</button>
             <select v-if="active.kind === 'group' && tasks.length" class="task-picker" aria-label="当前任务" :value="activeTask?.id" @change="router.push({query:{...route.query,taskId:($event.target as HTMLSelectElement).value}})">
               <option v-for="task in tasks" :key="task.id" :value="task.id">{{ task.title }}</option>
             </select>
@@ -684,6 +688,7 @@ onBeforeUnmount(() => {
         :mention-options="active.kind === 'group' ? members.map(a => ({id:a.id,label:a.name,insertText:`@${a.name} `})) : []"
         @send="sendFromComposer" @stop="control('stop')" @tool-trace-toggle="showThinking = !showThinking" @clear-reference="quoted = null" @error="error = $event" />
     </section>
+    <ComputerPanel v-if="computerOpen" :agents="computerAgents" @close="computerOpen=false" />
     <FloatingResourceSearch section="groups" label="搜索聊天" :items="searchItems" tabbed @select="select" />
     <PreviewModal v-if="preview" :item="preview" :items="media" @close="preview = null" />
     <ImagePreviewLightbox v-model="mediaIndex" :images="lightboxMedia" />

src/runner/agent.ts

@@ -3,6 +3,7 @@ import { DatabaseSync } from 'node:sqlite'
 import { realpath,access } from 'node:fs/promises'
 import { resolve, sep, join } from 'node:path'
 import {ComputerError} from './computers/container.js'
+import {z} from 'zod'
 import {ComputerRuntime,ComputerGateway,type ComputerTarget} from './worker/gateway.js'
 import { HttpError } from '../server/errors.js'
 import { UpstreamClient } from '../server/upstream.js'
@@ -50,7 +51,7 @@ export class RunnerAgent {
     if(url.hostname==='localhost')url.hostname='127.0.0.1'
     const response=await (isLocalAuthorizationTarget(url)&&this.fetchImpl===fetch?this.controlTransport.fetch.bind(this.controlTransport):this.fetchImpl)(url,{
       method:body===undefined?'GET':'POST',redirect:'error',signal:AbortSignal.any([this.controlAbort.signal,AbortSignal.timeout(25000)]),
-      headers:{Authorization:`Bearer ${this.config.token}`,'x-runner-instance':this.instance,'x-runner-protocol':'1','x-runner-features':this.computers?'computer-worker-v1,artifact-chunks-v1,helper-retirement-v1':'',...(this.serverEpoch?{'x-runner-epoch':this.serverEpoch}:{}),'Content-Type':'application/json',...(path==='poll'&&!this.connected?{'x-runner-reset':'1'}:{})},
+      headers:{Authorization:`Bearer ${this.config.token}`,'x-runner-instance':this.instance,'x-runner-protocol':'1','x-runner-features':this.computers?'computer-worker-v1,artifact-chunks-v1,helper-retirement-v1,computer-control-v1':'',...(this.serverEpoch?{'x-runner-epoch':this.serverEpoch}:{}),'Content-Type':'application/json',...(path==='poll'&&!this.connected?{'x-runner-reset':'1'}:{})},
       ...(body===undefined?{}:{body:JSON.stringify(body)}),
     })
     if(!response.ok)throw new HttpError(response.status,`执行节点连接被拒绝（HTTP ${response.status}）`,response.status===403?'runner_unauthorized':'runner_request_failed')
@@ -86,6 +87,19 @@ export class RunnerAgent {
   private async execute(command:RunnerCommand):Promise<any> {
     const p=command.payload
     if(!this.active||!this.connected)throw new Error('执行节点已断开')
+    if(command.kind==='computer.control'){
+      if(!this.computers)throw new HttpError(409,'电脑服务未配置','computer_unavailable')
+      const input=z.object({target:z.object({environmentId:z.string().uuid(),agentId:z.string().uuid(),ownerKey:z.string().regex(/^[a-f0-9]{64}$/)}).strict(),profile:z.string(),op:z.enum(['status','frame','take','input','giveback']),controlId:z.string().uuid().optional(),requestId:z.string().uuid().optional(),generation:z.number().int().optional(),frameId:z.string().uuid().optional(),action:z.unknown().optional(),notes:z.string().max(4000).optional()}).strict().parse(p)
+      this.requireProfile(input.profile)
+      const check=async()=>{if(!this.connected||!input.controlId||(await this.api('control-check',{controlId:input.controlId})).allowed!==true)throw new HttpError(403,'电脑控制授权已失效','computer_control_expired')}
+      const authorize=()=>{if(!this.connected||!this.active||Date.now()>command.expiresAt)throw new HttpError(403,'电脑请求已失效','computer_control_expired')}
+      await this.computers.ready
+      if(input.op==='status')return this.computers.controls.status(input.target)
+      if(input.op==='frame')return this.computers.controls.frame(input.target,authorize)
+      if(input.op==='take')return this.computers.controls.take(input.target,input.profile,input.controlId!,check)
+      if(input.op==='input')return this.computers.controls.input(input.target,input.controlId!,input.requestId!,input.generation!,input.frameId!,input.action)
+      return this.computers.controls.giveBack(input.target,input.controlId!,input.notes??'')
+    }
     if(command.kind==='computer.retire'){
       const meta=p as unknown as ComputerTarget
       if(!this.computers||![meta.environmentId,meta.agentId].every(value=>/^[0-9a-f-]{36}$/.test(value))||!/^[a-f0-9]{64}$/.test(meta.ownerKey))throw new HttpError(400,'电脑清理请求无效','computer_cleanup_invalid')
@@ -250,7 +264,7 @@ export class RunnerAgent {
       try{
         await this.disconnect().catch(()=>{})
         await Promise.allSettled(deliveries.values());await Promise.allSettled(this.inflight.values())
-        await this.computers?.pool.close()
+        await this.computers?.controls.close();await this.computers?.pool.close()
       }finally{this.target.client.close();this.controlTransport.close();this.db.close();this.status('执行节点已停止')}
     }
   }

src/runner/computers/container.ts

@@ -205,6 +205,18 @@ export class ContainerComputerProvider implements ComputerProvider {
       }
     })
   }
+  async capture(value:ComputerSpecification,authorize:()=>void):Promise<{data:string;width:number;height:number;capturedAt:number}> {
+    const spec=this.validateSpecification(value),state=await this.inspect(spec)
+    authorize();if(!state?.running)throw new ComputerError('computer_not_running','电脑尚未运行')
+    const file=`/tmp/yaoyao-view-${randomBytes(12).toString('hex')}.png`
+    const exec=(args:string[])=>this.run(this.runtime,['exec','--user','1000:1000','--',state.containerId,...args],{timeout:10000})
+    try{
+      await exec([CUA_DRIVER,'call','get_desktop_state','{}','--socket',CUA_SOCKET,'--screenshot-out-file',file]);authorize()
+      const data=(await exec(['base64','-w0',file])).stdout.trim(),bytes=Buffer.from(data,'base64')
+      if(bytes.length<24||bytes.subarray(1,4).toString()!=='PNG')throw new ComputerError('computer_frame_invalid','电脑画面无效')
+      authorize();return {data,width:bytes.readUInt32BE(16),height:bytes.readUInt32BE(20),capturedAt:Date.now()}
+    }finally{await exec(['rm','-f','--',file]).catch(()=>{})}
+  }
   async deleteWorkspace(value:ComputerSpecification):Promise<void>{
     const spec=this.validateSpecification(value)
     if(await this.inspectRaw(this.name(spec)))throw new ComputerError('computer_busy','删除工作区前必须移除电脑实例')

src/runner/computers/pool.ts

@@ -83,6 +83,21 @@ export class ComputerPool {
       catch(error){entry.status='uncertain';this.save(entry);throw error}
     })
   }
+  async transfer(lease:ComputerLease,holderId:string,authorize:()=>void):Promise<ComputerLease>{
+    return this.serial(lease.environmentId,async()=>{
+      this.assert(lease);authorize()
+      const active=this.active.get(lease.id)!
+      if(active.operations.size)throw new ComputerError('computer_busy','电脑仍有在途操作，不能移交控制')
+      const entry=this.require(lease),state=await this.provider.inspect(entry.spec)
+      this.assert(lease);authorize()
+      if(active.operations.size||!state?.running)throw new ComputerError('computer_busy','电脑尚未就绪，不能移交控制')
+      const next:ComputerLease={...lease,id:randomUUID(),holderId,generation:entry.generation+1}
+      active.detach();active.controller.abort();this.active.delete(lease.id)
+      this.active.set(next.id,{lease:next,authorize,controller:new AbortController(),operations:new Set(),detach:()=>{}})
+      entry.lease=next;entry.generation=next.generation;entry.expiresAt=this.now()+this.limits.ttlMs;this.save(entry)
+      return next
+    })
+  }
   authorize(lease:ComputerLease){this.assert(lease)}
   renew(lease:ComputerLease):void{this.assert(lease);const entry=this.require(lease);entry.expiresAt=this.now()+this.limits.ttlMs;this.save(entry)}
   use<T>(lease:ComputerLease,action:(context:{signal:AbortSignal;authorize():void})=>Promise<T>):Promise<T>{

src/runner/worker/control.ts

@@ -0,0 +1,122 @@
+import {randomUUID,createHash} from 'node:crypto'
+import {z} from 'zod'
+import {HttpError} from '../../server/errors.js'
+import {CUA_DRIVER,CUA_SOCKET,type ComputerSpecification} from '../computers/container.js'
+import type {ComputerLease} from '../computers/pool.js'
+import {ComputerPublicProxy} from '../network/computerProxy.js'
+import type {ComputerRuntime,ComputerGateway,ComputerTarget} from './gateway.js'
+import type {ComputerControlStatus,ComputerFrame} from '../../shared/computerControl.js'
+
+const actionSchema=z.discriminatedUnion('kind',[
+  z.object({kind:z.literal('click'),x:z.number().int().nonnegative(),y:z.number().int().nonnegative(),button:z.enum(['left','right','middle']).default('left'),count:z.number().int().min(1).max(2).default(1)}).strict(),
+  z.object({kind:z.literal('text'),text:z.string().max(16000)}).strict(),
+  z.object({kind:z.literal('key'),key:z.string().min(1).max(40).regex(/^[A-Za-z0-9_+ -]+$/),modifiers:z.array(z.enum(['ctrl','alt','shift','super'])).max(4).default([])}).strict(),
+  z.object({kind:z.literal('scroll'),direction:z.enum(['up','down','left','right']),amount:z.number().int().min(1).max(50).default(3)}).strict(),
+])
+interface Manual {
+  id:string;meta:ComputerTarget;profile:string;mode:ComputerControlStatus['mode'];valid:boolean
+  gateway?:ComputerGateway;lease?:ComputerLease;spec?:ComputerSpecification;proxy?:ComputerPublicProxy
+  pending?:Promise<void>;error?:string;check():Promise<void>;checking:boolean
+  stopping?:Promise<void>;actions:Map<string,{fingerprint:string;result:Promise<unknown>}>
+}
+export class ComputerControls {
+  private manual=new Map<string,Manual>()
+  private frames=new Map<string,ComputerFrame[]>()
+  private captures=new Map<string,Promise<ComputerFrame>>()
+  private timer:ReturnType<typeof setInterval>
+  constructor(readonly runtime:ComputerRuntime){
+    this.timer=setInterval(()=>{for(const state of this.manual.values()){
+      if(state.checking)continue;state.checking=true
+      void state.check().then(()=>{if(state.lease&&state.mode==='human')this.runtime.pool.renew(state.lease)}).catch(()=>this.stop(state)).finally(()=>{state.checking=false}).catch(()=>{})
+    }},5000);this.timer.unref()
+  }
+  private guard(state:Manual){if(!state.valid||this.manual.get(state.meta.environmentId)!==state)throw new HttpError(410,'电脑控制权已结束','computer_control_expired')}
+  private state(meta:ComputerTarget){const state=this.manual.get(meta.environmentId);if(state&&state.meta.ownerKey!==meta.ownerKey)throw new HttpError(403,'电脑归属不匹配','computer_owner_mismatch');return state}
+  async status(meta:ComputerTarget):Promise<ComputerControlStatus>{
+    const state=this.state(meta)
+    if(state)return {mode:state.mode,controlId:state.id,generation:state.lease?.generation,canResume:!!state.gateway,error:state.error}
+    const resource=this.runtime.pool.status(meta.ownerKey).find(item=>item.environmentId===meta.environmentId)
+    return {mode:resource?.status==='active'?'agent':'off',generation:resource?.generation}
+  }
+  async frame(meta:ComputerTarget,authorize:()=>void):Promise<ComputerFrame>{
+    this.runtime.assertTarget(meta);authorize()
+    const spec=this.runtime.pool.definition(meta.ownerKey,meta.environmentId)
+    if(!spec)throw new HttpError(409,'电脑尚未启动','computer_not_running')
+    const status=await this.status(meta),cached=this.frames.get(meta.environmentId)?.at(-1)
+    if(cached&&cached.generation===status.generation&&Date.now()-cached.capturedAt<700)return cached
+    const existing=this.captures.get(meta.environmentId);if(existing)return existing.then(frame=>{authorize();return frame})
+    const pending=this.runtime.provider.capture(spec,authorize).then(frame=>{
+      authorize();const result:ComputerFrame={...frame,id:randomUUID(),generation:status.generation??0}
+      const history=(this.frames.get(meta.environmentId)??[]).filter(frame=>Date.now()-frame.capturedAt<10000).slice(-4)
+      this.frames.set(meta.environmentId,[...history,result]);return result
+    }).finally(()=>this.captures.delete(meta.environmentId))
+    this.captures.set(meta.environmentId,pending);return pending
+  }
+  async take(meta:ComputerTarget,profile:string,id:string,check:()=>Promise<void>):Promise<ComputerControlStatus>{
+    this.runtime.assertTarget(meta);await check()
+    const old=this.state(meta)
+    if(old){if(old.id===id)return this.status(meta);throw new HttpError(409,'电脑已由另一控制会话接管','computer_control_busy')}
+    const gateway=this.runtime.gateways.get(meta.environmentId)
+    const state:Manual={id,meta,profile,mode:'pausing',valid:true,gateway:gateway?.active?gateway:undefined,check,checking:false,actions:new Map()}
+    this.manual.set(meta.environmentId,state)
+    state.pending=(async()=>{
+      if(state.gateway){state.lease=await state.gateway.takeControl(id,()=>this.guard(state));state.spec=state.gateway.specification}
+      else{
+        const resolved=await this.runtime.resolve(profile);await check();this.guard(state)
+        state.spec={id:meta.environmentId,ownerKey:meta.ownerKey,imageId:this.runtime.config.imageId,cwd:resolved.cwd,network:this.runtime.config.network??'none'}
+        await this.runtime.pool.configure(state.spec,()=>this.guard(state))
+        state.lease=await this.runtime.pool.acquire(state.spec,`human:${id}`,()=>this.guard(state))
+        if(this.runtime.config.network==='public-proxy')state.proxy=await ComputerPublicProxy.start(this.runtime.provider,state.spec,this.runtime.proxyScript,()=>this.runtime.pool.authorize(state.lease!),check,()=>{void this.stop(state)})
+      }
+      await check();this.guard(state);state.mode='human'
+    })().catch(async()=>{if(!state.valid)return;state.mode='error';state.error='接管未完成，请等待当前操作结束后重试';await this.stop(state).catch(()=>{})})
+    return this.status(meta)
+  }
+  async input(meta:ComputerTarget,id:string,requestId:string,generation:number,frameId:string,value:unknown):Promise<unknown>{
+    const state=this.state(meta)
+    if(!state||state.id!==id||state.mode!=='human'||!state.lease||!state.spec)throw new HttpError(403,'需要当前电脑的控制权','computer_control_required')
+    this.guard(state);await state.check()
+    if(state.lease.generation!==generation)throw new HttpError(409,'控制版本已变化','computer_control_stale')
+    const frame=this.frames.get(meta.environmentId)?.find(frame=>frame.id===frameId&&frame.generation===generation&&Date.now()-frame.capturedAt<10000)
+    if(!frame)throw new HttpError(409,'画面已过期，请刷新后操作','computer_frame_stale')
+    const action=actionSchema.parse(value),fingerprint=createHash('sha256').update(JSON.stringify({action,generation,frameId})).digest('hex')
+    const old=state.actions.get(requestId)
+    if(old){if(old.fingerprint!==fingerprint)throw new HttpError(409,'操作编号与内容不一致','idempotency_conflict');return old.result}
+    if(state.actions.size>=1000)throw new HttpError(429,'本次接管操作已达上限，请重新接管','computer_control_limit')
+    const common={scope:'desktop',session:`yaoyao-control-${state.id}`}
+    let tool:string,args:Record<string,unknown>
+    if(action.kind==='click'){
+      if(action.x>=frame.width||action.y>=frame.height)throw new HttpError(400,'点击位置超出画面','computer_point_invalid')
+      tool='click';args={...common,x:action.x,y:action.y,button:action.button,count:action.count}
+    }else if(action.kind==='text'){tool='type_text';args={...common,text:action.text}}
+    else if(action.kind==='key'){tool='press_key';args={...common,key:action.key,modifiers:action.modifiers}}
+    else {tool='scroll';args={...common,direction:action.direction,amount:action.amount}}
+    const result=this.runtime.pool.use(state.lease,async context=>{
+      this.guard(state);await state.check()
+      const response=await this.runtime.provider.execute(state.spec!,[CUA_DRIVER,'call',tool,JSON.stringify(args),'--socket',CUA_SOCKET],context)
+      this.guard(state);let parsed:any;try{parsed=JSON.parse(response.stdout)}catch{}
+      if(parsed?.code||parsed?.error||parsed?.isError)throw new HttpError(502,'电脑未接受这次输入，请刷新画面后重试','computer_action_failed')
+      return {ok:true,result:response.stdout}
+    })
+    state.actions.set(requestId,{fingerprint,result});return result
+  }
+  async giveBack(meta:ComputerTarget,id:string,notes:string){
+    const state=this.state(meta);if(!state||state.id!==id)throw new HttpError(410,'电脑控制权已结束','computer_control_expired')
+    await state.pending;this.guard(state);await state.check()
+    if(state.gateway){state.mode='resuming';await state.gateway.giveBack(notes);state.valid=false;this.manual.delete(meta.environmentId)}
+    else await this.stop(state)
+    return {ok:true}
+  }
+  private async stop(state:Manual){
+    if(state.stopping)return state.stopping
+    state.valid=false
+    state.stopping=(async()=>{
+      await state.proxy?.close()
+      if(state.gateway)await state.gateway.stopControl()
+      else if(state.lease)await this.runtime.pool.release(state.lease).catch(error=>{if(error.code!=='computer_lease_stale')throw error})
+      if(this.manual.get(state.meta.environmentId)===state)this.manual.delete(state.meta.environmentId)
+    })().catch(error=>{state.mode='error';state.error='电脑停止状态待确认';state.stopping=undefined;throw error})
+    return state.stopping
+  }
+  async close(){clearInterval(this.timer);await Promise.allSettled([...this.manual.values()].map(state=>this.stop(state)))}
+}

src/runner/worker/gateway.ts

@@ -10,12 +10,13 @@ import type {GatewayFrame} from '../../server/workspaceGateway.js'
 import type {RunnerConfiguration} from '../../shared/runner.js'
 import {ContainerComputerProvider,CUA_DRIVER,CUA_SOCKET,type ComputerSpecification} from '../computers/container.js'
 import {ComputerPool,type ComputerLease} from '../computers/pool.js'
+import {ComputerControls} from './control.js'
 import {ComputerPublicProxy} from '../network/computerProxy.js'
 import {HermesWorkerProcess,type WorkerModel,type WorkerTool,type WorkerFrame} from './process.js'
 
 export interface ComputerTarget {environmentId:string;ownerKey:string;agentId:string}
 export interface WorkerSession {id:string;profile:string;ownerKey:string;environmentId:string;cwd:string;configuredCwd:string;history:any[];outcome?:'complete'|'failed'|'uncertain'}
-interface Live {proxy?:ComputerPublicProxy;session:WorkerSession;model:WorkerModel;lease:ComputerLease;controller:AbortController;worker?:HermesWorkerProcess;finishing?:Promise<void>;stopping?:Promise<void>;releasing?:Promise<void>;images:any[];running:boolean;received:boolean}
+interface Live {workMarker?:string;paused?:boolean;pauseJob?:Promise<ComputerLease>;segment?:number;toolCalls:Set<Promise<unknown>>;journal:Map<string,{name:string;result:unknown}>;proxy?:ComputerPublicProxy;session:WorkerSession;model:WorkerModel;lease:ComputerLease;controller:AbortController;worker?:HermesWorkerProcess;finishing?:Promise<void>;stopping?:Promise<void>;releasing?:Promise<void>;images:any[];running:boolean;received:boolean}
 const object=(properties:Record<string,unknown>,required:string[])=>({type:'object',properties,required,additionalProperties:false})
 const text={type:'string'}
 const TOOLS:WorkerTool[]=[
@@ -28,10 +29,12 @@ const TOOLS:WorkerTool[]=[
 ]
 const pathField=z.string().min(1).max(4096).refine(value=>!value.includes('\0'))
 export class ComputerRuntime {
+  readonly controls:ComputerControls
   readonly provider:ContainerComputerProvider
   readonly pool:ComputerPool
   readonly ready:Promise<void>
   readonly config:NonNullable<RunnerConfiguration['computers']>
+  readonly gateways=new Map<string,ComputerGateway>()
   readonly workers=new Map<string,{environmentId:string;ownerKey:string;process:HermesWorkerProcess}>()
   readonly script:string
   readonly proxyScript:string
@@ -44,6 +47,7 @@ export class ComputerRuntime {
     this.pool=new ComputerPool(db,this.provider,{concurrent:this.config.maxConcurrent??2,environments:32,ttlMs:30000})
     db.exec('CREATE TABLE IF NOT EXISTS retired_computers(id TEXT PRIMARY KEY,owner_key TEXT NOT NULL,spec TEXT)')
     db.exec('CREATE TABLE IF NOT EXISTS computer_sessions(id TEXT PRIMARY KEY,value TEXT NOT NULL)')
+    this.controls=new ComputerControls(this)
     this.ready=this.pool.recover();void this.ready.catch(()=>{})
   }
   assertTarget(meta:ComputerTarget){if(this.db.prepare('SELECT id FROM retired_computers WHERE id=?').get(meta.environmentId))throw new HttpError(410,'临时电脑已退役','computer_retired')}
@@ -109,14 +113,14 @@ export class ComputerGateway {
       await this.authorized()
       const controller=new AbortController(),lease=await this.runtime.pool.acquire(this.spec(session),this.workId,()=>{this.guard();if(controller.signal.aborted)throw new Error('cancelled')},controller.signal)
       if(this.closed){controller.abort();await this.runtime.pool.release(lease);throw new Error('电脑通道已关闭')}
-      this.live={session,lease,controller,model:resolved.model as WorkerModel,images:[],running:false,received:false}
-      this.runtime.save(session)
+      this.live={session,lease,controller,model:resolved.model as WorkerModel,images:[],running:false,received:false,toolCalls:new Set(),journal:new Map()}
+      this.runtime.save(session);this.runtime.gateways.set(this.meta.environmentId,this)
       if(this.runtime.config.network==='public-proxy'){
         const live=this.live
-        try{live.proxy=await ComputerPublicProxy.start(this.runtime.provider,this.spec(session),this.runtime.proxyScript,()=>this.runtime.pool.authorize(lease),()=>this.authorized(),()=>{void this.interrupt().finally(()=>{if(!this.closed)this.onDisconnect()}).catch(()=>{})})}
+        try{live.proxy=await ComputerPublicProxy.start(this.runtime.provider,this.spec(session),this.runtime.proxyScript,()=>this.runtime.pool.authorize(live.lease),()=>this.authorized(),()=>{void this.interrupt().finally(()=>{if(!this.closed)this.onDisconnect()}).catch(()=>{})})}
         catch(error){await this.interrupt();throw error}
       }
-      this.timer=setInterval(()=>{if(this.checking)return;this.checking=true;void this.authorized().then(()=>this.runtime.pool.renew(lease)).catch(async()=>{await this.interrupt().catch(()=>{});if(!this.closed)this.onDisconnect()}).finally(()=>{this.checking=false})},5000);this.timer.unref()
+      this.timer=setInterval(()=>{if(this.checking)return;this.checking=true;void this.authorized().then(()=>this.runtime.pool.renew(this.live!.lease)).catch(async()=>{await this.interrupt().catch(()=>{});if(!this.closed)this.onDisconnect()}).finally(()=>{this.checking=false})},5000);this.timer.unref()
       return {session_id:session.id,stored_session_id:session.id,running:false,info:{profile_name:profile}}
     }
     const live=this.live
@@ -146,29 +150,39 @@ export class ComputerGateway {
     if(method==='prompt.submit'){
       if(live.running||live.received)throw new HttpError(409,'电脑会话已经提交过本轮','computer_session_busy')
       await this.authorized();this.guard()
-      const prompt=String(params.text??''),history=structuredClone(live.session.history)
-      live.running=true;live.session.outcome='uncertain';live.session.history.push({role:'user',content:prompt});this.runtime.save(live.session)
-      const directory=join(this.runtime.home,'workers',live.session.id,this.workId)
+      if(typeof params.workMarker==='string'&&/^\[yaoyao-run:[0-9a-f-]{36}:[0-9a-f-]{36}\]$/.test(params.workMarker))live.workMarker=params.workMarker
+      await this.startModel(live,String(params.text??''),structuredClone(live.session.history))
+      return {status:'streaming'}
+    }
+    throw new HttpError(409,'当前隔离 Worker 不支持此交互','computer_method_unavailable')
+  }
+  private async startModel(live:Live,prompt:string,history:any[]){
+      live.received=false;live.running=true;live.session.outcome='uncertain';live.session.history.push({role:'user',content:prompt});this.runtime.save(live.session)
+      const directory=join(this.runtime.home,'workers',live.session.id,this.workId,String(live.segment=(live.segment??0)+1))
       await mkdir(directory,{recursive:true,mode:0o700})
       const worker=new HermesWorkerProcess(this.runtime.config.python,this.runtime.script,{mode:'run',home:directory,hermesSource:this.runtime.config.hermesSource,model:live.model,sessionId:live.session.id,taskId:this.workId,cwd:live.session.cwd,network:this.runtime.config.network??'none',tools:[...TOOLS,...(this.team?.catalog??[])],prompt:live.images.length?[{type:'text',text:prompt},...live.images]:prompt,history})
       live.worker=worker
-      this.runtime.workers.set(this.workId,{environmentId:this.meta.environmentId,ownerKey:this.meta.ownerKey,process:worker});void worker.exited.then(()=>this.runtime.workers.delete(this.workId))
-      worker.onTool=(name,args,id)=>this.tool(live,name,args,id)
+      this.runtime.workers.set(this.workId,{environmentId:this.meta.environmentId,ownerKey:this.meta.ownerKey,process:worker});void worker.exited.then(()=>{if(this.runtime.workers.get(this.workId)?.process===worker)this.runtime.workers.delete(this.workId)})
+      worker.onTool=(name,args,id,callId)=>{
+        const task=this.tool(live,name,args,id).then(result=>{if(callId)live.journal.set(callId,{name,result});return result},error=>{if(callId)live.journal.set(callId,{name,result:{error:'操作未完成或已中断'}});throw error})
+        live.toolCalls.add(task);void task.finally(()=>live.toolCalls.delete(task)).catch(()=>{});return task
+      }
       worker.onEvent=frame=>{
+        if(frame.type==='checkpoint'&&Array.isArray(frame.messages)){live.session.history=frame.messages;this.runtime.save(live.session);return}
+        if(live.paused)return
         if(frame.type==='delta')this.event('message.delta',{text:frame.text})
         if(frame.type==='reasoning')this.event('reasoning.delta',{text:frame.text})
         if(frame.type==='complete'){live.received=true;live.finishing=this.finish(live,frame);void live.finishing.catch(()=>this.lost(live))}
         if(frame.type==='failed')void this.lost(live)
       }
-      void worker.exited.then(()=>{if(!live.received&&live.running&&!this.closed)void this.lost(live)})
-      return {status:'streaming'}
-    }
-    throw new HttpError(409,'当前隔离 Worker 不支持此交互','computer_method_unavailable')
+      void worker.exited.then(()=>{if(!live.paused&&!live.received&&live.running&&!this.closed)void this.lost(live)})
   }
   installTeamLease(id:string,catalog:WorkerTool[],call:(name:string,args:Record<string,unknown>,callId:string)=>Promise<unknown>){this.team={id,catalog,call}}
   removeTeamLease(id:string){if(this.team?.id===id)this.team=undefined}
   private async tool(live:Live,name:string,args:unknown,id:string):Promise<unknown>{
+    if(live.paused)throw new HttpError(409,'电脑正在等待人工操作','computer_paused')
     await this.authorized();this.guard()
+    if(live.paused)throw new HttpError(409,'电脑正在等待人工操作','computer_paused')
     this.event('tool.started',{tool_id:id,name})
     try{
       const team=this.team?.catalog.find(tool=>tool.name===name)
@@ -218,7 +232,7 @@ export class ComputerGateway {
     }finally{this.event('tool.completed',{tool_id:id,name})}
   }
   private async finish(live:Live,frame:WorkerFrame){
-    if(this.closed||live.controller.signal.aborted)return
+    if(this.closed||live.paused||live.controller.signal.aborted)return
     await this.authorized()
     this.runtime.pool.renew(live.lease)
     const success=frame.completed!==false&&!frame.interrupted
@@ -231,9 +245,44 @@ export class ComputerGateway {
   }
   private release(live:Live){clearInterval(this.timer);live.releasing??=(async()=>{await live.proxy?.close();await this.runtime.pool.release(live.lease)})();return live.releasing}
   private async lost(live:Live){
-    if(!live.running||this.closed||live.controller.signal.aborted)return
+    if(!live.running||live.paused||this.closed||live.controller.signal.aborted)return
     live.running=false;await this.interrupt().catch(()=>{});if(!this.closed)this.onDisconnect()
   }
+  get active(){return !!this.live?.running&&!this.live.received&&!this.closed}
+  get paused(){return this.live?.paused===true}
+  get lease(){return this.live?.lease}
+  get specification(){return this.live?this.spec(this.live.session):undefined}
+  async takeControl(controlId:string,authorize:()=>void):Promise<ComputerLease>{
+    const live=this.live
+    if(!live||!this.active)throw new HttpError(409,'Agent 已不在执行，请重新打开电脑','computer_not_active')
+    if(live.pauseJob)return live.pauseJob
+    live.paused=true;this.event('computer.paused',{stage:'pausing'})
+    live.pauseJob=(async()=>{
+      await live.worker?.close()
+      await Promise.allSettled([...live.toolCalls])
+      authorize();this.guard()
+      const history=structuredClone(live.session.history),done=new Set(history.filter(message=>message.role==='tool').map(message=>message.tool_call_id))
+      for(const message of [...history])for(const tool of message.tool_calls??[])if(!done.has(tool.id)){
+        const entry=live.journal.get(tool.id),value=entry?.result??{error:'用户接管时此操作未完成，继续前请检查电脑状态，不要盲目重复'}
+        const content=typeof value==='string'?value:JSON.stringify(value)
+        history.push({role:'tool',name:tool.function?.name??entry?.name,tool_call_id:tool.id,content});done.add(tool.id)
+      }
+      live.session.history=history;this.runtime.save(live.session)
+      live.lease=await this.runtime.pool.transfer(live.lease,`human:${controlId}`,authorize)
+      this.event('computer.paused',{stage:'human',generation:live.lease.generation})
+      return live.lease
+    })()
+    return live.pauseJob
+  }
+  async giveBack(notes:string):Promise<void>{
+    const live=this.live
+    if(!live?.paused)throw new HttpError(409,'Agent 没有处于接管状态','computer_not_paused')
+    await live.pauseJob;await this.authorized()
+    live.lease=await this.runtime.pool.transfer(live.lease,this.workId,()=>this.guard())
+    live.paused=false;live.pauseJob=undefined;live.journal.clear()
+    this.event('computer.resumed',{generation:live.lease.generation,notes})
+    await this.startModel(live,`用户已完成电脑接管并明确交还控制。请先检查当前电脑状态，再继续原任务；不要重复已完成的副作用。用户说明：${notes||'无额外说明'}\n${live.workMarker??''}`,structuredClone(live.session.history))
+  }
   async interrupt(){
     const live=this.live;if(!live)return
     if(live.stopping)return live.stopping
@@ -245,5 +294,6 @@ export class ComputerGateway {
     })()
     return live.stopping
   }
-  close(){this.closed=true;this.controller.abort();this.team=undefined;clearInterval(this.timer);return this.interrupt()}
+  async stopControl(){await this.interrupt();this.event('message.complete',{text:'电脑控制已结束，本轮已停止',status:'interrupted'})}
+  close(){if(this.runtime.gateways.get(this.meta.environmentId)===this)this.runtime.gateways.delete(this.meta.environmentId);this.closed=true;this.controller.abort();this.team=undefined;clearInterval(this.timer);return this.interrupt()}
 }

src/runner/worker/hermes_worker.py

@@ -84,6 +84,7 @@ def run_model():
             # Intercept before Hermes' sequential, parallel and inline branches.
             # In particular message_agent/delegate_task must never reach native code.
             from agent.tool_dispatch_helpers import make_tool_result_message
+            emit("checkpoint", messages=messages)
             for tool in assistant_message.tool_calls:
                 name = tool.function.name
                 try:
@@ -96,6 +97,7 @@ def run_model():
                 if isinstance(result, dict) and result.get("_multimodal"):
                     result = [{"type": "text", "text": result.get("text_summary", "")}, *result.get("content", [])]
                 messages.append(make_tool_result_message(name, result, tool.id))
+                emit("checkpoint", messages=messages)
 
         def _invoke_tool(self, function_name, function_args, effective_task_id, tool_call_id=None, **kwargs):
             if stopped.is_set() or function_name not in allowed:
@@ -104,7 +106,7 @@ def run_model():
             event = threading.Event()
             slot = {"event": event}
             pending[call_id] = slot
-            emit("tool", id=call_id, name=function_name, arguments=function_args)
+            emit("tool", id=call_id, callId=tool_call_id, name=function_name, arguments=function_args)
             try:
                 if not event.wait(90) or stopped.is_set():
                     stopped.set()

src/runner/worker/process.ts

@@ -15,7 +15,7 @@ export class HermesWorkerProcess {
   private pending=new Set<Promise<unknown>>()
   private waiters=new Map<string,{resolve(frame:WorkerFrame):void;reject(error:Error):void;timer:ReturnType<typeof setTimeout>}>()
   onEvent:(frame:WorkerFrame)=>void=()=>{}
-  onTool:(name:string,args:unknown,id:string)=>Promise<unknown>=async()=>{throw new Error('工具未授权')}
+  onTool:(name:string,args:unknown,id:string,callId?:string)=>Promise<unknown>=async()=>{throw new Error('工具未授权')}
   constructor(python:string,script:string,input:Record<string,unknown>,onDiagnostic?:(text:string)=>void){
     this.child=spawn(python,['-u',script],{stdio:'pipe',env:{PATH:process.env.PATH??'/usr/bin:/bin',HOME:homedir(),PYTHONNOUSERSITE:'1',PYTHONDONTWRITEBYTECODE:'1'}})
     if(onDiagnostic)this.child.stderr.on('data',chunk=>onDiagnostic(String(chunk)));else this.child.stderr.resume()
@@ -36,7 +36,7 @@ export class HermesWorkerProcess {
     if(waiting){clearTimeout(waiting.timer);this.waiters.delete(frame.type);waiting.resolve(frame)}
     if(frame.type==='failed')this.fail(new HttpError(502,'Hermes Worker 执行失败，请核对基础 Profile 与模型连接','computer_worker_failed'))
     if(frame.type==='tool'){
-      const work=Promise.resolve().then(()=>this.onTool(String(frame.name),frame.arguments,String(frame.id))).then(result=>this.send({type:'result',id:frame.id,result}),()=>this.send({type:'result',id:frame.id,result:{error:'工具执行失败或授权已失效'}}))
+      const work=Promise.resolve().then(()=>this.onTool(String(frame.name),frame.arguments,String(frame.id),typeof frame.callId==='string'?frame.callId:undefined)).then(result=>this.send({type:'result',id:frame.id,result}),()=>this.send({type:'result',id:frame.id,result:{error:'工具执行失败或授权已失效'}}))
       this.pending.add(work);void work.finally(()=>this.pending.delete(work)).catch(()=>{})
     }else this.onEvent(frame)
   }

src/server/app.ts

@@ -49,6 +49,7 @@ import {
   HermesChatNotificationResolver,
 } from './pushEvents.js'
 import { ChatCacheCoordinator, ChatCacheStore } from './chatCache.js'
+import {ComputerControlService} from './computerControls.js'
 import { RunnerHub } from './runnerHub.js'
 
 export interface ApplicationOptions {
@@ -208,6 +209,8 @@ export function createApplication(options: ApplicationOptions = {}): Application
   const workspaceNodes = new WorkspaceNodes(workspace, config, { url: config.upstream, client: upstream, session: upstreamSession }, pairings.nodeID)
   workspaceNodes.sourceAllowed = (owner, nodeId, profile) => auth.canUseSource(owner, nodeId, profile)
   const runners=new RunnerHub(workspace,auth,workspaceNodes.local)
+  const computerControls=new ComputerControlService(workspace,auth,workspaceNodes,runners)
+  runners.controlAllowed=(id,runnerId)=>computerControls.allowed(id,runnerId)
   workspaceNodes.runnerTarget=(owner,nodeId,computer)=>runners.target(owner,nodeId,computer)
   const workspaceRuntime = new WorkspaceRuntime(workspace, workspaceNodes, uploads, owner => auth.isUserActive(owner), owner => auth.pushAuthorizationVersion(owner) ?? 0)
   workspaceRuntime.retireHelper=(owner,helper)=>runners.retireHelper(owner,helper)
@@ -372,6 +375,7 @@ export function createApplication(options: ApplicationOptions = {}): Application
     } else await next()
   })
 
+  const computerRouter=computerControls.router();app.use(computerRouter.routes());app.use(computerRouter.allowedMethods())
   const applicationRouter = workspaceRouter(workspace, workspaceRuntime, workspaceNodes, workspaceAssets, uploads, auth, push)
   app.use(runners.adminRouter().routes())
   app.use(applicationRouter.routes())

src/server/computerControls.ts

@@ -0,0 +1,69 @@
+import {randomBytes,randomUUID,createHash,timingSafeEqual} from 'node:crypto'
+import Router from '@koa/router'
+import {z} from 'zod'
+import {HttpError} from './errors.js'
+import {parse,type WorkspaceStore} from './workspaceStore.js'
+import type {LocalAuthStore} from './localAuth.js'
+import type {WorkspaceAgent} from '../shared/workspace.js'
+import type {WorkspaceNodes} from './workspaceGateway.js'
+import type {RunnerHub} from './runnerHub.js'
+const hash=(value:string)=>createHash('sha256').update(value).digest()
+interface Grant {id:string;agentId:string;owner:string;requestId:string;runnerId:string;authVersion:number;expiresAt:number;tokenHash:string;sealedToken:string}
+export class ComputerControlService {
+  constructor(readonly store:WorkspaceStore,readonly auth:LocalAuthStore,readonly nodes:WorkspaceNodes,readonly hub:RunnerHub){}
+  private agent(owner:string,id:string){
+    const agent=this.store.require<WorkspaceAgent>(owner,'agent',id)
+    if(agent.archived||agent.execution!=='computer'||agent.remoteAgentId)throw new HttpError(409,'该 Agent 没有可用的隔离电脑','computer_unavailable')
+    this.nodes.requireSource(owner,agent);return agent
+  }
+  allowed(id:string,runnerId:string):boolean{
+    const grant=this.store.get<Grant>('_system','computer-control',id)
+    if(!grant||grant.runnerId!==runnerId||grant.expiresAt<=Date.now()||this.auth.pushAuthorizationVersion(grant.owner)!==grant.authVersion)return false
+    try{this.agent(grant.owner,grant.agentId);return true}catch{return false}
+  }
+  private token(owner:string,agentId:string,body:unknown){
+    const input=parse(z.object({controlId:z.string().uuid(),token:z.string().min(32).max(256)}).passthrough(),body)
+    const grant=this.store.get<Grant>('_system','computer-control',input.controlId)
+    if(!grant||grant.owner!==owner||grant.agentId!==agentId||!timingSafeEqual(Buffer.from(grant.tokenHash,'hex'),hash(input.token)))throw new HttpError(403,'需要当前电脑的控制凭据','computer_control_forbidden')
+    if(!this.allowed(grant.id,grant.runnerId))throw new HttpError(410,'电脑控制权已过期','computer_control_expired')
+    return grant
+  }
+  router(){
+    const router=new Router()
+    router.get('/api/app/agents/:id/computer',async ctx=>{
+      const owner=this.auth.require(ctx).id,agent=this.agent(owner,ctx.params.id)
+      ctx.body=await this.hub.computer(owner,agent,'status',{},()=>{this.agent(owner,agent.id)})
+    })
+    router.get('/api/app/agents/:id/computer/frame',async ctx=>{
+      const owner=this.auth.require(ctx).id,agent=this.agent(owner,ctx.params.id)
+      ctx.set('Cache-Control','no-store');ctx.body=await this.hub.computer(owner,agent,'frame',{},()=>{this.agent(owner,agent.id)})
+    })
+    router.post('/api/app/agents/:id/computer/take',async ctx=>{
+      const owner=this.auth.require(ctx).id,agent=this.agent(owner,ctx.params.id),input=parse(z.object({requestId:z.string().uuid()}).strict(),(ctx.request as any).body)
+      const old=this.store.list<Grant>('_system','computer-control').find(grant=>grant.owner===owner&&grant.agentId===agent.id&&grant.expiresAt>Date.now()&&this.allowed(grant.id,grant.runnerId))
+      if(old&&old.requestId!==input.requestId)throw new HttpError(409,'电脑已由另一个页面控制，关闭原页面或等待控制权到期','computer_control_busy')
+      const runnerId=this.hub.computerRunner(owner,agent).id
+      const token=old?this.nodes.open<string>(old.sealedToken):randomBytes(32).toString('base64url')
+      const grant:Grant=old??{id:randomUUID(),requestId:input.requestId,owner,agentId:agent.id,runnerId,authVersion:this.auth.pushAuthorizationVersion(owner)??0,expiresAt:Date.now()+30000,tokenHash:hash(token).toString('hex'),sealedToken:this.nodes.seal(token)}
+      this.store.put('_system','computer-control',grant.id,grant)
+      const status=await this.hub.computer(owner,agent,'take',{controlId:grant.id},()=>{if(!this.allowed(grant.id,runnerId))throw new HttpError(403,'电脑控制请求已失效','computer_control_expired')})
+      ctx.body={...status,controlId:grant.id,token,expiresAt:grant.expiresAt}
+    })
+    router.post('/api/app/agents/:id/computer/renew',ctx=>{
+      const owner=this.auth.require(ctx).id,grant=this.token(owner,ctx.params.id,(ctx.request as any).body)
+      grant.expiresAt=Date.now()+30000;this.store.put('_system','computer-control',grant.id,grant);ctx.body={expiresAt:grant.expiresAt}
+    })
+    router.post('/api/app/agents/:id/computer/input',async ctx=>{
+      const owner=this.auth.require(ctx).id,agent=this.agent(owner,ctx.params.id),input=parse(z.object({controlId:z.string().uuid(),token:z.string(),requestId:z.string().uuid(),generation:z.number().int().nonnegative(),frameId:z.string().uuid(),action:z.record(z.string(),z.unknown())}).strict(),(ctx.request as any).body)
+      const grant=this.token(owner,agent.id,input)
+      ctx.body=await this.hub.computer(owner,agent,'input',{controlId:grant.id,requestId:input.requestId,generation:input.generation,frameId:input.frameId,action:input.action},()=>{this.token(owner,agent.id,input)})
+    })
+    router.post('/api/app/agents/:id/computer/giveback',async ctx=>{
+      const owner=this.auth.require(ctx).id,agent=this.agent(owner,ctx.params.id),input=parse(z.object({controlId:z.string().uuid(),token:z.string(),notes:z.string().max(4000).default('')}).strict(),(ctx.request as any).body)
+      const grant=this.token(owner,agent.id,input)
+      ctx.body=await this.hub.computer(owner,agent,'giveback',{controlId:grant.id,notes:input.notes},()=>{this.token(owner,agent.id,input)})
+      grant.expiresAt=0;this.store.put('_system','computer-control',grant.id,grant)
+    })
+    return router
+  }
+}

src/server/runnerHub.ts

@@ -19,6 +19,7 @@ interface Lease { runnerId:string; input:LeaseInput; calls:Map<string,{fingerpri
 
 /** Outbound-only machine transport. Browser auth and machine auth remain separate. */
 export class RunnerHub {
+  controlAllowed:(id:string,runnerId:string)=>boolean=()=>false
   readonly epoch=randomUUID()
   private online = new Map<string,{instance:string;seen:number;epoch:string;features:string[];wake?:()=>void}>()
   private retired = new Map<string,Set<string>>()
@@ -134,6 +135,19 @@ export class RunnerHub {
         },
       }}
   }
+  computerRunner(owner:string,agent:import('../shared/workspace.js').WorkspaceAgent){
+    const record=this.records().find(record=>record.enabled&&record.sourceNodeId===agent.nodeId&&(record.sourceOwner==='_system'||record.sourceOwner===owner))
+    if(!record||!this.online.get(record.id)?.features.includes('computer-control-v1'))throw new HttpError(409,'执行节点不支持电脑查看与控制，请更新 Runner','computer_control_unavailable')
+    if(!record.allowedProfiles.includes(agent.profile)||!this.auth.canUseSource(owner,agent.nodeId,agent.profile))throw new HttpError(403,'电脑 Profile 未授权','computer_profile_forbidden')
+    return record
+  }
+  async computer(owner:string,agent:import('../shared/workspace.js').WorkspaceAgent,op:string,data:Record<string,unknown>,authorize:()=>void):Promise<any>{
+    const record=this.computerRunner(owner,agent),version=this.auth.pushAuthorizationVersion(owner)
+    const valid=()=>{try{authorize();return this.auth.isUserActive(owner)&&this.auth.pushAuthorizationVersion(owner)===version}catch{return false}}
+    const result=await this.request(record.id,'computer.control',{target:{environmentId:agent.id,agentId:agent.id,ownerKey:hash(owner)},profile:agent.profile,op,...data},valid)
+    if(!valid())throw new HttpError(403,'电脑请求授权已失效','computer_control_expired')
+    return result
+  }
   async retireHelper(owner:string,helper:import('../shared/workspace.js').WorkspaceAgent):Promise<void>{
     if(!helper.temporaryGoalId||!helper.archived||!helper.helperRunnerId)throw new HttpError(403,'只能清理已退役的任务助手','helper_cleanup_forbidden')
     const current=this.store.require<import('../shared/workspace.js').WorkspaceAgent>(owner,'agent',helper.id)
@@ -152,7 +166,7 @@ export class RunnerHub {
   }
   middleware():Koa.Middleware {
     return async(ctx,next)=>{
-      const match=/^\/api\/runner\/v1\/([0-9a-f-]{36})\/(poll|admit|result|event|tool|check|artifact)$/.exec(ctx.path)
+      const match=/^\/api\/runner\/v1\/([0-9a-f-]{36})\/(poll|admit|result|event|tool|check|artifact|control-check)$/.exec(ctx.path)
       if(!match)return next()
       const record=this.store.get<RunnerRecord>('_system','runner',match[1]!),token=ctx.get('authorization').replace(/^Bearer /,'')
       if(ctx.get('origin')||!record?.enabled||!ctx.get('authorization').startsWith('Bearer ')||!timingSafeEqual(Buffer.from(record.tokenHash),Buffer.from(hash(token))))
@@ -169,7 +183,7 @@ export class RunnerHub {
         retired.add(previous.instance);this.retired.set(record.id,retired);this.disconnect(record.id);previous=undefined
       }
       if(match[2]!=='poll'&&ctx.get('x-runner-epoch')!==previous?.epoch)throw new HttpError(409,'执行连接代次已改变','runner_epoch_changed')
-      const state=this.online.get(record.id)??{instance,seen:Date.now(),features:[],epoch:`${this.epoch}:${randomUUID()}`};state.seen=Date.now();if(match[2]==='poll')state.features=ctx.get('x-runner-features').split(',').filter(value=>['computer-worker-v1','artifact-chunks-v1','helper-retirement-v1'].includes(value));this.online.set(record.id,state)
+      const state=this.online.get(record.id)??{instance,seen:Date.now(),features:[],epoch:`${this.epoch}:${randomUUID()}`};state.seen=Date.now();if(match[2]==='poll')state.features=ctx.get('x-runner-features').split(',').filter(value=>['computer-worker-v1','artifact-chunks-v1','helper-retirement-v1','computer-control-v1'].includes(value));this.online.set(record.id,state)
       ctx.set('Cache-Control','no-store')
       if(match[2]==='poll') {
         if(ctx.method!=='GET')throw new HttpError(405,'仅允许 GET','method_not_allowed')
@@ -190,6 +204,7 @@ export class RunnerHub {
       let body:any
       try {body=JSON.parse(Buffer.concat(chunks).toString())}catch{throw new HttpError(400,'JSON 无效','invalid_json')}
       if(!body||typeof body!=='object'||Array.isArray(body))throw new HttpError(400,'请求必须是对象','invalid_json')
+      if(match[2]==='control-check'){ctx.body={allowed:typeof body.controlId==='string'&&this.controlAllowed(body.controlId,record.id)};return}
       if(match[2]==='artifact'){
         const value=parse(z.object({connectionId:z.string().uuid(),id:z.string().uuid(),phase:z.enum(['begin','append','finish']),name:z.string().min(1).max(240).optional(),size:z.number().int().min(0).max(25*1024*1024).optional(),digest:z.string().regex(/^[a-f0-9]{64}$/).optional(),index:z.number().int().min(0).optional(),data:z.string().max(1024*1024).optional()}).strict(),body)
         const connection=this.connections.get(value.connectionId)

src/server/workspaceRoutes.ts

@@ -67,6 +67,7 @@ export function workspaceRouter(
         'conversationTasks',
         'editableGroups',
         'agentTeamManagement',
+        'computerControl',
         'files',
         'voice',
         'context',

src/server/workspaceRuntime.ts

@@ -299,6 +299,7 @@ export class WorkspaceRuntime extends WorkspaceScheduler {
       if (settled || !runtimeId || frame.session_id !== runtimeId) return
       const p = frame.payload ?? {},
         type = frame.type
+      if(type==='computer.paused'||type==='computer.resumed'){const current=this.getWork(owner,run.id);current.status=type==='computer.paused'?'waiting':'running';this.saveWork(owner,current);return}
       if (type === 'message.delta') {
         resultMessage.content += String(p.text ?? p.delta ?? '')
         if (resultMessage.content.trim() && !NO_REPLY.startsWith(resultMessage.content.trim()) && !resultMessage.content.trim().startsWith('[[YAOYAO_')) resultMessage.visible = true
@@ -608,6 +609,7 @@ export class WorkspaceRuntime extends WorkspaceScheduler {
         try {
           await gateway.rpc('prompt.submit', {
             session_id: runtimeId,
+            ...(target.runner?.computer?{workMarker:`[yaoyao-run:${run.runId}:${resultMessage.id}]`}:{}),
             text: `${rules}\n\n${text}\n${attachmentRefs.join('\n')}`,
           })
           binding.contextSeq = Math.max(binding.contextSeq ?? 0, admission.contextThroughSeq)

src/shared/computerControl.ts

@@ -0,0 +1,7 @@
+export interface ComputerFrame {id:string;generation:number;data:string;width:number;height:number;capturedAt:number}
+export interface ComputerControlStatus {mode:'off'|'agent'|'pausing'|'human'|'resuming'|'error';generation?:number;controlId?:string;canResume?:boolean;error?:string}
+export type ComputerInput =
+  | {kind:'click';x:number;y:number;button?:'left'|'right'|'middle';count?:number}
+  | {kind:'text';text:string}
+  | {kind:'key';key:string;modifiers?:string[]}
+  | {kind:'scroll';direction:'up'|'down'|'left'|'right';amount?:number}

src/shared/runner.ts

@@ -11,7 +11,7 @@ export interface RunnerRecord {
 }
 export interface RunnerCommand {
   id: string
-  kind: 'computer.retire' | 'http' | 'gateway.open' | 'gateway.rpc' | 'gateway.close' | 'lease.create' | 'lease.bind' | 'lease.close'
+  kind: 'computer.control' | 'computer.retire' | 'http' | 'gateway.open' | 'gateway.rpc' | 'gateway.close' | 'lease.create' | 'lease.bind' | 'lease.close'
   payload: Record<string, unknown>
   expiresAt: number
 }

tests/e2e/team-tools.spec.ts

@@ -132,3 +132,32 @@ test('explains isolated execution and keeps the edit form intact when a computer
   await dialog.getByRole('combobox',{name:'执行环境',exact:true}).scrollIntoViewIfNeeded()
   await page.screenshot({path:testInfo.outputPath('computer-execution-mobile.png')})
 })
+
+test('computer viewing is read-only until takeover and returns control with a user note',async({page},testInfo)=>{
+  await page.goto('/conversations')
+  await page.getByRole('textbox',{name:'账号',exact:true}).fill('fixture')
+  await page.getByRole('textbox',{name:'密码',exact:true}).fill('fixture-pass')
+  await page.getByRole('button',{name:'登录',exact:true}).click()
+  await expect(page.locator('.desktop-sidebar .sidebar-create-trigger')).toBeVisible()
+  const seeded=await page.request.post('/__test/computer',{data:{}});expect(seeded.ok(),await seeded.text()).toBe(true)
+  const fixture=await seeded.json();expect(fixture.conversationId).toBeTruthy()
+  await page.goto(`/conversations/${fixture.conversationId}`)
+  await page.getByRole('button',{name:'电脑',exact:true}).click()
+  const panel=page.getByRole('dialog',{name:'隔离电脑',exact:true}),screen=panel.getByRole('img',{name:'隔离电脑当前画面'})
+  await expect(screen).toBeVisible()
+  await screen.click()
+  expect((await (await page.request.get('/__test/computer')).json()).actions).toEqual([])
+  await panel.getByRole('button',{name:'接管电脑',exact:true}).click()
+  await expect(panel).toContainText('人工控制中')
+  await panel.getByRole('textbox',{name:'输入到电脑',exact:true}).fill('人工输入验收')
+  await panel.getByRole('button',{name:'输入文字',exact:true}).click()
+  await expect.poll(async()=>(await (await page.request.get('/__test/computer')).json()).actions).toEqual([{kind:'text',text:'人工输入验收'}])
+  await panel.getByRole('textbox',{name:'交还说明',exact:true}).fill('已完成设置，请继续任务')
+  await page.setViewportSize({width:390,height:844})
+  await panel.getByRole('button',{name:'交还 Agent 并继续',exact:true}).scrollIntoViewIfNeeded()
+  await page.screenshot({path:testInfo.outputPath('computer-panel-mobile.png')})
+  await panel.getByRole('button',{name:'交还 Agent 并继续',exact:true}).click()
+  await expect(panel.getByRole('button',{name:'接管电脑',exact:true})).toBeVisible()
+  await panel.getByRole('button',{name:'关闭',exact:true}).click()
+  await expect(panel).not.toBeVisible()
+})

tests/fixtures/workspace-server.ts

@@ -349,6 +349,28 @@ runtime.app.use((ctx,next) => {
   runtime.workspace.saveMessage(owner,{id:randomUUID(),conversationId:source.id,seq:0,role:'system',content:`任务已有结果。[打开任务](/conversations/${team.id}?taskId=${first.id})`,reasoning:'',status:'complete',attachments:[],tools:[],createdAt:Date.now()})
   ctx.body={conversationId:team.id,firstTaskId:first.id,secondTaskId:second.id,sourceConversationId:source.id}
 })
+const computerFixtures=new Map<string,{mode:string;generation:number;controlId?:string;actions:unknown[]}>()
+const originalComputer=runtime.runners.computer.bind(runtime.runners),originalRunner=runtime.runners.computerRunner.bind(runtime.runners)
+runtime.runners.computerRunner=(owner,agent)=>computerFixtures.has(agent.id)?{id:'11111111-1111-4111-8111-111111111111'} as any:originalRunner(owner,agent)
+runtime.runners.computer=async(owner,agent,op,data,authorize)=>{
+  const state=computerFixtures.get(agent.id);if(!state)return originalComputer(owner,agent,op,data,authorize)
+  authorize()
+  if(op==='frame')return {id:'22222222-2222-4222-8222-222222222222',generation:state.generation,width:512,height:512,capturedAt:Date.now(),data:readFileSync(resolve('public/icons/icon-512.png')).toString('base64')}
+  if(op==='take'){state.controlId=String(data.controlId);state.mode='human';state.generation++}
+  if(op==='input')state.actions.push(data.action)
+  if(op==='giveback'){state.mode='agent';state.controlId=undefined;state.generation++}
+  return {mode:state.mode,generation:state.generation,controlId:state.controlId,canResume:true,ok:true}
+}
+runtime.app.use((ctx,next)=>{
+  if(ctx.path!=='/__test/computer')return next()
+  const owner=auth.require(ctx).id
+  if(ctx.method==='POST'){
+    const agent=runtime.workspace.createAgent(owner,{name:'电脑面板验收',profile:'default',execution:'computer'})
+    computerFixtures.set(agent.id,{mode:'agent',generation:1,actions:[]})
+    ctx.body={agentId:agent.id,conversationId:runtime.workspace.list<any>(owner,'conversation').find(c=>c.memberIds[0]===agent.id)!.id};return
+  }
+  ctx.body={actions:[...computerFixtures.values()].flatMap(state=>state.actions)}
+})
 runtime.app.use(serve(resolve('dist')))
 runtime.app.use((ctx) => {
   if (!ctx.path.startsWith('/api/')) {

tests/live/worker.live.test.ts

@@ -60,11 +60,11 @@ it.skipIf(!process.env.YAOYAO_HERMES_PYTHON||!process.env.YAOYAO_COMPUTER_IMAGE)
     if(req.headers.authorization!=='Bearer fixture-private-key'){res.statusCode=401;res.end('{}');return}
     commands.push(input)
     const user=input.messages.filter((message:any)=>message.role==='user').at(-1)?.content??''
-    const helper=/fixture:helper ([0-9a-f-]{36})/.exec(String(user)),team=String(user).includes('fixture:team'),slow=String(user).includes('fixture:slow')
+    const helper=/fixture:helper ([0-9a-f-]{36})/.exec(String(user)),team=String(user).includes('fixture:team'),slow=String(user).includes('fixture:slow'),control=String(user).includes('fixture:control')
     const current=input.messages.slice(input.messages.findLastIndex((message:any)=>message.role==='user')+1)
     const called=current.some((message:any)=>message.role==='tool')
     const exported=current.some((message:any)=>message.role==='tool'&&message.name==='computer_export')
-    const message=called&&!team&&!helper&&!exported?{role:'assistant',content:null,tool_calls:['result.txt','bulk.bin'].map(path=>({id:randomUUID(),type:'function',function:{name:'computer_export',arguments:JSON.stringify({path})}}))}:called?{role:'assistant',content:'隔离任务完成'}:{role:'assistant',content:null,tool_calls:[{id:randomUUID(),type:'function',function:{name:helper?'workspace_create_helper':team?'workspace_create_agent':'computer_shell',arguments:JSON.stringify(helper?{requestId:randomUUID(),goalId:helper[1],name:'临时核验助手',title:'临时核验',brief:'在隔离电脑完成 fixture:helper-work 并导出结果'}:team?{requestId:randomUUID(),name:'隔离创建成员',profile:'default'}:{command:slow?'sleep 30; printf forbidden > late-write.txt':(process.env.YAOYAO_WORKER_PUBLIC_NETWORK==='1'?'curl --fail --silent --show-error --max-time 20 https://example.com > public.html; ':'')+'printf worker-proof > result.txt; head -c 2100000 /dev/zero > bulk.bin; id -u; pwd'})}}]}
+    const message=called&&!team&&!helper&&!exported?{role:'assistant',content:null,tool_calls:['result.txt','bulk.bin'].map(path=>({id:randomUUID(),type:'function',function:{name:'computer_export',arguments:JSON.stringify({path})}}))}:called?{role:'assistant',content:'隔离任务完成'}:{role:'assistant',content:null,tool_calls:[{id:randomUUID(),type:'function',function:{name:helper?'workspace_create_helper':team?'workspace_create_agent':'computer_shell',arguments:JSON.stringify(helper?{requestId:randomUUID(),goalId:helper[1],name:'临时核验助手',title:'临时核验',brief:'在隔离电脑完成 fixture:helper-work 并导出结果'}:team?{requestId:randomUUID(),name:'隔离创建成员',profile:'default'}:{command:control?'sleep 2; printf paused > pause-proof.txt':slow?'sleep 30; printf forbidden > late-write.txt':(process.env.YAOYAO_WORKER_PUBLIC_NETWORK==='1'?'curl --fail --silent --show-error --max-time 20 https://example.com > public.html; ':'')+'printf worker-proof > result.txt; head -c 2100000 /dev/zero > bulk.bin; id -u; pwd'})}}]}
     if(input.stream){res.setHeader('Content-Type','text/event-stream');res.end('data: '+JSON.stringify({id:randomUUID(),object:'chat.completion.chunk',model:'fixture-model',choices:[{index:0,delta:{...message,...(message.tool_calls?{tool_calls:message.tool_calls.map((tool:any,index:number)=>({...tool,index}))}:{})},finish_reason:message.tool_calls?'tool_calls':'stop'}]})+'\n\ndata: [DONE]\n\n')}
     else res.end(JSON.stringify({id:randomUUID(),object:'chat.completion',model:'fixture-model',choices:[{index:0,message,finish_reason:message.tool_calls?'tool_calls':'stop'}],usage:{prompt_tokens:10,completion_tokens:10,total_tokens:20}}))
   })
@@ -112,6 +112,31 @@ it.skipIf(!process.env.YAOYAO_HERMES_PYTHON||!process.env.YAOYAO_COMPUTER_IMAGE)
     await expect.poll(()=>app.workspace.require<any>(owner,'run',next.id).status,{timeout:45000}).toMatch(/complete|failed/)
     expect(app.workspace.require<any>(owner,'run',next.id).status).toBe('complete')
     expect(commands.flatMap(input=>input.messages).some((message:any)=>message.role==='tool'&&String(message.content).includes('/projects/changed-fixture'))).toBe(true)
+    const controlled=app.workspaceRuntime.send(owner,direct.id,{requestId:randomUUID(),content:'fixture:control'})
+    await expect.poll(()=>app.workspace.messages(owner,direct.id).some(message=>message.runId===controlled.id&&message.tools.some(tool=>tool.name==='computer_shell')),{timeout:30000}).toBe(true)
+    const cookies=setup.headers['set-cookie'].map((cookie:string)=>cookie.split(';')[0]).join('; ')
+    const fresh=await request(app.app.callback()).get('/api/app/bootstrap').set('Host',`127.0.0.1:${app.config.port}`).set('Cookie',cookies).expect(200)
+    const endpoint=`/api/app/agents/${worker.id}/computer`
+    const api=(path:string,data:unknown)=>request(app.app.callback()).post(endpoint+path).set('Host',`127.0.0.1:${app.config.port}`).set('Origin',address).set('Cookie',cookies).set('X-CSRF-Token',fresh.body.csrfToken).send(data)
+    const taken=await api('/take',{requestId:randomUUID()}).expect(200),ticket={controlId:taken.body.controlId,token:taken.body.token}
+    const status=async()=>(await request(app.app.callback()).get(endpoint).set('Host',`127.0.0.1:${app.config.port}`).set('Cookie',cookies)).body
+    await expect.poll(async()=>(await status()).mode,{timeout:30000}).toBe('human')
+    expect(app.workspace.require<any>(owner,'run',controlled.id).status).toBe('waiting')
+    const capture=async()=>(await request(app.app.callback()).get(endpoint+'/frame').set('Host',`127.0.0.1:${app.config.port}`).set('Cookie',cookies)).body
+    let frame:any
+    await expect.poll(async()=>{frame=await capture();return frame.width},{timeout:20000}).toBeGreaterThan(0)
+    const click=await api('/input',{...ticket,requestId:randomUUID(),generation:frame.generation,frameId:frame.id,action:{kind:'click',x:568,y:875}}).expect(200)
+    expect(click.body.ok).toBe(true)
+    await new Promise(resolve=>setTimeout(resolve,1200));frame=await capture()
+    await mkdir('test-results/control',{recursive:true});await writeFile('test-results/control/after-click.png',Buffer.from(frame.data,'base64'))
+    const typed=await api('/input',{...ticket,requestId:randomUUID(),generation:frame.generation,frameId:frame.id,action:{kind:'text',text:'printf control-proof > /projects/changed-fixture/manual.txt'}}).expect(200)
+    await api('/input',{...ticket,requestId:randomUUID(),generation:frame.generation,frameId:frame.id,action:{kind:'key',key:'Return'}}).expect(200)
+    frame=await capture();await writeFile('test-results/control/after-text.png',Buffer.from(frame.data,'base64'))
+    await expect.poll(async()=>readFile(join(runnerHome,'computer-workspaces',worker.id,'manual.txt'),'utf8').catch(()=>''),{timeout:10000}).toBe('control-proof')
+    await api('/input',{...ticket,token:'x'.repeat(40),requestId:randomUUID(),generation:frame.generation,frameId:frame.id,action:{kind:'key',key:'Return'}}).expect(403)
+    await api('/giveback',{...ticket,notes:'用户已在终端完成验证，请继续并核对结果。'}).expect(200)
+    await expect.poll(()=>app.workspace.require<any>(owner,'run',controlled.id).status,{timeout:45000}).toMatch(/complete|failed/)
+    expect(app.workspace.require<any>(owner,'run',controlled.id).status).toBe('complete')
     const cancelling=app.workspaceRuntime.send(owner,direct.id,{requestId:randomUUID(),content:'fixture:slow'})
     await expect.poll(()=>app.workspace.messages(owner,direct.id).some(message=>message.runId===cancelling.id&&message.tools.some(tool=>tool.name==='computer_shell')),{timeout:30000}).toBe(true)
     await app.workspaceRuntime.stop(owner,cancelling.id)

tests/server/computerControl.test.ts

@@ -0,0 +1,39 @@
+// @vitest-environment node
+import {expect,it,vi} from 'vitest'
+import {DatabaseSync} from 'node:sqlite'
+import {mkdtempSync,rmSync} from 'node:fs'
+import {tmpdir} from 'node:os'
+import {join} from 'node:path'
+import {randomUUID} from 'node:crypto'
+import {ComputerRuntime} from '../../src/runner/worker/gateway'
+it('separates view from control and fences stale, duplicate and out-of-frame input',async()=>{
+  const home=mkdtempSync(join(tmpdir(),'yaoyao-control-')),db=new DatabaseSync(':memory:')
+  const runtime=new ComputerRuntime(db,{protocol:1,runnerId:randomUUID(),serverURL:'http://127.0.0.1',hermesURL:'http://127.0.0.1',token:'fixture',allowedProfiles:['default'],artifactRoots:[],computers:{runtime:'docker',imageId:'sha256:'+'a'.repeat(64),python:'/fixture/python',hermesSource:'/fixture/source',hermesHome:'/fixture/home'}},home)
+  let permitted=true
+  try{
+    await runtime.ready
+    vi.spyOn(runtime,'resolve').mockResolvedValue({type:'resolved',cwd:'/projects/fixture',configuredCwd:'/projects/fixture',model:{}})
+    vi.spyOn(runtime.provider,'ensure').mockResolvedValue({id:'fixture',containerId:'fixture',running:true,workspace:'/fixture',isolation:'container'})
+    vi.spyOn(runtime.provider,'stop').mockResolvedValue();vi.spyOn(runtime.provider,'remove').mockResolvedValue()
+    vi.spyOn(runtime.provider,'capture').mockImplementation(async()=>({data:'image-fixture',width:100,height:80,capturedAt:Date.now()}))
+    const execute=vi.spyOn(runtime.provider,'execute').mockResolvedValue({stdout:'{"ok":true}',stderr:''})
+    const meta={environmentId:randomUUID(),agentId:randomUUID(),ownerKey:'owner'},id=randomUUID()
+    await expect(runtime.controls.input(meta,id,randomUUID(),1,randomUUID(),{kind:'key',key:'Return'})).rejects.toMatchObject({code:'computer_control_required'})
+    await runtime.controls.take(meta,'default',id,async()=>{if(!permitted)throw new Error('revoked')})
+    await vi.waitFor(async()=>expect((await runtime.controls.status(meta)).mode).toBe('human'))
+    const frame=await runtime.controls.frame(meta,()=>{})
+    expect(execute).not.toHaveBeenCalled()
+    await expect(runtime.controls.input(meta,id,randomUUID(),frame.generation-1,frame.id,{kind:'key',key:'Return'})).rejects.toMatchObject({code:'computer_control_stale'})
+    await expect(runtime.controls.input(meta,id,randomUUID(),frame.generation,frame.id,{kind:'click',x:101,y:1})).rejects.toMatchObject({code:'computer_point_invalid'})
+    const requestId=randomUUID(),action={kind:'click',x:20,y:30}
+    const first=await runtime.controls.input(meta,id,requestId,frame.generation,frame.id,action)
+    expect(await runtime.controls.input(meta,id,requestId,frame.generation,frame.id,action)).toEqual(first)
+    expect(execute).toHaveBeenCalledTimes(1)
+    await expect(runtime.controls.input(meta,id,requestId,frame.generation,frame.id,{kind:'click',x:30,y:30})).rejects.toMatchObject({code:'idempotency_conflict'})
+    permitted=false
+    await expect(runtime.controls.input(meta,id,randomUUID(),frame.generation,frame.id,action)).rejects.toThrow('revoked')
+    await vi.waitFor(async()=>expect((await runtime.controls.status(meta)).mode).toBe('off'),{timeout:6500,interval:100})
+    expect(runtime.provider.stop).toHaveBeenCalled()
+    await expect(runtime.controls.input(meta,id,randomUUID(),frame.generation,frame.id,action)).rejects.toMatchObject({code:'computer_control_required'})
+  }finally{await runtime.controls.close();await runtime.pool.close();db.close();rmSync(home,{recursive:true,force:true})}
+},10000)

tests/server/computerControlRoutes.test.ts

@@ -0,0 +1,39 @@
+// @vitest-environment node
+import {expect,it,vi} from 'vitest'
+import Koa from 'koa'
+import {bodyParser} from '@koa/bodyparser'
+import request from 'supertest'
+import {mkdtempSync,rmSync} from 'node:fs'
+import {tmpdir} from 'node:os'
+import {join} from 'node:path'
+import {randomUUID} from 'node:crypto'
+import {WorkspaceStore} from '../../src/server/workspaceStore'
+import {WorkspaceNodes} from '../../src/server/workspaceGateway'
+import {loadServerConfig} from '../../src/server/config'
+import {ComputerControlService} from '../../src/server/computerControls'
+import {HttpError} from '../../src/server/errors'
+import type {LocalAuthStore} from '../../src/server/localAuth'
+import type {RunnerHub} from '../../src/server/runnerHub'
+it('returns a private control credential once, rejects other callers and invalidates revoked grants',async()=>{
+  const home=mkdtempSync(join(tmpdir(),'yaoyao-control-routes-')),store=new WorkspaceStore(home),runnerId=randomUUID()
+  let authVersion=1
+  const auth={require:(ctx:any)=>({id:ctx.get('x-fixture-owner')||'owner'}),pushAuthorizationVersion:()=>authVersion} as unknown as LocalAuthStore
+  const nodes=new WorkspaceNodes(store,loadServerConfig({HERMES_YAOYAO_HOME:home}),{} as any)
+  const calls=vi.fn(async(_owner,_agent,op,_data,authorize)=>{authorize();return {mode:op==='take'?'pausing':'agent'}})
+  const hub={computerRunner:()=>({id:runnerId}),computer:calls} as unknown as RunnerHub
+  const service=new ComputerControlService(store,auth,nodes,hub),app=new Koa()
+  app.use(async(ctx,next)=>{try{await next()}catch(error){ctx.status=error instanceof HttpError?error.status:500;ctx.body={code:(error as any).code}}});app.use(bodyParser());app.use(service.router().routes())
+  try{
+    const agent=store.createAgent('owner',{name:'电脑',profile:'default',execution:'computer'}),base=`/api/app/agents/${agent.id}/computer`
+    const taken=await request(app.callback()).post(base+'/take').send({requestId:randomUUID()}).expect(200)
+    expect(taken.body.token.length).toBeGreaterThan(32)
+    expect(JSON.stringify((await request(app.callback()).get(base).expect(200)).body)).not.toContain(taken.body.token)
+    expect(service.allowed(taken.body.controlId,runnerId)).toBe(true)
+    await request(app.callback()).post(base+'/renew').send({controlId:taken.body.controlId,token:'x'.repeat(40)}).expect(403)
+    await request(app.callback()).post(base+'/renew').set('x-fixture-owner','other').send({controlId:taken.body.controlId,token:taken.body.token}).expect(403)
+    await request(app.callback()).post(base+'/take').send({requestId:randomUUID()}).expect(409)
+    authVersion++
+    expect(service.allowed(taken.body.controlId,runnerId)).toBe(false)
+    await request(app.callback()).post(base+'/renew').send({controlId:taken.body.controlId,token:taken.body.token}).expect(410)
+  }finally{nodes.close();store.close();rmSync(home,{recursive:true,force:true})}
+})

tests/server/computerPool.test.ts

@@ -98,3 +98,15 @@ it('applies a changed authoritative cwd only after an environment is released',a
   expect(next.generation).toBeGreaterThan(lease.generation)
   await f.pool.close()
 })
+it('transfers an idle live computer with a new generation and rejects its previous holder',async()=>{
+  const f=fixture();await f.pool.recover();const resource=spec(),lease=await f.pool.acquire(resource,'agent',()=>{})
+  f.provider.inspect=vi.fn(async()=>({id:resource.id,containerId:'same-computer',running:true,workspace:'/fixture',isolation:'container'}))
+  const human=await f.pool.transfer(lease,'human',()=>{})
+  expect(human.generation).toBeGreaterThan(lease.generation)
+  expect(()=>f.pool.authorize(lease)).toThrow()
+  expect(f.provider.stop).not.toHaveBeenCalled()
+  f.pool.authorize(human)
+  const resumed=await f.pool.transfer(human,'agent',()=>{})
+  expect(resumed.generation).toBeGreaterThan(human.generation)
+  await f.pool.close()
+})