Browse Source

fix(bot): 服务器按云虚拟机方式操作,并放开 Hermes 工具

服务器与脚本机分开:先看真实桌面再操作,本机只碰夭夭所在电脑。Bot 可用 Hermes 全部工具,长期记忆仍用 Bot 自己的。
Samien 1 month ago
parent
commit
159943e

+ 1 - 1
src/client/components/app/HostToolsPanel.vue

@@ -69,7 +69,7 @@ onMounted(() => { void load(); void loadNames() })
     <p>这些开关对所有机器人生效,不再按单个机器人勾选。关掉之后,新开或恢复的会话才会按新设置执行。</p>
     <form @submit.prevent="save" @change="changed">
       <label class="toggle"><input v-model="scriptMachine" type="checkbox" :disabled="loading || saving" /><span><strong>脚本机</strong><small>客户连接的 Mac。机器人按名称点名,可以查看画面、操作、读写文件和执行命令。</small></span></label>
-      <label class="toggle"><input v-model="serverComputer" type="checkbox" :disabled="loading || saving" /><span><strong>服务器</strong><small>夭夭正在运行的这台电脑。查看本机只看它,不会跳到别的在线 Mac。</small></span></label>
+      <label class="toggle"><input v-model="serverComputer" type="checkbox" :disabled="loading || saving" /><span><strong>服务器</strong><small>夭夭正在运行的这台电脑。操作方式和云虚拟机一样:先看真实桌面,再点击、输入、按键或滚动。查看本机只看它。</small></span></label>
       <label class="toggle"><input v-model="vm" type="checkbox" :disabled="loading || saving" /><span><strong>虚拟环境</strong><small>服务端上的隔离桌面。聊天留在夭夭,操作派到虚拟机里。</small></span></label>
       <label class="toggle"><input v-model="cloud" type="checkbox" :disabled="loading || saving" /><span><strong>云虚拟机</strong><small>共享的 Grok Bot 电脑。先看真实桌面,再点击、输入、按键或滚动。</small></span></label>
       <label class="toggle"><input v-model="deny" type="checkbox" :disabled="loading || saving" /><span><strong>禁用服务端本机的终端和文件</strong><small>默认开启。开启后对话拿不到服务端的 terminal 和 file 工具。这和上面的电脑开关是分开的。</small></span></label>

+ 4 - 3
src/client/components/workspace/ComputerPanel.vue

@@ -24,7 +24,7 @@ const controllable=computed(()=>canSend.value&&!busy.value)
 const targets=computed(()=>{
   const agent=selectedAgent.value,envs=agent?agentEnvs(agent):{vm:false,cloud:false,desktop:false,browser:false},info=envTargets.value
   const list:{key:string;label:string;backend:'desktop'|'cloud'|'vm';host?:string;disabled?:boolean}[]=[]
-  if(envs.desktop)for(const h of info?.hosts??[])list.push({key:'desktop:'+h.id,label:(h.id==='local'?'本机':'Mac')+' · '+(h.name||'未命名'),backend:'desktop',host:h.id,disabled:!h.online})
+  if(envs.desktop)for(const h of info?.hosts??[])list.push({key:'desktop:'+h.id,label:(h.id==='local'?'服务器 · 本机':'脚本机')+' · '+(h.name||'未命名'),backend:'desktop',host:h.id,disabled:!h.online})
   if(envs.cloud)list.push({key:'cloud',label:'云端 · Grok Bot'+(info?.cloudConfigured?'':' · 未连接'),backend:'cloud',disabled:!info?.cloudConfigured})
   if(envs.vm)list.push({key:'vm',label:'本地虚拟机'+(info?.vmEnabled?'':' · 未启用'),backend:'vm',disabled:!info?.vmEnabled})
   return list
@@ -131,7 +131,7 @@ defineExpose({take,close,releaseControl:async()=>{await run(giveBack);if(ticket.
   <Teleport to="body" :disabled="embedded||standalone">
     <component :is="embedded||standalone?'section':'dialog'" ref="dialog" class="computer-panel" :class="{embedded,standalone}" :aria-label="state.backend==='local'?'本机':state.backend==='browser'?'浏览器':'隔离电脑'" @cancel.prevent="close">
       <header>
-        <div><strong>{{state.backend==='grok'?'Grok Bot 云端电脑':state.backend==='local'?(state.hostName??'本机桌面'):state.backend==='browser'?'浏览器 · '+state.hostName:'本地虚拟机'}}</strong><small>{{ labels[state.mode] }}</small></div>
+        <div><strong>{{state.backend==='grok'?'Grok Bot 云端电脑':state.backend==='local'?(state.hostName??'服务器桌面'):state.backend==='browser'?'浏览器 · '+state.hostName:'本地虚拟机'}}</strong><small>{{ labels[state.mode] }}</small></div>
         <select v-if="targets.length>1" :value="currentKey" aria-label="切换电脑环境" :disabled="!!ticket||busy" @change="switchTarget(($event.target as HTMLSelectElement).value)"><option v-for="target in targets" :key="target.key" :value="target.key" :disabled="target.disabled">{{ target.label }}</option></select>
         <select v-if="agents.length>1" v-model="selected" aria-label="选择电脑" :disabled="!!ticket"><option v-for="agent in agents" :key="agent.id" :value="agent.id">{{ agent.name }}</option></select>
         <span v-else-if="!embedded">{{ agents[0]?.name }}</span>
@@ -147,13 +147,14 @@ defineExpose({take,close,releaseControl:async()=>{await run(giveBack);if(ticket.
       </header>
       <BrowserToolbar v-if="state.backend==='browser'" :state="browserState" :enabled="controllable" @action="browserAction"/>
       <div class="computer-screen" :class="{controlling:canSend,'actual-size':actualSize}">
-        <img v-if="frame" :src="`data:image/png;base64,${frame.data}`" :alt="state.backend==='local'?'本机当前画面':state.backend==='browser'?'浏览器当前画面':'隔离电脑当前画面'" draggable="false" tabindex="0" title="点按后可使用键盘、滚轮和拖动;按 Esc 退出键盘控制" @pointerdown="down" @pointerup="up" @pointercancel="dragStart=undefined" @wheel="wheel" @keydown="keydown" @click="click" @contextmenu.prevent="click">
+        <img v-if="frame" :src="`data:image/png;base64,${frame.data}`" :alt="state.backend==='local'?'服务器当前画面':state.backend==='browser'?'浏览器当前画面':'隔离电脑当前画面'" draggable="false" tabindex="0" title="点按后可使用键盘、滚轮和拖动;按 Esc 退出键盘控制" @pointerdown="down" @pointerup="up" @pointercancel="dragStart=undefined" @wheel="wheel" @keydown="keydown" @click="click" @contextmenu.prevent="click">
         <p v-else>{{ state.mode==='off' ? '打开电脑后可查看桌面。' : '正在读取电脑画面…' }}</p>
         <span v-if="frame&&(!mine||frame.generation!==state.generation)" class="view-label">{{mine?'正在同步画面':state.mode==='off'?'上次画面':'仅查看'}}</span>
       </div>
       <p v-if="error||state.error" class="computer-error" role="alert">{{ error||state.error }}</p>
       <footer v-if="controlsOpen||(!embedded&&!ticket)">
         <p v-if="state.backend==='grok'">多个机器人共享同一台 Grok Bot 云端电脑、工作文件与浏览器资料。</p>
+        <p v-else-if="state.backend==='local'">服务器就是夭夭所在的这台电脑。先看真实桌面,再点击、输入、按键或滚动。</p>
         <p v-if="selectedAgent?.computerEnvironmentId">共享电脑:{{ selectedAgent.computerEnvironmentName }}。成员共用工作文件和浏览器资料,可同时执行命令与文件操作,桌面操作交替进行;人工接管时机器人暂停等待。</p>
         <p v-if="!ticket">查看不会输入任何内容。接管后机器人会暂停,已有操作结束后才允许人工操作。</p>
         <button v-if="!ticket&&state.mode!=='human'" type="button" :disabled="busy" @click="embedded?emit('controlRequest'):take()">{{ state.mode==='off'?'打开并控制':'接管电脑' }}</button>

+ 6 - 5
src/client/components/workspace/LocalVmChatPanel.vue

@@ -39,7 +39,8 @@ const desktopHosts=computed(()=>(native.value?.hosts??[]).filter(host=>host.id==
 const activeEnvLine=computed(()=>{
  const parts:string[]=[]
  if(modern.value){
-  if(computers.value.scriptMachine||computers.value.serverComputer)parts.push('脚本机')
+  if(computers.value.serverComputer)parts.push('服务器')
+  if(computers.value.scriptMachine)parts.push('脚本机')
   if(computers.value.cloud)parts.push('云端')
   if(computers.value.vm)parts.push('虚拟环境')
   return parts.join(' · ')
@@ -51,7 +52,7 @@ const activeEnvLine=computed(()=>{
 })
 const previewTargets=computed(()=>{
  const list:{key:string;label:string;backend:'desktop'|'cloud'|'vm';host?:string}[]=[]
- if(computers.value.scriptMachine||computers.value.serverComputer)for(const h of desktopHosts.value){if(h.online)list.push({key:'desktop:'+h.id,label:h.id==='local'?'本机 · '+(h.name||'桌面端'):h.name,backend:'desktop',host:h.id})}
+ if(computers.value.scriptMachine||computers.value.serverComputer)for(const h of desktopHosts.value){if(h.online)list.push({key:'desktop:'+h.id,label:h.id==='local'?'服务器 · '+(h.name||'本机'):'脚本机 · '+(h.name||'未命名'),backend:'desktop',host:h.id})}
  if(computers.value.cloud&&cloud.value?.configured)list.push({key:'cloud',label:'云端 · Grok Bot',backend:'cloud'})
  if(computers.value.vm)list.push({key:'vm',label:'虚拟环境',backend:'vm'})
  return list
@@ -188,16 +189,16 @@ onBeforeUnmount(()=>{closed=true;revision++;clearTimeout(timer)})
     <p class="hint">电脑开关在 Bot 设置里,对所有机器人一起生效。</p>
     <div class="computer-options">
     <template v-if="showDesktop">
-      <p>查看本机就是夭夭所在的这台电脑。其他脚本机按名称点名,不会因为只有一台在线就自动跳过去。</p>
+      <p>服务器就是夭夭所在的这台电脑,操作方式和云虚拟机一样:先看真实桌面,再点击、输入、按键或滚动。其他脚本机按名称点名,不会因为只有一台在线就自动跳过去。</p>
       <div v-for="host in desktopHosts" :key="host.id" class="host-permission" :class="{offline:!host.online}">
-       <div class="host-permission-head"><strong>{{host.id==='local'?'本机':host.name||'桌面主机'}}</strong><small>{{host.online?'在线':'离线'}}</small></div>
+       <div class="host-permission-head"><strong>{{host.id==='local'?'服务器 · 本机':host.name||'脚本机'}}</strong><small>{{host.online?'在线':'离线'}}</small></div>
        <div class="permission-list"><span>{{host.local.authorized?'✓':'○'}} 本机控制授权</span><span>{{host.local.screen?'✓':'○'}} 屏幕录制</span><span>{{host.local.accessibility?'✓':'○'}} 辅助功能</span><span>{{host.local.fullAuthorized?'✓':'○'}} 文件与命令</span></div>
        <button v-if="host.online&&!host.local.ready" class="primary" :disabled="busy||!isAdmin" @click="authorizeHost(host.id)">在桌面端授权</button>
        <p v-if="host.online&&!host.local.ready" class="hint">在 {{host.id==='local'?'这台':host.name}} Mac 上确认授权,并按系统提示开启权限;系统可能要求重新打开 App。</p>
        <button v-else-if="host.online&&!host.local.fullAuthorized" :disabled="busy||!isAdmin" @click="authorizeHost(host.id,'full')">授权文件与命令</button>
        <p v-if="host.online&&host.local.ready&&!host.local.fullAuthorized" class="hint">授权后,机器人可以在这台 Mac 的用户主目录内读写文件并执行命令;每台电脑单独授权,可随时撤销。</p>
       </div>
-      <button class="primary" :disabled="busy||!native?.online||!native?.local.ready" @click="openDesktop()">接管本机</button>
+      <button class="primary" :disabled="busy||!native?.online||!native?.local.ready" @click="openDesktop()">接管服务器</button>
     </template>
     <template v-if="showCloud">
      <p>多个机器人共用同一台 Grok Bot 云端虚拟机、工作文件和浏览器登录。基础机器人和聊天记录保持独立。</p>

+ 6 - 3
src/server/desktopEnvironments.ts

@@ -54,15 +54,18 @@ interface HostState {
 }
 const newHostState=(key:string,remote:boolean):HostState=>({key,remote,seen:0,pending:new Map(),queue:new Map(),manual:new Map(),paused:new Set(),frames:new Map(),generation:new Map(),notes:new Map(),activity:new Map()})
 export const DESKTOP_ENVIRONMENT_TOOLS=[
- {id:'desktop_environment_view',name:'desktop_environment_view',description:'查看当前已选电脑的真实截图,返回截图尺寸、环境类型和连接的电脑名称。',inputSchema:{type:'object',properties:{host:{type:'string',description:'可选,目标电脑的名称或编号;缺省用默认电脑'}},additionalProperties:false}},
- {id:'desktop_environment_action',name:'desktop_environment_action',description:'操作当前已选电脑。必须先查看新截图;kind 为 click、drag、text、key 或 scroll,坐标使用截图像素。可用 host 指定目标电脑。',inputSchema:{type:'object',properties:{host:{type:'string'},kind:{enum:['click','drag','text','key','scroll']},x:{type:'integer'},y:{type:'integer'},fromX:{type:'integer'},fromY:{type:'integer'},toX:{type:'integer'},toY:{type:'integer'},button:{enum:['left','right','middle']},count:{type:'integer'},text:{type:'string'},key:{type:'string'},modifiers:{type:'array',items:{enum:['ctrl','alt','shift','super']}},direction:{enum:['up','down','left','right']},amount:{type:'integer'}},required:['kind'],additionalProperties:false}},
+ {id:'desktop_environment_view',name:'desktop_environment_view',description:'查看电脑真实桌面截图,返回截图尺寸、环境类型和电脑名称。操作前必须先看。可用 host 指定目标电脑;省略或写「本机」是夭夭所在的服务器。',inputSchema:{type:'object',properties:{host:{type:'string',description:'可选,目标电脑的名称或编号;缺省用默认电脑'}},additionalProperties:false}},
+ {id:'desktop_environment_action',name:'desktop_environment_action',description:'操作电脑桌面。先查看真实桌面,再使用 click、text、key、scroll 或 drag;坐标用截图像素。可用 host 指定目标电脑。',inputSchema:{type:'object',properties:{host:{type:'string'},kind:{enum:['click','drag','text','key','scroll']},x:{type:'integer'},y:{type:'integer'},fromX:{type:'integer'},fromY:{type:'integer'},toX:{type:'integer'},toY:{type:'integer'},button:{enum:['left','right','middle']},count:{type:'integer'},text:{type:'string'},key:{type:'string'},modifiers:{type:'array',items:{enum:['ctrl','alt','shift','super']}},direction:{enum:['up','down','left','right']},amount:{type:'integer'}},required:['kind'],additionalProperties:false}},
  {id:'desktop_file_list',name:'desktop_file_list',description:'列出当前电脑允许范围内的目录内容,返回名称、类型、大小与修改时间。路径相对于用户主目录,可用 ~。可用 host 指定目标电脑。',inputSchema:{type:'object',properties:{path:{type:'string'},host:{type:'string'}},required:['path'],additionalProperties:false}},
  {id:'desktop_file_read',name:'desktop_file_read',description:'读取当前电脑上的一个文件,返回 base64 内容与大小。路径相对于用户主目录;超出允许范围会被拒绝。可用 host 指定目标电脑。',inputSchema:{type:'object',properties:{path:{type:'string'},host:{type:'string'}},required:['path'],additionalProperties:false}},
  {id:'desktop_file_write',name:'desktop_file_write',description:'把 base64 内容写入当前电脑上的文件(覆盖已存在文件)。路径相对于用户主目录;超出允许范围会被拒绝。可用 host 指定目标电脑。',inputSchema:{type:'object',properties:{path:{type:'string'},data:{type:'string'},host:{type:'string'}},required:['path','data'],additionalProperties:false}},
  {id:'desktop_shell',name:'desktop_shell',description:'在这台电脑的用户主目录执行一条 shell 命令,返回 stdout、stderr 与退出码。长任务会超时终止;不要用它代替文件工具读写大文件。可用 host 指定目标电脑。',inputSchema:{type:'object',properties:{command:{type:'string'},cwd:{type:'string'},timeoutMs:{type:'integer'},host:{type:'string'}},required:['command'],additionalProperties:false}},
 ]
 export const DESKTOP_FILE_TOOL_IDS=new Set(['desktop_file_list','desktop_file_read','desktop_file_write','desktop_shell'])
-export const DESKTOP_ENVIRONMENT_RULES='当前环境是脚本机。本机是夭夭正在运行的这台电脑。使用 desktop_environment_view/action 查看和操作;省略 host 或写「本机」只看这台电脑,其他脚本机用名称点名。先查看再操作,页面变化后重新获取截图;不得猜测旧坐标。desktop_file_list/read/write 在用户主目录范围内读写文件,desktop_shell 在主目录执行命令;这些是真实生效的操作。虚拟环境和云虚拟机都不是脚本机。人工接管时等待交还。电脑断开时报告原因,不得宣称操作成功。'
+/** Satellite Macs connected as script machines. */
+export const DESKTOP_ENVIRONMENT_RULES='当前环境是脚本机,也就是客户连接的 Mac。电脑操作、文件读写和命令必须使用 desktop_*,并用 host 写明电脑名称。先查看真实桌面,再使用 click、text、key、scroll 或 drag;页面变化后重新截图,不得猜测旧坐标。文件与 shell 在用户主目录范围内真实生效。服务器本机、虚拟环境和云虚拟机都不是脚本机。人工接管时等待交还。电脑断开时报告原因,不得宣称操作成功。'
+/** Yaoyao host Mac — same look-then-act pattern as the cloud computer. */
+export const SERVER_COMPUTER_RULES='当前环境是服务器,也就是夭夭正在运行的这台电脑。电脑操作、文件读写和命令必须使用 desktop_*;省略 host 或写「本机」只看这台电脑。先查看真实桌面,再使用 click、text、key、scroll 或 drag;页面变化后重新截图,不得猜测旧坐标。文件与 shell 在用户主目录范围内真实生效。脚本机、虚拟环境和云虚拟机都不是这台服务器。人工接管时等待交还。电脑断开时报告原因,不得宣称操作成功。'
 
 /** The loopback native transport is admitted only by serviceInstance's
  *  private capability; remote hosts are admitted by DesktopHostHub's bearer

+ 9 - 6
src/server/workspaceRuntime.ts

@@ -1,4 +1,4 @@
-import {DESKTOP_ENVIRONMENT_TOOLS,DESKTOP_ENVIRONMENT_RULES,DESKTOP_FILE_TOOL_IDS,type DesktopEnvironments} from './desktopEnvironments.js'
+import {DESKTOP_ENVIRONMENT_TOOLS,DESKTOP_ENVIRONMENT_RULES,SERVER_COMPUTER_RULES,DESKTOP_FILE_TOOL_IDS,type DesktopEnvironments} from './desktopEnvironments.js'
 import {GROK_COMPUTER_TOOLS,grokComputerRules,type GrokCloud} from './grokCloud.js'
 import {VM_COMPUTER_TOOLS,VM_COMPUTER_RULES,VmToolSession} from './vmComputer.js'
 import { randomUUID } from 'node:crypto'
@@ -35,8 +35,10 @@ function environmentMentions(text: string, hostNames: string[]) {
   const cloud = /云虚拟机|cloud_computer|Grok Bot 虚拟机/.test(text)
   const vm = /虚拟环境/.test(text) || /computer_(desktop|shell|action|read_file|write_file)/.test(text)
   const named = hostNames.some(name => name.length >= 2 && name !== '本机' && text.includes(name))
-  const desktop = /脚本机|本机|desktop_/.test(text) || named
-  return { desktop, vm, cloud }
+  const script = /脚本机/.test(text) || named
+  const server = /服务器|本机/.test(text) || (/desktop_/.test(text) && !script)
+  const desktop = server || script || /desktop_/.test(text)
+  return { desktop, server, script, vm, cloud }
 }
 
 import { WorkspaceScheduler, type Work, NO_REPLY, HOST_FALLBACK, WORKSPACE_CONCURRENCY_LIMIT, RESOURCE_WAIT_CODES } from './workspaceScheduler.js'
@@ -767,14 +769,14 @@ export class WorkspaceRuntime extends WorkspaceScheduler {
         const assignmentId = this.store.require<Run>(owner, 'run', run.runId).assignmentId
         const globals = globalComputers(this.store.home)
         const eligible = !agent.remoteAgentId && !agent.temporaryGoalId && !agent.archived
-        const open = [eligible && (globals.scriptMachine || globals.serverComputer) ? '脚本机' : '', eligible && globals.vm && agent.envs ? '虚拟环境' : '', eligible && globals.cloud ? '云虚拟机' : ''].filter(Boolean)
+        const open = [eligible && globals.scriptMachine ? '脚本机' : '', eligible && globals.serverComputer ? '服务器(本机)' : '', eligible && globals.vm && agent.envs ? '虚拟环境' : '', eligible && globals.cloud ? '云虚拟机' : ''].filter(Boolean)
         const hostNames = this.desktopEnvironments?.hostStates(owner).map(host => host.name).filter(Boolean) ?? []
         const prior = this.store.messages(owner, c.id, Number.MAX_SAFE_INTEGER, Number.MAX_SAFE_INTEGER, true, run.conversationTaskId)
           .filter(message => message.id !== resultMessage.id && message.seq <= run.triggerSeq)
           .map(message => `${message.content}\n${message.tools.map(tool => String(tool.name ?? tool.tool_name ?? '')).join(' ')}`)
           .join('\n')
         const hit = environmentMentions(`${trigger.content}\n${prior}`, hostNames)
-        const envDirective = `同一条对话里,只使用 Bot 设置里全局开放的环境:${open.join('、') || '无'}。没开放的不要用。`
+        const envDirective = `Hermes 的工具都可以用,包括网页、浏览器、终端和文件。不要使用 Hermes 的 memory 工具,长期记忆只用 Bot 自己的记忆。另外可以使用全局开放的电脑环境:${open.join('、') || '无'}。没开放的电脑环境不要用。这些环境是额外的桌面,不代替 Hermes 工具。`
         const rules = [
           envDirective,
           `你是 ${agent.name}。${agent.description ? `描述:${agent.description}\n\n` : ''}${personaSection(agent) ? `用户为它设定的工作规范:\n${personaSection(agent)}\n\n` : ''}以下是用户为这个独立机器人配置的角色与规则(版本 ${agent.revision}):\n${agent.remoteAgentId ? '配置由远端机器人管理。' : agent.instructions}`,
@@ -802,7 +804,8 @@ export class WorkspaceRuntime extends WorkspaceScheduler {
           root.projectId ? `当前项目 ID:${root.projectId}。只能使用当前项目记忆。` : '',
           `本次来源消息 ID:${run.messageId};当前会话 ID:${c.id}。`,
           plugins ? '本轮已挂载用户为当前 Bot 授权的插件工具,工具名以 plugin_ 开头,说明中包含实际服务和操作。仅按用户当前任务使用;连接或重新授权应用请让用户打开 Bot 模式的工具 → 已连接应用。不要索取 API Key 或在回复中展示凭据。' : '',
-          desktop && hit.desktop ? DESKTOP_ENVIRONMENT_RULES+(this.desktopEnvironments?.onlineHostsLine(owner)??'') : '',
+          desktop && hit.server && globals.serverComputer ? SERVER_COMPUTER_RULES+(this.desktopEnvironments?.onlineHostsLine(owner)??'') : '',
+          desktop && hit.script && globals.scriptMachine ? DESKTOP_ENVIRONMENT_RULES+(this.desktopEnvironments?.onlineHostsLine(owner)??'') : '',
           cloud && hit.cloud ? grokComputerRules() : '',
           dispatchedVm && hit.vm ? VM_COMPUTER_RULES : '',
           `[yaoyao-run:${run.runId}:${resultMessage.id}]`,