| 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326 |
- """Progressive subdirectory hint discovery: as the agent navigates into
- subdirectories via tool calls, load project context files (AGENTS.md, CLAUDE.md,
- .cursorrules) from them and append to the tool result — context arrives without
- touching the system prompt (prompt caching preserved). Complements the startup
- CWD-only loading in ``prompt_builder.py``."""
- import hashlib
- import logging
- import os
- import shlex
- from pathlib import Path
- from typing import Dict, Any, Optional, Set
- from agent.prompt_builder import _read_text_with_timeout, _scan_context_content, _truncate_content
- from agent.search_policy import SEARCH_PRUNE_DIR_NAMES
- logger = logging.getLogger(__name__)
- # Same filenames as prompt_builder.py, in priority order (first match wins per dir).
- _HINT_FILENAMES = ["AGENTS.override.md", "AGENTS.md", "agents.md", "CLAUDE.md", "claude.md", ".cursorrules"]
- # Per-file ceiling for on-demand subdirectory hints. 32 KiB matches Codex's `project_doc_max_bytes` default
- # (Claude Code and Cursor apply none); it is a guard against a stray huge CLAUDE.md in a vendored tree, not a
- # target — keep area AGENTS.md files well under it (~8k) because this text lands in a tool result on the first
- # touch of that directory. Over the ceiling: head+tail kept, marker with the path so the agent can read_file it,
- # and a WARNING in the log (the old 8k silent tail-chop cut apps/desktop/AGENTS.md for months unnoticed).
- _MAX_HINT_CHARS = 32_000
- _PATH_ARG_KEYS = {"path", "file_path", "workdir"}
- _COMMAND_TOOLS = {"terminal"}
- _MAX_ANCESTOR_WALK = 5 # ancestor levels walked per path — bounds deep-path scans
- # Shared with broad recursive search probes so context discovery and search never drift into
- # different dependency/cache/build trees (those hold *copies* of context files, never authoritative ones).
- _EXCLUDED_DIR_NAMES = SEARCH_PRUNE_DIR_NAMES
- def _digest(content: str) -> str:
- return hashlib.sha256(content.encode("utf-8")).hexdigest()
- def _resolved_hint_target(hint_path: Path, working_dir: Path) -> Optional[Path]:
- """Resolved hint-file target, or None when the file must not be loaded.
- ``is_file()`` and ``read_text`` follow symlinks, so a checked-in
- ``sub/AGENTS.md -> ~/.aws/credentials`` would inject an out-of-tree file
- into the tool result. The resolved target must stay inside the resolved
- working dir (same containment ``_within_working_dir`` applies to the
- directory itself) and pass the canonical read deny-list
- (``context_references`` applies it to explicit @-references), which also
- catches in-tree targets like a symlink to the project ``.env``.
- """
- try:
- resolved = hint_path.resolve()
- except (OSError, RuntimeError):
- return None
- try:
- inside = resolved.is_relative_to(working_dir)
- except (OSError, ValueError, RuntimeError):
- inside = False
- if not inside:
- return None
- try:
- from agent.file_safety import get_read_block_error
- blocked = get_read_block_error(str(resolved)) is not None
- except Exception:
- # Mirror context_references: a deny-list lookup that fails re-opens the
- # exact hole the check closes, so fail closed.
- return None
- return None if blocked else resolved
- def _first_hint_file(directory: Path):
- """``(path, stripped content)`` of the first readable non-empty hint file
- in *directory* (priority order), or None. Unreadable files are skipped."""
- from agent.safe_worker_policy import safe_worker_enabled
- if safe_worker_enabled():
- return None
- for filename in _HINT_FILENAMES:
- candidate = directory / filename
- try:
- if not candidate.is_file() or (target := _resolved_hint_target(candidate, directory)) is None:
- continue
- # Read the resolved target (not the link path) so a symlink swapped
- # between check and read still lands on the vetted file.
- content = target.read_text(encoding="utf-8-sig").strip()
- except (OSError, UnicodeDecodeError):
- continue
- return candidate, content
- return None
- _NAV_COMMANDS = frozenset({"cd", "pushd"})
- _SHELL_OPERATORS = frozenset({"&&", "||", "|", ";", "&", ";;", "|&", "(", ")"})
- def _nav_targets(cmd: str) -> list:
- """Operands of `cd` / `pushd` that begin a shell segment. `cd -` and bare `cd` yield nothing."""
- lexer = shlex.shlex(cmd, posix=True, punctuation_chars=True)
- lexer.whitespace_split = True
- try:
- tokens = list(lexer)
- except ValueError:
- return []
- targets, segment_start = [], True
- for idx, token in enumerate(tokens):
- if token in _SHELL_OPERATORS:
- segment_start = True
- continue
- if segment_start and token in _NAV_COMMANDS:
- operand = next((t for t in tokens[idx + 1:] if t in _SHELL_OPERATORS or not t.startswith("-")), None)
- if operand and operand not in _SHELL_OPERATORS:
- targets.append(operand)
- segment_start = False
- return targets
- # A container cwd is not a host path: a docker/local session whose working dir is the backend
- # container's default /home (or /root) home must not start hint discovery either — the same
- # vacuous-containment hazard as the desktop home pin, just container-shaped (#76902).
- _CONTAINER_HOMES = frozenset({Path("/home"), Path("/root"), Path("/workspace")})
- def _is_home_like_working_dir(working_dir: Path) -> bool:
- """True when ``working_dir`` is a home the desktop/container fell back to — not a project."""
- home = Path.home()
- if working_dir == home or working_dir in _CONTAINER_HOMES or working_dir.parent in _CONTAINER_HOMES:
- return True
- # Also the local-sandbox home mount: a docker backend binds the host home under a
- # container path whose basename stays the user name (e.g. /mnt/host-home/brooklyn).
- for part in working_dir.parts:
- if part in (".home", "host-home", "host_home"):
- return True
- return False
- class SubdirectoryHintTracker:
- """Track which directories the agent visits and load hints on first access.
- Usage: after each tool call, ``hints = tracker.check_tool_call(name, args)``
- and append the returned text to the tool result.
- """
- def __init__(self, working_dir: Optional[str] = None, *, enabled: bool = True):
- # ``enabled=False`` mirrors ``skip_context_files``: a session that opted out of
- # AGENTS.md/CLAUDE.md injection at startup must not get the same files spliced into
- # tool results later — cron jobs relaying exact stdout leaked them to chat (#9441).
- self.enabled = enabled
- self.working_dir = Path(working_dir or os.getcwd()).resolve()
- # $HOME is not a project (#76902): the packaged Desktop app with no default project
- # dir configured resolves its cwd (and TERMINAL_CWD) to home, and the containment
- # check is then vacuously true for the whole home subtree — every AGENTS.md /
- # CLAUDE.md / .cursorrules under ~ would be injected on tool calls. Hint discovery
- # runs only inside a real project workspace; a home cwd only resumes when the
- # session adopts a real project (see ``rebind_working_dir``).
- self._home_is_working_dir = _is_home_like_working_dir(self.working_dir)
- # The working dir is pre-marked loaded (startup context handles it).
- self._loaded_dirs: set[Path] = {self.working_dir}
- # Content digests already injected: the same file reached through
- # symlinks/hardlinks/copies is never re-sent. Seeded with the CWD hint
- # file prompt_builder already loaded.
- self._loaded_digests: set[str] = set()
- found = _first_hint_file(self.working_dir)
- if found and found[1]:
- self._loaded_digests.add(_digest(found[1]))
- def rebind_working_dir(self, working_dir: str) -> None:
- """Re-anchor hint discovery to *working_dir* (a session workspace adoption)."""
- try:
- new_dir = Path(working_dir).expanduser().resolve()
- if not new_dir.is_dir() or _is_home_like_working_dir(new_dir):
- return
- except (OSError, ValueError, RuntimeError):
- return
- if new_dir == self.working_dir:
- self._home_is_working_dir = False
- return
- self.working_dir = new_dir
- self._home_is_working_dir = False
- # Fresh anchor, fresh bookkeeping: the new project's directories get their
- # own first-visit hints (startup never loaded context files for it).
- self._loaded_dirs = {self.working_dir}
- self._loaded_digests = set()
- found = _first_hint_file(self.working_dir)
- if found and found[1]:
- self._loaded_digests.add(_digest(found[1]))
- def check_tool_call(self, tool_name: str, tool_args: dict[str, Any]) -> Optional[str]:
- """Return formatted hint text for newly visited directories, or None."""
- from agent.safe_worker_policy import safe_worker_enabled
- if not self.enabled or safe_worker_enabled():
- return None
- if self._home_is_working_dir:
- return None
- all_hints = [h for d in self._extract_directories(tool_name, tool_args) if (h := self._load_hints_for_directory(d))]
- return "\n\n" + "\n\n".join(all_hints) if all_hints else None
- def _extract_directories(self, tool_name: str, args: dict[str, Any]) -> list:
- """Extract directory paths from tool call arguments."""
- candidates: set[Path] = set()
- for key in _PATH_ARG_KEYS:
- val = args.get(key)
- if isinstance(val, str) and val.strip():
- self._add_path_candidate(val, candidates)
- cmd = args.get("command", "") if tool_name in _COMMAND_TOOLS else None
- if isinstance(cmd, str):
- self._extract_paths_from_command(cmd, candidates)
- return list(candidates)
- def _add_path_candidate(self, raw_path: str, candidates: set[Path]):
- """Add a raw path's directory and its ancestors (up to ``_MAX_ANCESTOR_WALK``
- levels, stopping at the first already-loaded dir) so reading
- ``project/src/main.py`` still discovers ``project/AGENTS.md``."""
- try:
- p = Path(raw_path).expanduser()
- if not p.is_absolute():
- p = self.working_dir / p
- p = p.resolve()
- if p.suffix or (p.exists() and p.is_file()):
- p = p.parent
- for _ in range(_MAX_ANCESTOR_WALK):
- if p in self._loaded_dirs:
- break
- if self._is_valid_subdir(p):
- candidates.add(p)
- if p.parent == p:
- break # filesystem root
- p = p.parent
- except (OSError, ValueError, RuntimeError):
- pass
- def _extract_paths_from_command(self, cmd: str, candidates: set[Path]):
- """Extract path-like tokens (contain / or .; not flags or URLs) from a shell command."""
- try:
- tokens = shlex.split(cmd)
- except ValueError:
- tokens = cmd.split()
- # `cd backend && ls`: a bare directory name has no `/` or `.`, so the generic filter below drops
- # it; the operand of a navigation command is a path by construction (#11032). Only a `cd` at the
- # START of a shell segment counts (`echo cd backend` is prose); punctuation-aware tokenizing keeps
- # a quoted `'backend;'` literal while splitting bare `backend;ls` at the operator.
- for target in _nav_targets(cmd):
- self._add_path_candidate(target, candidates)
- for token in tokens:
- if token.startswith(("-", "http://", "https://", "git@")) or ("/" not in token and "." not in token):
- continue
- self._add_path_candidate(token, candidates)
- def _within_working_dir(self, path: Path) -> bool:
- """Reject paths outside the working-dir tree: loading ~/.codex/AGENTS.md
- or ~/.claude/CLAUDE.md would mix another agent's instructions into this
- session. Falls back to an ancestor check when ``is_relative_to`` fails."""
- try:
- return path.is_relative_to(self.working_dir)
- except (OSError, ValueError):
- try:
- path.relative_to(self.working_dir)
- return True
- except ValueError:
- return False
- def _is_valid_subdir(self, path: Path) -> bool:
- """Directory inside the working-dir tree, not yet loaded, not an excluded copy dir."""
- try:
- if not path.is_dir():
- return False
- except OSError:
- return False
- return path not in self._loaded_dirs and self._within_working_dir(path) and not self._is_excluded(path)
- def _is_excluded(self, path: Path) -> bool:
- """True when a segment *below* the working dir is an excluded copy dir
- (a user deliberately working inside ``vendor/`` keeps that segment legitimate)."""
- try:
- rel_parts = path.relative_to(self.working_dir).parts
- except ValueError:
- return True # outside the tree — already rejected upstream
- return any(part in _EXCLUDED_DIR_NAMES for part in rel_parts)
- def _load_hints_for_directory(self, directory: Path) -> Optional[str]:
- """Load the first hint file in *directory*; formatted text or None."""
- self._loaded_dirs.add(directory)
- if self._home_is_working_dir or not self._within_working_dir(directory):
- logger.debug("Skipping hint files in %s — outside working_dir %s", directory, self.working_dir)
- return None
- for filename in _HINT_FILENAMES:
- hint_path = directory / filename
- try:
- if not hint_path.is_file():
- continue
- except OSError:
- continue
- if (target := _resolved_hint_target(hint_path, self.working_dir)) is None:
- continue
- try:
- content = (_read_text_with_timeout(target) or "").strip()
- if not content:
- continue
- digest = _digest(content)
- if digest in self._loaded_digests:
- logger.debug("Skipping duplicate hint content at %s (digest %s)", hint_path, digest[:12])
- return None
- self._loaded_digests.add(digest)
- # Same security scan as startup context loading.
- content = _scan_context_content(content, filename)
- rel_path = self._display_path(hint_path)
- content = _truncate_content(
- content, filename, max_chars=_MAX_HINT_CHARS, read_path=rel_path, queue_warning=False,
- )
- logger.debug("Loaded subdirectory hints from %s: %s", directory, [rel_path])
- return f"[Subdirectory context discovered: {rel_path}]\n{content}" # first match wins per directory
- except Exception as exc:
- logger.debug("Could not read %s: %s", hint_path, exc)
- return None
- def _display_path(self, hint_path: Path) -> str:
- """Working-dir-relative, else ``~/``-relative (POSIX rendering so Windows
- never shows ``~/AppData\\Local\\...`` chimeras), else absolute."""
- try:
- return str(hint_path.relative_to(self.working_dir))
- except (ValueError, RuntimeError):
- pass
- try:
- return "~/" + hint_path.relative_to(Path.home()).as_posix()
- except (ValueError, RuntimeError):
- return str(hint_path)
|