The signal was a coverage gap at the edge of the project, not in the central test tree. RAPTOR already compiled and tested core and packages, but the libexec wrapper tests lived in a separate path. If wrappers are how operators reach lower-level functionality, leaving those tests outside CI makes the assurance boundary weaker than the codebase appears.
The useful lesson is that CI collection is a security-adjacent boundary. A regression test that is not collected by the default gate is closer to documentation than enforcement.
ci: run libexec wrapper tests (merged 2026-05-05 16:19 SGT)gadievron/raptor #308 changed .github/workflows/tests.yml so CI now:
libexec/tests in the Python compile gate;pytest core packages to avoid pytest import-name collisions with other top-level tests packages.The vault also captured the maintainer-facing communication rule from the same work: public PR bodies should summarize diligence naturally, name the concrete adjustment that matters to maintainers, and keep private review-pass bookkeeping out of the public description.
Wrapper and adapter tests often sit outside the obvious application test tree. That makes them easy to miss in CI even when they cover important operational paths: command wrappers, integration shims, plugin glue, runner scripts, or tool-facing entrypoints.
The reusable pattern is evidence drift between local validation and shared validation. A local command can prove the wrapper path once, but CI is what keeps the proof alive after merge. If combining the suite into a broader pytest invocation creates import collisions, the right fix is not to drop the suite. Run it as a separate evidence boundary with a clear name and explicit validation.
sys.path / PYTHONPATH — useful background for why test layout and import mode can change collection behavior when multiple top-level tests packages exist.Security review should ask whether the proof path is enforced by the project’s default automation. For wrappers, CLIs, tool bridges, MCP servers, upload processors, and background runners, the high-risk path may live outside the main package test command. If those tests are not in CI, the project can look covered while a boundary-specific regression is only checked manually.
The second lesson is communication discipline. Maintainers do not need a transcript of every internal review pass. They need the concrete risk found during checking, the design choice made because of it, and the validation that passed. In this case, the maintainer-relevant detail was that the wrapper tests should be a separate pytest step because combining them with the larger suite can create import-name collisions.
tests package collisions before merging separate test trees into one pytest command.09 - GitHub Activity/GitHub Follow-up Fixes/GitHub Follow-up Fix - gadievron - raptor - PR 308.md.06 - Lessons/Takeaway - PR descriptions should summarize diligence without internal audit phrasing.md.05 - Workflows/Workflow - GitHub Review Follow-up and Patch Loop.md and 05 - Workflows/Workflow - Source Code Vulnerability Discovery Loop.md.The signal was a small correction with a large workflow implication. A documentation/security-process PR can be directionally right and still reduce reviewer trust if it cites the wrong requirement ID, stale title, appendix name, or related-document mapping.
For standards-style AI security work, reference integrity is part of the evidence boundary. The control is only easy to verify when the reader can follow the exact requirement map back to the canonical source.
None in this window.
The vault ingested the outcome from OWASP/APTS #47, where maintainer review corrected an incorrect prompt-injection requirement reference and several mismatched requirement titles before merge.
That outcome was routed into the research system instead of staying as a PR-thread detail:
Checklist - Meaningful SECURITY.md Review now requires requirement IDs, standard titles, appendix names, and related-requirement maps to be checked against the canonical source before submitting documentation or policy PRs.Checklist Change - 2026-05-04 documentation reference verification records the checklist change and why no duplicate checklist was needed.Lesson - Cross-document requirement IDs need source-of-truth validation captures the review lesson.Takeaway - Maintainer reference corrections should become pre-submit checks turns the maintainer correction into a repeatable pre-submit gate.The reusable pattern is evidence-path drift. In code, the review follows attacker input through transforms into a sink. In standards and security-program documentation, the review follows a claim through requirement IDs, appendix links, related controls, and implementation guidance.
If those references drift, the failure is not a runtime exploit. It is a verification failure: future reviewers may land on the wrong control, miss the intended scope, or spend trust on a document that should have been mechanically checked first.
Documentation-heavy security work still needs a proof shape. The proof is not a PoC or regression test; it is the ability for a maintainer to trace every cited requirement and appendix name back to the canonical source without correction.
This changes the pre-submit loop. Before opening a standards, SECURITY.md, policy, checklist, or appendix PR, the review should include a canonical-reference pass alongside link checks and Markdown validation. If a maintainer corrects an ID or title, the right response is not just to fix the typo. The correction should become a durable checklist item, because it exposed a review boundary that was too loose.
OWASP/APTS #47 maintainer feedback and merge record.06 - Lessons/Lesson - Cross-document requirement IDs need source-of-truth validation.md.06 - Lessons/Takeaway - Maintainer reference corrections should become pre-submit checks.md.05 - Workflows/Checklist - Meaningful SECURITY.md Review.md, 05 - Workflows/Checklist Change - 2026-05-04 documentation reference verification.md, and 05 - Workflows/Checklist Change Log.md.The signal was not a shipped patch. It was a review-quality boundary. Broad AI agents can generate many plausible vulnerability candidates, but the candidate only becomes useful when it states the attacker condition, server condition, concrete impact, security-policy fit, and proof status in a form that a skeptical reviewer can reject or reproduce.
None in this window.
The vault ingested Hyunseo Shin’s CyKor writeup on using an LLM multi-agent workflow to find open-source 0-days. The raw source note, advisory-case synthesis, and takeaway note were added to the research system, then folded back into existing workflows instead of creating a parallel checklist.
The source-code discovery workflow now treats candidate quality as an explicit contract: attacker control, server/environment prerequisite, security impact, project policy fit, and proof status must be written down before escalation. The quick-pass checklist gained the same false-positive gate. The authz checklist gained a sharper object-scope question: does the permission engine evaluate the exact target resource, tenant, workspace, or UID, or only a generic role/action?
The reusable pattern is a funnel, not a monolithic agent. Cheap discovery can stay broad and noisy. Semi-triage should kill candidates that cannot name the attacker, environment, impact, policy fit, or proof. Final verification should spend expensive model and human attention only on candidates that survived that contract.
For authorization review, the key invariant is scope binding. A check that proves “the actor has this action” is not enough when the dangerous operation affects a specific object. The review question has to follow the target identifier into the permission engine and confirm that the exact object or tenant scope is part of the decision.
The important part of LLM-assisted review is not that a model can point at suspicious code. It is whether the workflow makes the model produce a claim that can be tested. A candidate without attacker control, a reachable sink, a realistic server condition, and a concrete impact is only review noise. The false-positive gate should be structural, not a reminder buried in the prompt.
This also changes how authz candidates should be triaged. Generic permission checks are not inherently wrong, but they are incomplete evidence when the operation mutates or reveals a specific resource. The reviewer needs to compare route intent, action permission, object identifier, service-layer enforcement, storage namespace, and documented security model before claiming IDOR or privilege escalation.
SECURITY.md, advisory scope, and trust-model language before severity claims, especially when a behavior may be trusted-operator, admin-only, or intentionally out of scope.07 - Sources/Blog Posts/Source - CyKor - LLM multi-agent workflow for open-source 0-days.md.08 - Advisory Cases/Case - Tiered LLM multi-agent workflow for open-source 0-days.md.06 - Lessons/Takeaway - LLM discovery candidates need explicit attacker server impact contracts.md.05 - Workflows/Workflow - Source Code Vulnerability Discovery Loop.md, 05 - Workflows/Checklist - Source Code Discovery Quick Pass.md, and 05 - Workflows/Checklist - Authz Coverage Review.md now carry the reusable false-positive and object-scope gates.The useful signal was that agent and OSS hardening are not only about finding the bug. They are also about preventing quiet authority transfer: upload bytes redirected through a symlink, findings exported without stable structure, coverage assumed without a threshold, or autonomy changes handled without a prewritten downgrade record.
DeerFlow hardened upload and inbound attachment writes. Normal filename cleanup was not enough because the final destination could already be a symlink inside a writable thread uploads directory. The merged fix routes HTTP uploads and channel attachment ingestion through a shared no-symlink writer, rejects unsafe pre-existing destination entries, uses O_NOFOLLOW where available, skips unsafe destinations, and adds regressions for both HTTP and channel file paths.
RAPTOR added grouped Markdown findings export. Project reports now generate a findings/ directory with a project-level Markdown report, per-finding Markdown and JSON artifacts grouped by validation state, a manifest, and JSONL output. That turns a run into a more stable handoff object: confirmed findings, needs-review items, and ruled-out items no longer collapse into one machine-only blob.
RAPTOR also added a coverage threshold gate. raptor project coverage --fail-under <pct> computes review-item coverage from the existing summary, prints a pass/fail line, and exits non-zero when the configured floor is missed. That makes incomplete review coverage a CI/local workflow failure instead of an informal caveat.
APTS added an autonomy downgrade matrix template. It is informative, not normative, but it gives teams a concrete place to define downgrade triggers, temporary autonomy caps, approval paths, evidence preservation, incident-response activation, and re-authorization conditions before an incident or drift event makes the decision messy.
The common pattern was sink-side proof. The dangerous sink may be a filesystem write, but it can also be a report handoff, a CI gate, or an autonomy decision. In each case, the weak version depends on intent: “this filename was normalized,” “the reviewer probably covered enough,” “the findings are somewhere in the run output,” or “operators will know when to downgrade.” The stronger version makes the final authority point prove the invariant before it acts.
For uploads, the invariant belongs at the final open/write operation, not only at the string-normalization layer. For review tooling, coverage and findings need first-class artifacts that can fail, be linked, and be audited. For autonomy governance, downgrade criteria need to be written before the system is under pressure.
The DeerFlow fix reinforces that upload roots shared with sandbox-controlled or channel-controlled state must be reviewed as hostile storage, not as ordinary application folders. If the backend writes into that namespace, the final path component has to be checked as a filesystem object. A clean basename does not prove the destination is safe when a symlink, directory, special file, or shared inode can already exist there.
The RAPTOR changes sharpened the evidence side of the same review loop. A finding is easier to trust when its validation state, severity grouping, machine-readable record, and coverage floor are explicit. The coverage gate is especially useful because it turns “review breadth” into something automation can reject. That does not prove a target is safe, but it prevents a partial run from being presented as complete without friction.
The APTS matrix is a reminder that agent autonomy needs precommitted downgrade paths. Prompt-injection signals, connector overreach, model drift, audit gaps, and incomplete handoffs are easier to handle when the organization has already defined the cap, approver, evidence to preserve, and condition for re-authorization.
open()/write for upload destinations, CI exit status for coverage, generated artifacts for findings, and written matrices for autonomy downgrades.The useful AI-security pattern was sink-side authority. In agent, bot, and container workflows, the first untrusted object often looks ordinary: a URL, a filename, an outbox row, or a handoff field. The security question is where that object later gains file, network, or operator authority.
NanoClaw hardened the host/container outbox boundary. Container-owned outbound rows and files were previously reused by the host as path components for attachment reads and recursive cleanup. The fix validates message ids and filenames as simple path segments, rejects symlinks with lstat(), requires realpath() containment under the intended outbox directories, and adds regression coverage for traversal read, symlink read, escaped cleanup, and normal basename behavior.
Nanobot hardened DingTalk remote media fetching. The vulnerable path let a permitted control source or prompt-injection route supply a remote media URL, have the nanobot host fetch it, follow redirects, and upload the response bytes as DingTalk media. The fix validates the initial URL, refuses redirects by default, adds explicit same-host/cross-host redirect opt-ins, validates every redirect hop and final URL, caps remote media bytes, and covers private targets, private redirects, redirect policy, allowlists, and oversized responses in tests.
APTS added a shift handoff template appendix. It is informative rather than normative, but it still tightens the review surface: pending approvals, active scope, kill-switch authority, safety signals, connector state, and incoming-operator acceptance now have a concrete record format instead of living as vague process intent.
AI and automation systems often split admission from action. A caller may validate a URL, path, or control record early, but the later sink is where authority becomes real. Review should therefore follow the object until the privileged primitive actually acts: HTTP client, filesystem call, upload endpoint, browser action, process spawn, or human approval record.
The repeat lesson is that the sink owns the final boundary. Container-side discipline helps, but the host process must treat container-written outbox state as hostile when it performs file I/O. URL admission helps, but the channel-specific fetcher must validate the URL and every redirect target before accepting bytes. Human-oversight requirements help, but shift handoff only becomes reviewable when the artifact captures who accepted which authority, scope, and safety state.
The trade-off in both security fixes was compatibility without silent trust expansion. NanoClaw kept normal basename attachments working while rejecting path-like and symlinked inputs. Nanobot kept direct remote media support and gave operators an explicit redirect path, but made cross-host redirects allowlist-driven and kept private targets blocked. That is the shape I want: a narrow default, a named compatibility escape hatch, and tests that preserve the distinction.
attacker-controlled input -> transformation -> sink -> boundary, then place the strongest validation at the sink.signal -> observed pattern -> external reference -> takeaway -> repeat next time.DeerFlow hardened the legacy local Docker sandbox path. Before the fix, the launcher built Docker port mappings in the bare form -p <host_port>:8080, which Docker treats as a bind on all host interfaces. That can expose a localhost-oriented sandbox HTTP API to adjacent hosts when a developer, demo, or self-hosted machine is reachable on a LAN or shared network.
The merged fix binds localhost-compatible sandbox runs to 127.0.0.1:<port>:8080 by default, keeps the broader bind for Docker-outside-of-Docker style deployments such as host.docker.internal, and adds DEER_FLOW_SANDBOX_BIND_HOST for explicit operator override. It also documents the behavior and adds regression coverage for the default Docker path, the compatibility branch, explicit override, and Apple Container port formatting.
The useful part was not only “bind to loopback.” The lesson is that locality is an invariant, not a comment. If the product-level assumption is “this sandbox API is local/internal,” then the command-construction layer has to encode that assumption in the primitive that actually opens the socket. Docker’s default is operationally convenient, but security review has to treat that default as a boundary decision.
The trade-off was compatibility. A strict loopback-only change would have been cleaner, but it would also risk breaking deployments that intentionally need host-wide reachability from a containerized DeerFlow process. The better patch shape was secure by default for localhost/bare-metal runs, compatibility for explicit non-loopback sandbox hosts, and an override that makes the operator’s choice visible.
docker -p host:container is not neutral when the omitted bind host widens exposure.OpenHarness gained a gateway-level regression test for the remote /bridge spawn shell-execution boundary fixed in #208. The new test resolves /bridge from the real create_default_command_registry(), sends a concrete marker-file payload through OhmoSessionRuntimePool.stream_message(), and asserts the gateway returns the local-UI-only denial before a bridge session or marker file can be created.
The changed file was narrow: tests/test_ohmo/test_gateway.py. No runtime behavior changed beyond the earlier fix. The value of the PR is proof shape: it locks the security boundary to the path that originally made the issue exploitable, not to a synthetic command object that could pass while the real registry drifted.
The vault review loop keeps returning to the same discipline: map the actual surface before trusting the fix. A regression that tests only the convenient abstraction can become a false comfort when the vulnerable behavior depended on surrounding machinery. Here, the dangerous path was remote message -> slash-command parser -> default registry -> bridge handler -> shell subprocess. The test now exercises enough of that path to make future regressions harder to hide.
This is a useful boundary lesson. Security tests should not merely assert that a flag looks correct. They should prove the sensitive sink is unreachable through the realistic caller path, and they should check for side effects that would appear if denial happened too late. In this case, both conditions matter: no bridge session and no marker file.
OpenHarness tightened three separate local-control boundaries. Plugin uninstall now treats the plugin argument as an identifier, rejects traversal, nested, absolute, empty, and backslash-containing names, and requires the resolved deletion target to be a direct child of the user plugin directory before rmtree() is reached. The Feishu/Lark channel now treats inbound attachment filenames as remote metadata, sanitizes them before saving media, and verifies the resolved write path remains under the channel media directory. The /bridge command is now local-only by default, with a trusted-operator opt-in marker for deployments that intentionally expose it; remote gateway messages are denied before the bridge handler can spawn a shell session.
FastGPT moved MCP URL validation into both persistence and execution paths. The existing direct preview/run guard already rejected internal addresses, but stored MCP tool create/update and workflow execution could drift away from that policy. A shared guard now rejects internal MCP endpoints before storage and revalidates stored URLs before the backend runner connects to them.
RAPTOR hardened Mermaid diagram generation. The sanitizer now handles additional label and ID edge cases, normalizes more line separators, preserves collision resistance for sanitized IDs, and applies shared sanitization across context maps, flow traces, hypotheses, attack paths, attack trees, findings summaries, class assignments, subgraphs, and pie labels. Regression tests cover callback-shaped payloads, quoted-label breakouts, subgraph IDs, class assignments, and pie labels.
OWASP/APTS added an informative Authority Delegation Matrix template and linked it through the human-oversight guidance, appendix index, Getting Started map, and Rules of Engagement template. The artifact does not change requirements; it makes approval authority, escalation paths, emergency authority, and review history easier to compare against decision records and handoffs.
The useful review move was the same one from the vault loop: name the trust boundary, then follow the value until it reaches the operation that matters. The bugs were not just path traversal, SSRF, diagram escaping, or command exposure in isolation. They were places where a weakly typed value crossed into deletion, file write, network connection, renderer syntax, shell spawn, or approval authority without being forced back into the narrow form that sink actually accepts.
The strongest fixes did not rely on a caller remembering context. They put the constraint at the boundary that performs the dangerous action: direct-child validation before deletion, resolved-path containment before attachment writes, remote-invocation metadata before command dispatch, MCP URL validation before storage and before workflow execution, and shared Mermaid sanitization before diagram emission. The documentation PR follows the same pattern in a non-runtime form: approval authority should not be implied by scattered prose when an auditable matrix can make the delegated capability explicit.
The broader rule is capability discipline. Local-only features should stay local by default. Stored configuration should be rechecked when used, not trusted because an earlier interactive route had validation. Renderer output should be treated as a syntax sink, not inert text. Governance artifacts should make authority boundaries visible enough that reviewers can compare them against logs, handoffs, and risk levels.
value -> transformation -> sink -> capability map before deciding severity or patch shape.RAPTOR’s WebClient now normalizes the configured target origin, rejects absolute or protocol-relative request URLs that leave that origin, and handles redirects manually with allow_redirects=False. Same-origin redirects still work. Cross-origin redirects are blocked before the scanner sends a request to the redirected host, which also prevents configured cookies from leaking to an off-scope sink. Focused tests cover direct scope checks, redirect scope checks, and cookie non-leakage.
OpenViking’s image generation tool now routes local image reads through the session sandbox instead of calling Path(...).read_bytes() on host paths. The patch passes ToolContext into edit and variation parsing paths, adds a binary-safe read_file_bytes() sandbox API, enforces the existing direct-backend path restrictions before byte reads, updates schema text to say local image paths are sandbox-local, and adds regression tests for denied host paths plus supported data: and HTTP(S) inputs.
The vault review loop was the useful constraint here: map the input, transformation, sink, and trust boundary before deep-reading the patch. Both bugs were easy to understate if viewed as isolated parser details. In OpenViking, a string that looked like an image path became a host filesystem read and then outbound provider input. In RAPTOR, a URL that began inside the authorized target could become an off-scope network request after requests followed a redirect.
The clean fix in both cases was not to add a distant policy note and hope callers remember it. The boundary moved to the layer that performs the dangerous operation: sandboxed byte reads at the image-file sink, and target-origin checks in the HTTP client before direct requests or redirected requests leave scope. That keeps future callers covered even when they bypass higher-level discovery logic.
The same-day vault lesson on debug escape hatches also applies indirectly. Exceptions and convenience paths must be capability-scoped. A local image-path feature should not become arbitrary host-file read. A scanner redirect feature should not become off-scope traffic. A reveal/debug flag should not disable unrelated sanitizers. The repeatable rule is narrower than “be careful with inputs”: every escape hatch needs a named capability, a safe default, and tests proving it does not widen adjacent boundaries.
input -> transform -> sink -> boundary map before deciding whether a bug is only parsing, only routing, or only documentation.RAPTOR now redacts common secrets from web scanner request history, fuzzer finding URLs, and LLM/provider log sanitization by default. Operators can still reveal exact values for local debugging, but only through an explicit opt-in path: environment configuration or the web scanner flag. The change also adds centralized redaction logic and focused tests across the web and LLM paths.
APTS gained a set of non-normative review aids: a quick vendor review checklist, a completed evidence package manifest example, a completed conformance claim example, and a reader-path flowchart. These do not change the standard’s requirements. They reduce ambiguity around how a vendor, buyer, reviewer, or contributor should move through evidence, scope, traceability, and decision records.
Documentation can be a security boundary when it controls interpretation. The APTS PRs were useful because they made evidence shape visible without adding hidden requirements. A completed manifest example shows what provenance, redaction, custody, exports, and customer review questions look like together. A conformance claim example shows how scope and requirement-level traceability should be stated without implying certification. The quick checklist gives a short triage path before a team spends time on a full review.
The RAPTOR change carried the same lesson in executable form. Secret redaction is not just a logging nicety; it is an operational default. If debugging needs raw credentials, the exception should be deliberate, named, and test-covered. Otherwise the tool slowly trains users to preserve secrets in artifacts because it is convenient.
The vault’s review loop reinforced the constraint: keep claims narrower than the evidence. For code, that means proving the exact sink and boundary. For standards work, it means separating informative examples from normative requirements. For redaction, it means testing both sides of the operator escape hatch instead of assuming the safe default will survive future debugging pressure.