Skip to content

Fix recurring agent tool failures: proxy timeouts, report identifiers, page fetch errors, skill names - #1780

Open
bearsyankees wants to merge 6 commits into
mainfrom
fix/str-2710-harness-tool-failures
Open

bearsyankees wants to merge 6 commits into
mainfrom
fix/str-2710-harness-tool-failures

Conversation

@bearsyankees

@bearsyankees bearsyankees commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Fixes tool failures that recur across scans.

  • Proxy tools. The Caido SDK's GraphQL layer aborts each operation after 10 seconds with an exception that has no message, so agents saw list_requests failed: Source:. The timeout is now set explicitly (and reapplied after connect(), which rebuilds the GraphQL client). list_requests uses a metadata-only query and no longer downloads the raw request and response of every row. A timeout is reported as a timeout. Read-only calls are sent once more after a timeout, with a 20-second limit per attempt so both attempts fit inside the tool deadline; the default limit is restored after each call. Calls that send traffic (repeat_request) and invalid HTTPQL filters are not retried.
  • Reports. Placeholder CVE values (null, N/A, -) are treated as absent, and a CWE id in the CVE field moves to the CWE field when that field is empty.
  • web_get_contents. When no page is fetched, the error gives the per-URL reason and no longer says "Try again". One retry when the failure may be transient; both attempts share one time budget that fits inside the tool deadline.
  • Skills. A wrong category (technologies/nextjs) resolves when the name itself is an exact, unambiguous skill name. A misspelled name (tooling/nuklei) is still rejected, and the error now suggests the closest names.
  • stop_agent. Stopping an agent that has already finished is a successful no-op.

Tests are in tests/test_harness_tool_failures.py.

Link to Devin session: https://app.devin.ai/sessions/12149bec39f84d02a389080ebc42b1fa
Open in Devin Desktop: https://app.devin.ai/desktop/session/12149bec39f84d02a389080ebc42b1fa?variant=devin
Requested by: @bearsyankees

…, page fetch errors, skill names

- Proxy tools: raise the SDK's 10s GraphQL operation timeout to 60s, list
  requests without raw bodies, and report timeouts instead of an empty error.
- Reports: drop placeholder CVE values and move a CWE id out of the CVE field.
- web_get_contents: return the per-URL failure reason and retry once only
  when the failure may be transient.
- Skills: resolve a wrong category or a typo when exactly one skill matches.
- stop_agent: an already finished agent is a successful no-op.
… timeout after connect

- A request to the proxy now and then gets no answer while the same request
  succeeds right afterwards, so read-only calls are sent once more. Calls
  that send traffic are not repeated.
- The SDK rebuilds its GraphQL client on connect, which reset the timeout.
- A rejected query (invalid HTTPQL) is reported once, not twice.
@greptile-apps

greptile-apps Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge with the two previously reported, non-blocking retry deadline issues still open.

Findings

  1. P2 Proxy retries lose their window. ▶
  2. P2 Page retries exceed the deadline. ▶
Fix with agent prompt
### Issue 1
strix/tools/proxy/tools.py:79-84
`_call` now gives each attempt 60 seconds, but `view_request`, `list_sitemap`, `view_sitemap_entry`, and `scope_rules` still have 60-second tool deadlines. When a request stalls until the new timeout, the tool deadline arrives before the retry can finish, bypassing the intended recovery and clearer error. Give these reads a shorter per-attempt timeout or increase the outer deadline enough for both attempts and time spent waiting for the shared lock.

### Issue 2
strix/tools/web_search/tool.py:175-176
`_exa_page_text` can now make two calls, each with a 300-second HTTP timeout, while `web_get_contents` still stops waiting after 330 seconds. A slow empty response followed by a slow successful retry can therefore lose its result. Because the work runs through `asyncio.to_thread`, that HTTP call can also keep running after the tool returns a timeout. Bound both attempts to one total time budget that fits inside the tool deadline.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

This PR makes scan tools more forgiving of common input mistakes and improves recovery from slow external services.

  • Caido reads wait longer and request lists skip raw traffic data.
  • Skill requests can recover from a wrong category or clear typo.
  • Stopping an inactive agent now returns a successful no-op.
  • Finding reports accept missing CVEs and misplaced CWE identifiers.
  • Exa page fetches retry uncertain empty results and explain permanent failures.

No new findings were established. Both previous findings remain outstanding: proxy retries and page-fetch retries can exceed their tools’ deadlines.

Reviews (2) · Last reviewed commit: "Retry read-only proxy calls once after a..." · Reviewed by Greptile

Comment on lines +79 to +84
attempts = 2 if read_only else 1
for attempt in range(1, attempts + 1):
async with _CAIDO_CALL_LOCK:
caido_api.extend_graphql_timeout(client, timeout)
try:
return await fn(client)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Proxy retries lose their window. _call now gives each attempt 60 seconds, but view_request, list_sitemap, view_sitemap_entry, and scope_rules still have 60-second tool deadlines. When a request stalls until the new timeout, the tool deadline arrives before the retry can finish, bypassing the intended recovery and clearer error. Give these reads a shorter per-attempt timeout or increase the outer deadline enough for both attempts and time spent waiting for the shared lock.

Prompt To Fix With AI
This is a comment left during a code review.
Path: strix/tools/proxy/tools.py
Line: 79-84

Comment:
**Proxy retries lose their window.** `_call` now gives each attempt 60 seconds, but `view_request`, `list_sitemap`, `view_sitemap_entry`, and `scope_rules` still have 60-second tool deadlines. When a request stalls until the new timeout, the tool deadline arrives before the retry can finish, bypassing the intended recovery and clearer error. Give these reads a shorter per-attempt timeout or increase the outer deadline enough for both attempts and time spent waiting for the shared lock.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't block merge. When a read-only proxy call stalls until its 60-second GraphQL timeout, the tool's 60-second deadline can end the call before _call retries or returns its clearer error. The caller sees a timeout and can request the read again; the failed read leaves proxy records unchanged. The affected tools have 60-second deadlines and pass read_only=True to _call.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed—this is a resilience gap, not a merge blocker. The 60-second tool deadline can still preempt _call's retry for the affected reads, but a timed-out read leaves proxy records unchanged and can be requested again. I’d keep this as a non-blocking follow-up rather than require a change for this PR.

Comment thread strix/tools/web_search/tool.py Outdated
Comment on lines +175 to +176
for _ in range(2):
body = _exa_post(api_key, "https://api.exa.ai/contents", {"urls": urls, "text": True})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Page retries exceed the deadline. _exa_page_text can now make two calls, each with a 300-second HTTP timeout, while web_get_contents still stops waiting after 330 seconds. A slow empty response followed by a slow successful retry can therefore lose its result. Because the work runs through asyncio.to_thread, that HTTP call can also keep running after the tool returns a timeout. Bound both attempts to one total time budget that fits inside the tool deadline.

Prompt To Fix With AI
This is a comment left during a code review.
Path: strix/tools/web_search/tool.py
Line: 175-176

Comment:
**Page retries exceed the deadline.** `_exa_page_text` can now make two calls, each with a 300-second HTTP timeout, while `web_get_contents` still stops waiting after 330 seconds. A slow empty response followed by a slow successful retry can therefore lose its result. Because the work runs through `asyncio.to_thread`, that HTTP call can also keep running after the tool returns a timeout. Bound both attempts to one total time budget that fits inside the tool deadline.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't block merge. When the first page fetch returns no content slowly and the second succeeds after the 330-second tool deadline, the caller gets a timeout instead of the fetched text. _exa_page_text allows two HTTP calls of up to 300 seconds each, while web_get_contents runs them in a thread under that deadline. The timeout is visible, so the caller can request the pages again; the lost result does not alter saved work.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed—since the timeout is visible and the caller can retry without losing saved work, this doesn’t block merge. The deadline mismatch remains a retryable reliability issue, but I’ll withdraw the blocker framing.

@arcanist

arcanist Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Ready to merge Re-run Arcanist review

PR summary

Proxy request listings used to download raw bodies and could show an empty timeout error. They now fetch metadata only, retry read-only calls once with bounded attempts, and describe proxy failures more clearly. Page fetches used to return a generic failure when no content arrived. They now retry once within a shared time budget and report the reason for each failed URL. Reports, skill loading, and agent stopping used to reject placeholder identifiers, wrong-category skill names, and no-op stops. They now normalize those identifiers and skill names, reject misspelled skills with suggestions, and treat an already-finished stop as successful.

🟡 Medium blast radius.

Review history (3)
  • Ready to merge
  • Ready to merge
  • Ready to merge (this review)
Flow diagram
%%{init: {"theme": "neutral"}}%%
flowchart TD
  n0["Page tool"]
  n1["Fetch the requested pages"]
  n0 --> n1
  n2["Request page text"]
  n1 --> n2
  n3["Receive the response"]
  n2 --> n3
  n4{"Keep this response"}
  n3 --> n4
  n5{"Time remains for a retry"}
  n4 -->|"Retry a temporary failure"| n5
  n6["Try the page fetch once more"]
  n5 -->|"Yes"| n6
  n7["Receive the retry response"]
  n6 --> n7
  n8["No usable text raises a typed error"]
  n4 -->|"Yes"| n8
  n7 --> n8
  n5 -->|"Skip the retry"| n8
Loading
✅ Ruled out (5)
  • Can another proxy call break a retry by using the same connection?
    Both calls succeeded and the server saw three requests, so the shared connection remained usable throughout the retry.
  • Do request listings lose their timestamp timezone?
    Both produced the same local timestamp string, so the new metadata-only listing does not change timestamp rendering.
  • Can a qualified custom skill name load the packaged skill instead?
    The qualified custom name stayed unchanged and loaded the custom body, while the packaged same-named skill was not substituted.
  • Can a misspelled bare skill name silently load another skill?
    A bare name absent from both registry forms remains invalid and receives an error, so it is not replaced with a suggested skill.
  • Does stopping a finished agent hide the need to stop active descendants?
    The response still tells the caller to inspect the agent graph and stop active descendants individually, so that instruction is not removed.

@devin-ai-integration devin-ai-integration Bot changed the title Fix/str 2710 harness tool failures Fix recurring agent tool failures: proxy timeouts, report identifiers, page fetch errors, skill names Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant