Repository navigation
feat(runner): pin a target to an ip with a JSON input line - #2691
Conversation
A JSON input line pins its target to an ip, in the format of nuclei
target lines, so one list feeds both tools:
{"target": "https://app.example.com", "ip": "192.0.2.10"}
Like curl --resolve, but on every port of the host, connections to the
hostname go to the ip, redirects included, while the hostname stays in
the Host header and SNI and is not resolved. Each target and ip pair is
probed separately, and host_ip and the output show the ip.
An input line is parsed once, where it enters probing, and the code
after it uses the parsed target and ip. One value in the context of a
request describes its host: the ip it is pinned to and the ip it is
dialed to. -pa pins its ips the same way, instead of the fastdialer.IP
value, which sent every dial of a request to the ip, redirects to other
hosts and the proxy included.
Options that do not connect through the ip stop the run when a pinned
line is read, -pa is refused with -proxy, and a proxy from the
environment fails the request, reported once with a hint to use
NO_PROXY. The ips of a host are kept apart for -maxhr, -sr and -e cdn.
An invalid line is reported and skipped.
Closes projectdiscovery#2667
|
No actionable comments were generated in the recent review.
 |
| Layer / File(s) | Summary |
|---|---|
Target-line parsing and input runner/targetline.go, runner/runner.go, runner/targetline_test.go, README.md |
Input handling parses and validates JSON target lines, canonicalizes pinned entries, and applies option checks. Direct, file, and streamed inputs use this handling. Tests cover parsing, validation, and input loading; the README documents the input format and constraints. |
Pinned host request context and transport common/httpx/requesthost.go, common/httpx/httpx.go, common/httpx/tls.go, common/httpx/requesthost_test.go, runner/main_test.go |
Request contexts retain the hostname and optional pinned IP. The primary transport records matching dialed IPs and rejects proxy use for pinned hosts. TLS-grab requests use the pinned context. Tests cover context, proxy, and dial behavior. |
Pinned request execution and results runner/runner.go, runner/targetline_test.go |
The runner carries pinned targets through probing, redirects, error tracking, output, response storage, and supporting checks. Integration tests cover pinned IPs, DNS behavior, SNI, and related request paths. |
Priority:
Estimated code review effort: 4 (Complex) | ~60 minutes
Change: Feature
Sequence Diagram(s)
sequenceDiagram
participant Input
participant readInputLine
participant Runner
participant HTTPXTransport
participant PinnedEndpoint
Input->>readInputLine: JSON target and IP
readInputLine->>Runner: validated targetLine
Runner->>HTTPXTransport: request with pinned host context
HTTPXTransport->>PinnedEndpoint: dial pinned IP with hostname identity
Possibly related PRs
- projectdiscovery/httpx#2668: Implements the same JSON target-line pinning behavior; this PR continues and refines that implementation.
- projectdiscovery/httpx#2688: Updates
fastdialerandtlsxversions used by the pinning implementation.
Suggested reviewers: mzack9999
Merge Risk: 
a7248
-
No actionable merge-blocking issue remains in the reviewed change; pinned requests retain the intended IP and hostname behavior.
Pre-merge checks: 
-
Check name Status Explanation Resolution Out of Scope Changes check The AI summary of runner/runner.go describes a change that is not in the PR description and does not relate to pinned ips or issue #2667. Under automatic HTTP/HTTPS selection, a 400 Bad Request from t…Remove the 400 Bad Request to HTTPS retry from runner/runner.go and its tests, or move it to a separate pull request with its own issue. Either explain how exclusive response-file creation with numbered suffixes is required for per-ip stora… -
Check name Status Explanation Description Check Check skipped - CodeRabbit’s high-level summary is enabled. Title check The title clearly and concisely describes the main change: allowing runner targets to pin a hostname to an IP through a JSON input line. Linked Issues check Issue #2667requires a JSON input line that pins a hostname to an ip. The connection must go to the ip. The hostname must stay in the Host header and SNI. Redirects must follow the pin. IPv6, -pa, fav…Docstring Coverage Docstring coverage is 82.35% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentati… Full details: Linked Issues check
-
Explanation
Issue
#2667requires a JSON input line that pins a hostname to an ip. The connection must go to the ip. The hostname must stay in the Host header and SNI. Redirects must follow the pin. IPv6, -pa, favicon, ztls, and per-ip -maxhr behavior are also in scope.Evidence at head:
- runner/runner.go: pinned lines carry the ip as targetLine.IP. targetsOf emits httpx.Target{Host, CustomIP} without DNS. Pinned targets skip getDNSData, so a, aaaa, and cname stay empty.
- runner/runner.go analyze: httpx.WithHost(ctx, URL.Hostname(), target.CustomIP) pins the request. The URL keeps the hostname, so Host and SNI stay the hostname.
- runner/runner.go: pinnedKey(...) keys HostErrorsCache and response files per ip. Failures and -sr output are therefore per ip.
- runner/runner.go: HandleFaviconHash clones with httpx.PinnedContext(req.Request). The ztls path uses PinnedContext per the summary.
- runner/runner.go: host_ip uses httpx.DialedIP(req.Context()), with a fallback to the dialer.
- go.mod requires github.com/projectdiscovery/fastdialer v0.6.0, which the PR says provides WithResolveOverride. This matches the issue's dependency note.
- The PR description reports tests and a before/after table that cover pinned redirects, -pa per-ip probing, and IPv6 (skipped without IPv6). I did not run these tests.
Limits: the runner.go diff output was truncated at 16000 bytes. I did not read common/httpx/requesthost.go, runner/targetline.go, or the test files directly. Their behavior rests on the AI summary and the PR description. The requirement mapping is consistent with the visible runner.go hunks.
Full details: Docstring Coverage
-
Explanation
Docstring coverage is 82.35% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 34 functions across 8 files. (1 skipped: 1 unsupported.)
-
Full details: Out of Scope Changes check
-
Explanation
The AI summary of runner/runner.go describes a change that is not in the PR description and does not relate to pinned ips or issue
#2667. Under automatic HTTP/HTTPS selection, a 400 Bad Request from the HTTP attempt triggers one HTTPS attempt. If that attempt fails, the original HTTP response is restored. This changes protocol detection for all targets, not only pinned targets. The truncated runner.go diff did not show this hunk directly, so the finding rests on the summary. The summary is specific and matches the visible runner.go hunks.A second change also lacks a stated link to the issue. The summary says response files are now created exclusively. If the base filename exists, numbered suffixes are tried. This affects -sr storage for every target. The PR description says only that per-ip -sr file names change under -pa. It does not explain why exclusive creation is needed for pinned targets.
The other changes trace to the issue or the PR description. These include target-line parsing, host pinning, -pa per-ip behavior, option conflicts, host_ip, -maxhr and -sr keys, -e cdn, favicon and ztls pinning, and the test helpers.
Resolution
Remove the 400 Bad Request to HTTPS retry from runner/runner.go and its tests, or move it to a separate pull request with its own issue. Either explain how exclusive response-file creation with numbered suffixes is required for per-ip storage, and describe it in the PR description, or remove it from this pull request. Confirm that no other unlinked behavior change remains, then re-review the affected hunks.
-
Finishing Touches
-
🧪 Generate unit tests (beta)
-
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
A rabbit pins a hostname tight,
Then hops to where its IP might.
The Host and SNI stay in view,
While requests follow the address true.
Each result keeps the pin in sight,
And carrots mark a tested flight.
Comment @coderabbitai help to get the list of available commands.
Proposed changes
Probing a vhost on a given server, or an origin behind a CDN, needs the hostname in the Host header and SNI while the connection goes to another ip, and one host often has to be probed on several ips. httpx had no input for that: a
vhost,ipline builds the url from the ip and sends no SNI, and-paonly probes the ips DNS returns.A JSON input line can now pin its target to an ip, in the format of nuclei target lines (projectdiscovery/nuclei#7863), so one list feeds both tools:
Connections to the hostname go to the ip on every port, redirects included, like
curl --resolvebut not tied to one port. The hostname stays in the Host header and SNI and is not resolved. Each target and ip pair is probed separately, andhost_ipand the output show the ip.fastdialer.WithResolveOverridecomes from fastdialer v0.6.0, whichdevalready requires, sogo.modis unchanged.runner/targetline.goparses a line once, where it enters probing, andprocess,targetsOfandanalyzetake the parsed target and ip. Onlytargetandipare read; the keys nuclei uses to select templates are ignored. Any other key, a null or empty ip, or an IP, CIDR, ASN, wildcard orvhost,iptarget makes the line invalid, so a typo can't send the probe through DNS. An invalid line is reported and skipped. A line withoutipis the plain target, and equivalent lines are probed once.common/httpx/requesthost.gokeeps the host of a request in its context: the ip it is pinned to and the ip it was dialed to, whichhost_ipnow reports. Before,host_ipwas the last ip the dialer had dialed to the host for any request, which a pinned target of the same host changes.-papins its ips the same way. It used thefastdialer.IPvalue, which sent every dial of the request to the ip, redirects to other hosts and the proxy included, and failed for an IPv6 ip. The favicon request and the ztls handshake keep the pin.-proxy,-unsafe,-pipeline,-http2,-jarm,-screenshotand-vhost-input, which don't connect through the ip, and with-pr http2, which shares a connection between the ips of a host.-pais refused with-proxy. A proxy fromHTTP_PROXY/HTTPS_PROXYresolves the host itself, so a pinned request through it fails, and the first failure is reported with a hint to add the host toNO_PROXY.-maxhrcounts failures and-srnames stored responses per ip,-e cdnchecks the pinned ip, and a pinned line sends no DNS query, so itsa,aaaaandcnamestay empty.Without target lines, these change:
-pa -frresolves a redirect to another host instead of sending it to the probed ip.-pa -proxyis refused, and withHTTP_PROXY/HTTPS_PROXYset,-pafails for hosts not inNO_PROXY. Both used to dial the proxy's port on each ip.-pa,-maxhrand-srkeep to each ip, so stored file names change,-e cdnchecks each ip, and a-probefailure shows its ip, inhost_iptoo.host_ipis the ip the request was dialed to.-streamlines are trimmed like-llines, andRunner.Processreads its input as an input line.Closes #2667. Supersedes #2668, the same change in one commit after its review rounds.
Proof
Run with
app.testresolving to 127.0.0.1 and 127.0.0.2 through/etc/hosts, a server on each whose title names it and the Host header,redir.test(127.0.0.1) redirecting toother.test(127.0.0.3), and a proxy on 127.0.0.1:18500 whose title is the url it got;devat 668a37c, with-title -ip:dev-lwith lines pinninghttp://app.test:18080to .1 and, with"tags": ["cve"], to .2[srv=127.0.0.1 host=app.test:18080] [127.0.0.1],[srv=127.0.0.2 host=app.test:18080] [127.0.0.2]-lwith a pinned line, a malformed line and a plain url[ERR] Skipping input: invalid target line …, then the other two-pa -fr -sconhttp://redir.test:18201[302,200] [srv=127.0.0.1 host=other.test:18202] [127.0.0.1][302,200] [srv=127.0.0.3 host=other.test:18202] [127.0.0.1]-pa -sr -mc 200onhttp://app.test:18080-pa -probe -proxy http://127.0.0.1:18500[SUCCESS] [proxy got http://app.test:18080/] [127.0.0.1],[FAILED][FTL] Could not create runner: -pa/-probe-all-ips cannot be used with -proxy, …-proxy[FTL] Invalid input: target line "…" cannot be probed with options that do not keep to its ip: -proxyHTTP_PROXY=http://127.0.0.1:18500[ERR] Targets pinned to an ip … add their hosts to NO_PROXY; withNO_PROXY=app.test, both lines are probedrunner/targetline_test.goandcommon/httpx/requesthost_test.gorun local servers on 127.0.0.1 and 127.0.0.2 with the same port, a local DNS server, and a TLS server that records the SNI. Each fix has a test that fails when the fix is reverted.TestTargetLinePinsIPv6is skipped where IPv6 is not available.go test -race ./...passes, exceptTestRunner_probeall_targets, which needs direct DNS and fails ondevtoo.integration_tests/run.shandcmd/functional-testpass, andgolangci-lint run ./...reports nothing.Checklist
Summary by CodeRabbit