Repository navigation
Write a login bundle instead of a cookies.txt - #260
Merged
Merged
Conversation
The browser login handed the engine a Netscape jar through -%K, and set the browser identity as a global -F. The login's User-Agent therefore went to every host the crawl touched, CDNs included. A bundle carries the cookies and the identity together, and the engine sends that identity to the bundle's own sites alone. The bundle goes to private storage through --bundle-dir, which WinHTTrack does not need: our single -O makes the engine's path_log the shared mirror folder. The -%K file stays for the same reason. It names a private file, so the jar the engine saves at the end of the crawl does not land beside the mirror. Nothing in the app writes it. Two engine facts shaped the rest. A bundle's cookies reach the jar only when the engine accepts cookies, so the hand-off still ticks the box. And the engine refuses to run at all on a bundle it cannot read, so an address naming one that is gone loses the name. The landed address now keeps its query and loses only its fragment. A login that ends on another site is added with a warning rather than refused. Each change matches WinHTTrack. The crawl's own start hosts are asked for cookies too, along with the hosts the user visited. getCookie answers for any host, and one that shares a parent domain inherits that parent's cookies. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
The first cut rebuilt hts_bundle_from_paste in Java. That function is exported and sits in Android.mk's source list. It names the file and puts an international host into ASCII. It also refuses an address holding a password, applies the caps, and reads the file back through the engine's own loader. Our copy did none of those last three. A login on an international site could therefore never be saved, and a bundle we wrote could stop the mirror. So BrowserLoginBundle, the Java bundle name, the scope lines and both engine-wrapper natives are gone. One JNI call does the work. A review found three bugs. Each is closed by that move or beside it: - A landed address carrying a password wrote a bundle, then left the address field without its name. The crawl ran logged out while the dialog said the login was saved. The engine refuses that paste. - A scope line built from the cookie hosts could be one the engine refuses, which aborts the mirror. The engine takes the scope from the start URL, and the design notes say cookie domains must not widen it. - Unticking "Accept cookies" after saving a login left the bundle read and its cookies dropped into a throwaway jar, with only a hidden warning. A start URL naming a bundle now drops -b0. Forget takes the bundle names out of the address field. The engine stops a crawl whose start URL names a bundle that is gone. That is the agreed behaviour rather than a fault to work around. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
The sibling entry points carry the same line: no caller passes null today, and GetStringUTFChars(NULL) is undefined. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
The login's doc comment re-derived what withoutBundleNames already states, and one ternary packed a missing native library and an engine refusal behind the same null. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Two logins on one project left two bundles. A second sign-in rotates the session cookie, so the two files hold one cookie name with two values. The engine then refuses to start the crawl at all. The address field now names the newest bundle alone, and the login deletes the files the field no longer names. Three more things a review round found: - Forget cut from the bundle name to the end of the line, so a line holding two addresses lost the second one. Only the name goes now. - Use this login on a page that never committed passed a null address to the engine. It asks for the address first. - The dialog counted the cookies and hosts we offered the engine. It reports what the engine kept, because a cookie outside the bundle's scope is dropped, and the over-long count stays ours. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
A mutation run showed what the source-text guards could not see. Forcing the bundle scan to false left -b0 in the argv, so the crawl ran logged out, and no test went red. The same held for dropping --bundle-dir. So the decision moved into BrowserLoginPolicy.withLogin, where a test drives it directly. Three mutants now die there: the scan, the folder test and the removal. Two of my own tests asserted nothing useful: - One claimed the engine's paste reader drops comment lines. Inverting the engine to keep them left the test green, and this app depends on neither behaviour, because it writes no comment. Dropped. - One looked for the loader read-back anywhere in htsbundle.c. It now asks that the read-back sits inside hts_bundle_from_paste, that its failure stops the write, and that the write comes after it. The private-storage guard was defeated twice, last by a path built with new File(getFilesDir(), "../../../sdcard"). It now asks that the path comes from a StoragePaths helper, rather than listing calls it must not make. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The browser login now writes a login bundle instead of a cookies.txt, and the engine writes it.
hts_bundle_from_pastenames the file and puts an international host into ASCII. It also refuses an address holding a password, applies its own caps, and reads the file back through its own loader. The bundle carries the browser's identity with the cookies, so the engine sends that identity to the bundle's own sites. Our old-Fsent it to every host the crawl touched.The bundle goes to private storage through
--bundle-dir, because our single-Omakes the engine'spath_logthe shared mirror folder. The-%Kfile stays private for that same reason. Nothing in the app writes either file.A login that ends on another site is now added with a warning rather than refused.