[clementbiron]

Validating a collection in CI took 18 minutes for a Facebook declaration (user-rights, 46 documents in a combine) and 64 minutes for Instagram, with the vlopses collections cancelled at the 10-minute cap. Every browser fetch took about 11 seconds, whatever the page.

The browser fetcher awaited every CSS selector of a source document, select and remove alike, until it matched an element with non-empty text, up to waitForElementsTimeout (10 seconds). A single remove absent from the page, or targeting an element without text such as img[alt=""], made every fetch wait for the whole timeout. Nothing documents that choice: the first browser fetcher (#183) only passed select, #189 merged both lists without explanation, and #1159 then required a non-empty text, which an image can never satisfy. Waiting for an element that will be removed serves no purpose: if it is there, extraction removes it; if it is not, there is nothing to remove.

The first commit awaits only the select selectors. Measured on 50 browser-fetched documents from 7 collections, each fetched twice with and without the remove selectors: average fetch down from 9.7 to 5.0 seconds, no version differed between the two modes. On the 3-document Facebook sample, validation went from 2 minutes to 29 seconds.

That accidental delay did give scripts time to finish rendering on some pages: on TikTok "Human Rights Policy", 2 fetches out of 11 captured the page before a navigation block was injected, yielding a version with the same content but two lines in a different order. The second commit replaces the accident with a deliberate, bounded wait: once the selectors exist, the content is read when the DOM has not changed for fetcher.domIdleDelay milliseconds (500), up to fetcher.domIdleTimeout (2000). It costs 0.5 to 0.9 seconds per fetch and captured that block in 16 fetches out of 16. A network-idle wait of the same length cost 1.6 to 2 seconds without capturing more. The observer is installed from within the awaited function so that the wait survives a navigation triggered by the page.

[MattiSG]

Waiting for an element that will be removed serves no purpose: if it is there, extraction removes it; if it is not, there is nothing to remove.

I disagree. The current behaviour enables consistency of snapshots. Such an optimisation would make remove behaviour potentially inconsistent. From memory, I also remember a case from a super lazy-loading Facebook page where each section was loaded async and where remove selectors acted as markers of secondary content being loaded, which was much harder to achieve with select as there was already content in the parent container but the detailed text we needed to check loading was loaded alongside some illustration image we could remove.

Using domIdleDelay worries me as as having, on the contrary, potential side-effects where we wait forever for a page that constantly refreshes its ads, for example. To be convinced of the innocuity of such a fundamental change, I would need to see a much larger sample size than 16.

[MattiSG]

Based on #1290 (comment), wouldn't we already see significant gains by changing the way we wait for img rather than for remove selectors? 🙂 I would feel much more comfortable with that.