Repository navigation
feat(esri): page by OBJECTID range when the layer asks for it - #8
Merged
Merged
Conversation
A layer with request.paginate set to "oid" passes paginate_oid to EsriDumper, so it pages with OBJECTID where clauses instead of resultOffset. Hosted services fail or crawl at deep offsets (Ohio's statewide parcels fail past about 1.4M), while an OBJECTID range answers in about a second at the same depth. Layers without the setting keep offset paging. Claude-Session: https://claude.ai/code/session_01Qfy2CDLqR5FowD3b6rWkk8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A layer can now ask for OBJECTID-range paging with
"request": {"paginate": "oid"}. batch-machine passes that toEsriDumperaspaginate_oid, and pyesridump then pages withOBJECTID > a AND OBJECTID <= bwhere clauses instead ofresultOffset. No ticket; this unblocks the Ohio and New Jersey statewide parcels in ChopChop.Changes
openaddr/cache.py:EsriRestDownloadTask.download()readsrequest.paginatefrom the layer and passespaginate_oidtoEsriDumper.openaddr/tests/cache.py: tests that"paginate": "oid"reaches the dumper, and that layers without it keep offset paging.Decisions
request, next toheaders, which is the existing place for per-layer download options.Review notes
Why it is needed: Ohio's hosted statewide parcels view (6.3M features) fails with "Cannot perform query. Invalid query parameters" at offsets past about 1.4M, and offset pages ran at about 27 features/s.
OBJECTID > 2734000 AND OBJECTID <= 2735000on the same service returns 1,000 features in about a second. pyesridump already falls back from statistics to returnIdsOnly to geometry queries when a server cannot do min/max.https://claude.ai/code/session_01Qfy2CDLqR5FowD3b6rWkk8