diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index b50584f..2057033 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -7,6 +7,7 @@ This is a Make custom app in the local project format used by the Make Apps Edit - `scripts/generate.mjs` writes every JSON file under `general/`, `connections/`, `modules/`, `webhooks/`, `rpcs/` and `makecomapp.json` from `scripts/params.mjs`, the fixtures in `test/fixtures/` and the shared operation catalog. `scripts/catalog.json` is a vendored copy of the catalog used by the Zapier and n8n apps (`scripts/catalog.sha256` pins it; a test fails on drift) and `scripts/catalog-modules.mjs` translates it into modules, the webhook and the dynamic dropdowns. The generated files are committed. Edit the scripts, then run `node scripts/generate.mjs`. - `test/catalog.test.mjs` checks every catalog module (request, output, TCPA flag, billing text, guards, webhook attach and detach). - `test/structure.test.mjs` checks the manifest, file links, billing and compliance copy, and that no callback or wallet fields exist. +- `test/pagination.test.mjs` plays the six paged modules (Search Properties, Get Investor Properties, Find Investors, Get Property Permits, List Search Sessions, Watch New Search Sessions) against a fake BatchData endpoint the way Make pages: it checks the Limit is met, that no request asks for more records than the Limit has left (billing safety), that the whole body is repeated on every page, and the response paths used. `scripts/paging.mjs` builds the directives. - `test/replay.test.mjs` renders the request templates with a small IML evaluator (`test/iml.mjs`) and compares the request bodies and output mapping to the n8n and Zapier integrations. ``` @@ -20,8 +21,8 @@ Node 20 or later. There are no dependencies. 1. Create a Make account (the organisation that will own the app) and a Make API key with the `apps:read` and `apps:write` scopes. 2. Install the Make Apps Editor extension in VS Code and open this folder. 3. Put the API key in `.secrets/apikey` (the folder is git-ignored). -4. In `makecomapp.json`, set `origins[0].baseUrl` to your zone, for example `https://us1.make.com/api` or `https://eu1.make.com/api`. -5. Run **Deploy to Make** from the extension. The app is created as a private app with id `batchdata`. +4. `makecomapp.json` has one documented origin: the private app `batchdata-r6b83g` in organization 3655001 on `https://us2.make.com/api` (Make added the suffix to the name `batchdata`, so the connection and webhook remote names are `batchdata-r6b83g`). To deploy to another zone or organization, copy that entry, change `baseUrl` and `appId`, and clear the `idMapping`. +5. Run **Deploy to Make** from the extension. 6. Open the app in Make and run the checks below. ## What was and was not verified diff --git a/README.md b/README.md index c1fb5c7..aba7b99 100644 --- a/README.md +++ b/README.md @@ -1,66 +1,144 @@ # BatchData for Make -Use BatchData property data inside Make scenarios: search and look up properties, skip trace owners (V1 and V3, single and batch), reverse skip trace, verify and geocode addresses, check phone numbers, get permits, find investors by buy box, run search sessions and watch property monitors, with polling and instant (webhook) triggers and a generic API call module. +![BatchData](assets/logo-512.png) -Support: support@batchservice.com +BatchData gives you property, owner and address data through an API. This app brings it into your Make scenarios, so you can find properties, enrich them with owner contact details, verify addresses and phone numbers, and react to property changes without writing code. + +Typical uses: -> **Status: v0.2.0, not yet imported into Make.** It passes its offline tests. Most operations were smoke-tested live through the BatchData API; monitors and the instant trigger were not (see [What was and was not verified](#what-was-and-was-not-verified)). +- Build a list of properties that match your filters and send it to a spreadsheet, CRM or marketing tool. +- Look up one address and add the full property record, ownership and sale history to a lead. +- Find the owner's phone numbers and emails (skip trace) for a property, and check the numbers against Do Not Call and TCPA litigator lists. +- Clean up addresses before mailing, or turn an address into coordinates. +- Start a scenario the moment a property you are watching changes. -![BatchData](assets/logo.png) +Support: support@batchservice.com ## Connect -1. Create an API token in your BatchData account. -2. In a BatchData module, click **Add** next to Connection and paste the token. +1. Sign in to your BatchData account and create an API token. Tokens are available on plans with API access. +2. In Make, add any BatchData module to a scenario. +3. Next to **Connection**, click **Add**, paste the token into **API token** and save. -If the token is rejected, create a new one. For help, contact support@batchservice.com. +If the token is rejected, create a new one and try again. Use a sandbox token to try the app with sample data. Sandbox tokens return mock data and are not billed. Use a production token for real data. If you need help, contact support@batchservice.com. ## Modules -Billing text is also in each module's description in Make. - -| Module | Type | What it does | Billing | -| --- | --- | --- | --- | -| Search Properties | Search | Finds properties by location and filters. Up to 100 records per run. | Per record returned (0 is billed as 1) | -| Get Property Details | Search | Full property data for one address or ID. | Per matched property | -| Batch Look Up Properties | Search | Full records for up to 100 addresses in one call. | Per matched property | -| Skip Trace Owner (V1) | Search | Owner contact details, one result per person. | Per matched property | -| Skip Trace Owner (V3) | Search | Owner contact details, one result per property with a persons list. | Per matched property | -| Batch Skip Trace Properties (V3) | Search | Up to 100 properties in one call. | Per matched property | -| Reverse Skip Trace (Phone or Email) | Search | Finds the person behind a phone number or email. | Per matched result | -| Verify Address / Batch Verify Addresses | Search | Standardizes one or up to 100 US addresses. | Per address | -| Autocomplete, Geocode, Reverse Geocode | Search | Address suggestions, coordinates, nearest addresses. | This call does not bill property records | -| Verify Phone Numbers, Check Do Not Call, Check TCPA Litigator | Search | Phone checks. | Per phone number | -| Get Property Permits | Search | Building permits for one property, one result per permit. | See the module description | -| Find Investors (Buy Box), Get Investor Properties | Search | Investors who bought near a subject property, and their holdings. | See the module description | -| List, Create or Advance, Delete Search Session | Search, Action | Persistent search sessions that return only properties not yet delivered. | Create or Advance bills each record delivered | -| List, Get, Create, Delete Property Monitor | Search, Action | Property monitors with stream delivery (Kinesis or Azure Event Hubs). | According to your BatchData plan | -| Watch New Search Sessions | Polling trigger | New search sessions on your team. | Does not bill property records | -| Watch New Properties in a Search Session (Billed) | Polling trigger | New properties delivered by a session. | Every check bills each property delivered | -| Watch Property Monitor Events (Instant) | Instant trigger | Fires when a property is added to, removed from or changed in a monitor. | According to your BatchData plan | -| Watch Property Monitor Events with Property Record (Instant, Billed) | Instant trigger | Same, plus the full property record. | Plan billing, plus one property lookup per event | -| Make an API Call | Universal | Any other BatchData API path. Wallet, asynchronous and monitor create or edit paths are blocked, and so are bodies with callback URL fields. | Follows the endpoint | - -### Instant triggers (property monitor events) - -Turning a scenario on creates a V2 property subscription whose delivery is the scenario's own Make webhook URL. Turning it off (or deleting the scenario) deletes that subscription. You never enter a callback URL. Each event is `{eventId, propertyId, parcelHash, addressHash, addedToSubscriptionIds, removedFromSubscriptionIds, propertyChangedAt, sentAt}`. The event carries IDs only; use the module with **Property Record** to add the full record (one billed property lookup per event). Use `eventId` to skip repeats. - -Limits (also in each trigger's description): up to 5 monitors per team, a monitor cannot be edited after it is created (delete and recreate it), a new monitor can take up to 7 days to start delivering, and delivery is best effort (4 attempts per event). Because of the 5 monitor limit, avoid many scenarios that each create a monitor. - -There is no polling "new monitor" trigger in Make: a polling trigger needs a date or an ordered ID on each item, and monitors have neither. Use List Property Monitors on a schedule instead. The Zapier and n8n apps do include one. +### Properties -## Phone compliance +| Module | What it does | +| --- | --- | +| Search Properties | Finds properties by location and filters such as property type, bedrooms, owner details and prebuilt quick lists. | +| Get Property Details | Returns the full record for one address, property ID, hash or APN. | +| Batch Look Up Properties | Returns full records for up to 100 addresses in one call. | +| Get Property Permits | Returns the building permits for one property, one result per permit. | + +### Owner contact + +| Module | What it does | +| --- | --- | +| Skip Trace Owner (V1) | Owner contact details for a property, one result per person. | +| Skip Trace Owner (V3) | Owner contact details, one result per property with a list of persons. | +| Batch Skip Trace Properties (V3) | Skip trace for up to 100 properties in one call. | +| Reverse Skip Trace (Phone or Email) | Finds the person behind a phone number or email address. | +| Verify Phone Numbers | Checks whether phone numbers are valid and reachable. | +| Check Do Not Call | Checks phone numbers against the Do Not Call list. | +| Check TCPA Litigator | Checks phone numbers against known TCPA litigators. | + +### Addresses + +| Module | What it does | +| --- | --- | +| Verify Address, Batch Verify Addresses | Standardizes and validates one address, or up to 100 US addresses. | +| Autocomplete Address | Suggests addresses as you type. | +| Geocode Address | Turns an address into coordinates. | +| Reverse Geocode (Find Nearby Addresses) | Finds the addresses closest to a set of coordinates. | + +### Investors + +| Module | What it does | +| --- | --- | +| Find Investors (Buy Box) | Finds likely investors for a subject property, based on nearby comparable sales. | +| Get Investor Properties | Lists the properties an investor holds. | + +### Search sessions + +A search session remembers what it has already returned, so each run gives you only properties you have not received before. + +| Module | What it does | +| --- | --- | +| Create or Advance Search Session | Starts a session, or returns the next properties from an existing one. | +| List Search Sessions | Lists the sessions on your team. | +| Delete Search Session | Deletes a session. | + +### Property monitors + +A monitor reports when a property is added to it, removed from it or changed. + +| Module | What it does | +| --- | --- | +| Create Property Monitor (Stream Delivery) | Creates a monitor that delivers to AWS Kinesis or Azure Event Hubs. | +| List Property Monitors, Get Property Monitor | Reads your monitors. | +| Delete Property Monitor | Deletes a monitor. | -Skip trace and phone modules return data only. You are responsible for following the TCPA and Do Not Call rules before you contact anyone. Skip trace removes phone numbers flagged as TCPA litigators unless you switch on **Include phone numbers flagged as TCPA litigators**. +### Triggers -## What was and was not verified +| Module | Type | What it does | +| --- | --- | --- | +| Watch New Search Sessions | Polling | Runs when a new search session appears on your team. | +| Watch New Properties in a Search Session (Billed) | Polling | Runs for each new property a session delivers. | +| Watch Property Monitor Events (Instant) | Instant | Runs when a property is added to, removed from or changed in a monitor. | +| Watch Property Monitor Events with Property Record (Instant, Billed) | Instant | The same, and adds the full property record to each event. | + +### Other + +| Module | What it does | +| --- | --- | +| Make an API Call | Calls any other BatchData API path with your connection. | + +## Limit and paging + +Search Properties, Get Investor Properties, Find Investors (Buy Box), Get Property Permits, List Search Sessions and Watch New Search Sessions have a **Limit** field. It is the total number of records you want. Make fetches them in pages of up to 100 and never asks for more than your Limit, so the Limit is also the most you can be billed for in one run. Search Properties and List Search Sessions also have **Skip** to start part-way through the results. + +Each record comes out as its own bundle. + +## Billing + +Your BatchData plan decides what each call costs. Each module's description in Make says how it is billed. In general: + +- Search and lookup modules bill per record returned. Set a Limit to cap it. +- Skip trace modules bill per matched property. +- Address, phone and compliance checks bill per item checked. +- Autocomplete, geocoding and the modules that list, get or delete search sessions do not bill property records. +- Property monitors and the events they deliver are billed according to your BatchData plan. +- Get Property Permits bills per property requested, not per permit. A Limit above 100 sends more than one request. +- Watch New Properties in a Search Session (Billed) bills every property it delivers, on every scheduled run. +- Watch Property Monitor Events with Property Record (Instant, Billed) adds one property lookup per event. + +For pricing questions, contact support@batchservice.com. + +## Property monitor events + +When you turn a scenario on, the instant trigger creates a monitor that delivers to that scenario's own Make webhook. When you turn it off, or delete the scenario, the monitor is deleted. You do not enter a callback URL. + +Each event contains the event ID, property ID, address and parcel hashes, the IDs of the monitors that added or removed the property, and the time it changed. It does not contain the property record. Use the module with **Property Record** to add it. Use the event ID to skip repeats. + +Good to know: + +- You can have up to 5 monitors per team, so avoid many scenarios that each create one. +- A monitor cannot be edited. Delete it and create a new one. +- A new monitor can take up to 7 days to start delivering. +- Delivery is best effort, with 4 attempts per event, so the same event can arrive more than once. +- Make has no polling trigger for new monitors. Use List Property Monitors on a schedule instead. + +## Phone compliance -Verified live (production API, 1 record per call, through the same request shapes the other BatchData integrations use): property search, lookup, skip trace V1 and V3, reverse skip trace (no match), address verify, DNC, permits, investor buy box and properties, geocode, reverse geocode, autocomplete, and the search-session create, list and delete lifecycle. +Skip trace and phone modules return data only. You are responsible for following the TCPA and Do Not Call rules before you contact anyone. Skip trace leaves out phone numbers flagged as TCPA litigators unless you switch on **Include phone numbers flagged as TCPA litigators**. -Mocked only: property monitors and the instant triggers. The token used gets 403 on the subscription endpoints, so create, list, delete and event delivery need a sandbox token (mock data, no cost, from app.batchdata.com) or a token with that ability. +## Support -Offline tests (47) check the manifest, file links, copy rules (one support contact, billing text in every module, no callback fields) and the request body, query and output of every module against mocked responses. The offline IML evaluator is a subset of Make's engine, so Make itself has not run these modules; see `DEVELOPMENT.md` for the checks to run after import. +Email support@batchservice.com. ## License -The code in this repository is released under the MIT License (see `LICENSE`). The BatchData API and the data it returns are proprietary to BatchData (batchdata.io) and are subject to BatchData's terms of service; the MIT License does not cover them. Support: support@batchservice.com +The code in this repository is released under the MIT License (see `LICENSE`). The BatchData API and the data it returns are proprietary to BatchData (batchdata.io) and are subject to BatchData's terms of service. Development notes are in `DEVELOPMENT.md`. diff --git a/assets/logo-512.png b/assets/logo-512.png new file mode 100644 index 0000000..e9ae424 Binary files /dev/null and b/assets/logo-512.png differ diff --git a/makecomapp.json b/makecomapp.json index 0b102b2..7f30ea9 100644 --- a/makecomapp.json +++ b/makecomapp.json @@ -20,7 +20,7 @@ "searchProperties": { "moduleType": "search", "label": "Search Properties", - "description": "Finds properties that match your filters. Billed per record returned. A Max results value of 0 is billed as 1 record. One call returns at most 100 records.", + "description": "Finds properties that match your filters. Billed per record returned. Set a Limit to cap the records returned in total. Records are requested in pages of up to 100, and never more than the Limit are requested.", "actionCrud": "read", "connection": "batchdata", "codeFiles": { @@ -253,7 +253,7 @@ "investorProperties": { "moduleType": "search", "label": "Get Investor Properties", - "description": "Returns the properties of one investor, one result per property. Get the investor ID from Find Investors (Buy Box). Use the \"nextPageCursor\" value of the last result as Page Cursor to read the next page. Billed per property row returned (take sets the maximum cost of a call).", + "description": "Returns the properties of one investor, one result per property. Get the investor ID from Find Investors (Buy Box). Set a Limit and Make reads the pages for you; each result also carries the \"nextPageCursor\" of its page. Billed per property row returned (the Limit sets the maximum cost of a run).", "actionCrud": "read", "connection": "batchdata", "codeFiles": { @@ -331,7 +331,7 @@ "createMonitor": { "moduleType": "action", "label": "Create Property Monitor (Stream Delivery)", - "description": "Creates a property monitor that streams changes for matching properties to your own AWS Kinesis stream or Azure Event Hub. For webhook delivery use the \"New Property Monitor Event\" trigger, which sets up and removes its own monitor. Monitors and the events they deliver are billed according to your BatchData plan; contact support@batchservice.com for pricing. Limits: up to 5 monitors per team, a monitor cannot be edited after it is created (delete and recreate it), a new monitor can take up to 7 days to start delivering, and delivery is best effort (4 attempts per event).", + "description": "Creates a property monitor that streams changes for matching properties to your own AWS Kinesis stream or Azure Event Hub. For webhook delivery use the \"Watch Property Monitor Events (Instant)\" trigger, which sets up and removes its own monitor. Monitors and the events they deliver are billed according to your BatchData plan; contact support@batchservice.com for pricing. Limits: up to 5 monitors per team, a monitor cannot be edited after it is created (delete and recreate it), a new monitor can take up to 7 days to start delivering, and delivery is best effort (4 attempts per event).", "actionCrud": "create", "connection": "batchdata", "codeFiles": { @@ -381,7 +381,7 @@ "watchMonitorEvents": { "moduleType": "instant_trigger", "label": "Watch Property Monitor Events (Instant)", - "description": "Triggers when BatchData reports that a property matching your criteria was added to or removed from a monitor, or changed (instant). Turning the scenario on creates a monitor that delivers to this trigger's own hook URL; turning it off deletes that monitor. Each event carries the property ID and change details; use \"Watch Property Monitor Events with Property Record\" to also look up the full property record. Monitors and the events they deliver are billed according to your BatchData plan; contact support@batchservice.com for pricing. Limits: up to 5 monitors per team, a monitor cannot be edited after it is created (delete and recreate it), a new monitor can take up to 7 days to start delivering, and delivery is best effort (4 attempts per event).", + "description": "Triggers when BatchData reports that a property matching your criteria was added to or removed from a monitor, or changed (instant). Turning the scenario on creates a monitor that delivers to this trigger's own hook URL; turning it off deletes that monitor. Each event carries the property ID and change details. Monitors and the events they deliver are billed according to your BatchData plan; contact support@batchservice.com for pricing. To get the full property record with each event, use \"Watch Property Monitor Events with Property Record (Instant, Billed)\" instead. Limits: up to 5 monitors per team, a monitor cannot be edited after it is created (delete and recreate it), a new monitor can take up to 7 days to start delivering, and delivery is best effort (4 attempts per event).", "connection": "batchdata", "webhook": "monitorEvents", "codeFiles": { @@ -393,7 +393,7 @@ "watchMonitorEventsWithProperty": { "moduleType": "instant_trigger", "label": "Watch Property Monitor Events with Property Record (Instant, Billed)", - "description": "Triggers when BatchData reports that a property matching your criteria was added to or removed from a monitor, or changed (instant). Turning the scenario on creates a monitor that delivers to this trigger's own hook URL; turning it off deletes that monitor. Each event carries the property ID and change details; use \"Watch Property Monitor Events with Property Record\" to also look up the full property record. Monitors and the events they deliver are billed according to your BatchData plan; contact support@batchservice.com for pricing. Limits: up to 5 monitors per team, a monitor cannot be edited after it is created (delete and recreate it), a new monitor can take up to 7 days to start delivering, and delivery is best effort (4 attempts per event). This version also looks up the full property record for every event. The lookup is billed per event, per matched property, on top of the monitor charges.", + "description": "Triggers when BatchData reports that a property matching your criteria was added to or removed from a monitor, or changed (instant). Turning the scenario on creates a monitor that delivers to this trigger's own hook URL; turning it off deletes that monitor. Each event carries the property ID and change details. Monitors and the events they deliver are billed according to your BatchData plan; contact support@batchservice.com for pricing. Limits: up to 5 monitors per team, a monitor cannot be edited after it is created (delete and recreate it), a new monitor can take up to 7 days to start delivering, and delivery is best effort (4 attempts per event). This version also looks up the full property record for every event and adds it to the output as \"property\". The lookup is billed per event, per matched property, on top of the monitor charges.", "connection": "batchdata", "webhook": "monitorEvents", "codeFiles": { @@ -437,16 +437,16 @@ }, "origins": [ { - "label": "Make (set your zone)", - "baseUrl": "https://eu1.make.com/api", - "appId": "batchdata", + "label": "Make us2: batchdata-r6b83g (private app)", + "baseUrl": "https://us2.make.com/api", + "appId": "batchdata-r6b83g", "appVersion": 1, "apikeyFile": ".secrets/apikey", "idMapping": { "connection": [ { "local": "batchdata", - "remote": "batchdata" + "remote": "batchdata-r6b83g" } ], "module": [ @@ -585,7 +585,7 @@ "webhook": [ { "local": "monitorEvents", - "remote": "monitorEvents" + "remote": "batchdata-r6b83g" } ] } diff --git a/modules/getPermits/getPermits.communication.iml.json b/modules/getPermits/getPermits.communication.iml.json index baa1966..e1a628a 100644 --- a/modules/getPermits/getPermits.communication.iml.json +++ b/modules/getPermits/getPermits.communication.iml.json @@ -15,7 +15,7 @@ "countyFipsCode": "{{ifempty(parameters.countyFipsCode, undefined)}}" }, "options": { - "take": "{{ifempty(parameters.take, 25)}}" + "take": "{{min(100, ifempty(parameters.limit, 25))}}" } }, "response": { @@ -24,6 +24,28 @@ "{{...}}": "{{item}}", "property": "{{body.result.property}}", "searchMeta": "{{body.result.meta.results}}" - } + }, + "limit": "{{ifempty(parameters.limit, 25)}}" + }, + "pagination": { + "body": { + "request": { + "address": "{{ifempty(parameters.address, undefined)}}", + "status": "{{ifempty(parameters.status, undefined)}}", + "type": "{{ifempty(parameters.type, undefined)}}", + "description": "{{ifempty(parameters.description, undefined)}}", + "allTags": "{{ifempty(parameters.allTags, undefined)}}", + "issueDate": "{{ifempty(parameters.issueDate, undefined)}}", + "jobValue": "{{ifempty(parameters.jobValue, undefined)}}", + "propertyId": "{{ifempty(parameters.propertyId, undefined)}}", + "apn": "{{ifempty(parameters.apn, undefined)}}", + "countyFipsCode": "{{ifempty(parameters.countyFipsCode, undefined)}}" + }, + "options": { + "take": "{{min(100, ifempty(parameters.limit, 25) - (pagination.page - 1) * 100)}}", + "skip": "{{0 + (pagination.page - 1) * 100}}" + } + }, + "condition": "{{body.result.meta.results.resultsFound > 0 + (pagination.page - 1) * 100 && ifempty(parameters.limit, 25) > (pagination.page - 1) * 100}}" } } diff --git a/modules/getPermits/getPermits.mappable-params.iml.json b/modules/getPermits/getPermits.mappable-params.iml.json index ea0ad98..54cff46 100644 --- a/modules/getPermits/getPermits.mappable-params.iml.json +++ b/modules/getPermits/getPermits.mappable-params.iml.json @@ -248,14 +248,15 @@ ] }, { - "name": "take", - "label": "Max Results", - "help": "Whole number from 1 to 100. Default 25.", - "type": "integer", + "name": "limit", + "type": "uinteger", + "label": "Limit", "default": 25, + "required": true, "validate": { "min": 1, - "max": 100 - } + "max": 5000 + }, + "help": "The maximum number of permits to return in total (1 to 5000, default 25). They are requested in pages of up to 100, and never more than this number are requested. Each page is billed per property, so a Limit above 100 sends more than one billed request." } ] diff --git a/modules/investorBuybox/investorBuybox.communication.iml.json b/modules/investorBuybox/investorBuybox.communication.iml.json index efdd0a1..7a07935 100644 --- a/modules/investorBuybox/investorBuybox.communication.iml.json +++ b/modules/investorBuybox/investorBuybox.communication.iml.json @@ -14,7 +14,7 @@ } }, "options": { - "take": "{{ifempty(parameters.take, 25)}}" + "take": "{{min(100, ifempty(parameters.limit, 25))}}" } }, "response": { @@ -23,6 +23,27 @@ "{{...}}": "{{item}}", "searchMeta": "{{body.result.meta.results}}", "warnings": "{{body.result.warnings}}" - } + }, + "limit": "{{ifempty(parameters.limit, 25)}}" + }, + "pagination": { + "body": { + "searchCriteria": { + "investor": "{{ifempty(parameters.investor, undefined)}}", + "intel": "{{ifempty(parameters.intel, undefined)}}", + "{{...}}": "{{if(parameters.additionalCriteria, parseJSON(parameters.additionalCriteria), undefined)}}", + "compAddress": { + "street": "{{parameters.street}}", + "city": "{{ifempty(parameters.city, undefined)}}", + "state": "{{ifempty(parameters.state, undefined)}}", + "zip": "{{ifempty(parameters.zip, undefined)}}" + } + }, + "options": { + "take": "{{min(100, ifempty(parameters.limit, 25) - (pagination.page - 1) * 100)}}", + "skip": "{{0 + (pagination.page - 1) * 100}}" + } + }, + "condition": "{{body.result.meta.results.resultsFound > 0 + (pagination.page - 1) * 100 && ifempty(parameters.limit, 25) > (pagination.page - 1) * 100}}" } } diff --git a/modules/investorBuybox/investorBuybox.mappable-params.iml.json b/modules/investorBuybox/investorBuybox.mappable-params.iml.json index e01e228..547183e 100644 --- a/modules/investorBuybox/investorBuybox.mappable-params.iml.json +++ b/modules/investorBuybox/investorBuybox.mappable-params.iml.json @@ -96,14 +96,15 @@ "type": "text" }, { - "name": "take", - "label": "Max Results", - "help": "Whole number from 1 to 100. Default 25.", - "type": "integer", + "name": "limit", + "type": "uinteger", + "label": "Limit", "default": 25, + "required": true, "validate": { "min": 1, - "max": 100 - } + "max": 5000 + }, + "help": "The maximum number of investors to return in total (1 to 5000, default 25). They are requested in pages of up to 100, and never more than this number are requested." } ] diff --git a/modules/investorProperties/investorProperties.communication.iml.json b/modules/investorProperties/investorProperties.communication.iml.json index 74863b2..7ee3416 100644 --- a/modules/investorProperties/investorProperties.communication.iml.json +++ b/modules/investorProperties/investorProperties.communication.iml.json @@ -8,7 +8,7 @@ "includeChildEntities": "{{if(parameters.includeChildEntities == true, true, undefined)}}" }, "options": { - "take": "{{ifempty(parameters.take, 25)}}", + "take": "{{min(100, ifempty(parameters.limit, 25))}}", "pageCursor": "{{ifempty(parameters.pageCursor, undefined)}}" } }, @@ -18,6 +18,21 @@ "{{...}}": "{{item}}", "resultsFound": "{{body.result.resultsFound}}", "nextPageCursor": "{{body.result.nextPageCursor}}" - } + }, + "limit": "{{ifempty(parameters.limit, 25)}}" + }, + "pagination": { + "body": { + "searchCriteria": { + "investorId": "{{parameters.investorId}}", + "ownership": "{{ifempty(parameters.ownership, undefined)}}", + "includeChildEntities": "{{if(parameters.includeChildEntities == true, true, undefined)}}" + }, + "options": { + "take": "{{min(100, ifempty(parameters.limit, 25) - (pagination.page - 1) * 100)}}", + "pageCursor": "{{body.result.nextPageCursor}}" + } + }, + "condition": "{{length(ifempty(body.result.nextPageCursor, '')) > 0 && ifempty(parameters.limit, 25) > (pagination.page - 1) * 100}}" } } diff --git a/modules/investorProperties/investorProperties.mappable-params.iml.json b/modules/investorProperties/investorProperties.mappable-params.iml.json index f2ae100..c8834a2 100644 --- a/modules/investorProperties/investorProperties.mappable-params.iml.json +++ b/modules/investorProperties/investorProperties.mappable-params.iml.json @@ -31,20 +31,21 @@ "type": "boolean" }, { - "name": "take", - "label": "Max Results", - "help": "Whole number from 1 to 100. Default 25.", - "type": "integer", + "name": "limit", + "type": "uinteger", + "label": "Limit", "default": 25, + "required": true, "validate": { "min": 1, - "max": 100 - } + "max": 5000 + }, + "help": "The maximum number of properties to return in total (1 to 5000, default 25). They are requested in pages of up to 100, and never more than this number are requested." }, { "name": "pageCursor", - "label": "Page Cursor", - "help": "Optional. The nextPageCursor from the previous page, used with the same criteria.", + "label": "Start from page cursor", + "help": "Optional. Leave empty to start at the first page. The Limit already pages for you, so use this only to continue an earlier run from its Next Page Cursor.", "type": "text" } ] diff --git a/modules/listSearchSessions/listSearchSessions.communication.iml.json b/modules/listSearchSessions/listSearchSessions.communication.iml.json index 4b7d2c2..741bc2f 100644 --- a/modules/listSearchSessions/listSearchSessions.communication.iml.json +++ b/modules/listSearchSessions/listSearchSessions.communication.iml.json @@ -2,11 +2,19 @@ "url": "/v2/property/search-sessions", "method": "GET", "qs": { - "take": "{{ifempty(parameters.take, 50)}}", + "take": "{{min(100, ifempty(parameters.limit, 50))}}", "skip": "{{ifempty(parameters.skip, undefined)}}" }, "response": { "iterate": "{{body.results.sessions}}", - "output": "{{item}}" + "output": "{{item}}", + "limit": "{{ifempty(parameters.limit, 50)}}" + }, + "pagination": { + "qs": { + "take": "{{min(100, ifempty(parameters.limit, 50) - (pagination.page - 1) * 100)}}", + "skip": "{{ifempty(parameters.skip, 0) + (pagination.page - 1) * 100}}" + }, + "condition": "{{body.results.total > ifempty(parameters.skip, 0) + (pagination.page - 1) * 100 && ifempty(parameters.limit, 50) > (pagination.page - 1) * 100}}" } } diff --git a/modules/listSearchSessions/listSearchSessions.mappable-params.iml.json b/modules/listSearchSessions/listSearchSessions.mappable-params.iml.json index 4009694..c3b7019 100644 --- a/modules/listSearchSessions/listSearchSessions.mappable-params.iml.json +++ b/modules/listSearchSessions/listSearchSessions.mappable-params.iml.json @@ -1,13 +1,15 @@ [ { - "name": "take", - "label": "Max Results", - "type": "integer", + "name": "limit", + "type": "uinteger", + "label": "Limit", "default": 50, + "required": true, "validate": { "min": 1, "max": 100 - } + }, + "help": "The maximum number of search sessions to return in total (1 to 100, default 50). They are requested in pages of up to 100, and never more than this number are requested." }, { "name": "skip", diff --git a/modules/searchProperties/searchProperties.communication.iml.json b/modules/searchProperties/searchProperties.communication.iml.json index 0f02025..df307f0 100644 --- a/modules/searchProperties/searchProperties.communication.iml.json +++ b/modules/searchProperties/searchProperties.communication.iml.json @@ -2,10 +2,10 @@ "url": "/v1/property/search", "method": "POST", "body": { - "searchCriteria": "{{omit(parameters, 'take', 'skip')}}", + "searchCriteria": "{{omit(parameters, 'limit', 'skip')}}", "options": { - "take": "{{ifempty(parameters.take, 25)}}", - "skip": "{{ifempty(parameters.skip, undefined)}}" + "skip": "{{ifempty(parameters.skip, undefined)}}", + "take": "{{min(100, ifempty(parameters.limit, 25))}}" } }, "response": { @@ -13,6 +13,17 @@ "output": { "{{...}}": "{{item}}", "searchMeta": "{{body.results.meta.results}}" - } + }, + "limit": "{{ifempty(parameters.limit, 25)}}" + }, + "pagination": { + "body": { + "searchCriteria": "{{omit(parameters, 'limit', 'skip')}}", + "options": { + "skip": "{{ifempty(parameters.skip, 0) + (pagination.page - 1) * 100}}", + "take": "{{min(100, ifempty(parameters.limit, 25) - (pagination.page - 1) * 100)}}" + } + }, + "condition": "{{body.results.meta.results.resultsFound > ifempty(parameters.skip, 0) + (pagination.page - 1) * 100 && ifempty(parameters.limit, 25) > (pagination.page - 1) * 100}}" } } diff --git a/modules/searchProperties/searchProperties.mappable-params.iml.json b/modules/searchProperties/searchProperties.mappable-params.iml.json index fc9dab5..5c9339f 100644 --- a/modules/searchProperties/searchProperties.mappable-params.iml.json +++ b/modules/searchProperties/searchProperties.mappable-params.iml.json @@ -698,19 +698,21 @@ ] }, { - "name": "take", + "name": "limit", "type": "uinteger", - "label": "Max results", + "label": "Limit", "default": 25, + "required": true, "validate": { - "max": 100 + "min": 1, + "max": 5000 }, - "help": "Up to 100 per run. You are billed per record returned, and a value of 0 is billed as 1 record." + "help": "The maximum number of properties to return in total (1 to 5000, default 25). They are requested in pages of up to 100, and never more than this number are requested. You are billed per record returned." }, { "name": "skip", "type": "uinteger", "label": "Skip", - "help": "Number of records to skip, for paging." + "help": "Number of records to skip before the first one returned. Leave empty to start at the first record." } ] diff --git a/modules/watchSearchSessions/watchSearchSessions.communication.iml.json b/modules/watchSearchSessions/watchSearchSessions.communication.iml.json index adfa25d..d3333d5 100644 --- a/modules/watchSearchSessions/watchSearchSessions.communication.iml.json +++ b/modules/watchSearchSessions/watchSearchSessions.communication.iml.json @@ -1,9 +1,6 @@ { "url": "/v2/property/search-sessions", "method": "GET", - "qs": { - "take": 100 - }, "response": { "iterate": "{{body.results.sessions}}", "output": "{{item}}", @@ -12,6 +9,17 @@ "date": "{{item.createdAt}}", "id": "{{item.id}}", "order": "unordered" - } + }, + "limit": "{{ifempty(parameters.limit, 100)}}" + }, + "qs": { + "take": "{{min(100, ifempty(parameters.limit, 100))}}" + }, + "pagination": { + "qs": { + "take": "{{min(100, ifempty(parameters.limit, 100) - (pagination.page - 1) * 100)}}", + "skip": "{{0 + (pagination.page - 1) * 100}}" + }, + "condition": "{{body.results.total > 0 + (pagination.page - 1) * 100 && ifempty(parameters.limit, 100) > (pagination.page - 1) * 100}}" } } diff --git a/modules/watchSearchSessions/watchSearchSessions.static-params.iml.json b/modules/watchSearchSessions/watchSearchSessions.static-params.iml.json index fe51488..fc97192 100644 --- a/modules/watchSearchSessions/watchSearchSessions.static-params.iml.json +++ b/modules/watchSearchSessions/watchSearchSessions.static-params.iml.json @@ -1 +1,14 @@ -[] +[ + { + "name": "limit", + "type": "uinteger", + "label": "Limit", + "default": 100, + "required": true, + "validate": { + "min": 1, + "max": 100 + }, + "help": "The maximum number of search sessions to return in total (1 to 100, default 100). They are requested in pages of up to 100, and never more than this number are requested. A team has at most 100 search sessions, so the default returns all of them." + } +] diff --git a/package.json b/package.json index 452b08f..c5c4a68 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "batchdata-make", - "version": "0.2.0", + "version": "0.3.0", "private": true, "description": "BatchData custom app for Make", "type": "module", diff --git a/scripts/catalog-modules.mjs b/scripts/catalog-modules.mjs index fbb3c66..c00078d 100644 --- a/scripts/catalog-modules.mjs +++ b/scripts/catalog-modules.mjs @@ -2,6 +2,7 @@ // The catalog is the single source of truth shared with the Zapier and n8n integrations. import fs from 'node:fs'; import { root } from './lib.mjs'; +import { limitParam, offsetPaging, cursorPaging } from './paging.mjs'; export const catalog = JSON.parse(fs.readFileSync(`${root}/scripts/catalog.json`, 'utf8')); const fx = (n) => JSON.parse(fs.readFileSync(`${root}/test/fixtures/${n}.json`, 'utf8')); @@ -259,7 +260,7 @@ for (const op of catalog.ops) { actionCrud: CRUD(op), catalogKey: op.key, label: op.label, - description: describe(op), + description: describe(op).replace('the "New Property Monitor Event" trigger', 'the "Watch Property Monitor Events (Instant)" trigger'), communication: requestFor(op), params: paramsFor(op), sampleFrom: () => ({ ...op.sample, ...Object.fromEntries(Object.keys(op.result.attach ?? {}).filter((k) => !(k in op.sample)).map((k) => [k, SAMPLE_FILL[k] ?? {}])) }), @@ -268,6 +269,9 @@ for (const op of catalog.ops) { const staticFromFields = (op) => paramsFor(op); +// A team can hold at most 100 search sessions. +const SESSIONS_MAX = 100; +const SESSION_LIMIT_DEFAULT = 100; const trig = (key) => catalog.triggers.find((t) => t.key === key); { @@ -280,16 +284,22 @@ const trig = (key) => catalog.triggers.find((t) => t.key === key); communication: { url: api(t.http.path), method: 'GET', - qs: { take: t.fixedQuery.take }, response: { iterate: `{{body.${t.result.list}}}`, output: '{{item}}', trigger: { type: 'date', date: '{{item.createdAt}}', id: '{{item.id}}', order: 'unordered' }, }, }, - staticParams: [], + staticParams: [ + limitParam(SESSION_LIMIT_DEFAULT, { + max: SESSIONS_MAX, + what: 'search sessions', + extra: ' A team has at most 100 search sessions, so the default returns all of them.', + }), + ], sampleFrom: () => t.sample, }; + offsetPaging(catalogModules.watchSearchSessions.communication, { def: SESSION_LIMIT_DEFAULT, found: 'body.results.total', where: 'qs' }); } { // Search-session properties carry neither a date nor an ordered ID. The session itself never @@ -347,11 +357,14 @@ export const webhooks = { export const EVENT_LIMITS_NOTE = 'Limits: up to 5 monitors per team, a monitor cannot be edited after it is created (delete and recreate it), a new monitor can take up to 7 days to start delivering, and delivery is best effort (4 attempts per event), so the same event can arrive more than once. Use the Event ID to skip repeats.'; -const evBase = ev.description +const evCore = ev.description .replace('Turning the Zap or scenario on', 'Turning the scenario on') - .replace('turning it off', 'turning it off') - .replace(' turn on "Include Property Record" to also look up the full property record.', ' use "Watch Property Monitor Events with Property Record" to also look up the full property record.') + .replace('; turn on "Include Property Record" to also look up the full property record.', '.') .replace(/ The optional property lookup is billed per event, per matched property\./, ''); +if (evCore.includes('Include Property Record') || evCore.includes('Zap')) throw new Error('monitor event description still has catalog-only wording'); +// The plain trigger points to the property-record version; the property-record version describes itself. +const evBase = evCore.replace('Limits:', 'To get the full property record with each event, use "Watch Property Monitor Events with Property Record (Instant, Billed)" instead. Limits:'); + catalogModules.watchMonitorEvents = { moduleType: 'instant_trigger', catalogKey: ev.key, @@ -367,7 +380,7 @@ catalogModules.watchMonitorEventsWithProperty = { catalogKey: ev.key, webhook: 'monitorEvents', label: 'Watch Property Monitor Events with Property Record (Instant, Billed)', - description: `${evBase} This version also looks up the full property record for every event. The lookup is billed per event, per matched property, on top of the monitor charges.`, + description: `${evCore} This version also looks up the full property record for every event and adds it to the output as "property". The lookup is billed per event, per matched property, on top of the monitor charges.`, communication: { url: api(ev.hydrate.path), method: 'POST', @@ -380,6 +393,42 @@ catalogModules.watchMonitorEventsWithProperty = { sampleFrom: () => ({ ...ev.sample, property: fx('property-lookup-all-attributes').results.properties[0] }), }; +// ---------- Make limit and pagination for the catalog list modules ---------- +const swapTakeForLimit = (m, param) => { + const i = m.params.findIndex((x) => x.name === 'take'); + if (i < 0) throw new Error(`${m.label} has no take parameter`); + m.params[i] = param; +}; +{ + const m = catalogModules.getPermits; + swapTakeForLimit(m, limitParam(25, { what: 'permits', extra: ' Each page is billed per property, so a Limit above 100 sends more than one billed request.' })); + offsetPaging(m.communication, { def: 25, found: 'body.result.meta.results.resultsFound', where: 'body' }); +} +{ + const m = catalogModules.investorBuybox; + swapTakeForLimit(m, limitParam(25, { what: 'investors' })); + offsetPaging(m.communication, { def: 25, found: 'body.result.meta.results.resultsFound', where: 'body' }); +} +{ + // Cursor paging only (skip is rejected). A full last page still returns a cursor, and the request + // that follows it is empty and not billed. + const m = catalogModules.investorProperties; + swapTakeForLimit(m, limitParam(25, { what: 'properties' })); + m.description = m.description + .replace('Use the "nextPageCursor" value of the last result as Page Cursor to read the next page.', 'Set a Limit and Make reads the pages for you; each result also carries the "nextPageCursor" of its page.') + .replace('(take sets the maximum cost of a call)', '(the Limit sets the maximum cost of a run)'); + if (/Page Cursor to read|take sets/.test(m.description)) throw new Error('investorProperties description still has the old paging text'); + const cur = m.params.find((x) => x.name === 'pageCursor'); + cur.label = 'Start from page cursor'; + cur.help = 'Optional. Leave empty to start at the first page. The Limit already pages for you, so use this only to continue an earlier run from its Next Page Cursor.'; + cursorPaging(m.communication, { def: 25, cursor: 'body.result.nextPageCursor' }); +} +{ + const m = catalogModules.listSearchSessions; + swapTakeForLimit(m, limitParam(50, { max: SESSIONS_MAX, what: 'search sessions' })); + offsetPaging(m.communication, { def: 50, found: 'body.results.total', where: 'qs', skip: 'skip' }); +} + // ---------- RPCs for dropdowns ---------- export const rpcs = { listSearchSessions: { diff --git a/scripts/generate.mjs b/scripts/generate.mjs index 3849543..ef37b5d 100644 --- a/scripts/generate.mjs +++ b/scripts/generate.mjs @@ -1,6 +1,7 @@ import fs from 'node:fs'; import { root, write, toInterface } from './lib.mjs'; import * as P from './params.mjs'; +import { offsetPaging } from './paging.mjs'; import { catalogModules, catalogGroups, webhooks, rpcs } from './catalog-modules.mjs'; const fx = (n) => JSON.parse(fs.readFileSync(`${root}/test/fixtures/${n}.json`, 'utf8')); @@ -61,13 +62,13 @@ const modules = { searchProperties: { moduleType: 'search', label: 'Search Properties', - description: `Finds properties that match your filters. ${BILL.search} A Max results value of 0 is billed as 1 record. One call returns at most 100 records.`, + description: `Finds properties that match your filters. ${BILL.search} Set a Limit to cap the records returned in total. Records are requested in pages of up to 100, and never more than the Limit are requested.`, communication: { url: '/v1/property/search', method: 'POST', body: { - searchCriteria: "{{omit(parameters, 'take', 'skip')}}", - options: { take: '{{ifempty(parameters.take, 25)}}', skip: u('parameters.skip') }, + searchCriteria: "{{omit(parameters, 'limit', 'skip')}}", + options: { skip: u('parameters.skip') }, }, response: { iterate: '{{body.results.properties}}', @@ -194,6 +195,7 @@ const modules = { }, }; +offsetPaging(modules.searchProperties.communication, { def: 25, found: 'body.results.meta.results.resultsFound', where: 'body', skip: 'skip' }); Object.assign(modules, catalogModules); const baseGroups = [ @@ -348,17 +350,21 @@ write('makecomapp.json', { }, origins: [ { - label: 'Make (set your zone)', - baseUrl: 'https://eu1.make.com/api', - appId: 'batchdata', + // The private app that this source was uploaded to (organization 3655001, zone us2). + // Make suffixed the app name, so the remote connection and webhook names differ from the local ones. + // Put your own Make API token in .secrets/apikey. To deploy to another zone or organization, + // copy this entry, change baseUrl and appId, and clear the idMapping. + label: 'Make us2: batchdata-r6b83g (private app)', + baseUrl: 'https://us2.make.com/api', + appId: 'batchdata-r6b83g', appVersion: 1, apikeyFile: '.secrets/apikey', idMapping: { - connection: [{ local: 'batchdata', remote: 'batchdata' }], + connection: [{ local: 'batchdata', remote: 'batchdata-r6b83g' }], module: idMap, function: [], rpc: rpcIdMap, - webhook: webhookIdMap, + webhook: webhookIdMap.map((w) => ({ ...w, remote: 'batchdata-r6b83g' })), }, }, ], diff --git a/scripts/paging.mjs b/scripts/paging.mjs new file mode 100644 index 0000000..c84d695 --- /dev/null +++ b/scripts/paging.mjs @@ -0,0 +1,56 @@ +// Make "Limit" and pagination for the modules that return lists of billed records. +// +// BatchData bills per record returned, so every page asks for at most the records still missing +// from the user's Limit: min(100, Limit - records already requested). Make stops paging as soon as +// the Limit is met (response.limit) or the API reports that no more records exist +// (pagination.condition). A pagination request REPLACES the original body (it only merges query +// string and headers), so a POST body is repeated whole, including searchCriteria. + +export const PAGE_SIZE = 100; +export const LIMIT_MAX = 5000; // Make allows 1 first request + 50 pagination requests per module run + +const clone = (v) => JSON.parse(JSON.stringify(v)); +// Records already requested before the page that Make is about to ask for (pagination.page is 2 then). +const done = `(pagination.page - 1) * ${PAGE_SIZE}`; +const limitOf = (def) => `ifempty(parameters.limit, ${def})`; +const skipOf = (skipParam) => (skipParam ? `ifempty(parameters.${skipParam}, 0)` : '0'); + +export const limitParam = (def, { max = LIMIT_MAX, what, extra = '' }) => ({ + name: 'limit', + type: 'uinteger', + label: 'Limit', + default: def, + required: true, + validate: { min: 1, max }, + help: `The maximum number of ${what} to return in total (1 to ${max}, default ${def}). They are requested in pages of up to ${PAGE_SIZE}, and never more than this number are requested.${extra}`, +}); + +const takeFirst = (def) => `{{min(${PAGE_SIZE}, ${limitOf(def)})}}`; +const takeNext = (def) => `{{min(${PAGE_SIZE}, ${limitOf(def)} - ${done})}}`; +const moreAfter = (def) => `${limitOf(def)} > ${done}`; + +// Skip/take paging. `where` is 'body' (POST, options.take/skip) or 'qs' (GET, take/skip). +// `found` is the response path with the total number of matches. +export const offsetPaging = (comm, { def, found, where, skip }) => { + const nextOffset = `{{${skipOf(skip)} + ${done}}}`; + comm.response.limit = `{{${limitOf(def)}}}`; + const condition = `{{${found} > ${skipOf(skip)} + ${done} && ${moreAfter(def)}}}`; + if (where === 'qs') { + comm.qs = { ...comm.qs, take: takeFirst(def) }; + comm.pagination = { qs: { take: takeNext(def), skip: nextOffset }, condition }; + return; + } + const next = clone(comm.body); + comm.body.options = { ...comm.body.options, take: takeFirst(def) }; + next.options = { ...next.options, take: takeNext(def), skip: nextOffset }; + comm.pagination = { body: next, condition }; +}; + +// Cursor paging (the API returns nextPageCursor; skip is not accepted). +export const cursorPaging = (comm, { def, cursor }) => { + comm.response.limit = `{{${limitOf(def)}}}`; + const next = clone(comm.body); + comm.body.options = { ...comm.body.options, take: takeFirst(def) }; + next.options = { ...next.options, take: takeNext(def), pageCursor: `{{${cursor}}}` }; + comm.pagination = { body: next, condition: `{{length(ifempty(${cursor}, '')) > 0 && ${moreAfter(def)}}}` }; +}; diff --git a/scripts/params.mjs b/scripts/params.mjs index 14f4cb8..31e20a3 100644 --- a/scripts/params.mjs +++ b/scripts/params.mjs @@ -1,5 +1,6 @@ import fs from 'node:fs'; import { root } from './lib.mjs'; +import { limitParam } from './paging.mjs'; const C = JSON.parse(fs.readFileSync(`${root}/scripts/constants.json`, 'utf8')); @@ -89,15 +90,8 @@ export const searchParams = [ }, ], }, - { - name: 'take', - type: 'uinteger', - label: 'Max results', - default: 25, - validate: { max: 100 }, - help: 'Up to 100 per run. You are billed per record returned, and a value of 0 is billed as 1 record.', - }, - { name: 'skip', type: 'uinteger', label: 'Skip', help: 'Number of records to skip, for paging.' }, + limitParam(25, { what: 'properties', extra: ' You are billed per record returned.' }), + { name: 'skip', type: 'uinteger', label: 'Skip', help: 'Number of records to skip before the first one returned. Leave empty to start at the first record.' }, ]; export const lookupParams = [ diff --git a/test/catalog.test.mjs b/test/catalog.test.mjs index fc42c80..6309325 100644 --- a/test/catalog.test.mjs +++ b/test/catalog.test.mjs @@ -142,7 +142,7 @@ test('reverse skip trace sends one request and keeps only matched rows', () => { }); test('permits, geocoding and investors build the documented requests', () => { - const p = request('getPermits', { address: { street: '1 Main St' }, status: { equals: 'active' }, allTags: { inList: ['solar'] }, issueDate: { minDate: '2026-01-02' }, take: 10 }); + const p = request('getPermits', { address: { street: '1 Main St' }, status: { equals: 'active' }, allTags: { inList: ['solar'] }, issueDate: { minDate: '2026-01-02' }, limit: 10 }); assert.equal(p.url, '/v2/property/get-property-permits'); assert.deepEqual(p.body, { request: { address: { street: '1 Main St' }, status: { equals: 'active' }, allTags: { inList: ['solar'] }, issueDate: { minDate: '2026-01-02' } }, @@ -190,7 +190,7 @@ test('monitor creation is stream delivery only and never offers a webhook delive test('polling triggers: sessions by creation date, session properties by poll time, billing disclosed', () => { const s = comm('watchSearchSessions'); - assert.deepEqual(render(s.qs, {}), { take: 100 }); + assert.deepEqual(render(s.qs, { parameters: {} }), { take: 100 }); assert.equal(s.response.trigger.date, '{{item.createdAt}}'); const sample = read(files('watchSearchSessions').samples); assert.equal(render(s.response.trigger.id, { item: sample }), sample.id); diff --git a/test/fixtures/investor-buybox.json b/test/fixtures/investor-buybox.json new file mode 100644 index 0000000..b23e3a4 --- /dev/null +++ b/test/fixtures/investor-buybox.json @@ -0,0 +1,22 @@ +{ + "status": { + "code": 200, + "text": "OK" + }, + "result": { + "investors": [ + { + "id": "inv_1" + } + ], + "properties": [], + "meta": { + "apiVersion": "2", + "results": { + "resultCount": 25, + "resultsFound": 137 + }, + "requestId": "req_8f2a1c" + } + } +} diff --git a/test/fixtures/investor-properties.json b/test/fixtures/investor-properties.json new file mode 100644 index 0000000..989f0bd --- /dev/null +++ b/test/fixtures/investor-properties.json @@ -0,0 +1,20 @@ +{ + "status": { + "code": 200, + "text": "OK" + }, + "result": { + "properties": [ + { + "propertyId": "P-85021-7d3e9f" + } + ], + "resultsFound": 48, + "resultCount": 1, + "nextPageCursor": "HOJZT+Y+Tt/XkfGLHnpi5Q==", + "meta": { + "apiVersion": "6.94.0", + "requestId": "req_3c7d9e" + } + } +} diff --git a/test/fixtures/property-permits.json b/test/fixtures/property-permits.json new file mode 100644 index 0000000..5f890ec --- /dev/null +++ b/test/fixtures/property-permits.json @@ -0,0 +1,21 @@ +{ + "status": { + "code": 200, + "text": "OK" + }, + "result": { + "meta": { + "results": { + "resultCount": 1, + "resultsFound": 1 + }, + "apiVersion": "6.61.0", + "requestId": "xi8WLSU6AIgRRlY" + }, + "permits": [ + { + "permitNumber": "BLD-1" + } + ] + } +} diff --git a/test/fixtures/search-sessions.json b/test/fixtures/search-sessions.json new file mode 100644 index 0000000..7d03621 --- /dev/null +++ b/test/fixtures/search-sessions.json @@ -0,0 +1,20 @@ +{ + "status": { + "code": 200, + "text": "OK" + }, + "results": { + "sessions": [ + { + "id": "ssn_0001", + "totalDelivered": 120, + "lastRequestAt": "2026-04-02T10:00:00Z", + "createdAt": "2026-04-01T09:15:00Z" + } + ], + "total": 1, + "meta": { + "requestId": "req_1" + } + } +} diff --git a/test/iml.mjs b/test/iml.mjs index d977040..d4e62aa 100644 --- a/test/iml.mjs +++ b/test/iml.mjs @@ -5,6 +5,7 @@ const fns = { ifempty: (a, b) => (isEmpty(a) ? b : a), __if: (c, a, b) => (c ? a : b), omit: (o, ...keys) => Object.fromEntries(Object.entries(o || {}).filter(([k]) => !keys.includes(k))), + min: Math.min, length: (v) => (v === undefined || v === null ? 0 : v.length), first: (a) => (Array.isArray(a) ? a[0] : undefined), lower: (s) => String(s ?? '').toLowerCase(), diff --git a/test/pagination.test.mjs b/test/pagination.test.mjs new file mode 100644 index 0000000..d7dff12 --- /dev/null +++ b/test/pagination.test.mjs @@ -0,0 +1,207 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import path from 'node:path'; +import { render } from './iml.mjs'; + +const root = path.resolve(path.dirname(new URL(import.meta.url).pathname), '..'); +const read = (rel) => JSON.parse(fs.readFileSync(path.join(root, rel), 'utf8')); +const comm = (name) => read(`modules/${name}/${name}.communication.iml.json`); +const fixture = (name) => read(`test/fixtures/${name}.json`); + +const PAGED = ['searchProperties', 'investorProperties', 'investorBuybox', 'getPermits', 'listSearchSessions', 'watchSearchSessions']; + +// Response paths, checked against the BatchData OpenAPI examples and test/fixtures. +const has = (obj, p) => p.split('.').reduce((n, k) => (n === undefined || n === null ? undefined : n[k]), obj) !== undefined; +test('the response paths the pagination reads exist in real BatchData responses', () => { + assert.ok(has(fixture('property-search'), 'results.meta.results.resultsFound')); + assert.ok(has(fixture('property-search'), 'results.properties')); + for (const [file, p] of Object.entries({ + 'search-sessions': 'results.total', + 'investor-properties': 'result.nextPageCursor', + 'investor-buybox': 'result.meta.results.resultsFound', + 'property-permits': 'result.meta.results.resultsFound', + })) assert.ok(has(fixture(file), p), `${file} ${p}`); +}); + +// A fake BatchData endpoint that enforces the real rules and counts billed records. +const API = { + searchProperties: (n) => { + const rows = Array.from({ length: n }, (_, i) => ({ _id: `p${i}` })); + return (req, bill) => { + const { take = 25, skip = 0 } = req.body.options; + assert.ok(take >= 1 && take <= 100, `take ${take} outside 1..100`); + assert.ok(req.body.searchCriteria?.query, 'searchCriteria must be repeated on every request'); + const page = rows.slice(skip, skip + take); + bill(page.length); + return { results: { properties: page, meta: { results: { resultCount: page.length, resultsFound: n } } } }; + }; + }, + getPermits: (n) => { + const rows = Array.from({ length: n }, (_, i) => ({ permitNumber: `x${i}` })); + return (req, bill) => { + const { take = 25, skip = 0 } = req.body.options; + assert.ok(take >= 1 && take <= 100, `take ${take}`); + assert.ok(req.body.request?.propertyId === 'P-1', 'the property must be repeated on every request'); + bill(1); // billed per property requested, not per permit + const page = rows.slice(skip, skip + take); + return { result: { permits: page, property: {}, meta: { results: { resultCount: page.length, resultsFound: n } } } }; + }; + }, + investorBuybox: (n) => { + const rows = Array.from({ length: n }, (_, i) => ({ id: `i${i}` })); + return (req, bill) => { + const { take = 25, skip = 0 } = req.body.options; + assert.ok(take >= 1 && take <= 100, `take ${take}`); + assert.ok(req.body.searchCriteria, 'searchCriteria must be repeated on every request'); + const page = rows.slice(skip, skip + take); + bill(page.length); + return { result: { investors: page, meta: { results: { resultCount: page.length, resultsFound: n } } } }; + }; + }, + investorProperties: (n) => { + const rows = Array.from({ length: n }, (_, i) => ({ propertyId: `p${i}` })); + return (req, bill) => { + const { take = 25, pageCursor, skip } = req.body.options; + assert.equal(skip, undefined, 'skip is rejected by this endpoint'); + assert.ok(take >= 1 && take <= 100, `take ${take}`); + assert.equal(req.body.searchCriteria.investorId, 'inv-1', 'searchCriteria must be repeated on every request'); + const from = pageCursor ? Number(pageCursor.replace('c', '')) : 0; + const page = rows.slice(from, from + take); + bill(page.length); + // A full last page still returns a cursor. + return { result: { properties: page, resultsFound: n, nextPageCursor: page.length === take ? `c${from + take}` : undefined } }; + }; + }, + listSearchSessions: (n) => { + const rows = Array.from({ length: n }, (_, i) => ({ id: `s${i}` })); + return (req) => { + const { take = 50, skip = 0 } = req.qs; + assert.ok(take >= 1 && take <= 100, `take ${take}`); + return { results: { sessions: rows.slice(skip, skip + take), total: n } }; + }; + }, +}; +API.watchSearchSessions = API.listSearchSessions; + +// Plays a module the way Make does: first request, then pagination requests (page 2, 3, ...) +// until response.limit is reached, the condition is false, or the API returns nothing. +const run = (name, parameters, n, { requestBase } = {}) => { + const c = comm(name); + const endpoint = API[name](n); + let billed = 0; + const requests = []; + const send = (req) => { + requests.push(req); + return endpoint(req, (k) => (billed += k)); + }; + const scope = (extra = {}) => ({ parameters, ...extra }); + const items = []; + const list = (body) => render(c.response.iterate, { body }); + const limit = render(c.response.limit, scope()); + let body = send({ body: render(c.body, scope()), qs: render(c.qs, scope()) }); + items.push(...list(body)); + for (let page = 2; page < 60 && items.length < limit; page++) { + const ctx = scope({ body, pagination: { page } }); + if (!render(c.pagination.condition, ctx)) break; + const base = { body: render(c.body, scope()), qs: render(c.qs, scope()) }; + const req = { + body: c.pagination.body ? render(c.pagination.body, ctx) : base.body, + qs: c.pagination.qs ? { ...base.qs, ...render(c.pagination.qs, ctx) } : base.qs, + }; + body = send(req); + const got = list(body); + if (!got.length) break; + items.push(...got); + } + void requestBase; + return { items: items.slice(0, limit), requests, billed }; +}; + +const SCENARIOS = { + searchProperties: { params: { query: 'Phoenix, AZ' }, billedPerRecord: true }, + investorProperties: { params: { investorId: 'inv-1' }, billedPerRecord: true }, + investorBuybox: { params: { propertyId: 'P-1' }, billedPerRecord: true }, + getPermits: { params: { propertyId: 'P-1' }, billedPerRecord: false }, + listSearchSessions: { params: {}, billedPerRecord: false }, + watchSearchSessions: { params: {}, billedPerRecord: false }, +}; + +test('every paged module has a Limit parameter and sends limit and pagination directives', () => { + for (const name of PAGED) { + const c = comm(name); + const m = read('makecomapp.json').components.module[name]; + const params = read(m.codeFiles[m.moduleType === 'trigger' ? 'staticParams' : 'mappableParams']); + const limit = params.find((p) => p.name === 'limit'); + assert.ok(limit, `${name} has no Limit parameter`); + assert.equal(limit.label, 'Limit'); + assert.ok(limit.validate.min >= 1 && limit.validate.max >= limit.default, name); + assert.ok(!params.some((p) => p.name === 'take'), `${name} still has the old take parameter`); + assert.match(c.response.limit, /parameters\.limit/, name); + assert.ok(c.pagination?.condition, `${name} has no pagination condition`); + assert.ok(c.pagination.body || c.pagination.qs, name); + } +}); + +for (const name of PAGED) { + const { params, billedPerRecord } = SCENARIOS[name]; + const cap = name.includes('Sessions') ? 100 : 100000; + + test(`${name}: returns exactly the Limit and never asks for more records than the Limit`, () => { + for (const [limit, available] of [[1, 500], [25, 500], [100, 500], [101, 500], [250, 1000], [100, 100], [200, 200]]) { + if (name.includes('Sessions') && (limit > 100 || available > 100)) continue; + const r = run(name, { ...params, limit }, Math.min(available, cap)); + assert.equal(r.items.length, Math.min(limit, available), `${name} limit ${limit}`); + const taken = r.requests.map((q) => (q.body?.options ?? q.qs).take); + assert.ok(taken.every((t) => t >= 1 && t <= 100), `${name} take values ${taken}`); + assert.ok(taken.reduce((a, b) => a + b, 0) <= limit, `${name} requested ${taken} for limit ${limit}`); + if (billedPerRecord) assert.ok(r.billed <= limit, `${name} billed ${r.billed} for limit ${limit}`); + } + }); + + test(`${name}: stops when the data runs out and pages without duplicates`, () => { + if (name.includes('Sessions')) { + const r = run(name, { ...params, limit: 100 }, 37); + assert.equal(r.items.length, 37); + assert.equal(r.requests.length, 1); + return; + } + const r = run(name, { ...params, limit: 1000 }, 230); + assert.equal(r.items.length, 230); + assert.equal(new Set(r.items.map((i) => JSON.stringify(i))).size, 230, `${name} returned duplicates`); + assert.ok(r.requests.length <= 4, `${name} sent ${r.requests.length} requests`); + }); + + test(`${name}: the default Limit is a single request`, () => { + const r = run(name, { ...params }, 1000); + const def = name.includes('Sessions') ? (name === 'watchSearchSessions' ? 100 : 50) : 25; + assert.equal(r.items.length, Math.min(def, 1000)); + assert.equal(r.requests.length, 1); + }); +} + +test('searchProperties: the Skip offset is kept on every page', () => { + const r = run('searchProperties', { query: 'Tempe', limit: 150, skip: 40 }, 500); + assert.equal(r.items.length, 150); + assert.equal(r.items[0]._id, 'p40'); + assert.equal(r.items.at(-1)._id, 'p189'); + assert.deepEqual(r.requests.map((q) => [q.body.options.skip, q.body.options.take]), [[40, 100], [140, 50]]); +}); + +test('searchProperties: limit and skip are never sent as search filters', () => { + const r = run('searchProperties', { query: 'Tempe', limit: 150, skip: 40 }, 500); + for (const q of r.requests) assert.deepEqual(Object.keys(q.body.searchCriteria), ['query']); +}); + +test('investorProperties: a full last page ends the run without an extra request when the Limit is met', () => { + const r = run('investorProperties', { investorId: 'inv-1', limit: 200 }, 500); + assert.equal(r.items.length, 200); + assert.equal(r.requests.length, 2); + assert.equal(r.requests[1].body.options.pageCursor, 'c100'); +}); + +test('getPermits: one request covers up to 100 permits, so a small Limit is billed once', () => { + const r = run('getPermits', { propertyId: 'P-1', limit: 60 }, 500); + assert.equal(r.requests.length, 1); + assert.equal(r.billed, 1); +}); diff --git a/test/replay.test.mjs b/test/replay.test.mjs index 094d995..bcdb75b 100644 --- a/test/replay.test.mjs +++ b/test/replay.test.mjs @@ -19,7 +19,7 @@ const respond = (name, body, extra = {}) => { }; const iterate = (name, body) => render(comm(name).response.iterate, { body }); -test('search: only filled filters are sent, take defaults to 25', () => { +test('search: only filled filters are sent, the Limit defaults to 25', () => { const r = request('searchProperties', { query: 'Phoenix, AZ', quickLists: ['absentee-owner', 'not-vacant'], @@ -38,9 +38,9 @@ test('search: only filled filters are sent, take defaults to 25', () => { }); }); -test('search: take and skip are passed and kept out of the criteria', () => { - const r = request('searchProperties', { query: 'Tempe', take: 0, skip: 50 }); - assert.deepEqual(r.body, { searchCriteria: { query: 'Tempe' }, options: { take: 0, skip: 50 } }); +test('search: limit and skip are passed and kept out of the criteria', () => { + const r = request('searchProperties', { query: 'Tempe', limit: 10, skip: 50 }); + assert.deepEqual(r.body, { searchCriteria: { query: 'Tempe' }, options: { take: 10, skip: 50 } }); }); test('search: output is each property plus the result counts', () => { diff --git a/test/structure.test.mjs b/test/structure.test.mjs index ad0cd8a..de4a837 100644 --- a/test/structure.test.mjs +++ b/test/structure.test.mjs @@ -146,3 +146,24 @@ test('every module has samples that match its interface top-level names', () => assert.deepEqual(iface, sample, key); } }); + +test('makecomapp.json documents the us2 private app as an origin', () => { + const o = app.origins[0]; + assert.equal(o.baseUrl, 'https://us2.make.com/api'); + assert.equal(o.appId, 'batchdata-r6b83g'); + assert.equal(o.idMapping.connection[0].remote, 'batchdata-r6b83g'); + assert.equal(o.apikeyFile, '.secrets/apikey'); +}); + +test('the README is written for users', () => { + const text = fs.readFileSync(path.join(root, 'README.md'), 'utf8'); + assert.ok(!/not yet imported|BatchLeads|BatchDialer|zapier|n8n/i.test(text)); + assert.ok(!/\$\s?\d/.test(text), 'the README must not state prices'); + assert.match(text, /support@batchservice\.com/); + assert.match(text, /API token/); + assert.ok(fs.existsSync(path.join(root, 'assets/logo-512.png'))); +}); + +test('no module description refers to itself', () => { + for (const [key, m] of Object.entries(app.components.module)) assert.ok(!m.description.includes(`"${m.label}"`), `${key} mentions itself`); +});