Pulling Large Practice Datasets Efficiently: Paging, Deltas and Rate Limits

Two Jobs, Two Strategies

Most integrations that keep their own copy of practice data do two different jobs. The initial load reads everything once when a practice connects: every patient, every appointment in the window you care about, the ledger, treatment plans and reference tables. Ongoing sync keeps that copy current afterwards, ideally reading only what changed.

Treating them the same is the most common cause of slow integrations and rate-limit trouble: re-reading 20,000 patients every 15 minutes to find the twelve that changed. The CRMBridge API gives you separate tools for each job.

The Initial Load: Big Pages, Read Once

Every synced list endpoint — patients, appointments, transactions, treatment plans, notes, claims and the rest — pages with Skip and Take. Take defaults to 1,000 rows and can now be raised to 10,000, so a practice with 25,000 patients loads in three calls instead of twenty-five. Each response includes TotalCount, the number of matching rows across all pages, so you know up front how many pages to expect.

GET https://api.crmbridge.ai/v1/Getpatients
BusinessID / LocationId / APIToken headers
Content-Type: application/json

{ "Skip": 0, "Take": 10000 }

// Response (shortened)
{ "TotalCount": 24871, "Count": 10000, "Data": [ ... ] }

A few habits make the initial load faster and more reliable:

• Bigger is not always better. A page of 10,000 patients is roughly 12 MB of JSON and takes proportionally longer. If your HTTP client or serverless function has tight timeouts or memory, 2,000 to 5,000 per page is a comfortable middle.
• Window the time-based data. For appointments and transactions, send FromDate and ToDate and load in slices (a month or a quarter at a time) rather than asking for a decade in one request. Slices also give you natural restart points.
• Make it resumable. Record the last completed page or date slice. If the load is interrupted, continue from there instead of starting again.
• Note the start time. Write down when the initial load began, in UTC. It becomes the starting point of your first incremental pull, so nothing that changed during the load is missed.

Ongoing Sync: Ask Only for What Changed

Every list endpoint accepts LastModifiedFrom and LastModifiedTo. They filter on the LastModifiedTime field each row carries: the moment, in UTC, that the row last changed in CRMBridge’s copy of the practice data. Send the time of your last successful pull and only the rows that changed since then come back.

GET https://api.crmbridge.ai/v1/GetAppointments
BusinessID / LocationId / APIToken headers
Content-Type: application/json

{
  "LastModifiedFrom": "2026-10-29T08:15:00Z",
  "Skip": 0,
  "Take": 5000
}

• Use the time you started the previous pull, not the time it finished, and keep it in UTC. The window is inclusive, so a small overlap only means a row may arrive twice; upsert by id and it does no harm.
• Page the deltas too. After a busy day, or after the practice changes many records at once, an incremental pull can still be large.
• Match the practice’s sync schedule. The data behind these endpoints refreshes when the practice’s connector syncs. Polling every minute against a practice that syncs every 15 minutes mostly returns empty pages.

Deletions matter as much as changes. Subscribe to data-change webhooks (below) to hear about removed rows, or reconcile ids periodically for the entities where it matters to you.

Fewer Calls with Sideloading

Joining appointments to patients and providers in your own code means extra calls. The Include array asks for related fields in the same response: appointments with the patient’s contact details, patients with their provider’s name, transactions with the patient. It is supported on patients, appointments, transactions, treatment plans, notes and insurance claims.

{
  "FromDate": "2026-11-01",
  "ToDate": "2026-11-30",
  "Include": [ "Patient.FirstName", "Patient.LastName", "Patient.Cell",
               "Provider.ProviderFirstName", "Provider.ProviderLastName" ],
  "Take": 5000
}

Ask only for the fields you use: a request can join up to three related entities and twenty fields, and every included field makes each row larger.

Staying Inside the Rate Limits

Read endpoints share one budget per application and location: the Rate Limit set on your app in the developer portal, 60 calls a minute when left empty and up to 600. All of your servers calling for the same location draw from that one budget. Writes have their own allowance, so a large load never blocks a booking.

You never have to guess where you stand. Every limited response carries:

X-RateLimit-Limit: 60
X-RateLimit-Remaining: 12
X-RateLimit-Reset: 1793260860     (Unix seconds, UTC)
Retry-After: 23                   (on 429 only)

• When X-RateLimit-Remaining reaches 0, pause calls for that location until X-RateLimit-Reset.
• On a 429, wait Retry-After seconds and retry once. Do not retry in a tight loop.
• Get a token from POST /Auth once and reuse it for its 24-hour lifetime. Asking for a new token before every call wastes your budget and, when credentials are wrong, can get the calling address blocked.
• Larger pages are the cheapest way to save calls: loading 25,000 patients at 10,000 per page uses 3 calls of your budget instead of 25.

The API dashboard in the developer portal shows your calls per day by endpoint and location, and flags the patterns that waste budget.

Replace Polling with Webhooks

The most efficient pull is the one you do not make. Data-change webhooks send an Insert, Update or Delete event when a row reaches CRMBridge’s copy of the practice data, for the tables you subscribe to. A common pattern:

1. Run the initial load.
2. Subscribe to webhooks for the entities you keep, and apply events as they arrive.
3. Run a light incremental pull once or twice a day as a safety net, to catch anything missed while your endpoint was down.

To confirm your own writes, subscribe to the writeback outcome webhook instead of polling the writeback status.

Know How Fresh the Data Is

Synced endpoints answer from CRMBridge’s copy, so they keep working when the practice’s computer is off — and the data stops advancing. Every successful synced read tells you how current it is in its headers:

X-CRMBridge-Integrator-Status: Online
X-CRMBridge-Data-Status: Current
X-CRMBridge-Last-Sync: 2026-10-29T08:12:41Z
X-CRMBridge-Data-Age-Seconds: 197

If the status is Delayed or Stale, there is no point pulling harder; show users when the data was last updated instead. GET /GetIntegratorHealth gives the full picture for monitoring, and polling it every one to five minutes is plenty.

Synced Reads versus Live Reads

The /live/* endpoints ask the practice’s own system at the moment of the call. They are the right tool for “right now” questions — a patient added minutes ago, today’s schedule at the front desk, a current balance — and the wrong tool for bulk loading: practice-wide windows are capped at 31 days, each call counts against your read budget, and a practice answers at most four live queries at once. Load and sync with the synced endpoints; reach for live reads when a user is waiting on a single answer.

A Checklist

• Paging and filters go in the JSON body; loop until a page is shorter than Take.
• Initial load in large pages and date slices, resumable, with its start time recorded.
• Incremental pulls with LastModifiedFrom set to the start of the previous pull, in UTC; upsert by id.
• Include instead of follow-up calls, with only the fields you need.
• Read the X-RateLimit-* headers; honour Retry-After; reuse your token.
• Webhooks for changes, a daily incremental pull as a safety net.
• Check the freshness headers before deciding the data is wrong.