Incremental exports with recovery | Community
Skip to main content

Incremental exports with recovery

  • April 5, 2025
  • 2 replies
  • 0 views
Robert28

 

Hi all,

I'm implementing a data sync thing using Zendesk's Incremental Export APIs (specifically the cursor-based version), and I want to confirm that the approach I'm using is sound. I'm primarily interested in making the incremental sync recoverable. So, if everything blows up in the middle of paging over incremental requests and I lose a valid cursors, I want to be able to fallback to timestamp. My idea is to use the timestamp right before a successful end_of_stream response as a recovery point.

 

Here's the pattern I'm thinking:

Initial Full Sync:
Before fetching any data at all, I record a start_time timestamp. I then perform a full sync using the standard REST APIs to pull all existing records.
I record the timestamp before any requests to catch anything that may have changed during the full sync.

 

First incremental sync or recovery from timestamp (no cursor):
  I start the first incremental sync using either last_recovery_time or start_time, and loop through the results until end_of_stream.

     If no records are returned: I can retry using last_recovery_time or start_time (the API doesn’t return a cursor in this case).

     If there’s an error: I can retry from the same timestamp.

     If successful: I store the after_cursor and record a valid_recovery_time just before the successful end_of_stream response.

 

With cursor:
  I use the stored cursor to continue syncing until end_of_stream.

    On failure: I discard the cursor and can fall back to valid_recovery_time, or start_time if no recovery time is available.

    On success: I store the new after_cursor and record a valid_recovery_time just before the successful end_of_stream response.

 

I will probably buffer a few extra minutes into the timestamps especially in the case of an error recovery.

Is this a correct way to recover? Are there any pitfalls or edge cases I should be aware of?

 

Thanks!

      2 replies

      E
      • Erica26
      • April 7, 2025
      Hi Robert, 
       
      It does sound like you have solid understanding of the Incremental Exports API with cursor based pagination.
       
      However, we do not recommend continuously syncing with the incremental exports API. The intended behavior is: initial request made, all data from start time to one minute from request time is grabbed, then export that data accordingly. Then periodically or when needed, make a call for more data. 
       
      Full process: Make the initial call with start_time > receive first page with 1000 results > record after_cursor value > check returned end_stream_value > if false make call with cursor/if true end loop and return
       
      As for error handling, there also isn't a way to buffer necessarily meaning if the start_time you provide is 10/13/22 10:03:00AM and you're making that call on 4/7/25 11:06:32AM but an update was made at 11:06:00AM, you will not get that update. For both tickets and ticket events, no updates within one minute of request time will be returned. This behavior is intentional just to prevent race conditions. 
       
      If there is an issue when exporting I would recommend just using the last after cursor value. Otherwise, as you said you will have to look at the data you have successfully exported, record the time value of the last result received, then make an entirely new initial request with that as your start_time.
       
      If you're really worried about having an issue in the middle of that 1000 results per page, you could also change how many results you receive per page. 1000 is just the default and max. 
       
      Ruben14
      • Ruben14
      • October 17, 2025

      Your approach lines up with how the cursor-based Incremental Exports behave in 2025: treat the after_cursor as the primary high-water mark, and keep a timestamp checkpoint only as a recovery fallback. 

      The one-minute “safety window” is expected, so it’s normal that updates within 60 seconds of the request aren’t included to avoid race conditions (see the Cursor-based Incremental Exports notes on the one-minute lag and end_of_stream). 
      (https://developer.zendesk.com/api-reference/ticketing/ticket-management/incremental_exports/#cursor-based-incremental-exports)

       

      Two small tweaks I’ve found helpful at scale. First, persist both the last successful after_cursor and the updated_at of the last processed record from the final page before end_of_stream; if a process crashes mid-page, you can safely restart from the cursor, and if the cursor is lost, you can restart from that updated_at minus a small buffer and de-dupe on upsert. 

      Second, keep your sync idempotent and tolerant to duplicates, since recovering from a timestamp will replay data by design. Cursor pagination semantics and page sizing are covered here if you want finer control over batch size. (https://developer.zendesk.com/documentation/api-basics/pagination/paginating-through-lists-using-cursor-pagination/)

       

      A couple of edge cases to watch out for: clock skew between your system and record timestamps, which is why using the API’s returned updated_at is safer than wall-clock time; and partial failures after you’ve read a page but before persisting it, which is where idempotent writes and storing progress after durable commit make the difference. With those guardrails, your “cursor first, timestamp as recovery” plan is solid.

      Terms & ConditionsCookie settingsAccessibility statement