How to migrate an SFTP batch feed to an API

What matters here

• The file is a contract nobody wrote down. Recover its semantics before designing anything.
• A full-file replace means absence is deletion. No sequence of per-record API calls says that.
• Run both paths in parallel and diff them nightly. The diff is the specification you never had.
• Cut over one record type at a time, never the whole feed at once.
• Do not cut over until the diff has been empty across a month end. Month end is where the feed behaves differently.
• A truncated upload looks like a valid smaller file. Check the completeness marker before parsing.

Who this is for

You run platform or integration engineering. A batch file feeds something important, the API alternative has existed for years, and nobody has been willing to touch it.

How it works in practice

From a nightly file that nobody understands to an API integration somebody can change.

1. 1

   The file semantics are recovered

   Full replace or delta, ordering, key, and what an absent row means.

2. 2

   Completeness is checked before parsing

   A truncated upload is a valid smaller file and will be processed as one.

3. 3

   The API path is built to match those semantics

   Including the deletions that a file expresses by omission.

4. 4

   Both paths run in parallel

   The file stays authoritative and the API path writes to a shadow target.

5. 5

   The two are diffed every night

   And every difference is explained, not averaged away.

6. 6

   Cut over per record type, after a month end

   Because month end is where a batch feed behaves unlike any other day.

Paste these into Claude Code or Codex with the Tray Headless plugin installed. Each stage runs on its own. The systems named in them are the worked example rather than a requirement, and every prompt says so.

Once per project, run /tray-workflows:set-workspace to pick the workspace these build in. Point it at a sandbox first.

1. 1

   Set up and recover what the file actually means

   It is a contract, and nobody wrote it down.

   Headless skills build-workflow

   Copy

   Use build-workflow. The systems in play are NetSuite and the API that
   is replacing the feed, or whatever we run in those seats.
   
   Before designing anything, work out from real files what the feed
   actually means:
   
     Is each file a full replace of the record set, or a delta
     What is the key, and is it stable
     Does row order carry meaning
     What does an absent row mean: unchanged, or deleted
     Are there record types beyond the obvious one, distinguished by a
     column
     What happens on a day with no changes, an empty file or no file at all
   
   Take a month of files and diff them against each other. The answers are
   in the data, because they are not in a document and the person who knew
   them has left.
   
   The absent-row question is the one that matters most. If a full-file
   replace means absence is deletion, then a naive per-record API migration
   will never delete anything, and the target will accumulate records that
   should have gone years ago.

2. 2

   Check the file is complete before parsing it

   A truncated upload is a valid smaller file.

   Headless skills build-workflow

   Copy

   Use build-workflow. Before parsing anything:
   
     Look for a trailer record or control file with an expected row count,
     and reject a mismatch
     Reject a file whose size is wildly outside the recent range, which is
     usually a partial upload
     Reject a file that arrived while still being written, by requiring a
     marker file or a rename into place
     Handle no file at all as an alert, not as an empty change set
   
   A half-uploaded full-replace file is the worst case available here. It
   parses cleanly, contains a valid subset, and if absence means deletion it
   removes everything that had not uploaded yet.
   
   Record every file received with its checksum and row count, so a question
   about what arrived on a given night has an answer.

   Reject an unexpectedly small full-replace file rather than processing it. It is nearly always a partial upload, and processing it is not recoverable.

3. 3

   Build the API path to the same semantics, deletions included

   Absence is the part a per-record API cannot express.

   Copy

   Build the API path to reproduce what the file means, not what the API
   finds convenient:
   
     For a full replace, compute the set difference and issue explicit
     deletes for anything no longer present
     For a delta, respect ordering where the file has it
     Reproduce the same key, so records line up for comparison
     Handle the record types the file distinguishes by column value
   
   Keep the API path idempotent and keyed, so re-running a night is safe.
   You will re-run nights during the parallel period, repeatedly.
   
   Where the API cannot express something the file does, write that down and
   raise it rather than approximating. An approximation discovered during
   cutover is a rollback; the same approximation discovered during the
   parallel run is a design conversation.

4. 4

   Run both, diff nightly, and explain every difference

   The diff is the specification you never had.

   Headless skills tray-gotchas

   Copy

   Use tray-gotchas, then run both paths together. The file stays
   authoritative and writes to production; the API path writes to a shadow
   target.
   
   Every night, diff the two:
   
     Records in one and not the other
     Records present in both with different values, field by field
     Counts by record type
   
   Explain every difference. Do not summarise them as a match percentage.
   Each one is either a bug in the new path or a behaviour of the old feed
   nobody knew about, and both are worth the hour.
   
   Expect the interesting ones to appear at month end, on the day a record
   type appears that has not been seen in weeks, and on the day somebody
   sends a correction file. That is why the parallel period spans a month
   end rather than a fortnight.

5. 5

   Cut over one record type at a time

   Because a whole-feed cutover has one rollback and no diagnosis.

   Copy

   Cut over per record type, and only when that type has shown zero
   unexplained differences across a full month including month end.
   
   Keep the file path running and diffing after cutover, with the roles
   reversed: the API path is authoritative and the file becomes the check.
   Retire the file only when a quarter has passed without a difference.
   
   Report during the migration: differences by record type and by field,
   days since the last unexplained difference per type, file arrival times
   and completeness rejections, and which types have cut over.
   
   Days since the last unexplained difference is the number that decides the
   cutover. A team that cuts over on schedule instead of on that number is
   choosing the date over the evidence.

6. 6

   Check it end to end, then hand the cutover to platform engineering

   Because the sequence is a judgement, not a schedule.

   Headless skills tray-patterns

   Copy

   Run the per-step schema checks and the whole-workflow audit before this
   touches production. Feed a deliberately truncated file into the sandbox
   and confirm it is rejected rather than processed.
   
   Then open the same workflow in Tray Build so platform engineering can
   control which record types have cut over, adjust the completeness
   thresholds and read the nightly diff in the visual canvas.

What it connects to

The file is on one side, the API on the other, and the diff between them is the whole point.

Named systems are the ones most teams run, not the only ones that work. Each is an authentication in your Tray workspace, referenced by name, so the workflow never holds a credential. Where we have a connector page, the name links to it.

Questions people ask

Related guides