Skip to main content
Agents can pull inventory from customer-configured data sources and push mapping results to customer-configured data sinks. v1 supports two transport types — S3 and HTTPS webhook — for both inbound and outbound. SFTP and GCS are deferred.

​
Roles

RoleDirectionWhat gets moved
INVENTORY_SOURCEInboundCSV / JSON / XLSX files pulled on a schedule, uploaded via the existing inventory pipeline.
RESULTS_SINKOutboundMapping job results pushed on job complete (opt-in).

​
Inbound: pulling inventory

Required config:
  • bucket — bucket name (e.g. acme-inventory).
  • prefix — key prefix to watch (e.g. daily-uploads/).
  • region — AWS region (e.g. us-east-1).
  • secretRef — vault id containing accessKeyId + secretAccessKey (IAM keys with s3:GetObject + s3:ListBucket).
The agent calls pull_inventory_from_source(sourceId) either on its scheduleCron (default the agent’s cron) or on demand. New files since lastSyncedAt are downloaded, uploaded via the standard inventory pipeline, and recorded in agent_data_source_sync.manifest_json.

​
Auto-pull on schedule

Controlled by the rule set: 
"data_sources": {
  "auto_pull_on_schedule": true,
  "pause_source_after_failures": 3
}
After three consecutive failures, the source is auto-paused, a DATA_SOURCE_PULL_FAILED pending item is filed, and configured channels are notified.

​
Outbound: pushing results

Outbound auto-push is default off — data egress requires explicit opt-in. Enable it in the rule set:
"data_sinks": {
  "auto_push_on_job_complete": true,
  "retry_failed_push": 2
}
Same shape as the inbound config (bucket + prefix + region + secretRef), plus IAM keys with s3:PutObject permission. The agent writes {prefix}/job-{jobId}.{format} on successful job completion.
Push actions are not undoable — outbound webhooks are external side effects. Use the test endpoint before flipping auto-push on.

​
Secret handling

All credentials live in the existing secret vault used by BillingService. They are:
  • Never logged.
  • Never returned in full from the API (mt_byo_****1234 redaction).
  • Encrypted at rest.
  • Validated at save time via a low-cost ping (OPTIONS for webhooks, ListBucket for S3).

​
Testing a source / sink

Every source has a Test button (and a POST /api/v1/agents/{id}/data-sources/{sid}/test endpoint). It runs a read-only smoke check — no data moved, no sync record created. Returns a verbose pass/fail with the diagnostic message. Data sources tab

​
Sync history

The Inbound / Outbound subsections show the last N syncs per source. Per-row manifest, byte counts, errors. Drill in via GET /api/v1/agents/{id}/data-sources/{sid}/syncs.