Reconciliation

Keeping data in sync

Events get lost: a webhook fails while your service is being deployed, an outbox worker crashes, someone bulk-imports articles with a script that bypasses triggers. A reconciliation run compares the complete current data set with what was last sent and pushes only the differences. It is a safety net — schedule it nightly or weekly, and run it on demand after incidents.

How it works

1. Read the full data set from the ERP (a SELECT, not an API call to AIOTIC).
2. For each record compute the fingerprint of its AIOTIC shape.
3. Compare with the fingerprint stored from the last successful send:
   • different or unknown → PUT
   • identical → skip
4. Records that were sent before but are no longer in the ERP data set → DELETE.

Only steps 3 and 4 touch AIOTIC. A catalog of 200 000 articles with 50 changes results in 50 requests.

from aiotic.sync import ChangeEvent, HashStateStore, SyncEngine

engine = SyncEngine(client, state=HashStateStore("sync-state.db"), concurrency=6)

report = engine.reconcile(
    customers=(ChangeEvent.customer_upsert(c.no, name=c.name, city=c.city, vat_number=c.vat, email=c.email) for c in erp.active_customers()),
    products=(ChangeEvent.product_upsert(p.no, "nl", description=p.description) for p in erp.sellable_items()),
    mappings=(ChangeEvent.mapping_upsert(m.debtor, m.code, item_number=m.item, language_code="nl") for m in erp.customer_items()),
    delete_missing=True,
)
print(report)   # sent=52 deleted=3 unchanged=204871 failed=0 in 41.2s

aiotic sync reconcile --customers a.csv --products b.csv --mappings c.csv does the same from files.

First run after an existing integration

If AIOTIC already holds data that your state store has never seen, the first reconcile would re-send everything once. Avoid that by seeding the state from AIOTIC:

aiotic sync bootstrap-state     # reads /customer, /product, /customer-product and records fingerprints

Deleting what disappeared

delete_missing=True removes records that were sent earlier but are absent from the current data set. Make sure the data set you pass is complete for that kind — passing only "changed" rows with delete_missing=True would delete everything else. When in doubt run with delete_missing=False first and look at the report.

Fingerprints and what counts as a change

The fingerprint covers the AIOTIC fields only (what you would PUT). A change in an ERP field you do not map (e.g. a price) does not trigger a request. Empty strings are treated as null.

Scheduling

Data	Events available	Reconcile
Customers	yes	nightly
Customers	no	every 15–60 min (small table)
Products	yes	weekly + after bulk imports
Products	no	nightly (reading 200k rows locally is cheap; the diff is small)
Mappings	yes	nightly

Reconciliation is idempotent and safe to run concurrently with event-driven sync: both write the same fingerprints.