Anyone can generate 11 tables. Can the data survive 99 SQL assertions?
Every synthetic data tool produces something that looks right in a preview. The interesting question is what happens when a reviewer runs arbitrary JOINs and GROUP BYs against it. So we built an 11-table e-commerce schema with an M:N junction and a diamond dependency, wrote 99 assertions about what must be true of it, and had DuckDB check them. DuckDB had no part in generating the data, so the verifier and the generator share no code.
The two tools pass exactly the same 67 assertions. Misata passes 31 more, and there is no assertion Faker passes that Misata fails. Every one of those 31 differences sits in the same four categories: temporal causality, status implications, cross-table reconciliation, and geographic consistency. In other words, Faker is not bad at generating values. It is excellent at values. It simply has no concept of a fact that spans two rows.
| Category | Faker | Misata |
|---|---|---|
Structural Primary keys unique and not null, every foreign key resolving to a parent that exists. | 32/32 | 32/32 |
Domain Value ranges, formats, and enum membership. Is a price positive, is a state a real state, is a zip five digits. | 20/20 | 20/20 |
Temporal causality No child event predates the parent that caused it. An order before its customer signed up cannot happen. | 1/10 | 9/10 |
Status implications A status gates its dependent columns. An active subscription has no cancellation date; a cancelled order has no payments. | 0/10 | 10/10 |
Reconciliation Parent aggregates equal what child rows actually sum to, including through two joins, and child money never exceeds the parent's. | 0/8 | 8/8 |
Diamond dependency A denormalized copy agrees with its source. The price on a line item is the price of the product it points at. | 2/2 | 2/2 |
Geographic consistency One state per city, one city per zip, and the same city agreeing across tables. | 0/5 | 5/5 |
Derived arithmetic Computed columns satisfy the formula they were declared with. | 2/2 | 2/2 |
Distribution sanity The data is not degenerate: spread in the values, a heavy tail in the child counts, some parents with no children at all. | 10/10 | 10/10 |
| Total | 67/99 | 98/99 |
Where the 31 differences actually are
These four categories are the ones that need a generator to reason about more than one row at a time. Violating-row counts are from the real run, not illustrations.
Temporal causality Misata 9/10, Faker 1/10
- orders never precede customer signupMisata: pass · Faker: 806 violating rows
- subscriptions never precede customer signupMisata: pass · Faker: 248 violating rows
- support tickets never precede customer signupMisata: pass · Faker: 275 violating rows
- payments never precede their orderMisata: pass · Faker: 1,381 violating rows
- shipments never precede their orderMisata: pass · Faker: 904 violating rows
- returns never precede their orderMisata: pass · Faker: 144 violating rows
- order_items never reference a product created after the orderMisata: 253 violating rows · Faker: 633 violating rows · known red: Foreign-key sampling with temporal eligibility. Planned, and deliberately left visible.
- tickets resolved after they were createdMisata: pass · Faker: 253 violating rows
- subscriptions cancelled after they startedMisata: pass · Faker: 82 violating rows
- no order predates the shop's first customerMisata: pass · Faker: pass
Status implications Misata 10/10, Faker 0/10
- cancelled subscriptions have cancelled_atMisata: pass · Faker: 96 violating rows
- active subscriptions have no cancelled_atMisata: pass · Faker: 125 violating rows
- past_due subscriptions have no cancelled_atMisata: pass · Faker: 15 violating rows
- resolved tickets have resolved_atMisata: pass · Faker: 124 violating rows
- closed tickets have resolved_atMisata: pass · Faker: 55 violating rows
- open tickets have no resolved_atMisata: pass · Faker: 112 violating rows
- pending tickets have no resolved_atMisata: pass · Faker: 82 violating rows
- shipments only for shipped/completed/returned ordersMisata: pass · Faker: 268 violating rows
- returns only for returned/return_pending ordersMisata: pass · Faker: 270 violating rows
- cancelled orders have no paymentsMisata: pass · Faker: 132 violating rows
Reconciliation Misata 8/8, Faker 0/8
- orders.total_amount = sum(order_items.line_total), to the centMisata: pass · Faker: 2,500 violating rows
- customers.order_count = count(orders), exactlyMisata: pass · Faker: 451 violating rows
- customers.lifetime_value = sum(payments via orders), to the centMisata: pass · Faker: 500 violating rows
- grand total: sum(orders.total_amount) = sum(order_items.line_total)Misata: pass · Faker: 1 violating rows
- grand total: sum(customers.lifetime_value) = sum(payments.amount)Misata: pass · Faker: 1 violating rows
- grand total: sum(customers.order_count) = count(orders)Misata: pass · Faker: 1 violating rows
- refund never exceeds the order totalMisata: pass · Faker: 43 violating rows
- payments per order never exceed the order totalMisata: pass · Faker: 813 violating rows
Geographic consistency Misata 5/5, Faker 0/5
- customers: one state per cityMisata: pass · Faker: 5 violating rows
- customers: one city per zipMisata: pass · Faker: 1 violating rows
- addresses: one state per cityMisata: pass · Faker: 23 violating rows
- addresses: one city per zipMisata: pass · Faker: 3 violating rows
- city/state pairs agree across customers and addressesMisata: pass · Faker: 29 violating rows
Where both tools pass, and why that matters
structural, domain, diamond dependency, derived arithmetic, distribution sanity: both score full marks on all 66 assertions in these categories. This is the part of the benchmark that keeps it honest. A benchmark where the author wins every category is usually a benchmark designed to be won. Foreign key integrity, value formats, real state codes, derived formulas, and distribution shape are all solved problems if you write the script carefully, and Faker solves them.
The one assertion Misata fails
An order line can reference a product created after the order was placed, 253 rows of 16,128. Foreign key sampling does not yet filter candidate parents by temporal eligibility. It is on the roadmap, it fails visibly in every run, and continuous integration treats it as an expected failure while failing the build on any *other* regression. It also fails if it starts passing, so the roadmap list cannot go stale.
An acceptance test that quietly shrinks to fit the product is not an acceptance test.
Method, stated plainly
The schema. 11 tables and 16,128 rows: customers, addresses, categories, products, orders, order_items as a true M:N junction pointing at both orders and products, payments, shipments, returns, subscriptions, support_tickets.
The verifier. Every assertion is a SQL query returning a count of violating rows, run by DuckDB against the emitted frames. Zero is a pass. Equality checks on money carry a one cent tolerance in the SQL itself, so exact means exact in SQL, not in the generator's own bookkeeping.
The Faker script. A careful but ordinary script. Foreign keys are sampled from real parent ids, formats come from Faker's own providers, state codes use every fairness flag Faker offers, prices follow the same declared formula, and line totals are computed rather than drawn. What it does not do is hand-build a constraint solver, because at that point you would not be benchmarking Faker, you would be writing the correctness layer yourself. That is precisely the work being measured.
Tools not scored. SDV has a ready code path in the harness but was not run here, so it gets no number rather than an estimated one. Seedfast cannot be scored offline at all: it is a closed CLI requiring an account and a live Postgres database. Both are runnable by anyone who has them, against the same 99 assertions.
Run it yourself
The harness is in the repository, MIT licensed. Nothing here needs to be taken on trust.
git clone https://github.com/rasinmuhammed/misata.git cd misata pip install -e ".[dev]" python -m benchmarks.gauntlet # 98/99 python -m benchmarks.gauntlet_compare --tool faker # 67/99
Measured 2026-07-25 on misata 0.8.9.2, Faker 40.11.1. Every number on this page came from those two commands. When the engine changes, the page is regenerated from a fresh run rather than edited.