Search and the demo database

How the model is searched, and how to generate a synthetic database to try everything.

Search#

Hybrid search over the published model (or the draft, in the UI): every entity, field, calculated field, relationship, filter, check and query example is a chunk of text (custom properties and example questions included). Three rankings are merged with reciprocal rank fusion:

• meaning: vector similarity with the embedding model;
• keywords: PostgreSQL full-text search;
• exact codes: column names, abbreviations and record ids match exactly, shortest first.

Chunks are derived from the model and never edited; a chunk's text only changes when its item changes, so embeddings are reused across versions. For agents and Alfred, results follow the caller's entitlements: restricted fields and classes not granted are left out. Search in the UI covers the whole model's definitions, like the model pages. Searches by agents and Alfred that find nothing are kept as questions nobody could answer (Settings > Maintenance): a backlog of missing synonyms, terms or entities.

Without an embedding model (remote embeddings and no key), search is keyword and exact-code only.

The demo database#

A synthetic database generated from whatever model is published, to try everything before connecting real data. Admins build it on Data → Database (not while a connector is in use). It lives in the app's PostgreSQL as a separate database.

Options:

• Scale: small, medium or large (0.35×, 1×, 2.5× a base of 120 rows, times a factor per entity kind: reference tables are small, transactions and facts large; a code table gets one row per allowed code).
• Seed: the same seed and model give the same data.
• Realistic values from the AI model: one chat-model call per entity proposes fictional values for its free-text columns; only the model's names and descriptions are sent. Without a key, or unticked, values are guessed from types and column names.

How the data is generated (deterministic code, nothing about a particular model):

• Every entity is a table and every attribute a column (calculated fields are not columns).
• Keys are unique and follow the field's example or format (an example AB12345 gives AB + 5 digits), else <ENTITY>-000001.
• Every foreign key points at a real row. Every parent gets a child before any gets two, then a long tail (a few busy parents, many quiet ones). Required (many-to-one) links are never empty; optional ones sometimes are.
• Chains agree: when a row reaches the same ancestor through two links (an order line's order and its subscription), it is the same row; a parent row is added when none fits.
• Self-links form shallow trees (up to 4 levels).
• Allowed values are used (the first ones more often); other values are guessed from type and column name (emails, phones, names, cities, amounts, rates, counts, dates in order: created before closed).
• Personal and restricted values are fictional.

The build runs in the background with progress, then validates the result; its findings appear in the Model check. Rebuilding replaces all data. When the published model changes, the page says the database was built from an older version.

The demo database has 8 read-only database roles, one per combination of the confidential, personal and restricted entitlements, each with column grants matching it and an 8-second statement timeout: a second line of defence under the query checks.

Copy it to other databases to try connectors on the same data (start docker-compose.demo.yml first), from the app container:

python -m studio.synth.export sqlite  /app/data/samples/demo.sqlite
python -m studio.synth.export duckdb  /app/data/samples/demo.duckdb
python -m studio.synth.export mariadb host=sampledb-maria port=3306 user=root password=... database_prefix=
python -m studio.synth.export clickhouse host=sampledb-ch port=8123 user=default password=...

SQLite has no schemas, so its tables are flat (the demo for "generate a model from this database"); the others keep the model's schemas.