Only what
we measured.
These are the numbers we compute ourselves, on our own sample sets, with n shown. They describe the placeholder model (Qwen3-Embedding-8B), not toqan embed. Where a number is not measured yet, the page says so. There are no public leaderboard claims here.
meta.latency_msThe build
What was measured,
and on what.
- Status
- Build manifest not found yet.
Precomputed corpus vectors are real model output, built once and stored. Anything you type in the playground is embedded live. The catalog is synthetic and small, so every curve here is indicative, not a benchmark.
The dimension curve
How much quality survives
when vectors shrink.
| Dimensions | recall@10 vs 4,096-d | nDCG@10 vs hand labels | MB per million vectors |
|---|---|---|---|
| not yet measured | |||
How we measure it
The compress endpoint truncates the stored 4,096-d catalog vectors, re-normalises, and ranks the sample queries at each dimension. Recall is the overlap of the top 10 with the full-vector top 10. nDCG uses our hand-labelled relevant items. Memory is dimensions × 4 bytes × 1 million. The query set and catalog are small, so differences of a few points are noise.
Latency
How long a call takes,
from meta.latency_ms.
| Task | p50 (ms) | p95 (ms) | n |
|---|---|---|---|
| not yet measured | |||
Measured weaknesses
Where an embedding
gets it wrong.
From our research run on the placeholder model. Cosine similarity of two near-identical titles that differ in one decisive detail.
| Pair | What it shows | Cosine |
|---|---|---|
| “waterproof jacket” vs “non-waterproof jacket” | Negation barely moves the score | 0.873 |
| “1 liter” vs “2 liter” | Scores above the same size written another way | 0.935 |
| “1 liter” vs “1000 ml” | Same quantity, lower than the pair above | 0.909 |
| Translation pairs (EN with NL, DE, ES, PT) | Range of cosines for the same item in two languages, so a 0.9 threshold is wrong | 0.76 to 0.91 |
| Steer instructions on 4 queries × 8 products | Top-1 and top-2 did not change; scores spread wider (example 0.312 to 0.362) | n/a |
- Colour and size variants score almost as high as true duplicates, so the match tab recommends an embedding plus a structured attribute check.
- Exact identifiers such as SKUs return confident junk from a dense model alone; the hybrid tab shows the fix.
- Raw cosine moves with the instruction. Never compare scores across instruction settings.
Not yet run
Planned before any claim
appears here.
Instruct vs raw on Amazon ESCI
Does the query instruction help ranking quality on a public shopping set? Not yet run, so no number.
Dimension curve on ESCI
The same truncation curve as above, on a public set. Not yet run.
toqan embed scores
None until training finishes. The placeholder banner stays on every page until then.
We do not quote leaderboard ranks, language counts or model-card figures here. If we add one, it will carry a source link we checked.