NPUMark methodology — score 3.2

NPUMark measures how many TOPS each AI engine in a computer — the NPU, the GPU, the CPU, through every runtime that reaches it — can actually produce, and how fast it produces correct answers on real neural networks, and it proves where the work ran. This document is the complete definition of the score; anything that can move a number is versioned (§10).

1. Principle

The score is measured TOPS — no multiplier, no marketing figure. Peak TOPS means every multiplier busy every cycle on one large matrix multiply. NPUMark measures that peak itself, on the device, in every number format each engine runs, with the math checked — and reports beside it what real networks get, because real networks at one request at a time never reach the peak.

Precision is the vendor’s implementation choice; answer quality is the constraint. Engines differ in what they accelerate — INT8, FP16, BF16, AMD’s Block FP16, Qualcomm’s INT16 activations, 4-bit weights. NPUMark does not force one format on everyone: the fastest format that gets the answer right counts. The rules that decide what counts are fixed before any engine is measured, and no number is ever chosen because it looks good.

2. The score

NPUMark score      = max over engines of peak(engine)
peak(engine)       = max over the peak test's configurations of TOPS   (§3)
delivered(engine)  = geomean over the scored real networks of TOPS     (§4)
TOPS               = FLOPs per inference (2 ops per multiply-accumulate) × inferences/s

The NPUMark score is the most TOPS the device’s best engine produced, labeled with the engine and the number format: “49.3 TOPS · GPU (MLX) · BF16”. FLOPs use the 2-ops-per-MAC convention. Each engine’s peak and its delivered TOPS are reported beside the score; the gap between them is what real networks leave on the table.

An engine is a runtime’s target: on macOS, Core ML’s Neural Engine, GPU and CPU, and the GPU through MLX (Core ML’s GPU path doesn’t reach the M5 GPU’s per-core Neural Accelerators; MLX does); on Windows and Linux, each ONNX Runtime execution provider’s device.

Several engines can run on one processor — on a Windows laptop the NVIDIA GPU through DirectML, CUDA and TensorRT, the Intel integrated GPU through DirectML and OpenVINO. Results are therefore shown per processor (CPU, integrated GPU “iGPU”, discrete GPU “dGPU”, NPU), each with its best engine and the others beside it, and the score’s label names the processor and the runtime: “55.0 TOPS · dGPU · RTX 4070 Laptop (TensorRT) · FP16”. Every engine is still measured and scored on its own; the grouping changes no number.

Local-only engines. TensorRT for RTX runs by default where Windows ML offers it, but its results aren’t published (until NVIDIA’s written permission to publish them is in hand): the local report shows them and may lead with them, while the uploaded run, the server’s result page and the box total leave them out and are scored without them. The local report links the uploaded result once it is sent. A run with local-only results joins no local series (comparable: false).

3. The peak test

Chains of large matrix multiplies — the most favorable work there is:

x[S, D] -> L x (linear D->D, relu)        FLOPs = 2 · S · D² · L
shapes: 512×2048, 8 layers  and  2048×2048, 4 layers

Shapes were chosen by sweep on an M5 Max with native timing: the Neural Engine peaks on narrow chains (512×2048: 25.4 TOPS INT8, 13.9 FP16; at D = 4096 it drops to 14–16 INT8, 9 FP16), GPUs on wider ones (2048×2048: Core ML 27–30, MLX 47–51).

• Every format each engine runs: Core ML FP16 and INT8 (W8A8, the Neural Engine’s int8 path); ONNX FP32, FP16 and INT8 QDQ; MLX FP16 and BF16; LiteRT FP16 and INT8 (A8W8) on Android.
• Generated by NPUMark’s build pipeline from fixed seeds, shipped and hashed: weights are seeded (He-scaled); the pipeline builds every format once, with the canonical input, two eval inputs and the float32 reference outputs (stored as FP16), and ships them in the workload pack, where each file is hashed in the manifest and checked on every run (§8) like any other workload.
• Correct math: outputs over two eval inputs must reach 20 dB SNR against the shipped float32 reference (INT8 lands near 30 dB, FP16 above 50; a skipped layer or garbage lands far below), and the timed loop’s outputs must match the untimed ones. A confirmed pass is required — unverified is not enough.
• On the engine: on an NPU or GPU at least 90% of the work must be verified to have run there, or the TOPS would be credited to the wrong engine.
• Repeats: every configuration runs once; those within 90% of the engine’s best first attempt are measured until they have 5 attempts. When the fastest 3 of those 5 agree within 2%, the configuration’s peak is their median; otherwise it runs to 9 attempts and its peak is the median of its 5 fastest — a peak is what the engine reaches when a cold load does not land in a slow episode. An engine’s peak is its best configuration’s figure; its stability is still judged on every attempt it ran (§7).

3a. The large-batch peak (discrete GPUs)

The peak test’s shapes keep laptop NPUs and GPUs busy; a large discrete or data-centre GPU reads far below its rate at them. The large-batch peak runs the same chain at 8192×4096, 4 layers (ONNX FP32, FP16 and INT8 QDQ), with the same seeds, gates, attempt rules and math check, on every discrete GPU (on every GPU with --large-peak). It is reported per engine and per role as a labelled figure beside the score — never part of it: the NPUMark score stays the peak test’s (§2). Screening never compares the two tiers. The files ship in the Windows and Linux packs only.

3b. The box total (several cards)

A machine with two or more devices of one role — eight GPUs in a workstation, several NPU cards — gets one identity per card (identical cards are numbered in bus order: “dGPU 1 … dGPU 8”) and every card runs the whole plan, so each card’s peak and delivered TOPS are measured and shown. After every scored stage, the box stage runs each card’s own best solo peak configuration on all cards at the same time, each in its own worker, released together into one shared 5-second window:

box total(role) = median over 3 attempts of Σ over cards of TOPS inside the shared window
scaling         = box total ÷ Σ over cards of solo peak

Each card counts the calls that completed inside the window by their share of it, and checks one output against the reference; an attempt counts only when every card ran and passed. The box total is not the sum of the solo peaks: cards that share power, cooling or a PCIe link show it in the scaling figure. The CPU is never summed. Like the large-batch peak, the box total is a labelled figure beside the score, never part of it. The report draws a bar per card (solo and in the box run); a card under 90 % of the median of its identical cards is flagged (“check its cooling, power and PCIe link”). The results server recomputes the box total from its attempts and refuses one above 1.05 × the cards’ solo peaks.

4. Real networks (delivered TOPS)

Scored networks (suite 3.2.1): MobileNetV3-Large (classification), ResNet-50 (classification), DeepLabV3-MobileNetV3 (segmentation), Granite Embedding 30M (sentence embeddings). Granite Embedding 125M and SmolLM2-360M prefill/decode are measured on request (--extras) and reported, unscored. The language models are chosen for the terms of their training data as well as their licenses (Apache-2.0; IBM trained the Granite encoders only on permissively licensed data). Every shipped input is a CC0 Wikimedia Commons photo or text written for NPUMark, and int8 calibration uses CC0 photos no eval set contains. On macOS each network also ships as an MLX function (FP16; weight-only INT8 for the transformers), traced from the same weights as every other format.

A network result counts toward an engine’s delivered TOPS only if all hold:

1. its outputs over the workload’s eval set pass the standard gate (§5);
2. the variant is validated (passed its gate when built) and in the standard division (§9);
3. on an NPU or GPU, the backend verifies that at least 50% of the work ran on that engine (Core ML compute plan; ONNX Runtime graph assignment and profiler; LiteRT’s profiled run: its Invoke time minus the time of the ops that ran as CPU kernels, the op share kept as accel_ops_pct; MLX has no silent per-op fallback, so a result that ran ran on the GPU). Work is time, not op count: one heavy op on the CPU outweighs many light ones on the accelerator. Unknown placement is not eligible;
4. it wasn’t skipped (known runtime abort, §8) and it measured something.

Variant ladder (off by default; plan-shrink-02 Phase 7). A run may use the default ladder (data/default_ladder.json, --ladder default): on each engine a scored network runs one representative label per number family — Core ML FP16 and INT8 (A8W8), MLX FP16, ONNX Runtime FP32 and A8W8 on the CPU, FP16 on GPUs and the CoreML EP, A8W8 (and A16W8) on QNN; the peak test always runs every format. Every other variant runs with --all-variants or the app’s All variants setting. Shipped off: a default run measures every variant each engine supports, as above. Whether to switch it on is decided on the hardware runs; a run using it is labelled in its record.

An engine missing any scored network gets a partial figure — shown, never ranked. The high-accuracy figure applies the same rules with the near-lossless gate and requires a confirmed pass.

5. Accuracy gates

Every network ships an eval set (n inputs with full-precision reference outputs). After the timed loop, each variant runs the whole set (untimed) and is judged on the task’s own measure:

Gates are derived by rule, on CPU, before any accelerator is measured, and are identical for every vendor:

• near-lossless (strict) — the fp16 noise floor: PyTorch fp16 vs fp32 on CPU over the eval set, loosened by a fixed margin (KL ×2; SNR −6 dB; cosine/mIoU at half the remaining gap to 1). Continuous metrics only.
• standard — a neutral int8 reference: ONNX Runtime CPU running the pack’s plain int8 (A8W8) variant, same margins, plus a flip budget of max(1, ⌈5% of decisive samples⌉).

Each result publishes its gate margin (the binding bound and the slack).

Every network ships fp16 and int8 variants (int8 weights; int8 weights and activations). 4-bit weight variants — int4 (w4) and, for Core ML, 4-bit palettized (lut4, the 4-bit form Apple’s Neural Engine runs) — ship for the LLMs, and for any other network only if they pass the standard gate on CPU when the pack is built. A 4-bit result that fails its gate is shown (speed and accuracy) and never counts. Eval sets are curated so classifiers have ≥ 24 decisive samples. Eval images are CC0 photographs from Wikimedia Commons (sources listed in workloads/EVAL_SOURCES.md); text and prompts were written for NPUMark.

6. Timing

• Fresh load per attempt, untimed warm-up, then a time-based loop (2 s per attempt; at least 30 inferences, 10 for the peak test, whose calls take tens of milliseconds), so fast and slow models get comparable statistical weight.
• Compiled caches. Where the runtime compiles a model for its device (Core ML’s compiled models and the Neural Engine, ONNX Runtime’s EPContext models for QNN, OpenVINO and VitisAI, the CoreML EP’s model cache, LiteRT’s GPU programs), every attempt still loads afresh, from a compiled cache verified against its sidecar hash (§8) — checked once per run while the file is unchanged. Engines that compile are prepared once, untimed, before the first attempt, so attempt 1 loads the way every later one does. Load and compile times are shown under Run time; they are not part of any score.
• Timed the way apps run inference. The scored loop runs natively and leaves outputs in the runtime’s own buffers, as an app does: Core ML through prediction(from:) on prepared providers, ONNX Runtime through I/O binding, MLX through its compiled function, LiteRT through the compiled model’s buffers — each sample’s inputs are copied into the input buffers inside the timed region (what an app pays, as ONNX Runtime’s prepare-then-run does) and outputs are read in place. A binding that copies every output into a new array per call measures the copy, not the engine: for DeepLabV3’s 11 MB map that copy cost more than the GPU inference (Core ML GPU: 2.2 ms native, 7.6 ms through Python bindings). ONNX Runtime’s outputs are bound to host memory, so on a device EP (DirectML, CUDA, QNN) each timed Run includes the copy of the outputs back to the CPU — what an app reading its results pays; a conservative figure for an app that would keep outputs on the device.
• The loop cycles distinct eval inputs, so a runtime can’t serve a memoized answer to one repeated input; outputs of timed iterations are compared with the untimed outputs for the same inputs — a mismatch fails the gate (timed_check).
• Measurements run isolated from the UI with no scheduling sleeps inside the loop (on M5 Max the old 1.5 ms yields cost the GPU 13–30% throughput by letting it downclock): in a worker process on macOS, Windows and Linux, in a foreground service in its own process on Android, on a dedicated engine isolate on iOS (§8).
• Reported alongside: median and tail latency, burst latency (first 400 ms), drift (slowest 250 ms window vs median), sustained pace, and where the run’s time went (load, warm-up, timed loop, accuracy pass, placement), which is never part of a score.
• Diagnostics (separate run). The probes — concurrency, batching, idle-wake tax, and with --deep contention, the auto-scheduler and the compile cache — are not part of the scored run: they run as a diagnostics run of their own (npumark diagnose, or Run Diagnostics in the app), one attempt per engine and network on the fastest passing variant of the last benchmark, with a 2 s loop. They compare against probe_ips, the native single-call rate (one predict per call, timed by the same clock); py_ips is no longer measured and is recorded as null. A diagnostics record is saved locally, not comparable, and not part of the run history; probe ratios never feed a score.

7. Repeatability

Every engine is shared with the operating system. On an M5 Max, ten fresh-load attempts of the same ResNet-50 int8 measurement on the NPU ranged 1,087–1,558 inferences/s: interference arrives in episodes lasting seconds, and a single measurement can land in one. The GPU swings as well: MiniLM w8 (suite 2.3.0) on the GPU delivered 2.87 and 1.62 TOPS in two idle runs of the same pack. So, on every engine:

• each counted network result is the median of 3 attempts when those 3 range within 2% of their median, else the median of 5; each peak configuration the median of its fastest 3 of 5, else of its fastest 5 of 9 (§3). An entry with a crashed attempt among its first ones always runs to its maximum. Attempts run as later passes over the plan, and two attempts of one entry start at least 30 s apart (a short plan waits; nothing is reordered);
• a result whose first attempt shows it can never count (it failed its gate, or its work didn’t verifiably run on its accelerator) is not repeated, nor is a peak configuration under 90% of its engine’s best, nor (screening) a network variant whose first attempt is under 85% of the best first attempt of that network on that engine among the variants that can count. A screened variant is reported with its one attempt and is never the counted figure; if its first attempt beat the winner’s median, the network’s row is flagged (screened_outlier);
• apart from the peak’s fastest-half rule, no attempt is selected for being fast or steady — the choice of a network’s fastest passing variant and of an engine’s best peak configuration compares these figures, never single attempts;
• the attempts’ spread and robust CV (1.4826 × median absolute deviation / median) are published. A figure whose attempts — all that ran, also for the peak — have a robust CV above 5% is flagged unstable. Robust, so one stray attempt — which the median already ignores — doesn’t flag a figure; backtested on every repeated result of the 2.1–2.2 runs it flagged 3/10 GPU, 2/48 NPU and 0/9 CPU results, where a fastest-vs-slowest spread over 10% flagged 7, 6 and 2.

GPUs on Apple silicon additionally lower their clocks under single-request, latency-bound load. That is how single-request apps experience the GPU, so it is measured, not corrected; the diagnostics run’s concurrency probe reports the headroom.

8. Integrity

• Verified files. Every file a run reads — models, inputs, references, eval sets — is hashed every run and checked against the manifest, and the manifest against a hash pinned inside the NPUMark build. An edited pack (a loosened gate, a swapped model) runs but is never comparable.
• Verified peak and compiled models. The peak models and their references are pack files, hashed like every other: a modified model fails the math check, a modified reference fails the manifest hash. Compiled NPU graphs (ONNX Runtime EPContext caches, Core ML compiled models) carry a hash written at compile time and are rebuilt if they changed.
• Crash containment. A runtime abort costs one result, not the run. On desktop each attempt runs in a worker process and on Android in the measuring service’s own process: a crash is recorded as “crashed (worker died)” and the run continues. iOS allows no helper processes: the engine runs on its own isolate and the run journal is written before every attempt, so after a native abort the next launch resumes the run and records the attempt in flight as “crashed (app terminated)”. Combinations known to abort are skipped by default and recorded as skipped with the reason (re-test with --include-known-crashes). A later attempt that crashes, after an earlier one landed, keeps the result but is listed in the entry’s crashed_attempts ({attempt, why}), and the result shows it.
• Counted attempts. Each figure’s stats records attempts (all attempts run) and, for the peak, counted (how many entered the median: the fastest 3 of 5, or 5 of 9). The results server recomputes the median from the counted attempts, so a figure that disagrees with its own attempts is implausible; it also rejects an entry that stopped at 3 (or 5) attempts that do not agree within 2%, and a counted figure that is a single attempt.
• Cards. A card’s bus address (Linux) or adapter LUID (Windows) only orders identical cards. Uploads name a card by its model and number (“RTX 6000 Ada Generation #3”).
• Signed submissions. A shared run is signed (npumark-sig/v1) over the sha256 of its canonical payload: an HMAC with a build key that only production builds carry (held obfuscated in native code), an Ed25519 signature with a per-install key kept in the platform keystore and registered with the machine ID (up to 8 install keys per machine ID; a 9th after a 24-hour cooldown replaces the oldest; a refused key makes the install rotate to a new one and re-sign its queue), and App Attest (iOS, Mac App Store) or Play Integrity (Android) attestation where the platform offers it. The results server recomputes the digest, checks both signatures and the attestation, refuses replays, and applies plausibility rules (pinned and verified pack, known score version, peak below 1.25 × the chip family’s physical ceiling, attempts consistent with the reported medians, entries matching the platform’s plan). Limits: this stops edited JSON, replayed records and casual forgery; it cannot make a desktop build tamper-proof — a determined attacker who extracts the build key from a signed binary and patches the engine can still submit runs that pass plausibility. Such runs on Windows, Linux and the macOS .pkg build carry no attestation and are marked unattested; the leaderboard ranks only comparable, plausible runs.

9. Divisions

• Standard — NPUMark-built variants from public tools; the score.
• Optimized — vendor-built variants of the same source model under the same gates (see docs/OPTIMIZED_DIVISION.md); reported separately.

10. Comparability and versioning

A run is comparable — eligible for deltas, personal bests and the results database — only when the workload pack verified, the NPUMark score exists with verified math and is stable, and it isn’t a quick run. An unstable peak is a flag, not an exclusion: the results database ranks a run whose only reason is an unstable peak and marks it unstable. Delivered figures are ranked only when complete and stable. Any change that can move a number (workloads, models, gates, formula, timing) bumps the score version; scores compare only within one version.

• 3.2 (suite 3.2.1, beta; first published with suite 2.3.0): suite 2.4.0 replaces MiniLM-L6, BERT-base and Qwen2.5-0.5B with Granite Embedding 30M / 125M and SmolLM2-360M (licensing of their training data), calibrates int8 on CC0 photos instead of COCO, takes every canonical vision input from the eval set, and adds the large-batch peak. Beta 3.2 results from suite 2.3.0 are not comparable with suite 2.4 ones. Suite 2.4.1 runs the base SmolLM2-360M instead of -Instruct (whose fine-tuning data includes Llama outputs); only the unscored LLM extras change. Suite 3.2.1 is suite 2.4.1 renumbered to line up with the app: the same models, gates and inputs, so their results compare directly. Score 3.2 itself: machines with several identical cards are measured per card, and two labelled figures sit beside the headline — the large-batch peak on discrete GPUs (§3a) and the box total of every role with two or more devices (§3b). No 3.1 figure moves: the headline, per-engine peaks and delivered TOPS are computed as before. The record gains published fields, so 3.1 and 3.2 are separate series.
• 3.1 (suite 2.3.0): a default run takes far fewer attempts, with the same gates. Probes became a diagnostics run of their own (§6). Network variants under 85% of their network’s best first attempt on an engine are not repeated (screening), and every entry stops at its minimum attempts — 3 for a network, 5 for a peak configuration — when they agree within 2% (§3, §7); attempts of one entry start 30 s apart; compiled caches are verified once per run and compiling engines are prepared before attempt 1 (§6). The thresholds were set by backtest (tool/backtest_shrink.dart) on the 2.1 records (51 repeated NPU network entries) and a seeded synthetic peak set: at 85% / 2% neither changed a pick, a winner or an unstable flag, nor moved a scored figure by more than 0.4%, and the stricter 80% / 1.5% was clean too; 3% let a thermally declining peak configuration stop early and read stable, and 5% changed a best variant. Attempts saved on that corpus: 17% of every attempt, 27% of the synthetic peak attempts; the most on many-engine machines, where screening applies to every engine. 3.0 and 3.1 results are separate series: deltas, personal bests and the results database compare within one score version.
• 3.0, NPUMark 3 engine (suite 2.3.0): the engine was re-implemented in Dart with native backends (Core ML and MLX in Swift, ONNX Runtime and LiteRT in C++) for the cross-platform apps, with the same definitions, gates and formula; the score version is unchanged. Release requires a cross-implementation acceptance on the reference M5 Max (idle, on AC power; three default runs of each engine on the same pinned pack): the headline within 3 % of the previous engine’s median with its own spread ≤ 3 %, every scored delivered entry within 5 %, identical gate verdicts and residency figures, and the same record fields. The acceptance record is kept in docs/PARITY.md, and the M5 Max figures quoted in this document are re-measured with the new engine. Suite 2.3.0 adds the LiteRT (.tflite) format for Android and ships the peak models in the pack (§3); gates are unchanged. Before release, the peak test moved to 9 attempts per leading configuration with the median of the fastest 5 counted (§3, §7): a single slow cold load had flagged GPU peaks unstable.
• 3.0 (suite 2.2.0): the score became the best engine’s measured peak TOPS (§3), in plain TOPS; the former score, 1000 × geomean delivered TOPS of the NPU, became each engine’s delivered TOPS (§4), without the ×1000. MLX joined as a GPU runtime on macOS. Timed loops run natively (§6) — DeepLabV3 and other large-output networks read higher than in 2.x. Stability is judged by robust CV (§7).
• 2.2 (suite 2.1.2): GPU and CPU results became the median of 5 fresh-load attempts, like NPU results.
• 2.1.1 (suites 2.1.1–2.1.2): Core ML int8 packages re-encoded so all-zero weight groups decode correctly on the Neural Engine.

Gate values (suite 3.2.1)

Derivation per workload (gate_basis in the manifest): the near-lossless floor is the worst of PyTorch fp16, ONNX Runtime fp16 and fp32 against the fp32 reference on CPU; the standard bound comes from ONNX Runtime CPU int8 (A8W8; weight-only W8 for the LLMs). Where that neutral int8 is itself degenerate (MobileNetV3, DeepLabV3: it loses classes) the standard gate falls back to fixed floors — KL 0.05, mIoU 0.90, token cosine 0.98, mean cosine 0.99, p10 cosine 0.98. Manifest sha256 (pinned in the engine): fb81d029afd7c6d5ec6998466202ac79fd6e3de402a0dc7d07f0a4699c9ba888.