Run an MLPerf Tiny benchmark with Aidge#

This tutorial exports a TinyML model with Aidge, builds and flashes an MLPerf Tiny-compatible UART firmware, then uses the official MLCommons runner to measure accuracy and performance.

The example uses ResNet8 / CIFAR-10 on an STM32H7. The same flow also supports DS-CNN / Speech Commands, MobileNetV1 / Visual Wake Words, and the Deep AutoEncoder / ToyADMOS. Power measurement is not covered because it requires an external energy-measurement setup.

1. Understand the two parts of the workflow#

Aidge and MLCommons have separate responsibilities:

  1. Aidge loads the model, loads representative data, optionally quantizes it, exports C/C++ for Cortex-M, generates the MLPerf Tiny UART command loop, builds the firmware, and flashes the board.

  2. The MLCommons runner owns the serial port after flashing. It sends dataset samples to the board, requests inference, receives predictions or timing results, computes the metric, and writes a session report.

The --tinymlbenchmark option is important: instead of Aidge’s normal standalone benchmark entry point, it generates Src/tinyml_benchmark.hpp and a firmware that speaks the mlperf-tiny-uart/v1 protocol.

2. Prerequisites#

You need:

  • an Aidge environment containing aidge_core, aidge_onnx, aidge_quantization, aidge_export_arm_cortexm, and aidge_deploy;

  • a supported Cortex-M board connected through its debug probe and UART;

  • pyOCD and the target pack, plus pyserial;

  • Docker when using --use-docker, or a local Arm GNU toolchain otherwise;

  • Git and a separate Python virtual environment for the MLCommons runner.

[ ]:

from pathlib import Path
import shutil
import sys

REPO_CANDIDATES = [Path.cwd(), Path.cwd().parents[2]]
AIDGE_ROOT = next(
    (
        p
        for p in REPO_CANDIDATES
        if (p / "examples/benchmark/benchmark_model.py").is_file()
    ),
    None,
)
if AIDGE_ROOT is None:
    raise FileNotFoundError(
        "Run this notebook from the Aidge repository or its tutorials directory."
    )

BENCHMARK_DIR = AIDGE_ROOT / "examples" / "benchmark"
print("Aidge repository:", AIDGE_ROOT)
print("Python:", sys.executable)
print("pyocd:", shutil.which("pyocd") or "not found")
print(
    "arm-none-eabi-gcc:",
    shutil.which("arm-none-eabi-gcc") or "not found (OK when building with Docker)",
)

3. Choose the benchmark#

Aidge maps each registered model to the identifier expected by the Tiny runner:

Aidge model

MLPerf Tiny ID

Dataset

Runner device file

resnet8_cifar10

ic01

CIFAR-10

devices_kws_ic_vww.yaml

ds_cnn

kws01

Speech Commands

devices_kws_ic_vww.yaml

mobilenet_v1_vww

vww01

Visual Wake Words

devices_kws_ic_vww.yaml

deep_autoencoder

ad01

ToyADMOS

devices_ad.yaml

Use the same model/ID pair in the firmware and runner. This tutorial selects image classification.

[ ]:

MODEL = "resnet8_cifar10"
TINY_ID = "ic01"
TARGET = "stm32h7a3"
DEVICE_FILE = "devices_kws_ic_vww.yaml"
DTYPE = "int8"
USE_CMSIS_NN = True
USE_DOCKER = True
PROBE_UNIQUE_ID = None  # Fill this when several probes are connected.

assert (MODEL, TINY_ID) in {
    ("resnet8_cifar10", "ic01"),
    ("ds_cnn", "kws01"),
    ("mobilenet_v1_vww", "vww01"),
    ("deep_autoencoder", "ad01"),
}

4. Export, build, and flash with Aidge#

The benchmark script downloads a registered model when it is not already cached, loads dataset samples for host inference and INT8 calibration, generates the Tiny UART harness, builds the target project, and flashes it. Use real representative data for meaningful quantization; --mock-db is useful only for checking the pipeline.

The command normally continues by opening UART. Once flashing has completed and the board is waiting for Tiny commands, press Ctrl+C in that terminal so Aidge releases the serial port. The MLCommons runner cannot connect while another process owns it.

[ ]:

cmd = [
    sys.executable,
    "benchmark_model.py",
    "--target",
    TARGET,
    "--model-path",
    MODEL,
    "--dtype",
    DTYPE,
    "--tinymlbenchmark",
]
if USE_CMSIS_NN:
    cmd.append("--cmsis")
if USE_DOCKER:
    cmd.append("--use-docker")
if PROBE_UNIQUE_ID:
    cmd += ["--probe-unique-id", PROBE_UNIQUE_ID]

print("Run in a terminal so Ctrl+C can release UART after flashing:")
print(f"cd {BENCHMARK_DIR}")
print(" ".join(cmd))

If you use a board-farm API, replace the local connection with --backend api plus --api-url, --api-username, --api-password, and --api-board-name. The Tiny runner still needs a UART path to the flashed device; therefore, a remote API workflow only works when the runner can reach the corresponding serial transport. Do not save credentials in the notebook.

5. Download the official MLCommons Tiny repository#

Keep the runner in its own environment because its dependencies can differ from Aidge’s. The commands below clone the upstream repository and install the dependencies used by the runner. Check the current upstream benchmark/TUTORIAL.md and the README for your selected task if the repository layout or requirements change.

[ ]:

# Run these commands in a separate terminal.
# git clone https://github.com/mlcommons/tiny.git
# cd tiny
# python3 -m venv .venv
# source .venv/bin/activate
# python -m pip install --upgrade pip
# pip install -r benchmark/training/streaming_wakeword/requirements.txt pyusb libusb

6. Prepare the evaluation dataset#

The runner does not use Aidge’s calibration folder. It expects the MLPerf evaluation samples in tiny/evaluation/datasets/<benchmark-id>/, for example tiny/evaluation/datasets/ic01/. Dataset preparation is benchmark-specific, so follow the README below tiny/benchmark/training/<task>/ and the official tutorial.

Typical task directories are:

  • image_classification for CIFAR-10 / ic01;

  • keyword_spotting for Speech Commands / kws01;

  • visual_wake_words for Visual Wake Words / vww01;

  • anomaly_detection for ToyADMOS / ad01.

Some tasks provide ready-made binary samples; others require downloading raw data and running a conversion script. For example, the Visual Wake Words flow may require download_and_train_vww.sh followed by generate_y_labels.py. Do not substitute arbitrary images or random calibration tensors for an accuracy run: input shape, dtype, quantization, byte order, preprocessing, sample names, and labels must match the runner’s expected format.

[ ]:

# Example only; execute after reading the current upstream task README.
# cd tiny/benchmark/training/visual_wake_words
# ./download_and_train_vww.sh
# python generate_y_labels.py
#
# Then place/copy the generated runner inputs under:
# tiny/evaluation/datasets/vww01/

7. Configure the serial device#

Open tiny/benchmark/runner/devices_kws_ic_vww.yaml (or devices_ad.yaml) and set the device entry to the UART exposed by your board, such as /dev/ttyACM0 on Linux. Confirm the baud rate in the runner’s device configuration matches the target firmware.

Useful checks on Linux are pyocd list for the debug probe and python -m serial.tools.list_ports for UART ports. Add your user to the appropriate serial-port group if access is denied, then reconnect or log in again. Ensure no serial monitor, Aidge process, or IDE still has the port open.

8. Run accuracy and performance#

Reset the board after flashing, activate the Tiny environment, enter tiny/benchmark/runner, and run the two modes. Both modes must use the same firmware. Accuracy streams evaluation samples and checks model quality; performance measures inference latency through the Tiny protocol.

[ ]:

dataset_path = "../evaluation/datasets/"
accuracy_cmd = (
    f"python main.py --dataset_path={dataset_path} "
    f"--test_script=tests_accuracy.yaml --device_list={DEVICE_FILE} --mode=a"
)
performance_cmd = (
    f"python main.py --dataset_path={dataset_path} "
    f"--test_script=tests_performance.yaml --device_list={DEVICE_FILE} --mode=p"
)
print("cd tiny/benchmark/runner")
print(accuracy_cmd)
print(performance_cmd)

The runner writes logs and reports below tiny/benchmark/runner/sessions/. Archive the complete accuracy and performance session together with the Aidge export manifest, compiler version, board name, clock configuration, Aidge revision, Tiny revision, and dataset-generation procedure.

A successful local run is useful engineering evidence, but calling a result an official MLPerf result requires following the current MLCommons rules, using the prescribed model/data and submission process, and satisfying the benchmark’s quality target.

9. Troubleshooting#

  • Runner cannot open UART: stop the Aidge command/serial monitor, verify the YAML port and permissions, and reset the board.

  • Runner does not recognize the device: confirm that Aidge was invoked with --tinymlbenchmark and that the flashed export uses the expected Tiny ID.

  • Wrong outputs or poor accuracy: verify the model/ID pair, real calibration data, INT8 preprocessing, dataset folder structure, and that the runner sends exactly the bytes expected by the exported input tensor.

  • Build fails: run the target board_files/setup.sh, verify the target name and pyOCD pack, and check Docker or arm-none-eabi-gcc.

  • Performance run hangs: reset the board, close all other UART clients, verify baud rate, and inspect the runner log plus the generated Src/tinyml_benchmark.hpp.

  • Upstream command no longer exists: consult the current MLCommons Tiny tutorial and the selected task README; dataset scripts and requirements are maintained upstream.