Cross-compiling an XNNPACK export for a RISC-V (RVV) board#

Aidge generates a standalone C++ export of a model and builds it for a machine other than yours. This tutorial covers the full flow for a 64-bit RISC-V Linux board with the vector extension, using the XNNPACK kernels instead of Aidge’s portable C++ ones:

  1. declare the board in Aidge’s hardware model,

  2. adapt a ResNet-18 graph to the XNNPACK kernels,

  3. generate the export and cross-compile it on your machine,

  4. copy the binary to the board over SSH and run it,

  5. check the result against a host reference, then measure what XNNPACK gains.

Aidge selects the cross toolchain from the CPU architecture (riscv64), never from the board, so every step applies to any RVV 1.0 target. Two things change from one board to the next: the declaration in §2 and an SSH host.

Every number below comes from a Canaan CanMV K230: one C908 core visible to Linux (rv64imafdcvxthead), 490 316 kB of RAM, Ubuntu 24.04. That single core is also why we cross-compile instead of building on the board.

1. Prerequisites#

What

Why

aidge_core, aidge_onnx, aidge_backend_cpu, aidge_export_cpp

graph import, host reference inference, fallback kernels

aidge_export_xnnpack

registers ExportLibXNNPack, the XNNPACK kernel bindings

riscv64-linux-gnu-g++ 13 or newer on your PATH

cross-compiles the export; older ones have no RVV 1.0 intrinsics

SSH access to the board

deploy and run

XNNPACK itself is not in that list: CMake fetches it and cross-builds it with the export.

This tutorial cannot run on Binder: it needs a cross toolchain and a board reachable over SSH. Sections 1 to 5 run on any machine, only §6 onwards needs the board.

[ ]:

%pip install aidge-core \
    aidge-backend-cpu \
    aidge-onnx \
    aidge-export-cpp

aidge_export_xnnpack is not published on PyPI yet; install it from its repository alongside the other Aidge modules (same virtual environment):

git clone https://gitlab.eclipse.org/eclipse/aidge/aidge_export_xnnpack.git
pip install -e ./aidge_export_xnnpack

riscv64.toolchain compiles with -march=rv64gcv -mabi=lp64d, whose RVV 1.0 intrinsics need GCC 13+ (Ubuntu 22.04 ships GCC 11, without riscv_vector.h). If your distribution is too old, start the Jupyter kernel from a container that carries one, for instance ghcr.io/google/xnnpack/riscv, with your virtual environment mounted. The next cell prints the version it found.

Optional. If you already have a riscv64 XNNPACK build tree, export XNNPACK_ROOT=... before starting the notebook: export_xnnpack.cmake reads it from the environment and links against it instead of fetching. It expects the <checkout>/build/<config> layout.

[ ]:

from pathlib import Path
import subprocess
import numpy as np

import aidge_core
import aidge_onnx
import aidge_backend_cpu  # host reference inference
import aidge_export_cpp  # portable C++ kernels (baseline + XNNPACK fallback)
import aidge_export_xnnpack  # registers ExportLibXNNPack

from aidge_core.export_utils import (
    compile_export,
    export,
    generate_main_inference_time_cpp,
)
from aidge_core.export_utils.config import ExportConfig
from aidge_core.export_utils.export import prepare_export, write_export

aidge_core.Log.set_console_level(aidge_core.Level.Error)

# --- what you need to adapt to your setup ---------------------------------
TARGET = "k230"  # any target registered in hw_model
BOARD = "k230"  # ssh host alias, or user@ip
BOARD_TMP = "/tmp"  # writable dir on the board

NB_ITERATIONS, NB_WARMUP = 5, 1  # timed / untimed iterations for the latency run

WORK = Path(f"./{TARGET}_export").resolve()  # exports and their XNNPACK build trees
MODEL = WORK / "resnet18_imagenet_1k.onnx"
WORK.mkdir(exist_ok=True)

gcc = subprocess.run(
    ["riscv64-linux-gnu-g++", "-dumpversion"], capture_output=True, text=True
)
print("cross compiler:", gcc.stdout.strip(), "(13 or newer required)")

2. Declaring your board#

This section is the only board-specific code in the notebook. Aidge describes targets in aidge_core.hw_model, and the K230 is already declared upstream in aidge_core/hw_model/targets/linux/sbc.py: a LinuxTarget with a Riscv CPU, a Memory holding the board’s real MemTotal, and xnn_configurable = True.

Three things follow from that declaration:

  • the CPU is a Riscv, whose arch is riscv64. LinuxTarget resolves the toolchain as cmake/{arch}.toolchain, so every Riscv board gets riscv64.toolchain and is cross-compiled identically. That is what makes the rest of the notebook board-agnostic.

  • xnn_configurable = True lets the target accept the xnn switch. When it is on, "export_xnnpack" is prepended to the kernel library list: XNNPACK is tried first for every operator, and export_cpp covers whatever XNNPACK does not implement.

  • the capacity is the board’s real MemTotal, which the memory planner uses.

For another board, copy that class, change the name and the capacity, then set TARGET in §1. Declaring it from this notebook works as well as editing sbc.py.

[ ]:

from aidge_core.hw_model import get_target_by_name

for xnn in (False, True):
    device = get_target_by_name(TARGET, xnn_pack=xnn)
    print(
        f"xnn={xnn!s:5s} {device.name} {device.arch} "
        f"{device.board_toolchain_path.name} libs={device.lib}"
    )

3. Load and prepare ResNet-18#

Nothing target-specific here: import the ONNX model and apply the usual recipes. The export pipeline will do the heavy graph work (expand_metaops, adaptation, fusion) later.

[ ]:

aidge_core.utils.download_file(
    str(MODEL),
    "https://huggingface.co/EclipseAidge/resnet18/resolve/main/"
    "resnet18_imagenet_1k.onnx?download=true",
)


model = aidge_onnx.load_onnx(str(MODEL))
model.set_name("resnet18")
aidge_core.remove_flatten(model)
aidge_core.fuse_batchnorm(model)
print(f"{len(model.get_nodes())} nodes after the recipes")

4. The input tensor: set its data format#

XNNPACK kernels are NHWC only (ExportLibXNNPack._default_dformat). The export adapts the graph to NHWC and writes the sample input into data/<input>.h in that layout.

To permute it, Aidge needs to know the layout your tensor is in. A tensor built from a NumPy array carries the default format, not nchw, and Aidge only warns about it. Ignore that warning and the header is written unpermuted: NCHW bytes reach an NHWC pipeline. The export still builds and still runs, it just computes on scrambled data and the result is wrong. Label the tensor explicitly.

[ ]:

np.random.seed(1234)
sample = np.random.rand(1, 3, 224, 224).astype(np.float32)


def make_input(array):
    tensor = aidge_core.Tensor(array)  # float32 numpy -> cpu backend, float32 dtype
    tensor.set_data_format(aidge_core.dformat.nchw)  # <-- do not skip this
    return tensor


input_tensor = make_input(sample)
print("dformat:", input_tensor.dformat)

5. Export and cross-compile#

aidge_core.export_utils.export() does the whole thing in one call: it resolves the target, expands meta-operators, propagates dimensions, adapts the graph to the target’s kernel libraries, plans the activation memory, writes the export folder with its main.cpp, and builds it. Since the target’s arch is riscv64, that last step is a cross build against the cmake/riscv64.toolchain the export folder carries, so the binary comes out for the board and not for your machine.

ExportConfig is the only place where XNNPACK is switched on: xnn=True. We run it twice, once with XNNPACK and once without, to keep a baseline to compare against.

[ ]:

def config_for(variant, folder):
    # variant: "xnn" -> XNNPACK first, "cpp" -> portable C++ kernels only
    return ExportConfig(
        target=TARGET,
        xnn=(variant == "xnn"),
        dformat="nhwc",
        export_folder=str(folder),
        cuda=False,  # no backend_cuda needed for a float32 export
        clone_model=True,  # leave the loaded model untouched
    )


folders = {
    variant: export(
        model,
        config_for(variant, WORK / f"export_{variant}"),
        [make_input(sample)],
    )
    for variant in ("xnn", "cpp")
}

for name, folder in folders.items():
    binary = folder / "build" / "run_export"
    arch = subprocess.run(["file", str(binary)], capture_output=True, text=True).stdout
    assert "RISC-V" in arch, arch  # a cross build, not a host binary
    print(f"{name:4s} {binary.stat().st_size / 2**20:5.1f} MB  RISC-V")

What adaptation did#

Inside that call, adapt_graph rewrote the graph so every operator maps onto a kernel the target has, and fused the patterns each kernel library declares. XNNPACK’s fusion list (ExportLibXNNPack._fusion_list) covers PadConvAct, PadConvDWAct, PadAvgPool, AddAct and more: a Pad → Conv → ReLU chain becomes one node backed by a single fused XNNPACK operator, instead of three passes over memory.

Both libraries declare the patterns ResNet-18 contains, so the two exports end up with the same graph. Only the kernel behind each node differs.

What the generated code looks like#

This is where the two kernel libraries differ most. export_cpp emits a fully template-parameterized loop nest, where every shape, stride and memory offset is a compile-time template argument. The XNNPACK export instead builds one operator object per layer, in a generated header under dnn/include/layers/, and the forward just calls run() on it, as the next cell shows straight from the generated file.

Those objects are thin RAII wrappers (aidge_export_xnnpack/kernels/*_ctx.hpp) around XNNPACK’s C API: the constructor calls xnn_initialize and xnn_create_*_nhwc_f32, and run() does the reshape / setup / xnn_run_operator sequence. The heavy work, picking a micro-kernel for the CPU it runs on, happens once at construction.

[ ]:

forward = folders["xnn"] / "dnn" / "src" / "resnet18_forward.cpp"
lines = forward.read_text().splitlines()
first_run = next(i for i, line in enumerate(lines) if ".run(" in line)
print("\n".join(lines[first_run - 4 : first_run + 2]))

Where XNNPACK came from#

export_xnnpack.cmake reads XNNPACK_ROOT from the environment. Unset, FetchContent clones XNNPACK and builds it as a sub-project of the export: it inherits the toolchain file, and because that file sets CMAKE_SYSTEM_PROCESSOR riscv64, XNNPACK enables its RVV micro-kernels on its own. Set, the prebuilt tree is imported instead. Same binary either way; fetching just costs a few minutes of compilation per export folder.

6. Deploy and run#

XNNPACK is linked statically and the sample input is compiled into the binary, so a single scp is all it takes: no runtime, no data files, no Python on the board.

[ ]:

def run_on_board(binary, remote_name):
    subprocess.run(
        ["scp", "-q", str(binary), f"{BOARD}:{BOARD_TMP}/{remote_name}"], check=True
    )
    return subprocess.run(
        ["ssh", BOARD, f"cd {BOARD_TMP} && ./{remote_name}"],
        check=True,
        capture_output=True,
        text=True,
    ).stdout


out_xnn = run_on_board(folders["xnn"] / "build" / "run_export", "out_xnn")
out_cpp = run_on_board(folders["cpp"] / "build" / "run_export", "out_cpp")
print(out_xnn[:300])

7. Result 1: is it correct?#

The generated main prints the first 100 values of each output. We compare them against Aidge running the same ONNX file on backend_cpu, on the host.

[ ]:

def parse_output(text):
    # the generated main prints "<output name>:" then the first 100 values
    lines = [l for l in text.splitlines() if l.strip()]
    idx = next(i for i, l in enumerate(lines) if l.startswith("_30_FC_0_output_0"))
    return np.fromstring(lines[idx + 1].split("Output")[0], sep=" ")


model.set_backend("cpu")
model.set_datatype(aidge_core.dtype.float32)
model.forward_dims([[1, 3, 224, 224]], allow_data_dependency=True)
scheduler = aidge_core.SequentialScheduler(model)
reference = np.array(scheduler.forward(data=[make_input(sample)])[0]).flatten()[:100]

print(f"{'aidge backend_cpu':22s} {np.round(reference[:3], 4)}  reference")
for name, text in (("xnnpack", out_xnn), ("export_cpp", out_cpp)):
    values = parse_output(text)
    print(
        f"{TARGET + ' / ' + name:22s} {np.round(values[:3], 4)}  "
        f"max|diff| = {np.max(np.abs(values - reference)):.2e}"
    )

Both exports match the host reference to ~5e-6 in float32, and each other to 2e-6: the XNNPACK micro-kernels accumulate in a different order than the plain loop nest. Compare with a tolerance, never for equality.

8. Result 2: how fast?#

Same graph, same compiler, same flags, same board: the only variable is which kernel library the operators were mapped to.

Timing needs a different main.cpp, the one generate_main_inference_time_cpp emits, and that helper needs the adapted graph, which export() does not hand back. So this section uses the three functions export() is built from: prepare_export adapts and schedules the graph and returns a context without writing anything, write_export emits the folder from it, compile_export cross-compiles it.

[ ]:

timings = {}
for variant in ("xnn", "cpp"):
    ctx = prepare_export(
        model,
        config_for(variant, WORK / f"bench_{variant}"),
        input_tensors=[make_input(sample)],
    )
    write_export(ctx)
    generate_main_inference_time_cpp(
        ctx.export_folder, ctx.graph, NB_ITERATIONS, NB_WARMUP
    )
    compile_export(
        ctx.export_folder,
        build_system=ctx.device.build_system,
        arch=ctx.device.arch,
    )
    text = run_on_board(ctx.export_folder / "build" / "run_export", f"bench_{variant}")
    values = timings[variant] = np.fromstring(text.strip().splitlines()[-1], sep=" ")
    print(
        f"{variant:4s}: median {np.median(values):.3f} s "
        f"(min {values.min():.3f}, max {values.max():.3f}, n={len(values)})"
    )

print(f"\nspeed-up: {np.median(timings['cpp']) / np.median(timings['xnn']):.1f}x")

≈ 9.6x faster on one in-order RISC-V core (median 6.267 s for export_cpp, 0.651 s for export_xnnpack), for the same graph, the same numerical result and the same planned activation memory (5 619 712 bytes). The gain is entirely in the kernels: XNNPACK picks RVV micro-kernels at runtime, while export_cpp emits portable scalar loops that GCC vectorizes only where it can.

9. Going further#

  • float32 only. On aidge_export_xnnpack 0.0.1 these kernels are float32 (*_f32_ctx.hpp), so dtype="int8" silently falls back to export_cpp.

  • Per-layer timings. The generated forward wraps every layer in a ScopedTimer, compiled out unless ENABLE_TIMING is defined. compile_export takes no extra flags, so configure that build yourself: cmake -B build --toolchain cmake/riscv64.toolchain   -DCMAKE_CXX_FLAGS="-march=rv64gcv -mabi=lp64d -DENABLE_TIMING". Read the numbers from a warm forward, and note the timers are named after each node’s input tensor.

  • Build on the board instead of cross-compiling. aidge_core.export_utils.hardware_in_the_loop has an ssh_native backend that ships the export and runs CMake on the target. It avoids the toolchain-version question entirely, at the cost of compiling on the board: fine for a small model, painful for ResNet-18 on one core.

  • ``run_model_on_target`` takes the same context as §8 and drives deployment, warm-up and timed iterations, profiling and on-target validation for you. See the benchmark on target tutorial.

  • Other models. examples/exports/ has ready-made scripts for LeNet, ResNet-8/50, MobileNetV1, DS-CNN and more; swap the model in §3 and the rest of the notebook is unchanged.