Deploy acelerado por NPU: STM32N6 e Alif Ensemble E7#

This tutorial shows how to deploy a quantized Aidge model to an STM32N657 (Neural-ART) or an Alif Ensemble E7 / AKE7 (Ethos-U55). It focuses on the less visible part of the workflow: how Aidge creates NPU regions, how vendor compilers validate them, where CPU fallback is decided, and how to deliberately limit acceleration to a small region such as a single convolution.

The notebook is safe to read and inspect: cells that submit laboratory jobs are disabled by default. Never store an API password in the notebook; use environment variables.

1. A mental model of hybrid deployment#

FP32 Aidge model
  -> INT8 PTQ + Cortex-M adaptation
  -> candidate subgraph discovery
  -> NPU compiler accepts, rejects, or splits the region
  -> each accepted region becomes an Aidge MetaOperator
  -> Aidge scheduler interleaves CPU and NPU actions
  -> C/C++ generation, build, flash, execution, and validation

An NPU region is not a second model executed by another program. It replaces a subgraph with a compound node (NeuralArtRegion or EthosURegion) inside the hybrid graph. Aidge still schedules forward.cpp: it executes CPU/CMSIS-NN kernels, calls the NPU runtime, returns to the CPU when required, and continues. Region boundaries are INT8 tensors materialized in memory.

Fallback is decided during export. If an operation is not placed in a region, it is exported for the CPU. There is no attempt to run it on the NPU and migrate it to the CPU at runtime.

2. What changes between the boards#

Etapa	STM32N657	Alif AKE7
NPU	ST Neural-ART / ATON	Arm Ethos-U55
Front-end of subgraph	ONNX	TOSA
Compilator	ST Edge AI Core `stedgeai`	Arm Vela
MetaOperator	`NeuralArtRegion`	`EthosURegion`
CPU outside the region	Aidge C++ / CMSIS-NN	Aidge C++ / CMSIS-NN
Key Artifact	ecblobs, weights and interface STAI/ATON	command stream, read-only data e scratch Ethos-U
CLI target	`stm32n657`	`ake7`

AKE7 uses aidge_tosa to serialize a region and Vela to turn it into an Ethos-U custom operator. STM32N6 exports each region to ONNX and invokes ST Edge AI with the stm32n6 target. Any CPU code produced by the ST tool is discarded because CPU fallback remains under the control of the Aidge Cortex-M exporter.

3. Software and hardware prerequisites#

Common requirements#

an Aidge environment with aidge_core, quantization, the Cortex-M exporter, aidge_deploy, and the model loader;
aidge_export_neural_art for STM32N6;
aidge_tosa and aidge_export_arm_ethos for AKE7;
Docker when using --use-docker;
access to the laboratory API, a registered board, and its correct board name;
an INT8 model and representative calibration/validation samples.

STM32N657#

Install ST Edge AI Core 4.x or later and make stedgeai available in PATH, or configure STEDGEAI_PATH. The exporter requires version 4.x to avoid cases where residual networks are accepted but parts are silently lowered to ll_sw_forward_* software kernels. The installation must also contain the stm32n6.mpool memory-pool file. Local execution additionally requires the STM32 BSP/toolchain and probe/UART access; with the API backend, these dependencies are installed on the laboratory server.

Alif AKE7#

Install aidge_tosa, the Ethos exporter, Vela, a compatible Arm GNU Toolchain (12.3 or 13.3.Rel1 are recommended by the module), the Alif ML Embedded Evaluation Kit/BSP, and SETools to program MRAM. For local execution, the SETools directory must be writable. On the AppKit, follow the jumper positions and serial-port instructions in the board guide; the adjacent heterogeneous deployment tutorial describes the SEUART/UART workflow. With the API backend, this software stack must be installed and configured on the server.

[ ]:

# Local checks that do not compile or access hardware.
from __future__ import annotations

import importlib.util
import os
import shutil

modules = [
    "aidge_core",
    "aidge_deploy",
    "aidge_export_neural_art",
    "aidge_tosa",
    "aidge_export_arm_ethos",
]
for module in modules:
    print(f"{module:32} {'OK' if importlib.util.find_spec(module) else 'MISSING'}")

print(
    "stedgeai:",
    os.environ.get("STEDGEAI_PATH")
    or shutil.which("stedgeai")
    or "/opt/ST/STEdgeAI/4.0/Utilities/linux/stedgeai",
)
print("vela:", shutil.which("vela"))
print("arm-none-eabi-gcc:", shutil.which("arm-none-eabi-gcc"))

5. How automatic partitioning works#

After quantization and Cortex-M adaptation, Aidge traverses the DAG in topological order.
Constant producers (weights and biases) follow their consumers and do not break connectivity.
Connected supported operators form maximal candidates; branches, residual shortcuts, and merges can remain in the same region.
The largest candidate is submitted to the actual target compiler. Nominal operator-family support is not sufficient: shapes, quantization, memory, and hardware constraints also matter.
If the compiler rejects a candidate, the exporter shrinks or splits it and retries. Operators that never form a valid region remain on the CPU.
An accepted region is replaced by a MetaOperator and the scheduler is regenerated.

On STM32N6, initial discovery filters operator families registered as supported. The driver can exclude nodes mapped to software by ST Edge AI, split the candidate, and recompile. ST Edge AI is authoritative for liveness and memory-pool planning; summing every node input and output would overestimate memory and ignore tensor lifetimes.

On AKE7, each region is expanded into primitive operators, checked by the TOSA exporter, and compiled by Vela. It is accepted only when the output contains exactly one Ethos-U custom operator. The algorithm tries the entire candidate first and searches for the longest valid prefix after a failure.

6. Boundaries, cost, and current limitations#

The current hybrid direct-driver requires one INT8 data input and one INT8 output per region. Internal fan-in and fan-out are allowed: a residual block may contain its skip path and Add inside the region. If those paths cross the boundary, however, the signature becomes invalid and the candidate is rejected. Shapes must be resolved before partitioning.

More NPU placement does not always mean lower latency. Each boundary may require synchronization, descriptor preparation, layout conversion, and memory traffic. An isolated convolution can be slower than CMSIS-NN when launch overhead exceeds the compute savings. Large regions usually amortize this cost better, but may fail because of memory pressure or one incompatible operator. Automatic partitioning therefore maximizes compilable regions; it does not prove that placement is globally optimal for latency or energy.

The nb_op heuristic belongs to the general graph-adaptation and implementation-selection workflow. It does not replace compiler-guided NPU region selection.

7. Prepare and inspect the graph before programming the board#

benchmark_model.py already performs the public export/deploy sequence. The expanded cells below first use prepare_export() only for diagnostics: they inspect the adapted graph and preview compiler-guided partitioning in a disposable directory.

This diagnostic context is not accepted by deploy. Section 11 separately calls the public export() function and obtains the ExportArtifact used for optional hardware deployment.

[ ]:

from dataclasses import replace
from pathlib import Path
import sys

import aidge_core
from aidge_core.export_utils import ExportApplication, configure_logging
from aidge_core.export_utils.config import BenchmarkConfig, ExportConfig
from aidge_core.export_utils.export import (
    export,
    prepare_export,
    quantize_graph,
    resolve_inference_backend,
)

BENCHMARK_CANDIDATES = [
    Path("../../benchmark").resolve(),
    Path("examples/benchmark").resolve(),
    Path("aidge/examples/benchmark").resolve(),
]
BENCHMARK_DIR = next(
    path for path in BENCHMARK_CANDIDATES if (path / "dataset_loaders").is_dir()
)
if str(BENCHMARK_DIR) not in sys.path:
    sys.path.insert(0, str(BENCHMARK_DIR))

from dataset_loaders.dataset_loaders import get_dataset_loader
from utils import load_model, quantize_samples, set_sample_metadata

configure_logging(3)  # Equivalent to -vvv.

# Choose "stm32n657" (Neural-ART) or "ake7" (Ethos-U55).
TARGET = "stm32n657"
target = TARGET
export_cfg = ExportConfig(
    target=target,
    dtype="int8",
    dformat=None,
    heuristic="nb_op",
    cmsis=True,
    use_docker=True,
    cuda=False,
    clone_model=False,
    npu=True,
)
backend = resolve_inference_backend(export_cfg.cuda)
print(f"Python kernel: {sys.executable}", flush=True)
print(f"Inference/calibration backend: {backend}", flush=True)

model_path = BENCHMARK_DIR / "resnet8_cifar10"
if not model_path.is_file():
    raise FileNotFoundError(f"Cached benchmark model not found: {model_path}")
print(f"Loading cached ResNet8 model: {model_path}", flush=True)
model = load_model(str(model_path))
print("ResNet8 loaded; creating CIFAR-10 mock samples...", flush=True)
loader = get_dataset_loader("cifar10")
sample_arrays, float_tensors, labels = loader.load_samples(
    sample_count=20, backend=backend, data_root="./data", mock_db=True
)
print("Mock samples ready.", flush=True)
model.set_datatype(aidge_core.dtype.float32)
print("Model datatype set to float32.", flush=True)
model.set_backend(backend)
print(f"Model backend set to {backend}.", flush=True)
print("Running INT8 PTQ calibration...", flush=True)
quantize_graph(model, float_tensors)
model.set_backend("cpu")
_, int8_tensors = quantize_samples(sample_arrays, "int8")
print("PTQ complete. The graph is ready for the single public export step.", flush=True)

[ ]:

def summarize_graph(graph):
    rows = []
    for index, node in enumerate(graph.get_ordered_nodes()):
        if node.type() != "Producer":
            rows.append((index, node.name(), node.type()))
    return rows


if preview_ctx is None:
    print("Enable RUN_PARTITION_DIAGNOSTICS to inspect the adapted preview graph.")
else:
    for row in summarize_graph(preview_ctx.graph):
        print(f"{row[0]:3d}  {row[1]:50s}  {row[2]}")

8. Preview automatic partitioning#

These target compiler functions write preview artifacts but do not create the final Aidge project, build firmware or access hardware. They operate on the diagnostic graph and return a hybrid graph plus a compiler manifest. Use the disposable partition_preview directory only for auditing candidate regions.

[ ]:

if preview_ctx is None:
    preview_dir = None
    hybrid_graph = None
    manifest = {"runtime": "skipped", "regions": []}
elif target == "stm32n657":
    preview_dir = Path(preview_ctx.export_folder) / "partition_preview"
    from aidge_export_neural_art import partition_and_export_st

    hybrid_graph, manifest = partition_and_export_st(
        preview_ctx.graph, preview_dir, board=preview_ctx.device.name
    )
elif target == "ake7":
    preview_dir = Path(preview_ctx.export_folder) / "partition_preview"
    from aidge_export_arm_ethos.direct_driver import partition_and_export

    hybrid_graph, manifest = partition_and_export(
        preview_ctx.graph, preview_dir, board=preview_ctx.device.name
    )
else:
    raise ValueError(f"Target not covered by this NPU tutorial: {target}")

print("Runtime:", manifest.get("runtime"))
for region in manifest.get("regions", []):
    print(region.get("name"), region.get("nodes", region.get("node_names")))
if hybrid_graph is not None:
    print("Hybrid graph:")
    for row in summarize_graph(hybrid_graph):
        print(row)
else:
    print("Partition preview skipped.")

9. Fine-grained STM32N6 control: accelerate only one Conv#

The Neural-ART partitioner provides an experimental excluded_node_names interface. To keep only one convolution as a candidate, exclude every other supported node. Obtain node names from preview_ctx, after diagnostic adaptation may have fused or renamed operators.

This experiment only builds and displays a candidate graph. The final public export remains responsible for compiler invocation, target integration and the complete project.

[ ]:

if preview_ctx is None:
    print("Partition diagnostic skipped.")
elif target == "stm32n657":
    from aidge_export_neural_art.partition import (
        is_npu_supported_node,
        partition_graph_for_neural_art,
    )

    supported = [
        node
        for node in preview_ctx.graph.get_ordered_nodes()
        if is_npu_supported_node(node)
    ]
    conv_candidates = [node for node in supported if "Conv" in node.type()]
    if not conv_candidates:
        raise RuntimeError("No supported Conv was found after adaptation.")
    chosen_conv = conv_candidates[0]
    excluded = {node.name() for node in supported if node is not chosen_conv}
    one_conv_graph, regions = partition_graph_for_neural_art(
        preview_ctx.graph, excluded_node_names=excluded
    )
    print("Selected Conv:", chosen_conv.name(), chosen_conv.type())
    print("Proposed regions:", [r["nodes"] for r in regions])
else:
    print("This exclusion interface is specific to the Neural-ART exporter.")

You can also study granularity with single_node_regions=True, which creates one candidate region per supported operator. This does not select only one Conv: it attempts to accelerate every compatible node separately and usually increases overhead. Use it for diagnostics, not as the default configuration. split_after_node_names={"node_name"} forces a boundary after specific nodes and is useful for investigating memory limits or compiler failures.

[ ]:

if preview_ctx is not None and target == "stm32n657":
    diagnostic_graph, diagnostic_regions = partition_graph_for_neural_art(
        preview_ctx.graph, single_node_regions=True
    )
    print("Proposed single-node regions:", len(diagnostic_regions))
else:
    print("Single-node partition diagnostic skipped.")

10. Fine-grained control on AKE7#

The standard public Ethos-U path is deliberately compiler-guided: partition_and_export() tries the largest region and falls back to valid prefixes. Unlike Neural-ART, it does not currently expose excluded_node_names in its public API. For production, prefer automatic partitioning and audit the result through npu/manifest.json.

For a single-Conv experiment, the driver-supported approach is to provide an EthosURegion MetaOperator beforehand. The driver detects existing regions, compiles exactly their internal graph with TOSA/Vela, and keeps the remainder on the CPU. When building this MetaOperator, include weight and bias Producers, preserve input and output tensors, require a single INT8 input/output, and update ordered ports if the region touches the model’s public input or output. Because this transformation is topology-sensitive, it should live in a reusable project helper instead of being scattered across notebook cells or model-specific exporter conditions.

For most investigations, a safer alternative is to build a small Producer -> Conv -> consumer model, run it with --target ake7 --npu, and confirm in the manifest that the only region contains the Conv. Then compare this microbenchmark with the complete network. This measures launch and boundary costs without modifying the production graph.

11. Generate the complete NPU project and optionally deploy it#

The public export() call now performs the real target/NPU transformation, invokes ST Edge AI or Vela as needed, resolves the board BSP, generates the target-specific entrypoint and returns an ExportArtifact. It does not build, flash or execute.

Inspect artifact.path / "npu/manifest.json" to verify accepted regions. Do not conclude that the NPU was used merely because npu=True: zero accepted regions is a valid CPU fallback result.

[ ]:

import json
import aidge_deploy

bench_cfg = BenchmarkConfig(nb_warmup=5, nb_iterations=5, profiling=True)
application = ExportApplication(
    kind="benchmark",
    nb_warmup=bench_cfg.nb_warmup,
    nb_iterations=bench_cfg.nb_iterations,
    profiling=bench_cfg.profiling,
)
print("Running the single export/adaptation/partition step...", flush=True)
artifact = export(
    model,
    replace(
        export_cfg,
        application=application,
        model_prequantized=True,
        clone_model=False,
    ),
    input_tensors=[int8_tensors[0]],
)
print("Export complete.", flush=True)

export_dir = artifact.path
manifest_path = export_dir / "npu" / "manifest.json"
print(f"Buildable project: {artifact.path}")
print(f"Export manifest: {artifact.manifest_path}")
if manifest_path.is_file():
    persisted = json.loads(manifest_path.read_text())
    print(json.dumps(persisted, indent=2))
    if not persisted.get("regions"):
        print(
            "WARNING: no NPU region was accepted; execution will use the CPU fallback."
        )
else:
    print("The exporter did not emit an NPU compiler manifest.")

SERIAL_PORT = None
RUN_HARDWARE_DEPLOYMENT = False
if RUN_HARDWARE_DEPLOYMENT:
    deploy_cfg = aidge_deploy.DeployConfig(
        target=artifact.manifest.target,
        backend="local",
        benchmark=bench_cfg,
        serial_port=SERIAL_PORT,
        flash_method="setools" if target == "ake7" else "pyocd",
        use_docker=export_cfg.use_docker,
    )
    result = aidge_deploy.deploy(artifact, deploy_cfg)
    print("Deployment output:", result.output)
else:
    result = None
    print(
        "Project exported but not deployed. Enable RUN_HARDWARE_DEPLOYMENT when the board is ready."
    )

12. Validate correctly#

Validate in layers:

Host FP32 with real data.
Host INT8 to separate quantization error from deployment error.
Target CPU/CMSIS-NN without --npu.
Target hybrid execution with --npu.
Compare logits/outputs, accuracy, latency, memory, and energy.

Use representative calibration data. Random data can exercise the pipeline but produces poor ranges. Keep the same INT8 tensor and preprocessing in CPU and NPU runs. Warm up before measuring, report dispersion, and use the manifest to confirm which operations were actually accelerated.

13. Troubleshooting#

No regions in the manifest: confirm INT8, resolved shapes, supported types, and compiler logs. The workflow correctly continues on the CPU.
ST Edge AI not found or version rejected: configure STEDGEAI_PATH, check stedgeai --version, and verify that stm32n6.mpool exists in the same installation.
ST accepts the graph but ``ll_sw_forward_*`` appears: use ST Edge AI 4.x or later; software inside a region would violate the responsibility boundary of the hybrid workflow.
Vela rejects a region: check MetaOperator expansion, TOSA support, dtype/quantization, the single-input/single-output requirement, and the selected AKE7 profile.
A residual block is split: try to keep the branch, skip, and merge in the same region; a multi-input or multi-output boundary does not satisfy the current ABI.
The NPU is slower: compare a larger region, reduce boundaries, and measure launch/copy overhead. An isolated Conv is primarily a functional experiment.
Outputs diverge: first compare host INT8 with target CPU, then compare target CPU with hybrid execution using exactly the same input bytes.
The API job cannot find the board: --target describes the family (stm32n657 or ake7), while --api-board-name must match the name registered on the server.
Local Alif build or flashing fails: check the toolchain, BSP/ML kit, SETools, permissions, serial port, and jumpers.
No local STM32 logs: check the probe, UART, and BSP-specific configuration. The STM32N657 target is loaded into RAM and serial capture participates in the execution sequence.

14. Delivery checklist#

☐ model quantized to INT8 with a representative dataset;
☐ host FP32 and INT8 baselines recorded;
☐ Cortex-M baseline without NPU recorded;
☐ npu/manifest.json archived with the result;
☐ node list reviewed for every region;
☐ target outputs compared with the reference;
☐ latency measured after warmup and across multiple iterations;
☐ Aidge, ST Edge AI or Vela, toolchain, and BSP versions recorded;
☐ credentials absent from the notebook and logs;
☐ final test repeated with real data and without --mock-db.

For the Alif multicore workflow, continue with heterogeneous_deployment_alif_ake7.ipynb. For exporter details, read the aidge_export_neural_art, aidge_export_ethos, and aidge_tosa READMEs.