Aidge demonstration#
Aidge is a collaborative open source deep learning library optimized for exporting and processing deep learning algorithms on embedded devices. With Aidge, one you can create or import a computational graph from popular frameworks, apply modification to its structure, train it and export its architecture to various embedded devices. Aidge provides optimized functions for both inference and training, as well as many custom functionalities for the target device.
This notebook put in perspective the tool chain to import a Deep Neural Network from ONNX model and support its Inference in Aidge. The tool chain demonstrated is:
In order to demonstrate this toolchain, the MNIST digit recognition task is used.
Setting up the notebook#
[1]:
# First import some utility methods used in the tutorial:
import sys, os
sys.path.append(os.path.abspath(os.path.join('..')))
import tuto_utils
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
Cell In[1], line 4
2 import sys, os
3 sys.path.append(os.path.abspath(os.path.join('..')))
----> 4 import tuto_utils
ModuleNotFoundError: No module named 'tuto_utils'
(if needed) Download the model#
If you don’t have git-lfs, you can download the model and data using this piece of code
[2]:
# Download onnx model file
tuto_utils.download_material("101_first_step", "MLP_MNIST.onnx")
# Download input data
tuto_utils.download_material("101_first_step", "input_digit.npy")
# Download output data for later comparison
tuto_utils.download_material("101_first_step", "output_digit.npy")
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[2], line 2
1 # Download onnx model file
----> 2 tuto_utils.download_material("101_first_step", "MLP_MNIST.onnx")
3 # Download input data
4 tuto_utils.download_material("101_first_step", "input_digit.npy")
NameError: name 'tuto_utils' is not defined
Import Aidge#
In order to provide a colaborative environnement in the platform, the structure of Aidge is built on a core library that interfaces with multiple modules binded to python libraries.
aidge_core is the core library which offers all the basic functionnalities to create and manipulate the internal computational graph representation;
aidge_backend_cpu is a C++ module providing a generic C++ implementations for each component of the computational graph;
aidge_onnx is a module allowing to import ONNX to the Aidge framework;
aidge_export_cpp is a module dedicated to the generation of optimized C++ code.
This way, aidge_core is free of any dependencies and the user can install whatever they want depending on their use case.
[3]:
import aidge_core
# Conv2D Operator is available but only the "export_serialize" is available.
# This backend allow to generate C++ code but not to run inference.
# For this we would need "cpu" backend.
print(f"Available backends:\n{aidge_core.get_keys_Conv2DOp()}")
# note: Tensor is a special case as 'cpu' backend is provided in the core
# module to guarantee basic functionalities such as data accesss
print(f"Available backends for Tensor:\n{aidge_core.Tensor.get_available_backends()}")
Available backends:
{'export_serialize'}
Available backends for Tensor:
{'cpu'}
As one can see, only an export backend is available for the class Conv2D, which is export_serialize. One needs to import the aidge_backend_cpu module which automatically registers itself to aidge_core, giving access to a backend that is able to run an inference.
[4]:
import aidge_backend_cpu
print(f"Available backends:\n{aidge_core.get_keys_Conv2DOp()}")
Available backends:
{'export_serialize', 'cpu'}
For this tutorial, we will need to import aidge_onnx in order to load ONNX files, numpy in order to load data and matplotlib to display images.
[5]:
import aidge_onnx
import numpy as np
import matplotlib.pyplot as plt
ONNX Import#
[6]:
model = aidge_onnx.load_onnx("MLP_MNIST.onnx")
---------------------------------------------------------------------------
FileNotFoundError Traceback (most recent call last)
Cell In[6], line 1
----> 1 model = aidge_onnx.load_onnx("MLP_MNIST.onnx")
File /builds/eclipse/aidge/aidge/venv/lib/python3.10/site-packages/aidge_onnx/onnx_import.py:39, in load_onnx(filename, verbose)
36 aidge_core.Log.info(f"Loading ONNX {filename}")
38 # Load the ONNX model
---> 39 model = onnx.load(filename)
40 return _load_onnx2graphview(model, verbose)
File /builds/eclipse/aidge/aidge/venv/lib/python3.10/site-packages/onnx/__init__.py:212, in load_model(f, format, load_external_data)
191 def load_model(
192 f: IO[bytes] | str | os.PathLike,
193 format: _SupportedFormat | None = None, # noqa: A002
194 load_external_data: bool = True,
195 ) -> ModelProto:
196 """Loads a serialized ModelProto into memory.
197
198 Args:
(...)
210 Loaded in-memory ModelProto.
211 """
--> 212 model = _get_serializer(format, f).deserialize_proto(_load_bytes(f), ModelProto())
214 if load_external_data:
215 model_filepath = _get_file_path(f)
File /builds/eclipse/aidge/aidge/venv/lib/python3.10/site-packages/onnx/__init__.py:149, in _load_bytes(f)
147 else:
148 f = typing.cast(Union[str, os.PathLike], f)
--> 149 with open(f, "rb") as readable:
150 content = readable.read()
151 return content
FileNotFoundError: [Errno 2] No such file or directory: 'MLP_MNIST.onnx'
As you can see in the logs, aidge imported a Node as a GenericOperator:
- /Flatten_output_0 (Flatten | GenericOperator)
This is a fallback mechanism which allows Aidge to load the entirety of an ONNX graph without failing, even when encountering a node that is not yet available in Aidge.
The GenericOperator acts as a stub retrieving the node’s type and attributes from ONNX. This allows the user to provide these nodes an implementation in a user script or remove/replace them using Aidge’s recipes, as detailed hereafter. You can visualize the graph using the save method and the mermaid visualizer we have setup.
[7]:
model.save("myModel")
tuto_utils.visualize_mmd("myModel.mmd")
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[7], line 1
----> 1 model.save("myModel")
2 tuto_utils.visualize_mmd("myModel.mmd")
NameError: name 'model' is not defined
Graph transformation#
In order to support the graph for inference we need to support all operators. The imported model contains Flatten before the Gemm operator. The aidge.FC operator already supports the flatten operation. Graph transformation is required to support the graph for inference, i.e. remove the Flatten operator.
Aidge’s graph transformation toolchain is embedded inside recipes functions. These recipes are available in aidge_core.
Examples include:
fuse_batchnorm: Fuse BatchNorm inside Conv or FC operator;
matmul_to_fc: Fuse MatMul and Add operator into a FC operator;
conv_horizontal_tiling: replace a conv by an horizontal tilled version;
remove_flatten: Remove Flatten if it is before an FC operator;
adapt_to_backend: Adapt graph to the current backend by adding Transpose layer to match expected input/output data format;
constant_folding: Compute constant part of the graph and replace them by pre-computed values.
Let’s apply the remove_flatten recipe:
[8]:
# Use the remove_flatten recipe
aidge_core.remove_flatten(model)
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[8], line 2
1 # Use the remove_flatten recipe
----> 2 aidge_core.remove_flatten(model)
NameError: name 'model' is not defined
The flatten node is removed with the recipie. Let’s visualize the model:
[9]:
model.save("mySupportedModel")
tuto_utils.visualize_mmd("mySupportedModel.mmd")
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[9], line 1
----> 1 model.save("mySupportedModel")
2 tuto_utils.visualize_mmd("mySupportedModel.mmd")
NameError: name 'model' is not defined
Static analysis#
Static analysis can be applied anytime on a graph in order to measure its complexity in terms of memory and operations.
[10]:
import aidge_core.static_analysis
# Dims must be forwarded for static analysis!
model.forward_dims(dims=[[1, 1, 28, 28]], allow_data_dependency=True)
model_stats = aidge_core.static_analysis.StaticAnalysis(model)
model_stats.summary()
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[10], line 4
1 import aidge_core.static_analysis
3 # Dims must be forwarded for static analysis!
----> 4 model.forward_dims(dims=[[1, 1, 28, 28]], allow_data_dependency=True)
6 model_stats = aidge_core.static_analysis.StaticAnalysis(model)
7 model_stats.summary()
NameError: name 'model' is not defined
[11]:
model_stats.log_nb_ops_by_type("stats_ops.png", log_scale=True)
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[11], line 1
----> 1 model_stats.log_nb_ops_by_type("stats_ops.png", log_scale=True)
NameError: name 'model_stats' is not defined
Inference#
Create an input tensor#
In order to perform an inference pass, we will load an image from the MNIST dataset using Numpy.
[12]:
## Load input data & its output from the MNIST_model
digit = np.load("input_digit.npy")
plt.imshow(digit[0][0], cmap='gray')
---------------------------------------------------------------------------
FileNotFoundError Traceback (most recent call last)
Cell In[12], line 2
1 ## Load input data & its output from the MNIST_model
----> 2 digit = np.load("input_digit.npy")
3 plt.imshow(digit[0][0], cmap='gray')
File /builds/eclipse/aidge/aidge/venv/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py:451, in load(file, mmap_mode, allow_pickle, fix_imports, encoding, max_header_size)
449 own_fid = False
450 else:
--> 451 fid = stack.enter_context(open(os.fspath(file), "rb"))
452 own_fid = True
454 # Code to distinguish from NumPy binary files and pickles.
FileNotFoundError: [Errno 2] No such file or directory: 'input_digit.npy'
And in order to validate the result our model will provide, we will also load the output the PyTorch model povided for this image
[13]:
output_model = np.load("output_digit.npy")
print(output_model)
---------------------------------------------------------------------------
FileNotFoundError Traceback (most recent call last)
Cell In[13], line 1
----> 1 output_model = np.load("output_digit.npy")
2 print(output_model)
File /builds/eclipse/aidge/aidge/venv/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py:451, in load(file, mmap_mode, allow_pickle, fix_imports, encoding, max_header_size)
449 own_fid = False
450 else:
--> 451 fid = stack.enter_context(open(os.fspath(file), "rb"))
452 own_fid = True
454 # Code to distinguish from NumPy binary files and pickles.
FileNotFoundError: [Errno 2] No such file or directory: 'output_digit.npy'
Thanks to the Numpy interoperability we can create an Aidge Tensor using directly the numpy array storing the image.
[14]:
input_tensor = aidge_core.Tensor(digit)
print(f"Aidge Input Tensor dimensions: \n{input_tensor.dims()}")
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[14], line 1
----> 1 input_tensor = aidge_core.Tensor(digit)
2 print(f"Aidge Input Tensor dimensions: \n{input_tensor.dims()}")
NameError: name 'digit' is not defined
Configure the model for inference#
At the moment the model has no implementation, it is only a datastructure. To set an implementation we will set a dataype and a backend.
[15]:
# Configure the model
model.compile("cpu", aidge_core.dtype.float32, dims=[[1,1,28,28]])
# equivalent to set_datatype(), set_backend() and forward_dims()
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[15], line 2
1 # Configure the model
----> 2 model.compile("cpu", aidge_core.dtype.float32, dims=[[1,1,28,28]])
3 # equivalent to set_datatype(), set_backend() and forward_dims()
NameError: name 'model' is not defined
Create a scheduler and run inference#
The graph is ready to run ! We just need to schedule the execution, to do this we will create a Scheduler object, which will take the graph and generate an optimized scheduling using a consummer producer heuristic.
[16]:
# Create SCHEDULER
scheduler = aidge_core.SequentialScheduler(model)
# Run inference !
scheduler.forward(data=[input_tensor])
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[16], line 2
1 # Create SCHEDULER
----> 2 scheduler = aidge_core.SequentialScheduler(model)
4 # Run inference !
5 scheduler.forward(data=[input_tensor])
NameError: name 'model' is not defined
[17]:
# Assert results
for outNode in model.get_output_nodes():
output_aidge = np.array(outNode.get_operator().get_output(0))
print(output_aidge)
print('Aidge prediction = ', np.argmax(output_aidge[0]))
assert(np.allclose(output_aidge, output_model,rtol=1e-04))
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[17], line 2
1 # Assert results
----> 2 for outNode in model.get_output_nodes():
3 output_aidge = np.array(outNode.get_operator().get_output(0))
4 print(output_aidge)
NameError: name 'model' is not defined
It is possible to save the scheduling in a mermaid format using:
[18]:
scheduler.save_scheduling_diagram("schedulingSequential")
tuto_utils.visualize_mmd("schedulingSequential_forward.mmd")
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[18], line 1
----> 1 scheduler.save_scheduling_diagram("schedulingSequential")
2 tuto_utils.visualize_mmd("schedulingSequential_forward.mmd")
NameError: name 'scheduler' is not defined
Optimize network#
[19]:
quantized_model = model.clone()
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[19], line 1
----> 1 quantized_model = model.clone()
NameError: name 'model' is not defined
[20]:
import gzip
NB_SAMPLES = 100 # Number of samples to use for PTQ
# Use data stored in PTQ tutorial, make sure to download them using git lfs
samples = np.load(gzip.GzipFile('../PTQ_tutorial/mnist_samples.npy.gz', "r"))
for i in range(10):
plt.subplot(1, 10, i + 1)
plt.axis('off')
plt.tight_layout()
plt.imshow(samples[i], cmap='gray')
tensors = []
for sample in samples[0:NB_SAMPLES]:
sample = np.reshape(sample, (1, 1, 28, 28)).astype(np.float32)
tensor = aidge_core.Tensor(sample)
tensors.append(tensor)
---------------------------------------------------------------------------
FileNotFoundError Traceback (most recent call last)
Cell In[20], line 6
3 NB_SAMPLES = 100 # Number of samples to use for PTQ
5 # Use data stored in PTQ tutorial, make sure to download them using git lfs
----> 6 samples = np.load(gzip.GzipFile('../PTQ_tutorial/mnist_samples.npy.gz', "r"))
7 for i in range(10):
8 plt.subplot(1, 10, i + 1)
File /usr/lib/python3.10/gzip.py:174, in GzipFile.__init__(self, filename, mode, compresslevel, fileobj, mtime)
172 mode += 'b'
173 if fileobj is None:
--> 174 fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
175 if filename is None:
176 filename = getattr(fileobj, 'name', '')
FileNotFoundError: [Errno 2] No such file or directory: '../PTQ_tutorial/mnist_samples.npy.gz'
[21]:
import aidge_quantization
aidge_quantization.quantize_network(
quantized_model,
8,
tensors,
target_type = aidge_core.dtype.float32,
clipping_mode = aidge_quantization.Clipping.MSE,
no_quant = False,
optimize_signs = True,
single_shift = False,
use_cuda = False)
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[21], line 3
1 import aidge_quantization
2 aidge_quantization.quantize_network(
----> 3 quantized_model,
4 8,
5 tensors,
6 target_type = aidge_core.dtype.float32,
7 clipping_mode = aidge_quantization.Clipping.MSE,
8 no_quant = False,
9 optimize_signs = True,
10 single_shift = False,
11 use_cuda = False)
NameError: name 'quantized_model' is not defined
[22]:
quantized_model.save("quantizedModel")
tuto_utils.visualize_mmd("quantizedModel.mmd")
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[22], line 1
----> 1 quantized_model.save("quantizedModel")
2 tuto_utils.visualize_mmd("quantizedModel.mmd")
NameError: name 'quantized_model' is not defined
Export#
Now that we have tested the imported graph we can look at one of the main feature of Aidge, the export of computationnal graph to an hardware target using code generation.
Generate an export in C++#
In this example we will generate a generic C++ export. This export is not based on the cpu backend we have set before.
In this example we will create a standalone export which is abstracted from the Aidge platform.
[23]:
! rm -r myexport
rm: cannot remove 'myexport': No such file or directory
[24]:
!ls myexport
ls: cannot access 'myexport': No such file or directory
Generating a cpu export recquires the aidge_export_cpp module.
Once the module is imported you just need one line to generate an export of the graph.
[25]:
import aidge_export_cpp
# Configuration for the model + forward dimensions
model.compile("cpu", aidge_core.dtype.float32, dims=[[1, 1, 28, 28]])
# Export the model in C++ standalone
aidge_core.export_utils.scheduler_export(
scheduler,
"myexport",
aidge_export_cpp.ExportLibCpp,
memory_manager=aidge_core.mem_info.generate_optimized_memory_info,
memory_manager_args={"stats_folder": "myexport/stats", "wrapping": False }
)
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[25], line 4
1 import aidge_export_cpp
3 # Configuration for the model + forward dimensions
----> 4 model.compile("cpu", aidge_core.dtype.float32, dims=[[1, 1, 28, 28]])
5 # Export the model in C++ standalone
6 aidge_core.export_utils.scheduler_export(
7 scheduler,
8 "myexport",
(...)
11 memory_manager_args={"stats_folder": "myexport/stats", "wrapping": False }
12 )
NameError: name 'model' is not defined
The export_scheduler function will generate:
dnn/include/forward.hpp define API function to use the export;
dnn/include/kernels folders for kernels;
dnn/include/layers layers configuration;
dnn/include/parameters folder with parameters;
dnn/src/forward.cpp source code of forward function which call kernels;
Makefile To compile the main.cpp
[26]:
!tree myexport
/usr/bin/sh: 1: tree: not found
Generate main file#
Export scheduler only generates the export of the kernels and a forward function which calls the kernels in the order described by the scheduler.
From this point we can start building an application. In order to do so, Aidge proposes a utils function named generate_main_cpp, which generates a simple main.cpp, able to perform an inference pass based on an input tensor provided by the user.
[27]:
aidge_core.export_utils.generate_main_cpp("myexport", model)
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[27], line 1
----> 1 aidge_core.export_utils.generate_main_cpp("myexport", model)
NameError: name 'model' is not defined
[28]:
!cat myexport/main.cpp
cat: myexport/main.cpp: No such file or directory
(Optional) Generate an input file for tests#
To test the export we need to provide input data. The generate_main_cpp function automatically generates an input file using tensor set as an input.
This is the case here, has we set an input tensor when running the forward pass, so we don’t need to execute the following cell. However, if no input has been set you need to manually generate the input file, to do so we can export a Numpy array using:
aidge_core.export_utils.generate_input_file(export_folder="myexport", array_name="fc1_Gemm_input_0", tensor=aidge_core.Tensor(digit.reshape(-1)))
Compile the export#
Once the generation has been done, we can compile the export with a simple make command:
[29]:
!cd myexport && make
/usr/bin/sh: 1: cd: can't cd to myexport
Run the export#
[30]:
!./myexport/bin/run_export
/usr/bin/sh: 1: ./myexport/bin/run_export: not found