Runtime graph execution

Implementations and backends

In order to provide a modular platform (e.g. to switch easily from one hardware target to another), AIDGE separates the concepts of description and implementation. Operator and data descriptions are generic, while implementations are specific. Indeed, if the implemtation of a convolution calculation differs on a GPU or CPU, the description of the convolution itself (its inputs and parameters) does not change. In this example, the implementation changes according to the hardware target, but it can also change according to the library used. For example, on the NVIDIA GPU target, programming can be done either via CUDA or via TensorRT. That is why AIDGE introduces the notion of Backend to define both the hardware target and the library used for the implementation.

Note that to associate an implementation with an operator or data, you need to define more than just a backend, you also need to define a data type and a precision. The process of choosing an implementation will be described in the sections defining operators and data respectively.

Implementation selection

As previously mentioned, an operator is agnostic of the implementation, and to select an implementation, a register system (similar to the one used in the case of Tensor) is available. This selection depends on the following attributes:

• The Backend, defined by both the hardware target (e.g. CPU, GPU, …) and available libraries (e.g OpenCV);

• The Datatype (float, int, …) and Precision (8bits, 16bits, 32bits,…) of the inputs and outputs;

• The DataFormat (NCHW, NHWC, see this link for more details)

• The Kernel : the algorithm chosen to perform the computation.

As long as these attributes are not defined, the forward and backward functions of the operator will remain empty.

Scheduler

AIDGE internal graph describes topological structures such as deep neural networks using nodes and operators. In order to execute a graph, i.e. calling each operator function, the graph needs a sequencing strategy. The scheduler defines that execution sequencing strategy.

Default scheduler

AIDGE define a default scheduler which use a system of producer consumer. Operator which have enough data in input will consume them and produce data for the next one.

In order for this heuristic to work, the scheduler needs to know how much data the operator:

• needs as input;

• has actually consumed;

• has actually produced.