Basic Usage

Using your saved models

nnio provides four classes for loading models in different formats:

Loaded models can be simply called as functions on numpy arrays. Look at the example:

import nnio

# Create model and put it on TPU device
model = nnio.EdgeTPUModel(
    model_path='path/to/model_quant_edgetpu.tflite',
    device='TPU:0',
)
# Create preprocessor
preproc = nnio.Preprocessing(
    resize=(224, 224),
    dtype='uint8',
    padding=True,
    batch_dimension=True,
)

# Preprocess your numpy image
image = preproc(image_rgb)

# Make prediction
class_scores = model(image)

See also nnio.Preprocessing documentation.

Description of the basic model classes

class nnio.ONNXModel(model_path: str)

This class is used with saved onnx models.

Usage example:

# Create model
model = nnio.ONNXModel('path/to/model.onnx')
# Create preprocessor
preproc = nnio.Preprocessing(
    resize=(300, 300),
    dtype='uint8',
    batch_dimension=True,
    channels_first=True,
)

# Preprocess your numpy image
image = preproc(image_rgb)

# Make prediction
class_scores = model(image)

Using this class requires onnxruntime to be installed. See Installation.

__init__(model_path: str)
Parameters

model_path – URL or path to the .onnx model

forward(*inputs, return_info=False)

This method is called when the model is called.

Parameters
  • *inputs – numpy arrays, Inputs to the model

  • return_info – bool, If True, will return inference time

Returns

numpy array or list of numpy arrays.

get_input_details()
Returns

human-readable model input details.

get_output_details()
Returns

human-readable model output details.

class nnio.EdgeTPUModel(model_path: str, device='CPU')

This class works with tflite models on CPU and with quantized tflite models on Google Coral Edge TPU.

Using this class requires some libraries to be installed. See Installation.

__init__(model_path: str, device='CPU')
Parameters
  • model_path – URL or path to the tflite model

  • device – str. CPU by default. Set TPU or TPU:0 to use the first EdgeTPU device. Set TPU:1 to use the second EdgeTPU device etc.

forward(*inputs, return_info=False)

This method is called when the model is called.

Parameters
  • *inputs – numpy arrays, Inputs to the model

  • return_info – bool, If True, will return inference time

Returns

numpy array or list of numpy arrays.

get_input_details()
Returns

human-readable model input details.

get_output_details()
Returns

human-readable model output details.

property n_inputs

number of input tensors

property n_outputs

number of output tensors

class nnio.OpenVINOModel(model_bin: str, model_xml: str, device='CPU')

This class works with OpenVINO models on CPU, Intel GPU and Intel Movidius Myriad.

Using this class requires some libraries to be installed. See Installation.

__init__(model_bin: str, model_xml: str, device='CPU')
Parameters
  • model_bin – URL or path to the openvino binary model file

  • model_xml – URL or path to the openvino xml model file

  • device – str. Choose Intel device: CPU, GPU, MYRIAD If there are multiple devices in your system, you can use indeces: MYRIAD:0 but it is not recommended since Intel automatically chooses a free device.

forward(inputs, return_info=False)
Parameters
  • inputs – numpy array, input to the model

  • return_info – bool, If True, will return inference time

Returns

numpy array or list of numpy arrays.

class nnio.TorchModel(model_path: str, device: str = 'cpu')

This class is used with saved torchscript models.

For saving model, use torch.jit.trace (easier) or torch.jit.script (harder).

Usage example:

# Create model
model = nnio.TorchModel('path/to/model.pt')
# Create preprocessor
preproc = nnio.Preprocessing(
    resize=(300, 300),
    dtype='uint8',
    batch_dimension=True,
    channels_first=True,
)

# Preprocess your numpy image
image = preproc(image_rgb)

# Make prediction
class_scores = model(image)

Using this class requires torch to be installed. See Installation.

__init__(model_path: str, device: str = 'cpu')
Parameters
  • model_path – URL or path to the torchscript model

  • device – Can be either cpu or cuda.

forward(*inputs, return_info=False)

This method is called when the model is called.

Parameters
  • *inputs – numpy arrays, Inputs to the model

  • return_info – bool, If True, will return inference time

Returns

numpy array or list of numpy arrays.