Slow inference

[igor-yusupov]

Tried running the inference model and it runs very slow. The python code runs much faster. Can you please check what is causing this?

Weights: https://drive.google.com/file/d/1eEhrck8zzv5HP3vUf7wiimOFk_y5Gwn0/view?usp=sharing

code:

use rand::prelude::*;
use std::time::Instant;
use tract_onnx::{
    prelude::*,
    tract_hir::tract_ndarray::{Array3, Dim},
};

fn main() {
    let mut rng = thread_rng();
    let encoder = tract_onnx::onnx()
        .model_for_path(format!("weights/encoder.onnx"))
        .unwrap()
        .into_optimized()
        .unwrap()
        .into_runnable()
        .unwrap();

    let shape = Dim([1, 80, 3000]);
    let mel: Array3<f32> = Array3::from_shape_fn(shape, |_| rng.gen());
    let mel: Tensor = mel.into();
    let inputs = tvec!(mel.into());

    let start_time = Instant::now();
    let _encoder_out = encoder.run(inputs);
    let end_time = Instant::now();
    let elapsed_time = end_time.duration_since(start_time);
    println!("{:?}", elapsed_time.as_millis());
}

[abhemanyus]

Are you running it with the --release flag, and if so, is it running on the CPU or the GPU?

[igor-yusupov]

Yes, I run this model with --release flag. Without this flag it runs indefinitely. I guess tract works with CPU only

[cospectrum]

Tried running the inference model and it runs very slow. The python code runs much faster. Can you please check what is causing this?

Weights: https://drive.google.com/file/d/1eEhrck8zzv5HP3vUf7wiimOFk_y5Gwn0/view?usp=sharing

code:

use rand::prelude::*;
use std::time::Instant;
use tract_onnx::{
    prelude::*,
    tract_hir::tract_ndarray::{Array3, Dim},
};

fn main() {
    let mut rng = thread_rng();
    let encoder = tract_onnx::onnx()
        .model_for_path(format!("weights/encoder.onnx"))
        .unwrap()
        .into_optimized()
        .unwrap()
        .into_runnable()
        .unwrap();

    let shape = Dim([1, 80, 3000]);
    let mel: Array3<f32> = Array3::from_shape_fn(shape, |_| rng.gen());
    let mel: Tensor = mel.into();
    let inputs = tvec!(mel.into());

    let start_time = Instant::now();
    let _encoder_out = encoder.run(inputs);
    let end_time = Instant::now();
    let elapsed_time = end_time.duration_since(start_time);
    println!("{:?}", elapsed_time.as_millis());
}

What do you mean by "python code"?