TFLiteConverter produces model that doesn't conform to GPUv2 (TfLiteGpuDelegate Init: FULLY_CONNECTED: Amount of input channels should match weights width)

[gustavla]

System information

• Compilation: Ubuntu Linux
• Execution: Google Pixel 7 / Android 14 / Google Tensor G2
• Tensorflow 2.16.1

Standalone code to reproduce the issue

• Generate model with script below (note, this fails with the reshape, but the TFLiteConverter ends up with a model without the reshape)
• Run on GPUv2 on device (for instance via benchmark tool)
• Falls back to run 100% on CPU

Generate model script

import tensorflow as tf
import keras

input_shape = [7]
output_shape = [1, 21]

tf_input = keras.Input(input_shape[1:], batch_size=input_shape[0])

tf_z = tf.reshape(tf_input, (1, 7))
tf_output = tf.keras.layers.Dense(21)(tf_z)

model = keras.Model(inputs=[tf_input], outputs=[tf_output])

# Convert the model.
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()

# Save the model.
with open('model.tflite', 'wb') as f:
  f.write(tflite_model)

Any other info / logs

Runtime log (executed by https://aihub.qualcomm.com/)

[01/May/2024:07:48:20 +10:00: profiler/info] Android system property: ro.board.platform = gs201
[01/May/2024:07:48:20 +10:00: profiler/info] Android system property: ro.boot.hardware = panther
[01/May/2024:07:48:20 +10:00: profiler/info] Android system property: ro.boot.hardware.platform = gs201
[01/May/2024:07:48:20 +10:00: profiler/info] Android system property: ro.system.build.id = UP1A.231005.007.A1
[01/May/2024:07:48:20 +10:00: profiler/info] Android system property: ro.system.build.version.release = 14
[01/May/2024:07:48:20 +10:00: profiler/info] Android system property: ro.hardware = panther
[01/May/2024:07:48:20 +10:00: profiler/info] Android system property: ro.hardware.chipname = 
[01/May/2024:07:48:20 +10:00: profiler/info] Android system property: ro.product.board = panther
[01/May/2024:07:48:20 +10:00: profiler/info] Android system property: ro.product.brand = google
[01/May/2024:07:48:20 +10:00: profiler/info] Android system property: ro.product.device = panther
[01/May/2024:07:48:20 +10:00: profiler/info] Android system property: ro.product.build.fingerprint = google/panther/panther:14/UP1A.231005.007.A1/10762838:user/release-keys
[01/May/2024:07:48:20 +10:00: profiler/info] Android system property: ro.product.manufacturer = Google
[01/May/2024:07:48:20 +10:00: profiler/info] Android system property: ro.product.model = Pixel 7
[01/May/2024:07:48:20 +10:00: profiler/info] Android system property: ro.product.name = panther
[01/May/2024:07:48:20 +10:00: profiler/info] Android system property: ro.soc.manufacturer = Google
[01/May/2024:07:48:20 +10:00: profiler/info] Android system property: ro.soc.model = GS201
[01/May/2024:07:48:20 +10:00: profiler/info] [Manager] DeviceManager::DeviceManager
[01/May/2024:07:48:20 +10:00: profiler/info] [Manager] findAvailableDevices
[01/May/2024:07:48:20 +10:00: profiler/info] [Manager] Found interface google-edgetpu (version = 2.0)
[01/May/2024:07:48:20 +10:00: profiler/info] [Manager] Found interface google-armnn (version = ArmNN)
[01/May/2024:07:48:20 +10:00: profiler/info] NNAPI devices: google-edgetpu,google-armnn,nnapi-reference
[01/May/2024:07:48:20 +10:00: profiler/info] GPU device: ARM Mali-G710
[01/May/2024:07:48:20 +10:00: profiler/info] OpenGL Version: OpenGL ES 3.2 v1.r43p0-01eac0.ff5c643eda65d3f5ba9886fdffb12673
[01/May/2024:07:48:20 +10:00: profiler/info] OpenCL Version: OpenCL C 1.2 v1.r43p0-01eac0.ff5c643eda65d3f5ba9886fdffb12673
[01/May/2024:07:48:20 +10:00: profiler/info] -=- Tungsten Running Task: Loading -=-
[01/May/2024:07:48:20 +10:00: profiler/info] Detected chipset 3101, made by 3000.
[01/May/2024:07:48:20 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] Loading tflite model Models/model.tflite
[01/May/2024:07:48:20 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] Malloc VM size before: 28856.0 kB, allocated: 18204.7 kB, slack: 10651.3 kB.
[01/May/2024:07:48:20 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] Current memory baseline range: 61424.7-72076.0 kB.
[01/May/2024:07:48:20 +10:00: profiler/debug] [job_id: j1p3y37mg] [model.tflite] Runtime metadata not found in Models/model.tflite/trt_metadata.json or Models/model.tflite/trt_metadata.pb
[01/May/2024:07:48:20 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] TF Lite version 2.16.1. Loading model from Models/model.tflite.
[01/May/2024:07:48:20 +10:00: profiler/debug] [job_id: j1p3y37mg] [model.tflite] Mapping resource file in Models/model.tflite
[01/May/2024:07:48:20 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] Loaded model. Minimum TF Lite version = 1.5.0.
[01/May/2024:07:48:20 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] No delegates specified; using compute unit=cpu_and_gpu.
[01/May/2024:07:48:20 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] [tflite] Initialized TensorFlow Lite runtime.
[01/May/2024:07:48:20 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] GPUV2 delegate requested. OpenCL detected.
[01/May/2024:07:48:20 +10:00: profiler/debug] [job_id: j1p3y37mg] [model.tflite] Enabling delegate cache in dir=/data/user/0/ai.tetra.tungsten/cache/1714513700731/ai.tetra.runtime/0.6.0/model.tflite_8945450969824422876_1714513695446/gpuv2.
[01/May/2024:07:48:20 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] [tflite] Created TensorFlow Lite delegate for GPU.
[01/May/2024:07:48:20 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] [tflite] Replacing 1 out of 1 node(s) with delegate (TfLiteGpuDelegateV2) node, yielding 1 partitions for the whole graph.
[01/May/2024:07:48:20 +10:00: profiler/warning] [job_id: j1p3y37mg] [model.tflite] [tflite] TfLiteGpuDelegate Init: FULLY_CONNECTED: Amount of input channels should match weights width
[01/May/2024:07:48:20 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] [tflite] Created 0 GPU delegate kernels.
[01/May/2024:07:48:20 +10:00: profiler/warning] [job_id: j1p3y37mg] [model.tflite] [tflite] TfLiteGpuDelegate Prepare: delegate is not initialized
[01/May/2024:07:48:20 +10:00: profiler/warning] [job_id: j1p3y37mg] [model.tflite] [tflite] Node number 1 (TfLiteGpuDelegateV2) failed to prepare.
[01/May/2024:07:48:20 +10:00: profiler/warning] [job_id: j1p3y37mg] [model.tflite] [tflite] Restored original execution plan after delegate application failure.
[01/May/2024:07:48:20 +10:00: profiler/error] [job_id: j1p3y37mg] [model.tflite] Delegate GPUV2/OpenCL failed to prepare. Retrying without this delegate.
[01/May/2024:07:48:20 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] TfLite XNNPACK delegate requested.
[01/May/2024:07:48:20 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] [tflite] Created TensorFlow Lite XNNPACK delegate for CPU.
[01/May/2024:07:48:20 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] [tflite] Replacing 1 out of 1 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 1 partitions for the whole graph.
[01/May/2024:07:48:20 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] Applied 1 delegates: XNNPACK. Model is fully delegated=true.
[01/May/2024:07:48:20 +10:00: profiler/debug] [job_id: j1p3y37mg] [model.tflite] Saving delegate selection for subsequent steps.
[01/May/2024:07:48:20 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] Malloc VM size after: 28856.0 kB, allocated: 18272.2 kB, slack: 10583.8 kB.
[01/May/2024:07:48:20 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] Status Successfully Loaded Cold with t = 16935 us and usage: before = 72076.0 kB; peakBefore = 72076.0 kB; mallocUnusedBefore = 10651.3 kB; after = 72232.0 kB; peakAfter = 72144.0 kB; mallocUnusedAfter = 10583.8 kB; increase = 0.0-223.5 kB; peak = 68.0-10719.3 kB
[01/May/2024:07:48:21 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] Saving results to /storage/emulated/0/Android/data/ai.tetra.tungsten/files/Results/job_j1p3y37mg/job_j1p3y37mg_results.bin
[01/May/2024:07:48:21 +10:00: profiler/info] -=- Tungsten Running Task: Loading -=-
[01/May/2024:07:48:21 +10:00: profiler/info] Detected chipset 3101, made by 3000.
[01/May/2024:07:48:21 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] Loading previously saved results in /storage/emulated/0/Android/data/ai.tetra.tungsten/files/Results/job_j1p3y37mg/job_j1p3y37mg_results.bin
[01/May/2024:07:48:21 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] Loading tflite model Models/model.tflite
[01/May/2024:07:48:21 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] Malloc VM size before: 28336.0 kB, allocated: 17488.8 kB, slack: 10847.2 kB.
[01/May/2024:07:48:21 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] Current memory baseline range: 55556.8-66404.0 kB.
[01/May/2024:07:48:21 +10:00: profiler/debug] [job_id: j1p3y37mg] [model.tflite] Runtime metadata not found in Models/model.tflite/trt_metadata.json or Models/model.tflite/trt_metadata.pb
[01/May/2024:07:48:21 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] TF Lite version 2.16.1. Loading model from Models/model.tflite.
[01/May/2024:07:48:21 +10:00: profiler/debug] [job_id: j1p3y37mg] [model.tflite] Mapping resource file in Models/model.tflite
[01/May/2024:07:48:21 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] Loaded model. Minimum TF Lite version = 1.5.0.
[01/May/2024:07:48:21 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] TfLite XNNPACK delegate requested.
[01/May/2024:07:48:21 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] [tflite] Replacing 1 out of 1 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 1 partitions for the whole graph.
[01/May/2024:07:48:21 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] Applied 1 delegates: XNNPACK. Model is fully delegated=true.
[01/May/2024:07:48:21 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] Malloc VM size after: 29368.0 kB, allocated: 18832.3 kB, slack: 10535.7 kB.
[01/May/2024:07:48:21 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] Status Successfully Loaded Warm with t = 5937 us and usage: before = 66404.0 kB; peakBefore = 66404.0 kB; mallocUnusedBefore = 10847.2 kB; after = 66848.0 kB; peakAfter = 66644.0 kB; mallocUnusedAfter = 10535.7 kB; increase = 0.0-755.5 kB; peak = 240.0-11087.2 kB
[01/May/2024:07:48:21 +10:00: profiler/info] [job_id: j1p3y37mg] [model.tflite] Saving results to /storage/emulated/0/Android/data/ai.tetra.tungsten/files/Results/job_j1p3y37mg/job_j1p3y37mg_results.bin