Prediction model works real-time on-device or based on cloud?

[TimS70]

Dear ClearBuds-Team from U Washington,

thank you for creating such an awesome tool as Clearbuds! We already ordered our CB-PCB. For the time we are waiting, I was taking a closer look on your iOS app. I have a major question about how the prediction model of your prototype works (https://clearbuds.cs.washington.edu/).

In your paper you report a run-time performance of 21.4ms on an iPhone 12 (Table 2 and text). Moreover, you write about your goal of optimizing the Neural Network to run in real-time on smart devices and the benefits of such versus cloud transmission (Chatterjee et al., 2022).

However, in your iOS app, I could not find a reference to your Neural Network in a form of a .tflite model or similar (CB Neural Network is referenced here: https://github.com/vivjay30/clearbuds). Instead, the recorded audio data seems to be pushed on your servers via AWS (ViewController.swift line 526 ff., line 617). Is the filtering process of the recorded audio performed on a cloud instead of the mobile device?

• If yes, does the runtime of 21.4ms correspond to the performance including cloud transmission or another kind of implementation? Can this runtime test still be found in this repo?
• If not, could you please explain, whether and where in this Repo I can find the exact on-device execution of the prediction model?

Thank you very much for your help!

Chatterjee, I., Kim, M., Jayaram, V., Gollakota, S., Kemelmacher, I., Patel, S., & Seitz, S. M. (2022, June). ClearBuds: wireless binaural earbuds for learning-based speech enhancement. In Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services (pp. 384-396).