On-device AI on mobile: NPU, Whisper and real privacy

Every time a mobile app ships your voice to a server to “understand” what you said, there’s a hidden cost: latency, an inference bill that grows with every user, and personal data traveling to a cloud the user doesn’t control. The alternative has been maturing for years and is now viable on mid-range hardware: run the model inside the phone. This is what I’ve learned bringing on-device AI on mobile to real applications, from recommenders in React Native to voice transcription with Whisper.

What “on-device” means and why it matters

On-device means inference happens on the device itself, with no network. It sounds obvious, but it changes three equations at once:

• Real privacy, not declared privacy. If the audio or the image never leaves the phone, there’s nothing to leak, intercept or retain. It’s the same principle I applied in RecruitSecure AI, where candidate matching runs 100% local with Transformers.js precisely so CVs never move to any server. The difference from a “we don’t store your data” promise is that here there’s no data to store.
• Zero marginal cost. A call to a transcription API costs money per minute. Multiply that by 50,000 active users and the business model suffers. On-device, the user’s device pays for the compute: your inference bill is zero.
• Offline and low latency. With no network round-trip, the response is instant and works on the subway, out in the field, or with patchy coverage. For an offline-first field app like appxdata this isn’t a luxury, it’s the requirement.

The price you pay is working with limited compute and the fragmentation of Android hardware. That’s where the real engineering lives.

The runtime: TFLite, NNAPI and the GPU delegate

On Android the workhorse is still TensorFlow Lite (now LiteRT), and the decision that affects performance the most isn’t the model, it’s the delegate: which unit of the SoC you hand inference off to.

• CPU (XNNPACK). The safe path. It works on every device, and for small models quantized to int8 it’s usually more than enough. It’s the floor I always start from.
• GPU delegate. For vision models heavy on convolution, moving inference to the GPU lowers latency and frees the CPU for the UI. The cost is an initial warm-up (compiling the shaders) that’s best done off the critical thread.
• NNAPI / NPU. The Android layer that routes to the neural accelerator (the NPU) when one exists. When it works, it’s the most energy-efficient option. The problem is fragmentation: the same model can fly on a Pixel and silently fall back to CPU on another manufacturer. That’s why I never assume the NPU: I treat it as an opportunistic optimization, with a fallback to GPU and then to CPU.

The practical rule I follow: quantize first, accelerate later. Going from float32 to int8 cuts the size to a quarter and often delivers more real improvement on mid-range hardware than wrestling with an exotic delegate. And always measure end-to-end latency —preprocessing included—, not just inference time, a lesson I learned deploying computer vision with YOLOv8 at the edge that reappears identically on mobile.

I have a direct precedent for this in IoT: in PlantsCare the plant-image classification runs on EfficientNet-B0 over TensorFlow Lite, a model chosen for its size/accuracy ratio precisely to live close to the edge. Mobile is the same problem in a different enclosure.

React Native without reaching for the cloud

In React Native the pattern that works for me is clear: the model and the heavy preprocessing live in a native module (Kotlin/Swift or C++ via JSI), and JavaScript orchestrates. It’s exactly the architecture of Cherry TV, where the on-device recommender and the fuzzy search over thousands of channels run on the device, with the AI logic in TypeScript and the native modules in Kotlin for anything touching the hardware. Hermes and the New Architecture help, but the underlying decision is not to cross the JS bridge for every frame or every token.

Whisper on the device: real local STT

Voice transcription (STT) is the case where on-device shines most, because voice is among the most sensitive data an app handles. OpenAI’s Whisper can run locally thanks to whisper.cpp, which ports it to C/C++ with aggressive quantization and bindings for mobile.

The decisions that really matter when integrating local Whisper:

• Pick the right size. tiny and base are the realistic ones on mobile; small already demands a high-end device. For short commands or dictation, quantized base strikes a sensible balance between accuracy and consumption. The “just in case” large model is the usual trap.
• GGML/int8 quantization. It’s what makes the model fit in RAM without frying the battery. Unquantized, Whisper on a phone isn’t practical.
• Chunk the audio with VAD. Don’t transcribe long streams in one go: use voice activity detection (VAD) to feed the model only when someone is speaking. You cut compute and avoid hallucinations during silences, a classic Whisper failure.
• Pin the language if you know it. Forcing a known language stops the model from spending steps on language detection and reduces errors on short phrases.

The result is dictation and voice commands that work on a plane, without sending a single byte of audio anywhere. For a fitness app where the user wants to log sets “with their hands full”, this is the difference between a feature that gets used and one that depends on having good coverage in the basement gym.

Closing the loop: TTS and avatar

The flip side of STT is speech synthesis (TTS). Android and iOS ship decent, free native TTS engines (expo-speech exposes them directly in the Expo ecosystem), enough for spoken responses with no cost and no network. If you also want an avatar that moves its lips, the cheap piece is mapping the TTS visemes to the blend shapes of a 3D model; you don’t need a generative model on the device to make it believable. My criterion: use the native, free option as long as it holds up, and reserve a neural voice model only if naturalness is part of the product.

An on-device architecture that survives production

I pull it all together into a pattern I repeat project after project:

1. Runtime abstraction layer. The app code doesn’t talk to TFLite or whisper.cpp directly, but to an InferenceEngine interface. Swapping a delegate or a backend shouldn’t touch the UI (dependency-inversion principle, which incidentally makes it easy to test with a mock).
2. Cascading degradation. NPU → GPU → CPU, detected at startup and cached. If the accelerator isn’t there, the app still works, just slower. Never a crash because of hardware.
3. Lazy model loading. Models are heavy; don’t block startup. Load them in the background after the first render and show status in the meantime.
4. Battery and thermal budget. Continuous inference heats up the phone and drains the battery. For looping tasks (always-on voice, live camera) set frequency limits and respect power-saving mode.
5. Measure on real, cheap devices. The emulator lies, and so does your high-end phone. The truth lives in a three-year-old mid-range Android.

On-device isn’t “the same thing but serverless”. It’s a change of constraints: you trade cloud cost and privacy risk for a fight with the user’s hardware. Almost always worth it.

When NOT to go on-device

For technical honesty: on-device isn’t the answer to everything. Large generative language models, tasks that need fresh centralized data, or tasks where absolute accuracy outweighs privacy and cost still call for the cloud. The hybrid pattern —the sensitive and frequent stuff local, the heavy and occasional stuff on the server— is usually the grown-up answer. I apply it in DevFlow AI, which is local-first but keeps cloud AI as an option, not a dependency.

Closing

On-device AI on mobile stopped being a lab experiment. With TFLite/LiteRT and the right delegates, quantized Whisper for voice and native TTS for the response, you can now build private, instant experiences with no marginal cost that three years ago demanded an expensive cloud. The hard work isn’t training the model: it’s the systems engineering around it —quantization, fallbacks, thermal budget, native modules— that makes it run on a real user’s real phone.

If you’re building a mobile app and want to put AI inside the device —for privacy, for cost, or because it has to work with no coverage— and you’d like an honest assessment of what’s viable on your users’ hardware, let’s talk. It’s exactly the kind of problem I enjoy.