# Optional on-device LLM (Gemma). Dimmy Help. > Optional on-device LLM (Gemma). Step-by-step help for Dimmy, the native voice dictation app for macOS, Windows, and Linux. Source: https://dimmy.app/help/local-llm/ [← Back to Help center](https://dimmy.app/help/) Providers & Models # Optional on-device LLM Run the LLM enhancement pass locally via llama.cpp. No cloud, no key, no per-token cost. Pick a model to match your GPU memory. By default the rewrite pass uses a cloud provider, or is Off. Optionally, Dimmy can run it on your own machine through llama.cpp. No key, no upload, no per-token cost. The catalogue is 15 GGUF models and the default is Phi-4 Mini Q4, 2.5 GB, 3.8B params. info The on-device LLM is a build feature. The installers on dimmy.app include it, with Metal on macOS and Vulkan on Windows. The Linux AppImage does not: it ships local speech to text only. ## Turn it on 1. Open Settings → Output. On Windows the card is LLM MODE, with Local and Cloud radios. On macOS it is LLM provider → Mode, with On device and Cloud. 2. Pick Local, or On device. 3. Choose a model from the list and click Download. Files stay on disk until you remove them, and a partial download resumes where it stopped. 4. Run a dictation. The pill shows the style icon while the model works on the text. Every row in the picker tells you two things: a green check when the file is already on disk, and a coloured dot for whether it fits the memory Dimmy can see. The recap picker uses the same rows, so you can run a different local model for meeting recaps than for dictation. See [Meeting recaps](https://dimmy.app/help/use-meeting-recaps/). ## The models | Model | Size | Notes | | ------------------------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Phi-4 Mini Q4 | 2.5 GB | Default. Fast fallback, multilingual, 3.8B params. | | Gemma 4 E2B QAT Q4 | 2.5 GB | Recommended. Quantization-aware, so better quality on less memory than the plain build. 5B params. | | TranslateGemma 4B Q4 | 2.4 GB | Best for translation, and the strongest local model on the rewrite styles. | | Qwen 3 4B Q4 | 2.4 GB | Slower, but extracts more detail. Tends to answer in English whatever you spoke. | | Gemma 4 E2B Q4 | 3.1 GB | The plain build of the same 5B. Fits a 4 GB card. | | Gemma 4 E2B Q5 | 3.7 GB | Better quality than Q4, still fits a 4 GB card. | | Gemma 4 E4B QAT Q4 | 4.0 GB | Quantization-aware 8B. Wants around 6 GB. | | Gemma 4 E4B Q3 | 4.1 GB | Tight on a 4 GB card, and roughly twice as slow on a recap. | | Gemma 4 12B Q2 (compact) | 4.7 GB | The closest a 12B gets to a 4 GB card. | | Gemma 4 E4B Q4 | 5.0 GB | Best balance once you have 6 GB or more. | | Gemma 4 12B QAT Q4 | 6.4 GB | Quantization-aware 12B. Around 8 GB of VRAM, or a 16 GB Mac. | | TranslateGemma 12B Q4 | 7.0 GB | The 12B translation model. Around 8 GB of VRAM, or a 16 GB Mac. | | Gemma 4 12B Q4 | 7.1 GB | Dense 12B. Wants around 9 GB. | | Gemma 4 E4B Q8 | 8.2 GB | Maximum quality of the 8B family. Wants 10 GB or more. | | Gemma 4 26B-A4B QAT Q4 | 13.6 GB | Best quality. Mixture of experts: 26B of knowledge, 4B active per token, so it reasons like a big model at roughly 4B speed. All 13.6 GB has to stay resident, so in practice a 32 GB Mac. | tip If you are undecided, start on the default Phi-4 Mini, then try Gemma 4 E2B QAT. Quantization-aware training is not marketing: on the same 35-minute meeting the QAT build used 1224 MiB against 1408 MiB for the plain Q4, and pulled out 23 decisions instead of 2. ## Language matters more than size here The -it in the file names means instruction tuned, not Italian. One model covers every language, which is why an Italian meeting comes back recapped in Italian without being asked. Two measured exceptions are worth knowing. TranslateGemma 4B came out best of every local model on the rewrite styles, with zero wrong-language and zero unchanged answers across 48 trials, where Gemma 4 E2B QAT handed the input straight back on Professional and Imbruttito. Qwen 3 4B extracts the most detail of any of them and took 108 seconds where Gemma 4 E2B took 44 on the same meeting, but it tends to answer in English even when you spoke Italian. ## How much memory it really takes * Dimmy budgets roughly the file size times 1.5, plus about 1 GB. That is deliberately generous: a 4 GB model measured 5.9 GB resident on a Mac. * If loading the LLM would push past a third of your machine memory, Dimmy unloads the speech model first and brings it back afterwards. * A model left idle for 5 minutes is unloaded, so the memory comes back to you between dictations. * If the GPU has no room, Dimmy says exactly that and points you at a smaller model. It is not reported as a network or HTTP error. ## Local recaps A local model can write meeting recaps too, and it streams into the meeting window token by token rather than appearing all at once. The whole transcript stays on the machine. Local recaps are capped at 4096 tokens of output, which is plenty for a summary and not meant for a transcript dump. ## Where cloud is still better On-device is private and free, but it is not the equal of the cloud flagships. For Correct and Summarize the small models are fine. For Prompt, Elaborate and long recaps, GPT-5.6 Sol or Claude Opus 5 are noticeably better. The 12B and 26B builds close part of that gap and charge you memory and speed for it. tip Speed depends on your machine far more than on the model name, and a GPU that quietly throttles can halve it. Download two, run the same dictation through both, keep the one you prefer. See [Which model should you use?](https://dimmy.app/help/choose-your-model/). Was this helpful? ## In this category [ Providers & ModelsWhich model should you use?Four questions decide it: what machine you have, what must not leave it, which subscriptions you already pay for, and which language you speak. Answer them and the rest follows.](https://dimmy.app/help/choose-your-model/) [ Providers & ModelsLocal mode: transcription on your own machineFour speech engines that run on your device. No API key, no upload, no per-minute cost. Works on a plane.](https://dimmy.app/help/local-mode/) [ Providers & ModelsCloud providers comparedGroq is fastest, OpenAI is the gold standard, Deepgram leads on streaming. All optional. Bring your own key.](https://dimmy.app/help/cloud-providers/) [ Providers & ModelsAdd your API keyOpen Settings → Providers & keys, find the provider card, paste the key. It is encrypted on your machine. You can update keys at any time.](https://dimmy.app/help/api-keys/) [ Providers & ModelsChoose a local modelThe whole on-device catalogue: nine Whisper sizes, Parakeet TDT v3, four Qwen3-ASR variants and Whistle. What each one weighs and what it runs on.](https://dimmy.app/help/whisper-models/) Last updated: 2026-05-22 On this page 1. [01 Turn it on](#turn-it-on) 2. [02 The models](#the-models) 3. [03 Language matters more than size here](#language-matters-more-than-size-here) 4. [04 How much memory it really takes](#how-much-memory-it-really-takes) 5. [05 Local recaps](#local-recaps) 6. [06 Where cloud is still better](#where-cloud-is-still-better)