A Medium post examines Apple Intelligence's on-device 20-billion-parameter model, which is stored in flash memory and activates only a few experts per prompt. The sparse mixture-of-experts design trades compute for memory and battery efficiency on Macs. The analysis offers practitioners a look at how Apple fits large models onto consumer hardware.
A YouTube video demonstrates a 397-billion-parameter AI running on an iPhone with the network off, claiming on-device AI has reached that scale. The video explains the mixture-of-experts design that keeps 96% of the model inactive, cites an Apple paper predicting this three years ago, and notes the flash-moe team's caching code deletion that improved speed by 38%. It also acknowledges limitations in speed, storage, and heat.
A YouTube video demonstrates a 397-billion-parameter AI running on an iPhone with the network off, claiming on-device AI has reached that scale. The video explains the mixture-of-experts design that keeps 96% of the model inactive, cites an Apple paper predicting this three years ago, and notes the flash-moe team's caching code deletion that improved speed by 38%. It also acknowledges limitations in speed, storage, and heat.
A developer released TurboFieldfare, a custom Swift + Metal runtime that runs the instruction-tuned Gemma 4 26B-A4B model in about 2 GB of RAM on any M-series MacBook, including 8 GB models. This enables local inference of a 26-billion-parameter model on memory-constrained devices.
A developer released TurboFieldfare, a custom Swift + Metal runtime that runs the instruction-tuned Gemma 4 26B-A4B model in about 2 GB of RAM on any M-series MacBook, including 8 GB models. This enables local inference of a 26-billion-parameter model on memory-constrained devices.
A new resource provides measured LLM inference speeds on Apple Silicon, releasing raw data under CC BY 4.0. The data enables practitioners to compare performance across models and hardware configurations.
A new resource provides measured LLM inference speeds on Apple Silicon, releasing raw data under CC BY 4.0. The data enables practitioners to compare performance across models and hardware configurations.
Apple's M5 chip introduces dedicated Neural Accelerators on each GPU core, exposed via Metal 4 tensor API. The BaseRT runtime, a native Metal inference engine, uses hand-written tensor-core kernels to outperform llama.cpp and MLX on Apple hardware. The paper details dense and MoE GEMM and flash-attention prefill kernels.
Apple's M5 chip introduces dedicated Neural Accelerators on each GPU core, exposed via Metal 4 tensor API. The BaseRT runtime, a native Metal inference engine, uses hand-written tensor-core kernels to outperform llama.cpp and MLX on Apple hardware. The paper details dense and MoE GEMM and flash-attention prefill kernels.
A Medium article reviews Meta's MobileLLM-Flash and presents five counter-intuitive insights about deploying LLMs on mobile chips. It argues conventional compression wisdom breaks down in this setting, with FLOPs as a misleading metric and deep architectures stalling. Practitioners deploying LLMs to mobile devices are the target audience.
A Medium article reviews Meta's MobileLLM-Flash and presents five counter-intuitive insights about deploying LLMs on mobile chips. It argues conventional compression wisdom breaks down in this setting, with FLOPs as a misleading metric and deep architectures stalling. Practitioners deploying LLMs to mobile devices are the target audience.