Google's AI Edge team announced the open-source release of ML Drift, a GPU compute engine for on-device AI/ML inference, under the Apache 2.0 license. It abstracts hardware and low-level APIs across OpenGL ES, OpenCL, Metal, and WebGPU, letting developers build real-time ML experiences on multiple platforms. It serves as the core GPU acceleration engine inside LiteRT and is also available standalone.
PyTorch models can now run on the GPU inside any WebGPU-enabled browser via a new ExecuTorch backend, eliminating conversions to formats like GGUF, ONNX, or TFLite. Developers export a model with torch.export to a single .pte file that executes in Chrome and Safari and natively on Metal, Vulkan, and D3D12. This simplifies shipping in-browser inference for practitioners working with PyTorch.
PyTorch models can now run on the GPU inside any WebGPU-enabled browser via a new ExecuTorch backend, eliminating conversions to formats like GGUF, ONNX, or TFLite. Developers export a model with torch.export to a single .pte file that executes in Chrome and Safari and natively on Metal, Vulkan, and D3D12. This simplifies shipping in-browser inference for practitioners working with PyTorch.
Hugging Face released open-source WebGPU kernels claimed to be the fastest for local AI, covering 200+ common ML operations that run entirely in the browser. The team plans to upstream the optimizations into Transformers.js, ONNX Runtime Web, and LiteRT.js, making fast local inference more accessible to web developers.
Hugging Face released open-source WebGPU kernels claimed to be the fastest for local AI, covering 200+ common ML operations that run entirely in the browser. The team plans to upstream the optimizations into Transformers.js, ONNX Runtime Web, and LiteRT.js, making fast local inference more accessible to web developers.
Cormac Brick of Google AI Edge argues that RAM, not compute, is the primary constraint for edge AI, noting that phone makers are shipping less RAM and a 6GB Raspberry Pi now costs 2.5x its launch price. His team has quantized a 2B-parameter Gemma model to 2.9 bits per weight, achieving ~8 tokens/second on a Raspberry Pi and fast enough vision processing on Qualcomm NPUs. They also target tiny models from 500M down to 50M parameters for older laptops and cheap devices.
Cormac Brick of Google AI Edge argues that RAM, not compute, is the primary constraint for edge AI, noting that phone makers are shipping less RAM and a 6GB Raspberry Pi now costs 2.5x its launch price. His team has quantized a 2B-parameter Gemma model to 2.9 bits per weight, achieving ~8 tokens/second on a Raspberry Pi and fast enough vision processing on Qualcomm NPUs. They also target tiny models from 500M down to 50M parameters for older laptops and cheap devices.
A developer launched a browser-based runtime that executes PyTorch-style code via WebGPU, with no installation, server, or account. The TypeScript-based tool supports both model training and inference and was built to let students run torch code in classes without setup.
A developer launched a browser-based runtime that executes PyTorch-style code via WebGPU, with no installation, server, or account. The TypeScript-based tool supports both model training and inference and was built to let students run torch code in classes without setup.
embedl.com (by the creator of HF Viewer) hosts 1000+ models optimized for edge devices like NVIDIA Orin Nano Super, AGX Thor, and RTX GPUs. Users can run models remotely on real devices through a free device farm with detailed profiling, or on their own hardware. It targets developers who need to benchmark edge LLM deployment without owning the hardware.
embedl.com (by the creator of HF Viewer) hosts 1000+ models optimized for edge devices like NVIDIA Orin Nano Super, AGX Thor, and RTX GPUs. Users can run models remotely on real devices through a free device farm with detailed profiling, or on their own hardware. It targets developers who need to benchmark edge LLM deployment without owning the hardware.