Samsung Debuts zHBM Prototype, Stacking Memory Directly on AI Accelerators

A new storage-inspired memory technology could enable GPUs to reach multiple terabytes of memory, according to a report from The Register. The approach leverages storage-class memory concepts to dramatically increase GPU memory capacity, potentially transforming AI workloads that are currently limited by VRAM sizes.

Developments·Hardware

Micron Explores Near-GPU NAND Flash to Run Bigger LLMs

TechPowerUp reports Micron is exploring placing NAND flash near GPUs to expand memory capacity for running larger LLMs. The approach could let models exceed VRAM limits, with commenters noting particular interest for unified-memory devices.

TechPowerUp reports Micron is exploring placing NAND flash near GPUs to expand memory capacity for running larger LLMs. The approach could let models exceed VRAM limits, with commenters noting particular interest for unified-memory devices.

Developments·Breakthrough

Chinese researchers develop single-electron memory chip to break AI memory bottleneck

Chinese researchers have created a memory chip that stores data using a single electron, dramatically reducing power consumption. The technology could enable AI chat models to run on phones without losing memory during conversations, potentially eliminating the need for cloud connectivity.

Chinese researchers have created a memory chip that stores data using a single electron, dramatically reducing power consumption. The technology could enable AI chat models to run on phones without losing memory during conversations, potentially eliminating the need for cloud connectivity.

A new storage technology called High-Bandwidth Flash (HBF) has been proposed for efficiently storing large AI model weights. The approach aims to reduce latency and power consumption compared to traditional storage methods, potentially enabling faster model loading and inference. The concept was detailed in an IEEE Spectrum article shared on Hacker News.

SK hynix, with SanDisk, announced a new High Bandwidth Flash (HBF) standard aimed at easing AI inference bottlenecks by delivering up to 3TB/s bandwidth. The standard could enable faster local model inference, though cost concerns remain.