Ollama not utilizing more than 50GB VRAM

A user with an AMD Ryzen AI Max+ 395 system (128GB RAM, 96GB UMA VRAM) reports that Ollama only uses 7% CPU and 93% GPU for a 35B parameter model, failing to fully leverage the unified memory. The model loads and runs, but the GPU utilization is not 100%, indicating a potential optimization issue with Ollama on this hardware.

A developer detailed a CPU-only RAG stack using Postgres, pgvector, and small ONNX models, reporting high latency (~35s per grounded answer). The community discussion focuses on optimization strategies for CPU-bound retrieval and generation.

Social·Community

User struggles to offload DeepSeek-V4-Flash-DSpark to RAM in vLLM

A Reddit user reports difficulty running the 156 GB DeepSeek-V4-Flash-DSpark model on 96 GB VRAM with vLLM's CPU offloading, despite adding `--cpu-offload-gb 256` and reducing `--max-model-len` to 32768. The model crashes with a CUDA out-of-memory error, suggesting vLLM's offload feature may not work properly with this MoE model. The user notes that llama.cpp works fine but seeks a vLLM solution.

A Reddit user reports difficulty running the 156 GB DeepSeek-V4-Flash-DSpark model on 96 GB VRAM with vLLM's CPU offloading, despite adding `--cpu-offload-gb 256` and reducing `--max-model-len` to 32768. The model crashes with a CUDA out-of-memory error, suggesting vLLM's offload feature may not work properly with this MoE model. The user notes that llama.cpp works fine but seeks a vLLM solution.

Social·Community

llama.cpp users hit silent stalls and stoppages loading models near max VRAM on Windows CUDA

A Reddit user reports that loading a model with a context close to max VRAM in llama.cpp on Windows (CUDA 13.4) causes prompt processing to stall or generation to silently stop, with no OOM or error in server logs. The issue matters to practitioners running large-context inference on fully utilized GPUs.

A Reddit user reports that loading a model with a context close to max VRAM in llama.cpp on Windows (CUDA 13.4) causes prompt processing to stall or generation to silently stop, with no OOM or error in server logs. The issue matters to practitioners running large-context inference on fully utilized GPUs.

A user documents running DeepSeek V4 Flash in UD-Q3_K_M quantization on a single RTX 4090 with 64k context, providing exact hardware and software configuration. The setup requires 128 GB of RAM as a minimum, with the model file at 121 GB. This serves as a reference for others attempting similar configurations.