An Open Intelligence article examines Tencent's AngelSlim, an open toolkit for shrinking large AI models without major accuracy loss. It covers quantization, speculative decoding, sparse attention, token pruning, caching, and distillation as complementary compression techniques. For practitioners, it maps which methods fit inference-cost reduction across serving scenarios.
Medium post lays out a practical roadmap to teacher-student distillation for local AI
Medium post lays out a practical roadmap to teacher-student distillation for local AI
An engineering team posted a writeup on stress-testing massive LLM context systems using FP8/INT4 KV cache compression, addressing PagedAttention memory fragmentation at 1M+ token contexts. They claim dynamic KV cache optimization preserves perplexity and throughput while cutting VRAM overhead and inference latency, and invite community discussion.
An engineering team posted a writeup on stress-testing massive LLM context systems using FP8/INT4 KV cache compression, addressing PagedAttention memory fragmentation at 1M+ token contexts. They claim dynamic KV cache optimization preserves perplexity and throughput while cutting VRAM overhead and inference latency, and invite community discussion.
An arXiv study overtrains ~1B-parameter dense transformers across tokenization schemes (tokens, bytes, bytes-with-EOT) and objectives (distillation vs cross-entropy) on up to 1T bytes of data. Byte models start weaker but surpass token models with more compute, reaching higher performance ceilings, greater data efficiency, and lower logit storage costs. Extrapolated scaling laws predict distilled byte models beating Llama 3.2-1B and Gemma baselines on downstream tasks.
An arXiv study overtrains ~1B-parameter dense transformers across tokenization schemes (tokens, bytes, bytes-with-EOT) and objectives (distillation vs cross-entropy) on up to 1T bytes of data. Byte models start weaker but surpass token models with more compute, reaching higher performance ceilings, greater data efficiency, and lower logit storage costs. Extrapolated scaling laws predict distilled byte models beating Llama 3.2-1B and Gemma baselines on downstream tasks.
Tencent released an open-source 780B-parameter mixture-of-experts model. The model uses 256 experts, supports a 1-million-token context window, and weighs 814 GB even after FP8 quantization. The release gives practitioners access to frontier-scale open weights, though the size makes deployment demanding.
Tencent released an open-source 780B-parameter mixture-of-experts model. The model uses 256 experts, supports a 1-million-token context window, and weighs 814 GB even after FP8 quantization. The release gives practitioners access to frontier-scale open weights, though the size makes deployment demanding.
An arXiv paper introduces SpecQuant, a training-free inference framework that derives INT4, FP8, and FP16 variants from a shared base model and routes queries by predicted complexity. Evaluated on Qwen2.5 models, it delivers 35-43% speedups with under 2% accuracy loss, targeting on-device LLM deployment on consumer hardware.
An arXiv paper introduces SpecQuant, a training-free inference framework that derives INT4, FP8, and FP16 variants from a shared base model and routes queries by predicted complexity. Evaluated on Qwen2.5 models, it delivers 35-43% speedups with under 2% accuracy loss, targeting on-device LLM deployment on consumer hardware.