Researchers introduce HoliBench, an open-source benchmarking and deployment toolkit that jointly characterizes accuracy, latency, and energy for foundation models across heterogeneous platforms from single-board computers to GPU servers. It addresses the gap left by capability-only benchmarks and mutually incompatible hardware profilers, enabling deployment decisions for CPS-IoT applications where energy and latency matter as much as accuracy.
Researchers introduce Hydra, a phase-aware workload characterization framework for LLM inference on edge SoCs. It instruments HuggingFace Transformers and llama.cpp with a shared per-prompt timing schema, fusing records with hardware telemetry to analyze prefill and decode phases. The framework enables multi-dimensional characterization of performance, resource utilization, and efficiency across edge platforms.
Researchers introduce Hydra, a phase-aware workload characterization framework for LLM inference on edge SoCs. It instruments HuggingFace Transformers and llama.cpp with a shared per-prompt timing schema, fusing records with hardware telemetry to analyze prefill and decode phases. The framework enables multi-dimensional characterization of performance, resource utilization, and efficiency across edge platforms.
An arXiv paper presents a controlled measurement study of self-hosted LLM inference across edge and near-edge hardware, balancing quality, latency, model footprint, and energy. It tests an NVIDIA Jetson AGX Orin and a near-edge server in both CPU-only and GPU-enabled modes. The work gives practitioners empirical grounding for choosing where and how to serve LLM-backed web services.
An arXiv paper presents a controlled measurement study of self-hosted LLM inference across edge and near-edge hardware, balancing quality, latency, model footprint, and energy. It tests an NVIDIA Jetson AGX Orin and a near-edge server in both CPU-only and GPU-enabled modes. The work gives practitioners empirical grounding for choosing where and how to serve LLM-backed web services.
Researchers introduced CARB, a characterization-guided framework for predicting CNN inference cost—energy, latency, and peak memory—before deployment. It is based on a workload characterization study of 13,419 CNN configurations on RTX 5090 and RTX 3080 GPUs, revealing distinct scaling behaviors for each cost metric.
Researchers introduced CARB, a characterization-guided framework for predicting CNN inference cost—energy, latency, and peak memory—before deployment. It is based on a workload characterization study of 13,419 CNN configurations on RTX 5090 and RTX 3080 GPUs, revealing distinct scaling behaviors for each cost metric.
A developer released Quantprobe, an open-source memory allocation optimization framework that lets users run large local LLMs on old hardware. It achieves 30B parameter models at 22 tokens per second on a 6GB GPU with 16GB RAM. The tool aims to maximize performance for specific use cases.
A developer released Quantprobe, an open-source memory allocation optimization framework that lets users run large local LLMs on old hardware. It achieves 30B parameter models at 22 tokens per second on a 6GB GPU with 16GB RAM. The tool aims to maximize performance for specific use cases.