Releases·Tool Release
Djembed: open-source Java embedding + rerank server (ONNX Runtime) with Cohere/OpenAI APIs, benchmarked vs TEI and Infinity
A RAG backend developer released Djembed, a Java server built on ONNX Runtime that exposes both Cohere- and OpenAI-compatible embedding and rerank APIs and batches across concurrent requests. Benchmarks on an RTX 3090 Ti with bge-m3 and bge-reranker-v2-m3 show throughput competitive with or exceeding Hugging Face TEI and Infinity under concurrency, with some caveats on single-client and p99 latency. The project targets practitioners self-hosting RAG inference infrastructure.
2 engagement1 sourceTue, Sep 29, 2026, 07:57 PM
AI Overview
- Djembed runs any ONNX model with cross-request batching on RTX 3090 Ti; bge-m3 embeddings match baselines at cosine ≥ 0.9999
- Vs TEI 1.9 with bge-reranker-v2-m3: 0.85–0.97× fp32 single-client, 1.59–1.78× at 128 clients, 0.98–1.46× fp16
- Vs Infinity 0.0.77 fp16: 4.5–6× on single short queries, 1.03–1.17× on batches; p99 runs 7–17% higher under saturation
- Currently NVIDIA/CUDA only; CPU works with unpacked ONNX models, AMD support planned; benchmarks reproduce with one command on GitHub
Related
Social·Community
For those self-hosting embeddings and rerankers (TEI, Infinity, Ollama) - what are you actually running?
A practitioner asked production retrieval operators which self-hosted inference server they settled on for embedding and reranking models, citing dissatisfaction with README claims. The thread weighs TEI, Infinity, Ollama, and SIE on multi-model density, reranking support, and operational overhead. It matters to teams deciding between one-model-per-container and multi-model GPU serving.
A practitioner asked production retrieval operators which self-hosted inference server they settled on for embedding and reranking models, citing dissatisfaction with README claims. The thread weighs TEI, Infinity, Ollama, and SIE on multi-model density, reranking support, and operational overhead. It matters to teams deciding between one-model-per-container and multi-model GPU serving.
10 engagement1 sourceredditTue, Sep 15, 2026, 03:43 AM
Social·Community
Went through most of the ways to self-host embeddings and reranking, notes on each
A practitioner posted notes from months of experience self-hosting embedders and rerankers, ranking options by operational pain. They cover DIY sentence-transformers, Ollama, TEI, Infinity, and SIE, landing on Infinity for retrieval. The key insight is that GPU utilization, not features, drives the choice.
A practitioner posted notes from months of experience self-hosting embedders and rerankers, ranking options by operational pain. They cover DIY sentence-transformers, Ollama, TEI, Infinity, and SIE, landing on Infinity for retrieval. The key insight is that GPU utilization, not features, drives the choice.
4 engagement1 sourceredditThu, Sep 24, 2026, 03:27 AM
Products·building
TEI-compatible bge-m3 embeddings on AMD RDNA consumer GPUs
A workaround approach to run TEI-compatible bge-m3 embeddings on AMD consumer RDNA GPUs (like Strix Halo and Radeon R9 7000), which Hugging Face's TEI does not officially support. The author needed higher embedding throughput for large-scale RAGFlow ingestion. It addresses the bottleneck where consumer AMD cards can't run TEI directly despite usable ROCm support.
A workaround approach to run TEI-compatible bge-m3 embeddings on AMD consumer RDNA GPUs (like Strix Halo and Radeon R9 7000), which Hugging Face's TEI does not officially support. The author needed higher embedding throughput for large-scale RAGFlow ingestion. It addresses the bottleneck where consumer AMD cards can't run TEI directly despite usable ROCm support.
1 engagement1 sourceredditSun, Sep 13, 2026, 08:31 PM
Social·Community
Dev consolidates doc-QA agent's four model services onto Superlinked's inference engine, weighing ops savings against blast radius
A Reddit user describes a document-QA agent whose four small models — bge-m3, a cross-encoder reranker, GLiNER, and a small Qwen drafter — run as four separate containerized services. The poster asks whether anyone has consolidated the stack, citing OOM incidents and maintenance overhead, and reports the four deployments consume most of their week.
A Reddit user describes a document-QA agent whose four small models — bge-m3, a cross-encoder reranker, GLiNER, and a small Qwen drafter — run as four separate containerized services. The poster asks whether anyone has consolidated the stack, citing OOM incidents and maintenance overhead, and reports the four deployments consume most of their week.
10 engagement1 sourceredditTue, Sep 8, 2026, 03:42 AM
Releases·Tool Release
embedcache: a Go caching proxy for Ollama embeddings cuts GPU time by avoiding recomputation
A developer released embedcache, a single Go binary that acts as a caching proxy for embedding backends like Ollama, vLLM, and TEI. It caches computed embeddings in memory, serving duplicates in under a millisecond and only sending new text to the GPU. The author measured 49.7% duplicate tokens in a real RAG workload, suggesting significant GPU cost savings.
A developer released embedcache, a single Go binary that acts as a caching proxy for embedding backends like Ollama, vLLM, and TEI. It caches computed embeddings in memory, serving duplicates in under a millisecond and only sending new text to the GPU. The author measured 49.7% duplicate tokens in a real RAG workload, suggesting significant GPU cost savings.
2 engagement1 sourceredditWed, Jul 15, 2026, 05:00 PM