slaren/dsh-llm-llamacpp

Adds llama.cpp servers as DSH model providers and shows live prompt and decode progress in the composer dock.

Registers one or more llama.cpp servers as provider routes, each with its own base URL, optional alias and optional credential reference, and surfaces what the server is actually doing. Model facts come from the server rather than a static table: the model list from /v1/models, loaded context, output cap and modalities from /props, and reasoning levels from the server's own template list. The composer dock shows a pill with prompt progress and an ETA, current prompt and decode tokens per second, per-slot context and KV use, cache hit rate, and the raw metrics. Sessions are pinned to a stable llama.cpp slot and idle slots are reused without evicting active conversations, which is what keeps prompt caches warm when several sessions run at once. Streaming behaviour matches the stock route, including retryable failure codes, assistant reasoning replay, image handling, parallel tool calls and disjoint usage accounting, and nothing is intercepted or rewritten behind your back.

Models & Providers ★ 0
View on GitHub ↗

Install

dsh plugin --profile web add github:slaren/dsh-llm-llamacpp

Private package, not published to npm, so the GitHub form is the only install path; the author states it was tested with DSH 0.2.x and a current llama-server build. Servers are declared by hand in the plugin's servers map rather than auto-discovered, and removing a route is a hand edit of that map. The README shows how to launch llama-server with the flags the plugin reads: --metrics for throughput, prompt-cache hit rate and speculative decoding, --slots for the slot view and session-context line, and --jinja.

Compatibility

DSH 0.2.x with a llama-server exposing the OpenAI-compatible API. It expects /v1/models for the model list, /props for loaded context, output cap and modalities, the server's own /apply-template list for reasoning levels, and /metrics and /slots for the live readings. Headless use works for routes, discovery and streaming; the dock pill and dialog simply do not mount.

Details

Alternatives

NOirBRight/dsh-llm-ollama · shengbinxu/dsh-llm-latency · btspoony/dsh-llm-fallbacks

More plugins in Models & Providers

Browse more in Models & Providers

Guides for Models & Providers plugins