lucebox

Luce-Org/lucebox

Not yet reviewedDeveloper tools

LLM speculative inference server for heterogeneous hardware & consumer GPUs

View on GitHubCompare 12 alternatives How we review projects

Topics

cuda cuda-kernels dflash heterogeneous-computing kernel llama-cpp local-ai luce megakernel pflash poolside qwen r9700 rocm rtx3090 spark

Install

Not available through Techandria. A library has no Techandria install path. Running an arbitrary program is a different risk from adding a reviewed skill.

You can still get the code from the source and read it yourself:

git clone https://github.com/Luce-Org/lucebox

What we checked

Not yet reviewed

Discovery metadata only. Nothing has been scanned or approved, so this is a listing — not a recommendation to run the code.

Nothing has been scanned for this repository yet, so there is nothing to report. Listing here is not permission or a recommendation to run the code.

Maintain this project? Add the review badge to your README

The badge shows this project's current review state and updates on its own — including if the state changes. It links back to this page.

[![Techandria](https://techandria.com/badge/Luce-Org/lucebox.svg)](https://techandria.com/project/Luce-Org/lucebox)

README

Loading README from GitHub…

Speculative inference for heterogeneous machines and consumer GPUs.
Custom kernels, speculative prefill and decoding, tuned for each model and hardware target.

Inference Engine Optimizations

Optimization	Measured setup	Result
DFlash2	Qwen 3.8 27B on one R9700	208.1 tok/s average, 227.8 tok/s peak
DSpark	DeepSeek V4 on Strix Halo, native top-6	32.7 tok/s high-acceptance median; 27.9 tok/s mixed-eval average
PFlash + KVFlash	Laguna XS 2.1 33B at 256K on RTX 3090	6.1× prefill, 411 s to 67.3 s
Luce Spark	Laguna XS.2 33B on RTX 3090	~100 tok/s in 14.6 GiB
KVFlash	Laguna XS 2.1 33B at 256K on RTX 3090	152.3 tok/s with an 8K pool
Heterogeneous execution	DeepSeek V4 on R9700 + Strix Halo	86 tok/s decode; 788 tok/s prefill at 2K
Paged attention + continuous batching	Qwen 3.8 27B + DFlash2 on R9700; DeepSeek V4 Flash AR on Strix Halo	300.9 tok/s total at 5 clients (Qwen); 48.4 tok/s output-window at 4 clients (DeepSeek)
Three-tier experts	DeepSeek V4.1 Flash on R9700 + Strix Halo + SSD	25-26 tok/s decode on code; 118-121 tok/s prefill at 4K, at the profile's 128K context
Megakernel	Qwen 3.5 0.8B on RTX 3090	413 tok/s, 1.87 tok/J
Image generation	MOSS-Image 5B at 1024×1024, 8 steps, on one R9700	3.1 s per image in luce_server; 1.5-1.8× a DGX Spark on ComfyUI, 4.0× stable-diffusion.cpp
Vision (image input)	Qwen 3.8 27B vision on R9700 + DeepSeek V4 Flash Vision on Strix Halo, together	3.2× the image-question throughput of a DGX Spark (58 vs 18 a minute); 2.4× faster at 8 users with the same model file

Supported Models and Drafters

Each badge opens the exact weights used by the measured setup. Drafter badges open the published quant, or the source checkpoint when conversion is required.

Model	Optimization (blog)	Phase	Result
Qwen 3.5 0.8B		Prefill + decode	1.9× prefill; 1.55× decode
Qwen 3.8 27B _{on R9700}		Decode	6.4× vs Lucebox AR; 3.8× vs llama.cpp with the same drafter
Laguna XS 2.1 33B		Prefill	6.1×, 411 s to 67.3 s at 256K
Laguna XS 2.1 33B		Decode	1.7× at 256K
Gemma 4 26B-A4B		Decode	1.31×
Gemma 4 31B IT		Decode	3.2×
DeepSeek V4 Flash _{on Strix Halo}		Decode	42 tok/s at 8K and 39 tok/s on code and math with the plain launch (PR #729)
DeepSeek V4.1 Flash _{R9700 + Strix Halo + SSD, ds41-halo (applied by default)}		Decode	25.4-25.8 tok/s on code, 16.8-17.5 tok/s after a 4K prompt, at the profile's 128K context (guide)
DeepSeek V4.1 Flash _{R9700 + Ryzen AI Max+ PRO 495 192 GB, --profile ds41-r9700-gorgon}		Prefill + decode	985-1,019 tok/s prefill at 14K-25K, 48.7 tok/s decode on code, every expert resident (guide)
Ling 3.0 Flash 124B-A5.1B _{on DGX Spark}	Autoregressive	Decode	34.6 tok/s median AR
Qwen 3.8 Flash-Next _{on Strix Halo}		Decode	40 tok/s on a short reply, 1.8× autoregressive decode (guide)
Qwen 3.8 Flash-Next _{R9700 + Strix Halo, --profile qwen-next-r9700-strix}		Prefill + decode	75 tok/s decode, 1.86× the Strix Halo alone; 1,820 tok/s prefill at 16K (guide)
Qwen 3.8 27B Vision _{on R9700}		Image questions	2.4× vs llama.cpp on DGX Spark at 8 users (10.5 s vs 25.4 s); 2.1× for one user
DeepSeek V4 Flash Vision _{encoder on R9700, 4 users at once}		Image questions	1.5× sooner first token with 16 images when the R9700 encodes
MOSS-Image SFT 5B _{on R9700, text to image}		Image generation	3.1 s per 1024×1024 image at 8 steps in luce_server; 1.5-1.8× a DGX Spark on ComfyUI (results)

Tested Machines (GPU/APU)

The engine is not tied to one reference card. NVIDIA architectures are selected by CMake; HIP builds should target the device's exact gfx architecture.

	Architecture	Hardware	Runtime	Details
	RDNA4 `gfx1201`	Radeon AI PRO R9700	ROCm 7.2	Qwen 3.8 R9700 quick start
	RDNA3.5 `gfx1151`	Ryzen AI MAX+ 395 / Strix Halo	ROCm 7.2	DeepSeek V4 Strix profile
	RDNA3 `gfx1100`	Radeon RX 7900 XT / XTX	ROCm 6+	DeepSeek V4 dual AMD profile
	Ampere `sm_86`	RTX 3090	CUDA 12+	Qwen 3.8 NVLink result and Megakernel results
	Blackwell `sm_120`	RTX 5090	CUDA 12.8+	Qwen 3.8 single-GPU result
	Blackwell `sm_121`	DGX Spark / GB10	CUDA 12.9	Qwen 3.5 NVFP4 results
	Ada `sm_89`	RTX 4090	CUDA 12+	Linux and WSL2 community runs
	Turing `sm_75`	RTX 2080 Ti	CUDA 12.0	DFlash results
	Volta `sm_70`, Pascal `sm_61`	V100, P40	CUDA 12.0	CUDA quick start
Not pictured	Blackwell `sm_110`	Jetson AGX Thor	CUDA 13.0	Thor quick start

Single-device results

Hardware	Model	Measured result
R9700	Qwen 3.8 27B UD-IQ4_XS + DFlash2 source	208.1 tok/s HumanEval average; 227.8 tok/s best request
Strix Halo	DeepSeek V4 ROCmFPX MIX Strix + DSpark Q4RMFP4	42 tok/s decode and 320 tok/s prefill at 8K, 36 tok/s at 123K, 39 tok/s on code and math, 25 tok/s on prose, all six routed experts, plain launch (PR #729)
RTX 5090	Qwen 3.8 27B	110.6 tok/s for a 26,758-token prompt and 1,024-token continuation (PR #637)

Heterogeneous and parallel results

Hardware	Configuration	Measured result
2x RTX 3090 + NVLink	Qwen 3.8 target tensor parallel + DFlash2	79.7 tok/s, 2.16× autoregressive decode (PR #637)
RX 7900 XT + Strix Halo	DeepSeek V4 with all six experts + DSpark verification width 4	45.0 to 47.7 tok/s decode; 111.2 tok/s prefill at 132,981 tokens (PR #604)
R9700 + Strix Halo	DeepSeek V4 across both AMD devices	86 tok/s decode; 788 tok/s prefill at 2K
R9700 + Strix Halo	Qwen 3.8 Flash-Next, routed experts on the Strix Halo + MTP	75.5 tok/s decode (1.86× the Strix Halo alone); 44-45 tok/s after a 16K prompt; 1,820 tok/s prefill at 16K (PR #827)

These runs use different prompts, quantizations, and inference policies. They show which configurations work; they are not a cross-hardware ranking.

Recommended Setups

See Recommended server setups for the model and hardware matrix, including single-GPU and mixed-GPU profiles.

The DS4 guide also documents the Strix long-context sparse-verifier profile and Qwen3-0.6B PFlash integration. PFlash is lossy prompt compression; keep it off for exact-retrieval and matched true-context benchmarks.

Client Harnesses

harness/ runs Lucebox through popular coding clients and checks server compatibility.

Client	Launcher
Claude Code	`run_claude_code.sh`
Codex	`run_codex.sh`
OpenCode	`run_opencode.sh`
Hermes	`run_hermes.sh`
OMP	`run_omp.sh`
Pi	`run_pi.sh`
OpenClaw	`run_openclaw.sh`
Open WebUI	`run_openwebui.sh`

Set the server binary and model paths, then run a launcher:

LUCE_SERVER_BIN=server/build/luce_server \
LUCE_TARGET=server/models/Qwen3.8-27B-UD-IQ4_XS.gguf \
LUCE_DRAFT=server/models/draft/qwen38-dflash2-q8_0.gguf \
MAX_CTX=32768 \
harness/clients/run_codex.sh

See the harness guide for setup, no-draft targets, and benchmarks.

Quick Start With Docker

Prebuilt images on GHCR track main. Mount the weights and serve the OpenAI-compatible API on :8000.

GPU	Image tag
NVIDIA (CUDA 12+)	`:cuda12`
AMD (ROCm 6+)	`:rocm`

Put the target in server/models/ and its matching drafter in server/models/draft/.

Lucebox prebuilt Docker images for NVIDIA and AMD

Run the image for your GPU:

# NVIDIA
docker run --rm --gpus all -p 8000:8080 \
  -v "$PWD/server/models:/opt/lucebox-hub/server/models" \
  ghcr.io/luce-org/lucebox-hub:cuda12

# AMD
docker run --rm --device /dev/kfd --device /dev/dri \
  --group-add video --group-add render --security-opt seccomp=unconfined \
  -p 8000:8080 -v "$PWD/server/models:/opt/lucebox-hub/server/models" \
  ghcr.io/luce-org/lucebox-hub:rocm

The container picks the GPU the model fits on (a discrete card before an integrated one, else the largest) and sizes the context