🌍 English | Türkçe | العربية | 日本語 | 中文
Fine-tune and post-train LLMs in one command. No SSH, no config hell.
Website · Quick Start · Web UI · Config · Docs · Commands · Models · Discord · Telegram · Product Hunt
Soup turns the pain of LLM fine-tuning into a simple workflow. One config, one command, done.
pip install "soup-cli[train]" # add [train] to fine-tune; bare `soup-cli` is the light CLI
soup init --template chat
soup trainFine-tune an 8B model on a 4 GB laptop GPU. Layer streaming keeps the frozen base out of
VRAM and feeds it to the GPU one decoder layer at a time. Measured on an RTX 3050 Laptop 4 GB:
Llama-3.1-8B-Instruct + NF4 at 119.6 tok/s at 952 MHz, 3.32 GB peak (pre-repair) — bit-exact
against a normal resident run, and reproduced independently on an H100 at 113.00 tok/s in the same
3.32 GB. (Both figures were measured on v0.72.2, before the v0.73.0 correctness repair that cost
−4.8% at 32B. The post-repair run, on another RTX 3050 Laptop 4 GB (@umran666, #1103), reads
208.6 tok/s at 1935–1957 MHz in a 2.40 GB peak. That is 59% of its same-session GEMM ceiling
against 68%: about 14% slower per unit clock, so the raw 1.74x gain is the clock, not a speedup.
Large-layer streaming also changed the memory pattern between the two rows, so even the fraction
is not like-for-like — the record.) Opt-in (stream_layers: true)
and still BETA —
how it works ·
all measurements · paper ·
check it yourself on a free Colab T4 (caps the process to
4 GB, then asserts a streamed model is bit-identical to a normal one)
Llama-3.1-8B-Instruct + NF4, LoRA, batch 1, seq 512 on an RTX 3050 Laptop 4 GB — 3.32 GB peak, 119.6 tok/s at 952 MHz (pre-repair, v0.72.2; post-#331: 208.6 tok/s at 1935 MHz in 2.40 GB, not directly comparable, @umran666, #1103). Full video (90s)
Training LLMs is still painful. Even experienced teams spend 30-50% of their time fighting infrastructure instead of improving models. Soup fixes that.
- Zero SSH. Never SSH into a broken GPU box again.
- One config. A simple YAML file is all you need.
- Auto everything. Batch size, GPU detection, quantization — handled.
- Works locally. Train on your own GPU with QLoRA. No cloud required.
v0.77.1 — a patch: soup data keeps its input, soup plan agrees with soup train, and
GRPO trains on Apple Silicon. Nothing in v0.77.0 is taken back. That release refuses
settings v0.76 warned about or ignored, so a config that loaded on v0.76 can stop loading:
coming from 0.76.x, run soup doctor --config soup.yaml first and read the
v0.77.0 release notes.
soup datacommands no longer write over their input. Eighteen of them (dedup,filter,convert,merge,sample,pii,toxicityand the rest) refuse an--outputthat is the same file as an input, under any spelling, before they read anything, and a default output name that the input already has is refused too. Write to a new file instead.soup planandsoup applyrefuse the configssoup trainrefuses, with the same message, and write no plan.- GRPO trains natively on Apple Silicon.
task: grpowithbackend: mlxruns Soup's own MLX loop on the objective the transformers path trains, with the built-inaccuracyorformatreward.soup infer --backend mlxgenerates from MLX bases and native text LoRA adapters without Torch. - One peak-VRAM estimate for
soup train's pre-flight,soup profileandsoup plan, calibrated on 39 measured runs (record). For dense models up to about 3.5B parameters, the range it was fitted on, the pre-flight now refuses some long-sequence, large-vocabulary runs it let through, and--allow-oom-attemptstill launches them. Outside that range (a larger or mixture-of-experts model,mediumorselectivecheckpointing,use_liger/use_cut_ce,backend: unsloth) a run that does not fit gets the estimate as a warning and starts, unless its weights, the CUDA context and the gradients and optimizer state of the parameters it trains are more than the card holds (a 7B full fine-tune on 80 GB, Llama-3.1-70B in 4-bit on 24 GB): that run is refused there too. LoRA runs and runs that train a subset (Spectrum, LISA, freezing) are judged on fewer terms, and--allow-oom-attemptstill launches a refused run. - External programs start by an absolute path found on
PATH. The current directory and relativePATHentries are no longer searched: a tool that only lives there needs its directory onPATH. - Also new.
soup ship --progress jsonl; unknown config keys written the way another library names them get the Soup field (learning_ratesuggestslr); and the Docker example below passes--yes, which a non-interactivesoup trainneeds since v0.77.0.
Older highlights live on the GitHub Releases page.
Soup is a command-line application, so the cleanest install gives it its own
environment and puts soup on your PATH:
# Light core: CLI + config + data tools, no PyTorch
pipx install soup-cli
uv tool install soup-cli # same idea, if you already use uv
# Add the training stack (torch, transformers, peft, trl, datasets, …)
pipx install "soup-cli[train]"
# Everything (train + serve + ui + data) in one shot
pipx install "soup-cli[all]"
# Or from GitHub (latest dev)
pipx install "git+https://github.com/MakazhanAlpamys/Soup.git"Already inside a virtualenv, a Colab notebook, or a Docker image? Use pip
directly, with the same names and extras:
pip install soup-cli
pip install "soup-cli[train]"
pip install "soup-cli[all]"
pip install git+https://github.com/MakazhanAlpamys/Soup.gitUse pip rather than pipx if you also want to import soup_cli from your own
code, since pipx deliberately isolates the application from everything else.
The full extras table (fast, mlx, serve, eval, ui, vision, audio, …) lives in
docs/models.md.
error: externally-managed-environment? That is PEP 668, not a Soup problem. Debian 12, Ubuntu 23.04 and later stoppipfrom writing into the system Python, becauseaptmanages those files too.pipxanduv toolsidestep it by giving Soup its own environment, which is why they are listed first above.python3 -m venv .venv && source .venv/bin/activatethen plainpipworks just as well.
Double quotes, not single.
"soup-cli[train]"is the only spelling that works in every shell —cmd.exe, PowerShell, bash and zsh. If you copied'soup-cli[train]'from an older tutorial and pip rejected it, that is the reason: why, and the exact error.
soup init, soup data …, and the other data/inspection commands work on the light install.
Fine-tuning (soup train) needs the [train] extra.
soup init # interactive wizard
soup init --template chat # or start from a templateTemplates: chat, code, tool-calling, medical, reasoning, vision, kto, orpo,
simpo, ipo, bco, rlhf, pretrain, moe, longcontext, embedding, audio.
soup train --config soup.yaml # LoRA, quantization, batching — all handled
soup chat --model ./output # talk to your model
soup push --model ./output --repo you/my-model
soup merge --adapter ./output # merge LoRA into the base
soup export --model ./output --format gguf --quant q4_k_m # GGUF for Ollama / llama.cppMore export targets (ONNX, TensorRT, BitNet) and deployment options live in
docs/serving-and-export.md.
Prefer a browser? soup ui serves a local dashboard for experiments,
training setup, live metrics, dataset exploration and model chat.
pip install "soup-cli[ui]"
soup ui
# Opens http://127.0.0.1:7860/?token=<token> and prints the same URL
A complete soup.yaml:
base: meta-llama/Llama-3.1-8B-Instruct
task: sft
# backend: unsloth # 2-5x faster, pip install "soup-cli[fast]"
data:
train: ./data/train.jsonl
format: alpaca
val_split: 0.1
training:
epochs: 3
lr: 2e-5
batch_size: auto
lora:
r: 64
alpha: 16
quantization: 4bit
output: ./outputconfig/schema.py is the single source of truth for every field. Advanced data, training,
and PEFT options are documented under Documentation.
Unknown config keys are rejected since v0.75. A key no model declares — a typo like
quantizaton, or a field that only exists on a newer Soup — used to validate clean and be discarded, so the run proceeded with the setting simply not applied. v0.74 reported it at load with the field you probably meant; from v0.75 the same config fails to load, so fix or remove the key rather than relying on it being ignored. See Unknown config keys.
The full feature reference lives in docs/. Start here:
| Guide | Covers |
|---|---|
| Training tasks & methods | SFT, DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/BCO, tool-calling, PRM, pre-training, distillation, classification, vision/audio/TTS, unlearning, RAFT/RA-DIT, loop-hardening detectors |
| PEFT, long context & efficiency | DoRA, LoRA+, rsLoRA, VeRA, OLoRA, NEFTune, PiSSA, ReLoRA, optimizer & PEFT zoo, LLaMA Pro, GaLore, YaRN/LongLoRA, packing, curriculum, auto-tuning |
| Performance & quantization | QAT, FP8, Quant Menu (I + II), KV-cache, NVFP4, save formats, Cut Cross-Entropy, gradient checkpointing, kernels, activation offloading, layer streaming, multi-GPU / DeepSpeed / FSDP |
| Data engineering | Formats, the Axolotl/LF-parity pipeline, data tools, synthetic generation & forge, quality scorecards, trace tooling, remote datasets, mixing, recipe DAGs |
| Evaluation & probes | Eval design/gate, eval-gated training, benchmarks, NLG metrics, calibration, Elo arena, diagnose, post-train X-ray probes, A/B, drift, tunability, soup advise |
| Serving & export | OpenAI-compatible server, batch inference, benchmarking, merge/export, Anthropic Messages endpoint, speculative decoding (train + measure your own draft), deploy autopilot, Web UI, Agent Forge |
| Adapters, registry & governance | Adapter lifecycle/management, model registry, Soup Cans, the data flywheel (soup loop), knowledge editing, steering, supply-chain controls (scan/sign/BOM/attest/audit/airgap) |
| Compliance & governance quickstart | HIPAA/SOC2/EU-AI-Act/SR-11-7 init templates, provenance (BOM/attest/repro-receipt), audit log, air-gap, model-card autogen (soup card), CI gate (soup ci init) |
| Backends, platform & ops | MLX/Unsloth backends, alternative hubs, HF Hub integration, autopilot, experiment tracking, plan/apply, env lockfiles, hardware-fit, completions, plugins, utility commands |
| Command reference | The full soup command list |
| Supported models & extras | Recommended model families, the VRAM size guide, the pip extras matrix |
Alpaca, ShareGPT, ChatML, preference pairs (DPO / ORPO / SimPO / IPO / KTO), vision, audio,
ASR, plaintext, embedding, RAFT and more — all auto-detected from JSONL, JSON, CSV, Parquet or
TXT, so in most cases you point data.train at a file and nothing else changes. Schemas with a
worked example per format, plus the data pipeline (remote URIs, streaming, sharding,
interleaving, vocab expansion, document ingestion), are in
docs/data.md, with native Sesame CSM audio training documented separately.
soup train --config soup.yaml # train (SFT/DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/...)
soup infer --model ./output --input prompts.jsonl --output results.jsonl # batch inference
soup chat --model ./output # interactive chat
soup serve --model ./output # OpenAI-compatible API server
soup ui # local browser dashboard
soup merge --adapter ./output # merge LoRA into the base model
soup export --model ./output --format gguf # export for deployment
soup eval benchmark --model ./output # evaluate
soup data inspect ./data/train.jsonl # dataset stats
soup recipes list # 100+ ready-made model recipes
soup autopilot --model <id> --data d.jsonl --goal chat # zero-config
soup doctor # check GPU / deps / environmentThe complete command list is in docs/commands.md.
Soup works with any text-generation model on the
HuggingFace Hub — if it loads with
AutoModelForCausalLM, it works, zero config changes. Llama 3.x/4, Qwen 2.5/3, Gemma 3, Mistral,
Mixtral, DeepSeek R1/V3, Phi-4, and 100+ others ship as ready-made recipes (soup recipes list).
| VRAM | Max model (QLoRA 4-bit) | Example |
|---|---|---|
| 8 GB | ~7B | Llama-3.1-8B, Mistral-7B |
| 16 GB | ~14B | Phi-4-14B, Qwen2.5-14B |
| 24 GB | ~34B | CodeLlama-34B, Yi-1.5-34B |
| 48 GB | ~70B | Llama-3.3-70B |
| 80 GB+ | 70B+ (full) or MoE | Mixtral-8x22B, DeepSeek-V3 |
Full model + vision tables and the optional-extras matrix are in docs/models.md.
Run Soup without installing CUDA or PyTorch locally (image published to GHCR on every release):
docker pull ghcr.io/makazhanalpamys/soup:latest
docker run --gpus all -v $(pwd):/workspace ghcr.io/makazhanalpamys/soup train --config soup.yaml --yes
docker compose up # or build locally- Python 3.11, 3.12 or 3.13 (those are the versions CI tests; 3.14+ is not supported yet because the PyTorch stack has not been validated there)
- GPU with CUDA (recommended), Apple Silicon (MPS), or CPU (experimental — very slow)
- 8 GB+ VRAM for 7B models with QLoRA
All training tasks run on CPU for testing (quantization auto-disabled). Optional extras
(train, all, fast, vision, qat, serve, serve-fast, ui, eval, deepspeed,
liger, mlx, onnx, tensorrt, …) are listed in
docs/models.md.
soup doctor # GPU, system resources, dependencies, and version in one placeCUDA wheels, version mismatches: docs/backends-and-ops.md.
git clone https://github.com/MakazhanAlpamys/Soup.git
cd Soup
pip install -e ".[dev]"
ruff check src/soup_cli/ tests/ # lint
pytest tests/ -v # unit tests (fast, no GPU)
pytest tests/ -m smoke -v # smoke tests (downloads a tiny model, trains)
pytest tests/ -m gpu --no-cov -v # GPU tests (need a CUDA card; report results, see CONTRIBUTING.md)
pre-commit install # optional: ruff lint+format on commitSee CONTRIBUTING.md for the full workflow and SECURITY.md to
report a vulnerability. Telemetry is strictly opt-in (SOUP_TELEMETRY=1, default off; see Privacy Policy).
Soup is Apache-2.0 and free — and stays that way. It is built and maintained in the open on a single 4 GB laptop, which is why every performance number in these docs is measured rather than claimed.
If Soup saved you a training run, starring the repo helps most, and it costs nothing. If you would like to fund the work directly:
❤️ Donate — one-off, any amount (use Change amount on the checkout page). Payments are processed by Stripe under the maintainer's registered business, MePlay, Inc. — that name, not "Soup", is what appears on the checkout page and on your card statement.
Donations buy GPU time for the hardware-gated work — multi-GPU, 8B+ validation, Apple Silicon — that a single 4 GB laptop cannot reach.
The other way to move exactly those items is hardware itself. They ship behind honest
"requires <hardware>" gates rather than unverified claims, so if you have access to a bigger
box — or GPU credits going unused — running one of the
help wanted
issues and posting the numbers helps as much as funding the GPU time would. Those issues say
exactly what is blocked on hardware today.
Built by the community ❤️ — thank you to everyone who has contributed. See CONTRIBUTORS.md.
Bugs and feature requests belong in the issue tracker, questions in Discussions — both get answered faster and help the next person with the same problem.
For live chat, setup help, and everything that reads better as a conversation, join the Discord or the Telegram community. Anything that should still be findable in six months belongs in Issues or Discussions — a Discord answer helps one person, an issue helps everyone who hits the same thing. The Code of Conduct applies there too.
For anything that does not fit in public — security reports (see SECURITY.md), Code of Conduct matters, or press — email team@trysoup.dev. That is the project address and the right one for anything Soup-related. makazanalpamys@gmail.com is the maintainer's personal address; it reaches the same person and is a fine fallback.
Layer streaming — training an 8B model on a 4 GB laptop GPU by streaming the frozen base from host RAM one decoder layer at a time — is described in a preprint, together with the correctness protocol that verifies a streamed run against a resident one (forward and backward stated separately, because they are two claims and not one).
Makazhan, A. (2026). Exact Layer Streaming: LoRA Fine-Tuning of an 8B Model on a 4 GB Laptop GPU (v3). Zenodo. https://doi.org/10.5281/zenodo.21918325
Version 3 (13 August 2026) is current. The title and the claim are unchanged — 8B on 4 GB — and no measured number has changed since v1. What v3 does is withdraw an explanation we had published, which is also the shortest way to describe what the paper is for:
- Retracted in v3: "layer streaming is bound by host-to-device transfer, not by the GPU." That was an inference from the H100 replication below, and it had never been measured. We measured it on 11 August and it is false at the published configuration: deleting every host-to-device byte buys 1.4%, the compute stream waits on a copy for 0.20% of the step, and the step runs at 71.3% of that card's same-session GEMM ceiling. The largest streaming-specific cost is the per-layer NF4 dequantisation, at 9.8% (the record). Every measurement stands; the replication survives in a weaker form — the constraint is common to both machines and is not the GPU's compute.
- Replication on hardware nothing like the original (added in v2): 119.6 tok/s on the RTX 3050 (at 952 MHz) against a median 113.00 on an H100, at the same 3.32 GB peak. Both pre-