A Multilingual, Multimodal Translation Model Family
Multimodal Translation (Vision & Audio), Controlled Dubbing, and Long-Document Translation
🌐 Demo ·
⚡ Free API ·
🤗 Hugging Face ·
ModelScope ·
📚 Report ·
🤗 Papers ·
🤗 Benchmarks ·
🐧 QQ
Important
New: Index-Translate-Omni 2B / 9B / 35BA3B brings together text translation, image translation, Chinese/English speech recognition, and bidirectional Chinese–English speech-to-text translation. Downloads and benchmarks.
| Model | Hugging Face | ModelScope | Release status |
|---|---|---|---|
| Index-Translate-Omni 2B | 2B | 2B | Available |
| Index-Translate-Omni 9B | 9B | 9B | Available |
| Index-Translate-Omni 35BA3B | 35BA3B | 35BA3B | Available |
The checkpoints use the native package layout. Follow the native inference guide and each model card.
| Model | FLORES COMET22 ↑ | WMT24++ COMET22 ↑ | WMT26 Judge ↑ | MEME ↑ |
|---|---|---|---|---|
| Index-Translate 2B | 0.8655 | 0.8489 | 60.26 | 0.6443 |
| Index-Translate-Omni 2B | 0.8655 | 0.8489 | 60.26 | 0.6443 |
| Index-Translate 9B | 0.8789 | 0.8601 | 75.35 | 0.7387 |
| Index-Translate-Omni 9B | 0.8789 | 0.8601 | 75.35 | 0.7387 |
| Index-Translate 35BA3B (preview) | 0.8794 | 0.8586 | 76.76 | 0.7405 |
| Index-Translate-Omni 35BA3B | 0.8795 | 0.8593 | 78.01 | 0.7405 |
| Model | instTrans Quality / IFscore ↑ | instTrans_minor Quality / IFscore ↑ / Off-target ↓ | IFMTBench XCOMET-XXL / IFscore ↑ |
|---|---|---|---|
| Index-Translate 2B | 0.5391 / 0.7569 | 0.3050 / 0.6586 / 3.97% | 0.7712 / 0.7584 |
| Index-Translate-Omni 2B | 0.5391 / 0.7569 | 0.3050 / 0.6586 / 3.97% | 0.7712 / 0.7584 |
| Index-Translate 9B | 0.6771 / 0.8209 | 0.5222 / 0.7725 / 3.47% | 0.7957 / 0.8760 |
| Index-Translate-Omni 9B | 0.6771 / 0.8209 | 0.5222 / 0.7725 / 3.47% | 0.7957 / 0.8760 |
| Index-Translate 35BA3B (preview) | 0.6901 / 0.8336 | 0.5151 / 0.7715 / 4.05% | 0.7926 / 0.8991 |
| Index-Translate-Omni 35BA3B | 0.6867 / 0.8371 | 0.5163 / 0.7843 / 3.62% | 0.7934 / 0.9070 |
Original Index-Translate figures come from the published benchmarks. The original 35BA3B reference is the published preview checkpoint.
CER is a percentage (lower is better). BLEU and chrF2 are higher-is-better. Translation cells show BLEU / chrF2 / target CER.
| Model | Chinese ASR CER ↓ | English ASR CER ↓ | English → Chinese | Chinese → English |
|---|---|---|---|---|
| Index-Translate-Omni 2B | 13.8113 | 34.2560 | 36.982 / 30.357 / 43.3452 | 30.747 / 51.152 / 48.0670 |
| Index-Translate-Omni 9B | 17.8917 | 14.4097 | 43.507 / 36.312 / 41.3767 | 34.656 / 55.376 / 46.3720 |
| Index-Translate-Omni 35BA3B | 13.0053 | 15.4950 | 40.074 / 32.862 / 44.8097 | 32.411 / 53.452 / 46.1700 |
35BA3B uses one shared audio encoder and connector for all four tasks, with repetition penalty 1.1 for English ASR and the original decoding for the other tasks. Results use the same 30 fixed samples per task. One English → Chinese case reached the 256-token limit with no target translation and remains in the score. Sample counts differ across model sizes.
COMET22 × 100 (higher is better). Each Omni model is compared with its Qwen3.5 baseline on paired evaluated subsets.
| Model | MMTIT ↑ | FLORES-rendered ↑ | Vistra ↑ |
|---|---|---|---|
| Qwen3.5-2B | 52.164 | 50.187 | 55.326 |
| Index-Translate-Omni 2B | 65.809 | 53.612 | 65.596 |
| Qwen3.5-9B | 73.814 | 67.804 | 68.963 |
| Index-Translate-Omni 9B | 77.604 | 68.438 | 71.815 |
| Qwen3.5-35B-A3B | 75.602 | 70.005 | 71.067 |
| Index-Translate-Omni 35BA3B | 77.011 | 69.891 | 71.482 |
35BA3B reuses the verified image scores for the same weights. Results cover the evaluated subsets; the original failure and UNKNOWN scoring rules are retained.
Tip
🚀 Free Public API Now Available! Call Index-Translate-35B-A3B directly with zero GPU setup. Fully OpenAI-compatible at https://index-translate.bilibili.com/v1. Try it in seconds with python inference/llm/call_api.py "Hello, world!" --target zh! 👉 API Quick Start
Index-Translate is a family of multilingual, multimodal translation models built on Qwen3.5. The text models cover 150 languages and follow translation instructions such as terminology, formatting, and content-preservation requirements. The family extends this unified multilingual foundation to multimodal translation (visual images and audio speech), syllable-controlled translation, and full-document translation.
- Index-Translate (Text & Multimodal Vision): Covers 150 languages for text translation, with native support for multimodal image translation (native vision encoder, zero OCR dependency), structured content, hard/soft instruction following, and community expressions.
- Index-Echo (Multimodal Audio & Speech): End-to-end speech-to-subtitles (S2TT) and cross-lingual voice-cloned dubbing (S2ST), preserving the source speaker's voice characteristics while translating.
- Index-Homura (Controlled Dubbing): Dynamically adjusts translations to match a specified target syllable budget for syllable-level timing alignment.
- Index-NativeLong (Long Document): Supports long-context input for complete documents, maintaining term and character name consistency across paragraphs.
The radar includes 35B-A3B (preview), 9B, and 2B, with fixed per-axis min–max ranges across all 14 models. Its seven axes are WMT, FLORES, instruction following, low-resource translation, subtitles, MEME, and books/fiction. Instruction following averages instTrans and IFMTBench IFscore. The normalized scale is not an accuracy percentage. The gray dashed line combines the best non-Index score on each axis and does not represent one model. Raw category scores · Figure notes · Individual benchmark results.
Omni · News · ⚡ Free API · ✨ Multimodal Translation · Models · Quick start · Instruction Following · Examples · Evaluation · Benchmarks · Applications · TODO · Papers and citation
- 2026-11-05: introduced Index-Translate-Omni 2B / 9B / 35BA3B for text/image translation, Chinese/English ASR, and bidirectional Chinese–English speech translation. See the pinned downloads, release status, and benchmarks.
- 2026-10-10: launched official Multimodal Image Translation support! The full family (Index-Translate 2B / 9B / 35B-A3B) natively supports end-to-end image translation. The free public API, Web Demo, Python script (call_api.py), and GGUF
mmprojweights all support image input with zero external OCR dependencies. See the Tutorial Guide. - 2026-10-04: released free public API endpoints on index-translate.bilibili.com/v1 for Index-Translate-35B-A3B. Fully OpenAI-compatible. Try it with call_api.py.
- 2026-10-04: released four Index-Translate benchmarks, with datasets/metadata on Hugging Face and evaluation scripts and guides on GitHub.
- 2026-10-03: released official quantized builds across the family on Hugging Face and ModelScope — GGUF for llama.cpp local inference (including visual
mmprojweights), alongside FP8 (W8A8) and NVFP4 (W4A4, Blackwell-optimized) for vLLM serving. - 2026-09-30: released Index-Translate, with 2B / 9B / 35B-A3B (preview) text-model weights on Hugging Face and ModelScope, the technical report, and the online demo.
- Release official quantized builds (GGUF for llama.cpp local inference, FP8 and NVFP4 for vLLM serving) for the whole family.
- Release instTrans, MEME, SandGlass and NAtIveLong data/metadata and evaluation code; see Benchmarks.
- Release Index-Translate-Omni with text and image translation, Chinese and English ASR, and bidirectional Chinese–English speech-to-text translation.
- Add native video and long-document translation to Index-Translate-Omni.
- Index-Echo-S2ST: add support for more languages.
- Scaling out (100B+).
The links below include the Omni native packages and the original 2B, 9B, and 35B-A3B (preview) text-model checkpoints. Evaluation results are included in the comparison tables.
| Category | Model | Task and released package coverage | Hugging Face | ModelScope | Inference Guide |
|---|---|---|---|---|---|
| Text, Vision & Speech | Index-Translate-Omni | Text/image translation, Chinese/English ASR, and Chinese ↔ English speech translation | 2B · 9B · 35BA3B | 2B · 9B · 35BA3B | Release status and native guide |
| Text & Multimodal Vision | Index-Translate | Text & multimodal image translation across 150 languages, instruction following | 2B · 9B · 35B-A3B (preview) (⚡Free API) | 2B · 9B · 35B-A3B (preview) | Guide · Free API · Image Guide |
| Multimodal Audio & Speech | Index-Echo S2TT | Speech → subtitles; packaged script: zh→en/ja/es | 2B · 9B | 2B · 9B | Guide |
| Multimodal Audio & Speech | Index-Echo S2ST | Speech → speech; zh→en/es/ja, en→zh/es/ja (voice cloning) | 2B · 9B | 2B · 9B | Guide |
| Controlled Dubbing | Index-Homura | Translation with a target syllable count | 2B · 9B | 2B · 9B | Guide |
| Long Document | Index-NativeLong | Long documents; fixed templates: zh↔en, zh↔ja | 2B · 9B | 2B · 9B | Guide |
Naming: Index-NativeLong is published under the model IDs IndexTeam/Index-Nailong-2B and IndexTeam/Index-Nailong-9B. Use those IDs in commands. Language support for the speech and long-document packages is listed separately from the text models' 150-language coverage.
Quantized builds: the original Index-Translate, Index-Homura, Index-NativeLong, and Index-Echo families below are also published in GGUF (llama.cpp local inference; all bit-widths in one repository per model, vision mmproj included where applicable), FP8 (compressed-tensors W8A8, ready for vLLM serving), and FP4 (compressed-tensors NVFP4 W4A4, for Blackwell GPUs). For the Index-Echo speech models, the GGUF repositories contain the text LLM backbone only, while the FP8/FP4 repositories ship the complete pipeline with a quantized LLM. All repositories are mirrored on ModelScope.
| Model | GGUF (llama.cpp) | FP8 (vLLM) | FP4 (vLLM, Blackwell) |
|---|---|---|---|
| Index-Translate | 2B · 9B · 35B-A3B (preview) | 2B · 9B · 35B-A3B (preview) | 2B · 9B · 35B-A3B (preview) |
| Index-Homura | 2B · 9B | 2B · 9B | 2B · 9B |
| Index-NativeLong | 2B · 9B | 2B · 9B | 2B · 9B |
| Index-Echo S2TT | 2B · 9B (LLM backbone) | 2B · 9B (full pipeline) | 2B · 9B (full pipeline) |
| Index-Echo S2ST | 2B · 9B (LLM backbone) | 2B · 9B (full pipeline) | 2B · 9B (full pipeline) |
You can call our free online API directly without local GPUs. We recommend using our zero-dependency Python script inference/llm/call_api.py:
# 1. Quick command-line translation (zero dependencies, works out-of-the-box)
python inference/llm/call_api.py "你好，世界。今天天气不错，我们去公园散步吧。" --target en
# 2. Multimodal image translation (all sizes 2B / 9B / 35B natively supported, zero OCR dependency)
python inference/llm/call_api.py --image path/to/screenshot.png --target zh # Default 35B flagship
python inference/llm/call_api.py --image photo.jpg --target zh -m Index-Translate-2B # 2B edge lightweight
python inference/llm/call_api.py --image diagram.png --target en -m Index-Translate-9B # 9B workstation balanced
# 3. Local proxy bridge for browser extensions (e.g. Immersive Translate / 沉浸式翻译)
# Browser extensions require a local bridge proxy due to CORS/WAF headers and thinking tag suppression:
python inference/llm/call_api.py --serve
# Then configure Immersive Translate:
# - Service: Custom / OpenAI
# - API URL: http://127.0.0.1:8080/v1
# - Model: Index-Translate-35B-A3B
# - API Key: Any string (e.g. index)
# Option A: Standard Chat Completions API (OpenAI-compatible)
curl https://index-translate.bilibili.com/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Index-Translate-35B-A3B",
"messages": [{"role": "user", "content": "请将以下文本翻译为英语，直接输出翻译结果，不要进行任何解释。\n\n你好，世界。"}]
}'
# Option B: Multimodal Image Translation API (OpenAI image_url format, base64 supported)
curl https://index-translate.bilibili.com/v1/chat/completions \
-H "Content-Type: application/json" \
-H "User-Agent: Index-Translate-Client/1.0" \
-d '{
"model": "Index-Translate-35B-A3B",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Translate the text in the image into Chinese."},
{"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII="}}
]
}
]
}'
# Option C: Modern OpenAI Responses API (compatible with client.responses.create / POST /v1/responses)
curl https://index-translate.bilibili.com/v1/responses \
-H "Content-Type: application/json" \
-d '{
"model": "Index-Translate-35B-A3B",
"input": "请将以下文本翻译为英语，直接输出翻译结果，不要进行任何解释。\n\n你好，世界。"
}'Tip
✨ Multimodal Translation is Fully Live (Vision & Audio Speech)!
- 🖼️ Vision Multimodal (Image Translation): All sizes (2B / 9B / 35B) natively supported by our built-in vision encoder, zero OCR dependency, preserving layout and multimodal context. 👉 【Tutorial Guide】Multimodal Image Translation Complete Guide (API, Self-Hosting & Cases)
- 🎙️ Audio Multimodal (Speech Subtitles & Dubbing): Index-Echo supports end-to-end speech-to-subtitles (S2TT) and voice-cloned dubbing (S2ST). 👉 S2TT Subtitle Guide · S2ST Dubbing Guide
Tip
🌐 Bilingual Web Browsing? Immersive Translate is supported out-of-the-box (Windows / macOS / Linux)!
Browser extensions benefit from our zero-config local bridge proxy (Windows users can simply double-click run_proxy_windows.bat in the root folder).
👉 【Setup Guide】Immersive Translate Configuration Tutorial (Windows / macOS / Linux, with Screenshots & Troubleshooting)
Requires a CUDA GPU. You can serve models with either vLLM or SGLang:
git clone https://github.com/bilibili/Index-Translate.git
cd Index-Translate
pip install -r inference/llm/requirements.txt
# Option A: serve with vLLM (default port 8000)
./inference/llm/serve_vllm.sh translate-9b
# Option B: serve with SGLang (default port 30000, supports --mtp native speculative decoding, +20%-28% speedup)
./inference/llm/serve_sglang.sh translate-9b --mtp⚡ MTP Speculative Decoding Acceleration: Index-Translate model checkpoints contain built-in Multi-Token Prediction (MTP) heads. On the 9B model, MTP speculative decoding achieves ~85% draft token acceptance and 1.22x ~ 1.28x decode acceleration (100+ tok/s)! See MTP Speculative Decoding & Acceleration Guide.
From the same repository directory in another terminal:
python inference/llm/translate.py \
"你好，世界。今天天气不错，我们去公园散步吧。" \
--target en --model IndexTeam/Index-Translate-2BAn output recorded with the released 2B model is:
Hello, world. The weather is nice today. Let's go for a walk in the park.
See captured cases, text inference, and prompt examples. The 4,096-token setting above is a short-text example; the serving presets are listed in the table below.
For audio, use the dedicated S2TT subtitle guide or S2ST dubbing guide.
Index-Translate deeply integrates instruction-following capabilities (instTrans). It supports both hard constraints (format preservation, strict terminology glossary enforcement) and soft constraints (tone & style, domain disambiguation) out of the box via translate.py and syllable_translate.py:
# Hard constraint 1: Strict terminology glossary enforcement (-g / --glossary)
python inference/llm/translate.py \
"王平仲采用了更加昂贵的碳纤维材料。碳纤维的好处就是它抗裂缝。" \
--target en -g "碳纤维:carbon fiber, 抗裂缝:crack resistance"
# Hard constraint 2: Untranslatable text & span lock (-g supports identity mapping e.g. "term:term")
python inference/llm/translate.py \
'<span translate="no">我喜欢</span>花蓮的山海澎湖的潮汐' \
--target en -g "我喜欢:我喜欢" -H "保留源文中的 HTML 格式标记不变"
# Hard constraint 3: Format & structure preservation (-H / --hard)
python inference/llm/translate.py \
'{"user_id": 1024, "event": "purchase", "message": "您的订单已支付完成。"}' \
--target en -H "保留源文中的 JSON 格式标记不变"
# Soft constraint 1: Tone and style adjustment (-S / --soft) + genre (-d / --genre)
python inference/llm/translate.py \
"今天下午的会议临时取消了，改天我们再碰一下商量。" \
--target en -d "商务邮件" -S "调整为严谨、正式、礼貌的商务公文风格"
# Soft constraint 2: Domain context and word-sense disambiguation (-S / --soft)
python inference/llm/translate.py \
"The plant is operating at full capacity after the spring upgrade." \
--target zh -d "工业制造" -S "语境为工业制造与重工厂房领域，准确消歧专有名词（如 plant 译为工厂而非植物）"
# Syllable control synergy: Index-Homura strictly respects syllable budgets while embedding glossaries
python inference/llm/syllable_translate.py \
"我们今天去看电影吧" --syllables 7 --target en --glossary "电影:cinema"instTrans Specification & Constraint Details:
- Canonical Prompt: Automatically formatted by the client into the instTrans benchmark structure (
【源文】+ numbered1. 【硬性要求】.../2. 【注意】...+ suffix instructions).- Hard Constraints: Structural formatting (JSON/CSV/code/placeholders), terminology glossaries, social elements, and syllable ordering. Binary gated (
$g_{\mathrm{hard}}$ ); any single failure zeroes the instance score.- Soft Constraints: Tone/style adaptation, contextual sense disambiguation, cross-sentence consistency, and LaTeX preservation. Evaluated on a graded scale (
$q_{\mathrm{soft}}$ ).- See the text inference guide and instruction cases for full examples.
This is the shared reference for the repository clients and the released Echo packages. Translate cover