Sarvam 105B is a strong fit when your product must answer in Indian languages, romanized text, and code-mixed user turns—not English-only demos. This tutorial shows how to evaluate it for RAG before you standardize model IDs or compare against Sarvam 30B and other frontier options such as DeepSeek R1 via comparison pages.
Sarvam documents OpenAI-compatible chat completions, a 128K context window on 105B, streaming, and Apache 2.0 licensing. Always confirm the live model ID and endpoint in Sarvam’s model docs.
1. Define the RAG job in product language
Write a one-paragraph success definition, for example:
“Support agents in Hindi, Tamil, and Hinglish can ask questions about our policy PDFs and get cited answers with ≤X% critical errors.”
Capture:
- Languages and scripts you must support
- Whether voice transcripts / romanization appear
- Citation requirements
- Latency and cost budgets
- Whether coding or tool use is in scope (105B is also positioned for reasoning/coding—not only chat)
2. Build a tiny but nasty evaluation corpus
Do not evaluate with English Wikipedia only. Create 30–50 questions across:
| Slice | Why it matters |
|---|---|
| Native script | Hindi/Tamil/etc. policy questions |
| Romanized | Mobile-typing patterns |
| Code-mixed | Real user mixes |
| Retrieval traps | Near-duplicate policies, outdated clauses |
| Refusal / privacy | Requests for other customers’ data |
Store gold answers or acceptable citation IDs. Keep PII out of the shared eval set.
3. Fix retrieval before you blame the LLM
For each question, log:
- Top-k chunks and scores
- Whether the gold passage was retrieved
- Answer faithfulness given only those chunks
If retrieval misses, fix chunking, language-aware embedding, or hybrid search before swapping sarvam-105b for another model.
4. Call Sarvam with an OpenAI-compatible chat shape
Use the official chat-completions path and the exact model ID from current docs (sarvam-105b vs conversational variants such as sarvam-105b-conversations when you need dialogue-oriented behavior). Sketch:
from openai import OpenAI
client = OpenAI(
api_key="YOUR_SARVAM_KEY",
base_url="https://api.sarvam.ai/v1", # confirm in current Sarvam docs
)
def answer(question: str, chunks: list[str]) -> str:
context = "\n\n".join(chunks)
resp = client.chat.completions.create(
model="sarvam-105b",
messages=[
{
"role": "system",
"content": (
"Answer using only the provided context. "
"Cite chunk numbers. If context is insufficient, say so."
),
},
{
"role": "user",
"content": f"Context:\n{context}\n\nQuestion: {question}",
},
],
temperature=0.2,
)
return resp.choices[0].message.content
Verify base_url, auth headers, and model IDs against Sarvam’s current documentation before production.
5. Score with humans + simple automatic checks
For each item, score:
- Correctness in the user’s language
- Citation validity (chunk exists and supports the claim)
- Language appropriateness (did it inappropriately switch to English?)
- Safety (no cross-tenant leakage)
Track vendor-reported benchmarks separately from this application eval.
6. Decide 105B vs 30B routing
Sarvam positions 30B as lower-latency / lower-cost for many real-time chat turns (64K context documented) and 105B for harder reasoning and long-form quality (128K). A practical pattern:
- Route simple FAQ turns to 30B after it clears your language bar.
- Escalate long-context or high-risk answers to 105B.
- Keep a manual review queue for low-confidence citations.
7. Ship gate
- Eval set covers native, romanized, and code-mixed traffic
- Retrieval hit-rate on gold passages measured
- Faithfulness scored with fixed chunks
- 30B vs 105B latency/quality trade-off recorded
- Logging redacts raw user PII where required