A multilingual embedding model is a candidate, not proof that German service questions retrieve the right English manual. Compare direct cross-language retrieval with query translation on paired, expert-reviewed cases. Keep document revisions, permission scope and evidence judgements fixed; measure retrieval before judging German answer fluency. Select the route that meets each language-pair quality floor within its latency budget.
Consider the illustrative question “Wie entlüfte ich die Kühlmittelpumpe nach dem Dichtungswechsel?” The relevant English section might say “bleeding the coolant circuit after seal replacement”, while another section describes normal pump operation. Both sound related. Only one answers the maintenance decision. This is a constructed example, not a customer observation or a validated operating instruction.
The engineering question is narrower than “Does this model support German?” It is whether language transformation loses the exact evidence needed for an authorised procedure. That requires a controlled comparison, not a translated demo and not a blended multilingual leaderboard score.
Define the evidence identity before the language experiment
Freeze a corpus snapshot and identify each source by document ID, revision, section and evidence span. Store source language separately from interface language. When a German and English manual genuinely describe the same revision and procedure, attach an evidence-equivalence ID after domain review. Do not infer equivalence from matching filenames or translated headings: local editions may contain different limits, warnings or component variants.
For each case, retain an intent ID, natural German question, expert-checked English equivalent, acceptable evidence IDs, applicability constraints, permission fixture and expected answerability. Two reviewers can resolve disagreements on high-impact cases. If one language edition lacks the relevant warning, mark that as a corpus-coverage difference, not an embedding failure. Keep these unpaired cases in a separate operational slice rather than deleting them from the evaluation.
Start with a bounded set that a domain expert can actually label. Cover procedure questions, symptoms, German compounds, abbreviations, mixed-language shop-floor phrasing and identifier-bearing questions. Include hard negatives: the right assembly but wrong procedure, an obsolete revision, a similar component and a passage where a negation changes the instruction. A small set is useful for debugging; it cannot establish general retrieval superiority.
Language-pair matrix: isolate what changed
Use paired intents for the diagnostic matrix below. DE → EN means a German query against English documents, not German answer generation. The German control requires equivalent German evidence; when that does not exist, label the control unavailable. Evaluate the actual mixed-language production corpus in an additional run so that cross-language distractors are not hidden by language-restricted test indexes.
| Slice | Query and corpus | Hold constant | What a failure suggests |
|---|---|---|---|
| EN → EN | English question; English manual | Same English source and intent | Extraction, relevance or baseline retrieval defect |
| DE → EN | German question; same English manual | Same acceptable evidence as EN → EN | Cross-language representation or query wording loss |
| DE → DE | German question; equivalent German manual | Reviewed procedure and revision equivalence | German terminology or corpus-localisation defect |
| Translated DE → EN | Machine-translated question; English manual | Translator version and protected identifiers | Translation helped or changed the task |
| Mixed-language | Original question; deployed language mix | Permissions, revisions and candidate budget | Duplicate crowding or language-routing bias |
A machine translation of every English benchmark question is not a representative German query set. It can preserve English syntax and vocabulary while missing the shorthand technicians use. Collect naturally authored German cases and use reviewer-checked pairing for diagnosis. Keep near-duplicate paraphrases in the same development or holdout group; otherwise the test rewards memorised wording.
The multilingual E5 technical report evaluates multilingual retrieval and cross-lingual bitext matching. Those are useful reasons to test the model. They are not measurements on German service questions over your English manuals. The reported MIRACL language table does not provide a German industrial-manual slice, and sentence matching is not equivalent to finding all clauses of a maintenance procedure.
Two competing routes and one protected identity path
Compare the routes below on identical authorised evidence. The diagram is a proposed evaluation architecture, not a measured deployment. Branch A searches using the original German question in a shared multilingual embedding space. Branch B translates the question into English and searches the English corpus. Both retain an independent original-identifier path; neither translation nor semantic similarity is allowed to redefine an equipment identity.
First isolate the translation effect by keeping the same multilingual encoder, English index, candidate budget and reranker configuration in both routes. Then, if justified, evaluate a separately versioned English-specialist retrieval stack. That second comparison measures an entire alternative stack, not translation alone. Report it accordingly.
For the shared-space baseline, the intfloat/multilingual-e5-large model card requires the literal prefixes query: and passage: even for non-English text. Encode “query: Wie entlüfte ich …” and “passage: Bleeding the coolant circuit …”; do not translate the prefixes. The card demonstrates attention-mask-aware mean pooling and normalised embeddings. Pin the tokenizer, model revision, pooling and normalisation together. The instruct variant is a different model contract, not a drop-in excuse to mix prompt formats.
The same card limits long inputs to 512 tokens and cautions about weaker low-resource-language performance. That general warning is not evidence that German is low-resource or that this German task will fail. Measure the actual tokenised input, including prefixes, injected titles and special tokens. A chunk that fits a character limit can still be truncated; German compounds need not have the same token length as their English equivalents. Log pre-truncation length and whether annotated evidence survives. Split or reject overlength inputs instead of silently discarding the final warning. The foundational chunking evaluation explains why chunk size and embedder must be tested together; do not copy a longer-context model's chunk size into E5.
Query translation needs a preservation contract
Treat translation as a versioned retrieval transformation. Preserve the original query; extract verified identifiers into protected fields; require an output schema containing the translated question and unchanged protected values. Compare those values deterministically before retrieval. A schema-valid translation can still invert a negation or replace a domain term, so review semantic failures separately. Do not assume an LLM follows “never change part numbers” reliably enough to omit the check.
Use an approved terminology glossary for compound terms and abbreviations, with ambiguity left visible. “Dichtung” can denote a seal or gasket depending on the assembly. If multiple interpretations change applicability, clarify rather than silently choosing the most common English word. Do not mask every number indiscriminately: pressures, dimensions and error codes need distinct handling, and removing all quantities can erase the question's meaning.
An external translator also changes the data boundary. Verify whether queries may leave the environment, what logs are retained and how tenant scope enters cache keys. Pin translator model, prompt and glossary revision in the run manifest. Record translation latency and timeout separately from embedding and search. On translation failure, a validated direct route may be a labelled degraded mode; otherwise request clarification. Never treat missing authorisation as a translation fallback condition.
Keep original-identifier lookup and lexical retrieval as controls, not as a new claim that fusion solves language. Microsoft's hybrid ranking documentation describes RRF as merging ranked lists and places semantic reranking after fusion. Its rank constant is distinct from the vector candidate count. Adding a translated-query branch changes the set of ranked lists and may change score magnitudes; an old absolute score threshold is not portable. Deduplicate repeated evidence and inspect both pre-fusion candidate coverage and final context coverage.
Measure evidence loss stage by stage
Define recall@10 for each answerable query as annotated relevant evidence units retrieved in the first ten unique units divided by all annotated relevant units. Average within each slice and publish its query count. If several spans are interchangeable alternatives, group them into one required evidence unit; if a procedure requires both a prerequisite and a warning, score them as separate requirements. Also report complete-evidence success: did every required unit reach the context? One useful paragraph is not enough for a multi-clause instruction.
Use MRR for the first useful result and nDCG only where graded judgements exist. Keep no-answer cases out of recall's zero denominator; report correct abstention and false-evidence selection separately. When running both routes, retain the evidence IDs after initial retrieval, fusion, reranking and context construction. If the right English section enters the candidate union but disappears after a reranker, changing the embedding model addresses the wrong stage.
Report DE → EN minus EN → EN on paired intents as a diagnostic language gap, not a universal multilingual penalty. Resample by independent intent or manual family when estimating uncertainty; translated variants of one question are not independent readers or independent tasks. Preserve slice-level counts and unresolved judgements. Do not let a large English slice conceal a German procedure regression.
Measure retrieval wall time and full answer latency separately, including translation, cache misses and errors. Declare corpus size, hardware or service tier, concurrency, candidate windows and warm/cold-cache policy. Track cost per attempted request, not just successful translations. This article supplies no measured latency or accuracy winner because no representative customer corpus or retrieval load test was available.
Decision table: choose by the failed constraint
| Observed evidence | Decision to test | Required release guard |
|---|---|---|
| Direct route meets every slice floor | Keep the shared multilingual encoder | Preserve German holdout and token-length checks |
| Translation repairs reviewed terminology failures | Enable translation for that query class | Identifier preservation, semantic review and latency budget |
| Both routes fail on the same source span | Repair extraction or evidence units first | Confirm the needed clause is indexed and survives truncation |
| Relevant evidence is lost after candidate retrieval | Revisit fusion, reranking or context selection | Compare stage-level evidence IDs at fixed budgets |
| Translation changes identity, negation or applicability | Block that route for affected cases | Clarification or validated direct fallback |
| German control lacks an equivalent source revision | Mark the comparison unpaired | Report corpus coverage separately; do not invent equivalence |
Set quality floors and tolerable paired regressions with the service owner before tuning. A stricter threshold is appropriate where a wrong procedure has higher impact, but retrieval evaluation is not safety approval to execute maintenance. High-impact instructions still need the applicable operational approval process. The RAG evaluation harness provides the broader release-control framework; this experiment contributes language-pair fixtures and a translation manifest to that gate.
Release checklist and trade-offs
Evidence check: source ID, revision, language and applicability are explicit; equivalent-language labels are reviewed; inaccessible or missing evidence cannot count as a successful match.
Encoding check: prefix contract, model revision, tokenizer, pooling and normalisation are pinned; token counts include added metadata; required spans survive truncation.
Translation check: original query is preserved; protected identifiers compare exactly; terminology, negations and quantities have reviewed test cases; timeout and cache-scope behaviour are explicit.
Ranking check: candidates and final context use the same permission scope; equivalent translations do not crowd out complementary evidence; reranker support for the evaluated language pair is tested, not assumed.
Release check: run paired holdout comparisons; inspect every high-impact regression; canary the chosen routing policy; retain rollback to the prior index, encoder and translation configuration. Monitor no-result, clarification and evidence-selection failures by language pair without retaining sensitive raw query text unnecessarily.
A shared encoder avoids a query translation dependency but can miss specialised wording. Translation can improve lexical alignment but adds another semantic failure point, latency and maintenance of glossary rules. Running both routes increases candidate work and duplicate pressure. None eliminates poor extraction, stale manuals or missing annotations. Choose the smallest architecture that passes the relevant evidence and operational gates.
What to build first
Bring one manual family, naturally authored German questions, the relevant English sections and your permission and latency constraints. Build the paired evidence matrix before adding a translator. I can help your team review the fixtures, trace where evidence is lost and define a release decision that separates retrieval quality from fluent German output. The deliverable should be a reproducible comparison with explicit failure boundaries—not a claim that multilingual support guarantees a correct procedure.
