[{"data":1,"prerenderedAt":171},["ShallowReactive",2],{"blog:\u002Fblog\u002F2026-07-06-the-reranker-that-looked-like-bert":3},{"id":4,"title":5,"body":6,"date":161,"description":162,"extension":163,"meta":164,"navigation":165,"path":166,"seo":167,"stem":168,"tag":169,"__hash__":170},"blog\u002Fblog\u002F2026-07-06-the-reranker-that-looked-like-bert.md","The reranker that looked like BERT",{"type":7,"value":8,"toc":154},"minimark",[9,13,17,37,42,53,84,91,94,101,105,116,119,122,136,139],[10,11,5],"h1",{"id":12},"the-reranker-that-looked-like-bert",[14,15,16],"p",{},"I ported a reranker onto the Apple Neural Engine, and the hardest bug in the project turned out to be one line in a config file rather than anything about the hardware.",[14,18,19,20,27,28,32,33,36],{},"The model is ",[21,22,26],"a",{"href":23,"rel":24},"https:\u002F\u002Fhuggingface.co\u002FBAAI\u002Fbge-reranker-base",[25],"nofollow","BAAI\u002Fbge-reranker-base",", a 278M-parameter cross-encoder with BERT-shaped guts: 12 layers, 768 hidden, GELU. So I reached for the BERT playbook: WordPiece-style assumptions, ",[29,30,31],"code",{},"[CLS]","\u002F",[29,34,35],{},"[SEP]",", and a single separator between the query and the document. It ran fine, but the relevance scores were quietly wrong.",[38,39,41],"h2",{"id":40},"one-line-changes-everything-downstream","One line changes everything downstream",[14,43,44,45,48,49,52],{},"The tell was in ",[29,46,47],{},"config.json",": ",[29,50,51],{},"model_type: xlm-roberta",". If you pattern-match on the architecture you will never see it, because that one line rewires everything the model expects at the input:",[54,55,56,60,78],"ul",{},[57,58,59],"li",{},"the tokenizer is SentencePiece-Unigram, not WordPiece",[57,61,62,63,66,67,66,70,73,74,66,76],{},"the special tokens are ",[29,64,65],{},"\u003Cs>"," \u002F ",[29,68,69],{},"\u003C\u002Fs>",[29,71,72],{},"\u003Cpad>"," (ids 0\u002F2\u002F1), not ",[29,75,31],{},[29,77,35],{},[57,79,80,81],{},"the paired input uses a doubled separator: ",[29,82,83],{},"\u003Cs> query \u003C\u002Fs>\u003C\u002Fs> document \u003C\u002Fs>",[14,85,86,87,90],{},"Missing the doubled ",[29,88,89],{},"\u003C\u002Fs>\u003C\u002Fs>"," doesn't break anything. There is no crash and no warning; the model just scores against malformed input.",[14,92,93],{},"The part I didn't expect was how invisible this is to normal testing. A \"does it run\" check passes, and the scores even look plausible in isolation. The test that catches it is numeric equivalence against the reference implementation. Two models can share a shape and still disagree on tokenization, separators, and position-id math, and those disagreements never show up as errors, only as wrong numbers.",[14,95,96,97,100],{},"The lesson travels well past Apple Silicon: read the config, not the architecture diagram. ",[29,98,99],{},"model_type"," is the real contract.",[38,102,104],{"id":103},"once-the-input-was-right-the-port-held","Once the input was right, the port held",[14,106,107,108,111,112,115],{},"Getting the model resident on the Neural Engine took more than a conversion flag. A naive Core ML conversion of a BERT-family encoder lands on CPU\u002FGPU. To keep it on the ANE you rewrite the graph the way Apple's ",[29,109,110],{},"ane_transformers"," reference does: every Linear becomes a 1×1 Conv2d, tensors carry a ",[29,113,114],{},"(B, C, 1, S)"," layout, and LayerNorm is swapped for an ANE-friendly version.",[14,117,118],{},"Then you verify it stayed there. My build gate asserts an exact CPU-dispatch fingerprint (the 31 ops that physically can't run on the ANE, like the gather over a 250k-token vocab, casts, and mask math) and zero GPU fallback. If anything drifts off that fingerprint, the build fails.",[14,120,121],{},"The payoff, on my machine:",[54,123,124,127,130,133],{},[57,125,126],{},"2.62ms per pair p95 on the ANE vs 6.92ms on CPU+GPU at batch=20\u002Fseq=128, about 2.6x faster",[57,128,129],{},"at seq=256 it's 6.54ms vs 12.16ms, about 1.9x",[57,131,132],{},"FP16 Core ML, a precision conversion, not quantization",[57,134,135],{},"an MTEB SciDocs regression check showing the FP16 conversion cost +0.0005 nDCG@10 vs FP32, which is nothing",[14,137,138],{},"Two caveats, because numbers without them are marketing: this is a single-machine benchmark (50 warmup + 100 timed iterations per cell), not independently reproduced. And this is a port of an existing model, not a new reranker; the work was the graph rewrite, the residency gate, and getting the input format right.",[14,140,141,142,147,148,153],{},"The conversion code is at ",[21,143,146],{"href":144,"rel":145},"https:\u002F\u002Fgithub.com\u002Ftcashel\u002Fjuice-bge-reranker-coreml",[25],"github.com\u002Ftcashel\u002Fjuice-bge-reranker-coreml",", and the converted model is on Hugging Face as ",[21,149,152],{"href":150,"rel":151},"https:\u002F\u002Fhuggingface.co\u002Ftcashel\u002Fbge-reranker-base-coreml",[25],"tcashel\u002Fbge-reranker-base-coreml"," if you're doing on-device inference on Apple Silicon.",{"title":155,"searchDepth":156,"depth":156,"links":157},"",3,[158,160],{"id":40,"depth":159,"text":41},2,{"id":103,"depth":159,"text":104},"2026-07-06","BERT-shaped guts, an XLM-RoBERTa config, and relevance scores that were silently wrong.","md",{},true,"\u002Fblog\u002F2026-07-06-the-reranker-that-looked-like-bert",{"title":5,"description":162},"blog\u002F2026-07-06-the-reranker-that-looked-like-bert","ML","1PRQQtNDA6J90X0mYILvt7B-9SbbQe7awdfi2LwekvA",1786110827757]