Toward Production: Better Every Day
After my previous post about evaluating embedding models for our use case, I felt like I was missing some key evaluations to really understand how these models tick..
DK Joo
Domain Separation / Overlap
Since I don't use fine-tuned or instructor-enabled models, understanding how well an embedding model separates across our domains is super important. It's all about that generalized power. So I dug deeper into domain overlapping tests, and honestly? It also reveals a lot about how each model was trained.
The results were surprisingly good, to be honest. I could also see exactly where models get confused when there's limited info available. But here's the thing: what do you do with those struggling cases? "Damn, this model can't help?" Don't panic!
Some domains are just inherently confusing, even for us humans. Plus, some queries naturally fit into multiple domains anyway. So it's not always a bad thing. The key is understanding the detailed results first.
What's really interesting is that those "bad" test results actually reveal valuable business intelligence. Our day-to-day text is confusing too, and we humans struggle with it as well. So the business can actually act on these insights to reach customers better based on these numbers.
k Avg Overlap Min Max
---------------------------------------------
Overall 0.019 (1.9%) 0.000 0.267
1 0.017 (1.7%) 0.000 0.267
3 0.017 (1.7%) 0.000 0.244
5 0.018 (1.8%) 0.000 0.233
10 0.024 (2.4%) 0.000 0.217
Noise Robustness
Next up: robustness to noise (or as the academics call it, "perturbation"). I make typos ALL the time. Partly because English is my second language (though I make tons of typos in my native language too 😄), partly because my typing speed can't keep up with how fast I think or want to speak. Whenever my daughter (only 3 years old) melts down, I feel sorry for her, awkwardly concerned that I'm passing down my bad temper... Sorry, Irene!
I also do this a lot when I'm screaming at my AI agent to STOP as it's screwing up my entire codebase with stupid changes. Today it changed the variable name from "emb" to "embs," claiming it would fix an issue because now it "generates a list type as return." Sigh...😦
So how do these models handle this kind of noise? I added noise at multiple levels (light, moderate, heavy) to queries and measured the degradation. Turns out the smart folks who trained these models had this in mind. Degradation was only 0.6-3.1%. Not bad at all.
Here's something interesting: since I'm combining keyword search to capture lexical signals along with semantic signals, using NLTK tokenization directly was less helpful for noisy query text. So to better support this (not systematically handling the noise though, just supporting it), I turned on stemming and boom! The hit ratio jumped from 2% to 10%. Very rewarding!
MRR S1 S3 S5 S10 P1 P3 P5 P10
---------------------------------------------------------------------------------------
Clean 0.869 0.819 0.904 0.923 0.967 0.819 0.817 0.801 0.734
---------------------------------------------------------------------------------------
Light 0.864 0.816 0.895 0.919 0.951 0.816 0.804 0.789 0.720
Drop% 0.6 0.4 1.0 0.4 1.6 0.4 1.6 1.5 1.9
---------------------------------------------------------------------------------------
Moderate 0.856 0.807 0.891 0.904 0.937 0.807 0.797 0.776 0.711
Drop% 1.5 1.5 1.4 2.1 3.1 1.5 2.4 3.2 3.2
---------------------------------------------------------------------------------------
Heavy 0.842 0.791 0.870 0.895 0.933 0.791 0.777 0.765 0.704
Drop% 3.1 3.4 3.7 3.0 3.4 3.4 4.9 4.6 4.1
---------------------------------------------------------------------------------------
The Main Event: Hybrid Search Magic
Now for the main game. Combining semantic signals (captured by embedding models and cosine similarity, basically the dot product of two normalized unit vectors) with lexical signals (captured by BM25, which is essentially a tf-idf thing).
I normalized the embedding cosine similarity score from [-1, 1] to [0, 1], which is pretty straightforward. If you understand basic trigonometry covering 360 degrees: 1 for the same direction (0 degrees), -1 for the opposite direction (180 degrees).
For BM25 scores that range from [0, +inf], I clipped at 7 (based on the 90th percentile from my tests) and normalized to [0,1]. I'm not going to dive into the other BM25 hyperparams here... Now both metrics share the same score scale and are easy to fuse with weighting:
score = alpha × semantic_score + (1-alpha) × bm25_score
I ran searches with varying alpha values. 0 means pure keyword search, 1 means pure semantic search. I focused on precision@k (for recommender or matching systems, precision@k can be more important than MRR (Mean Reciprocal Rank) or Success@k).
Surprisingly, with most models and my test data, semantic-only was enough. Little to no contribution from BM25. This actually makes sense since our domains are inherently subtle, confusing, and ambiguous. Semantic signals rule over keywords in this case.
But I don't want to lose that clear keyword match signal, so I set the weights at 0.85 and 0.15 for semantic and BM25 respectively to be conservative, even though my test results said 0.95 and 0.05 showed the best precision@5 numbers. I trust my gut on these seemingly suboptimal weights. BM25 should work as a defense mechanism. Chasing small accuracy improvements on somewhat limited test datasets is sometimes meaningless, even dangerous. Getting live population-synced test data is super hard. Almost impossible. Plus, the nature of the data always shifts over time...
Semantic BM25 MRR S@1 S@5 S@10 P@1 P@3 P@10
-------------------------------------------------------------------
0.50 0.50 0.8397 0.786 0.905 0.947 0.786 0.775 0.680
0.60 0.40 0.8413 0.784 0.907 0.953 0.784 0.785 0.691
0.70 0.30 0.8507 0.796 0.916 0.956 0.796 0.798 0.702
0.75 0.25 0.8576 0.805 0.919 0.960 0.805 0.804 0.710
0.80 0.20 0.8665 0.819 0.919 0.954 0.819 0.811 0.719
0.85 0.15 0.8691 0.819 0.923 0.967 0.819 0.817 0.734
0.90 0.10 0.8881 0.844 0.940 0.965 0.844 0.836 0.765
0.95 0.05 0.8950 0.856 0.946 0.967 0.856 0.846 0.801
1.00 0.00 0.8781 0.839 0.916 0.954 0.839 0.833 0.803
One thing to note: I also tried RRF (Reciprocal Rank Fusion) strategy using the approach below below, evaluating varying k values. But since the semantic signal is too strong and the lexical keyword signal is too weak, it didn't prove to work better than the weighted fusion approach. Even though it has a good advantage (we don't need to normalize scores since it only looks at ranking), it didn't work well for our case.
ranked_score = 1/(k + semantic_rank) + 1/(k + bm25_rank)
O(M x N x K)
One new thing to note: in my previous article, I mentioned that search target items can have multiple embedding vectors to avoid embedding dilution (many industry systems use this approach like one vector for title, another for reviews, etc.). So searches run over N items × K vectors = O(N×K).
Now, even queries can have multiple embedding vectors from chunking to overcome the embedding model's context length limitations. We could mean or max pool to get a single aggregated vector from multiple chunk vectors, but we'd lose or weaken signals in that process. Better approach: max aggregation for query chunk searches. This becomes O(M × N × K), which could seriously impact system load and response time. But since M will be 1 for most queries, it should be fine. Fortunately, negligible impact.
Good Experiments
Good stuff! Maybe even great. I enabled the entire evaluation pipeline for this business logic with the underlying embedding component (its API service is already in prod). The end-to-end pipeline uses Docker and I can run testing across models easily in a fully automated manner. Plus, since I enabled parallel execution in many areas of my evaluation logic, even though I increased my test data size 3×, execution time for a single model eval dropped by more than 5×.
The Real Challenge: Taking It to Production
Finally, probably the most important question: How do I carry this business logic (hybrid semantic and keyword search) to the production search system (whose in-flight version is in my dev environment)?
We can't use the in-memory approach I used for testing and evaluation. If I did this at the app level with minimal database interaction, the system would be SUPER slow! So here's my idea for taking this to prod using Postgres:
For the semantic search part:
- pgvector extension for vector similarity search
- HNSW or IVFFlat indexing for fast approximate nearest neighbor search
- Multiple embeddings per item supported with MAX aggregation (N×K search)
- Multiple query embeddings supported via MAX aggregation (M×N search)
- Case normalization: lowercase all text before embedding for consistency
For lexical keyword search:
- Built-in PostgreSQL FTS instead of reimplementing BM25
- GIN (Generalized Inverted Index) for fast text search
- tsvector and tsquery for efficient tokenization and search
- Built-in stemming support via language configurations
- Case-insensitive by default
Combining it all together:
- Single SQL query combining multiple scoring approaches
- Database-level fusion reduces network overhead and latency
- Scalable scoring architecture using CTEs for modular components (CTEs rock! No more temp table nonsense!)
- Transparent handling of single or multiple query embeddings
- Extensible design for easy addition of new scoring signals
Here's what a sample SQL query would look like:
-- Extensible hybrid fusion with configurable weights
WITH query_embeddings AS (
SELECT unnest($1::vector[]) AS query_emb -- M query vectors (usually 1, occasionally 2-3)
),
-- Scorer 1: Semantic similarity
semantic AS (
SELECT ie.item_id,
MAX(1 - (ie.embedding <=> qe.query_emb)) AS score
FROM item_embeddings ie
CROSS JOIN query_embeddings qe
GROUP BY ie.item_id
LIMIT 100
),
-- Scorer 2: Lexical (BM25-like)
lexical AS (
SELECT item_id, ts_rank_cd(doc_vector, plainto_tsquery($2)) AS score
FROM item_documents
WHERE doc_vector @@ plainto_tsquery($2)
LIMIT 100
),
-- Scorer 3: Review rating (example additional scorer)
review_rating AS (
SELECT item_id,
(avg_rating / 5.0) AS score -- Normalized 0-5 rating to 0-1 score
FROM item_reviews
WHERE review_count >= 3 -- Only include items with sufficient reviews
),
-- Scorer 4: Geographic relevance (example additional scorer)
geographic AS (
SELECT item_id,
CASE
WHEN location @> point($3, $4) THEN 1.0 -- Exact location match
WHEN earth_distance(location, point($3, $4)) < 50000 THEN 0.5 -- Within 50km
ELSE 0.0
END AS score
FROM item_locations
)
-- Fusion with configurable weights
SELECT
COALESCE(s.item_id, l.item_id, r.item_id, g.item_id) AS item_id,
(
0.70 * COALESCE(s.score, 0) + -- Semantic weight
0.15 * LEAST(COALESCE(l.score, 0) / 4.0, 1.0) + -- Lexical weight (normalized)
0.10 * COALESCE(r.score, 0) + -- Review rating weight
0.05 * COALESCE(g.score, 0) -- Geographic weight
) AS final_score
FROM semantic s
FULL OUTER JOIN lexical l USING(item_id)
LEFT JOIN review_rating r USING(item_id)
LEFT JOIN geographic g USING(item_id)
ORDER BY final_score DESC
LIMIT 20;
Quick note on that divisor of 4.0 for ts_rank_cd normalization: it's a conservative estimate based on typical PostgreSQL FTS score distributions. My BM25 testing showed the 90th percentile at 7.0, but ts_rank_cd typically produces lower scores (2-4 range for good matches). This should definitely be tuned based on actual ts_rank_cd distribution analysis with your production data.
And that's the journey so far! From a quick follow up about domain overlaps to building a hybrid search that can handle my terrible typing. Sometimes the best solutions come from just admitting your weaknesses and building around them.