Building Blocks in Production: My First Semantic Text Embedding Service

Today (8/17/2025) marked a milestone I’ve been working toward for months - deploying my first production service component: a microservice for semantic reasoning using open-source text embedding models. While it might seem small on the surface, it’s actually one of the core backbones of my entire service stack. And honestly? Building it felt like playing with sophisticated Lego blocks and proudly showing off the creation… kind of like my daughter running to call her mom and me to share her latest ‘achievement’!

Why Microservices?

Before you roll your eyes and think "this guy is building everything with limited resources and talking about microservices?! What the heck is he doing???" - let me explain. I'm well aware of the perils of over-engineering and premature optimization.

I'm not doing this for scale or to recreate fancy corporate architectures I used to build in the corporate world. I'm doing it for complexity management and focus. Trying to jam everything into a monolithic system, even early-stage, was becoming unwieldy. Breaking things into focused services lets me tackle each piece surgically while managing limited resources and attention more effectively.

Plus, I constantly reevaluate this approach. If it stops serving me, I'll change it.

The Engineering Fun

Before diving into the AI/ML part, let’s quickly look at what I did on the engineering side for the infrastructure (excluding the ML experiment pipeline). Honestly, this turned out to be way more fun than I expected!

• FastAPI design with robust security and system load management
• Comprehensive testing with unit and integration tests (pytest)
• Dockerized development and deployment via GitHub Actions
• AWS infrastructure including EC2 instance optimization, CloudWatch monitoring with alarms, S3 logging through Fluent Bit (side car), and Parameter Store integration

Huge shoutout to Amazon’s generous startup credits. They let me experiment freely with different configurations. 😄 Thanks again!

Alright, now that the engineering side is quickly covered, let’s dive into the main discussion.

Why Text Embeddings Still Matter in the LLM Era

Text embedding is a classic concept - projecting text into semantic space so we can understand what words mean "semantically" and how they relate to each other. This approach helps overcome the limitations of keyword-based methods, especially in retrieval and ranking.

With LLMs seemingly offering enormous semantic discrimination power, you might wonder why bother with embeddings? Here's the reality: LLM inference is expensive. While inference is definitely lighter than training (no gradient calculation or backpropagation), it's still computationally demanding compared to typical jobs. For small companies trying to prove their ideas with limited budgets, it's often not practical, unless their use cases involve using LLM as a single choice. Also, deploying at scale is a huge financial investment and risky decision when financial gain isn't clearly promising.

The future might bring ultra-cheap LLM inference (come on, Trainium! You can do better!) or new model paradigms that don't rely on resource-intensive O(n²) attention mechanisms - maybe O(n) or even O(1)! Until then, embedding models remain everywhere - even powering tools like Cursor, Claude Code, and GitHub Copilot (btw, this one sucks and stay away from it. Disabling it in VS Code was one of the best things I’ve done recently) for finding relevant code snippets.

By the way, modern open-source embedding models heavily use BERT-style transformer architectures as well. But, they are still a lot less than typical LLM models (though, Alibaba's recent Qwen3 embedding model families supporting 32,000 tokens are relatively big with 0.6B ~ 8B parameters and 1024 ~ 4096 dims)

My Evaluation Process: Not Academic, Just Practical

I want to be upfront - this isn't rigorous academic research evaluation. It's my way of documenting what I do for this "fun" work and making informed decisions for my specific use case.

Key Evaluation Criteria

Context Length Reality Check: I evaluated models at both 512 and 8192 token context lengths. Since my use case involves embedding entire documents, longer context length reduces heavy chunking (which can dilute embeddings, though not always).

CPU vs GPU Trade-offs: GPU instances provide faster, more efficient inference, but I'm using CPU instances for now since this is inference-only and I expect moderate load traffic initially. My custom stack using sentence-transformers + PyTorch can switch between CPU and GPU instantly via environment variables.

Domain-Specific Testing: Having good test data for your domain is crucial as I'm using general models without fine-tuning (I'm honestly so negative about fine-tuning... it's hard to evaluate whether it actually helped, and creating high-quality fine-tuning data is not so easy). Since each model trains on different data, their generalization power varies across domains. Also, larger models aren't always better for your specific domain, as my results show. Numbers will guide you and debunk the myths.

The Models I Tested

Performance Results Summary

Here are all the models I evaluated with their key metrics:

Human-in-the-Loop Evaluation

Beyond aggregated metrics like Top-K accuracy, precision, and recall, I built an interactive tool to see how each model behaves qualitatively (me as a tester in Human-in-the-Loop process) and its resulting ranking in retrieval. This human evaluation revealed interesting patterns:

• Nomic AI models showed a tendency to focus heavily on details (helpful in some cases, problematic in others)
• Alibaba GTE models demonstrated more balanced behavior between high-level categories and details

While the Alibaba model didn't always show better aggregated metrics (across all different test data sets - not mentioned here), I preferred its responses in my interactive testing.

Load Testing Setup

I used k6 for load testing on an AMD Ryzen machine (8 cores, 16 virtual cores, 32GB memory) with 2 workers and 3 PyTorch threads, intentionally without CPU/memory caps. I kept the worker count low even though I had more computing power available, as this better simulates my planned deployment environment and helped me choose the right AWS instance specifications.

The Winner: Alibaba-NLP/gte-base-en-v1.5

After evaluating all models across accuracy, performance, and qualitative behavior, I selected Alibaba-NLP/gte-base-en-v1.5 for production.

This model struck the perfect balance for my needs:

• Strong accuracy (91.05% top-1 accuracy)
• Reasonable response times (0.45s average, 1.26s P95 for single user)
• Excellent 8192 token context length for document embedding
• Manageable resource requirements (1.9GB memory)

This API's /info endpoint returns this:

{
  "model_name": "Alibaba-NLP/gte-base-en-v1.5",
  "embedding_dimension": 768,
  "max_context_length": 8192,
  "device": "cpu",
  "has_normalize_layer": false
}

What's Next?

Score Scaling Optimization: Need to scale similarity scores properly. As I mentioned in previous write-ups, all embedding similarity scores cluster around 60%~90%, so probably min-max scaling would work better than the current approach. But let's see what the data shows.

BM25 Integration: I need to experiment with integrating BM25 for accuracy improvement. Probably BM25 first, then reranking with embedding models might not work well. So weighted fusion between BM25 and embedding scores might be the better approach. But what would be the optimal weight? Need to experiment. Since I built the entire dockerized E2E pipeline for experiment/evaluation, once I create the core logic for this, evaluation itself should be fast via full automation.

Vector Storage Strategy: I'll need to maintain multiple embeddings for each search target (to avoid embedding dilution), so need to experiment with optimization (as the number of vectors to compare with increases by n times) to see whether I can use pgvector extension with PostgreSQL or should rely on specialized vector databases, with approximate indexing approach for both cases. I guess pgvector extension should be good enough... but will see why people just use vector databases all the time. Yes, another topic to bust myth. By the way, don't get me wrong. I know vector database does a lot more.

Integration Time: Once all these experiments are done, integrating everything into the core search/ranking service is the natural next step. A lot more fun infrastructure for this already exists in my dev environment now. 😄

The journey from "just an embedding service" to a full semantic search stack continues. Each piece teaches me something new about the trade-offs between performance, cost, and complexity.

Straight Talk

• This microservices book is really good (a bit old, though). If you want to actually understand what people are talking about when they say 'microservices' and figure out whether they really know it or are just throwing buzzwords around like we see all the time at the office, in media, or wherever, it’s worth taking a look. But here’s the thing: there are also plenty of people who do understand the concept… and still don’t do anything. They just talk. Don’t be one of them. Be a doer, not a talker or naysayer. If something doesn’t work, come up with ideas to make it work instead of whining that it doesn’t. Both are useless.
• Kudos to all the researchers and engineers who keep sharing powerful open-source models. This is what real science and engineering are about. The AI/ML boom happened because people shared everything openly… until OpenAI started doing their ‘ClosedAI’ thing, even though they piggybacked on Google’s Attention and Transformer papers. By the way, to be clear, I’m not blindly praising open-source efforts, nor am I blaming commercial providers. Commercial AI services (ChatGPT, Claude, Gemini, etc.) have a legitimate role: delivering high-quality services, operational excellence, and competitive pricing. I also use commercial services a lot, for example, in my personal diary/social feed app, Scribble. It is much cheaper and reliable for this case. But from a research, ideation, or startup perspective, open-source models lower the barrier to entry and enable experimentation. That’s why I appreciate them so much.
• Big fuck to the so-called sovereign AI folks in Korea. It’s pathetic hearing them whine about not having enough GPUs, claiming they couldn’t do research or build models ‘optimized’ for Korean and Koreans (whatever that even means), and that this somehow justifies sovereign AI funding. Total BS. They didn’t even buy the damn GPUs, and even if they couldn’t, there’s AWS, Azure, Oracle, Google, or whatever, plenty of ways to get GPU power. Look at what Chinese companies/academies are doing with far fewer GPUs and less powerful hardware. Real research and real models get built. AI isn’t just about LLMs or fancy hardware anyway; universities manage to do real research without giant clusters. In reality, they just decided not to invest or bother, and now they’re trying to spin excuses into a patriotic-sounding tax grab. Even if they somehow built the model, who’s paying to host and maintain it? That’s right, the taxpayers. Fuck that nonsense. It’s only a matter of time before some coupon called ‘한국사랑 AI 머니’ shows up.

Full evaluation report (.md) and its raw data (.json) are available here: