The fundamental problem we tried to solve with this system is to summarize the news for each symbol on our customer-facing app. The "real" fundamental problem is that we must deliver an "AI"-based system, just because.
So I built this with the mindset of implementing a simple AI system without specific expectations. However, it turned into a quite decent system, in my humble, subjective opinion of course. Along the way I learned and implemented concepts such as token budgeting and LLM-as-Judge.
I started by picking the use case: one of the most common yet useful applications of LLMs, summarization. This project started circa early 2025, when LLMs were already pretty good at coding, so the LLMs wrote most of the system. I was only responsible for the architecture (JK, it's 100% LLM, my only job was to prompt it). Regardless, let's see what I've built.
Architecture
For context, I worked at Valbury, a trading platform for foreign exchange, index, and commodity products. As we grew, we also started serving US stocks. On the existing mobile app, before I built this, each Product Display Page (PDP) already had a list of news related to the product. Well, good news (pun intended), we already had the data sources.
From these data sources, creating the summary itself is straightforward: everyone knows LLMs can do it better than we can. But the problem is that nobody wants to be the one who acknowledges it. Who can say that a summary generated by an LLM is good enough? Furthermore, there are dozens of futures exchange symbols and thousands of US stock symbols. Should we check them one by one? I'm sure the Market Analyst team would turn into bears even when the market is bulls if I handed them the job of evaluating every single summary.
That's when I learned about the LLM-as-Judge concept: evaluating the results of LLM-generated summaries, also using an LLM.
Nice, we've got the full architecture now. Let's code prompt.
We can see that under the hood, it's an ordinary FastAPI service on top of Postgres, with an LLM API doing most of the work. I'm just a Prompt Engineer indeed.
Data Fetch and Token Budgeting
Since we already have an existing news dataset, I didn't need to crawl data from outside sources. That said, I did prepare an endpoint to crawl news from a given link using Firecrawl. We then cleanse the latest n news items and append them into one object for summarization.
But what if the news turns out to be too long to handle? That's where I implemented token budgeting using Tiktoken. We count the total tokens for the prompt plus the appended news, and if it exceeds the TOKEN_COUNT_LIMIT, we drop the earliest news item. We didn't want to mess up the context by truncating the text mid-article.
current_limit = len(rows)
while current_limit > 0:
sliced_rows = rows[:current_limit]
formatted_news = [format_news(row) for row in sliced_rows]
joined_news = " --- ".join(formatted_news)
total_text = summarizer_prompt + "\n" + joined_news
token_count = count_tokens(total_text)
if token_count <= self.TOKEN_COUNT_LIMIT:
return {"news": joined_news, "token_count": token_count}
logger.warning(f"Token count {token_count} exceeds limit, reducing news limit and retrying...")
current_limit -= 1
raise ValueError(f"Unable to fit news and prompt within {self.TOKEN_COUNT_LIMIT} tokens.")The data was ready to be summarized.
Summarize
The summarize module is straightforward: run the prompt against the appended news with a strict output format, and fingers crossed, hope the LLM produces an accurate summary. But in the back, of course, I didn't trust it. So we built format validation on top. Nothing could compromise the format stored in the DB, because the Frontend Engineer would consume it to determine the UI on the customer-facing feature.
def format_summary_to_json(raw_text: str) -> dict:
"""Clean and parse the raw LLM output into proper JSON format."""
cleaned_text = raw_text.strip()
if cleaned_text.startswith('```json'):
cleaned_text = cleaned_text[7:]
if cleaned_text.endswith('```'):
cleaned_text = cleaned_text[:-3]
data = json.loads(cleaned_text)
VALID_SENTIMENTS = {"bullish", "bearish", "neutral"}
sentiment = data.get("sentiment", "").lower()
if sentiment not in VALID_SENTIMENTS:
raise ValueError(f"Invalid sentiment value: '{sentiment}'")
return {
"summary": data.get("summary", ""),
"bullet_points": data.get("bullet_points", []),
"sentiment": sentiment
}Fortunately, most LLM providers already offer a structured output parameter on their API, so this cleansing task is an easy one nowadays.
Anyway, we got the summary.
LLM-as-Judge: G-Eval
There are many state-of-the-art (SOTA) LLM-as-Judge approaches out there, but for this project, I implemented G-Eval as the framework.
G-Eval proposes 4 independent metrics:
- Relevance
- Coherence
- Consistency
- Fluency
A separate LLM call scores each one 1-5. According to the paper, this approach outperforms both conventional reference-based metrics, such as BLEU and ROUGE, and prior LLM-based reference-free evaluators, achieving a much higher correlation with human judgment. I tailored the prompts to accommodate Bahasa Indonesia.
EVALUATION_METRICS = {
"Relevance": (
"Relevance(1-5) - pemilihan konten penting dari sumber. Ringkasan hanya boleh mencakup informasi penting dari dokumen sumber.",
"""
1. Baca ringkasan dan dokumen sumber dengan cermat.
2. Bandingkan ringkasan dengan dokumen sumber dan identifikasi poin-poin utama.
3. Evaluasi seberapa baik ringkasan mencakup poin-poin utama.
4. Berikan skor relevansi 1 hingga 5.
"""
),
# ...Coherence and Consistency follow the same shape
"Fluency": (
"Fluency(1-5): kualitas ringkasan dalam hal tata bahasa, ejaan (EYD - Ejaan yang Disempurnakan), tanda baca, pemilihan kata, dan struktur kalimat."
"1: Sangat Buruk. Ringkasan memiliki banyak kesalahan yang membuatnya sangat sulit dipahami atau tidak koheren."
"2: Buruk. Ringkasan memiliki kesalahan signifikan yang mengganggu kelancaran dan kejelasan makna."
"3: Cukup. Ringkasan memiliki beberapa kesalahan yang mempengaruhi kualitas teks tapi poin utama masih terpahami."
"4: Baik. Ringkasan memiliki sedikit kesalahan minor yang tidak mengganggu pemahaman secara signifikan."
"5: Sangat Baik. Ringkasan hampir sempurna dalam struktur bahasa, mudah dibaca, dan mengikuti semua aturan EYD.",
"Baca ringkasan dan evaluasi kelancarannya berdasarkan kriteria yang diberikan. Berikan skor 1 hingga 5."
)
}We average the four scores, and if the result falls below the MIN_AVG_SCORE threshold, we regenerate the summary from scratch rather than patch it. Of course, we didn't want an infinite loop, so we do graceful degradation: after RETRY_SUMMARIZE_COUNT failed attempts, we return the best-effort result instead of hard-failing.
There's still room to improve here. We could do a lot of things, for example, using the crawler I mentioned earlier on every retry to pull in more news and enrich the source data.
Once we got the summarized news with a valid format and a passing score, we stored the data in the DB, and the rest was up to the UI/UX, BE, and FE to shine!
Thoughts
The highlight of this project for me was the evaluation method, the LLM-as-Judge. From a business POV, benchmarks and evals are a never-ending debate. I mean, the stakeholders always want a single answer to a question: is this system good enough? Meanwhile, we as engineers would ask: good enough based on which metrics? Cost? Compute? Storage? Accuracy? Quality?
With this feedback loop of quality control, at least we have an answer. I don't know for which business metric, but at least we have one.
There are so many things I'd do differently next time. One year (as I write this, it's 2026) feels like a decade in AI development after all.
- Utilize structured output and crawling as mentioned above
- Explore other LLM-as-Judge methods
- Mitigate LLM-as-Judge bias
- Caching: prompt caching, result caching
- A more advanced personal use case: trading signals
In terms of fun, I found this project a bit too basic, so frankly it was only okay. I'll talk about a project that brought me more joy next time, whenever that ends up being.
In terms of learning, this built strong fundamentals for me on building a production-grade AI system. So this project is valuable.
In terms of philosophy, looking at the Google Analytics events, well, someone, somewhere, was using this project of mine. So the feeling of being a little bit useful to society still applies here. Good.