Agentifying schemes.sg: rebuilding around a tool-calling agent

Singapore's social service sector supports 400,000+ people through 500+ agencies, yet finding the right help is still difficult. Schemes.sg makes 600+ schemes from 200+ organisations searchable in multiple languages. Today, it is a live product with about 400 monthly active users, and that number is growing.

We borrowed Perplexity's answer-first pattern: ask a question, get a grounded response, then refine it through follow-ups. But Schemes.sg is not just an answer engine; people inspect and filter a catalogue. That creates a less familiar UX: each follow-up may need to change the answer, filters, and visible schemes together.

We started with one-shot RAG and ended up rebuilding Schemes.sg around one tool-calling agent. Agentification was not a framework migration. LangGraph gave us the loop, but most of the work was elsewhere: tool design, retrieval, application state, evaluation, and UX.

The implementation is open source on GitHub, so you can inspect the code alongside the architecture. If you are interested in our journey, or are thinking about agentifying your own product, read on.

The project is built by a group of passionate volunteers from the better.sg community. I am the tech lead for the full-stack team, working across product, frontend, backend, and agent architecture. The implementation details below are pinned to release v4.1.0, released on 15 July 2026.

---

TL;DR

• Agentification was a product rebuild, not a framework migration. LangGraph provided the loop; tools, retrieval, application state, evals, and UX made it work as a product.
• One tool-calling agent was enough. Better retrieval improved relevance, and an eval harness catches regressions in search, tool routing, and answers. Trust still depends on good data, careful UX, and people keeping more than 600 schemes current.

Before the agent

Before v4, Schemes.sg used a conventional RAG pipeline. A query searched an in-memory Chroma index for the top-k scheme IDs. The application fetched those records from Firestore, placed them in the model's context window, and asked it to summarize them.

Chat could explain that fixed set, but it could not search again. Reindexing was manual and took about an hour with validation.

We began with manual relevance checks by the team and volunteers, then used foundation models to help curate a ground-truth set and built an eval harness around it.

The harness tracked Precision@k and Recall@k for retrieval; separately, an LLM judge evaluated chat responses at scale. The evaluation work deserves its own post.

Our evals confirmed what users told us. Some schemes were outdated. Some searches returned irrelevant results, including non-financial schemes for financial-assistance queries. Data quality still needs work.

User interviews and usability studies surfaced another problem. Users wanted filters to apply automatically as they described their needs. They also questioned why every search returned the same number of results. A broad query and a narrow query should not produce lists of the same length.

That expanded the work beyond improving relevance. The rebuilt flow lets follow-ups search again, refine the active set, and update the visible results.

How the agent architecture works

Before getting into why we chose one agent over a MAS, it helps to see the agent path end to end. The diagram is deliberately scoped to conversational search, not the entire Schemes.sg platform. It follows a query through the Next.js client, Firebase streaming function, and LangGraph ReAct loop, then shows how tool updates return to the screen.

Outside this path are other API endpoints, Cloud Run services, scheduled batch jobs, and a human-in-the-loop data pipeline that posts proposed schemes to Slack for review. We will not cover those systems in this post.

Read it from left to right: client, streaming boundary, agent loop, then the services behind its tools. The frontend is hybrid rather than purely server-side rendered (SSR) or client-side rendered (CSR). Next.js App Router prerenders the initial HTML. The route is a Server Component; ChatHome and ChatPage are Client Components.

After hydration, the browser owns the live chat state, visible results, and response stream. It posts directly to Firebase, without a Next.js API route in between.

Firebase verifies the user, creates or reuses a sessionID, and opens a server-sent event (SSE) response. The same ID becomes the LangGraph thread_id, allowing a Firestore checkpoint to restore the conversation across turns.

At the center, RouterAgentGraph is a conventional single-agent ReAct loop. A model call either answers or requests tools. LangGraph's ToolNode executes those calls, adds the results to the same thread, and invokes the same model again. The loop ends when the model stops calling tools.

The follow-up generator runs once after the answer. It has no tools and no loop, so it is a post-processing step rather than a second agent.

The tools search the index, refine the active set, load complete records, and read public sources. Search and refinement tools also publish a typed schemes_update event. The browser applies that event directly to the cards and result count, so a conversational refinement changes the application state, not just the answer.

Azure OpenAI supplies chat and embeddings. Firestore stores scheme records and graph checkpoints. Langfuse records traces. These services surround one decision loop; only RouterAgentGraph chooses the next product action.

One agent, not a MAS

My bias is simple: distributed coordination should be earned. Schemes.sg search is sequential and stateful. A search creates the active result set; each refinement changes it; the answer and cards must stay aligned. Splitting those writes across agents would add handoffs around the same state.

"Avoid multi-agent architectures early: Simple single-agent systems can handle more complexity than you might expect."

Andrew McNamara, Shopify Engineering, Building production-ready agentic systems

Specialization lives in tools and skills. Tool calling gives each capability a narrow boundary without another agent loop; MCP extends that pattern to remote capabilities. Search, refine, inspect, and verify are executable tools.

load_skills supplies guidance for clarifying vague requests, writing compact mobile replies, or drafting an email. The same agent switches among tools and skills as intent changes.

MAS still makes sense for independent, breadth-first work such as Anthropic's parallel research system. Schemes.sg writes to one shared result set, so one ReAct loop was enough.

Tools follow intent

The tool boundaries came from user behavior, not an agent framework. Early usability planning and demo feedback surfaced confusing filter labels, interest in multi-select, and a need for conversational follow-ups. Users should not need to translate a real situation into the product's taxonomy before they can search.

That research became a product requirement: distinguish a new search from a refinement or a detail question, preserve context across turns, and update the result list while explaining the change. That led to five capability groups:

• Discover: search_schemes searches the index for a new need.
• Refine: filter_rerank_by_directive filters or reorders the active result set from a follow-up such as "only those for seniors."
• Inspect: retrieve_schemes_by_ids loads the complete stored record for a known scheme.
• Verify: fetch_webpage reads the official page, then duckduckgo_web_search finds a source when the database is not enough.
• Adapt: load_skills loads task-specific guidance for clarifying, summarising, or drafting an email.

The old pipeline had one path: run a vector search and summarize the results. The agent can now search, refine, inspect, verify, or load task guidance as the conversation changes. That is the adaptability we wanted from agentification.

Why search relevance still matters after agentification

For a discovery query, the agent still starts with search. If that result set is weak, every later step starts with weak evidence. Search relevance therefore remains a product priority.

I considered training a neural recommender, but we had anonymous query logs rather than reliable outcomes such as applications or successful matches. This was an intent-retrieval problem, not a personalization problem.

We chose hybrid search. Firestore vector search handles different ways of describing the same need, while BM25 restores exact qualifiers such as "seniors", "low-income", or "disability".

A relevance cutoff lets broad and narrow queries return different result counts. Keeping the vectors in Firestore also removed the Chroma index we had to package and refresh.

We ran the legacy and current search over the same 684-scheme corpus. For six representative queries, we had labelled which schemes were relevant. The table compares how well each search recovered that ground truth:

• Precision@K: Of the first K results, how many were relevant?
• Recall@K: Of all known relevant schemes, how many appeared in the first K?

We track precision because irrelevant results erode trust. We track recall because a missed scheme can mean missed support.

Version	P@1	P@3	P@5	P@10	R@5	R@10
Legacy: mpnet + FAISS, vector only	0.50	0.72	0.67	0.47	0.28	0.40
Current: Azure + Firestore, hybrid	1.00	0.94	0.83	0.83	0.36	0.72
Change (relative lift)	+0.50 (+100%)	+0.22 (+31%)	+0.16 (+24%)	+0.36 (+77%)	+0.08 (+29%)	+0.32 (+80%)

The current search put a relevant scheme first for all six queries, up from three. In the top 10, relevant results rose from 47% to 83%, while coverage of the known relevant set rose from 40% to 72%.

The averages hide where the gains came from. P@5 shows the quality of the first five results. R@10 shows how much of the labelled set appeared in the first ten.

Query	Known relevant	Legacy P@5	Current P@5	Legacy R@10	Current R@10
Single parent + financial aid	10	0.20	0.80	0.10	0.70
Healthcare subsidies for seniors	8	0.60	0.60	0.62	0.75
Education grants for low-income families	11	0.60	0.60	0.36	0.64
Seniors or caregivers	15	1.00	1.00	0.33	0.67
Disability or transport support	13	0.80	1.00	0.38	0.77
Legal or safety support	13	0.80	1.00	0.62	0.77
Average		0.67	0.83	0.40	0.72

The results suggest why:

• The large gain for "single parent + financial aid" likely comes from combining complementary signals. Dense embeddings connect different ways of describing single-parent needs, while BM25 gives exact financial-assistance terms more weight. The legacy vector-only search was easier to pull towards generic family support.
• Healthcare and education remain harder because weighted fusion is not a strict AND. A scheme matching "seniors" and "subsidy" can rank well without being healthcare support. The same happens with general education aid that is not specifically for low-income families.
• The broad OR queries are easier because either branch can count and their terms are explicit enough for BM25 to match. Their R@10 scores also hit the limit of the metric: only ten of the 13–15 labelled schemes can appear in the first ten results.

We also ran the real LangGraph agent through an eval harness. It measured tool-use accuracy, plus RAGAS faithfulness and response relevancy. Together, these act as a gate against regressions in routing and answer quality, not proof of production quality.

The agent UX

Agentification is also a UX change. I cannot do justice here to all the UX work behind v4. I am proud of the care the team put into it.

Agent calls take longer than one-shot search, so the waiting state matters. While the agent searches, refines, or checks a source, Firebase streams short action_message events. The UI shows the current action, then keeps the completed trace under How I found these.

We show what the agent did, not its raw chain-of-thought.

Result changes have their own cues. Desktop keeps chat and cards side by side. On mobile, the Schemes tab pulses, and a tappable result count opens the updated list.

Every completed answer also has copy, thumbs-up, and thumbs-down actions. A rating updates immediately, can be undone, and is stored against the session and message.

Takeaway

LangGraph gave us an agent loop, not an agentic product. The rest came from the boundaries around it: focused tools, better retrieval, explicit application state, evals, feedback, and UX that makes the agent's actions legible.

The evaluation follows the same boundaries: precision and recall for retrieval, expected tool sets for routing, and RAGAS faithfulness and response relevancy for answers. Ratings and traces add production signals, but no metric is sufficient alone.

Users do not benefit from agent count. They benefit when the answer, cards, sources, and result count agree.

We are still a long way from our vision for Schemes.sg. The goal is not to build a convincing agent. It is to make finding help easier and more trustworthy.

Architecture alone cannot keep more than 600 schemes current, define relevance for every situation, or tell us whether a result actually helped someone. Those are people and process challenges as much as technical ones. They require sustained user research, data stewardship, evaluation, partner input, and volunteer coordination.

This architecture gives us a better foundation. It does not finish the work.

If you use Schemes.sg, there are two concrete ways to help:

• Share feedback on what worked, what was inaccurate, or where the experience broke down.
• Contribute a scheme if you know of support we are missing.