FinAgent

Building: two LangGraph agents — a router/synthesizer and a mixture-of-experts with parallel fan-out — run behind one env-swappable runner protocol, so the harness is provably reusable rather than wired to a single design. The same tools are served over MCP, so external clients reuse them without duplicated logic.

Evaluating: an LLM-as-judge harness scores a golden dataset with Ray fanning out the runs, an optimizer proposes and adopts better prompts against that score, and PyTorch fine-tunes on the judge-graded transcripts — so quality is measured rather than asserted.

Observing: LangFuse traces every run, Prometheus scrapes a /metrics endpoint into Grafana, and an Airflow DAG runs canaries against a rolling baseline with Alertmanager firing on drift — so a degrading agent pages someone before a user reports it.