Grafbase just launched Nexus, an open-source AI Router that unifies MCP servers and LLMs through a single endpoint. Designed for enterprise-grade governance, control, and observability, Nexus helps teams manage AI complexity, enforce policies, and monitor performance across their entire stack.
Built to work with any MCP server or LLM provider out-of-the-box, Nexus is designed for developers who want to integrate AI with the same rigor as production APIs.
Seems quite similar to the commercial nexos.ai platform, which also focuses on routing, governance, and observability for AI workloads, but as a proprietary solution rather than open source
Here are a few key differentiators vs LiteLLM today:
- Nexus does MCP server aggregation and LLM routing - LiteLLM only does LLM routing
- The Nexus router is a standalone binary that can run with minimal TOML configuration and optionally Redis - LiteLLM is a whole package with dashboard, database etc.
- Nexus is written in Rust - LiteLLM is written in Python
That said, LiteLLM is an impressive project, but we're just getting started with Nexus so stay tuned for a steady barrage of feature launches the coming months:)
Yeah they definitely belong in the same space. Nexus is an LLM Gateway, but early on, the focus has been on MCP: aggregation, authentication, and a smart approach to tool selection. There is that paper, and a lot of anecdotal evidence, pointing to LLMs not coping well with a selection of tools that is too large: https://arxiv.org/html/2411.09613v1
So Nexus takes a tool search based approach to solving that, among other cool things.
Disclaimer: I don't work on Nexus directly, but I do work at Grafbase.
The main difference is that while you can get Nexus to list all tools, by default the LLM accesses tools by semantic search — Nexus returns only the relevant tools for the what the LLM is trying to accomplish. Also, Nexus speaks MCP to the LLM, it doesn't translate like litellm_proxy seems to do (I wasn't familiar with it previously).
I'm curious, what issue does that solve? I'm only working on agents that make tool calls via HTTP in a home baked way but I can't imagine how resolving the tools from 2 MCP servers is harder than 1.
The issue is when you have many MCP tools the context becomes too large for the LLM. So Nexus indexes all the tools and lets you search for the right tool and then execute it.
Thanks, I think I get it now. In our case I've dealt with this problem by refactoring the monolithic agent into smaller agents, smaller, more specific prompts, fewer, more relevant tools.
We've found that monolithic agents just don't perform that well.
Listen to this article (with local TTS)
Today, we're excited to introduce Nexus - a powerful AI router designed to optimize how AI agents interact with multiple MCP tools and Large Language Models. Nexus serves as a central hub that aggregates Model Context Protocol (MCP) servers while providing intelligent LLM routing, security and governance capabilities.
Nexus is an AI router that solves two critical challenges in the AI ecosystem:
MCP Server Aggregation: Instead of managing connections to multiple MCP servers individually, Nexus consolidates them into a single, unified interface
Intelligent LLM Routing: Nexus intelligently routes requests to the most appropriate language model based on the task, cost considerations, and performance requirements
As AI applications become more sophisticated, they increasingly need to interact with multiple external services, APIs, and different language models. This creates several pain points:
Context: Helps the LLM select from a potentially large number of MCP tools without overwhelming it
Cost: Lack of strategic model selection can lead to unnecessary expenses
Observability: Provides insights into the performance and behavior of the AI system, enabling better decision-making and troubleshooting
Security: Ensures that requests are routed securely and in compliance with governance policies
Nexus acts as a proxy layer that connects to multiple MCP servers simultaneously. When your AI agent needs to access external tools or data sources, it makes a single request to Nexus, which then:
Helps the LLM to identify the appropriate MCP server(s) for the request
Handles authentication and connection management
Aggregates responses from multiple sources when needed
Provides a consistent API interface regardless of the underlying MCP implementations
Nexus is designed to integrate seamlessly into your existing AI workflow. Whether you're building customer service bots, code generation tools, or complex reasoning systems, Nexus can help streamline your architecture while improving performance and reducing costs.
This is just the beginning for Nexus. We're working on additional features including:
Advanced routing algorithms
Real-time analytics and monitoring dashboards
Custom routing rules and policies
Client-based rate limiting and observability
Enhanced security and compliance features
The future of AI applications lies in intelligent orchestration, and Nexus is our contribution to making that future more accessible and efficient for developers everywhere.
Want to learn more about Nexus or get early access to enterprise features? Reach out to our team - we're excited to show you what's possible when AI routing is done right.
