MCP Front Door (agentgateway)¶

Every Model Context Protocol server on the estate is reached through one front door, https://mcp.mdapi.ch, served by agentgateway (standalone, database mode) in the agentgateway namespace. Clients never hold an upstream credential: the gateway injects it.

Until 2026-11-02 this role belonged to LiteLLM's MCP gateway. That feature was retired here (the fleet config carries no mcp_servers, and no client permission lists one) and the front door moved to agentgateway, which is what kagent and Backstage already used internally. What changed is not the routing but the audience: the front door is now reachable from every seat and every agent, not only from inside the cluster.

Why a front door rather than one endpoint per server¶

A direct MCP endpoint authenticates with the upstream's own token — the Joplin token, the Windmill token, the cluster-admin kubeconfig behind k8s-mcp. Handing those to each client means the credential and the authorization live in the same place, so revoking a client means rotating what every other client uses.

Behind the gateway the two are separate. A client presents one key that identifies the client, the gateway decides which servers that client may reach, and the upstream credential never leaves the cluster. Revoking a client is one key; narrowing a client is one row; neither touches the upstream.

What is served today¶

Path	Upstream	Tools
`/memsearch/mcp`	`memsearch` — the estate's memory brain	4
`/kubernetes/mcp`	`kubernetes-mcp-server`, read-oriented	11
`/backstage/mcp`	Backstage catalog	4
`/notes/mcp`	Joplin notes	16
`/windmill/mcp`	Windmill scripts and flows	155
`/gitlab/mcp`	GitLab — pipelines and jobs, merge requests, repositories	36

GitLab is the one upstream whose credential is not an estate token: its built-in MCP server accepts only an OAuth token carrying the mcp scope, or a fine-grained PAT holding the MCP tool with Execute (legacy, project and group tokens are refused with 403 insufficient_scope). So a dedicated PAT lives at /mdapi/gitlab/mcp-token and is rotated on its own schedule.

Two more MCP servers serve per-user data and stay outside the front door on purpose (see Per-user data below): the household document, mail and photo servers that Open WebUI reaches per user, and the direct notes-mcp endpoint used for personal Joplin sessions.

Two authentication lanes¶

API keys — machines, seats and agents. Authorization: Bearer <key>, dispatched by header on the public listener. One key per client, stored in Akeyless under /mdapi/agentgateway-system/mcp-clients/<client>; the cluster injects it into the client that needs it, and no key is shared between two clients. This is the lane for a seat, a CI job, or an agent such as kagent.

OAuth — a human session acting through a public client. https://mcp.mdapi.ch/oauth/<server>/mcp publishes protected-resource metadata so a client discovers the authorization server itself (Keycloak, realm tilloch, public client mcp-gateway, role mdapi-mcp-oauth). This is the lane claude.ai and similar hosted clients use. Note what it is not: the gateway still injects one machine credential, so an OAuth session says who may reach the server, not as whom the server is reached. Anything that must act as the individual user stays on a per-user endpoint.

What has an OAuth route is a deliberate choice, not a consequence: five of the six servers do (windmill, notes, kubernetes, memsearch, backstage), while GitLab is not published that way here — a hosted client reaches GitLab's own OAuth-protected MCP endpoint directly instead.

Where the authorization actually lives¶

Per-server routes are rows in agentgateway's own database, and each row carries both the matching and the grants:

matches[].path.pathPrefix              "/notes/"
policies.apiKey.keys[].metadata        {"client": "seat-…", "servers": ["notes", …]}
policies.authorization.rules[].allow   '"notes" in apiKey.servers'

A grant counts only on the route that serves that server — the same servers list rides on every row of that client, so a pair named on a foreign row authorizes nothing.

These rows are not in git. The fleet bundle seeds a baseline read-only configuration and the UI writes an overlay on top of it; an attempted update of an overlay row by path is refused as file-owned. The consequence is deliberate and worth stating plainly: the front door's route table is restored from the CNPG backup of its database, not from the repository. The fleet repository holds what surrounds it — namespace, database, network policy, ingress, Keycloak wiring — and nothing about its content.

The admin API likewise carries no authentication of its own. It is reachable only from inside the cluster and every operation is done through a port-forward, never by exposing it.

What the guard rails are¶

Three things make the invisible part visible, all shipped together on 2026-11-12:

A catalog — ConfigMap agentgateway-catalog, rewritten every 15 minutes: every API-key server and its tools, for clients that need to discover what exists.
A grant matrix — ConfigMap agentgateway-grants plus Pushgateway metrics from the same job: which client may reach which server. Because the matrix is a database row, this is the only machine-readable copy that exists.
Five alerts — export stale, export failed, a grant claimed on a route that does not serve it, a client that lost every grant, and the matrix disappearing entirely.

Both jobs run in the agentgateway-catalog-export CronJob, which is the only workload allowed to reach the admin API besides ingress-nginx. That is also why the guard lives there: a worker outside that allow-list cannot reach the admin API or the MCP listener at all.

Connecting a client¶

Outside the cluster, point the client at https://mcp.mdapi.ch/<server>/mcp. Inside the cluster, use the service directly — the network policy admits only the namespaces that are meant to call it:

http://mcp.agentgateway.svc.cluster.local:8080/<server>/mcp

The older mcp.agentgateway-system.svc name was an ExternalName stand-in during the move out of the agentgateway-system namespace; it is gone, along with the namespace, so a note that still shows it is stale.

On a seat, the client configuration is derived rather than hand-written. agw-mcp-sync reads the grant matrix, renders the entry for each granted server into the harness configs, and reports drift; a session that starts out of sync says so instead of failing later with silent 401s:

agw-mcp-sync            # report drift, non-zero exit when out of sync
agw-mcp-sync --apply    # write, with a timestamped backup

Operating it¶

# read the live routes and grants (never expose this endpoint)
kubectl -n agentgateway port-forward svc/agentgateway-ui :4001
curl -s http://127.0.0.1:4001/api/config/resources | jq '.resources[] | {id, kind}'

# add or update a route
curl -X PUT http://127.0.0.1:4001/api/config/resources/traffic.route \
  -H 'content-type: application/json' -d '{"resources":[{"value":{…}}]}'

# prove a server answers, and with what
curl -s http://mcp.agentgateway.svc.cluster.local:8080/notes/mcp \
  -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \
  -H 'Authorization: Bearer …' \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"probe","version":"1"}}}'

A tools/list without a session answers 400 session header is required: call initialize first and send the returned Mcp-Session-Id on every later request.