Namespace as a Service — the MCP front door

An application team that needs a namespace does not file a form and does not talk to the platform team. They connect their own agent (Claude Code, Cursor, an internal chat agent) to the lab’s MCP server and say one sentence:

“I need a namespace. My BSN is _______.”

Everything else — the registry record, the support group, the eligible clusters, the LDAP group, the name, the quota — is fetched, derived, or recommended by the platform. The requester confirms a plan once; the platform executes. This is the lab’s working simulation of the production IssueOps namespace flow, rebuilt agent-first and proven end-to-end (POC 2026-10-12 on the tooling plane, ported to CRC 2026-10-13, both MCP servers on the real ServiceNow PDI since 2026-10-15). The ServiceNow side of the story — the registry, the catalog-to-git group flow, the keep-alive — has its own page: ServiceNow PDI & the MCP servers.

Part A — How it is set up

The one rule

The agent is the front door, never the authority. The agent conducts the conversation, gathers intent, and explains outcomes. It never decides whether a validation passes: every rule is enforced by deterministic code on the server side, re-run at plan time and again at creation time. A hallucinating or manipulated agent cannot create an invalid namespace, because the MCP server re-checks everything itself before it acts, and what gets created is the confirmed plan — never the conversation.

The flow, step by step

1. Connect. The team’s agent connects to the MCP server over streamable HTTP (https://mcp.rhels.com/mcp). The server runs on the tooling plane (Argo CD app agentops-mcp-server) and a second instance serves CRC; both are the same code from rhels/lab-agentops.

2. Log in. Login is an HTTP call — POST /login on the MCP server, not an MCP tool. It performs a real LDAP bind against the lab directory (identity.md) and returns an opaque session token (8-hour TTL), which the client then sends as Authorization: Bearer <token> on every MCP request. Every tool derives the caller’s identity from that token alone. No token → refused; wrong password → 401. Identity comes from the bind, never from anything typed into the chat. The call itself, so Part A is self-contained:

   curl -sS -X POST https://mcp.rhels.com/login \
     -H 'Content-Type: application/json' \
     -d '{"username": "<your directory uid>", "password": "<your directory password>"}'

   There is no separate administrator login for this flow and no admin-only MCP principal: an administrator walking this setup logs in with the same kind of seeded directory account as any other caller (Part B, “Which uid and password?”). For the worked app1001 flow that is the seeded fixture user dev1; what a caller may do is decided server-side by group membership, not by a special login. No page in this book can give you a password: directory accounts are seeded by the lab administrator, and the password reaches you only through that seeding handoff, outside this book. Treat the handoff as a hard prerequisite — if you do not already hold a uid and password, stop here and get one seeded first. Fixture passwords are held in Vault at secret/lab/ldap/fixture-users — that path is a custody record for readers who already have lab Vault access, not a way to obtain one. The directory-admin credential at secret/lab/ldap/admin administers the directory itself — it is not an MCP login, and this flow never asks for it. Nothing in this flow can be done anonymously.

3. Discover. get_bsn_details reads the BSN’s registry record — application name, business segment, the dev-support and operations groups, the prod / non-prod LDAP groups, the eligible clusters. The record’s source is the real PDI registry table; a simulator service with the same API shape stands in for exercises that should not touch the real instance (one configuration switch, SNOW_BACKEND).

4. Verify. verify_requester checks the caller against the BSN’s dev-support group: the registry names the group, the directory says who is in it. Both halves must agree, server-side.

5. Plan. plan_namespace runs all validations and assembles the complete change set without writing anything: BSN exists · requester is a member · cluster is eligible for the segment and environment · namespace name is free · LDAP group state · quota values are valid quantities. Any failure returns here, in conversation, with the remediation (“you are not a member of <group>; request access, then say continue”) — before anything exists anywhere.

6. Missing group? File, don’t fail. If the LDAP group does not exist, ensure_ldap_group files the ServiceNow “Create LDAP Group” catalog request, fire-and-forget. The PDI’s Business Rule commits the group file to git; the runner workflow reconciles it into the directory; the namespace flow proceeds in parallel and never waits (the full mechanics are on the ServiceNow page).

7. Create — as a pull request. create_namespace_request opens a branch and pull request in rhels/lab-agentops carrying the request files (below). Merging the PR is the actuation; there is no other write path.

8. Argo CD applies. Two ApplicationSets with automated sync and prune watch the merged files: agentops-bsn-projects turns each projects/<bsn>.yaml into one AppProject per BSN, and agentops-namespace-requests turns each requests/<bsn>-<suffix>.yaml into an Application of the namespace-request chart, which renders the Namespace (BSN metadata as labels and annotations), a ResourceQuota from the requested CPU/memory, a RoleBinding for the LDAP group (admin in non-production, view in production), and — when Vault is enabled on the request — the vault-auth ServiceAccount and a VaultAuth resource (Kubernetes auth method, role convention <bsn>-role).

9. Workload lands. The namespace is ready for the team’s own deployment. The POC proved this with a dummy workload pod Running in each created namespace.

The request file

One file per namespace request is the whole contract between the MCP server and GitOps:

# requests/app1001-cursor.yaml — as committed by the POC run
bsn: app1001
appName: Payments Demo
ldapGroup: app1001-nonprod
clusterType: nonprod
namespaceSuffix: cursor        # namespace name: app1001-cursor
supportContact: payments-team
cpu: "500m"
memory: "512Mi"
vault:
  enabled: false

The chart fails loudly on a missing BSN, group, or an invalid clusterType — a malformed request file cannot half-render.

Components and where they live

Security model

• The MCP server holds no cluster-admin. Its only cluster credential is a read-only ClusterRole used to aggregate status. Its one write path is git, through the GitHub API, with a token delivered as an out-of-band Kubernetes Secret — never committed.
• Authorization is server-side and per-tool. Discovery tools need a valid directory identity. The mutating tool additionally requires dev-support-group membership for the BSN in question, verified at call time. Proven negatively: a valid directory user outside the group gets a plan failure and a create refusal — and no commit results.
• No secrets pass through the agent. Backend credentials (the PDI’s, the directory bind’s) live in the server’s own secret references in-cluster. The agent sees tool results, never credentials.
• Injection hardening is regression-tested. The POC’s independent security review found one critical flaw — YAML injection through the quota fields into the committed request file — plus five high-severity issues. All were fixed on the server (strict quantity validation, safe scalar rendering) and re-proven; the hardened path is itself a fixture request (app1001-secure). The fixes belonged in the deterministic server; the front door needed no change. That is the design working as intended.
• Audit is the PR. Every request is a branch, a diff, a review, and a merge record — plus per-tool call and validation-failure counters scraped by Prometheus (Prometheus & Alertmanager).

Honest status: three completions, three clocks

A request being “done” is three independent completions, and get_request_status reports each one separately instead of pretending they are synchronous:

1. Namespace live — the PR merged and Argo CD synced.
2. Team can log in — the LDAP group exists and a group-sync cycle has carried it into the cluster.
3. Workloads can read secrets — the VaultAuth resource exists from day one, but the per-BSN Vault policy and auth role land on the Vault-side flow’s own clock.

PR-only, for every environment (owner decision, 2026-10-26)

Every namespace request — non-production included — travels as a pull request, and so does its cleanup. (History, in one line: the original POC committed non-prod requests directly and parked only prod requests for review; that split is retired.) Cleanup is a PR that removes the request file — or a revert PR — and the ApplicationSets’ prune then tears down the Application, the AppProject (when the BSN’s last namespace goes), and the namespace contents. The LDAP group deliberately survives: the directory is shared infrastructure, and group removal is a directory change of its own (identity.md).

The standing exercise

The lab’s six-hour end-to-end cycle exercises this flow on every run under a standing authorization (lab-monorepo issue #30) — no per-cycle human approval: a fresh test BSN goes through login, plan, request PR, merge, namespace verification, and the cleanup PR, and the cycle names the stage if any part fails. The challenge that set this up, and the gap report it produced, live on that issue.

Part B — How to use it

Connect and log in

Point your agent’s MCP client at https://mcp.rhels.com/mcp. Login is not one of the nine MCP tools — it is a plain HTTP call on the same server, made once, before the tool calls:

curl -sS -X POST https://mcp.rhels.com/login \
  -H 'Content-Type: application/json' \
  -d '{"username": "<your directory uid>", "password": "<your directory password>"}'

A successful bind returns {"token": "…", "user": "<uid>", "expires_in": 28800} — an opaque session token with an 8-hour TTL. Send it on every MCP request as the header Authorization: Bearer <token>; every tool below derives your identity from that token alone. No token → refused; wrong password → 401. If a call fails on auth, test the same credentials against the directory itself first — the MCP server has no credentials of its own to be wrong about.

Which uid and password? Your lab directory account (identity.md) — the same uid and password you would bind to the directory itself with. There is no self-service signup, and no page in this book can give you a password: directory accounts are seeded by the lab administrator, and the password reaches you only through that seeding handoff, outside this book. Treat the handoff as a hard prerequisite — if you do not already hold a uid and password, stop here and get one seeded first; nothing in this chapter can be done anonymously, and login cannot start until the handoff has happened. The worked request below runs as the seeded fixture user dev1, a member of app1001-devsupport, using the password from its seeding handoff. Fixture passwords are also held in Vault at secret/lab/ldap/fixture-users — that is a custody record for people who already have lab Vault access, not a way to obtain one: needing a password for this login grants no Vault access, and Vault access is not the bootstrap for this login.

The nine tools

A worked request

The POC’s conversational run, condensed (the requesting “user” was the Cursor agent itself):

1. “I need a namespace, BSN app1001.” → the agent calls get_bsn_details (Payments Demo; segment and groups returned) and verify_requester (caller is in app1001-devsupport — pass).
2. list_eligible_clusters recommends the non-prod cluster; the user picks the suffix cursor and accepts the default quota (500m / 512Mi).
3. plan_namespace — all validations green; the LDAP group app1001-nonprod exists, so no ticket is needed. The plan shows the name app1001-cursor, the quota, the admin RoleBinding, and the PR that will be opened.
4. The user confirms. create_namespace_request opens the PR adding requests/app1001-cursor.yaml.
5. The PR merges; the ApplicationSet creates the Application; Argo syncs the chart. get_request_status reports the namespace live, the workload pod Running, Argo Synced/Healthy.

Checking status, changing, retiring

• Status: get_request_status for one namespace or request; list_my_namespaces for your whole footprint. A group still pending shows as pending with its ticket number — not as an error.
• Change quota: edit the request file’s cpu / memory in a PR; merge; Argo applies. (A dedicated quota tool is a roadmap item, not a current tool.)
• Retire: open the cleanup PR (remove the request file, or revert the creating PR). Prune removes the namespace and its platform resources. The LDAP group and the registry record remain — say so in the PR so nobody expects them gone.

Contribute from day one

• Repo map: everything lives in rhels/lab-agentops — services/ (the MCP server and its sibling modules: cmdb, snow-sim, group-sync, openldap), charts/ (what a request renders), gitops/ + gitops-crc/ (how the modules are deployed), and the request/project files the ApplicationSets consume. The design document and POC evidence travel with the platform team’s goal record for the IssueOps simulation.
• Add or change a tool: a tool is one decorated function in services/mcp-server/app.py. Put every validation in the shared plan-time validation path, so the plan, the create path, and a hand-authored PR can never drift apart on policy. A tool that writes may only write through the request-PR path — never give a tool a cluster write or a second git path.
• How requests are tested: the POC proof pattern is the standard — a scripted MCP client run and a conversational agent run against the lab, including the negative cases (non-member refusal, wrong password, wrong-segment plan). Use SNOW_BACKEND=sim when an exercise must not touch the real PDI. The hourly auth health suite is the standing regression net for the login paths, and the six-hour cycle (above) exercises the full request flow unattended.
• Change discipline: code and GitOps changes land by PR on lab-agentops like everything else in the lab; a change to the flow updates this page in the same pass — the book is part of the definition of done (Docs system).