OpenLDAP — Identity & Access

The lab has exactly one identity store: OpenLDAP on the tooling plane. Every human and agent login in the lab resolves to it, directly or through one hop of federation. If a service grows its own user database, that is a bug to file, not a pattern to copy.

• Front door: ldap.rhels.com (LDAPS, L4 — see Part A).
• Git source: rhels/lab-agentops, gitops/modules/ldap/ — the seed file is github.com/rhels/lab-agentops/blob/main/gitops/modules/ldap/ldap.yaml (the openldap-seed ConfigMap inside it carries the LDIF).
• Consumers and break-glass: below, in Part A and the end of Part B.

Part A — How it is set up

Deployment

• Where: namespace agentops on the tooling cluster (Mac mini), Deployment openldap, image osixia/openldap:1.5.0, one replica, Recreate strategy. Data and server config live on two PVCs (openldap-data, openldap-config), so a pod restart never wipes the directory.
• GitOps: the mini’s Argo CD app agentops-ldap syncs gitops/modules/ldap from rhels/lab-agentops (automated, prune + self-heal). One file, ldap.yaml, carries everything: Deployment, Service, both PVCs, the seed ConfigMap, and the seed Job.
• Shape: base dc=agentops,dc=local; users under ou=users,dc=agentops,dc=local as inetOrgPerson entries (uid, cn, sn, mail, userPassword as an {SSHA} salted hash — never plaintext); groups under ou=groups,dc=agentops,dc=local as groupOfNames entries whose member attributes are full user DNs.
• Fixture users: dev1, dev2, ops1 — seeded test identities used by the login checks and day-to-day sign-in. A fourth user, svc-smoketest, was added post-bootstrap for the health suite and codified into the seed on 2026-10-24 (only its {SSHA} hash is in git, like the others; the plaintext is held in Vault and in the health suite’s private env file). Fixtures are not people; treat their passwords as low-value test credentials.
• Listeners: plain LDAP on 389 inside the cluster (every current consumer binds this way); LDAPS on 636 since 2026-10-24, with a Let’s Encrypt certificate for ldap.rhels.com issued by cert-manager (Certificate ldap-rhels-tls in the agentops namespace — see certificates.md). The cert mounts at /certs, not inside the config tree: the image’s startup script sweeps ownership of its config directory and dies on a read-only mount inside it (proven the hard way, 2026-10-24). TLS is additional, not enforced — plaintext binds are unchanged.

How the seed reaches the server — and what actually propagates

The osixia image’s own bootstrap-LDIF mechanism is not used. The image initializes its config and database on the empty PVCs at first start only; after that, restarts reuse the PVC state (this is also why changing directory passwords via container environment alone does nothing after first init — the values live in the database).

Seeding is a separate, deliberate step:

1. The LDIF lives in the openldap-seed ConfigMap (key seed.ldif) inside ldap.yaml.
2. An Argo CD PostSync hook Job (openldap-seed, hook-delete-policy BeforeHookCreation, so it is re-created on each completed sync) waits for the server, then runs ldapadd -c as the directory admin against it.
3. ldapadd -c means add, continue on error. That one flag defines exactly what git can and cannot change in the live directory:

Two operational caveats, both verified on 2026-10-24:

• The hook runs only when a sync operation completes. The agentops-ldap sync for the LDAPS change sat in Running (“waiting for healthy state of Deployment”) long after the new Deployment was live and serving, and the in-cluster seed Job was still the run from 2026-10-15 — the hook had not re-run. Lesson: after pushing a seed change, check the seed Job’s age and logs (kubectl -n agentops get jobs, kubectl -n agentops logs job/openldap-seed); if the change must land now, apply it directly with ldapadd/ldapmodify (Part B) and let git remain the record.
• Seed vs live, as of the 2026-10-24 codification. The entries that arrived through runtime paths — the svc-smoketest user and the labtest-* groups — were codified into the seed LDIF, so the seed now restores the directory exactly as it stands (verified by a full seed-vs-live comparison: no unexplained differences; re-verified entry-for-entry and member-for-member on 2026-10-26). The one remaining by-design difference is the app1002-prod group, which is deliberately unseeded (a fixture — see the group machinery below) and arrives through the ldap-groups/ request path. One earlier drift entry is gone entirely: the app1003-nonprod group was removed from the live directory on 2026-10-24 at the owner’s direction, and no app1003-* group exists live or in the seed — the fixture premise is that the whole app1003 segment stays absent so the segment-failure proof path works. A cold rebuild re-seeds only what git holds; see disaster-recovery.md.

The group machinery around the directory

• agentops-group-sync (Argo CD app, same repo, gitops/modules/group-sync): a CronJob every 2 minutes running a read-only checker. For each group in its ConfigMap list (group-sync-groups) it searches the directory, compares members, writes the result to the group-sync-status ConfigMap (with a failing_since timestamp per failing group, so Alertmanager can escalate only groups failing for a long time), and pushes metrics to the tooling plane’s Pushgateway. It never writes to LDAP. A group that does not exist yet fails alone — that isolation is the point: app1002-prod was deliberately unseeded so its absence proved the checker, until a request created it.
• Group creation from git (ldap-groups/ in rhels/lab-agentops): one small YAML file per group (group, bsn, requested_for, ticket). A GitHub Actions workflow (ldap-group-apply.yml, on the self-hosted mini-k8s runners — only they can reach the directory) runs scripts/apply_ldap_groups.py on every push touching ldap-groups/**: missing groups are created with requested_for as first member; existing groups are left exactly as-is. The ServiceNow PDI’s “Create LDAP Group” catalog item commits these files via the GitHub API (see tools/servicenow-pdi.md); humans may commit them directly too.
• Projection into OpenShift: on CRC, group-sync and openshift-group-sync CronJobs in the agentops namespace project LDAP groups into OpenShift groups, so authorization follows the directory. The OpenShift group projection is a one-shot sync — membership changes appear there when that sync re-runs; authentication itself is unaffected.

Who consumes it

Credential custody

Directory credentials live in Vault (see vault.md), never in git and never in these docs:

• secret/lab/ldap/admin — the directory admin credential; delivered to the CRC agentops namespace by a Vault Secrets Operator tenant. On the mini, the same value reaches the pod and the seed Job from the agentops-ldap-admin Kubernetes Secret (owner-placed).
• secret/lab/ldap/fixture-users — the fixture users’ passwords; this is also where the health suite’s expectations come from.
• Per-consumer bind credentials sit next to their consumers (secret/lab/ldap/openshift-bind, secret/lab/harbor/ldap-bind).

The seed carries {SSHA} hashes by design — a private repo and low-value fixtures, but anything committed is permanent: the fixture passwords were rotated on 2026-10-15 (the seed commit records the new hashes; the plaintexts of record are in Vault), and the pre-rotation hashes remain in history as dead credentials. Never put a plaintext password, or a hash of a password used anywhere else, into the seed.

Part B — How to use it

All examples run against the live server by executing the directory tools inside the running pod on the tooling host (its environment already carries the admin password — never print it):

ssh <tooling-host> '/opt/homebrew/bin/kubectl -n agentops exec deploy/openldap -- sh -c "<command>"'

Look up a user or a group

# one user (dn + attributes)
ldapsearch -x -H ldap://localhost:389 \
  -D "cn=admin,dc=agentops,dc=local" -w "$LDAP_ADMIN_PASSWORD" \
  -b "ou=users,dc=agentops,dc=local" "(uid=dev1)"

# who is in a group
ldapsearch -x -H ldap://localhost:389 \
  -D "cn=admin,dc=agentops,dc=local" -w "$LDAP_ADMIN_PASSWORD" \
  -b "ou=groups,dc=agentops,dc=local" "(cn=app1001-nonprod)" member

Other pods can also browse via the openldap Service on port 389; writes always need the admin bind.

Add a user — the verified flow

1. Choose the password and hash it. Inside the pod: slappasswd -s '<password>' prints an {SSHA} value. Record the plaintext in Vault under secret/lab/ldap/fixture-users (or the consumer’s own path) — git gets the hash only.

2. Edit the seed in rhels/lab-agentops (gitops/modules/ldap/ldap.yaml, the seed.ldif inside the openldap-seed ConfigMap) — add the user entry:

   dn: uid=dev4,ou=users,dc=agentops,dc=local
   objectClass: inetOrgPerson
   cn: dev4
   sn: Four
   uid: dev4
   mail: dev4@LAB-DOMAIN
   userPassword: {SSHA}<hash from step 1>

   To record intended membership, also add member: uid=dev4,ou=users,dc=agentops,dc=local lines to the groups in the same LDIF — knowing step 3’s limit:

3. Understand what the sync will and will not do. On the next completed agentops-ldap sync, the PostSync seed Job creates the new user entry. It will not add the member lines to groups that already exist (see the propagation table in Part A).

4. Apply the membership live with ldapmodify (this is the step that makes the user actually able to do anything):

   dn: cn=app1001-nonprod,ou=groups,dc=agentops,dc=local
   changetype: modify
   add: member
   member: uid=dev4,ou=users,dc=agentops,dc=local

   ldapmodify -x -H ldap://localhost:389 \
     -D "cn=admin,dc=agentops,dc=local" -w "$LDAP_ADMIN_PASSWORD" -f change.ldif

   If the sync is stuck or the user is needed immediately, create the user entry the same way with ldapadd and the LDIF from step 2 — git stays the record either way.

5. Verify (below): the search shows the entry and the membership, and a bind as the new user succeeds.

6. Commit and push the seed edit if you have not already — a rebuild must reproduce this user.

Change a password

ldappasswd -x -H ldap://localhost:389 -D "cn=admin,dc=agentops,dc=local" -w "$LDAP_ADMIN_PASSWORD" "uid=dev4,ou=users,dc=agentops,dc=local" -s '<new-password>' — then regenerate the {SSHA} hash with slappasswd, update the seed LDIF to match, and update the plaintext of record in Vault. Editing the hash in git alone changes nothing live (Part A).

Create a group

• Standing/seeded groups: add the groupOfNames entry to the seed LDIF (a new DN — the seed Job will create it), commit, sync.
• Request-driven groups: add one file to ldap-groups/ in rhels/lab-agentops and push; the ldap-group-apply workflow creates the group with its first member within the run. This is also exactly what the ServiceNow catalog item does.
• Then add the group name to the group-sync-groups ConfigMap list if the checker should watch it, and expect the OpenShift projection to pick it up on its next sync run.

Remove a user

ldapdelete the entry live, remove their member: lines from every group (live, via ldapmodify, and in the seed), then remove the entry from the seed LDIF and commit. Deleting from git alone leaves the live user fully able to log in.

Verify a login works

# bind test — prints the DN on success, "Invalid credentials" on failure
ldapwhoami -x -H ldap://localhost:389 \
  -D "uid=dev4,ou=users,dc=agentops,dc=local" -w '<password>'

Then prove the real path: sign in to one consumer (the OpenShift console or the tooling-plane Argo CD) as the new user. The health suite’s login checks are the standing proof for the fixture users — a green suite means every row of the consumer table worked end to end within the hour.

Break-glass paths

LDAP being down must not lock the lab. Each plane keeps exactly one non-LDAP path, and each is deliberately awkward:

• CRC: the kubeadmin account, retrievable on the CRC host itself.
• Harbor: the built-in admin, whose password is Vault-managed (it was rotated off the old committed default on 2026-10-24 — see vault.md).
• Tooling-plane Argo CD: the local admin, whose initial secret is recorded in Vault (secret/lab/argocd/mini-initial-admin).
• Grafana on CRC: no local login at all; recovery is grafana-cli inside the pod, or editing the GitOps config back.