VoiceOps: Webex Calling User Provisioning via GitOps

The Problem

Provisioning users in Webex Calling means clicking through Control Hub: create the person, assign the calling license, set the location, punch in an extension, pick the calling privileges. It’s fine for one user. It’s tedious and error-prone for a batch, and there’s no history — no diff, no review, no record of why someone’s extension changed six months ago.

I kept thinking: this is exactly the problem Kubernetes solved for infrastructure. Declare desired state, let a controller reconcile it. So I built VoiceOps — user lifecycle management for Webex Calling driven entirely by Git.

This is a public architecture overview of a private project. The manifests and engine live in a private repo; the write-up and the bot flow are documented at github.com/affragak/voiceops-overview.

The Idea: Users as Manifests

Every user is a YAML manifest, deliberately shaped like a Kubernetes CRD:

apiVersion: voiceops.io/v1alpha1
kind: WebexUser
metadata:
  name: max-mustermann-mainhq       # unique resource name (lowercase kebab-case)
  namespace: voice-provisioning
spec:
  state: present                    # present = create/update, absent = delete
  firstName: "Max"
  lastName: "Mustermann"
  email: "[email protected]"
  location: "Main-HQ"               # must exist in LOCATION_MAP
  extension: "8101-1999"
  callingPrivileges: "International" # International | National | Local | Internal

The Git repository is the source of truth. Want to onboard someone? Add a manifest. Off-board? Flip state: absent. The state of your telephony estate is now a reviewable, versioned, git blame-able set of files.

The GitOps Flow

A merge to main is the only thing that changes Webex. A GitHub Actions workflow runs the provisioning engine (provision.py) against only the manifests that changed:

IT Admin: "Add Katrin Müller, ext 1986…"
         │
         ▼
   Translation agent  →  valid YAML manifest (voiceops.io/v1alpha1)
         │
         ▼
   branch → commit → gh pr create → Pull Request
         │
         ▼
   Human review & merge → main
         │
         ▼
   voiceops-sync.yml fires (only when manifests/users/ changed)
   ├─ reconciles ONLY the changed manifest(s)
   └─ applies state to the Webex API (create / update / delete)

Two things I care about here: a push only reconciles the user(s) that changed — never unrelated users — and a human merge gates every single change. Editing scripts, config or docs does not trigger a provisioning run.

The Provisioning Engine

provision.py is idempotent — users are looked up by email and updated in place, never duplicated:

State	Action
`present`	Look up by email → create if new, update if existing → assign license + location + extension + privileges
`present` (location changed)	Detect the mismatch → run a Webex Move Users job
`absent`	Delete from Webex → move the manifest to `manifests/archive/`

The Move-Users Gotcha

This one cost me real debugging time. A person’s locationId is immutable on the Webex person object — you can’t relocate someone with a plain update. You have to drive the Webex Move Users job: validate → initiate → poll → done.

Except the job reported COMPLETED while the user never actually moved. The catch: a move reassigns the user’s identity-group membership, so the admin token needs the identity-group write scope on top of the telephony scopes. Without it the job “succeeds” but silently does nothing. Now the engine reads the job’s per-user errors endpoint and reports the real result instead of trusting the top-level status.

Lesson: with cloud APIs, a COMPLETED job status is not the same as the outcome you asked for. Verify the effect, not the acknowledgement.

A Webex Chat Bot Front-End

Editing YAML is great for me, but not for the average IT admin. So the same translation agent is wrapped as a native Webex bot. An admin DMs it in plain language; it generates the manifest, presents it on an Adaptive Card with Approve / Cancel, and on approval opens the pull request. The human merge still gates everything — the bot never touches the Webex admin API itself.

Admin (Webex)  ──DM──▶  bot  ──▶  translation agent (Claude)
                         │        └─ natural language → YAML manifest
                         ▼
               Adaptive Card: 📄 manifest  [Approve] [Cancel]
                         │ approve
                         ▼
               branch → commit → open PR  ──▶  human merge  ──▶  provision

Each example below is a real DM: the request, the generated manifest on the card, the PR the bot opens, and the confirmation it posts once CI provisions the change.

Add a user

Move a user to another location

Delete a user

Drift Detection & Nightly Reconcile

Best-effort provisioning has a trade-off: a license or extension assignment can fail with a warning rather than blocking the batch, which means a user can be present in Git yet have no working dial tone in Webex. To catch that, provision.py --check compares a manifest against live Webex read-only and flags the mismatches a plain audit misses — including “calling declared but no Professional license assigned.” It exits 0 in sync and 2 on drift, so it doubles as a CI gate.

For continuous reconcile, the workflow also runs nightly (03:00 UTC) via reconcile.py, re-asserting every manifest — so a manual change made directly in Control Hub drifts back into line, ArgoCD/Flux-style. Each nightly run writes a dated Markdown report (reports/reconcile-YYYY-MM-DD.md) and commits it back, giving a versioned record of every reconcile. A workflow concurrency group serializes runs so overlapping merges can’t race on Webex writes.

Least-Privilege by Design

Credentials are split by concern so no single token can do everything:

Purpose	Scope
Webex chat (bot)	Chat-only token, no admin scopes — can’t create or delete users
GitHub PRs	Fine-grained token, single repo (Contents + Pull requests)
Claude (agent)	API key, only on the bot host
Webex admin (CI)	`people_read` + `people_write`, stored as an Actions secret

The bot that talks to humans holds no power to change Webex; the pipeline that changes Webex never talks to humans. The only bridge between them is a reviewed pull request.

Lessons Learned

Treat telephony like infrastructure. Declarative YAML + Git review + a reconcile loop turns “who changed this extension?” from a mystery into a git log.
Verify effects, not acknowledgements. A Webex job can report success while doing nothing — inspect the per-item result.
Idempotency is what makes reconcile safe. Look users up by email and update in place; then re-running the whole set nightly is a feature, not a risk.
Put the LLM at the edge, not in the control path. The agent only drafts a manifest; a human merge still provisions. Natural language lowers the barrier without handing the keys to a model.
Separate credentials by blast radius. Chat token, PR token, admin token, LLM key — four keys, four concerns, no god-token.

Same GitOps muscle memory from the Hugo blog pipeline, pointed at Cisco Webex instead of a container image. Declare it, review it, let the robots reconcile it.