JIT elevation requests time out or never get approved: trace the workflow

TL;DR — When a just-in-time elevation request never gets granted, the break is usually in one of five places: the request never reaches the approval service, the approval notification never reaches the approver, the policy engine rejects it, the worker/queue is stalled, or the granted token/certificate cannot be applied on the target host. Start with the request ID in app logs, then trace HTTP callback/redirects, queue depth, policy decision logs, and target-host time skew; the single most common fix is restarting or unblocking the async worker that processes approval events. Reading time: ~6 min

The scenario

It is 14:20 on a Tuesday and you need temporary sudo on a production box to inspect a bad deploy. You submit an elevation request in the internal portal, the UI says "Pending approval", and your approver swears they clicked Approve ten minutes ago. The request eventually flips to "Timed out" or just stays pending forever, while the PAM dashboard still shows the service as healthy. You need to know whether the request died in the browser-to-app flow, the approval callback, the queue worker, the policy engine, or on the target host where the grant should be applied.

Symptoms

• UI stays on Pending approval until expiry.
• Request status changes to Timed out, Expired, or Approval window elapsed.
• Approver sees a link that opens an error page or spins forever.
• API returns 202 Accepted for request creation, but no later state transition occurs.
• Approval callback returns 401 Unauthorized, 403 Forbidden, 404 Not Found, or 410 Gone.
• Worker logs contain lines like:

ERROR approval-consumer failed to process message request_id=7f3d... err="context deadline exceeded"
WARN queue message age high queue=elevation-approvals oldest=1842s
ERROR policy decision deny request_id=7f3d... reason="missing approver group membership"
ERROR target grant failed host=app-03 err="certificate not yet valid"

• Mail logs or webhook delivery logs show retries, bounces, or TLS failures.
• Target host audit logs show no sudoers update, no SSH cert principal match, or no session token issuance.
• curl against approval/callback endpoints shows redirect loops or wrong hostnames:

$ curl -I https://pam.example.com/approve/7f3d
HTTP/2 302
location: http://pam.example.com/login

$ curl -I http://pam.example.com/login
HTTP/1.1 301 Moved Permanently
Location: https://pam.example.com/login

Likely causes

Step-by-step diagnosis

1. Find the request ID and last known state.

   grep -R "elevation request" /var/log/pam /var/log/app 2>/dev/null | tail -n 50
   

   This is your problem if you can see creation (state=pending) but no later approved, denied, or granted event for the same request_id. If you have the ID, continue to step 2.

2. Trace the approval URL end to end from a shell.

   curl -I -L https://pam.example.com/approve/<request-id>
   

   This is your problem if you see a redirect loop, a downgrade to http://, a 401/403 before the approval page, or a 404/410 for a still-pending request. Jump to Fixes → Approval callback/redirect URL is wrong or blocked by auth/proxy.

3. Check whether the approval event is stuck in the queue.

   rabbitmqctl list_queues name messages_ready messages_unacknowledged consumers | grep -E 'elevation|approval|grant'
   

   This is your problem if messages_ready keeps growing, consumers is 0, or messages_unacknowledged is high and not moving. If you use another broker, the equivalent check is queue depth and consumer lag. Jump to Fixes → Async worker or queue backlog stalls state transitions.

4. Check worker logs for the request ID.

   grep -R "<request-id>" /var/log/pam /var/log/app /var/log/containers 2>/dev/null | tail -n 100
   

   This is your problem if you see context deadline exceeded, DB timeouts, broker reconnect loops, or repeated retries without success. Usually the same fix as the queue/worker section applies.

5. Check policy decision logs after approval.

   grep -i "<request-id>\|deny\|reason=" /var/log/pam/policy.log /var/log/app/policy*.log 2>/dev/null | tail -n 50
   

   This is your problem if approval exists but policy says deny, missing group, outside allowed window, host not in scope, or max duration exceeded. Jump to Fixes → Policy engine denies after approval due to group/rule mismatch.

6. Check notification delivery if the approver claims they never saw it.

   grep -i "<request-id>\|status=bounced\|deferred\|connect to .* timed out" /var/log/mail.log /var/log/maillog 2>/dev/null | tail -n 50
   

   This is your problem if you see SMTP deferrals, bounces, or webhook retries. If approval can also happen in-app, verify whether the in-app task exists before spending time on email. Jump to Fixes → Notification delivery to approver fails or arrives too late.

7. Check time sync on the app node and target host.

   chronyc tracking && ssh target-host 'date -u && chronyc tracking || timedatectl timesync-status'
   

   This is your problem if clocks differ by more than a few seconds and logs show token expired, not yet valid, or x509: certificate has expired or is not yet valid. Jump to Fixes → Time skew breaks short-lived tokens/certs on app or target host.

8. Check TTL and approval window settings versus observed latency.

   grep -R "approval_timeout\|request_ttl\|grant_ttl\|max_pending" /etc /opt/app 2>/dev/null
   

   This is your problem if approval timeout is, for example, 60s while mail/webhook delivery or queue lag is several minutes. Jump to Fixes → Request TTL/approval window is shorter than processing latency.

[Requester Browser] ---- 1. create request ----> [PAM API]
[PAM API] ---- 2. enqueue approval ----> [Queue]
[Worker] <---- 3. consume event ---- [Queue]
[Worker] ---- 4. notify approver ----> [Mail/Webhook/Chat]
[Approver Browser] ---- 5. approve link ----> [PAM API]
[PAM API] ---- 6. policy check ----> [Policy Engine]
[PAM API] ---- 7. issue grant ----> [Target Host]

Fixes

Async worker or queue backlog stalls state transitions

If the queue has messages piling up or no consumers, restore consumers first.

systemctl status pam-worker
systemctl restart pam-worker
journalctl -u pam-worker -n 100 --no-pager

If the worker is containerized:

kubectl get deploy,po -n pam
kubectl rollout restart deploy/pam-worker -n pam
kubectl logs deploy/pam-worker -n pam --tail=100 -f

If broker connectivity is the issue, verify credentials and reachability from the worker node:

nc -vz rabbitmq.internal 5672
grep -R "amqp://\|broker_url" /etc /opt/app 2>/dev/null

Verify it worked:

rabbitmqctl list_queues name messages_ready messages_unacknowledged consumers | grep -E 'elevation|approval|grant'

Queue depth should fall and the request should transition within one processing cycle.

Approval callback/redirect URL is wrong or blocked by auth/proxy

Common failures: wrong external base URL, proxy not forwarding X-Forwarded-Proto, or auth middleware protecting the callback path.

Check the app base URL and proxy config:

grep -R "base_url\|external_url\|public_url" /etc /opt/app 2>/dev/null
nginx -T 2>/dev/null | grep -nE "server_name|proxy_set_header X-Forwarded-Proto|location /approve|location /callback"

Typical nginx fix:

server {
  listen 443 ssl http2;
  server_name pam.example.com;

  location / {
    proxy_pass http://pam-api;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  }

  location /approve/ {
    proxy_pass http://pam-api;
    auth_request off;
  }
}

Reload safely:

nginx -t && systemctl reload nginx

Verify it worked:

curl -I -L https://pam.example.com/approve/<request-id>

You should end on 200 or the expected app login flow over https, not loop or downgrade.

Policy engine denies after approval due to group/rule mismatch

Fix the rule or the requester/approver group mapping. Typical sources are stale group sync, wrong host scope, or duration exceeding policy.

Inspect the effective rule inputs:

grep -i "<request-id>\|principal=\|groups=\|resource=\|duration=" /var/log/pam/policy.log 2>/dev/null | tail -n 50

Then update the policy config. Example YAML:

rules:
  - name: prod-shell
    requesters: ["devops", "backend-oncall"]
    approvers: ["sre-leads"]
    resources: ["prod-app-*", "prod-db-ro-*"]
    max_duration: "1h"
    require_ticket: true

Reload the policy service:

systemctl reload pam-policy || systemctl restart pam-api

Trade-off: broadening groups or resource globs fixes incidents fast but expands blast radius. Prefer adding the exact missing group/resource rather than prod-* everywhere.

Verify it worked:

grep -i "<request-id>\|decision=allow" /var/log/pam/policy.log 2>/dev/null | tail -n 5

Notification delivery to approver fails or arrives too late

If the approver never receives the request, fix SMTP/webhook delivery and provide an in-app fallback if available.

SMTP checks:

postqueue -p
grep -i "status=bounced\|deferred\|SASL\|TLS" /var/log/mail.log | tail -n 50
openssl s_client -starttls smtp -connect smtp.internal:587 -servername smtp.internal </dev/null

Common fixes are correcting relay credentials, sender domain, or firewall rules to the relay. For webhook-based approval notifications, test the endpoint directly:

curl -i -X POST https://hooks.example.internal/pam-test -H 'Content-Type: application/json' -d '{"ping":"ok"}'

Verify it worked:

grep -i "status=sent" /var/log/mail.log | tail -n 10

Or confirm the approver receives a fresh request within the normal delivery window.

Time skew breaks short-lived tokens/certs on app or target host

Short-lived grants fail hard when clocks drift.

Resync time on Linux hosts:

chronyc makestep
systemctl restart chronyd || systemctl restart systemd-timesyncd
chronyc tracking

If the target host validates SSH certs or JWTs, check both ends:

date -u
ssh target-host 'date -u'

Verify it worked:

ssh target-host 'date -u' && chronyc tracking

Offsets should be small, and retries should no longer show not yet valid/expired errors.

Request TTL/approval window is shorter than processing latency

Raise the pending/approval timeout above your worst-case delivery plus queue lag.

Example config:

{
  "approval_timeout": "15m",
  "request_ttl": "30m",
  "grant_ttl": "1h"
}

Apply and restart:

systemctl restart pam-api pam-worker

Trade-off: longer pending windows reduce false timeouts but increase the period during which stale requests can be approved. Pair this with explicit request expiry messaging and audit logs.

Verify it worked:

grep -R "approval_timeout\|request_ttl" /etc /opt/app 2>/dev/null

Then create a test request and confirm it survives expected queue/mail delays.

Prevention

• Add queue lag alerts for approval/grant pipelines.

  rabbitmqctl list_queues name messages_ready messages_unacknowledged consumers
  

  Alert when messages_ready > 50 for 5 minutes or consumers = 0 on approval queues.

• Add a synthetic approval-path probe that follows redirects and asserts final status.

  curl -fsSL -o /dev/null -w '%{http_code} %{url_effective}\n' https://pam.example.com/health/approval-link-test
  

  Run it every minute from outside the cluster and alert on non-200 or unexpected hostnames.

• Pin proxy headers in config and test them in CI.

  proxy_set_header Host $host;
  proxy_set_header X-Forwarded-Proto https;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  

  CI check:

  nginx -T | grep -q 'X-Forwarded-Proto https'
  

• Add a policy regression test for real requester/approver/resource tuples.

  {"requester":"alice","approver":"bob","resource":"prod-app-03","duration":"30m","expected":"allow"}
  

  Run the policy engine against fixtures on every rules change.

• Monitor NTP drift on app and target hosts.

  chronyc tracking | awk -F: '/Last offset|RMS offset/{print $1,$2}'
  

  Alert when absolute offset exceeds 2s on any host participating in grant issuance or validation.

• Log and expose request lifecycle timestamps: created_at, notified_at, approved_at, granted_at, expired_at. A single histogram of approved_at - created_at and granted_at - approved_at will tell you whether the problem is delivery, policy, or target-host application.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.