AWS X-Ray Enterprise Guide: Distributed Tracing, Service Maps, and Secure Observability on AWS

Overview

AWS X-Ray is a managed distributed tracing service for applications running on AWS, hybrid infrastructure, and container platforms. It collects trace segments and subsegments from instrumented services, then builds end-to-end request views, service maps, and latency breakdowns that help teams understand application behavior under real production load.

Enterprises use AWS X-Ray to reduce mean time to resolution, validate microservice dependencies, and correlate failures across API Gateway, Lambda, ECS, EKS, EC2, and downstream AWS services. It is especially useful where logs alone cannot show causal relationships between services or where teams need sampling-based observability with low operational overhead.

Architecture

Core components

• X-Ray SDK: Instruments application code and creates segments and subsegments.
• X-Ray daemon or ADOT collector: Receives trace data locally and forwards it to AWS.
• X-Ray service: Stores traces, generates service maps, and supports filtering and analytics.
• IAM: Controls who can write telemetry and who can read traces.
• CloudWatch and CloudTrail: Complement X-Ray with metrics, logs, and audit events.

Deployment models

• Lambda-native tracing: Active tracing enabled on Lambda and API Gateway.
• EC2 or on-premises: X-Ray daemon runs as a service or container sidecar.
• ECS/EKS: Sidecar daemon or AWS Distro for OpenTelemetry collector.
• Hybrid observability: Applications emit traces from private data centers over AWS endpoints.

Data flow

1. A client request enters API Gateway, ALB, or an application endpoint.
2. The instrumented service creates a segment and propagates the trace header.
3. Downstream services create subsegments or child segments.
4. The daemon batches UDP/TCP traffic and sends it to the X-Ray API.
5. X-Ray processes traces into service maps, latency histograms, and error views.

Implementation Guide

1. Create an IAM policy for writers:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "xray:PutTraceSegments",
        "xray:PutTelemetryRecords"
      ],
      "Resource": "*"
    }
  ]
}

1. Attach the policy to an EC2 instance profile, ECS task role, or EKS service account.
2. Install the daemon on Amazon Linux 2:

sudo rpm -Uvh https://s3.dualstack.us-east-1.amazonaws.com/aws-xray-assets.us-east-1/xray-daemon/aws-xray-daemon-3.x.rpm
sudo systemctl enable xray
sudo systemctl start xray
sudo systemctl status xray

1. Configure the daemon at /etc/amazon/xray/cfg.yaml:

TotalBufferSizeMB: 64
Concurrency: 16
Region: us-east-1
Socket:
  UDPAddress: 0.0.0.0:2000
Logging:
  LogRotation: daily
  LogLevel: info
LocalMode: false

1. Instrument a Python service:

pip install aws-xray-sdk flask
export AWS_XRAY_TRACING_NAME=orders-api
export AWS_XRAY_DAEMON_ADDRESS=127.0.0.1:2000

1. For Lambda, enable active tracing:

aws lambda update-function-configuration --function-name orders-handler --tracing-config Mode=Active

1. Validate ingestion:

aws xray get-service-graph --start-time 2026-10-28T09:00:00Z --end-time 2026-10-28T10:00:00Z

Code Examples

1. Install and verify daemon

sudo systemctl restart xray
tail -f /var/log/xray/xray.log
ss -ulpn | grep 2000

2. ECS sidecar definition

xray-daemon:
  image: public.ecr.aws/xray/aws-xray-daemon:3.3.3
  cpu: 32
  memoryReservation: 256
  portMappings:
    - containerPort: 2000
      protocol: udp
  environment:
    - name: AWS_REGION
      value: us-east-1

3. Python Flask instrumentation

from flask import Flask
from aws_xray_sdk.core import xray_recorder, patch_all
from aws_xray_sdk.ext.flask.middleware import XRayMiddleware

patch_all()
xray_recorder.configure(service='orders-api', sampling=True)
app = Flask(__name__)
XRayMiddleware(app, xray_recorder)

@app.route('/health')
def health():
    subsegment = xray_recorder.begin_subsegment('health-check')
    subsegment.put_annotation('component', 'api')
    xray_recorder.end_subsegment()
    return {'status': 'ok'}

Security Hardening

• Use least-privilege IAM: writers need xray:PutTraceSegments and xray:PutTelemetryRecords; readers should be scoped to operational roles.
• Encrypt related logs in CloudWatch with KMS and restrict access to trace-linked log groups.
• Avoid storing secrets, tokens, or regulated personal data in annotations or metadata.
• Use VPC endpoints or controlled egress where required by compliance policy.
• Enable CloudTrail to audit changes to tracing configuration and IAM permissions.
• Standardize sampling rules to reduce unnecessary exposure of sensitive transaction context.

Comparison

Troubleshooting

Error 1: daemon cannot send segments

Log sample:

2026-10-28T09:14:22Z [ERROR] Send batch failed with: RequestError: send request failed caused by: Post "https://xray.us-east-1.amazonaws.com/TraceSegments": dial tcp 52.94.228.248:443: i/o timeout

Fix: verify outbound network access, proxy settings, DNS resolution, and VPC endpoint routing.

Error 2: missing IAM permissions

Log sample:

2026-10-28T09:17:03Z [ERROR] Failed to send telemetry records: AccessDeniedException: User: arn:aws:sts::123456789012:assumed-role/app-role/i-0123456789 is not authorized to perform: xray:PutTelemetryRecords

Fix: add xray:PutTelemetryRecords and xray:PutTraceSegments to the execution role.

Error 3: no traces visible from app

Log sample:

SegmentNotFoundException: cannot find the current segment/subsegment, please make sure you have a segment open

Fix: initialize middleware early, confirm incoming trace headers, and avoid creating subsegments outside request context.

Best Practices

Do

• Trace critical paths such as checkout, authentication, and payment flows.
• Use annotations for indexed fields like tenantId or env=prod to support filtering.
• Correlate with logs by injecting trace IDs into structured application logs.
• Adopt ADOT if you need OpenTelemetry portability across AWS and non-AWS systems.

Don't

• Do not capture secrets in metadata, request bodies, or SQL statements.
• Do not oversample high-volume endpoints; for example, sample 1-5% of health checks, not 100%.
• Do not grant broad read access to traces in regulated environments.
• Do not rely on tracing alone; pair X-Ray with CloudWatch metrics, alarms, and centralized logs.