Private endpoint resolves publicly and times out: diagnosis and fixes

TL;DR — If a private endpoint resolves to a public IP, start by checking which resolver answered and whether your private DNS zone is actually linked to the network/VPC/VNet your client is on. The most common fix is to point the client at the intended internal resolver or correctly attach the private DNS zone so the hostname returns RFC1918/ULA addresses instead of a public record. Reading time: ~6 min

The scenario

You deploy a service that should only be reachable over a private endpoint. From your laptop on VPN, curl to the service hostname just hangs and eventually times out; from a cloud VM in the same environment, it sometimes works, sometimes not. dig shows the hostname resolving to a public address, even though the endpoint was created as private and the app team swears the DNS zone is in place. It is 3:40 PM, the deploy window closes in an hour, and someone has already “fixed” it once by hardcoding /etc/hosts.

Symptoms

• curl hangs until timeout:

$ curl -I --connect-timeout 5 https://api.internal.example.com
curl: (28) Failed to connect to api.internal.example.com port 443 after 5001 ms: Timeout was reached

• DNS returns a public IP instead of RFC1918/ULA/private space:

$ dig +short api.internal.example.com
203.0.113.42

• Different answers from different resolvers:

$ dig +short api.internal.example.com @1.1.1.1
203.0.113.42
$ dig +short api.internal.example.com @10.0.0.2
10.42.3.17

• nslookup or resolvectl shows the wrong DNS server in use:

$ resolvectl status | sed -n '/Current DNS Server/,+4p'
Current DNS Server: 192.168.1.1
       DNS Servers: 192.168.1.1
        DNS Domain: ~.

• Traceroute goes toward the internet edge instead of staying internal:

$ traceroute -T -p 443 api.internal.example.com
 1  192.168.1.1
 2  198.51.100.1
 3  * * *

• TLS/SNI mismatch when the public endpoint answers but is not your private service:

$ openssl s_client -connect api.internal.example.com:443 -servername api.internal.example.com </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer
subject=CN = public-lb.example.net
issuer=CN = R3

• Application logs show connect timeouts rather than auth failures:

dial tcp 203.0.113.42:443: i/o timeout
connect ETIMEDOUT 203.0.113.42:443

Likely causes

| Private DNS zone/record exists but is not linked/associated to the client network | Very common | In your provider's dashboard: Private DNS/Hosted Zones → select zone → check linked VPC/VNet/network attachments |
| Split-horizon DNS missing: public record exists, private override does not | Common | ```bash
dig +short api.internal.example.com @<internal-dns-ip>
``` |
| DNS cache is stale on client, node, or local forwarder | Common | ```bash
dig +trace api.internal.example.com
``` |
| Private endpoint IP is correct, but route/firewall/NSG/SG blocks traffic to it | Common | ```bash
nc -vz -w3 <private-ip> 443
``` |
| CNAME chain points to a public hostname outside the private zone | Less common | ```bash
dig api.internal.example.com
``` |
| `/etc/hosts`, CoreDNS, or container DNS override is forcing the wrong answer | Less common | ```bash
getent hosts api.internal.example.com
``` |

## Step-by-step diagnosis

1. Check what IP the hostname resolves to from the failing client.
```bash
dig +short api.internal.example.com
getent hosts api.internal.example.com

If you get a public IP (for example 203.0.113.42) or getent disagrees with dig, this is your problem. Jump to Fixes → Wrong DNS resolver, Split-horizon DNS missing, or Hosts/container DNS override.

1. Check which DNS server the client is actually using.

resolvectl status
cat /etc/resolv.conf

On macOS:

scutil --dns | sed -n '/resolver #1/,/resolver #2/p'

If you see public resolvers (1.1.1.1, 8.8.8.8) or a local router instead of your corporate/VPN/internal resolver, jump to Fixes → Wrong DNS resolver.

1. Compare public vs internal resolver answers.

dig +short api.internal.example.com @1.1.1.1
dig +short api.internal.example.com @<internal-dns-ip>

If public returns a public IP and internal returns a private IP, DNS split-horizon is working but the client is not using it. Jump to Fixes → Wrong DNS resolver. If both return the public IP, jump to Fixes → Private DNS zone not linked or Split-horizon DNS missing.

1. Inspect the full record chain.

dig api.internal.example.com

What you are looking for:

;; ANSWER SECTION:
api.internal.example.com. 30 IN CNAME public-lb.example.net.
public-lb.example.net.    30 IN A     203.0.113.42

If the private name CNAMEs to a public hostname, jump to Fixes → CNAME chain points public.

1. Check whether the internal zone is attached to the right network. Use your provider dashboard or CLI to inspect the private zone and its network links. The exact UI varies, but the path is typically: DNS/Private DNS/Hosted Zones → select the zone → look for linked VPC/VNet/network attachments. If the client network is absent, this is your problem. Jump to Fixes → Private DNS zone not linked.

2. Verify whether the private IP itself is reachable.

nc -vz -w3 10.42.3.17 443
curl -vk --connect-timeout 5 https://10.42.3.17/ -H 'Host: api.internal.example.com'

If DNS is returning a private IP but nc times out, this is not a DNS problem anymore. Jump to Fixes → Route/firewall blocks private endpoint.

1. Eliminate cache and local overrides.

getent hosts api.internal.example.com
grep -n 'api.internal.example.com' /etc/hosts

If /etc/hosts returns the public IP, jump to Fixes → Hosts/container DNS override. If dig @<internal-dns-ip> is correct but plain dig is wrong, flush caches per Fixes → DNS cache stale.

Fixes

Wrong DNS resolver

Point the client at the internal resolver delivered by VPN/DHCP, or configure conditional forwarding for the private zone.

Linux with systemd-resolved on a VPN interface:

sudo resolvectl dns tun0 10.0.0.2 10.0.0.3
sudo resolvectl domain tun0 '~internal.example.com' '~example.private'
resolvectl flush-caches

Temporary one-off test without changing system DNS:

curl -I --resolve api.internal.example.com:443:10.42.3.17 https://api.internal.example.com

If you run a local DNS forwarder, add a conditional forward. Example dnsmasq:

server=/internal.example.com/10.0.0.2
server=/example.private/10.0.0.2

Then restart:

sudo systemctl restart dnsmasq

Verify it worked:

dig +short api.internal.example.com

Expected: private IP only.

Private DNS zone not linked

Attach the private DNS zone to the VPC/VNet/network where the client runs. In your provider dashboard, go to the private zone and add the client network under linked networks/associations. If you use IaC, define the association explicitly so it survives re-creation.

Generic Terraform pattern:

resource "example_private_dns_zone" "svc" {
  name = "internal.example.com"
}

resource "example_private_dns_zone_network_link" "app" {
  zone_id    = example_private_dns_zone.svc.id
  network_id = example_network.app.id
}

If the endpoint auto-created a record in a provider-managed zone, confirm your workload network is linked to that exact zone, not a similarly named one in another account/project/subscription.

Verify it worked:

dig +short api.internal.example.com @<internal-dns-ip>

Expected: private IP.

Split-horizon DNS missing

Create a private record that overrides the public one inside the internal zone.

Example BIND zone snippet:

$ORIGIN internal.example.com.
api 30 IN A 10.42.3.17

Example CoreDNS override:

internal.example.com:53 {
    hosts {
        10.42.3.17 api.internal.example.com
        fallthrough
    }
    forward . 10.0.0.2
}

If you already have a public A/CNAME, keep it if external clients need it; just add the private override in the internal view/zone.

Verify it worked:

dig +short api.internal.example.com @<internal-dns-ip>

Expected: private IP from the internal resolver, public IP only from public resolvers.

DNS cache stale

Flush caches on the client and any local forwarder.

Linux systemd-resolved:

sudo resolvectl flush-caches
sudo systemctl restart systemd-resolved

macOS:

sudo dscacheutil -flushcache
sudo killall -HUP mDNSResponder

nscd:

sudo nscd -i hosts

dnsmasq:

sudo systemctl restart dnsmasq

Kubernetes CoreDNS:

⚠️ Restarting CoreDNS can briefly impact name resolution cluster-wide.

kubectl -n kube-system rollout restart deployment coredns
kubectl -n kube-system rollout status deployment coredns

Verify it worked:

dig +short api.internal.example.com

Expected: new private answer.

Route/firewall blocks private endpoint

Open the path from the client subnet to the private endpoint IP on the required port, and confirm return traffic is allowed.

Quick checks:

ip route get 10.42.3.17
nc -vz -w3 10.42.3.17 443
traceroute -T -p 443 10.42.3.17

If ip route get picks the wrong interface or default route, add/repair the route. Example:

sudo ip route add 10.42.0.0/16 via 10.0.0.1 dev tun0

If a host firewall blocks egress:

sudo iptables -I OUTPUT -d 10.42.3.17/32 -p tcp --dport 443 -j ACCEPT

If the service is behind security groups/NSGs/firewall rules, allow the client subnet to the endpoint subnet on the target port in your provider dashboard or IaC.

Verify it worked:

nc -vz -w3 10.42.3.17 443

Expected: succeeded or open.

CNAME chain points public

Replace the public CNAME target with a private record or a CNAME that stays inside the private zone.

Bad:

api.internal.example.com. 30 IN CNAME public-lb.example.net.

Good:

api.internal.example.com. 30 IN A 10.42.3.17

Or:

api.internal.example.com. 30 IN CNAME svc-01.internal.example.com.
svc-01.internal.example.com. 30 IN A 10.42.3.17

If you need the same hostname internally and externally, use split-horizon views rather than a single public CNAME.

Verify it worked:

dig api.internal.example.com

Expected: no public target in the answer chain from the internal resolver.

Hosts/container DNS override

Remove the stale override from /etc/hosts, container config, or cluster DNS.

Host override:

sudo sed -i.bak '/api\.internal\.example\.com/d' /etc/hosts
getent hosts api.internal.example.com

Docker Compose explicit DNS override example to remove:

services:
  app:
    dns:
      - 8.8.8.8

Kubernetes pod with custom dnsPolicy/dnsConfig can bypass cluster DNS; inspect with:

kubectl get pod <pod> -o yaml | sed -n '/dnsPolicy/,+10p'

If CoreDNS has a stale hosts plugin entry, remove it and reload.

Verify it worked:

getent hosts api.internal.example.com

Expected: private IP from the intended resolver path.

Prevention

• Add a CI smoke test that fails if an internal hostname resolves publicly from a runner on the private network:

ip=$(dig +short api.internal.example.com | tail -n1)
python3 - <<'PY'
import ipaddress, os
ip = ipaddress.ip_address(os.environ['IP'])
assert ip.is_private, f'expected private IP, got {ip}'
PY

Run with IP="$ip".

• Pin conditional forwarding on developer VPN clients or office resolvers for private zones:

server=/internal.example.com/10.0.0.2
server=/example.private/10.0.0.2

• Monitor DNS answers from both public and internal vantage points. Example cron probe:

printf '%s public=%s internal=%s\n' "$(date -Is)" "$(dig +short api.internal.example.com @1.1.1.1 | tail -n1)" "$(dig +short api.internal.example.com @10.0.0.2 | tail -n1)"

Alert if the internal answer is empty or not private.

• Manage private DNS zone links in IaC, not by hand, and review them in code review:

resource "example_private_dns_zone_network_link" "app" {
  zone_id    = example_private_dns_zone.svc.id
  network_id = example_network.app.id
}

• Add a post-deploy connectivity test that checks both DNS and TCP reachability:

set -euo pipefail
host=api.internal.example.com
ip=$(dig +short "$host" | tail -n1)
nc -vz -w3 "$ip" 443
curl -skI --connect-timeout 5 --resolve "$host:443:$ip" "https://$host" | head -n1

• Ban ad-hoc /etc/hosts fixes in runbooks and shell history. If you need an emergency bypass, use curl --resolve for the single test instead of creating a persistent local override.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.