The Problem: Selective Network Outages Over Tailscale
When users reported that database access through a Tailscale exit node was failing while RDP connections remained functional, the initial symptoms suggested a routing or firewall issue. However, the real culprit was much more subtle: DNS resolution was being silently dropped at the exit node level.
Why This Setup Exists at All: Azure Private Link Resolves Differently Depending on Where You’re Standing
Before getting into the dnsmasq “bug” itself, it’s worth explaining why this infrastructure exists in the first place, because the root problem isn’t Tailscale or dnsmasq, it’s that Azure Private Link’s DNS behavior is context-dependent, and a VPN mesh network doesn’t naturally live in any single context.
Azure SQL (and most PaaS services behind Private Link) publish a CNAME chain, not a static A record. Querying <server>.database.windows.net from anywhere on the public internet returns something like:
<server>.database.windows.net → CNAME → <server>.privatelink.database.windows.net
<server>.privatelink.database.windows.net → CNAME → <redirect>.<region>.database.windows.net
<redirect>.<region>.database.windows.net → CNAME → <trafficmanager>.trafficmanager.net
<trafficmanager>.trafficmanager.net → A → <public gateway IP>
That public gateway IP is a shared, multi-tenant TDS redirect endpoint. It works, but it means your traffic leaves the VNet, hits a public Microsoft frontend, and gets redirected back in — the “TDS redirect” problem, with the associated latency and firewall surface area.
Inside a VNet that has the correct Private DNS Zone linked (privatelink.database.windows.net), the second hop in that chain resolves differently. Instead of continuing to the public trafficmanager record, it terminates at a private A record auto-created by Azure when the Private Endpoint is provisioned, pointing straight at the private endpoint’s NIC address inside your VNet (e.g., 10.5.0.4). Same hostname, same client, completely different answer, purely a function of which DNS resolver context answered the query.
The DNS record matters for a second, separate reason beyond just reachability: Azure SQL inspects the server name presented in the TDS pre-login packet, the FQDN the client actually asked to connect to, and uses it for redirect/SNI-style routing. Connect by raw IP instead of the FQDN and the gateway rejects the login outright with a “Cannot open server” error, even if the IP is fully reachable and correct. So the client always has to connect using the hostname; what changes based on DNS context isn’t whether the hostname is used, only which IP that hostname resolves to. It is worth mentioning this, because it means switching where the private answer comes from (dnsmasq vs. a different resolver path) has no bearing on this constraint — the client-side connection string never changes either way, and it is not simply a networking solution.
This is the actual constraint that makes split DNS non-optional here: a client sitting on a laptop in a coffee shop or remotely at the worker’s house has no way to be “inside the VNet” for DNS purposes. It needs to ask a resolver that has that VNet context on its behalf, get the private answer back, and then route the resulting private IP over the tailnet to a node that can actually reach it. That’s the whole reason for standing up a subnet router with a forwarder in front of Azure’s 168.63.129.16 metadata resolver in the first place, there’s no way to get a client-side resolver to “just know” the private answer without something in the loop that’s actually topologically inside the VNet.
Has Anyone Solved This Before? Is dnsmasq the Right Tool?
Before settling on dnsmasq as the forwarder, it’s worth asking whether this is a solved problem elsewhere, since running a second DNS daemon on infrastructure you also have to keep patched, monitored, and surviving reboots is its own maintenance cost. Later in this post I’ll detail where I think dnsmasq failed me, it’s a stable tool but integrating anything has its downsides you can’t foresee.
Tailscale’s own Split DNS feature handles routing which domains go where — that part is native and doesn’t need a workaround. What it doesn’t do is resolve those domains against a VNet-scoped answer; it just forwards the query verbatim to whatever nameserver IP you configure as the split-DNS destination. Something on the receiving end still has to actually be able to produce the private answer, which means something in that VNet, using Azure’s 168.63.129.16 resolver (which itself only answers with private-zone data when the querying source sits in a VNet with the matching Private DNS Zone linked).
That leaves a genuine question: does the “something” doing that resolution need to be a general-purpose forwarder like dnsmasq, or can 168.63.129.16 be pointed to directly as the split-DNS destination, skipping the extra daemon entirely? On paper this is appealing — fewer moving parts, no --local-service startup race to fight, one less package to patch and monitor.
I opened a support ticket with Tailscale to ask this directly, whether there’s a supported way to do domain-scoped conditional forwarding to a private upstream resolver without a self-managed forwarder in front of it, and whether the exit-node-vs-subnet-router distinction changes the answer (their first response leaned on exit-node DNS semantics, which don’t apply to a subnet-router-only setup like this one, since the two features have materially different DNS interaction models).
In parallel, rather than wait on that, I tested it directly. 168.63.129.16 isn’t a normal routable address — it’s a per-VNet special-cased endpoint that Azure documents as static and permanent, it has its own callout in Azure docs, so I’d presume it’s very stable, but whether it would answer correctly for traffic arriving via a routed path through a subnet router, rather than traffic originating directly from a NIC physically inside the VNet was an open question. I temporarily advertised 168.63.129.16/32 as a route through the exit node, approved it, and queried it directly from a client, bypassing dnsmasq entirely:
dig @168.63.129.16 prod.database.windows.net
Off Tailscale (querying the public anycast address directly), the answer falls through the full public CNAME chain to a 20.40.x.x trafficmanager IP — expected, since the querying source there has no VNet context at all. On Tailscale, routed through the subnet router, the same query terminates cleanly at the private A record, 10.5.0.4 — the correct private answer, same as dnsmasq has been returning. Confirmed again running the identical query directly from the exit node’s own shell, for a same-box baseline. All three results were consistent with the theory: the deciding factor is which VNet the querying source is associated with, and a subnet-router-forwarded packet from a Tailscale client is treated as VNet-sourced correctly.
That’s one successful test, from one client, at one point in time — not a production-grade validation. Before decommissioning dnsmasq, I’m running this in parallel for a burn-in period and keeping dnsmasq as a fallback, rather than cutting over fully on the strength of a single manual query.
Identifying the Root Cause of my outage woes
The breakthrough came from examining system logs on the exit node. The dnsmasq daemon was logging repeated messages indicating it was rejecting DNS queries from Tailscale IP addresses:
Jul 16 19:32:15 cus-vm-tailscale-exit-node-1 dnsmasq[302922]: ignoring query from non-local network 100.119.38.25
Jul 16 19:32:15 cus-vm-tailscale-exit-node-1 dnsmasq[302923]: ignoring query from non-local network 100.119.38.25
Jul 16 19:48:04 cus-vm-tailscale-exit-node-1 dnsmasq[302988]: ignoring query from non-local network 100.118.220.47
This was the issue: dnsmasq was configured to only accept DNS queries from local network interfaces, rejecting queries from Tailscale’s virtual network (100.x.x.x addresses). Users could establish TCP connections (like RDP) directly because those bypass DNS entirely, but any service requiring hostname resolution (databases, APIs, etc.) would timeout.
Understanding dnsmasq’s Listen Interface Restrictions
By default, dnsmasq restricts DNS queries to localhost or specific physical interfaces to prevent it from being used as an open resolver. However, this legitimate security measure becomes problematic when you want an exit node to serve DNS for a VPN network. It’s a great default for VMs that might be public, but not for an ephemeral VM that starts up and eventually creates the tailscale0 interface.
The configuration issue typically manifests in one of two places:
- Interface binding in dnsmasq.conf:
# Restrictive configuration (the problem)
interface=eth0
interface=lo
# Does not include Tailscale interfaces
- Listen address restrictions:
# Only listening on specific addresses
listen-address=127.0.0.1
listen-address=192.168.1.1
# Tailscale IPs not included
In this case it was neither of those, it was the Ubuntu-packaged --local-service default, which is a separate, less commonly documented restriction layered on top: it doesn’t check config-file interface/listen-address directives, it scans live interface addresses at dnsmasq’s own startup and only treats a source IP as “local” if it falls within a subnet dnsmasq saw attached at that moment. That distinction matters, because it’s the reason the fix below isn’t just “add the interface to a config file and restart” — it’s a startup-ordering problem, not a static configuration gap.
The Solution: Configure dnsmasq for Tailscale Networks
Step 1: Identify Your Tailscale Interface
First, determine which network interface Tailscale is using:
ip addr show | grep -E "(tailscale|wg0|100\.)" # Depending on your setup
ip link show
You can also check Tailscale’s configuration:
tailscale status
tailscale ip -4 # View your Tailscale IPv4
Step 2: Update dnsmasq Configuration
Modify /etc/dnsmasq.conf (or create /etc/dnsmasq.d/tailscale.conf) to accept queries on the Tailscale interface:
# Allow DNS queries from Tailscale clients
interface=lo
interface=eth0
interface=tailscale0 # Add your Tailscale interface
# Alternatively, listen on all interfaces and restrict by source
listening-address=127.0.0.1
listening-address=0.0.0.0 # Listen on all interfaces (more permissive)
# Forward requests appropriately
server=8.8.8.8
server=8.8.4.4
# Cache DNS responses locally
cache-size=1000
Step 3: Restart dnsmasq
Apply the configuration changes:
sudo systemctl restart dnsmasq
Step 4: Verify the Fix
Test DNS resolution from a Tailscale client:
# From a client on the Tailscale network
nslookup google.com <exit-node-tailscale-ip>
dig @<exit-node-tailscale-ip> example.com
Monitor the dnsmasq logs for rejection messages:
sudo journalctl -u dnsmasq -f # Follow logs in real-time
sudo tail -f /var/log/syslog | grep dnsmasq
After the fix, rejection messages should cease and normal DNS responses should appear in the logs.
The Part That Actually Matters: Why “Add the Interface Directive” Wasn’t Enough
Adding interface=tailscale0 to the config and restarting dnsmasq once will make it work — right up until the next reboot. That’s the trap. On a manually-managed box you’d never notice, because you set it up once and never touched it again. On infrastructure provisioned by Terraform + cloud-init, the box gets rebuilt, redeployed, or rebooted by automation on a schedule you don’t control, and the fix has to survive that without a human re-running commands.
The actual failure mode: dnsmasq starts at boot via systemctl enable --now dnsmasq, early in the provisioning sequence. tailscaled also starts around the same time, but the tailscale0 interface doesn’t get its 100.x.x.x/32 address until tailscale up completes — which requires an OAuth token exchange and device registration over the network, and in this setup runs later in the same cloud-init script, well after dnsmasq is already live. If dnsmasq’s interface scan happens before that address exists, it never recognizes tailscale0 as local network for the remaining lifetime of that process. A plain systemctl restart dnsmasq, run any time after the interface exists, fixes it immediately with zero config change, which is itself the diagnostic tell: if restarting alone fixes it, the config was never wrong, the timing was. Of course, on a code-defined ephemeral type instance: this doesn’t work.
The two tempting fixes turned out to be wrong or incomplete:
After=tailscaled.service/Wants=tailscaled.servicesystemd ordering. This only orders dnsmasq’s start againsttailscaled.servicebecoming active — meaning the daemon process exists. It says nothing about whethertailscale uphas finished authenticating and the interface has an actual address. Ordering against the wrong milestone just moves the race, it doesn’t close it.- Stripping
--local-servicefrom the packaging script (sed -i 's/ --local-service//' /usr/share/dnsmasq/init-system-common). This was the original approach in the cloud-init config and never actually took effect — confirmed viaps auxshowing the flag present on the live process well after the “fix” had supposedly applied. Turned out to be moot anyway: once startup ordering is correct,--local-servicecorrectly recognizestailscale0on its own. The flag was never the real defect, just a debugging distraction.
The fix that actually holds up: don’t rely on systemd unit ordering to approximate “after tailscale up succeeds.” Make it literal, imperative, sequential shell logic in the same provisioning script that runs tailscale up in the first place:
tailscale up \
--authkey='${oauth_secret}' \
--advertise-routes=${advertise_routes} \
--advertise-tags=tag:exit-node \
--accept-dns=false \
--hostname='${hostname}'
echo "=== tailscale up exit code: $? ==="
systemctl restart dnsmasq
tailscale status
One line added, systemctl restart dnsmasq, placed after tailscale up returns and before the script exits. This guarantees dnsmasq only ever (re)starts once the tailnet interface is confirmed up with an address, regardless of whatever order systemd happened to pick at boot. It survives reboots, survives full VM rebuilds from the same cloud-init template, and doesn’t add any new units, drop-ins, or dependency chains that could themselves race or silently fail to apply — the exact failure mode that made the original sed-based fix invisible until a real outage surfaced it.
But again, I didn’t like how fragile-seeming this was so I asked Tailscale for any ideas, and simultaneously arrived at the 168.63.129.16 advertisement discovery described above.
Prevention: Exit Node Deployment Checklist
When deploying Tailscale exit nodes or subnet routers with a DNS forwarder in front of them, ensure your DNS configuration includes:
- Exit node’s primary network interface(s) configured
- Tailscale interface explicitly added to dnsmasq listeners
- Upstream DNS servers specified (Google, Cloudflare, or internal resolvers)
- DNS query logging enabled initially for debugging
- Firewall rules updated to permit DNS (UDP/TCP port 53) if applicable
- Test DNS resolution from multiple Tailscale IPs before production use
- DNS forwarder restart is sequenced after
tailscale upsucceeds, not just aftertailscaled.servicestarts, verify this by rebooting the box and testing from a real peer client, not from loopback on the box itself - Confirm the fix survives a full redeploy from IaC, not just a live-patched running instance — a manual fix on a running VM proves the mechanism works, it doesn’t prove the provisioning script produces it
- If relying on Azure Private DNS Zones for the private answers, confirm the zone is actually linked to the VNet your forwarder/router lives in — a correctly configured forwarder pointed at
168.63.129.16still returns public answers if that link is missing, which looks identical to a DNS forwarding bug from the client’s side
Why This Matters
This subtle configuration issue demonstrates why exit nodes require careful attention to network services. While the connection itself works fine, breaking DNS silently means:
- Users see timeout errors rather than clear DNS failures
- Troubleshooting points toward network/firewall issues rather than DNS
- Database connections, API calls, and hostname-based services all fail mysteriously
- A fix verified by hand on a live box can still be invisible to the automation that’s supposed to reproduce it — the boot-order race in particular only shows up after a reboot, so a fix that “works” in the same session it was applied hasn’t actually been tested
By understanding how DNS restrictions interact with VPN networks, and how Azure Private Link’s answers depend on the resolver’s own network context rather than just the hostname being queried, you can prevent similar issues in your own infrastructure and debug faster when they do occur.