Advanced Traceroute Techniques for Network EngineersTraceroute remains a cornerstone utility for diagnosing network path and latency issues. For network engineers working in complex, multi-domain environments, basic traceroute output often isn’t enough. This article digs into advanced techniques, tools, and analysis methods that turn traceroute from a simple hop list into actionable insight for performance tuning, capacity planning, and troubleshooting.
When basic traceroute falls short
Basic traceroute (ICMP/UDP-based on many systems, or UDP/tcp/ICMP depending on options) provides hop-by-hop Round-Trip Time (RTT) and intermediate IP addresses. But there are frequent limitations:
- Asymmetric routing can show only one path direction.
- Intermediate devices may deprioritize or block TTL-expired packets, causing missing or delayed responses.
- MPLS, VPNs, and load balancers can hide real topology.
- Per-flow load balancing (ECMP) may split packets across multiple paths, producing misleading hop sequences.
- ICMP rate limiting at hops skews latency measurements.
Advanced techniques target these issues to reveal true topology and root causes.
Use the right probe type and ports
Different devices treat ICMP, UDP, and TCP differently. Experimenting with probe types helps evade filtering and reflect the behavior of actual traffic.
- ICMP probes are often filtered or deprioritized on routers but useful when hosts respond to ICMP.
- UDP (classic traceroute) may reach destinations that expect UDP traffic, but many networks block high-numbered UDP ports.
- TCP probes (e.g., to destination port ⁄443) are often most representative for troubleshooting application connectivity and traverse firewalls that allow web traffic.
Tools: traceroute (Linux) supports -I (ICMP), -T (TCP) on modern versions; tcptraceroute; tracepath; Windows tracert is ICMP-only; MTR supports different probe types.
Example: use TCP-SYN to port 443 to emulate HTTPS flow:
tcptraceroute example.com 443 Control packet timing and retry behavior
Default traceroute timing can mask transient issues. Adjust probes per hop, timeouts, and intervals:
- Increase probe count per hop (e.g., 3→5 or more) to expose variability.
- Reduce inter-probe interval to detect short-lived congestion bursts.
- Raise timeout to capture slow replies from overloaded devices.
MTR is particularly useful for continuous sampling over time to spot intermittent packet loss and jitter.
Correlate with flow-based measurements (NetFlow/IPFIX/sFlow)
Traceroute gives topology and RTT; flow telemetry provides volumetric context. When traceroute shows high latency at a hop, check flow records to see if that interface is experiencing high utilization, heavy flows, or microbursts.
Workflow:
- Run traceroute toward the affected prefix.
- Identify suspected egress interface or AS hop.
- Query NetFlow/IPFIX/sFlow for top talkers, flows, and timestamps matching traceroute observations.
- Correlate with interface counters and queue drops.
Handling ECMP and path churn
Per-flow load balancing (ECMP) can cause traceroute to alternate paths, producing multiple different hop sequences. To address:
- Use a fixed 5-tuple for probes (source/dest IP, source/dest port, protocol). TCP probes with fixed ports are effective.
- Increase probe counts and map multiple observed paths — treat traceroute output as path ensemble rather than single path.
- Use Paris-traceroute (or modern traceroute implementations using Paris technique) to maintain consistent flow hashing and reveal actual per-flow path.
Commands:
paris-traceroute -P tcp -p 443 example.com Reveal MPLS, tunnels, and hidden hops
MPLS and various tunneling technologies can hide true hop counts. Techniques:
- Look for MPLS label stack entries in traceroute output (some routers return MPLS label info).
- Use TTL-limited probes beyond destination to see intermediate decapsulation points when possible.
- Combine traceroute with MPLS-aware tools (e.g., vendor-specific commands or SNMP queries for MPLS LSP state).
- Check for sudden large RTT jumps that suggest an encapsulating/decapsulation event.
Path MTU discovery and fragmentation-aware tracing
ICMP fragmentation-needed messages are often filtered; fragmentation issues can cause application problems even when traceroute looks fine.
- Use tracepath (Linux) or tracepath6 to test path MTU progressively without relying on ICMP FragNeeded from middleboxes.
- Use hping3 with varying packet sizes and DF bit to determine the packet size that provokes fragmentation:
hping3 -S -p 443 -d 1400 --setdf example.com
Use IPv6 traceroute best practices
IPv6 networks often differ in filtering and ICMPv6 handling. Use ICMPv6 probes when appropriate and be mindful that some middleboxes will treat ICMPv6 differently. Use tools that support flow-label handling and check for NDP-related path issues.
Automated analysis and visualization
Large-scale environments benefit from automated parsing and visualization:
- Store traceroute samples in a database (timestamps, probe type, hop IPs, RTTs).
- Use graph visualization (Graphviz, Gephi) to show path changes, multiple observed paths, and AS-level mapping.
- Heatmaps for per-hop RTT and packet loss help prioritize investigation.
- Example processing pipeline: scheduled probes → parse JSON output → enrich with GeoIP/ASN → visualize.
AS-level and inter-domain troubleshooting
When traceroute crosses AS boundaries and shows issues:
- Map IPs to ASNs (using RIR/WHOIS or local BGP view) to identify which operator controls the problematic hop.
- Check BGP paths and updates around the incident time — route flaps can cause transient path changes and packet drops.
- Use looking glasses and RIPE Atlas anchors/clients for cross-domain validation from other vantage points.
Using distributed measurement platforms
Leverage public and commercial measurement platforms for broader visibility:
- RIPE Atlas, CAIDA Ark, and Looking Glasses let you run traceroutes from many networks worldwide to compare paths and confirm whether an issue is local or global.
- Commercial NPMD tools offer scheduled, multi-vantage traceroute and integrated alerts.
Interpreting tricky symptoms
- Repeated asterisks (*) at one hop but normal later hops: often ICMP rate limiting or ACL drops at that hop — check subsequent hop RTT and path continuity.
- Early high RTT that persists: likely congestion/queuing at that hop — correlate with interface counters.
- High variance across probes at a single hop: intermittent queuing or CPU load on that router.
- Different final IPs across probes to the same hostname: DNS round-robin or load balancer behavior — use TCP probe to the application port to see actual service path.
Practical checklist for a traceroute investigation
- Choose probe type (TCP to app port if possible).
- Increase probe samples and adjust timeouts.
- Use Paris-traceroute to avoid ECMP artifacts.
- Correlate with flow telemetry and interface stats.
- Map problematic hops to ASNs and vendors.
- Validate from alternate vantage points (RIPE Atlas / looking glasses).
- Visualize results and store for trend analysis.
Example advanced traceroute commands
- Paris-traceroute TCP to 443:
paris-traceroute -P tcp -p 443 example.com - tcptraceroute:
tcptraceroute -n example.com 443 - MTR with TCP probes:
mtr --tcp --port 443 --report example.com - Tracepath for PMTU:
tracepath example.com
Limitations and ethical considerations
Traceroute is a diagnostic tool — avoid excessive probing that could be interpreted as scanning or DOS by remote operators. Respect organizational policies and rate limits.
Advanced traceroute is about using the right probe, sampling enough, correlating with telemetry, and validating across vantage points. With these techniques you can turn noisy hop lists into precise, actionable evidence for network performance and routing problems.
Leave a Reply