Description
Book SynopsisThe definitive guide to troubleshooting today’s complex BGP networks
This is today’s best single source for the techniques you need to troubleshoot BGP issues in modern Cisco IOS, IOS XR, and NxOS environments. BGP has expanded from being an Internet routing protocol and provides a scalable control plane for a variety of technologies, including MPLS VPNs and VXLAN. Bringing together content previously spread across multiple sources, Troubleshooting BGP describes BGP functions in today’s blended service provider and enterprise environments.
Two expert authors emphasize the BGP-related issues you’re most likely to encounter in real-world deployments, including problems that have caused massive network outages. They fully address convergence and scalability, as well as common concerns such as BGP slow peer, RT constraint filtering, and missing BGP routes. For each issue, key concepts are presented, along with basic configuration, detailed troubleshooting methods, and clear illustrations. Wherever appropriate, OS-specific behaviors are described and analyzed.
Troubleshooting BGP is an indispensable technical resource for all consultants, system/support engineers, and operations professionals working with BGP in even the largest, most complex environments.
· Quickly review the BGP protocol, configuration, and commonly used features
· Master generic troubleshooting methodologies that are relevant to BGP networks
· Troubleshoot BGP peering issues, flapping peers, and dynamic BGP peering
· Resolve issues related to BGP route installation, path selection, or route policies
· Avoid and fix convergence problems
· Address platform issues such as high CPU or memory usage
· Scale BGP using route reflectors, diverse paths, and other advanced features
· Solve problems with BGP edge architectures, multihoming, and load balancing
· Secure BGP inter-domain routing with RPKI
· Mitigate DDoS attacks with RTBH and BGP Flowspec
· Understand common BGP problems with MPLS Layer 3 or Layer 2 VPN services
· Troubleshoot IPv6 BGP for service providers, including 6PE and 6VPE
· Overcome problems with VXLAN BGP EVPN data center deployments
· Fully leverage BGP High Availability features, including GR, NSR, and BFD
· Use new BGP enhancements for link-state distribution or tunnel setup
This book is part of the Networking Technology Series from Cisco Press, which offers networking professionals valuable information for constructing efficient networks, understanding new technologies, and building successful careers.
Table of ContentsForeword xxii
Introduction xxiii
Part I BGP Fundamentals
Chapter 1 BGP Fundamentals 1
Border Gateway Protocol 1
Autonomous System Numbers 2
Path Attributes 3
Loop Prevention 3
Address Families 3
BGP Sessions 4
Inter-Router Communication 5
BGP Messages 6
OPEN 6
Hold Time 6
BGP Identifier 7
KEEPALIVE 7
UPDATE 7
NOTIFICATION Message 8
BGP Neighbor States 8
Idle 9
Connect 9
Active 10
OpenSent 10
OpenConfirm 10
Established 10
Basic BGP Configuration 11
IOS 11
IOS XR 12
NX-OS 13
Verification of BGP Sessions 14
Prefix Advertisement 17
BGP Best-Path Calculation 20
Route Filtering and Manipulation 21
IBGP 22
IBGP Full Mesh Requirement 24
Peering via Loopback Addresses 25
EBGP 26
EBGP and IBGP Topologies 28
Next-Hop Manipulation 30
IBGP Scalability 31
Route Reflectors 31
Loop Prevention in Route Reflectors 33
Out-of-Band Route Reflectors 33
Confederations 34
BGP Communities 37
Route Summarization 38
Aggregate-Address 39
Flexible Route Suppression 40
Selective Prefix Suppression 40
Leaking Suppressed Routes 40
Atomic Aggregate 40
Route Aggregation with AS_SET 42
Route Aggregation with Selective Advertisement of AS-SET 42
Default Route Advertisement 42
Default Route Advertisement per Neighbor 42
Remove Private AS 43
Allow AS 43
LocalAS 43
Summary 44
References 45
Part II Common BGP Troubleshooting
Chapter 2 Generic Troubleshooting Methodologies 47
Identifying the Problem 47
Understanding Variables 48
Reproducing the Problem 49
Setting Up the Lab 49
Configuring Lab Devices 52
Triggering Events 56
Sniffer-Packet Capture 57
SPAN on Cisco IOS 58
SPAN on Cisco IOS XR 60
SPAN on Cisco NX-OS 62
Remote SPAN 63
Platform-Specific Packet Capture Tools 65
Netdr Capture 66
Embedded Packet Capture 68
Ethanalyzer 70
Logging 74
Event Monitoring/Tracing 77
Summary 81
Reference 81
Chapter 3 Troubleshooting Peering Issues 83
BGP Peering Down Issues 83
Verifying Configuration 84
Verifying Reachability 87
Find the Location and Direction of Packet Loss 88
Verify Whether Packets Are Being Transmitted 89
Use Access Control Lists to Verify Whether Packets Are Received 90
Check ACLs and Firewalls in Path 91
Verify TCP Sessions 94
Simulate a BGP Session 95
Demystifying BGP Notifications 96
Decode BGP Messages 99
Troubleshoot Blocked Process in IOS XR 103
Verify BGP and BPM Process State 104
Verify Blocked Processes 105
Restarting a Process 106
BGP Traces in IOS XR 106
BGP Traces in NX-OS 108
Debugs for BGP 110
Troubleshooting IPv6 Peers 112
Case Study–Single Session Versus Multisession 113
Multisession Capability 114
Single-Session Capability 115
BGP Peer Flapping Issues 115
Bad BGP Update 115
Hold Timer Expired 116
Interface Issues 116
Physical Connectivity 117
Physical Interface 117
Input Hold Queue 117
TCP Receive Queue 119
MTU Mismatch Issues 120
High CPU Causing Control-Plane Flaps 125
Control Plane Policing 127
CoPP on NX-OS 129
Local Packet Transport Services 134
Dynamic BGP Peering 138
Dynamic BGP Peer Configuration 139
Dynamic BGP Challenges 142
Misconfigured MD5 Password 142
Resource Issues in a Scaled Environment 142
TCP Starvation 142
Summary 143
References 143
Chapter 4 Troubleshooting Route Advertisement and BGP Policies 145
Troubleshooting BGP Route Advertisement 145
Local Route Advertisement Issues 145
Route Aggregation Issues 147
Route Redistribution Issues 150
BGP Tables 152
Receiving and Viewing Routes 154
Troubleshooting Missing BGP Routes 156
Next-Hop Check Failures 157
Bad Network Design 160
Validity Check Failure 162
AS-Path 162
Originator-ID/Cluster-ID 165
BGP Communities 167
BGP Communities: No-Advertise 167
BGP Communities: No-Export 169
BGP Communities: Local-AS (No Export SubConfed) 170
Mandatory EBGP Route Policy for IOS XR 172
Filtering of Prefixes by Route Policy 173
Conditional Matching 174
Access Control Lists (ACL) 174
Prefix Matching 175
Regular Expressions (Regex) 177
UnderScore _ 179
Caret ^ 180
Dollar Sign $ 181
Brackets [ ] 181
Hyphen - 182
Caret in Brackets [^] 182
Parentheses ( ) and Pipe | 183
Period . 183
Plus Sign + 183
Question Mark ? 184
Asterisk * 184
Looking Glass and Route Servers 185
Conditionally Matching BGP Communities 185
Troubleshooting BGP Router Policies 185
IOS and NX-OS Prefix-Lists 186
IOS and NX-OS AS-Path ACLs 188
Route-Map Processing 191
IOS and NX-OS Route-Maps 192
IOS XR Route-Policy Language 196
Incomplete Configuration of Routing Policies 198
Conditional BGP Debugs 199
Summary 203
Further Reading 204
References in This Chapter 204
Chapter 5 Troubleshooting BGP Convergence 205
Understanding BGP Route Convergence 205
BGP Update Groups 207
BGP Update Generation 212
Troubleshooting Convergence Issues 216
Faster Detection of Failures 218
Jumbo MTU for Faster Convergence 219
Slow Convergence due to Periodic BGP Scan 219
Slow Convergence due to Default Route in RIB 222
BGP Next-Hop Tracking 223
Selective Next-Hop Tracking 225
Slow Convergence due to Advertisement Interval 226
Computing and Installing New Path 226
Troubleshooting BGP Convergence on IOS XR 227
Verifying Convergence During Initial Bring Up 227
Verifying BGP Reconvergence in Steady State Network 228
Troubleshooting BGP Convergence on NX-OS 234
BGP Slow Peer 237
BGP Slow Peer Symptoms 238
High CPU due to BGP Router Process 238
Traffic Black Hole and Missing Prefixes in BGP table 238
BGP Slow Peer Detection 239
Verifying OutQ value 240
Verifying SndWnd 240
Verifying Cache Size and Pending Replication Messages 241
Workaround 242
Changing Outbound Policy 242
Advertisement Interval 243
BGP Slow Peer Feature 245
Static Slow Peer 245
Dynamic Slow Peer Detection 245
Slow Peer Protection 246
Slow Peer Show Commands 246
Troubleshooting BGP Route Flapping 246
Summary 250
Reference 250
Part III BGP Scalability Issues
Chapter 6 Troubleshooting Platform Issues Due to BGP 251
Troubleshooting High CPU Utilization due to BGP 251
Troubleshooting High CPU due to BGP on Cisco IOS 252
High CPU due to BGP Scanner Process 253
High CPU due to BGP Router Process 255
High CPU Utilization due to BGP I/O Process 256
Troubleshooting High CPU due to BGP on IOS XR 258
Troubleshooting High CPU due to BGP on NX-OS 262
Capturing CPU History 265
Troubleshooting Sporadic High CPU Condition 265
Troubleshooting Memory Issues due to BGP 267
TCAM Memory 269
Troubleshooting Memory Issues on Cisco IOS Software 269
Troubleshooting Memory Issues on IOS XR 274
Troubleshooting Memory Issues on NX-OS 278
Restarting Process 281
Summary 281
References 282
Chapter 7 Scaling BGP 283
The Impact of Growing Internet Routing Tables 283
Scaling Internet Table on Various Cisco Platforms 285
Scaling BGP Functions 288
Tuning BGP Memory 290
Prefixes 290
Managing the Internet Routing Table 290
Paths 292
Attributes 293
Tuning BGP CPU 295
IOS Peer-Groups 295
IOS XR BGP Templates 295
NX-OS BGP Peer Templates 296
BGP Peer Templates on Cisco IOS 297
Soft Reconfiguration Inbound Versus Route Refresh 298
Dynamic Refresh Update Group 302
Enhanced Route Refresh Capability 305
Outbound Route Filtering (ORF) 309
Prefix-Based ORF 309
Extended Community—Based ORF 309
BGP ORF Format 310
BGP ORF Configuration Example 312
Maximum Prefixes 316
BGP Max AS 318
BGP Maximum Neighbors 322
Scaling BGP with Route Reflectors 322
BGP Route Reflector Clusters 324
Hierarchical Route Reflectors 331
Partitioned Route Reflectors 332
BGP Selective Route Download 339
Virtual Route Reflectors 342
BGP Diverse Path 346
Shadow Route Reflectors 349
Shadow Sessions 355
Route Servers 357
Summary 364
References 365
Chapter 8 Troubleshooting BGP Edge Architectures 367
BGP Multihoming and Multipath 367
Resiliency in Service Providers 370
EBGP and IBGP Multipath Configuration 370
EIBGP Multipath 372
R1 373
R2 374
R3 374
R4 375
R5 376
AS-Path Relax 377
Understanding BGP Path Selection 377
Routing Path Selection Longest Match 377
BGP Best-Path Overview 379
Weight 380
Local Preference 380
Locally Originated via Network or Aggregate Advertisement 380
Accumulated Interior Gateway Protocol (AIGP) 381
Shortest AS-Path 383
Origin Type 383
Multi-Exit Discriminator (MED) 384
EBGP over IBGP 386
Lowest IGP Metric 386
Prefer the Oldest EBGP Path 387
Router ID 387
Minimum Cluster List Length 388
Lowest Neighbor Address 388
Troubleshooting BGP Best Path 389
Visualizing the Topology 390
Phase I–Initial BGP Edge Route Processing 391
Phase II–BGP Edge Evaluation of Multiple Paths 392
Phase III–Final BGP Processing State 394
Path Selection for the Routing Table 394
Common Issues with BGP Multihoming 395
Transit Routing 395
Problems with Race Conditions 397
Peering on Cross-Link 402
Expected Behavior 403
Unexpected Behavior 406
Secondary Verification Methods of a Routing Loop 409
Design Enhancements 411
Full Mesh with IBGP 412
Problems with Redistributing BGP into an IGP 413
Summary 417
References 418
Part IV Securing BGP
Chapter 9 Securing BGP 419
The Need for Securing BGP 419
Securing BGP Sessions 420
Explicitly Configured Peers 421
IPv6 BGP Peering Using Link-Local Address 421
BGP Session Authentication 424
BGP Pass Through 426
EBGP-Multihop 427
BGP TTL Security 428
Filtering 429
Protecting BGP Traffic Using IPsec 431
Securing Interdomain Routing 431
BGP Prefix Hijacking 432
S-BGP 439
IPsec 439
Public Key Infrastructure 439
Attestations 441
soBGP 442
Entity Certificate 442
Authorization Certificate 443
Policy Certificate 443
BGP SECURITY Message 443
BGP Origin AS Validation 443
Route Origination Authorization (ROA) 445
RPKI Prefix Validation Process 446
Configuring and Verifying RPKI 449
RPKI Best-Path Calculation 460
BGP Remote Triggered Black-Hole Filtering 463
BGP Flowspec 467
Configuring BGP Flowspec 469
Summary 479
References 480
Part V Multiprotocol BGP
Chapter 10 MPLS Layer 3 VPN (L3VPN) 481
MPLS VPNs 481
MPLS Layer 3 VPN (L3VPN) Overview 483
Virtual Routing and Forwarding 483
Route Distinguisher 485
Route Target 485
Multi-Protocol BGP (MP-BGP) 486
Network Advertisement Between PE and CE Routers 487
MPLS Layer 3 VPN Configuration 487
VRF Creation and Association 488
IOS VRF Creation 488
IOS XR VRF Creation 489
NX-OS VRF Creation 490
Verification of VRF Settings and Connectivity 492
Viewing VRF Settings and Interface IP Addresses 492
Viewing the VRF Routing Table 494
VRF Connectivity Testing Tools 495
MPLS Forwarding 495
BGP Configuration for VPNv4 and PE-CE Prefixes 497
IOS BGP Configuration for MPLS L3VPN 497
IOS XR BGP Configuration for MPLS L3VPN 499
NX-OS BGP Configuration for MPLS L3VPN 500
Verification of BGP Sessions and Routes 502
Troubleshooting MPLS L3VPN 506
Default Route Advertisement Between PE-CE Routers 508
Problems with AS-PATH 509
Suboptimal Routing with VPNv4 Route Reflectors 514
Troubleshooting Problems with Route Targets 520
MPLS L3VPN Services 524
RT Constraints 534
MPLS VPN Label Exchange 538
MPLS Forwarding 541
Summary 542
References 542
Chapter 11 BGP for MPLS L2VPN Services 543
L2VPN Services 543
Terminologies 545
Virtual Private Wire Service 548
Interworking 549
Configuration and Verification 550
VPWS BGP Signaling 558
Configuration 560
Virtual Private LAN Service 561
Configuration 562
Verification 564
VPLS Autodiscovery Using BGP 569
VPLS BGP Signaling 580
Troubleshooting 586
Summary 588
References 589
Chapter 12 IPv6 BGP for Service Providers 591
IPv6 BGP Features and Concepts 591
IPv6 BGP Next-Hop 591
IPv6 Reachability over IPv4 Transport 596
IPv4 Routes over IPv6 Next-Hop 601
IPv6 BGP Policy Accounting 604
IPv6 Provider Edge Routers (6PE) over MPLS 607
6PE Configuration 611
6PE Verification and Troubleshooting 615
IPv6 VPN Provider Edge (6VPE) 620
IPv6-Aware VRF 622
6VPE Next-Hop 623
Route Target 624
6VPE Control Plane 624
6VPE Data Plane 626
6VPE Configuration 627
6VPE Control-Plane Verification 629
6VPE Data Plane Verification 633
Summary 639
References 639
Chapter 13 VxLAN BGP EVPN 641
