Wednesday, 4 June 2014

ASR1k architechture and toubleshooting packet loss

ASR1k architechture and toubleshooting packet loss











Points of Packet Drops ->>
Module   Functional Component
SPA   Dependent on the interface type
SIP    IO Control Processor (IOCP) SPA Aggregation ASIC Interconnect ASIC
ESP   Cisco QuantumFlow Processor (QFP) Forwarding Control Processor (FECP) Interconnect ASIC QFP subsystem. QFP subsystem consists of these components:
  -  Packet Processor Engine (PPE)
  - Buffering, Queuing, and Scheduling (BQS)
  - Input Packet Module (IPM)
  - Output Packet Module (OPM)
  - Global Packet Memory (GPM)

RP    Linux Shared Memory Punt Interface (LSMPI) Interconnect ASIC

SPA
    show interfaces <interface-name>
    show interfaces <interface-name> accounting
    show interfaces <interface-name> stats

SIP
    show platform hardware port <slot/card/port> plim statistics
    show platform hardware subslot {slot/card} plim statistics
    show platform hardware slot {slot} plim statistics
    show platform hardware slot {0|1|2} plim status internal
    show platform hardware slot {0|1|2} serdes statistics

ESP
    show platform hardware slot {f0|f1} serdes statistics
    show platform hardware slot {f0|f1} serdes statistics internal
    show platform hardware qfp active bqs 0 ipm mapping
    show platform hardware qfp active bqs 0 ipm statistics channel all
    show platform hardware qfp active bqs 0 opm mapping
    show platform hardware qfp active bqs 0 opm statistics channel all
    show platform hardware qfp active statistics drop | exclude _0_
    show platform hardware qfp active interface if-name <Interface-name> statistics
    show platform hardware qfp active infrastructure punt statistics type per-cause | exclude _0_
    show platform hardware qfp active infrastructure punt statistics type punt-drop | exclude _0_
    show platform hardware qfp active infrastructure punt statistics type inject-drop  | exclude _0_
    show platform hardware qfp active infrastructure punt statistics type global-drop | exclude _0_
    show platform hardware qfp active infrastructure bqs queue output default all
    show platform hardware qfp active infrastructure bqs queue output recycle all
    !--- The if-name option requires full interface-name

RP
    show platform hardware slot {r0|r1} serdes statistics
    show platform software infrastructure lsmpi
To display statistics of packets that are according to protocol, use this command:
  Router#show interfaces TenGigabitEthernet 1/0/0 accounting
To display statistics of packets that were process switched, fast switched, or distributed switched:
   Router#show interfaces TenGigabitEthernet 1/0/0 stats
To display per port queue drop counters on SPA Aggregation ASIC, use this command:
 
Router#show platform hardware port 1/0/0 plim statistics
To display per SPA counters on SPA Aggregation ASIC, use this command:
   
Router#show platform hardware subslot 1/0 plim statistics
To display all SPA counters on SPA Aggregation ASIC, use this command:
    
Router#show platform hardware slot 1 plim statistics
To display aggregated rx/tx counters to/from Interconnect ASIC on SPA Aggregation ASIC, use.. Rx counter means the input packet from SPA; the Tx counter means output packet to SPA.
   
Router#show platform hardware slot 1 plim status internal
To display rx counters from RP, SIP Interconnect ASIC on ESP Interconnect ASIC, use this command:
    
Router#show platform hardware slot F0 serdes statistics
To display internal link packet counters and error counters, use this command:
   
Router#show platform hardware slot F0 serdes statistics internal
To check mapping for the Input Packet Module (IPM) channel and other components, use this command:
   
Router#show platform hardware qfp active bqs 0 ipm mapping
To display statistical information for each channel in Input Packet Module (IPM), use:
   
Router#show platform hardware qfp active bqs 0 ipm statistics channel all
To check mapping for the Output Packet Module (OPM) channel and other components, use:
   
Router#show platform hardware qfp active bqs 0 opm mapping
To display statistical information for each channel in Output Packet Module (OPM), use:
   
Router#show platform hardware qfp active bqs 0 opm statistics channel all
To display statistics of drops for all interfaces in Packet Processor Engine (PPE), use:  (start here)
   
Router#show platform hardware qfp active statistics drop
To clear statistics of drops for all interfaces in Packet Processor Engine (PPE), use.
   
Router#show platform hardware qfp active statistics drop clear
To display statistics of drops for each interface in the Packet Processor Engine (PPE), use..
This counter is cleared every 10 seconds.
   
Router#show platform hardware qfp active interface if-name TenGig1/0/0 statistics
To check cause of packet punted to RP, use this command:
   
Router#show platform hardware qfp active infrastructure punt statistics type per-cause
To display the statistics of drops for punt packets (ESP to RP), use this command:
   
Router#show platform hardware qfp active infrastructure punt statistics type punt-drop
To display the statistics of drops for inject packets (RP to ESP), use this command..
Inject packets are sent from the RP to the ESP. Most of them are generated by IOSD. They are L2 keep alives, routing protocols, management protocols like SNMP, etc.
   
Router#show platform hardware qfp active infrastructure punt statistics type inject-drop
To display the statistics of global drops packets, use this command:
   
Router#show platform hardware qfp active infrastructure punt statistics type global-drop
To display statistics of default queues/schedules of Buffering, Queuing, and Scheduling (BQS) for each interface, use this command:
Router#show platform hardware qfp active infrastructure bqs queue output default all
To display statistics of Recycle queues/schedules of Buffering, Queuing, and Scheduling (BQS) for each interface, use this command. Recycle queues hold packets that are processed more than once by QFP. For example, fragment packets and multicast packets are placed here.
   
Router#show platform hardware qfp active infrastructure bqs queue output recycle all
The RP processes these types of traffic:
- Mgmt traffic that comes through the gigabit Ethernet management port on the route processor.
-Punt traffic in system(through the ESP),which includes all control-plane traffic recd on any SPA
- Older protocol traffic, DECnet, Internet Packet Exchange (IPX), etc.
To display rx counters from ESP Interconnect ASIC on RP Interconnect ASIC, use this command:
 Router#show platform hardware slot r0 serdes statistics
To display the statistics for the Linux Shared Memory Punt Interface (LSMPI) on the router, use this command. LSMPI offers a way to do zero-copy transfer of packets between the network and IOSd for high performance. In order to achieve this, share (memory map) a region in the Linux kernel virtual memory between the LSMPI module and IOSd.
Router#show platform software infrastructure lsmpi
***************
~ Packet Drops on SPA
* Error Packet
-> If a packet has an error, these packets are dropped on SPA. This is common behavior, not only on Cisco ASR 1000 Series Routers, but on all platforms.
    Router#show interfaces TenGigabitEthernet 1/0/0
419050 input errors, 419050 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     1 packets output, 402 bytes, 0 underruns
~ Packet drops on SIP
* High Utilization of QFP
-> In case of high utilization of QFP, packets are dropped in each interface queue on SIP by backpressure from QFP. In this case, a pause frame is also sent from the interface.
   
Router#show platform hardware port 1/0/0 plim statistics
    Interface 1/0/0
      RX Low Priority
        RX Drop Pkts 21344279    Bytes 1515446578
~ Packet Drops on ESP
*Oversubscription
-> If you send packets that exceed the wire rate of the interface, the packets are dropped at the egress interface.
   
Router#show interfaces GigabitEthernet 1/1/0
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 48783
On QFP, these drops can be checked as Taildrop.
    Router#show platform hardware qfp active statistics drop | exclude _0_
    ----------------------------------------------------------------
    Global Drop Stats                         Octets         Packets
    ----------------------------------------------------------------
      TailDrop                            72374984           483790
* Overload by Packet Fragment
If packets are fragmented due to the MTU size, even if the ingress interface is less than the wire rate, wire rate can be exceeded at the egress interface. In this case, the packet is dropped at the egress interface.
    Router#show interfaces gigabitEthernet 1/1/0
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 272828
On QFP, these drops can be checked as Taildrop.
   
Router#show platform hardware qfp active statistics drop | exclude _0_
    ----------------------------------------------------------------
    Global Drop Stats                         Octets         Packets
    ----------------------------------------------------------------
      TailDrop                           109431162          272769
* Performance Limit by Fragment Packets
In QFP, Global Packet Memory (GPM) is used for reassembly for the fragmented packet. If GPM runs out in the reassembly of large numbers of fragmentation packets, these counters show the number of packet drops. In many cases, this is a performance limit.
    Router#show platform hardware qfp active statistics drop | ex _0_
    ----------------------------------------------------------------
    Global Drop Stats                         Octets         Packets
    ----------------------------------------------------------------
      ReassNoFragInfo                  39280654854        57344096
      ReassTimeout                          124672             128
* Forwarding to Null0 Interface
The packets to Null0 interface are dropped on ESP and not punted to RP. In such a case, possibly you are not unable to check the counter by the traditional command (show interfaces null0). Check the ESP counter, in order to know the number of packet drops. If the “clear” and “exclude _0_” options are used at the same time, you can check only new drop packets.
   
Router#show platform hardware qfp active statistics drop clear | ex _0_
    ----------------------------------------------------------------
    Global Drop Stats                         Octets         Packets
    ----------------------------------------------------------------
      Ipv4Null0                              11286              99
* RP Switchover with HA Nonsupport Feature
In the case of RP switch over, these packets are dropped until the new active RP reprograms the QFP. All packets are dropped if the new active RP was not synced with the old active RP before the switch over. Packets are processed by High Availability (HA) nonsupport features.
   
Router#show platform hardware qfp active statistics drop | ex _0_
    ----------------------------------------------------------------
    Global Drop Stats                         Octets         Packets
    ----------------------------------------------------------------
      Ipv4NoAdj                            6993660          116561
      Ipv4NoRoute                        338660188         5644337
* Punt Packets
On the Cisco ASR 1000 Series Routers, packets that cannot be handled by ESP are punted to RP. If there are too many punt packets, the TailDrop of QFP drop statistics increases.
   
Router#show platform hardware qfp active statistics drop | ex _0_
    ----------------------------------------------------------------
    Global Drop Stats                         Octets         Packets
    ----------------------------------------------------------------
      TailDrop                            26257792           17552
Check the Buffering, Queuing, and Scheduling (BQS) queue output counter in order to specify the dropped interface. The “internal0/0/rp:0” shows the interface to punt from ESP to RP.
    Router#show platform hardware qfp active infrastructure bqs queue output default all
    Interface: internal0/0/rp:0
     Statistics:
        tail drops (bytes): 26257792            ,          (packets): 17552
        total enqs (bytes): 4433777480          ,          (packets): 2963755
queue_depth (bytes): 0
In such a case, the Input queue drop is counted on the ingress interface.
    Router#show interfaces TenGigabitEthernet 1/0/0
Input queue: 0/375/2438309/0 (size/max/drops/flushes); Total output drops: 0
The reason for the punt can be shown by this command:
    Router#show platform hardware qfp active infrastructure punt statistics type per-cause
    Global Per Cause Statistics
Counter ID  Punt Cause Name                   Received      Transmitted
  ------------------------------------------------------------------------
  00          RESERVED                          0             0
  01          MPLS_FRAG_REQUIRE                 0             0
  02          IPV4_OPTIONS                      2981307       2963755
You can also check the show ip traffic command.
   
Router#show ip traffic
IP statistics:
  Rcvd:  2981307 total, 15 local destination
         0 format errors, 0 checksum errors, 0 bad hop count
         0 unknown protocol, 0 not a gateway
         0 security failures, 0 bad options, 2981307 with options
*Punt Limit by Punt Global Policer
In case too many punt packets are destined to the router itself, the Taildrop counts with PuntGlobalPolicerDrops by the QFP drop counter. The Punt Global Policer protects RP from an overload. These drops are seen not by the transit packet but by the FOR_US packet.
 
 Router#show platform hardware qfp active statistics drop | ex _0_
    ----------------------------------------------------------------
    Global Drop Stats                         Octets         Packets
    ----------------------------------------------------------------
      PuntGlobalPolicerDrops                155856             102
      TailDrop                          4141792688         2768579
The reason for the punt can be known by this command:
    Router#show platform hardware qfp active infrastructure punt statistics type per-cause
    Global Per Cause Statistics
Counter ID  Punt Cause Name                   Received      Transmitted
  ------------------------------------------------------------------------
11          FOR_US                            5197865       2428755
* Packet Drops on RP
Packet Errors on LSMPI
On the Cisco ASR 1000 Series Routers, the packet is punted from ESP to RP through the Linux Shared Memory Punt Interface (LSMPI). LSMPI is the virtual interface for the packet transfer between the IOSd and Linux kernel on RP through the Linux shared memory. Packets punted from the ESP to the RP are received by the Linux kernel of the RP. The Linux kernel sends those packets to the IOSD process through LSMPI. If you see error counters up on the LSMPI, this is a software defect. Open a TAC case.
    Router#show platform software infrastructure lsmpi
Lsmpi0 is up, line protocol is up
  Hardware is LSMPI
1 input errors, 0 CRC, 3 frame, 0 overrun, 0 ignored, 0 abort

Thanks to - Harbaksh Singh
Shared for Reading

1 comment: