Table of Contents

How CAPWAP works in HiveOS and how to debug issues

CAPWAP - Control And Provisioning of Wireless Access Points

When your Access Point get disconnected from HiveManager (both Classic or NG), you have to check few basic stuff before going into a deep debug.

show capwap client

CAPWAP client:   Enabled
CAPWAP transport mode:  UDP
RUN state: Connected securely to the CAPWAP server
CAPWAP client IP:        10.10.1.3
CAPWAP server IP:        54.154.134.72
HiveManager Primary Name:hmng-prd-ie-cwps-01.aerohive.com
HiveManager Backup Name: hmng-prd-ie-cwpm-01.aerohive.com
CAPWAP Default Server Name: redirector.aerohive.com
Virtual HiveManager Name: VHM-IKILYXPE
Server destination Port: 12222
CAPWAP send event:       Enabled
CAPWAP DTLS state:       Enabled
CAPWAP DTLS negotiation: Disabled
     DTLS next connect status:   Enable
     DTLS always accept bootstrap passphrase: Enabled
     DTLS session status: Connected
     DTLS key type: passphrase
     DTLS session cut interval:     5 seconds
     DTLS handshake wait interval: 60 seconds
     DTLS Max retry count:          3
     DTLS authorize failed:         0
     DTLS reconnect count:          0
Discovery interval:      5 seconds
Heartbeat interval:     30 seconds
Max discovery interval: 10 seconds
Neighbor dead interval:105 seconds
Silent interval:        15 seconds
Wait join interval:     60 seconds
Discovery count:         0
Max discovery count:     3
Retransmit count:        0
Max retransmit count:    2
Primary server tries:    0
Backup server tries:     0
Keepalives lost/sent:    4/2835
Event packet drop due to buffer shortage: 0
Event packet drop due to loss connection: 4

Sometime is better to have all the counters cleared:

clear capwap client counter

In the output message, you can easily recognize

Your Access Point IP

CAPWAP client IP:        10.10.1.3

The reverse IP address of CAPWAP server

CAPWAP server IP:        54.154.134.72

The FQDN of your assigned CAPWAP server

HiveManager Primary Name:hmng-prd-ie-cwps-01.aerohive.com

and the FQDN of your assigned CAPWAP backup server

HiveManager Backup Name: hmng-prd-ie-cwpm-01.aerohive.com

and also the FQDN of your default redirector server

CAPWAP Default Server Name: redirector.aerohive.com

If you’ve accidentally shutdown the CAPWAP client, you should have a RED led on your AP and the “Disabled” status on your output. Normal state is, of course, enabled:

CAPWAP client: Enabled

Depending on what the Access Point discovered in the boot phase, the CAPWAP transport could be UDP or HTTP over TCP. The default mode id UDP:

CAPWAP transport mode: UDP

if your firewall is blocking UPD/12222 (this is the RFC port/proto used for CAPWAP), you can manually force the HTTP transport over TCP:

capwap client transport HTTP

Now, if your Access Point is still with RED led status, check the default networking settings: your management interface !

show interface mgt0

Admin state=enabled; Operational state=up;
DHCP client=disabled;
Default IP subnet=192.168.0.0/255.255.0.0;
IP addr=10.10.1.3; Netmask=255.255.255.0; Default Gateway:10.10.1.1;
IPV6 address autoconfig enable
IPV6 link local addr=fe80::9e5d:12ff:fe5b:32c0/64
VLAN id=1;  Native vlan id=1;
MAC addr=9c5d:125b:32c0; MTU=1500;
Rx packets=24189; errors=0; dropped=0;
Tx packets=15452; errors=0; dropped=0;
Rx bytes=4902310 (4.675 MB); Tx bytes=4149386 (3.957 MB);

you should at least reach your default gateway (10.10.1.1 in my case) and, before call the security team, ensure your DNS is working too!

ping redirector.aerohive.com

PING redirector.aerohive.com (54.172.0.252) 56(84) bytes of data.

if you’re not able to resolve any FQDN, well, you found the problem. Probably just need to add manually the DNS server ip address:

dns server-ip 8.8.8.8

but, if your access point is in DHCP mode, you have to check the networking settings:

show interface mgt0 dhcp client

HM=HiveManager; SLS=system log server;

DHCP client: Enabled
Timeout for applying static IP or default IP: 20 secs
State: Network configuration received; Get IP address 10.10.2.10 from the server 10.10.2.30; Relay agent 0.0.0.0
Lease time: 600 seconds; VLAN: 1
Duration time: 299 seconds

Get options from server:
Netmask (option number 1): 255.255.255.0
Router (option number 3):     10.10.2.30
DNS server (option number 6): 8.8.8.8
Log server (option number 7):
DNS domain (option number 15): local.lan
NTP server (option number 42):
HM string (option 43 suboption 225):
HM IP (option 43 suboption 226):
SLS string (option 43 suboption 227):
SLS IP (option 43 suboption 228):
PPSK server IP (option 43 suboption 229):
RADIUS server IP (option 43 suboption 230):
RADIUS accounting server IP (option 43 suboption 231):
Backup HM string (option 43 suboption 232):
Backup HM IP (option 43 suboption 233):

If the networking layer is fully working, probably it’s better to check for any interface errors

show interface mgt0 | include errors

Rx packets=25184; errors=0; dropped=0;
Tx packets=16118; errors=0; dropped=0;

show interface eth0 | include errors

Rx packets=10384584; errors=0; dropped=0;
Tx packets= 8443112; errors=0; dropped=0;

If CAPWAP is working as expected, you should have 0 CAPWAP packet lost:

show capwap client | include Keepalives
Keepalives lost/sent: 0/3253

Sometime you need to re-initialize the CAPWAP process, starting from the beginning, but instead do a reset config (factory default of the device) you can clear the actual capwap session with:

no capwap client server enable
no capwap client server name
no capwap client server backup name
no capwap client vhm-name
no capwap client transport 

and re-enable the client:

capwap client server enable

Communication between Aerohive access devices such as APs and HiveManager uses three types of channels or protocols:

CAPWAP HTTPS SCP

CAPWAP and HTTPS are predominantly used for most transfers (SCP is no longer used in NG).

Aerohive devices send a keep alive message every 30 seconds to HiveManager over CAPWAP. CAPWAP is also used for sending network management commands, client connection events and AP status events. APs indicate the availability of statistical data files to HM based on report sampling intervals. HiveManager queries reporting data once every 30 minutes for client and AP level statistics (default).

If everything is working as expected, you should be able to performa a capwap ping to your server

capwap ping hmng-prd-ie-cwps-01.aerohive.com

CAPWAP ping parameters:
    Destination server: hmng-prd-ie-cwps-01.aerohive.com (54.154.134.72)
    Destination port: 12222
    Count: 5
    Size: 56(82) bytes
    Timeout: 5 seconds
--------------------------------------------------
CAPWAP ping result:
    82 bytes from 54.154.134.72 udp port 12222: seq=1 time=51.939 ms
    82 bytes from 54.154.134.72 udp port 12222: seq=2 time=47.258 ms
    82 bytes from 54.154.134.72 udp port 12222: seq=3 time=47.393 ms
    82 bytes from 54.154.134.72 udp port 12222: seq=4 time=47.400 ms
    82 bytes from 54.154.134.72 udp port 12222: seq=5 time=49.844 ms
    ------- hmng-prd-ie-cwps-01.aerohive.com CAPWAP ping statistics -------
    5 packets transmitted, 5 received, 0.00% packet loss, time 5282.97ms
    rtt min/avg/max = 47.258/48.766/51.939 ms

Be aware: CAPWAP ping is not ICMP !

    ping              Perform a CAPWAP ping (Note: A CAPWAP ping does not use
                      ICMP echo requests, but UDP packets similar to those
                      used for CAPWAP heartbeats.)

If you see something different in the log (show log buffer | include capwap) like:

capwap: CAPWAP:Exit current DTLS connect!
capwap: CAPWAP_HM:Ready connecting to HM 255.255.255.255
capwap: CAPWAP_HM:use broadcast, ip=0.0.0.0, port=12222
capwap: CAPWAP_HM:get capwap server ip (54.172.0.252) for name (redirector.aerohive.com)
capwap: CAPWAP_HM:can not find the ip address of hostname  (hivemanager),reason:Unknown host
capwap: CAPWAP_HM:user doesn't config primary and backup HM's name,use fixed server name and pre-defined server name to try
capwap: CAPWAP_HM:get hivemanager name from scd (first:, second:).
capwap: CAPWAP_HM:Choose HM IP or name for connecting....
capwap: CAPWAP: capwap predefine server name file isn't exist.
capwap: CAPWAP: capwap client doesn't receive echo packet 2 times, CAPWAP client reconnect
capwap: CAPWAP: capwap client reconnect because the neighbor dead interval elapsed during the previous CAPWAP session (reason id: 1006)
capwap: CAPWAP: capwap client doesn't receive echo packet 2 times
capwap: CAPWAP: capwap client doesn't receive echo packet 1 times

it means that there isn’t the Serial Number in the global Aerohive redirector.

If you need to manually add the Access Point into a specific instance or into a specific HiveManager, you can easly configure through the local https instance or through the CLI:

capwap client server name YOUR-HIVEMANAGER-FQDN-HERE
capwap client vhm-name VHM-NAME-HERE

Just use question mark after the command to get help as usual (capwap client ?)

My last advice for the capwap debug: remove any DPI (Deep Packet Inspection) rule, SSL inspection, SSL Control, HTTPS proxy/reverse proxy/manipulation and so on…DTLS (Datagram Transport Layer Security) is VERY susceptible!

—– TO BE CONTINUED: proxy config, deep debug, throubleshooting, transport ….stay tuned
Enjoy capwap!