How CAPWAP works in HiveOS and how to debug issues
CAPWAP - Control And Provisioning of Wireless Access Points
When your Access Point get disconnected from HiveManager (both Classic or NG), you have to check few basic stuff before going into a deep debug.
show capwap client
CAPWAP client: Enabled CAPWAP transport mode: UDP RUN state: Connected securely to the CAPWAP server CAPWAP client IP: 10.10.1.3 CAPWAP server IP: 220.127.116.11 HiveManager Primary Name:hmng-prd-ie-cwps-01.aerohive.com HiveManager Backup Name: hmng-prd-ie-cwpm-01.aerohive.com CAPWAP Default Server Name: redirector.aerohive.com Virtual HiveManager Name: VHM-IKILYXPE Server destination Port: 12222 CAPWAP send event: Enabled CAPWAP DTLS state: Enabled CAPWAP DTLS negotiation: Disabled DTLS next connect status: Enable DTLS always accept bootstrap passphrase: Enabled DTLS session status: Connected DTLS key type: passphrase DTLS session cut interval: 5 seconds DTLS handshake wait interval: 60 seconds DTLS Max retry count: 3 DTLS authorize failed: 0 DTLS reconnect count: 0 Discovery interval: 5 seconds Heartbeat interval: 30 seconds Max discovery interval: 10 seconds Neighbor dead interval:105 seconds Silent interval: 15 seconds Wait join interval: 60 seconds Discovery count: 0 Max discovery count: 3 Retransmit count: 0 Max retransmit count: 2 Primary server tries: 0 Backup server tries: 0 Keepalives lost/sent: 4/2835 Event packet drop due to buffer shortage: 0 Event packet drop due to loss connection: 4
Sometime is better to have all the counters cleared:
clear capwap client counter
In the output message, you can easily recognize
Your Access Point IP
CAPWAP client IP: 10.10.1.3
The reverse IP address of CAPWAP server
CAPWAP server IP: 18.104.22.168
The FQDN of your assigned CAPWAP server
HiveManager Primary Name:hmng-prd-ie-cwps-01.aerohive.com
and the FQDN of your assigned CAPWAP backup server
HiveManager Backup Name: hmng-prd-ie-cwpm-01.aerohive.com
and also the FQDN of your default redirector server
CAPWAP Default Server Name: redirector.aerohive.com
If you’ve accidentally shutdown the CAPWAP client, you should have a RED led on your AP and the “Disabled” status on your output. Normal state is, of course, enabled:
CAPWAP client: Enabled
Depending on what the Access Point discovered in the boot phase, the CAPWAP transport could be UDP or HTTP over TCP. The default mode id UDP:
CAPWAP transport mode: UDP
if your firewall is blocking UPD/12222 (this is the RFC port/proto used for CAPWAP), you can manually force the HTTP transport over TCP:
capwap client transport HTTP
Now, if your Access Point is still with RED led status, check the default networking settings: your management interface !
show interface mgt0
Admin state=enabled; Operational state=up; DHCP client=disabled; Default IP subnet=192.168.0.0/255.255.0.0; IP addr=10.10.1.3; Netmask=255.255.255.0; Default Gateway:10.10.1.1; IPV6 address autoconfig enable IPV6 link local addr=fe80::9e5d:12ff:fe5b:32c0/64 VLAN id=1; Native vlan id=1; MAC addr=9c5d:125b:32c0; MTU=1500; Rx packets=24189; errors=0; dropped=0; Tx packets=15452; errors=0; dropped=0; Rx bytes=4902310 (4.675 MB); Tx bytes=4149386 (3.957 MB);
you should at least reach your default gateway (10.10.1.1 in my case) and, before call the security team, ensure your DNS is working too!
PING redirector.aerohive.com (22.214.171.124) 56(84) bytes of data.
if you’re not able to resolve any FQDN, well, you found the problem. Probably just need to add manually the DNS server ip address:
dns server-ip 126.96.36.199
but, if your access point is in DHCP mode, you have to check the networking settings:
show interface mgt0 dhcp client
HM=HiveManager; SLS=system log server; DHCP client: Enabled Timeout for applying static IP or default IP: 20 secs State: Network configuration received; Get IP address 10.10.2.10 from the server 10.10.2.30; Relay agent 0.0.0.0 Lease time: 600 seconds; VLAN: 1 Duration time: 299 seconds Get options from server: Netmask (option number 1): 255.255.255.0 Router (option number 3): 10.10.2.30 DNS server (option number 6): 188.8.131.52 Log server (option number 7): DNS domain (option number 15): local.lan NTP server (option number 42): HM string (option 43 suboption 225): HM IP (option 43 suboption 226): SLS string (option 43 suboption 227): SLS IP (option 43 suboption 228): PPSK server IP (option 43 suboption 229): RADIUS server IP (option 43 suboption 230): RADIUS accounting server IP (option 43 suboption 231): Backup HM string (option 43 suboption 232): Backup HM IP (option 43 suboption 233):
If the networking layer is fully working, probably it’s better to check for any interface errors
show interface mgt0 | include errors
Rx packets=25184; errors=0; dropped=0; Tx packets=16118; errors=0; dropped=0;
show interface eth0 | include errors
Rx packets=10384584; errors=0; dropped=0; Tx packets= 8443112; errors=0; dropped=0;
If CAPWAP is working as expected, you should have 0 CAPWAP packet lost:
show capwap client | include Keepalives Keepalives lost/sent: 0/3253
Sometime you need to re-initialize the CAPWAP process, starting from the beginning, but instead do a reset config (factory default of the device) you can clear the actual capwap session with:
no capwap client server enable no capwap client server name no capwap client server backup name no capwap client vhm-name no capwap client transport
and re-enable the client:
capwap client server enable
Communication between Aerohive access devices such as APs and HiveManager uses three types of channels or protocols:
CAPWAP HTTPS SCP
CAPWAP and HTTPS are predominantly used for most transfers (SCP is no longer used in NG).
Aerohive devices send a keep alive message every 30 seconds to HiveManager over CAPWAP. CAPWAP is also used for sending network management commands, client connection events and AP status events. APs indicate the availability of statistical data files to HM based on report sampling intervals. HiveManager queries reporting data once every 30 minutes for client and AP level statistics (default).
If everything is working as expected, you should be able to performa a capwap ping to your server
capwap ping hmng-prd-ie-cwps-01.aerohive.com
CAPWAP ping parameters: Destination server: hmng-prd-ie-cwps-01.aerohive.com (184.108.40.206) Destination port: 12222 Count: 5 Size: 56(82) bytes Timeout: 5 seconds -------------------------------------------------- CAPWAP ping result: 82 bytes from 220.127.116.11 udp port 12222: seq=1 time=51.939 ms 82 bytes from 18.104.22.168 udp port 12222: seq=2 time=47.258 ms 82 bytes from 22.214.171.124 udp port 12222: seq=3 time=47.393 ms 82 bytes from 126.96.36.199 udp port 12222: seq=4 time=47.400 ms 82 bytes from 188.8.131.52 udp port 12222: seq=5 time=49.844 ms ------- hmng-prd-ie-cwps-01.aerohive.com CAPWAP ping statistics ------- 5 packets transmitted, 5 received, 0.00% packet loss, time 5282.97ms rtt min/avg/max = 47.258/48.766/51.939 ms
Be aware: CAPWAP ping is not ICMP !
ping Perform a CAPWAP ping (Note: A CAPWAP ping does not use ICMP echo requests, but UDP packets similar to those used for CAPWAP heartbeats.)
If you see something different in the log (show log buffer | include capwap) like:
capwap: CAPWAP:Exit current DTLS connect! capwap: CAPWAP_HM:Ready connecting to HM 255.255.255.255 capwap: CAPWAP_HM:use broadcast, ip=0.0.0.0, port=12222 capwap: CAPWAP_HM:get capwap server ip (184.108.40.206) for name (redirector.aerohive.com) capwap: CAPWAP_HM:can not find the ip address of hostname (hivemanager),reason:Unknown host capwap: CAPWAP_HM:user doesn't config primary and backup HM's name,use fixed server name and pre-defined server name to try capwap: CAPWAP_HM:get hivemanager name from scd (first:, second:). capwap: CAPWAP_HM:Choose HM IP or name for connecting.... capwap: CAPWAP: capwap predefine server name file isn't exist. capwap: CAPWAP: capwap client doesn't receive echo packet 2 times, CAPWAP client reconnect capwap: CAPWAP: capwap client reconnect because the neighbor dead interval elapsed during the previous CAPWAP session (reason id: 1006) capwap: CAPWAP: capwap client doesn't receive echo packet 2 times capwap: CAPWAP: capwap client doesn't receive echo packet 1 times
it means that there isn’t the Serial Number in the global Aerohive redirector.
If you need to manually add the Access Point into a specific instance or into a specific HiveManager, you can easly configure through the local https instance or through the CLI:
capwap client server name YOUR-HIVEMANAGER-FQDN-HERE capwap client vhm-name VHM-NAME-HERE
Just use question mark after the command to get help as usual (capwap client ?)
My last advice for the capwap debug: remove any DPI (Deep Packet Inspection) rule, SSL inspection, SSL Control, HTTPS proxy/reverse proxy/manipulation and so on…DTLS (Datagram Transport Layer Security) is VERY susceptible!
—– TO BE CONTINUED: proxy config, deep debug, throubleshooting, transport ….stay tuned Enjoy capwap!