AP NOT Joined

[antonioxud80]

Good Evening,

I have a problem with a cisco 5508 controller, no ap are unable to connect.

*spamApTask2: Nov 27 18:14:50.099: 00:5f:86:1e:66:e0 Join Priority Processing status = 0, Incoming Ap's Priority 1, MaxLrads = 100, joined Aps =0
*spamApTask2: Nov 27 18:14:50.099: 00:5f:86:1e:66:e0 Primary Discovery Response sent to 10.40.94.199:15203

*spamApTask6: Nov 27 18:14:50.249: c4:0a:cb:5c:7a:90 Primary Discovery Request from 10.40.94.114:48716

*spamApTask6: Nov 27 18:14:50.249: c4:0a:cb:5c:7a:90 Join Priority Processing status = 0, Incoming Ap's Priority 1, MaxLrads = 100, joined Aps =0
*spamApTask6: Nov 27 18:14:50.250: c4:0a:cb:5c:7a:90 Primary Discovery Response sent to 10.40.94.114:48716

*spamApTask6: Nov 27 18:14:50.250: c4:0a:cb:5c:7a:90 Primary Discovery Request from 10.40.94.114:48716

*spamApTask6: Nov 27 18:14:50.250: c4:0a:cb:5c:7a:90 Join Priority Processing status = 0, Incoming Ap's Priority 1, MaxLrads = 100, joined Aps =0
*spamApTask6: Nov 27 18:14:50.250: c4:0a:cb:5c:7a:90 Primary Discovery Response sent to 10.40.94.114:48716

*spamApTask4: Nov 27 18:14:50.251: c4:0a:cb:2d:c3:d0 Primary Discovery Request from 10.40.94.59:1321

Do you have any suggestions?

many thanks

Regards Antonio

[balaji.bandi]

What is the model of the AP ? - what Code WLC running here ?

is this new Setup or working one broken.

Connect the console to AP, post complete boot log to understand the issue.

[antonioxud80]

Cisco Controller) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 7.4.121.0
Bootloader Version............................... 1.0.1
Field Recovery Image Version..................... 6.0.182.0
Firmware Version................................. FPGA 1.3, Env 1.6, USB console 1.27
Build Type....................................... DATA + WPS

System Name...................................... WLC-3
System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.1069
Redundancy Mode.................................. Disabled
IP Address....................................... 10.40.92.9
Last Reset....................................... Software reset
System Up Time................................... 0 days 21 hrs 22 mins 46 secs
System Timezone Location.........................
System Stats Realtime Interval................... 5
System Stats Normal Interval..................... 180

Configured Country............................... Multiple Countries:IT,US
Operating Environment............................ Commercial (0 to 40 C)

--More-- or (q)uit
Internal Temp Alarm Limits....................... 0 to 65 C
Internal Temperature............................. +40 C
External Temperature............................. +31 C
Fan Status....................................... OK

State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 6
Number of Active Clients......................... 0

Memory Current Usage............................. Unknown
Memory Average Usage............................. Unknown
CPU Current Usage................................ Unknown
CPU Average Usage................................ Unknown

Burned-in MAC Address............................ 68:EF:BD:94:0C:A0
Power Supply 1................................... Present, OK
Power Supply 2................................... Absent
Maximum number of APs supported.................. 100

AP models: AIR-LAP1142N-E-K9 and AIR-CAP2602I-E-K9

thanks

[ajc]

after validation, you are running the right country code.

[antonioxud80]

all three controllers are configured with county IT and US
two work without any problems, but on the third controller there are no APs

I know the iOS version is very old, but it has been running smoothly for a few years and then right now I can't change it.

thanks

[ajc]

FINAL NOTE:

Another Issue if yourSHA-1 certificate on the WLC is expired and the fallback option of using SHA-2 does not work because your AP's have faulty SHA-2 certificates installed.

Issue: Wireless Controllers have a SHA-1 device cert that is valid for 10 years since manufacture date. When the SHA-1 certificate expires, APs should still be able to join using the SHA-2 certificate. 

Observed Impact:. I see APs unable to join WLC's once the controller certificate has expired. In theory, it should fall back in using the SHA-2 certificate. some APs manufactured between August 2014 and October 2014 have faulty SHA-2 certificate. Since our SHA-1 certificate expired on these controllers, AP with faulty SHA-2 certificates are unable to join.

There is a tool that identifies the APs with faulty certificates based on the serial number. 

https://www.cisco.com/c/en/us/support/docs/field-notices/639/fn63916.html

Potential Workarounds:

1.       ​​​​​RMA controller with expired certificate. 

2.       Redistribute faulty APs to WLC that does not have expired certificate

3.       Adjust the clock on the WLC. This may mess up logging so not recommended

[ajc]

Check this links and run a debug. Post the results/output of the debug.

https://community.cisco.com/t5/wireless/ap-can-t-join-dtls-connection-closed-by-controller/td-p/1871401

https://community.cisco.com/t5/wireless/wlc-and-ap/td-p/1912477

[antonioxud80]

debug capwap errors enable

spamApTask1: Nov 27 21:04:21.019: 00:f2:8b:10:57:1b Deleting AP 10.40.94.207 which has not been plumbed
*spamApTask1: Nov 27 21:04:21.023: 00:f2:8b:10:57:1b DTLS connection was closed
*spamApTask1: Nov 27 21:04:21.023: 00:f2:8b:10:57:1b Discarding non-ClientHello Handshake OR DTLS encrypted packet from 10.40.94.207:1394)since DTLS session is not established

*spamApTask1: Nov 27 21:04:21.023: 00:f2:8b:10:57:1b DTLS connection closed event receivedserver (10:40:92:9/5246) client (10:40:94:207/1394)
*spamApTask1: Nov 27 21:04:21.024: 00:f2:8b:10:57:1b No entry exists for AP (10:40:94:207/1394)
*spamApTask1: Nov 27 21:04:21.024: 00:f2:8b:10:57:1b No AP entry exist in temporary database for 10.40.94.207:1394
*spamApTask5: Nov 27 21:04:21.074: 28:94:0f:05:cd:a6 DTLS connection not found, creating new connection for 10:40:94:137 (23772) 10:40:92:9 (5246)

*spamApTask5: Nov 27 21:04:21.216: 28:94:0f:05:cd:a6 Deleting AP 10.40.94.137 which has not been plumbed
*spamApTask5: Nov 27 21:04:21.220: 28:94:0f:05:cd:a6 DTLS connection was closed
*spamApTask5: Nov 27 21:04:21.220: 28:94:0f:05:cd:a6 Discarding non-ClientHello Handshake OR DTLS encrypted packet from 10.40.94.137:23772)since DTLS session is not established

debug capwap errors enable

*spamApTask2: Nov 27 21:07:33.677: c4:64:13:00:53:c5 Deleting AP 10.40.94.165 which has not been plumbed
*spamApTask2: Nov 27 21:07:33.682: c4:64:13:00:53:c5 DTLS connection was closed
*spamApTask2: Nov 27 21:07:33.682: c4:64:13:00:53:c5 Discarding non-ClientHello Handshake OR DTLS encrypted packet from 10.40.94.165:1341)since DTLS session is not established

*spamApTask1: Nov 27 21:07:34.274: c4:64:13:00:53:95 Deleting AP 10.40.94.38 which has not been plumbed
*spamApTask1: Nov 27 21:07:34.278: c4:64:13:00:53:95 DTLS connection was closed
*spamApTask1: Nov 27 21:07:34.279: c4:64:13:00:53:95 Discarding non-ClientHello Handshake OR DTLS encrypted packet from 10.40.94.38:1339)since DTLS session is not established

*spamApTask3: Nov 27 21:07:36.241: c4:64:13:11:40:25 Deleting AP 10.40.94.47 which has not been plumbed
*spamApTask3: Nov 27 21:07:36.246: c4:64:13:11:40:25 DTLS connection was closed
*spamApTask3: Nov 27 21:07:36.246: c4:64:13:11:40:25 Discarding non-ClientHello Handshake OR DTLS encrypted packet from 10.40.94.47:5124)since DTLS session is not established

*spamApTask1: Nov 27 21:07:36.310: 00:3a:7d:83:af:ff Deleting AP 10.40.94.224 which has not been plumbed
*spamApTask1: Nov 27 21:07:36.315: 00:3a:7d:83:af:ff DTLS connection was closed
*spamApTask1: Nov 27 21:07:36.315: 00:3a:7d:83:af:ff Discarding non-ClientHello Handshake OR DTLS encrypted packet from 10.40.94.224:15105)since DTLS session is not established

*spamApTask2: Nov 27 21:07:36.591: 00:f2:8b:10:58:77 Deleting AP 10.40.94.201 which has not been plumbed
*spamApTask2: Nov 27 21:07:36.595: 00:f2:8b:10:58:77 DTLS connection was closed
*spamApTask2: Nov 27 21:07:36.596: 00:f2:8b:10:58:77 Discarding non-ClientHello Handshake OR DTLS encrypted packet from 10.40.94.201:1417)since DTLS session is not established

[MHM Cisco World]

first check the time/date in WLC
what type of AP auth, is it MIC with authorize local ?

[ajc]

CHECK for MIC expiration date on those AP's and ALSO on your WLC's. I have recently ISSUES on BOTH sides regarding certificate expiration. Let me provide you the commands for verification on the WLC and AP's since that you are running OLD WLC's and OLD AP's

[ajc]

On the AP side, privilege mode, run this

AP_CLI#sh crypto pki certificates

AND look for the section of the certificate, 

Certificate
Status: Available
Certificate Serial Number: 728AF4350000001E4C89
Certificate Usage: General Purpose
Issuer:
cn=Cisco Manufacturing CA
o=Cisco Systems
Subject:
Name: C1130-001c58b5b3a4
ea=support@cisco.com
cn=C1130-001c58b5b3a4
o=Cisco Systems
l=San Jose
st=California
c=US
CRL Distribution Points:
http://www.cisco.com/security/crl/cmca.crl
Validity Date:
start date: 04:22:10 UTC Jul 11 2007
end date: 04:32:10 UTC Jul 11 2017
Associated Trustpoints: Cisco_IOS_MIC_cert

AP_CLI >show auth-list
...
AP with Self-Signed Certificate................ yes
...

All AP SSC's have an expiration date of January 1st, 2020.

[antonioxud80]

ov 28 11:59:09.003: %CAPWAP-3-ERRORLOG: Go join a capwap controller
*Nov 28 11:58:04.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.40.92.9 peer_port: 5246
*Nov 28 11:58:04.128: %PKI-3-CERTIFICATE_INVALID_EXPIRED: Certificate chain validation has failed. The certificate (SN: 3948E4B6000000317F37) has expired. Validity period ended on 05:40:06 UTC Jul 2 2020Peer certificate verification failed 001A

*Nov 28 11:58:04.128: %CAPWAP-3-ERRORLOG: Certificate verification failed!
*Nov 28 11:58:04.128: DTLS_CLIENT_ERROR: ../capwap/base_capwap/capwap/base_capwap_wtp_dtls.c:447 Certificate verified failed!
*Nov 28 11:58:04.128: %DTLS-5-SEND_ALERT: Send FATAL : Bad certificate Alert to 10.40.92.9:5246
*Nov 28 11:58:04.128: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.40.92.9:5246

[MHM Cisco World · Accepted solution]

your debug show the problem and here the solution as cisco suggest.

[antonioxud80 · Accepted solution]

for me this workaround worked to solve the certificate problem

(Cisco Controller) >config time manual MM/DD/YY HH:MM:SS

I disabled the ntp server and set the time manually (one year earlier)

Many thanks for your support

[MHM Cisco World]

Bro you are so so welcome any time.