Cisco ASA AAA Failure Debug

I recently came across an issue where our team was unable to log into one of our Cisco ASA firewalls running code version 9.2(4)5 to manage the firewall. Shortly after we were notified that AnyConnect clients were unable to authenticate. SSH is configured to authenticate using TACACS and AnyConnect using RADIUS, not the same protocols but still both functions of AAA . We reviewed our remote logging server (a must have tool!) for any output from the firewall and found the following log message:

%ASA-3-113001: Unable to open AAA session. Session limit [2048] reached

After discussing the issue with Cisco TAC, they provided the following debug command to assist with diagnostics:

debug menu aaa 61

The output of this command looked something like this:

IN USE AUTH HANDLE STATS
Max Sessions: 2048
In Use List Count: 2047
In Use List Head: 247
In Use List Tail: 765

The issue ultimately came down to the firewall not properly tearing down AAA sessions to the AAA servers and eventually hitting the max session limit where it stopped performing further AAA functions (in our case SSH login and AnyConnect VPN authentication requests). The immediate resolution was to reboot each the firewall which cleared the sessions. We were running a HA pair and AAA sessions are not HA replicated so we were able to reboot them one at a time which allowed us to avoid outage time while resolving the AAA issue.

A quick search of the Cisco Bug site using the syslog event message ID (ASA-3-113001) reveals several known bug IDs for this event (CSCud50997, CSCuj10655, CSCtg28821, and others). We were not able to nail down exactly which bug was responsible. The short answer for us was the Cisco ASA platform has bugs (as does every platform from every vendor) and regularly patching (at least to the latest interim/minor version) is good network hygiene. We patched to the latest 9.2 minor/interim code release and have not seen another occurrence of this issue since..

Important follow-up note… the day after dealing with the issue above, we inexplicable experienced a software bug which repeatedly crashed each firewall in the HA pair until both crashed at the same time causing an outage. After the simultaneous crash occurred, both firewalls recovered and now appear stable. I have to assume whatever caused the crashing was rooted in some HA replicated information which is why they are stable after the dual reboot (which is the only way to fully clear the HA replicated state stable information from both firewalls). We were never able to find a root cause of the crash events. While I have no reason other than timing to think the two issues are related, consider this a warning… if you experience the AAA issue above, do not be surprised if it is followed shortly there after by a full firewall crash & reboot (maybe even consider enabling coredump on one of the firewalls in the HA pair just in case).

Advertisements

About kludgebomb

Network engineer with an obsession for the details.
This entry was posted in Bugs, Cisco, Networking and tagged , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s