IOS & Question Marks (?)

The question mark (?) is almost always used in Cisco IOS to present the help context menu. However, it can also be used as a special character in regex strings and EEM scripts. It can also be used in meta-information fields such as a VLAN name or interface description though I would considered it best to avoid this practice (but someone always has a “good” reason for everything). Should you ever have the need to insert a question mark (?), the escape sequence Ctrl + V must first be entered then the question mark (?) can be inserted as a string character.

Here is Cisco references for IOS and NX-OS, respectively.

Posted in Cisco | Leave a comment

Cisco ASA AAA Failure Debug

I recently came across an issue where our team was unable to log into one of our Cisco ASA firewalls running code version 9.2(4)5 to manage the firewall. Shortly after we were notified that AnyConnect clients were unable to authenticate. SSH is configured to authenticate using TACACS and AnyConnect using RADIUS, not the same protocols but still both functions of AAA . We reviewed our remote logging server (a must have tool!) for any output from the firewall and found the following log message:

%ASA-3-113001: Unable to open AAA session. Session limit [2048] reached

After discussing the issue with Cisco TAC, they provided the following debug command to assist with diagnostics:

debug menu aaa 61

The output of this command looked something like this:

Max Sessions: 2048
In Use List Count: 2047
In Use List Head: 247
In Use List Tail: 765

The issue ultimately came down to the firewall not properly tearing down AAA sessions to the AAA servers and eventually hitting the max session limit where it stopped performing further AAA functions (in our case SSH login and AnyConnect VPN authentication requests). The immediate resolution was to reboot each the firewall which cleared the sessions. We were running a HA pair and AAA sessions are not HA replicated so we were able to reboot them one at a time which allowed us to avoid outage time while resolving the AAA issue.

A quick search of the Cisco Bug site using the syslog event message ID (ASA-3-113001) reveals several known bug IDs for this event (CSCud50997, CSCuj10655, CSCtg28821, and others). We were not able to nail down exactly which bug was responsible. The short answer for us was the Cisco ASA platform has bugs (as does every platform from every vendor) and regularly patching (at least to the latest interim/minor version) is good network hygiene. We patched to the latest 9.2 minor/interim code release and have not seen another occurrence of this issue since..

Important follow-up note… the day after dealing with the issue above, we inexplicable experienced a software bug which repeatedly crashed each firewall in the HA pair until both crashed at the same time causing an outage. After the simultaneous crash occurred, both firewalls recovered and now appear stable. I have to assume whatever caused the crashing was rooted in some HA replicated information which is why they are stable after the dual reboot (which is the only way to fully clear the HA replicated state stable information from both firewalls). We were never able to find a root cause of the crash events. While I have no reason other than timing to think the two issues are related, consider this a warning… if you experience the AAA issue above, do not be surprised if it is followed shortly there after by a full firewall crash & reboot (maybe even consider enabling coredump on one of the firewalls in the HA pair just in case).

Posted in Bugs, Cisco, Networking | Tagged , , | Leave a comment

Cisco IOS AAA Custom Login Prompts

While likely more fun than function (though I am sure there are some truly justified reasons to do this, feel free to share in comments if you have one), those who are running Cisco IOS can customize the echoed text given when IOS prompts for either a username or password, you can use the following AAA configuration attributes:

aaa authentication username-prompt text-string
aaa authentication password-prompt text-string

Additional information about these configuration attributes can be found on the website here.

Posted in Cisco | Tagged , , | Leave a comment