The Silo Chronicles
The not so exciting adventures of a Network Engineer

As anyone who has ever applied an access-list to an interface knows you must specify the direction in which you wish to apply the access-list.  This direction is used to determine which side is to be used as the source or destination.  While this sounds fairly straight forward it remains a black magic area for many engineers.  Today we are going to demystify this directionality and make it simple and straight forward.

Background
The first thing we must acknowledge is that unicast communications is always between two parties.  As such there is always a direction that communications is flowing.  For the purpose of this post we are going to define those directions as local and remote.  The local side is the interface that has the access-list on it and the remote side is whoever is on the other end of that interface.

Since all unicast communications is between two parties there is always a source and a destination.  These are the terms used within the access-list creation process.  The question is which side local or remote is the source and which side local or remote is the destination.  This is where the direction statement you specify on the interface comes into play.

Which Direction Am I Going?
If you apply the direction IN on an interface’s access-list the interface (local side) considers the source address to be the remote side.  If you apply the direction OUT on an interface’s access-list the interface (local) considers the destination access to be the remote side.  Thus the packet is flowing IN from the remote side or OUT to the remote side.

Why is this important?
Directionality is very critical into the effectiveness of access-lists.  If you do not understand which side is the source or destination your access-lists might result in being more open than originally intended or even worse longer than necessary.

Recently I was provided a challenge.  A customer has a call center with a large number of DIDs that all point to a single script and all calls are handled by a single group of agents.  The unusual part is that they did not want to determine the queue based off which number however they wanted the prompts to change depending on the DID that it came in on.

Obviously one could add a trigger for every number but this customer wanted to be able to determine all the prompts to play based off a database table and the number in that table is the DID not a trigger.  While I could simply extend the table I wanted to enable them to add new numbers without having to make changes to UCCX.  This ended up actually being fairly straight forward.

The first thing you need to do is ensure the number hits CUCM un-altered.  This means no translating on an H323 gateway.  Doing this means you would traditionally have a translation pattern for that number going to the UCCX Trigger.  Instead of doing that add a CTI Route Point and have that CTI Route Point forward all call to the trigger.  What this does is populate the “Last Redirected Number” field of Call Contact Info.  As a result you can now problematically access what originally dialed number even though all calls hit the single trigger.

This is related to yesterday’s post about Cisco changing the phone firmware signatures in 8.5.2(SR1).  To quickly recap the underlying issue is that a phone with firmware prior to 8.5.1 trying to connect to a system with firmware 8.5.2(SR1) or later will fail it’s upgrade due to “Auth Fail”.  This is because Cisco changed the firmware package signature in 8.5.2(SR1) and only firmware 8.5.1 and 8.5.2 have the new signature as well as the old signature.  This means you must upgrade to 8.5.2 before you can upgrade to 8.5.2(SR1) or later.

This trickles down to a factory defaulted phone that runs a firmware prior to 8.5.1.  The way factory defaulting works is once you default the phone it knows to request a specific file from the TFTP server to figure out what versions of what to load.  This file is named “termXX.default.loads”.  The XX is the two digit model number in question. The problem is created when a device pack or service pack is installed on the server that includes firmware 8.5.2(SR1) or newer.  Along with all the firmware files the termXX.default.loads files are overwritten automatically by the latest version installed.

This means if you install 8.5.4 but go to Device Defaults and specify the 8.5.2 version it won’t matter for a factory defaulted phone because the termXX.default.loads file is from 8.5.4 pack.  The reverse is also true.  If you are running 8.5.4 but due to this issue go and install 8.5.2 your termXX.default.loads will now reflect 8.5.2 so all defaulted phones will also go to 8.5.2 and then have to upgrade again to 8.5.4 even if they were already at 8.5.4 previously.  This means if you install a downgraded version of firmware you need to make sure to go back and re-upload the currently installed versions of the termXX.default.loads files.

Checking Your Default Loads Files
This is actually pretty straight forward since the file is in plain text.  Simply use a TFTP client and request the “termXX.default.loads” file for the model of phone you are curious about.  Open it in a text editor and you should see the filenames it will instruct the phone to request.  From those filenames you should see the version number.

Downloading A Different Version of Default Loads Files
If you were like me when you first discovered you found that you needed an 8.5.2 file but the file reference 8.5.4 and the 8.5.2 was already installed.  This isn’t an issue since you can get the correct file from CCO.  If you haven’t installed the 8.5.2 simply backup (Via TFTP) the current version of the file, install the old version and then re-upload the backed up version of the file to restore it to the original state.  If you don’t need the firmware just the file simply go to CCO and find the phone in question.  There should be three files in each model’s area.  One is a .sgn which is for CUCM 5+, One is a .exe for CCM 4.x and the last is a .zip which says firmware files only.  You want to download the .zip firmware files only one.  Once you download it unzip it and you will find copies of your default loads files.

Replacing the Default Loads File
You need to log into the OS Administration section and then go to Software Updates/TFTP File Management.  If you search for termXX you can find the existing file there.  Simply upload the new version and it will automatically over-write the existing version of the file.

NOTE: Make sure you backup the existing one first so you can restore it later.

NOTE: You must restart the TFTP server after uploading the file for the new version to be offered.

You’re Done
At this point the phones that won’t boot should request the new termXX.default.loads file and download the 8.5.2 version of firmware.  Once that is done they should request the latest version assuming the device defaults references a newer version.

One of my current joys I keep getting subjected to is the upgrade Auth Fail messages on brand new phones from Cisco. It seems that with the introduction of 8.5.2(SR1) Cisco requires that you know the new signature in order to load the firmware. The only problem is that brand new phones from Cisco are still shipping with 8.3.1 which doesn’t know the new signature. What does this mean? This means if you are running a current version of Communications Manager you are going to have to manually upgrade the firmware to version 8.5.2 and then let it upgrade to the current version on your system otherwise it will be stuck at the original version.

The Fix

NOTE: You have to run a back-leveled version of firmware on the phone first. Installing this version for the first time will override the default device loads for ALL phones. As such make sure to go to “Devices\Device Settings\Device Defaults” and find the model in question taking note of what the load it currently is set to.  We need this so after we install the back-leveled version of firmware we can change back the default load to the more recent version already installed.

The first thing you will need to do is go download version 8.5.2 from CCO.  Do NOT download the SR1 version as the SR1 version uses the new signature.  Once it is download install it on your Communication Manager Cluster and restart the TFTP Service on all your servers that offer TFTP.  If you don’t restart the TFTP server the system won’t recognize and offer up the new file we just installed.  Once it is installed the default device load for that model will be changed to that version, something liek “SCCP42.8-5-2S”.  You will change to change this back to the latest version you have installed on your system otherwise you risk downgrading all of your phones.

Once you have 8.5.2 install on your system and your default device load changed back to the current version simply find the phone in question under “Device\Phones”.  Inside the phone configuration is a variable called Phone Load Name.  Put the 8.5.2 load name, the value the default device load was automatically changed to, click save and then reset the phone.  Once the phone resets it will upgrade to 8.5.2 and re-register with the system.  Once it is upgraded to 8.5.2 simply remove the value from Phone Load Name, save and reset the phone again.  This time the phone will go ahead and upgrade to the current version.

Long story short, pretty much all Communication Manager systems will need to have 8.5.2 firmware installed for all their phone models and be prepared to manually upgrade to 8.5.2 all new phones on the system.

It seems that almost weekly I am being asked “How can I tell how many calls are actually going on at any one point”". The driving force behind these questions recently has been the push to SIP. With the economy being what it is everyone is looking for cost savings and SIP is one of those things that not only saves money but in most cases saves a lot of money.

With traditional telephony circuits you size the number of talk-paths based off the physical location. With SIP the physical location boundary is removed and instead you size based off total con-current talk-paths needed. This is the source of most savings. For example if you have 10 offices each with their own PRI you have 230 talk-paths. The chances that all 10 offices max their PRIs out at one time is very slim. Instead you might find that only a fraction of those might be in use all at the same time.

So again the question becomes what is my actual usage? The first stop for most customers is the telephony circuit provider. They get mixed results there. Typically they are not left feeling comfortable with what they are told and instead would like to get their own numbers. This is where I typically get tasked to explain to them how to get this data.

When using Cisco CallManager there is no easy way to get this data unfortunately. While we can get this data we have to jump through some hoops. First lets cover the list of gotchas:

• You can’t run a report and collect historical data.
• You have to capture this data on a per gateway basis and then merge the data together based off time interval
• You can only capture 6 gateways per PC. You most likely will have to run multiple copies of the software to capture the data across all your gateways.

So how do we accomplish this? Follow these steps:

• Download and install Cisco Real Time Monitoring Tool. It can be found under Application/Plugins in the Communications Manager Web Interface.
• Once this is installed point it to the subscriber that the gateway is registered too and log in with an administration account.
• Click on Performance
• From this point if you want to look at an H323 gateway then expand Cisco H323 and select CallsActive, if MGCP expand Cisco MGCP Gateways and select PRIChannelsActive or FXOPortsActive.
• After Selecting this information you will be prompted to select a gateway. Select the gateway you wish to monitor and click add
• At this point you should have between 1 and 6 gateways shown on your screen. Click on the first gateway (A box should appear around the graph to show you it is highlighted). Right click and click Start Counter(s) Logging. Enter a unique name and click Ok.
• Repeat for all other gateways.

At this point the gateways are generating six CSV files found under your user directory in the .jrtmt/logs folder. You can get the exact location shown to you when the dialog box to enter a filename is displayed. You will most likely have to repeat this on multiple PCs until you are collecting logs on all your PCs

What will result is a series of CSV files. Each time there is at least one call going on a row is added to the log. The gateways are polled every 10 seconds and the chances are that all the PCs will not have the exact same polling time. As such you will need to figure out what your polling window will be. I typically recommend rounding down or up to the base tens (10, 20, 30, etc…). This will generate the picture of gateway utilization and concurrent talk-paths in use across all your gateways.

Like I said it wasn’t very pretty but it will get you accurate data.

One of the long standing annoyances with using a PRI in an H323 gateway in conjunction with CallManager was the lack of the ability to support Caller-ID names. If the PRI was tied to CCM via MGCP or SIP you were good but H323 you were SOL. The good news is that recently Cisco has resolved this lack of support recently. The first step is going to be upgrading your router to either 12.4(20)T, 12.4(11)XW or a later release in either train. Once that is done you need to add the following commands.

This will enable support system wide. If you wish you can also add the command to a voice class as well. Once this is added you need to enable the supplementary service on the PRI. You can do this by entering the following command on the PRI’s serial interface.

At this point you should be passing caller-id names as long as you have subscribed to the service with your provider. It should be noted that if you don’t have this service you can also break the PRI by adding the supp-service command making it always returned an un-subscribed service when placing or receiving calls. To troubleshoot this and see if you are seeing the caller-id and or figure out why calls aren’t working issue the following command:

This will give you all the signaling that goes back and forth. You should see the caller-id name text in the messages if the provider is sending it.

It’s been awhile since I last posted so I figured I would start with a command of the day post. Today’s post is actually going to be several commands. These are commands that are not very common but I find useful when working with the Cisco Autonomous Access Points.

dot11 arp-cache [optional]
This command is entered in the global configuration mode. It enables client ARP caching on the access point. ARP caching on the access point reduces the traffic on your wireless LAN and increases client battery life by stopping ARP requests for client devices at the access point. Instead of forwarding ARP requests to client devices, the access point responds to requests on behalf of associated client devices and drops ARP requests that are not directed to clients associated to the access point. When ARP caching is optional, the access point responds on behalf of clients with IP addresses known to the access point but forwards through its radio port any ARP requests addressed to unknown clients. When the access point knows all the IP addresses for associated clients, it drops any ARP requests not directed to its clients. In its beacon, the access point includes an information element to alert client devices that they can safely ignore broadcast messages to increase battery life.

dot11 [Interface] carrier busy
This has to be one of my favorite commands for deploying new access points. This command is entered in the EXEC mode and will cause the access point to do a carrier busy test on each channel resulting in a report showing you how busy each of the channels are allowing you to select a free channel. Two things to note:

1. This will disassociate users from the access point so make sure it isn’t in production or people are aware of an outage before issuing the command.

2. To see the report you need to issue a “show log” command or if connected via console console logging enabled.

show dot11 associations
This command will show you a list of devices currently associated with the access point. You might find it useful to make this an alias on the access point.

Last night I was had a simple task of adding a new 3750 to an existing stack. I went in thinking it would be a cake walk as they auto-upgrade and basically I was just going to have to rack it and connect some stack cables. Like most things recently there was more to it. This customer happen to have a very old 1.5U 3750 with 16mb of flash. It happened to be the master of the stack as well. This proved to be very annoying when trying to get the switch in the stack. The first problem was the current revision of code was too old and since the new switch was a later hardware revision it refused to auto-upgrade since it wasn’t sure it could support it. The customer used the web interface so I did the archive upgrade reloaded the stack and when it came back up only the first two switches actually upgraded. The new switch was still on the old revision. I then attempted to manually start the upgrade only to run into “Unable to create temp dir “flash:update”" error. This was quickly resolved with a “delete /recursive /force flash:update”. Unfortunately this is when I discovered that since it only had 16mb of flash it didn’t actually have enough to store the image for auto upgrades. After about an hour and a half I said screw it, loaded the bins on them and reloaded the stack… Victory was mine!

Morals of this story:

1. Look at which switch is your master. If it is an old hardware revision be prepared for some issues
2. Check the amount of available flash on your master switch. If it is only 16mb auto-upgrade most likely will not work.
3. When push comes to shove you can also just load the .bin files on all members manually (TIP: Use copy flash1: flash2: etc… to save time when loading it between stack members)

As you are most likely aware there are 512mb AIM-CUE cards floating around. Unfortunately you can not run any of the recent releases on these cards as they all need 1GB. The pickle is how do you tell which card is installed? The answer is quite easy in fact however it is far from obviously. If you want to see which card you have installed do the following:

1. Connect to Router
2. Session to the module. (Example: service-module service-Engine 1/0 session). For more instructions on this see the previous post on how to password recover the CUE module.
3. Issue the command ‘’show software license”. It will give you the following output.
   cue> show software license
   Core:
   – Application mode: CCME
   – Total usable system ports: 6

   Voicemail/Auto Attendant:
   – Max system mailbox capacity time: 840
   – Default # of general delivery mailboxes: 10
   – Default # of personal mailboxes: 25

   – Max # of configurable mailboxes: 35

   Languages:
   – Max installed languages: 1
   – Max enabled languages: 1

4. What we are looking for is the ”Max system mailbox capacity time”. If it says 840 you have a 1GB module, if it says 480 you have the 512mb module.

Like I said, not very hard but not very obvious.

My adventures in Network land today brought me to a poor ignored AIM-CUE module running CUE 2.1. The customer was interested in us doing some upgrades to their CCME and CUE so I was poking around to get some basic info on the deployment. The only problem was the customer did not know the username/password. Luckily he knew the login/password for the router which makes access to CUE very simple.

1. Connect to the Router and establish level 15 access
2. You need to determine what the ServiceEngine number for CUE is. You do that by issuing a show ip interface brief. It should look similar to the following:
   Router#show ip interface brief
   Interface IP-Address OK? Method Status Protocol
   FastEthernet0/0 192.168.10.1 YES NVRAM up up
   FastEthernet0/1 unassigned YES NVRAM administratively down down
   Service-Engine1/0 192.168.10.4 YES TFTP up up
3. Once you know the number type the following command service-module service-Engine 1/0 session. In this example CUE is installed at ServiceEngine1/0. Hit enter a few times and you should see a new command prompt.
4. You now what to take a look at the users and see which accounts are there and which ones are administrators. To get a list of users type show users.
   cue>show users
   admin
   johndoe
   davidsmith

   To get a list of Administrators type show group detail groupname Administrators.

   cue>show group detail groupname Administrators
   Full Name: Administrators
   Description:
   Phone:
   Phone(E.164):
   Language: systemDefault(en_US)
   Owners:
   Members: admin johndoe
   Privileges: superuser ManagePrompts ManagePublicList ViewPrivateList
5. If the username is “admin” you can reset the password by issuing the following command user admin password cisco.

As you can see it is very easy to do. If you are concerned about resetting a password you can also create a new administrator account by using the following commands:

As you can see if you have level 15 access to the router you can session over to the CUE module and reset any password you like. I would highly recommend you make sure that the service-module command stays strictly restricted to level 15.