The Silo Chronicles
The not so exciting adventures of a Network Engineer

It seems that almost weekly I am being asked “How can I tell how many calls are actually going on at any one point”". The driving force behind these questions recently has been the push to SIP. With the economy being what it is everyone is looking for cost savings and SIP is one of those things that not only saves money but in most cases saves a lot of money.

With traditional telephony circuits you size the number of talk-paths based off the physical location. With SIP the physical location boundary is removed and instead you size based off total con-current talk-paths needed. This is the source of most savings. For example if you have 10 offices each with their own PRI you have 230 talk-paths. The chances that all 10 offices max their PRIs out at one time is very slim. Instead you might find that only a fraction of those might be in use all at the same time.

So again the question becomes what is my actual usage? The first stop for most customers is the telephony circuit provider. They get mixed results there. Typically they are not left feeling comfortable with what they are told and instead would like to get their own numbers. This is where I typically get tasked to explain to them how to get this data.

When using Cisco CallManager there is no easy way to get this data unfortunately. While we can get this data we have to jump through some hoops. First lets cover the list of gotchas:

• You can’t run a report and collect historical data.
• You have to capture this data on a per gateway basis and then merge the data together based off time interval
• You can only capture 6 gateways per PC. You most likely will have to run multiple copies of the software to capture the data across all your gateways.

So how do we accomplish this? Follow these steps:

• Download and install Cisco Real Time Monitoring Tool. It can be found under Application/Plugins in the Communications Manager Web Interface.
• Once this is installed point it to the subscriber that the gateway is registered too and log in with an administration account.
• Click on Performance
• From this point if you want to look at an H323 gateway then expand Cisco H323 and select CallsActive, if MGCP expand Cisco MGCP Gateways and select PRIChannelsActive or FXOPortsActive.
• After Selecting this information you will be prompted to select a gateway. Select the gateway you wish to monitor and click add
• At this point you should have between 1 and 6 gateways shown on your screen. Click on the first gateway (A box should appear around the graph to show you it is highlighted). Right click and click Start Counter(s) Logging. Enter a unique name and click Ok.
• Repeat for all other gateways.

At this point the gateways are generating six CSV files found under your user directory in the .jrtmt/logs folder. You can get the exact location shown to you when the dialog box to enter a filename is displayed. You will most likely have to repeat this on multiple PCs until you are collecting logs on all your PCs

What will result is a series of CSV files. Each time there is at least one call going on a row is added to the log. The gateways are polled every 10 seconds and the chances are that all the PCs will not have the exact same polling time. As such you will need to figure out what your polling window will be. I typically recommend rounding down or up to the base tens (10, 20, 30, etc…). This will generate the picture of gateway utilization and concurrent talk-paths in use across all your gateways.

Like I said it wasn’t very pretty but it will get you accurate data.

One of the long standing annoyances with using a PRI in an H323 gateway in conjunction with CallManager was the lack of the ability to support Caller-ID names. If the PRI was tied to CCM via MGCP or SIP you were good but H323 you were SOL. The good news is that recently Cisco has resolved this lack of support recently. The first step is going to be upgrading your router to either 12.4(20)T, 12.4(11)XW or a later release in either train. Once that is done you need to add the following commands.

This will enable support system wide. If you wish you can also add the command to a voice class as well. Once this is added you need to enable the supplementary service on the PRI. You can do this by entering the following command on the PRI’s serial interface.

At this point you should be passing caller-id names as long as you have subscribed to the service with your provider. It should be noted that if you don’t have this service you can also break the PRI by adding the supp-service command making it always returned an un-subscribed service when placing or receiving calls. To troubleshoot this and see if you are seeing the caller-id and or figure out why calls aren’t working issue the following command:

This will give you all the signaling that goes back and forth. You should see the caller-id name text in the messages if the provider is sending it.

It’s been awhile since I last posted so I figured I would start with a command of the day post. Today’s post is actually going to be several commands. These are commands that are not very common but I find useful when working with the Cisco Autonomous Access Points.

dot11 arp-cache [optional]
This command is entered in the global configuration mode. It enables client ARP caching on the access point. ARP caching on the access point reduces the traffic on your wireless LAN and increases client battery life by stopping ARP requests for client devices at the access point. Instead of forwarding ARP requests to client devices, the access point responds to requests on behalf of associated client devices and drops ARP requests that are not directed to clients associated to the access point. When ARP caching is optional, the access point responds on behalf of clients with IP addresses known to the access point but forwards through its radio port any ARP requests addressed to unknown clients. When the access point knows all the IP addresses for associated clients, it drops any ARP requests not directed to its clients. In its beacon, the access point includes an information element to alert client devices that they can safely ignore broadcast messages to increase battery life.

dot11 [Interface] carrier busy
This has to be one of my favorite commands for deploying new access points. This command is entered in the EXEC mode and will cause the access point to do a carrier busy test on each channel resulting in a report showing you how busy each of the channels are allowing you to select a free channel. Two things to note:

1. This will disassociate users from the access point so make sure it isn’t in production or people are aware of an outage before issuing the command.

2. To see the report you need to issue a “show log” command or if connected via console console logging enabled.

show dot11 associations
This command will show you a list of devices currently associated with the access point. You might find it useful to make this an alias on the access point.

Last night I was had a simple task of adding a new 3750 to an existing stack. I went in thinking it would be a cake walk as they auto-upgrade and basically I was just going to have to rack it and connect some stack cables. Like most things recently there was more to it. This customer happen to have a very old 1.5U 3750 with 16mb of flash. It happened to be the master of the stack as well. This proved to be very annoying when trying to get the switch in the stack. The first problem was the current revision of code was too old and since the new switch was a later hardware revision it refused to auto-upgrade since it wasn’t sure it could support it. The customer used the web interface so I did the archive upgrade reloaded the stack and when it came back up only the first two switches actually upgraded. The new switch was still on the old revision. I then attempted to manually start the upgrade only to run into “Unable to create temp dir “flash:update”" error. This was quickly resolved with a “delete /recursive /force flash:update”. Unfortunately this is when I discovered that since it only had 16mb of flash it didn’t actually have enough to store the image for auto upgrades. After about an hour and a half I said screw it, loaded the bins on them and reloaded the stack… Victory was mine!

Morals of this story:

1. Look at which switch is your master. If it is an old hardware revision be prepared for some issues
2. Check the amount of available flash on your master switch. If it is only 16mb auto-upgrade most likely will not work.
3. When push comes to shove you can also just load the .bin files on all members manually (TIP: Use copy flash1: flash2: etc… to save time when loading it between stack members)

As you are most likely aware there are 512mb AIM-CUE cards floating around. Unfortunately you can not run any of the recent releases on these cards as they all need 1GB. The pickle is how do you tell which card is installed? The answer is quite easy in fact however it is far from obviously. If you want to see which card you have installed do the following:

1. Connect to Router
2. Session to the module. (Example: service-module service-Engine 1/0 session). For more instructions on this see the previous post on how to password recover the CUE module.
3. Issue the command ‘’show software license”. It will give you the following output.
   cue> show software license
   Core:
   – Application mode: CCME
   – Total usable system ports: 6

   Voicemail/Auto Attendant:
   – Max system mailbox capacity time: 840
   – Default # of general delivery mailboxes: 10
   – Default # of personal mailboxes: 25

   – Max # of configurable mailboxes: 35

   Languages:
   – Max installed languages: 1
   – Max enabled languages: 1

4. What we are looking for is the ”Max system mailbox capacity time”. If it says 840 you have a 1GB module, if it says 480 you have the 512mb module.

Like I said, not very hard but not very obvious.

My adventures in Network land today brought me to a poor ignored AIM-CUE module running CUE 2.1. The customer was interested in us doing some upgrades to their CCME and CUE so I was poking around to get some basic info on the deployment. The only problem was the customer did not know the username/password. Luckily he knew the login/password for the router which makes access to CUE very simple.

1. Connect to the Router and establish level 15 access
2. You need to determine what the ServiceEngine number for CUE is. You do that by issuing a show ip interface brief. It should look similar to the following:
   Router#show ip interface brief
   Interface IP-Address OK? Method Status Protocol
   FastEthernet0/0 192.168.10.1 YES NVRAM up up
   FastEthernet0/1 unassigned YES NVRAM administratively down down
   Service-Engine1/0 192.168.10.4 YES TFTP up up
3. Once you know the number type the following command service-module service-Engine 1/0 session. In this example CUE is installed at ServiceEngine1/0. Hit enter a few times and you should see a new command prompt.
4. You now what to take a look at the users and see which accounts are there and which ones are administrators. To get a list of users type show users.
   cue>show users
   admin
   johndoe
   davidsmith

   To get a list of Administrators type show group detail groupname Administrators.

   cue>show group detail groupname Administrators
   Full Name: Administrators
   Description:
   Phone:
   Phone(E.164):
   Language: systemDefault(en_US)
   Owners:
   Members: admin johndoe
   Privileges: superuser ManagePrompts ManagePublicList ViewPrivateList
5. If the username is “admin” you can reset the password by issuing the following command user admin password cisco.

As you can see it is very easy to do. If you are concerned about resetting a password you can also create a new administrator account by using the following commands:

As you can see if you have level 15 access to the router you can session over to the CUE module and reset any password you like. I would highly recommend you make sure that the service-module command stays strictly restricted to level 15.

As a Cisco Partner I have had several occasions to password recover some devices. Today brought an usual request of wanting to harvest the configuration not the password. A customer had a Cisco router provided by a 3rd party that they wanted to replace and return but unfortunately the vendor would/did not give them the information they needed to bypass the router and have the services handled directly by their equipment.

For some reason I always get a chuckle when doing these. The task is simple enough to complete. When you password recover the device it also provides you the current configuration, including passwords, except you are in enable mode and you can make any changes you deem necessary. To password recovery an IOS device do the following:

1. Connect a console cable to the device and power the device on.
2. Press CTRL-BRK to enter rommon.
3. Type “confreg 0×2142″ and press enter.
4. Type “reset” and press enter.
5. Upon booting the device will ask you to enter a configuration setup wizard. Answer NO to this question. The device is now booted with no configuration.
6. Type “enable” and press enter.
7. Type “configure memory”

At this point you should see the hostname change and if you issue a “show running-config” you will see the complete configuration, including passwords. In the case of today the vendor did not run password encryption so all their passwords were in plain text. Ruh Roh!

I ran into a very annoying software glitch today with my 3G iPhone. I charge my phone every night so when I started out at 06:00 it was fully charged. I got to my first client of the day at around 9:00 and noticed that my phone was already at half power. The best part was I actually hadn’t even used it yet. After another 2 hours my phone was completed dead and powered off. I popped up my laptop and started re-charging it and it turned back on. My laptop ended up going to sleep and when I turned around it was dead again. I repeated the process this time when it powered on I properly powered off the phone and then powered it back on. Same thing it drained in a matter of minutes. I finally made it back to my wall charger and was shocked to find that after I plugged it in the thing still managed to drain itself and power off. Yes, it powered itself off while still plugged in. I had to unplug and plug it back in to get it to start charging again.

Another engineer at my company had a similar problem a few weeks back and I talked to him about it and he set to reset the iPhone and delete all settings and content. I did this, which took about an hour, and when it came back up it wanted to be synced with iTunes, after all that it seems to have charged and is ok now.

Long story short is there definitely seems to be a software glitch since erasing my iPhone seems to have fixed the issue. One item of note is that when I first plugged it in my iTunes prompted me to restore from backup but I chose to just start over.

Today’s post is a blast from the past. While studying for the CCNP I came across the CEF section which reminded me of my first introduction to CEF. It started with a report that users that some servers were down. The helpdesk was very confused because when they tried to talk to the servers it worked. The first technician tried to take control of the user’s workstation but couldn’t yet another technician could.

When I heard of the issue I knew what the root problem was. The location in which the user was at had dual equal cost paths and one of the paths was down. What made me pause for a second was the network engineer asking me a very simple question “how can I tell which path will be selected?”. It was a perfect case of knowing how a technology worked but not how to prove what I was seeing.

The first place to start is checking to make sure you actually have redudant paths recognized by CEF and the routing table. To start with you can check the CEF table and issue the following command:

As you can see there are two entries for the 192.168.1.0 network. You can verify that both routes are in the routing table by taking the network and doing the following:

This output shows you the two routes and the details about the routes. The one with the * next to it will be the default selected path. Now that you know there are in fact dual paths you can get more details about the CEF effects. The first command to enter will be:

The important part to get from this output is “per-destination” this tells you which type of load balancing is in use. This tells you that the path is selectd on a per-destination basis instead of changing it up on a per-packet basis. This is very important. If you are passing order sensative data such as voice packing arriving out of order is very bad. As such you want to make sure per-destination is selected. The way load balancing is done is there are 16 bit buckets created inside the router and those bit buckets are evenly split up between the avaialble paths. To see how the buckets are distributed enter the following command:

Now that you have verified that dual paths exist and you understand what method the router is using to select paths the question that arises is how can I see which path will be used under certain conditions. This can be done via the following command:

The first argument is the source IP and the second is the destination IP. This lets you ask the router if it were to process a packet with that source and that destination which path it would select. As you can see with this example with the destination changing the link selected has changed.

Under the conditions I created in my lab and the output shown this scenario would not exist. That is beause the routes are populated via EIGRP and as such when one of the paths is broken the routes will be automatically removed and the path depricated.

Today marked the start of my CCNP studying. The first topic I found on the Blue Print was EIGRP. While I have used EIGRP for years now I have to admit did not fully understand how the protocol worked and how to get the most out of it. Like I did with my CCVP I will be making posts of useful information.

Neighbor Table

Each router is responsible for keeping state information about their adjacent neighbors. When a neighbor is discovered the address and interface of the neighbor is recorded. Each of the protocol dependent modules has their own neighbor table. During the Hello phase a HoldTime is advertised. This time is the amount of time a router waits between Hellos before marking the neighbor as unreachable and non-operational. If this occurs DUAL is informed of the topology change. Along with this information sequence numbers are stored to help detect out of order packets. A transmission list is used to queue packets for retransmission on a per neighbor basis.

Topology Table

The Topology Table is managed by the protocol dependent modules and accessed by the DUAL finite state machine. It contains the destinations advertised by neighboring routers. Each entry is contains the destination address and a list of neighbors that advertise the destination. For each neighbor the advertised metric is recorded. This is the metric that the neighbor stores in its routing table.

The router then takes the sum of the best advertised metric from all neighbors plus the link cost to the best neighbor. This is the metric that the router uses in its own routing table.

Feasible Successors

Feasible Successors are routes that can be installed into the routing table if a route goes away. As long as the neighbor is considered downstream it will consider it a feasible successor. This allows a new route to be selected without having to re-compute the routing table.

Viewing Route Details

To view the detailed information regarding the routes in the router you will want to issue the following command. If you have two routes installed in the route table to the same destination it will tell you which route will be used.

The route with the * next to it will be selected for the traffic at that destination. By default the route selection is determined by Delay and Bandwidth. If they metric isn’t the same for two routes you will only see a single route installed. You can influence these selections by various means which I will cover in a later post.