The Silo Chronicles
The not so exciting adventures of a Network Engineer

As anyone who has ever applied an access-list to an interface knows you must specify the direction in which you wish to apply the access-list.  This direction is used to determine which side is to be used as the source or destination.  While this sounds fairly straight forward it remains a black magic area for many engineers.  Today we are going to demystify this directionality and make it simple and straight forward.

Background
The first thing we must acknowledge is that unicast communications is always between two parties.  As such there is always a direction that communications is flowing.  For the purpose of this post we are going to define those directions as local and remote.  The local side is the interface that has the access-list on it and the remote side is whoever is on the other end of that interface.

Since all unicast communications is between two parties there is always a source and a destination.  These are the terms used within the access-list creation process.  The question is which side local or remote is the source and which side local or remote is the destination.  This is where the direction statement you specify on the interface comes into play.

Which Direction Am I Going?
If you apply the direction IN on an interface’s access-list the interface (local side) considers the source address to be the remote side.  If you apply the direction OUT on an interface’s access-list the interface (local) considers the destination access to be the remote side.  Thus the packet is flowing IN from the remote side or OUT to the remote side.

Why is this important?
Directionality is very critical into the effectiveness of access-lists.  If you do not understand which side is the source or destination your access-lists might result in being more open than originally intended or even worse longer than necessary.

This is related to yesterday’s post about Cisco changing the phone firmware signatures in 8.5.2(SR1).  To quickly recap the underlying issue is that a phone with firmware prior to 8.5.1 trying to connect to a system with firmware 8.5.2(SR1) or later will fail it’s upgrade due to “Auth Fail”.  This is because Cisco changed the firmware package signature in 8.5.2(SR1) and only firmware 8.5.1 and 8.5.2 have the new signature as well as the old signature.  This means you must upgrade to 8.5.2 before you can upgrade to 8.5.2(SR1) or later.

This trickles down to a factory defaulted phone that runs a firmware prior to 8.5.1.  The way factory defaulting works is once you default the phone it knows to request a specific file from the TFTP server to figure out what versions of what to load.  This file is named “termXX.default.loads”.  The XX is the two digit model number in question. The problem is created when a device pack or service pack is installed on the server that includes firmware 8.5.2(SR1) or newer.  Along with all the firmware files the termXX.default.loads files are overwritten automatically by the latest version installed.

This means if you install 8.5.4 but go to Device Defaults and specify the 8.5.2 version it won’t matter for a factory defaulted phone because the termXX.default.loads file is from 8.5.4 pack.  The reverse is also true.  If you are running 8.5.4 but due to this issue go and install 8.5.2 your termXX.default.loads will now reflect 8.5.2 so all defaulted phones will also go to 8.5.2 and then have to upgrade again to 8.5.4 even if they were already at 8.5.4 previously.  This means if you install a downgraded version of firmware you need to make sure to go back and re-upload the currently installed versions of the termXX.default.loads files.

Checking Your Default Loads Files
This is actually pretty straight forward since the file is in plain text.  Simply use a TFTP client and request the “termXX.default.loads” file for the model of phone you are curious about.  Open it in a text editor and you should see the filenames it will instruct the phone to request.  From those filenames you should see the version number.

Downloading A Different Version of Default Loads Files
If you were like me when you first discovered you found that you needed an 8.5.2 file but the file reference 8.5.4 and the 8.5.2 was already installed.  This isn’t an issue since you can get the correct file from CCO.  If you haven’t installed the 8.5.2 simply backup (Via TFTP) the current version of the file, install the old version and then re-upload the backed up version of the file to restore it to the original state.  If you don’t need the firmware just the file simply go to CCO and find the phone in question.  There should be three files in each model’s area.  One is a .sgn which is for CUCM 5+, One is a .exe for CCM 4.x and the last is a .zip which says firmware files only.  You want to download the .zip firmware files only one.  Once you download it unzip it and you will find copies of your default loads files.

Replacing the Default Loads File
You need to log into the OS Administration section and then go to Software Updates/TFTP File Management.  If you search for termXX you can find the existing file there.  Simply upload the new version and it will automatically over-write the existing version of the file.

NOTE: Make sure you backup the existing one first so you can restore it later.

NOTE: You must restart the TFTP server after uploading the file for the new version to be offered.

You’re Done
At this point the phones that won’t boot should request the new termXX.default.loads file and download the 8.5.2 version of firmware.  Once that is done they should request the latest version assuming the device defaults references a newer version.

It’s been awhile since I last posted so I figured I would start with a command of the day post. Today’s post is actually going to be several commands. These are commands that are not very common but I find useful when working with the Cisco Autonomous Access Points.

dot11 arp-cache [optional]
This command is entered in the global configuration mode. It enables client ARP caching on the access point. ARP caching on the access point reduces the traffic on your wireless LAN and increases client battery life by stopping ARP requests for client devices at the access point. Instead of forwarding ARP requests to client devices, the access point responds to requests on behalf of associated client devices and drops ARP requests that are not directed to clients associated to the access point. When ARP caching is optional, the access point responds on behalf of clients with IP addresses known to the access point but forwards through its radio port any ARP requests addressed to unknown clients. When the access point knows all the IP addresses for associated clients, it drops any ARP requests not directed to its clients. In its beacon, the access point includes an information element to alert client devices that they can safely ignore broadcast messages to increase battery life.

dot11 [Interface] carrier busy
This has to be one of my favorite commands for deploying new access points. This command is entered in the EXEC mode and will cause the access point to do a carrier busy test on each channel resulting in a report showing you how busy each of the channels are allowing you to select a free channel. Two things to note:

1. This will disassociate users from the access point so make sure it isn’t in production or people are aware of an outage before issuing the command.

2. To see the report you need to issue a “show log” command or if connected via console console logging enabled.

show dot11 associations
This command will show you a list of devices currently associated with the access point. You might find it useful to make this an alias on the access point.

Last night I was had a simple task of adding a new 3750 to an existing stack. I went in thinking it would be a cake walk as they auto-upgrade and basically I was just going to have to rack it and connect some stack cables. Like most things recently there was more to it. This customer happen to have a very old 1.5U 3750 with 16mb of flash. It happened to be the master of the stack as well. This proved to be very annoying when trying to get the switch in the stack. The first problem was the current revision of code was too old and since the new switch was a later hardware revision it refused to auto-upgrade since it wasn’t sure it could support it. The customer used the web interface so I did the archive upgrade reloaded the stack and when it came back up only the first two switches actually upgraded. The new switch was still on the old revision. I then attempted to manually start the upgrade only to run into “Unable to create temp dir “flash:update”" error. This was quickly resolved with a “delete /recursive /force flash:update”. Unfortunately this is when I discovered that since it only had 16mb of flash it didn’t actually have enough to store the image for auto upgrades. After about an hour and a half I said screw it, loaded the bins on them and reloaded the stack… Victory was mine!

Morals of this story:

1. Look at which switch is your master. If it is an old hardware revision be prepared for some issues
2. Check the amount of available flash on your master switch. If it is only 16mb auto-upgrade most likely will not work.
3. When push comes to shove you can also just load the .bin files on all members manually (TIP: Use copy flash1: flash2: etc… to save time when loading it between stack members)

As a Cisco Partner I have had several occasions to password recover some devices. Today brought an usual request of wanting to harvest the configuration not the password. A customer had a Cisco router provided by a 3rd party that they wanted to replace and return but unfortunately the vendor would/did not give them the information they needed to bypass the router and have the services handled directly by their equipment.

For some reason I always get a chuckle when doing these. The task is simple enough to complete. When you password recover the device it also provides you the current configuration, including passwords, except you are in enable mode and you can make any changes you deem necessary. To password recovery an IOS device do the following:

1. Connect a console cable to the device and power the device on.
2. Press CTRL-BRK to enter rommon.
3. Type “confreg 0×2142″ and press enter.
4. Type “reset” and press enter.
5. Upon booting the device will ask you to enter a configuration setup wizard. Answer NO to this question. The device is now booted with no configuration.
6. Type “enable” and press enter.
7. Type “configure memory”

At this point you should see the hostname change and if you issue a “show running-config” you will see the complete configuration, including passwords. In the case of today the vendor did not run password encryption so all their passwords were in plain text. Ruh Roh!

Today’s post is a blast from the past. While studying for the CCNP I came across the CEF section which reminded me of my first introduction to CEF. It started with a report that users that some servers were down. The helpdesk was very confused because when they tried to talk to the servers it worked. The first technician tried to take control of the user’s workstation but couldn’t yet another technician could.

When I heard of the issue I knew what the root problem was. The location in which the user was at had dual equal cost paths and one of the paths was down. What made me pause for a second was the network engineer asking me a very simple question “how can I tell which path will be selected?”. It was a perfect case of knowing how a technology worked but not how to prove what I was seeing.

The first place to start is checking to make sure you actually have redudant paths recognized by CEF and the routing table. To start with you can check the CEF table and issue the following command:

As you can see there are two entries for the 192.168.1.0 network. You can verify that both routes are in the routing table by taking the network and doing the following:

This output shows you the two routes and the details about the routes. The one with the * next to it will be the default selected path. Now that you know there are in fact dual paths you can get more details about the CEF effects. The first command to enter will be:

The important part to get from this output is “per-destination” this tells you which type of load balancing is in use. This tells you that the path is selectd on a per-destination basis instead of changing it up on a per-packet basis. This is very important. If you are passing order sensative data such as voice packing arriving out of order is very bad. As such you want to make sure per-destination is selected. The way load balancing is done is there are 16 bit buckets created inside the router and those bit buckets are evenly split up between the avaialble paths. To see how the buckets are distributed enter the following command:

Now that you have verified that dual paths exist and you understand what method the router is using to select paths the question that arises is how can I see which path will be used under certain conditions. This can be done via the following command:

The first argument is the source IP and the second is the destination IP. This lets you ask the router if it were to process a packet with that source and that destination which path it would select. As you can see with this example with the destination changing the link selected has changed.

Under the conditions I created in my lab and the output shown this scenario would not exist. That is beause the routes are populated via EIGRP and as such when one of the paths is broken the routes will be automatically removed and the path depricated.

Today marked the start of my CCNP studying. The first topic I found on the Blue Print was EIGRP. While I have used EIGRP for years now I have to admit did not fully understand how the protocol worked and how to get the most out of it. Like I did with my CCVP I will be making posts of useful information.

Neighbor Table

Each router is responsible for keeping state information about their adjacent neighbors. When a neighbor is discovered the address and interface of the neighbor is recorded. Each of the protocol dependent modules has their own neighbor table. During the Hello phase a HoldTime is advertised. This time is the amount of time a router waits between Hellos before marking the neighbor as unreachable and non-operational. If this occurs DUAL is informed of the topology change. Along with this information sequence numbers are stored to help detect out of order packets. A transmission list is used to queue packets for retransmission on a per neighbor basis.

Topology Table

The Topology Table is managed by the protocol dependent modules and accessed by the DUAL finite state machine. It contains the destinations advertised by neighboring routers. Each entry is contains the destination address and a list of neighbors that advertise the destination. For each neighbor the advertised metric is recorded. This is the metric that the neighbor stores in its routing table.

The router then takes the sum of the best advertised metric from all neighbors plus the link cost to the best neighbor. This is the metric that the router uses in its own routing table.

Feasible Successors

Feasible Successors are routes that can be installed into the routing table if a route goes away. As long as the neighbor is considered downstream it will consider it a feasible successor. This allows a new route to be selected without having to re-compute the routing table.

Viewing Route Details

To view the detailed information regarding the routes in the router you will want to issue the following command. If you have two routes installed in the route table to the same destination it will tell you which route will be used.

The route with the * next to it will be selected for the traffic at that destination. By default the route selection is determined by Delay and Bandwidth. If they metric isn’t the same for two routes you will only see a single route installed. You can influence these selections by various means which I will cover in a later post.

I have been playing around with GNS3 (Dynamips) on my Windows VMWare image and I decided to get it working natively on my Mac OSX.  Unfortunately unlike the Windows installer the Mac OSX distribution does’nt include the necessary applications.  After poking around I figured out a way to get all the parts installed and working.  This post is an step by step of what I did to make it all work.

Step 1 – Apple Xcode Tools

The first thing you need to do is download and install the Apple Xcode Tools which are freely available from Apple at the following URL.

http://developer.apple.com/technology/xcode.html

Step 2 – Macports

Macports is a port application for Mac OSX. This is needed to install libpcap which is needed. The installation is pretty straight forward. Download the binary package from the following URL and install it.

http://www.macports.org/install.php

Once installed make sure you are updated by running the following command:

Step 3 – libpcap/dynamips

This is pretty straight forward and easy to do. Run the following:

Step 4 – libelf

Most likely the port install dynamips failed with a rom2c error. This package will correct that issue. Go to a temporary directory such as your Downloads and do the following:

Step 5 – dynamips

We are now ready to download a current version of Dynamips. You might want to check to see if a newer version is available. This is an example of the process:

Step 6 – Download & Install GNS3

Last but not least GNS3 itself. Simply go to the following URL, download the DMG and copy the application to your Applications folder. Once you are done point it to “/opt/local/bin/dynamips” and click Test. You should be ready to go now.

http://www.gns3.net/download

Optimizations

One of the easy things you can do to speed up booting of the IOS images is unzipping the images. To do this do the following, simply replace the filename with your filename.

Now point GNS3 to that new image and your router will boot significantly faster.

Summary

At this point everything should be working for you. If you run into problems let me know.

Anyone that decided to renew SmartNet on Cisco IP Phones has discovered that unfortunately there is no easy way to get the serial numbers of all the Cisco IP Phones that are registered to the CUCM cluster. What you can get is the IP address of the phones. If you pull the webpage up for the phones you can find the serial number for the phone. While this isn’t that big of a deal for small phone deployments if you have a medium or large deployment this simply becomes an unrealistic approach.

This is where the CUCM AXL Phone Scanner comes into play. This is a small application I wrote that polls the CUCM cluster via AXL for a list of all phones registered with the cluster and their IPs. Once it has this information it goes through the phones and extracts the serial numbers from their web pages. Obviously for this to work you need to have the web pages turned on and have access to the phone’s webpage from where you can scanning it.

I have released this application for free via my blog. Please give it a whirl and let me know if you run into bugs or have some general feedback. You can find it’s page at the link below:

http://blog.crimsonsilo.com/cucm-axl-phone-scanner/

Last week was a slow week at work and I had some time to spend on lab gear.  I playing around with IPSec/GRE tunnels and routing protocols when our principle engineer saw what I was doing and decided to challenge me with a strange question he had on his CCIE lab.  The question was:

Configure a router with 10 loopback adapters.  The first loopback should be on network 192.168.1.0/24 with the other loopbacks incrementing their networks by one (192.168.2.0/24, 192.168.3.0/24, etc).  Once configured ensure that only the odd numbered routes were injected into EIGRP by adding only a single line to the EIGRP block and a single line ACL.

I first laughed at how off the wall the question was and then I spent a little bit thinking about it.  Obviously the single line in the EIGRP block is a distrubution list but a single line ACL For only odd networks that took me a bit to figure out.

With ACLs you classify traffic using wildcards which are bit based selections.  Normally you use wildcards in the subnet portion of the ACL statement.  Wanting to distribute only odd routes means the routes need to be classified into a single bit statement.  That is when I remembered that wild cards are not restricted to just the subnet portion but can be used in the network portion as well.  I popped the following configuration into the router and it worked like a charm.

All in all it was a very simple question but it was exactly the kind of questions I like the best.  Simple answer but something that you don’t typically do everyday.