The Silo Chronicles
The not so exciting adventures of a Network Engineer

I noticed that my CS-MARS box was no longer auto-updating its IPS signatures so I figured I would take a look at it. I am using CCO to get my updates and it was working fine but we had changed the CCO password it was using a few weeks ago and I forgot to update it here. I typed the new password in, clicked Test Connectivity and got a rejected message. After clicking on More Info it says File Not Found.

It seems that somehow CS-MARS lost the certificate for the connection and when you click “Test Connectivity” that routine will not present the certificate to you. You must click the “Update Now” button to get presented with the new certificate. Once accepted the test works fine and all is well. The moral of this story is to not rely solely on “Test Connectivity”.

One comes to learn that there are many a great commands hidden from the view within the bowels of Cisco IOS. While some know of the common uses of such a command they do not know the potential these commands hold. One such command is the almighty reload command.

Virtually every person in the world knows that in order to reboot a Cisco IOS device one issues the reload command. The thing that most of them never bother to notice is that you can actually specify several useful parameters.

Scheduling Options
There you two options you can use when scheduling a reload. The first option is the “at” command and the second is the “in” command. The “at” command lets you specify a specific point in time you wish to reload at. For example:

This specifies to reload a router at 7:00PM local time. You should always check the current time on the device to ensure your scheduled time is when you expect it to be. You can do this via “show clock”. Also as you notice the time should be entered in military format.

The second option is the “in” command. This specifies an amount of time to wait until a reload is executed. For example:

This specifies to reload a router in 65 minutes. You can enter the time in the minute format or also the hour:minute format.

Reason
As a bonus to this post I am going to include the reason parameters as well. You can enter a line at the end of any variation of the reload option. This reason is passed on to the logs and can be used for justifications at a later time.

A year ago I took the CCVP QoS test to make a work goal but never really pursued the CCVP track past that test. That changed tonight when me and a buddy got together and started reviewing the CVoice exam. I have been working with Cisco VoIP for five years now and my buddy has been working with it for about a year now. We both had a fun time going over the RUF (Random Useless Fact) questions. We went through a little over a hundred questions on a review guide we had. I quickly found that going thru this material is going to create endless amounts of source for various blog posts. Today’s post is about MOS.

Mean Opinion Score is a metric designed to grade the quality of a voice call. The valid ranges are 1 to 5 with 1 being bad and 5 being excellent. Each codec has a MOS score associated with it. The common Cisco codecs are as follows:

A quick search of Google will give you a complete list of codecs and their MOS.

After moving to CUCM 6.1 I lost my ability to directly access the CDR records and started having to rely on the CAR application. So far it hasn’t given me much grief but I did discover something recently that I figured I would share. If you want to get more then 100 records outputted you must export the data. After doing an export I noticed that after about 3/4 of the way thru the file I suddenly lost my dateConnect and duration fields, they were all 0. I did some larger range exports and found the same thing.

After doing some poking around I discovered that there were no problems with the data just how Cisco exports the data. On our cluster we log zero duration calls. It seems that when you export CDR records Cisco exports non-zero duration calls in order and then zero duration calls in order, thus the last part of your file appears to be in order expect with no durations. It took me a minute to catch on to this because of the lovely epoch time they export into.

Moral of the story is make sure to resort the file based off globalCallID_callID if you want to see all the records in order.

When it comes to open source network tools you will find a fairly wide selection. However a few months back I discovered one particular tool that I feel stands out. ZipTie is a client/server device management tool. You install a server and then set it up via the client. Once you have it populated you can run a variety of tools and custom command jobs against the devices. They support a very wide range of products and there are a large number of custom jobs and tools available in their ZipForge. The recent version that was released includes some pretty nice improvements including reports, tagging and job device selection improvements. I highly recommend you give it a whirl.

NOTE: Make sure to use perl 5.8 instead of 5.10. Perl 5.10 has some issues with ZipTie.

I got to start my morning off with figuring this out. Apparently someone is disgruntled with the company I work for and they are calling various numbers harassing people. Apparently they called one person 31 times in an hour. Needless to say people are annoyed. After some quick poking around I came up with a clean solution that is pretty slick. Basically you have to create a translation-rule, translation-profile and add two lines to your inbound DID dial-peers you want the block to be used on. For example:

The translation-rule is pretty straight forward. Any translation regular expression will work here. Be careful though as you want to only block particular people. The profile is where you tell the rule to bind. You can use either calling for the caller or called for the DID called. You can actually block both DIDs and Callers via the same profile. The dial-peer part is the part where you tie it to the profile and also what message you play when a match is found.

That is all there is to it. I had some fun playing around with it and yet again was able to be the hero of the day by stopping a person from annoying people.

I received word back from Cisco on the issue I discovered last week between CS-MARS and IPS 6.1. It seems that CS-MARS is currently using RDEP protocol to perform the Test Connectivity function to Cisco IDS/IPS devices. Starting in IPS v6.1 RDEP is now disabled by default (it was replaced with SDEE starting in IDS 5.0/IPS 5.1). The reason the functionality of it still worked is that CS-MARS uses SDEE to pull data not RDEP.

The CS-MARS BU has flag this issue and said the fix for it will be released in 6.0.1. As a work around you can just ignore the Test Connectivity button or simply enable RDEP. To enable RDEP do the following:

The more prompt is something that we all encounter on a daily basis. Whether you are looking at the configuration or some sort of output we all run into it on a regular basis. The one thing that Cisco doesn’t seem to advertise is that you can do more then press enter and space. In fact there are some very useful options you can use. Below is a table of some of the options:

With IOS devices pretty much any key quits the display however with ASA you must use the q key to quit the output. If you happen across more options please let me know!

I upgraded an IDSM-2 and an AIP-SSM-10 today to the new 6.1. Let me start by saying finally. They finally added an auto-update signatures from CCO. Along with the new release comes a new release of IME and wow is it spiffy. The purpose of this quick entry though isn’t to go one about how nice the new release is but instead give you a heads up to an issue I discovered.

There appears to be an issue with CS-MARS (5.3.4) integrating with the IPS 6.1 version. When you click test connectivity it will prompt you for a new certificate like normal but after clicking yes it will tell you it couldn’t talk to the sensor. The thing is that it is actually still talking to the sensor. I spoke with Cisco about it today and they were able to replicate the issue and acknowledge it is a bug and that it is cosmetic only.

I am not sure which CS-MARS versions are affected other than 5.3.4. Once they give me the BugID I will upset this post.

UPDATE: Bug ID CSCsq07003

One of the frustrations of running a corporate network is what I call the ninety percent syndrome. It seems that once I get a project about ninety percent done I get pulled off the project to start working on something else. By the time it is ninety percent typically it is fully working it just isn’t tweaked, compared to best practices and fully documented. This is one such story and how just because it works doesn’t mean it is working the way it should be.

When I started working for my current company we were running EIGRP over our Frame Relay WAN network. A year later we moved to a MPLS network. This introduced BGP where before we had simply run EIGRP. At the time I was working several other major projects and I had never really even looked at the EIGRP deployment as I was more than busy with various projects and “it just worked”. Our service provider gave us some snippets and again “it just worked”. We were having some fairly serious fragmentation and QoS issues with the MPLS network and by the time I was able to get those resolved I was way behind schedule and I found myself pulled off the project. Our routing protocol implementation ended up on my list of follow-up items and it sat there for the next three years.

Flash forward to this week. We recently added a new engineer who has been doing really good. This with the combination of finally finishing my existing network re-design projects left me at a point I could finally sit down with our lab equipment and start playing.

The first thing I wanted to accomplish was to black hole RFC 1918 networks that were not present on our network. Over the last year we started seeing an increasingly amount of random RFC 1918 network traffic hitting our firewalls. Basically our laptop community is finally getting more network attached devices at their homes, installing the devices on the work laptops and lucky us we get see the constant attempts to talk to these devices while their laptop is on our network. I figured this was easy enough and I would be able to implement this with very little work. My plan was to simply add the following lines to each of our edge routers to kill off the traffic as close to the source as possible.

ip route 10.0.0.0 255.0.0.0 null0
ip route 172.16.0.0 255.240.0.0 null0
ip route 192.168.0.0 255.255.0.0 null0

These are very generic and as long as a more specific route is found it will use that. To make sure more specific routes were present I ran a quick show ip route on one of our edge routers and what I found made me sigh. Our EIGRP deployment used a network 10.0.0.0 statement which picked up all the locally connected networks on each router. The thing is that the BGP snippet wasn’t using any network statement, instead it was using a redistribute eigrp command. What resulted was every network from every router on our network was sharing every connected route. Long story short our network which consists of 19 locations (total of about 40 buildings) generating a BGP table with over 370 entries.

While this number isn’t outrageous and easily handled I was fairly embarrassed. I had completly redesigned the ip numbering system to base the entire system off subnets to easily be able to separate locations into subnet boundaries and I had forgotten to even look at routing and leveraging these improvements in our routing table. The ninety percent syndrome strikes again!

The first thing I wanted to do was to consolidate the BGP entires. There is no need to distribute every network on a router when all the routes are within a single subnet boundary such as 10.25.0.0/16. As you know unlike EIGRP BGP must find the exact network specified in the routing table before it will distribute it. To accomplish this I setup a static route such as:

ip route 10.25.0.0 255.255.0.0 null0

With this statement in the router I was able to use a network 10.25.0.0 mask 255.255.0.0 statement in my BGP section. This resulted in consolidating on average 20 networks in the BGP table into a single BGP entry. The beauty is that again the most specific route always wins. Even though I am null routing the entire range all of these networks have a more specific route on the router so by the time the packet receives the specified router the null0 route never gets matched for valid networks at the location.

Using this method I should be able to shrink our BGP table from over 370 entries to less than 80 entries. I am a very big proponent of the simplicity is golden principal. Our networks are getting more and more complex technologies everyday so whenever possible keep things as simple as possible.