All this came about due to us being acquired by another company and needing a way to extend our L3VPNs to their network quickly.

Our simple topology follows. It’s a 7200 that has a customer VRF, a 7600, and a Juniper that also has the other side of the customer VRF (small lab).

One of the things we wanted to do is limit the vpnv4 prefixes that are shared between ASNs. We didn’t want anyone to have to configure anything on our 7609. To do this, we came up with additional route targets to add to the vpn routes.

Our 7200 config looks like this

ip vrf NNI-TEST1
 rd 65100:1001
 route-target export 65100:1001
 route-target export 65100:65200
 route-target import 65100:1001

interface Loopback101
 ip vrf forwarding NNI-TEST1
 ip address 10.10.10.1 255.255.255.255

router bgp 65100
 address-family ipv4 vrf NNI-TEST1
  no synchronization
  redistribute connected
  redistribute static
 exit-address-family

Pretty simple stuff. You can see where we add the extra route-target in the vrf (65100:65200).

Our 7609

router bgp 65100
 neighbor NNI:US-TO-THEM peer-group
 neighbor NNI:US-TO-THEM remote-as 65200
 neighbor x.x.156.127 peer-group NNI:US-TO-THEM
 address-family ipv4
 no neighbor NNI:US-TO-THEM activate
 no neighbor x.x.156.127 activate
 address-family vpnv4
 neighbor NNI:US-TO-THEM activate
 neighbor NNI:US-TO-THEM next-hop-self
 neighbor NNI:US-TO-THEM send-community both
 neighbor NNI:US-TO-THEM route-map NNI:IMPORT in
 neighbor NNI:US-TO-THEM route-map NNI:EXPORT out
 neighbor x.x.156.127 peer-group NNI:US-TO-THEM

ip extcommunity-list 101 permit _RT:65100:65200_
ip extcommunity-list 102 permit _RT:65200:65100_

route-map NNI:IMPORT permit 10
 match extcommunity 102
!
route-map NNI:IMPORT deny 90

route-map NNI:EXPORT permit 10
 match extcommunity 101
!
route-map NNI:EXPORT deny 90

Finally, the Juniper

set interfaces lo0 unit 250 family inet address 10.10.10.2/32

set protocols mpls interface lo0.250
set protocols mpls interface fe-0/3/0.2500
set protocols mpls interface lo0.0

set routing-options autonomous-system 65200
set protocols bgp keep all
set protocols bgp group NNI:PEER type external
set protocols bgp group NNI:PEER family inet-vpn unicast
set protocols bgp group NNI:PEER neighbor x.x.156.126 import NNI:IMPORT
set protocols bgp group NNI:PEER neighbor x.x.156.126 export NNI:EXPORT
set protocols bgp group NNI:PEER neighbor x.x.156.126 peer-as 65100

set policy-options policy-statement CUST1-export term 1 then community add NNI:EXPORT
set policy-options policy-statement CUST1-export term 1 then community add CUST1:RT
set policy-options policy-statement CUST1-export term 1 then accept
set policy-options policy-statement CUST1-export term 2 then reject
set policy-options policy-statement CUST1-import term 1 from community CUST1:RT
set policy-options policy-statement CUST1-import term 1 then accept
set policy-options policy-statement CUST1-import term 2 then reject
set policy-options policy-statement NNI:IMPORT term 1 from community NNI:IMPORT
set policy-options policy-statement NNI:IMPORT term 1 then accept
set policy-options policy-statement NNI:IMPORT term 2 then reject
set policy-options policy-statement NNI:EXPORT term 1 from community NNI:EXPORT
set policy-options policy-statement NNI:EXPORT term 1 then accept
set policy-options policy-statement NNI:EXPORT term 2 then reject

set policy-options community CUST1:RT members target:65100:1001
set policy-options community NNI:IMPORT members target:65100:65200
set policy-options community NNI:EXPORT members target:65200:65100

set routing-instances CUST1 instance-type vrf
set routing-instances CUST1 interface lo0.250
set routing-instances CUST1 route-distinguisher 65100:1001
set routing-instances CUST1 vrf-import CUST1-import
set routing-instances CUST1 vrf-export CUST1-export

Now, let’s check this out on our 7200.

NW-7206-2#sh ip route vrf NNI-TEST1 10.10.10.2

Routing Table: NNI-TEST1
Routing entry for 10.10.10.2/32
  Known via "bgp 65100", distance 200, metric 0
  Tag 65200, type internal
  Last update from x.x.156.127 00:34:23 ago
  Routing Descriptor Blocks:
  * x.x.156.127 (default), from x.x.156.2, 00:34:23 ago
      Route metric is 0, traffic share count is 1
      AS Hops 1
      Route tag 65200
      MPLS Required

Woot!

NW-7206-2#ping vrf NNI-TEST1 ip 10.10.10.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.10.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms

From our Juniper:

junos@NNI-TESTING# run ping routing-instance CUST1 10.10.10.1
PING 10.10.10.1 (10.10.10.1): 56 data bytes
64 bytes from 10.10.10.1: icmp_seq=0 ttl=255 time=0.943 ms
64 bytes from 10.10.10.1: icmp_seq=1 ttl=255 time=0.988 ms
64 bytes from 10.10.10.1: icmp_seq=2 ttl=255 time=0.911 ms
64 bytes from 10.10.10.1: icmp_seq=3 ttl=255 time=0.990 ms
^C
--- 10.10.10.1 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.911/0.958/0.990/0.033 ms

junos@NNI-TESTING# run show route table CUST1

CUST1.inet.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.10.10.1/32      *[BGP/170] 00:34:58, localpref 100
                      AS path: 65100 ?
                    > to x.x.156.126 via fe-0/3/0.2500, Push 57
10.10.10.2/32      *[Direct/0] 11:28:14
                    > via lo0.250

And, on our 7609

NW-7609-1#sh ip bgp vpnv4 rd 65100:1001
BGP table version is 134, local router ID is x.x.156.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 65100:1001
*>i10.10.10.1/32    x.x.156.7              0    100      0 ?
*> 10.10.10.2/32    x.x.156.127                          0 65200 i

Here’s the l3vpn bgp table on the Juniper

junos@NNI-TESTING# run show route table bgp.l3vpn.0 extensive

bgp.l3vpn.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)
65100:1001:10.10.10.1/32 (1 entry, 0 announced)
        *BGP    Preference: 170/-101
                Route Distinguisher: 65100:1001
                Next hop type: Router, Next hop index: 541
                Next-hop reference count: 3
                Source: x.x.156.126
                Next hop: x.x.156.126 via fe-0/3/0.2500, selected
                Label operation: Push 57
                State: <Active Ext>
                Local AS:  65200 Peer AS: 65100
                Age: 44:42
                Task: BGP_65100.x.x.156.126+179
                AS path: 65100 ?
                Communities: target:65100:65200 target:65100:1001
                Import Accepted
                VPN Label: 57
                Localpref: 100
                Router ID: x.x.156.2
                Secondary Tables: CUST1.inet.0

65100:1001:10.10.10.2/32 (1 entry, 1 announced)
TSI:
Page 0 idx 0 Type 1 val 14845048
        *Direct Preference: 0
                Next hop type: Interface
                Next-hop reference count: 2
                Next hop: via lo0.250, selected
                State: <Secondary Active Int>
                Age: 11:20:01
                Task: IF
                Announcement bits (1): 0-BGP RT Background
                AS path: I
                Communities: target:65200:65100 target:65100:1001
                Primary Routing Table CUST1.inet.0

This was pretty simple. So much so that I was over thinking it for quite a bit. Hopefully this helps someone else in the future.

For the second message about line 102, that’s easy enough to fix. Search for line 103 in the Reconfig.pm module and comment out lines 103 through 106.

Essentially, change this:

                                # this really shouldn't happen.  But it does.
                               die unless $prev eq "!\n" || $prev =~ /^!.*<removed>$/;
                               die unless $indent == 0;
                               $ciscobug = 1;
                               $indent = $in;

To this:

                               # this really shouldn't happen.  But it does.
                               #die unless $prev eq "!\n" || $prev =~ /^!.*<removed>$/;
                               #die unless $indent == 0;
                               #$ciscobug = 1;
                               #$indent = $in;

I am not too familiar with the Cisco::Reconfig code, but I have been running scripts with this modification against 600+ routers for over a year without issues. If uncommented, it always crashes against our GSRs. It must be left over code from some old IOS configuration quirk (Cisco never changes stuff around from version to version!). Cisco’s anti-automation engineers will probably see this and the next IOS release will break all my scripts now.

The second error about line 212 always seems to happen when you’re searching for route-maps or ip access-lists (not regular access-lists). Basically, anything where you have a sub query such as:

my @EACL = $::MYCONFIG->get('ip access-list extended SOMEACL', 'permit')->alltext;

If the SOMEACL ip access-list doesn’t exist, Cisco::Reconfig bombs out with the error at line 212 because it doesn’t have any error handling for this situation. Line 212 has an explicit “die” if the two sequences aren’t true. Doesn’t even print a message – just crashes.

So, if you run into this, you can work around it by doing your own testing first. I’ll show you how I test for whether an ip access-list exists before deciding to try to get the following lines.

                                my $eaclline = $::MYCONFIG->get($aclcmd);
                                chomp ($eaclline);
                                if ($eaclline !~ /$aclcmd/)
                                {
                                        $::REPORTID = 14;
                                        $::MESSAGE = "CRITICAL: Missing '$aclcmd'!";
                                        InsertLogMessage();
                                        ClearERRs();
                                        next;
                                }

In the above code, $aclcmd is pulled from my database and basically would look like ‘ip access-list standard SSM’. All I care about is searching for that one line before doing a more complex search. If the output doesn’t match the $aclcmd variable, we report an error, insert it into the report table, and move on.

If it does exist, we can compare the contents of the ip access-list.

                                if ($deny == 1)
                                {
                                        my @EACLDt = $::MYCONFIG->get($aclcmd, 'deny')->alltext;
                                        foreach (@EACLDt)
                                        {
                                                my $line1 = $_;
                                                push @aclrestemp, split (/\n/, $line1);
                                        }
                                }
                                if ($permit == 1)
                                {
                                        my @EACLPt = $::MYCONFIG->get($aclcmd, 'permit')->alltext;
                                        foreach (@EACLPt)
                                        {
                                                my $line2 = $_;
                                                push @aclrestemp, split (/\n/, $line2);
                                        }
                                }

If you’re curious about the complete code for this function, here it is.

                my $aclsql = "SELECT extended,acl,name,contents FROM AccessLists WHERE chassisid = '$::CHASSIS' AND rtypeid = '$::TYPE'";
                my $aclqry = $dbh->prepare($aclsql);
                $aclqry->execute();

                while (@aclrow = $aclqry->fetchrow_array())
                {
                        my $isExtended  = $aclrow[0];
                        my $aclcmd      = $aclrow[1];
                        my $aclname     = $aclrow[2];
                        my $aclcontents = $aclrow[3];

                        my $deny = 0;
                        my $permit = 0;

                        my @aclarr;
                        my @acltmp = split('\n', $aclcontents);
                        for (@acltmp)
                        {
                                my $aclline = $_;
                                chomp ($aclline);
                                $aclline =~ s/^\s+|\s+$//g;
                                if ($aclline =~ /^deny/)
                                {
                                        $deny = 1;
                                }
                                if ($aclline =~ /^permit/)
                                {
                                        $permit = 1;
                                }
                                push @aclarr, $aclline;
                        }

                        my @aclrestemp;

                        if ($isExtended == 0)          ### regular accesss-list
                        {
                                @aclrestemp = $::MYCONFIG->get($aclcmd)->all;
                        }
                        if ($isExtended == 1)          ### ip access-list
                        {
                                my $eaclline = $::MYCONFIG->get($aclcmd);
                                chomp ($eaclline);
                                if ($eaclline !~ /$aclcmd/)
                                {
                                        $::REPORTID = 14;
                                        $::MESSAGE = "CRITICAL: Missing '$aclcmd'!";
                                        InsertLogMessage();
                                        ClearERRs();
                                        next;
                                }

                                if ($deny == 1)
                                {
                                        my @EACLDt = $::MYCONFIG->get($aclcmd, 'deny')->alltext;
                                        foreach (@EACLDt)
                                        {
                                                my $line1 = $_;
                                                push @aclrestemp, split (/\n/, $line1);
                                        }
                                }
                                if ($permit == 1)
                                {
                                        my @EACLPt = $::MYCONFIG->get($aclcmd, 'permit')->alltext;
                                        foreach (@EACLPt)
                                        {
                                                my $line2 = $_;
                                                push @aclrestemp, split (/\n/, $line2);
                                        }
                                }
                        }

                        my @aclres;
                        for (@aclrestemp)
                        {
                                my $restemp = $_;
                                $restemp =~ s/^\s+|\s+$//g;
                                chomp ($restemp);
                                push @aclres, $restemp;
                        }

                        # some quantum sexiness going on here
                        my @fACL = eigenstates(any(@aclres) ne all(@aclarr));
                        my @rACL = eigenstates(any(@aclarr) ne all(@aclres));

                        my $fcount = @fACL;
                        my $rcount = @rACL;

                        # If the first element is empty, don't try to insert stuff into the logs
                        if ($fACL[0] ne "")
                        {
                                for (@fACL)
                                {
                                        $::REPORTID = 14;
                                        $::MESSAGE = "$aclname extra: $_";
                                        InsertLogMessage();
                                }
                        }

                        if ($rACL[0] ne "")
                        {
                                for (@rACL)
                                {
                                        $::REPORTID = 14;
                                        $::MESSAGE = "$aclname missing: $_";
                                        InsertLogMessage();
                                }
                        }

                        ClearERRs();
                }
        }

Our AccessList table looks like this (click to enlarge):

Because we have some “Core” routers that are GSRs and some that are 7600s, the table has a chassisid and rtypeid. That allows us to have different ACLs for platforms with different functions.

The acl column is the actual command that is searched for in the $aclcmd variable. The name column is used when inserting an error message into our report table. Finally, the contents column is a text type and contains the official access-list used to audit against.

I hope this post helps those of you frustrated by the cryptic errors with Cisco::Reconfig. It is very useful, but it would be nice if it actually told you why it crashed. Even better would be if it had more enhanced error handling. Unfortunately, since it was last updated 3 years ago (Feb 2007), it seems to be abandoned. A shame for such a valuable module.

If anyone stumbles on this post and has a different error than the two identified, please leave a comment and I’ll see what I can do to help you out.

rtrcommander can be downloaded here: http://www.synacknetworks.com/scripts/rtrcommander.txt

rtrcommander can be used as follows:

pts/4 jrowley@toolbox:/home/audit/scripts/rtrcmd $> rtrcommander

New and Improved Router Commander 2.0

        Usage:
        ./rtrcommander [-h] -u <username> [-p <password>] -r <rtrlist> -c <cmdlist> [-l <loglocation>]

        -h                : prints this message
        -u                : username
        -p                : password - if not specified, will be prompted
        -r                : file containing list of routers
        -c                : file containing commands to run
        -l                : file where we should log to; defaults to "ipaddress.log"

        Examples:
        rtrcmd -u username -p password -r routerlist -c commandlist
        rtrcmd -u username -r routerlist -c commandlist -l mycombinedlogfile.txt

For security, if a password is not specified on the command line, you will be prompted for one.

This shows how we retrieved memory being held by the Logger process while checking to see our wide spread our memory leak was (see yesterday’s post).

pts/4 jrowley@toolbox:/home/audit/scripts/rtrcmd $> ./rtrcommander -u jrowley -r routers -c commands -l mylog.txt
password: ********

HOSTNAME: router1
Sending: show conf | i ^hostname
Sending: show proc mem sort | i Holding|Logger|Malloc

HOSTNAME: router2
Sending: show conf | i ^hostname
Sending: show proc mem sort | i Holding|Logger|Malloc

HOSTNAME: router3
Sending: show conf | i ^hostname
Sending: show proc mem sort | i Holding|Logger|Malloc

We create two files. One is a list of IP addresses or hostnames of routers we want to run these commands on. The second is a list of commands to run.

routers file:

router1
router2
router3

command file:

show conf | i ^hostname
show proc mem sort | i Holding|Logger|Malloc

If you want to push out some configuration, your command file may look similar to:

config t
ip access-list standard 1
permit 10.20.114.0 0.0.1.255
permit 10.20.164.0 0.0.3.255
end
quit

The combined log file is optional. If you leave it off, it will log everything into one file per router. If you have a small set of routers, you can use either way. Since I typically run stuff against our 600+ routers, I typically combine everything into one log file, otherwise my home directory gets pretty cluttered.

Our combined log file looks like:

pts/4 jrowley@toolbox:/home/audit/scripts/rtrcmd $> more mylog.txt
hostname router1
 PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
  44   0   15619876     170496    8664528      50760          0 Logger
   0   0          0          0    7257808          0          0 *MallocLite*
hostname router2
 PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
  44   0  393743528        188  180671388      71064          0 Logger
   0   0          0          0  139126148          0          0 *MallocLite*
hostname router3
 PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
   0   0          0          0     393528          0          0 *MallocLite*
  44   0   10641592   10607792      30148      10152          0 Logger
pts/4 audit@toolbox:/home/audit/scripts/rtrcmd $>

Non combined logging provides just a bit more details since we use raw Net::Telnet logging

pts/4 jrowley@toolbox:/home/jrowley $> more rtrcmds/2009-11-06/router.log

router line 2

User Access Verification

Username: jrowley
Password:

router#term len 0
router#config t
Enter configuration commands, one per line.  End with CNTL/Z.
router(config)#ip access-list extended OSPF_ROUTES_VOICE
router(config-ext-nacl)#permit ip 10.20.114.0 0.0.1.255 any
router(config-ext-nacl)#permit ip 10.20.164.0 0.0.3.255 any
router(config-ext-nacl)#end
router#quit

This is useful for pushing out the same configuration changes to many routers, or to run commands to capture output, such has how much memory the Logging process is holding.

Hopefully this will be of use for others.

NfSen is an invaluable open source tool that can be used to identify and alert on specific attacks. It does this by receiving Netflow data from a Cisco or Juniper router.

NfSen is available from: http://nfsen.sourceforge.net
NfDump (1.5.8 or greater)  is available from: http://nfdump.sourceforge.net

Besides the above applications, you will also need the following:

• Perl > 5.6.0
• PHP > 4.1
• Mail::Header and Mail::Internet (from CPAN)
• RRDTool (for graph generation)
• a RTBH (Real Time Black Hole) system to drop the traffic as close to ingress as possible.

Once you have those installed, we can move onto plugins. By itself, NfSen gives you visibility into your network, but alone, it doesn’t alert you about anomalies. That’s where my Dynamic DDoS Detector (DDD) plugin comes in.

DDD can be downloaded from http://www.synacknetworks.com/ddd/ddd.zip. This zip file contains the Perl module (ddd.pm) and the MySQL structure.

DDD is loosely based on a plugin by John Fraizer named newddosdetect. Other than the concept, it’s been almost entirely rewritten, hence the new name.

The main reason for this plugin is that we recently deployed dedicated Netflow probes through our network. These take a feed from an inline tap and generate Netflow data, which is then relayed to our collectors. These probes work fairly well, however, they continue to see an attack even though it may have been mitigated via RTBH.

Since they continue to see the attack traffic, our collector continued to alert, causing a flood of emails until we manually edited the existing plugin and reloaded NfSen. Now, all we need to do is insert the attacker’s IP in our exclusion list.

To install, you just need to do the following steps:

1) Create the MySQL database from the ddd.mysql file. You can use the cli or phpMyAdmin.

2) Add any IP blocks you want to exclude (in CIDR format)

3) Adjust the filters, if needed

4) Drop the ddd.pm file into your NfSen plugins directory

5) Edit ddd.pm and change the following

### database login info
my $dsn         = 'dbi:mysql:ddd:127.0.0.1:3306';
my $user        = 'dbuser';
my $pass        = 'dbpass';

# Specify the notification email(s).
if ($DEBUG > 0)
{
        $emails = "user\@example.com";
        print "Emails = $emails\n\n";
}
else
{
        $emails = "distro-list\@example.com";
}

# beginning of the subject line for alerts
my $subject     = "";
if ($DEBUG > 0)
{
        $subject        = '[DEBUG] DDoS Alert:';
}
else
{
        $subject        = "DDoS Early Warning:";
}

6) Edit nfsen.conf to include the plugin.

@plugins = (
    # profile    # module
        [ 'live', 'ddd'],
);

7) Reload NfSen

bin/nfsen reload

Once done, NfSen will run the plugin every expire cycle (default is every 5 minutes). If you watch /var/log/messages, you should see it run with output similar to the following sanitized log.

Jan 11 01:45:15 hostname nfsen[9261]: 90 channels/alerts to profile
Jan 11 01:45:16 hostname nfsen[9261]: Update profile Compromised-Customers in group .
Jan 11 01:45:17 hostname nfsen[9261]: Update profile botnet in group .
Jan 11 01:45:18 hostname nfsen[9261]: Update profile live in group .
Jan 11 01:45:18 hostname nfsen[9261]: Update profile AKAMAI in group BREAKDOWN
Jan 11 01:45:19 hostname nfsen[9261]: Update profile DNS-SERVERS in group BREAKDOWN
Jan 11 01:45:20 hostname nfsen[9261]: Update profile IRC-Probable-Targets in group BREAKDOWN
Jan 11 01:45:21 hostname nfsen[9261]: Update profile Private-IPs in group BREAKDOWN
Jan 11 01:45:22 hostname nfsen[9261]: Update profile Protocols-at-a-glance in group BREAKDOWN
Jan 11 01:45:23 hostname nfsen[9261]: Update profile Services-at-a-glance in group BREAKDOWN
Jan 11 01:45:23 hostname nfsen[9477]: Plugin Cycle: Time: 201001110140, Profile: live, Group: ., Module: PortTracker,
Jan 11 01:45:25 hostname nfsen[9477]: Plugin Cycle: Time: 201001110140, Profile: live, Group: ., Module: ddd,
Jan 11 01:45:25 hostname nfsen[9477]: ddd.pm -------------------------------------------------------------
Jan 11 01:45:25 hostname nfsen[9477]: ddd.pm exclusions: (NOT net A.B.C.D/32 )
Jan 11 01:45:25 hostname nfsen[9477]: ddd.pm run: Checking Packets > 50K/flow-set
Jan 11 01:45:25 hostname nfsen[9477]: ddd.pm run: NF Filter: ((packets > 50000) AND NOT (proto tcp AND NOT (port 110 OR port 25)) AND NOT proto esp) AND (NOT net A.B.C.D/32 )
Jan 11 01:45:25 hostname nfsen[9477]: ddd.pm end: Packets > 50K/flow-set
Jan 11 01:45:25 hostname nfsen[9477]: ddd.pm -------------------------------------------------------------
Jan 11 01:45:25 hostname nfsen[9477]: ddd.pm exclusions: (NOT net A.B.C.D/32  AND NOT net A.B.C.D/32 AND NOT net A.B.C.D/30 AND NOT net A.B.C.D/23 AND NOT net A.B.C.D/22 AND NOT net A.B.C.D/24 AND NOT net A.B.C.D/23  AND NOT net A.B.C.D/23 AND NOT net A.B.C.D/23)
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: Checking UDP Packets > 80K/dest
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: NF Filter: proto udp and not out if 0 AND (NOT net A.B.C.D/32  AND NOT net A.B.C.D/32 AND NOT net A.B.C.D/30 AND NOT net A.B.C.D/23 AND NOT net A.B.C.D/22 AND NOT net A.B.C.D/24 AND NOT net A.B.C.D/23  AND NOT net A.B.C.D/23 AND NOT net A.B.C.D/23)
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: No matching flows found for UDP Packets > 80K/dest
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm end: UDP Packets > 80K/dest
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm -------------------------------------------------------------
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm exclusions: (NOT net A.B.C.D/28  AND NOT net A.B.C.D/32)
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: Checking TCP Packets > 100K/dest
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: NF Filter: proto tcp and not out if 0 AND (NOT net A.B.C.D/28  AND NOT net A.B.C.D/32)
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: No matching flows found for TCP Packets > 100K/dest
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm end: TCP Packets > 100K/dest
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm -------------------------------------------------------------
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm exclusions:
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: Checking ICMP Packets > 50K/dest
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: NF Filter: proto icmp and not out if 0
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: No matching flows found for ICMP Packets > 50K/dest
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm end: ICMP Packets > 50K/dest
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm -------------------------------------------------------------
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm exclusions: (NOT net A.B.C.D/28  AND NOT net A.B.C.D/32 AND NOT net A.B.C.D/32 AND NOT net A.B.C.D/32 AND NOT net A.B.C.D/32)
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: Checking TCP SYN Storm
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: NF Filter: proto tcp and packets eq 1 and flags S and not flags A and bpp 48 and not out if 0 AND (NOT net A.B.C.D/28  AND NOT net A.B.C.D/32 AND NOT net A.B.C.D/32 AND NOT net A.B.C.D/32 AND NOT net A.B.C.D/32)
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: No matching flows found for TCP SYN Storm
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm end: TCP SYN Storm
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm -------------------------------------------------------------
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm exclusions: (NOT net A.B.C.D/23  AND NOT net A.B.C.D/30 AND NOT net A.B.C.D/32 AND NOT net A.B.C.D/23 AND NOT net A.B.C.D/22 AND NOT net A.B.C.D/24 AND NOT net A.B.C.D/23 AND NOT net A.B.C.D/23)
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: Checking UDP Packets > 70K / IP
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: NF Filter: proto udp and not out if 0 AND (NOT net A.B.C.D/23  AND NOT net A.B.C.D/30 AND NOT net A.B.C.D/32 AND NOT net A.B.C.D/23 AND NOT net A.B.C.D/22 AND NOT net A.B.C.D/24 AND NOT net A.B.C.D/23 AND NOT net A.B.C.D/23)
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm end: UDP Packets > 70K / IP
Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm -------------------------------------------------------------
Jan 11 01:45:26 hostname nfsen[9261]: Run expire at Mon Jan 11 01:45:00 2010
Jan 11 01:45:26 hostname nfsen[9261]: Expire profile Compromised-Customers group . low water mark: 90%%
Jan 11 01:45:27 hostname nfsen[9261]: Expire profile botnet group . low water mark: 90%%
Jan 11 01:45:27 hostname nfsen[9261]: Expire profile live group . low water mark: 90%%
Jan 11 01:45:27 hostname nfsen[9261]: Expire profile AKAMAI group BREAKDOWN low water mark: 90%%
Jan 11 01:45:27 hostname nfsen[9261]: Expire profile IRC-Probable-Targets group BREAKDOWN low water mark: 90%%
Jan 11 01:45:27 hostname nfsen[9261]: End expire at Mon Jan 11 01:45:00 2010

You shouldn’t see any errors here. If you do, most likely the IPs in your exclusion table aren’t in CIDR format. Even a host needs to be in that format (i.e. 192.168.1.1/32).

When an attack is detected, we send an email to the address listed in the plugin with the body composed of the relevant details. The following is an alert from last year indicating a UDP attack against our customer on port 113.

Date flow start          Duration Proto      Src IP Addr:Port          Dst IP Addr:Port   Flags Tos  Packets    Bytes      pps      bps    Bpp Flows
2009-11-16 03:07:15.269   133.556 UDP     AA.AAA.AAA.AAA:35198 ->     XX.XXX.XX.XX:113   ......   0   187678    8.6 M     1405   517127     46     2

Summary: total flows: 2, total bytes: 8.6 M, total packets: 187678, avg bps: 517127, avg pps: 1405, avg bpp: 46
Time window: 2009-11-16 03:03:28 - 2009-11-16 03:10:06
Total flows processed: 803235, Blocks skipped: 0, Bytes read: 43449712
Sys: 0.174s flows/second: 4590622.6  Wall: 0.170s flows/second: 4724217.0

Additionally, the alert is stored in the database for future queries.

With this information, we can then use NfSen to pull up details for the suspected attack. We’ll want to do a query such as “host XX.XXX.XX.XX”, where xx is the destination of the attack (our customer). This allows us to see all sources because some attackers could be below our thresholds.

Chances are that if we go back to the beginning of the attack (or a few cycles before), we’ll find IRC traffic directly preceding the attack. IRC is typically on ports 6660 through 6669, or port 7000).

This usually indicates one of two things. 1) Someone was talking smack on IRC and got slapped, or 2) This customer got infected by an old piece of malware and when it tried to login to the command and control channel, it had an old password, resulting in an automatic attack.

In either case, we mitigate the attack in our RTBH system, apply an IRC ACL to the customer interface, and send a ticket to our security group to contact the customer so they can clean the infection.

Some other ideas we’ve been kicking around for future versions of DDD is the ability to automatically inject a prefix into our mitigation VRF so that our Arbor implementation can scrub the traffic instead of manually blackholing it.

Also, version 1.5 will include the ability to auto expire attack exclusions, in addition to permanent exclusions. You don’t want to permanently exclude an attackers or attack destination because you’ll be blind to future attacks. Additionally, NfSen can only accept filters that are so long.

This is a fairly simple, yet effective way to identify attacks on your network. Since NfSen is always looking 5 minutes into the past, you won’t be able to prevent an outage, but you can shorten the duration, lessening the impact.

Be cautious what you blackhole with this. Sure, most alerts are true attacks, but some can just be anomalous, yet valid traffic. Things like breaking news could possibly trigger an alert if a lot of users suddenly start visiting a small number of web sites. Be sure you know what an IP is associated with before null routing it. You don’t want to create an outage. You also probably don’t want to null route a customer unless an attack is affecting large portions of the network and is absolutely necessary.

You should also get buy-in from management before implementing new policies, such as ACLs to block IRC traffic, or even RTBH.

You’ll notice that I don’t use vendor specific router shapes. They are fine for sales or marketing diagrams. For functional network diagrams, they are pretty useless. I’d rather have a colored box with the hostname and loopback IP inside than have that text outside and cluttering up the diagram. It’s a personal preference, use whatever you want.

Note: Sorry for the lack of details, but I can’t zoom in any closer without giving away any proprietary data on the network.

A snippet of our core diagram looks like this

Instead of using the curved line tool to create the curved links, we use the circle shape tool and split it to connect routers. Additionally, we also have hidden layers that we use to align routers.

To work with layers, click View and then Layer Properties from the file menu.

This layer is called “Drawing Aids” and is only used when modifying the drawing. We don’t want to print it on the plotter, so make sure Print is unchecked.

Now, draw a series of circles with the Ellipse tool from the Drawing toolbar. A tip here is to hold the shift key while sizing the circle to prevent it from turning into an oval. I usually use red for the color because it stands out enough at any zoom level.

After the circles are layed out, we want to put them into the layer. Select one or more circles and then right click. Now, click Format and then Layer. You can select an existing layer, or create a new one here.

Now, we are ready to create an arc that we can use to connect routers aligned along the circle. For this, I usually create a blank document or new tab and draw a circle (holding shift) and then draw straight lines through it like this:

Make sure the circle and all lines are selected with Ctrl-A (if this is a new temp tab or drawing). To split the shape, click Shape, Operations, and Fragment from the file menu.Then move the new fragment away.

To remove the straight lines, we need to use the pencil tool. Select it and then click the corner right on top of the handle. It looks like you are going to move or size the shape, but that’s ok. You should see the corner turn red after clicking. Now, press delete.

Voila! You should have something like the above. To remove the remaining line, click the circle in the middle of the line (it should turn red too) and press delete.

Now, we have an arc that we can use to follow our layouts for market to market connections. The last thing to do is remove the fill, otherwise you will cover up overlapping shapes. In the toolbar, click the fill bucket and select No Fill.

Now, we can copy this and paste it into our drawing. I use these to create circular links between market routers when I want several lines that line up easily. If you want a curve that’s a bit more granular, you can use the pencil tool to draw a straight line and curve it. To keep it straight at first, hold the shift key.

Now that we have a line, we want to click the round point in the middle and drag it outward. This will curve the line.

The nice thing about this is we no longer need the drawing toolbox. If you want to reposition the arc, just do so with the pointer tool. You can also adjust the arc to line it up with your layout circles.

I use the same layer trick for our market detail diagrams. Only, instead of circles, I just use straight lines to line up the routers.

With my layers

The trick to keeping everything in line is to zoom in. I go as close as 200%, occasionally 400%. This allows you to use the arrow keys to move the shapes around a lot more granular than you can at 100%. I also use the Align Shapes button to line up a stack of routers.

For lines that have angles, I typically curve the corners.

Just select your line and right click, then Format and Line. I select the 2nd round corner since it is the smallest curve.

Finally, you should have a legend somewhere on the map. Ours indicates what the different color links are. It also contains our default OSPF cost. Link costs are always on the links, but we use a dark red to indicate any that deviate from the standard costs so they are easy to notice.

Once you install them via CPAN, just add references at the top of your script

#!/usr/bin/perl
use Cisco::Reconfig;
use Quantum::Superpositions;

You should also have a Cisco configuration file available. If you use my backup script, copy one to your script directory and uncompress it with ‘gzip -d file.gz’.

Reading the Cisco configuration requires just one line

$MYCONFIG = readconfig("/tmp/myciscorouter.conf");

Make sure you keep this in appropriate scope since we only want to have to parse the entire configuration once. This allows us to quickly search for portions of configuration.

In this example, we’ll pretend we have a standard access-list applied to our VTYs and we want to verify the contents. We can load it like this:

my @ACL61 = $MYCONFIG−>get('access−list 62')−>all;

This creates an array that contains every line inside of access-list.

Now, let’s say we have a text file containing an ACL we want to verify against. Inside, it contains what the ACL should look like on our router.

pts/0 jrowley@toolbox:/home/jrowley/scripts/test $> cat ACL62
access-list 62 permit 10.132.0.0 0.0.1.255
access-list 62 permit 10.137.0.0 0.0.0.255
access-list 62 permit 10.148.152.0 0.0.0.255
access-list 62 permit 10.159.0 0.0.3.255
access-list 62 permit 10.178.100.0 0.0.1.255
access-list 62 permit 10.199.144.0 0.0.0.255
access-list 62 permit 10.199.146.0 0.0.1.255
access-list 62 permit 10.202.50.0 0.0.1.255

We can read this into another array like this

my $FILENAME = "/home/audit/FILES/ACL62";
open (ACL, $FILENAME);
my @REFACL62 = <acl>;
close(ACL);

Now, we could loop through both array’s comparing each line, but that’s too tedious. That’s where Quantum::Superpositions comes into play.

my @RESULTSf	= eigenstates(any(@ACL62)    ne all(@REFACL62));
my @RESULTSr	= eigenstates(any(@REFACL62) ne all(@ACL62));

Basically, what this does is compares the two arrays both ways. The first @RESULTSf array contains anything that is in @REFACL6, but missing from @ACL62. The @RESULTSr array contains anything that is in @ACL62, but missing from @REFACL6.

We just used two lines to do what normally would be a dozen or so lines. With the addition of a couple print statements, we can do a check to see if it works as expected. Here is the entire example script:

#!/usr/bin/perl
use Cisco::Reconfig;
use Quantum::Superpositions;

$MYCONFIG = readconfig("/tmp/myciscorouter.conf");

my @ACL62 = $MYCONFIG->get('access−list 62')->all;

print "ACL on Router\n";
print "@ACL62\n";

my $FILENAME = "ACL62";
open (ACL, $FILENAME);
my @REFACL62 = ;
close(ACL);

print "ACL in reference file\n";
print "@REFACL62\n";

my @RESULTSr    = eigenstates(any(@ACL62)    ne all(@REFACL62));
my @RESULTSf    = eigenstates(any(@REFACL62) ne all(@ACL62));

print "Items missing from router ACL\n";
print "@RESULTSf\n";

print "Items in router but not in reference file\n";
print "@RESULTSr\n";

Running it gives us this

pts/0 jrowley@toolbox:/home/jrowley/scripts/test $> perl test
ACL on Router
 access-list 62 permit 10.132.0.0 0.0.1.255
 access-list 62 permit 10.137.0.0 0.0.0.255
 access-list 62 permit 10.148.152.0 0.0.0.255
 access-list 62 permit 10.159.0 0.0.3.255
 access-list 62 permit 10.78.100.0 0.0.1.255
 access-list 62 permit 10.199.144.0 0.0.0.255
 access-list 62 permit 10.199.146.0 0.0.1.255
 access-list 62 permit 10.202.50.0 0.0.1.255
 access-list 62 permit 192.168.2.0.0 0.0.0.255

ACL in reference file
 access-list 62 permit 10.132.0.0 0.0.1.255
 access-list 62 permit 10.137.0.0 0.0.0.255
 access-list 62 permit 10.148.152.0 0.0.0.255
 access-list 62 permit 10.159.0 0.0.3.255
 access-list 62 permit 10.178.100.0 0.0.1.255
 access-list 62 permit 10.199.144.0 0.0.0.255
 access-list 62 permit 10.199.146.0 0.0.1.255
 access-list 62 permit 10.202.50.0 0.0.1.255

Items missing from router ACL
 access-list 62 permit 10.178.100.0 0.0.1.255

Items in router but not in reference file
 access-list 62 permit 10.78.100.0 0.0.1.255
 access-list 62 permit 192.168.2.0.0 0.0.0.255

As we can see, the router configuration is missing one line (10.178.0.0 is typo’d in the router) and it contains an extra line that isn’t in our reference file. This could indicate that your ACLs aren’t as controlled as you might expect.

While we’ve only checked a single router, it’s trivial to read a list from a database, read the backups generated by my backup script [see post #2], and loop through all of them.

For a large network, consistency is key. Sure, occasionally, you will have different ACLs on different routers, but this will help you standardize your deployment for things that should be the same everywhere. You probably don’t want to discover at 3:00am that the router you need to access doesn’t permit your vpn address. It may take a while to standardize all of your routers, but in the end, it is time very well spent.

With these two perl modules, you can easily audit 99% of your router configurations. The remainder is user accounts and vty passwords. You can also check those with an MD5 module and Cisco::Hash (level 7 password decrypter).

Happy scripting!

Before you begin, make sure you install the Net::Telnet::Cisco and DBD::mysql CPAN modules.

Download the backup script here.

Database

We use a database to store our router information [see the soon to be released Mr Audit]. The relevant table is included in the file db.sql [download]. Simply create a database (we use 'auditdb') and import the db.sql file. It includes some sample data, so be sure to change that too.

You only need to be concerned with following columns:

• id – primary key
• iptext – IP Address of the router
• hostname – Hostname of the router
• status – up or down
• company – company name in case of multiple networks
• backup – status field; used by script to indicate if backup was successful or not.

The others are not used by the backup script.
The Script

The only items that need to be changed are listed below.

### database login info
my $dsn         = "dbi:mysql:auditdb:127.0.0.1:3306";
my $user        = "dbuser";
my $pass        = "dbpass";

### tacacs/radius login info
my $username    = "tacacsuser";
my $password    = "tacacspass";

### this is where all the config backups are stored
$::DIRECTORY = "/backups/prod";

my $dbh = DBI->connect($dsn, $user, $pass) or die "Cantt connect to the DB: $DBI::errstr\n";

# change company field to whatever you use
$sql = "SELECT id,iptext,hostname FROM routers WHERE status='up' AND company='synacknetworks';

Once that is finished, just do a "chmod +x filename"  and you can either run it now, or add a cron job to run every so often. Until Mr Audit is released, you can view your backups with the following command.

zcat /backups/prod/hostname/hostname-config.2009-10-25.gz |less

Our production deployment has over 400 days worth of backups for more than 1000 routers. Your mileage may vary depending on disk space.

Limited disk environments

If you don't have much storage, you can run the following command to remove anything older than x days fro the backups.  This example uses 7 days

find /backups/prod -name "*.gz"; -mtime +7 -exec rm {} \;

There must be a space between {} and \; Future Over the next few months, I'll be showing how to use the tools included with Mr Audit to automate network audits. Stay tuned.

This utility is a simple perl script that checks all interfaces in a Cisco router. It is useful to run before and after router maintenance to ensure customers that were up prior, are up after. This script checks both VRF and non-VRF customers. It also supports /30 and /31 interface addresses.

Requirements

It requires the following perl modules. Running the script also requires sudo access due to sending ICMP packets.

• Net::Telnet::Cisco
• Net::Ping
• NetAddr::IP
• Cisco::Reconfig
• Getopt::Std
• Text::CSV

Install these via CPAN. The only things that you will need to change are the username and password variables. These should be low privilege accounts, preferably TACACS or RADIUS. The script itself can be downloaded here.

Usage

Typing ‘pingcheck’ by itself prints out the command line options.

pts/2 jrowley@toolbox:/home/jrowley $> sudo pingcheck

Cisco Ping Checker Thingy Version 1.0

        Usage:
        /APPS/sandbox/usr/local/bin/pingcheck [-h] -w init|recheck -r
        no copyright bs. just supply caffeine and cigs

        -h                : prints this message
        -w init|recheck   : initialize or recheck
        -r                : hostname or IP address

        Examples:

        Grabs router config and performs pre-check
                /APPS/sandbox/usr/local/bin/pingcheck -w init -r routername
        Re-checks interfaces from pre-check
        You must use the same router name as used to initialize
                /APPS/sandbox/usr/local/bin/pingcheck -w recheck -r routername

If you are doing something like an IOS upgrade, be sure you do a ‘wr mem’ first. This also has the benefit of obtaining all customer configs if your company has a policy of only writing the configuration at certain times of day.

Usage Example

The following shows what the output looks like when running against a production router

pts/2 jrowley@toolbox:/home/jrowley $> sudo pingcheck -w init -r exampleca1
Beginning initial check
Connecting to exampleca1
Grabbing config from exampleca1
Interface                              Local IP           Remote IP          Physical Logical  ICMP
Tunnel106474                           172.16.0.5         172.16.0.6         up       up       up
Tunnel175011                           172.16.0.5         172.16.0.6         up       up       up
Tunnel192359                           172.16.0.1         172.16.0.2         up       up       up
Tunnel201345                           172.16.30.9        172.16.30.10       up       up       up
Tunnel201964                           172.16.0.5         172.16.0.6         up       up       up
Tunnel213904                           172.16.0.5         172.16.0.6         up       up       up
Tunnel213980                           172.16.0.5         172.16.0.6         up       up       up
[snip]
Serial4/1/24:0                         xx.xxx.xx.51       xx.xxx.xx.50       up       up       down
Serial4/1/25:0                         NONE               NONE               up       up       down
Serial4/1/26:0                         NONE               NONE               up       up       down
Serial4/1/27:0                         NONE               NONE               up       up       down
Serial4/1/28:0                         NONE               NONE               up       up       down

Ping check complete!

Total Number of interfaces checked: 135
Number of interfaces up: 134
Number of interfaces down: 1
Number of interfaces ignored: 14

Essentially, what happens is that the script logs into the router and captures the startup configuration. Then it leaves the router connection open while it starts to parse all the interfaces. For each interface, the script will check the physical and logical status and then try to ping the customer side of the interface (/30 or /31 supported). If the interface is not in a VRF, it sources the ping from the server it is run on. If the interface is in a VRF, it sources the ping from the router inside the VRF. In the above example, we checked 135 configured interfaces (plus 14 skipped interfaces because they had no IP Address). Out of those 135, only one was down. In our case, it’s likely from preprovisioning a customer. The script saves the status of the interfaces and the ping status in a CSV. When we use the recheck flag, it reads this file instead of having to reparse the configuration resulting in a much faster check. Now, let’s pretend we did some maintenance and recheck the router.

pts/2 jrowley@toolbox:/home/jrowley $> sudo pingcheck -w recheck -r exampleca1
Beginning recheck
Connecting to exampleca1
Interface                              Local IP           Remote IP          Physical Logical  ICMP
Serial4/1/24:0                         xx.xxx.xx.51       xx.xxx.xx.50       up       up       up

Recheck complete!

Number that failed recheck: 0

If our down interface was a circuit issue that was corrected by the maintenance, it would show up in the recheck. Hopefully nothing fails recheck that worked before. If any do fail, troubleshoot them and keep rerunning the recheck until you see the failed recheck count get to zero.

Hints

If you get any weird errors, make sure you didn’t forget the -w and -r flags.

pts/2 jrowley@toolbox:/home/jrowley/d $> sudo pingcheck -w init exampleca1
Beginning initial check
Use of uninitialized value in pattern match (m//) at /usr/lib/perl5/site_perl/5.8.5/Net/Telnet/Cisco.pm line 39.
Use of uninitialized value in concatenation (.) or string at /APPS/sandbox/usr/local/bin/pingcheck line 393.

Connecting to
ERROR: Not a Cisco router!!!

This error is because -r was missing before the hostname we wanted to check. If you did include the -w and -r flags, you likely found a bug in the interface parser logic. Send me a message and I’ll include a fix in the next release.

Cisco::Reconfig note

In the current Cisco::Reconfig module, you will likely need to comment out 4 lines. Try without first. If you don’t receive any errors, comment these lines in Reconfig.pm

                                # this really shouldn't happen.  But it does.
#                               die unless $prev eq "!\n" || $prev =~ /^!.*$/;
#                               die unless $indent == 0;
#                               $ciscobug = 1;
#                               $indent = $in;

They start on line 103, just after the comment “# this really shouldn’t happen.  But it does”. I guess it doesn’t really happen.

Summary

Hopefully this script makes your life simpler. It could help prevent outages when making configuration changes or doing upgrades. If it prevents even one outage, writing it was effort well spent. Let me know if it is useful or if you would like to see any features added.