The Journey of a Network Engineer


May 1, 2011  5:20 AM

Solving the stackwise problem in Cisco Catalyst 3750.

Sulaiman Syed Profile: Sulaiman Syed

In our previous entry, we had the stack issue. The stack connection dropped between two switches. This caused a major concern. Going through the Cisco documentation about stack troubleshoot. We did change the stack cable, but it did not solve the problem. Since the switch was in our data center, any troubleshoot efforts might become an issue in case of downtime.

It was decided that to resolve this issue, the best course of action would be to remove the stack. We took a downtime, and we have removed the stack wise connection. The moment we took the stack cables out. The isolated switch became stand alone switch with the current running configuration. Since, we figured that the switch should be removed from the network so that the real (master) switch and the one just removed (slave) no to cause issues in the network by having similar configurations.

March 29, 2011  8:46 AM

CCNA vs JNCIA-Junos

Sulaiman Syed Profile: Sulaiman Syed

Lets look at CCNA and JNCIA-Junos in terms of knowledge that you gain as an entry level engineer into the realm of TCP/IP. I can garuntee that the rabbit hole goes deep, too deep to be accurate (that is still subjective term. I fail to see how subjective term can be accurate, lol).

JNCIA-Junos is not the best certificate for entry level engineers from my prospective. Why? here are the reasons although they are few. First, it assumes you are familiar with Cisco devices and Cisco CLI. Second, It is exam to for Junos and Juniper. It assumes you already learned or familiar with TCP/IP. I think any entry level certification should cover the topics of TCP/IP. The foundation of the Network, the OSI layers, Subnetting, IP addresses.

I think the biggest concern i have with entry level certificates is that basic of TCP/IP should be covered, and the exam should focus on it. It should not focus on How to run Junos. If i hire a network engineer who knows how to use Junos, but he has no clue of Networking, what use i have of him? Yes, the certification does require you to have the basic knowledge. But from what i gathered in the studying guides, they are not covered in the exam.

Once i pass the exam, I’ll update this entry, or make a new one. Just to be sure that I’m representing accurate facts.


March 23, 2011  3:13 AM

Running Juniper with GNS3

Sulaiman Syed Profile: Sulaiman Syed

It was rather a very intense Google journey to find out all the details on how to emulate Junos. Since i intend to learn juniper, i needed a platform to work on. after two days of research and work, i managed to a results.

There are various things required to make things work. I would list them down here so they are easy to find. VMware Player, GNS3, Cisco router IOS, and VMware Olive (Google is your friend). Once you have all these, you are ready to start!

Running the VMware machine will be an easy task, but connecting the VMware Olive with Cisco in GNS3 is the one requiring some work. But with my guide, it should be as easy as 1,2 and 3.

After installing VMware Player, check for adapter settings in windows.

Network Adapters

By default VMware player will install to VMware virtual Ethernet adapters, i’m not sure what are their numbers. but for my case, they were vmnet1 and vmnet8. These are significant to know how to connect VMware machine to Cisco router in GNS3.

Open the .vmx file in notepad. Here we can edit the fields in order to make VMware Olive machine use the virtual Ethernet adapters in windows.

VMware Network Interface

The Olive VMware has three network interfaces, two are bridged and first one is in “costum” we change the adapter to the one to fit the Ethernet adapter in our network devices (from the first figure). I already highlighted it in red. Ethernet 0 will be reflected as interface em0 in Junos. ethernet1, and ethernet2 will be bridge on the virtual Interface, so you can connect to other Olive Machines to ethernet1, ethernet2 (em1, em2). my assumption is, if you want to connect say Olive1 and Olive 3 using em2 then you change ethernet2 in vmx file of both olive 1 and 3 to a bridge mode with a common adapter.

Topology

That is the topology i created for the simulation. basic two Juniper routers connected to a Cisco router. and the two Juniper routers are connected as well (virtually in VMware). It was tested, and pings were working.

In GNS3, add the VMware as a Cloud, of course the cloud will be not associated with the VMware Olive till you select the adapter that you set up in the vmx file. In a screen shot, you will see that i have Chosen Vmnet1 for this particular Olive.

Cloud (VMware Olive) Settings

Last step, would be to do the appropriate configurations in the Olives, and Cisco Router, here is the screenshot of the sample configuration i used to ping.

Junos Configuration

Don’t forget to add the following before you can commit any configuration into Juniper Router

set system root-authentication plain-text-password

Cisco’s Configuration as simple:

interface FastEthernet0/0
ip address 192.168.1.2 255.255.255.252
duplex auto
speed auto
!
interface FastEthernet1/0
ip address 192.168.1.10 255.255.255.252
duplex auto


March 22, 2011  7:30 AM

Configuring IP DHCP Snooping.

Sulaiman Syed Profile: Sulaiman Syed

The other day, a smart user (i consider him evil) attached a ADSL modem to the network port. What he didn’t realize that his device was programmed to work as DHCP server. As a result, the whole vlan started getting the wrong IP address, and connectivity was lost. After investigation, we realized a rouge DHCP server, tracked it, and finally blocked the port manually.

Of course, the best was to enable DHCP snooping in the switch, and not worry about anyone attaching any funny thing to the network. To get an idea with DHCP snooping is, please read Cisco’s document. I will quote the basic idea.

DHCP snooping is a DHCP security feature that provides security by filtering untrusted DHCP messages and by building and maintaining a DHCP snooping binding table. An untrusted message is a message that is received from outside the network or firewall and that can cause traffic attacks within your network.

The DHCP snooping binding table contains the MAC address, IP address, lease time, binding type, VLAN number, and interface information that corresponds to the local untrusted interfaces of a switch; it does not contain information regarding hosts interconnected with a trusted interface. An untrusted interface is an interface that is configured to receive messages from outside the network or firewall. A trusted interface is an interface that is configured to receive only messages from within the network.

DHCP snooping acts like a firewall between untrusted hosts and DHCP servers. It also gives you a way to differentiate between untrusted interfaces connected to the end-user and trusted interfaces connected to the DHCP server or another switch.

Just following the normal command line will surely hinder the network, in fact it blocked all DHCP requests. the  information option should be disabled. That what i realized, and i found in many other forums.

Second issue i encounter, that the DHCP snooping will not be enabled on any switch with VTP mode other than transparent. What i found that it was disabled in other modes for security reasons. If a VTP domain is compromised, then the attacker can remove vlans or add vlans, thus compromising the integrity of vlans and this will cause an issue with IP DHCP snooping. Since DHCP snooping will start effecting other vlans, etc.

The following should be used to configure DHCP Snooping

ip dhcp snooping

ip dhcp snooping vlan vlan-number

no ip dhcp snooping information option

int gig 0/1 \\ this is the uplink

ip dhcp snooping trust

here is the output for various commands:

L2CS-B851-01#show ip dhcp snooping
Switch DHCP snooping is enabled
DHCP snooping is configured on following VLANs:
851
DHCP snooping is operational on following VLANs:
851
DHCP snooping is configured on the following L3 Interfaces:

Insertion of option 82 is disabled
circuit-id default format: vlan-mod-port
remote-id: 2893.fef7.f280 (MAC)
Option 82 on untrusted port is not allowed
Verification of hwaddr field is enabled
Verification of giaddr field is enabled
DHCP snooping trust/rate is configured on the following Interfaces:

Interface                  Trusted    Allow option    Rate limit (pps)
———————–    ——-    ————    —————-
GigabitEthernet0/1         yes        yes             unlimited
Custom circuit-ids:
L2CS-B851-01#show ip dhcp snooping statistics
Packets Forwarded                                     = 415328
Packets Dropped                                       = 7601
Packets Dropped From untrusted ports                  = 0

L2CS-B851-01#show ip dhcp snooping binding
MacAddress          IpAddress        Lease(sec)  Type           VLAN  Interface
——————  —————  ———-  ————-  —-  ——————–
00:1A:80:EE:79:8A   10.80.51.38      420810      dhcp-snooping   851   FastEthernet0/5
E0:CB:4E:06:FC:3E   10.80.51.99      372172      dhcp-snooping   851   FastEthernet0/45
00:21:9B:E2:87:C6   10.80.51.80      421750      dhcp-snooping   851   FastEthernet0/4
00:26:6C:78:00:F3   10.80.51.174     425902      dhcp-snooping   851   FastEthernet0/36
00:1B:38:AF:81:DD   10.80.51.186     423185      dhcp-snooping   851   FastEthernet0/43
Total number of bindings: 5

Applying IP dhcp snooping in the access switch is enough, as long as the uplinks are trusted. It is not required to apply these configurations to distribution switch (assuming that no one has access to them. )

Over all, this should have been used ages ago. I am very glad that i implemented this in our network.


March 21, 2011  6:52 AM

Juniper Certification Track

Sulaiman Syed Profile: Sulaiman Syed

I’ve decided that since I shall start learning Juniper, widen the horizon. First anyone starting Juniper will have to get the associate certificate, JNCIA-JUNOS. The topics that are covered can be seen from the website exam topics.

They include:

Junos OS Fundamentals

  • Explain the Junos OS architecture
  • Describe the functions of the control and forwarding planes

User Interface Options

  • Describe the functions of the CLI modes
  • List options for getting help and filtering output
  • Explain options for moving around in the CLI
  • Explain the Junos batch configuration model
  • Modify and manage configuration files
  • Describe the J-Web user interface

Junos Configuration Basics

  • Describe initial configuration steps and system maintenance
  • Save and restore a rescue configuration
  • Describe network interfaces and their associated properties
  • Configure various types of interfaces
  • List and describe user authentication methods
  • Configure user accounts with default and custom login classes
  • Describe, configure and monitor syslog and tracing
  • Describe, configure and monitor configuration tasks, such as NTP, SNMP, and configuration archival

Operational Monitoring and Maintenance

  • Explain how to view relevant interface statistics and errors in the CLI
  • List ways of gathering state and health information for a Junos device
  • Explain the purpose and syntax for various network utilities such as ping, traceroute, telnet, ssh etc.
  • Identify the procedure for installing or upgrading Junos
  • Explain how to reset a lost root password

Routing Fundamentals

  • Define route preference and list some default route preferences
  • Explain how to view the contents of the routing tables
  • Explain how the routing and forwarding tables differ
  • Describe and configure static routing
  • List the advantages and applications of a dynamic routing protocol

Routing Policy and Firewall Filters

  • Define a term
  • Explain when import and export policies are evaluated in relation to the learning and advertising of prefixes
  • List several match criteria and actions for firewall filters and routing policy
  • Explain some common match criteria and
  • Identify the results of a route or packet with a given filter or policy
  • Configure a routing policy and firewall filters

Class of Service

  • Describe class of service

Networking Fundamentals

  • Convert decimal to binary and binary to decimal
  • Define the term broadcast domain and the purpose of a router
  • Describe Ethernet operation
  • Describe the purpose of the subnet mask
  • Compare and contrast connection oriented and connnectionless protocols
  • Explain how to segment large networks into smaller ones
  • Identify the longest match from a routing table for a given destination

For the rest of track, please refer to the official website.


March 16, 2011  4:21 AM

Manage Engine IT360 Review – part 3

Sulaiman Syed Profile: Sulaiman Syed

In the previous two entries, we have Introduced Manage Engine IT360, and Dashboard of Manage Engine IT360. In this Entry, we would talk about the management of Network Devices in Manage Engine IT360

IT360 uses opmanager module for the network management service. This tap when clicked will show a window that gives a very comprehensive view of the whole network. From the figure below, it shows the various elements. First it will show the devices (It supports many other devices, a list can be seen from their website). A map can be uploaded to show a geographical representation of the devices. Event Summary will show all the monitors applied and if anything has been triggered. in our case we have plenty of violations (mostly link over utilization). it will show the top 10 for CPU utilization, Memory Utilization, Interface Traffic, and Bandwidth. These were the monitors configured in the start. If more monitors were configured, we expect to see them in this window as well.

Network-1
Network-2
Figure : Network Tap

Clicking on any device will give a general view of the device. when the SNMP configured properly, it managed to show the device name, ip address, vendor, type, and general description. on the right: availability, response time, packet loss, CPU utilization, Memory Utilization, and Backplane utilization. More monitors can be added from the action tap. Scrolling down, we see the monitors, notifications, and interface. I picked the interfaces and then we can see a detailed view of all the interfaces, from name, send and received traffic, errors and a real time monitor. Real time monitor can be configured to update per second for 5 mins. the interfaces will show green (up/up, or administratively down), Red (up/down) and orange for any violation. all this can be seen from the figure Below

Device-1
Device-2
Figure : Device Tap

Going to the interface. it will show interface details, utilization graphs. day’s traffic that can be modified to show up to a month. Errors and Discards can be seen as well. It will also show packets per day, and bytes per second. The one issue I found it to be very strange. The Real time traffic showing me traffic exceeding 1Gbps, where the link is only 1Gbps.

Interface-1
Interface-2
Figure : Interface Tap

In the case more monitors are needed to be added. here a brief list of all the possible built in monitors.

Monitor-1
Monitor-2
Figure : Adding Monitors.

The OpManager and IT360 do provide a good solution for the Network services.


March 13, 2011  4:26 AM

Manage Engine IT360 Review – part 2

Sulaiman Syed Profile: Sulaiman Syed

Following on my previous entry of IT360, where we have Introduced Manage Engine IT360. I would like to go on how the dashboard looks like. I think that once the alarms and monitors configured properly. the dashboard will give a real over view of every element in the network that the IT360 support. I would like to apologize for the zoom out images. I wanted to fit all information into a single webpage.

Once logged into IT360, you can see the dashboard, refer to figure 1. In there The business can be configured (grouped) so all services follow a logical pattern. Practices from ITIL will help to define the group of services, and making them into a logical processes with process owners, who can go and log into IT360 to find the appropriate data that they are monitoring, and operating.

Dashboard

The dashboard will also give the option to see the networks, servers, or applications that are monitored. But i will leave that for some other time. So clicking on the IT360 business (it was the default one). It will take you by default into the Summary tap. as shown in figure 2. The main variables to see are Up-time, recent alarms, application response time, server response time, CPU utilization, link utilization (it is not shown since the network data was not available, misconfiguration?). These readings are very accurate. To this moment of the review, it seems that IT360 provide actual proper data. (our own motoring tool without naming it. wont pull these data, or pull them wrong).

Business

Figure 3 is showing the inventory. This tap will just give a brief idea of health, up-time for all the items that were added. This can be configured to show only the important inventory, or to show everything. when adding routers, it will show the interfaces as well. which is good just to see if everything is up or not. I don’t find the data here interesting, since it is presented in other taps as well.

Inventory

Business View is how you categorize the Items. As seen from image. It is really good touch, to see which service is down effecting which business. Again, since i haven’t configured this. all shown in one single business.

Business View

Last tap, the troubleshoot. will show the status per hour. the period can be changed from hours, to days, to custom. When there is a ticket/alarm clicking on the box will take window to the alarm. This will help to see what was the issue, was it resolved, and who is responsible.

troubleshoot

In our next entry, I will go through the network tap. and see all the possible things to monitor, and how to do that. I have not yet done proper review of it. but from the first glance. I really liked it. I think a more deep digging will give me a good insight. but i can tell that it can do everything what our current solution (whatsup) can do. This can be seen in the last article on Manage Enginer IT360.


March 12, 2011  5:13 AM

Manage Engine IT360 Review – part 1

Sulaiman Syed Profile: Sulaiman Syed

We are evaluating manage engine IT360 in the organization. We still don’t have a centralized monitoring point. Every team does its own monitoring using their own devices. The management wants a transparent operation, while no downtime, and proper monitoring to know which and when went down.  I do agree with this policy. Here Manage Engine IT360 comes to play.

I have downloaded the Demo, installed it. it was very easy to do so. straight forward installation. Once installed, the application will take some time to initialize all the components, it did took a while.

Once started through its console (the web), you would need to do the first initial setup. that include usernames, passwords, adding devices, editing monitors, alarms, proxy setup, sms setup, etc. The screen shot shows the components in the admin tap.

Admin-General

Figure : Admin Tap

Clicking on networks, will take all the possible configurations that be done for network devices. The figure shows the options available.

Network Tap

Figure : Network Tap

Adding devices made simple when you setup the credentials. Once that is done, devices can be added in Bulk, or even discovery of devices can be used for a range of IP addresses. SwitchPort Mapper will accept End users or Servers. I was expected a fully switch port mapper, which will show me which device connected to the port with mac and ip addresses. MiB browser can be used to add or edit the MiB files. Adding network Devices is straight simple. so i will not show how it is done here.

Application and Servers tap have similar interface, by clicking on add new monitor. we can add various servers and applications. Since the list was long, i have took multiple screenshots.
List of applications -1
List of applications -3
List of applications -2

Figure : list of applications

As seen, they are really almost every important element any enterprise have.  Before full evaluation, the list seems good enough.

Lastly, we have the traffic tap. This will make use of netflow. At first glance, it seems that this will be used for billing purposes. Since our organization is not a ISP. This feature wont be required.

This application comes with Service Desk. This follows the ITIL standards. It will be very good if our help desk can make use of it, since the organization really have no centralized  point to see tickets, issues, requests, etc.

Newer entries explaning the Dashboard of Manage Engine IT360, and Network Monitor in Manage Engine IT360 are followed in this series.


March 11, 2011  4:57 AM

Troubleshoot: distribution Switch acting wierd.

Sulaiman Syed Profile: Sulaiman Syed

Before i Begin, Lets not talk bout how bad the design is. I know all the issues with our design, and that it is faulty in every possible way. Lets say that I still don’t have the managerial power to alter the design yet. The design will change, hopefully sooner than later. I’ll explain how the infrastructure is in detail, then i would say what is the problem in detail. I really appreciate any input, cause i just can’t figure it out.

logical Drawing

We have two Core switches 6509 that are connected with trunk links. The server farm distribution switches are 3750 are also connected with trunks. The connection is a square one. Then we have our Wireless Lan Service Modules (WLSM) connected to 6506 switches that is connected to our Server Farm Distribution. Everything is connected in Trunks. Only the Link to ASA firewalls are Access, and they are in the Server Farm Vlan.

Looking at the drawing, it is shown that the management vlan (used for us to access the devices) with 10.10.0.0/16. while the Server Farm Vlan is 10.40.0.0/16. Both Core, Server Farm Dist, and WLSM (6506) switches are running HSRP. The Server Farm Vlan is spanning to the core. Spanning tree is functioning well, without any issues.

Now, the problem is. If we shutdown the Server Farm Interface in any WLSM switch, then we lose connectivity to Firewalls (after exactly 3.30 Mins). We checked that no spanning tree issue, or routing issues. All proper ports were open (unblocked), and all the routes are showing right. We didn’t lose connectivity to any Server that is connected to the Server Farm Access switches. Only the Firewalls that are connected to our Cores.

Here is the output of traceroute, ping, extended ping to give a good idea.

C:\Documents and Settings\Administrator>Tracert 10.40.0.100

Tracing route to  10.40.0.100 over a maximum of 30 hops

1   1 ms    1 ms   1 ms    10.14.0.64
2   <1 ms <1 ms <1 ms 10.14.255.255
3   3 ms   <1 ms <1 ms  10.10.40.1
4   <1 ms <1 ms <1 ms 10.40.0.100

The reason above we see 10.14.255.255 is that the links between Building Distribution and the Cores are Routed (not trunked). so 10.14.255.255 is the interface in Core. This is normal behavior, after shutting down.

C:\Documents and Settings\Administrator>Tracert 10.40.0.100

Tracing route to  10.40.0.100 over a maximum of 30 hops

1   1 ms    1 ms   1 ms    10.14.0.64
2   <1 ms <1 ms <1 ms 10.14.255.255
3   3 ms   <1 ms <1 ms  10.10.40.1
4  *          *        *

5  *          *        *

L3CS-SF-02#ping
Protocol [ip]:
Target IP address: 10.40.0.100
Repeat count [5]:
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface: 10.10.40.1
Type of service [0]:
Set DF bit in IP header? [no]:
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.40.0.100, timeout is 2 seconds:
Packet sent with a source address of 10.10.40.1
…..
Success rate is 00 percent (0/5).

L3CS-SF-02#ping
Protocol [ip]:
Target IP address: 10.40.0.100
Repeat count [5]:
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface: 10.40.0.1
Type of service [0]:
Set DF bit in IP header? [no]:
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.40.0.100, timeout is 2 seconds:
Packet sent with a source address of 10.40.0.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
L3CS-SF-02#

What i Found is that, if the Server Farm Dist switched pinged Firewall using the management vlan, the ping will not work, while a ping from source of server farm vlan will work. My question is,why the traffic that came into the Dist switch through management vlan (routing vlan) wont propagate into the server farm vlan. After Shutting down the WLSM server farm vlan interface! (any interface from the two)


February 28, 2011  7:58 AM

Troubleshooting Switch Stacks.

Sulaiman Syed Profile: Sulaiman Syed

So, today in the morning we received this entry in our syslog server:

Feb 27 17:15:58: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 1 Switch 2 has changed to state DOWN

Feb 27 17:15:58: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 3 has changed to state DOWN

naturally, we get the feeling that the stack link went to down. but why and how? these questions remain un-answered. Further reading in to CIsco documentation. it showed that the only reason it could happen is for faulty connection. usually a bad cable. To confirm that the link is down we used the following commands

Switch#show switch detail
Current
Switch#  Role      Mac Address     Priority     State
——————————————————–
*1       Master    0019.e71f.6a80     10        Ready
2       Member    f4ac.c14e.3100     1         Ready
3       Member    04fe.7fc5.2980     3         Ready

Stack Port Status             Neighbors
Switch#  Port 1     Port 2           Port 1   Port 2
——————————————————–
1        Ok         Ok                2        3
2       Down        Ok              None       1
3        Ok        Down               1      None

As it can be seen, one link is indeed is down. and since we are using cross connection (full redundant connection) the stack did not break. Which is functioning right now.

We are investigating on how to fix the issue without causing the switches to reboot or any downtime. even 5 mins is critical for this server, since it is connected in our server farm.

We set up a test bed with two 3750 switches, going to do all possible testings before we even try to fix the current Live setup. all findings will be posted.


Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to: