The long road ahead…

Every journey begins with a first step.  With this post, I begin a new journey of self reflection and discovery.  Going forward, I intend to regularly share experiences from working with clients and other IT pro’s.  I’ll include a healthy amount of technical information, along with the design decisions driving the architecture and overall solution discussed.  I also hope to step beyond the geek speak at times to help bridge the gap between technical and business goals/objectives, as they don’t always line up directly.

My hope is that what I share can help others if they encounter similar scenarios or issues.  I will never claim to be the expert at everything, and I will call out others who have helped out along the way.  It’s important to realize that behind every successful individual, there is usually a team that has played a part in the success, whether directly or indirectly.  In many cases, individual success reflects the success of the supporting team as well.

For an initial thought to consider, how important is security-in-depth within a network architecture?

Today I presented an executive overview of an edge network restructuring project to a customer to finally wrap up the project. At the end of the presentation, the CIO openly stated that he did not care for the security services between network zones/segments.  In this case, it is a higher education customer, with student, internal, and other zones.  We had created zones on the firewall (SonicWall NSA e7500) to segment the traffic and provide UTM to prevent students from hacking the internal network.  SonicWall Unified Threat Management includes Firewall Integrated IPS/AV/AntiSpyware/Content Filtering/Traffic Policies and Wikipedia has a more general description of UTM.  This was all described in the original project scope as approved 6 months ago, and the firewall has been in place for over 3 months.

Some more discussion uncovered that they were concerned about bottlenecks that a 3rd party network analysis had reported, in particular regarding the new firewall.  The 3rd party recommended architecture was to have a core switch do the routing between segments instead of the firewall.  Well, in a traditional corporate network, I would likely agree.  In this case though, there are real security concerns about using basic ACL’s on the core switch to control the traffic flow.  With UTM monitoring the traffic allowed between zones, application access can be granted, yet traffic is scanned to verify it is not malicious first.

Additionally, this firewall is sized to handle the production traffic, which basic network link reporting backed up.  In reality, the existing Cisco switches in the edge network were only 100MB, and the firewall is running with 1 GB interfaces.  Thus, the there were times that that monitoring showed the switch interfaces to the firewall maxing out, thus the switches were the bottleneck.

Finally, some of the 3rd party testing results were inconsistent (I question their test point placement), and some of the recommendations were made based on assumptions regarding firewall performance.  From the recommendations, I suspect the 3rd party is not familiar with the actual equipment installed, since they were recommending a competitive platform.

Another point of customer concern, and my mutual concern was a traffic shaper that was put in, but not performing properly and causing network interruptions.  This unit had been down several times over the past few months, and it happened to go down yesterday and was powered off, since it still was causing slowness when in bypass mode.  Basically, the customer said that they can not handle more troubleshooting in the production network and either it had to go or we did.  Also, they expressed concern about their primary LOB vendor not being involved in the shaper deployment, as they had said they could not provide support for the application if there was a shaper on the network.  This should have been addressed during the pre-sales cycle, and personally if this was a concern, they should have brought it up months ago before signing onto the project.  We said we would have a hard discussion with the manufacturer in the morning, and moved on…

At this point, the discussion then shifted to the customer needing a proposal for an internal network architecture review and planning session by 5pm tomorrow (less than 24 hours)…  This was something I’d already discussed with various members of the technical team, and apparently staff had been having separate discussions at a management level.  Interestingly, the two paths crossed, and now we shift from defending the architecture to an opportunity to provide architecture recommendations for the rest of the campus network within a 15 minute span.  Amazing how an open discussion and willingness to listen to the customer can change the tone.

Lessons learned

• Make certain that proposals are approved beyond middle-management and meet C-level expectations and business requirements.  Technical features are not always valued by management unless they directly solve a business issue.  For every technical requirement in a proposal, there needs to be a corresponding business requirement to clearly link the value proposition.
• Be open minded and willing to listen to additional concepts or architectural options.  UTM is a relatively new concept, and it has not received the blessing of all vendors (primarily due to performance impact if not backed by solid hardware).  This has lead to customer confusion and a secondary post-install sales cycle to convince the customer that the features are beneficial…
• Wrap everything with monitoring so the customer can tell what is going on and be able to respond appropriately.  In this case, the customer hadn’t fully discovered all the monitoring capabilities, thus was not able to validate/invalidate the 3rd party reports of firewall overloading.  The monitoring also provides the benefit of being able to validate your own design/architecture.
• Learn about competitive products and alternative architectures.  There are times you may need to work with them, and there are other times you will need to be able to clearly distinguish between the architectures in a competitive situation.  Just saying your way is better doesn’t cut it, you need to be able to build a point-by-point case.  If you can’t, then maybe you aren’t offering the best solution…  The concept of walking away from an opportunity because the competition has a better solution will be an interesting discussion for another day…

Another day gone

I hope this has been a useful post.  Look for more soon.  Until then, keep on learning!