Troubleshooting archives - IT Trenches

IT Trenches:

troubleshooting

Oct 15 2009   6:44PM GMT

Free Training - Laura Chappell presents: Wireshark 201 Jumpstart - Filtering on the Good, the Bad, the Ugly



Posted by: Troy Tate
network analysis, protocol analysis, packet analysis, packet capture, training, education, wireshark, ethereal, tcp/ip, trace files, Networking, tools, Monitoring, reporting, IT education, performance monitoring, troubleshooting, howto, Metrics, analysis, Laura Chappell

Laura Chappel, the BitGirl, is at it again with another in her series of Wireshark Jumpstart webinars. The next one is called Wireshark Jumpstart 201: Filtering on the Good, the Bad, the Ugly. It will be held on October 27 - 10:00am-11:00am PDT (GMT-7). If you manage networks or want to manage a network, a good understanding of protocol and packet analysis will help you immensely with your career.

Some things you will learn in this webinar:

  • Using the Default Capture and Display Filters
  • Creating a Few Hot Capture Filters
  • Filtering Tips and Tricks for Troubleshooting
  • Filtering Tips and Tricks for Security

Even if you are very familiar with Wireshark or other packet capture and protocol decode tools, Laura’s seminars are well worth attending. You might even find out a little tidbit here or there because Repetition is one of the keys of learning. Unfortunately I will not be able to attend this webinar since I will be on a golf vacation in North Carolina. So, if you attend this event, please come back and share with me and other IT Trenches readers what you learned and how valuable the webinar was for you.

Thanks for reading and let’s continue to be good network citizens!

Jul 20 2009   6:36PM GMT

Wireshark quickstart tutorial - learn to capture network traffic



Posted by: Troy Tate
network analysis, protocol analysis, packet analysis, packet capture, training, education, wireshark, ethereal, tcp/ip, trace files, Networking, tools, Monitoring, reporting, IT education, performance monitoring, troubleshooting, howto, Metrics, analysis, Laura Chappell

There are more upcoming sessions in the Laura Chappell seminar series called Wireshark 101Jumpstart tutorials. Check out the schedule at Chappell University website. Some of the things you will learn include:

  • Wireshark elements and capabilities
  • Tapping into the wired or wireless network
  • Capturing and filtering basics
  • Graphing basics

If you cannot attend the seminar, you can still register and download the seminar notes and gain access to the trace files used in the session. If you manage a network, you should learn this stuff! Be sure to register and attend early. The sessions are limited to 1000 viewers and these fill up FAST!

See my entry

Repetition is one of the keys of learning

for a how attending one of these seminars helped address an issue I was having with using Wireshark.

Thanks for reading and lets continue to be good network citizens!


Jun 29 2009   8:15PM GMT

Another threat to watch out for - Ants in the keyboard!



Posted by: Troy Tate
computer vulnerability, troubleshooting, hardware, hardware failure, diagnostics, research, computer peripheral, keyboard failure, keyboard

On Friday I posted a tongue-in-cheek type posting about a worm taking down a laptop. Not necessarily big news but something different to see as a risk to computing equipment.

Another similar story came to my attention today. It has to do with ants in a membrane switch type keyboard. Check out the article Ant Farm In The Keyboard. Hey… isn’t it about sharing troubleshooting information between peers? wink-wink - nudge-nudge

Thanks for reading. Have a great day and let’s continue to be good network citizens!


Jun 26 2009   5:22PM GMT

Friday news alert - Worm causes computer crash! - Troubleshooting tip



Posted by: Troy Tate
hardware, hardware troubleshooting, computer failure, troubleshooting, crash analysis

Well… that’s probably news every day somewhere but in this case it was a real worm and I’m not talking about an electronic worm! I guess I need to ensure I don’t leave my computer out on the patio on the swing. I need to also watch my cat.

Check out the story Worm Causes Computer to Crash. You will see what I am saying is true.

Thanks for reading and let’s continue to be good network citizens.


May 26 2009   7:34PM GMT

Repetition is one of the keys of learning



Posted by: Troy Tate
network analysis, protocol analysis, packet analysis, packet capture, training, education, wireshark, ethereal, tcp/ip, trace files, Networking, tools, Monitoring, reporting, IT education, performance monitoring, troubleshooting, howto, Metrics, analysis, Laura Chappell

I recently posted an update about Laura Chappell’s Chappell University Online seminars. I attended one of these seminars today. What a great experience! I always try to attend Laura’s events and always pickup a tidbit that makes my life as a network manager easier. She gives you information about tools you can use to fight the battle of “the network is down”. Most of the time the network is behaving as designed. It’s poorly written applications or too high user expectations that create issues. So, if you want be the expert on fighting the network is “bad” syndrome - check out Laura’s presentations - I did and I learned something new… Continued »


May 21 2009   12:57PM GMT

Master key tasks for network troubleshooting - Chappell University Online Seminars



Posted by: Troy Tate
network analysis, protocol analysis, packet analysis, packet capture, training, education, wireshark, ethereal, tcp/ip, trace files, Networking, tools, Monitoring, reporting, IT education, performance monitoring, troubleshooting, howto, Metrics, analysis

I’m a huge fan of Laura Chappell. She has a great sense of humor and is a great educator about all things packet oriented. Previous posts about Laura have included:

Is protocol analysis or network management your thing?

ARP as a network auditing tool

Did you see this? - Latest Laura Chappell Newsletter

Did you see this? - the viral bitgirl

She has now started a new online seminar series. Some of the presentation are free and others are accessible for a fee of $99. If you cannot get away for education, then this is an excellent alternative and you can gain a great amount of knowledge from this packet analysis expert. I recommend that you visit Chappell Online University and sign up for the free Wireshark Jumpstart: Master Key Tasks for Network Troubleshooting seminar to get a feel for the seminars.

Thanks for reading and let’s continue to be good network citizens!


Feb 11 2009   8:08PM GMT

Tracking down that user/computer that locks AD accounts



Posted by: Troy Tate
Data security, administration, analysis, antivirus, anti-virus, diagnostics, howto, information security, malicious activity, malware, Microsoft, Microsoft Windows, Active Directory, AD, network security, Password, policy enforcement, reporting, risk, risks, scanning, search, Security, security notification, tools, troubleshooting, Windows, password management, account management

With an environment spanning 18+ sites and more than 3000 computers around the globe, you could understand how challenging it would be to track down what device/user might be locking user accounts. There are tools out there that you can pay for that can help do this. However, Microsoft has some free tools that with a little testing and use will permit you to quickly track down where the account is being locked and address the situation.

We had a situation recently where malicious software got onto a couple of machines and attempted to use the Administrator account to login. We have account lockout on our Windows 2003 AD domain, so after the appropriate number of invalid tries the Administrator account was locked out in the domain. This is because the machines were members of the domain and the malware did not distinguish the local administrator account from the domain administrator when attempting to elevate authority. Note that we use least user authority in our environment so the malware was not able to spread beyond these two machines. We suspect the machines became infected due to out of date antivirus signatures.

Unfortunately, the antivirus we use did not alert us to the situation. The way we were alerted was by our Microsoft Systems Center Operations Manager (SCOM) implementation. It notified the SCOM admin that the domain Administrator account was locked. The operations team was then tasked with tracking down what or who was locking this account. This is where the Microsoft Account Lockout and Management Tools came in use and helped isolate the cause. Continued »


Jan 29 2009   9:13PM GMT

Nifty tools for tracking down that “interesting” network traffic



Posted by: Troy Tate
pstools, Sysinternals, Microsoft, Routers, Cisco, troubleshooting, toolkit, Security, WAN, LAN, malware, network analysis, network monitor, network troubleshooting

My previous posting was meant to help you determine the source of potentially dangerous network traffic at your network’s edge. This post is meant to help you identify applications and traffic on your local network that seems to be “interesting”. I define “interesting” as something that you don’t know much about but would find it interesting to learn more about and maybe take some action to shutdown.

As you may already know, I work at an international company with sites around the globe. There are over 2500 computer nodes not including printers, servers, switches, etc. Sometimes it is necessary to identify what traffic is crossing the network links between the sites. There are lots of tools and processes that can be used to gather this information. I will outline a couple here.

Our WAN edge routers are from Cisco. One of the features that can be enabled on a Cisco router is the ip cache flow feature. The show ip cache flow command returns some very useful information. An example is shown below.

show ip cache flow
IP packet size distribution (116972772 total packets):
1-32   64   96  128  160  192  224  256  288  320  352  384  416  448  480
.000 .375 .090 .023 .010 .007 .006 .003 .002 .014 .011 .010 .009 .005 .004
512  544  576 1024 1536 2048 2560 3072 3584 4096 4608
.004 .003 .006 .028 .378 .000 .000 .000 .000 .000 .000
IP Flow Switching Cache, 278544 bytes
64 active, 4032 inactive, 4367569 added
80215342 ager polls, 0 flow alloc failures
Active flows timeout in 30 minutes
Inactive flows timeout in 15 seconds
IP Sub Flow Cache, 21640 bytes
0 active, 1024 inactive, 0 added, 0 added to flow
0 alloc failures, 0 force free
1 chunk, 1 chunk added
last clearing of statistics never
Protocol         Total    Flows   Packets Bytes  Packets Active(Sec) Idle(Sec)
--------         Flows     /Sec     /Flow  /Pkt     /Sec     /Flow     /Flow
TCP-Telnet         724      0.0         7   430      0.0       6.1      15.4
TCP-FTP          13859      0.0         9    93      0.0       6.7       3.4
TCP-WWW        3537205      0.8        14  1021     12.2       3.7       9.7
TCP-SMTP           290      0.0       104   989      0.0       5.5       1.8
TCP-X                3      0.0         2    42      0.0       0.3       1.3
TCP-BGP             18      0.0         1    43      0.0       0.0      13.9
TCP-Frag           112      0.0        37    78      0.0      18.3      15.5
TCP-other       684674      0.1        12   831      2.0       6.4       7.0
UDP-DNS           1973      0.0         1    72      0.0       0.1      15.4
UDP-NTP            248      0.0         1    77      0.0       0.0      15.4
UDP-Frag             3      0.0         1    45      0.0       0.0      15.6
UDP-other        10247      0.0         1   210      0.0       0.8      15.4
ICMP             97640      0.0        19    83      0.4      18.6      15.4
GRE              20509      0.0      2598   150     12.4     165.6      14.5
Total:         4367505      1.0        26   593     27.2       5.2       9.4
SrcIf         SrcIPaddress    DstIf         DstIPaddress    Pr SrcP DstP  Pkts
Tu0           10.aa.20.254    Fa0/0         10.bb.21.1      01 0000 0000    20
Tu0           10.cc.12.200    Fa0/0         10.bb.21.1      01 0000 0000    20
Tu0           10.dd.12.8      Fa0/0         10.bb.12.150    06 0D0A 0871   467
Tu0           10.ee.12.200    Fa0/0         10.bb.ee.140    06 0A23 01BD     1
Tu0           10.ff.12.150    Fa0/0         10.bb.ee.130    06 048A 07DA     1
Tu0           10.gg.20.254    Fa0/0         10.bb.21.1      01 0000 0000    20
Tu0           10.hh.20.254    Fa0/0         10.bb.21.1      01 0000 0000    20
Tu0           10.ff.12.150    Fa0/0         10.bb.ee.11     06 048A 04A7     1
Tu0           10.oo.12.210    Fa0/0         10.bb.12.200    11 0035 EA0B     1
Tu1           203.151.20.17   Fa0/0         10.bb.50.200    06 0050 055D     5
Tu1           203.151.20.17   Fa0/0         10.bb.50.200    06 0050 055E    10

As you can see it includes statistics about the packet size distribution, the various protocols and amount of traffic for each protocol and then a summary listing of the traffic through the various interfaces on the router. In this case, the traffic is passing through a couple of encrypted tunnel interfaces. This is where things get interesting when troubleshooting traffic on a link. The first column is the source interface, then the source IP address. The third column is the destination interface followed by the destination IP address. The next 3 columns give some critical information about the traffic between the source and destination hosts. These values are all given in HEX. There is the protocol number (e.g. 01 - ICMP, 06 - TCP, 11 - UDP). See the protocol listing at IANA for more information on these numbers - remember to convert from HEX to decimal.

The next two columns are the source port and destination port pairing. These values are also in HEX. So, converting values like 01BD to 445 indicates that the traffic is Microsoft DS according to the port number listing at IANA. Port 0035 (53 decimal) would be DNS traffic. Port 0050 (80 decimal) would be http traffic. Port 01BB (443 decimal) would be https. So, as you can see, lots of information is right there on the router and no sniffing is required to see what traffic is on your network.

Once you find an “interesting” source and destination pair that concerns you, you might consider finding out what application is generating the traffic between that source / destination pair. This can be done unobtrusively using some of the excellent tools from the Microsoft/Sysinternals toolkit. For example, the following command will list the current tcp & udp connections on a remote computer (10.xx.50.81) - note that you must have administrative access to the remote computer to run this command (netstat is not a Sysinternals tool but is built into the Windows operating system):

psexec \\10.xx.50.81 netstat -ano

The output would look something like this:

PsExec v1.94 - Execute processes remotely

Copyright (C) 2001-2008 Mark Russinovich

Sysinternals - www.sysinternals.com

Active Connections

Proto Local Address Foreign Address State PID

TCP 0.0.0.0:135 0.0.0.0:0 LISTENING 852

TCP 0.0.0.0:445 0.0.0.0:0 LISTENING 4

TCP 0.0.0.0:5800 0.0.0.0:0 LISTENING 1748

TCP 0.0.0.0:5900 0.0.0.0:0 LISTENING 1748

TCP 0.0.0.0:8085 0.0.0.0:0 LISTENING 1456

TCP 10.xx.50.81:139 0.0.0.0:0 LISTENING 4

TCP 10.xx.50.81:445 10.bb.50.64:1826 ESTABLISHED 4

TCP 10.xx.50.81:1221 10.xx.12.200:135 ESTABLISHED 608

TCP 10.xx.50.81:1222 10.xx.12.200:1026 ESTABLISHED 608

TCP 10.xx.50.81:1822 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1823 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1827 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1828 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1829 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1830 10.xx.50.241:8080 ESTABLISHED 3756

TCP 10.xx.50.81:1831 10.xx.50.241:8080 ESTABLISHED 3756

TCP 127.0.0.1:1068 0.0.0.0:0 LISTENING 2412

UDP 0.0.0.0:445 *:* 4

UDP 0.0.0.0:500 *:* 608

netstat exited on 10.xx.50.81 with error code 0.

So, these results show that the host has various tcp & udp connections that are in an established state. It shows the source & destination ports (again like the show ip cache flow results). The other very useful piece of information that is shown is the PID or process identifier. This number matches a process running on the remote computer. So, to find out what the various running processes are and their PID’s, run the following command:

pslist \\10.xx.50.81

The results returned are like the following:

pslist v1.28 - Sysinternals PsList

Copyright ¬ 2000-2004 Mark Russinovich

Sysinternals

Process information for 10.xx.50.81:

Name Pid Pri Thd Hnd Priv CPU Time Elapsed Time

Idle 0 0 1 0 0 0:37:20.984 0:00:00.000

System 4 8 67 316 0 0:00:48.343 0:00:00.000

smss 464 11 3 21 164 0:00:00.015 4:43:15.698

csrss 528 13 15 545 2520 0:00:13.484 4:43:14.792

winlogon 552 13 19 524 9488 0:00:04.265 4:43:14.370

services 596 9 16 295 1876 0:00:04.281 4:43:14.183

lsass 608 9 20 428 4160 0:00:02.843 4:43:14.167

svchost 792 8 17 193 3284 0:00:00.796 4:43:13.667

svchost 852 8 10 371 2144 0:00:35.421 4:43:13.370

svchost 916 8 70 2092 16500 0:00:54.359 4:43:13.292

svchost 968 8 6 84 1596 0:00:00.921 4:43:13.245

svchost 992 8 15 292 3044 0:00:00.843 4:43:12.714

spoolsv 1196 8 12 142 3492 0:00:00.296 4:43:12.277

stormliv 1324 8 9 163 4952 0:00:08.343 4:43:04.339

EngineServer 1444 8 3 35 576 0:00:00.078 4:43:03.995

FrameworkService 1456 8 21 356 20632 0:00:37.203 4:43:03.573

VsTskMgr 1504 8 19 243 7128 0:00:29.578 4:43:02.714

MDM 1556 8 4 86 1092 0:00:00.140 4:43:02.495

mfevtps 1580 8 6 126 6848 0:00:02.609 4:43:02.370

ArchivingORBService 1636 8 4 88 3304 0:00:15.031 4:43:01.964

svchost 1696 8 5 118 2608 0:00:00.453 4:43:01.777

CcmExec 1836 8 13 810 14688 0:00:12.796 4:43:01.214

Mcshield 1880 13 26 182 45316 0:02:15.078 4:42:59.464

naPrdMgr 1964 8 6 130 208448 0:01:05.328 4:42:57.902

mfeann 1968 8 8 151 2264 0:00:01.625 4:42:57.855

alg 2412 8 5 102 1256 0:00:00.109 4:42:17.303

wmiprvse 2876 8 4 140 4132 0:00:00.781 4:42:09.979

wmiprvse 2660 8 7 146 1996 0:00:00.828 4:39:42.549

explorer 3676 8 12 442 17392 0:01:01.828 3:59:34.124

hkcmd 4092 8 2 86 896 0:00:00.140 3:59:30.406

igfxpers 816 8 3 93 868 0:00:00.078 3:59:30.343

UdaterUI 3388 8 5 115 1648 0:00:00.859 3:59:27.390

shstat 3252 8 10 98 2160 0:00:00.812 3:59:27.093

ctfmon 3968 8 1 67 984 0:00:00.156 3:59:25.828

Then if we need to remotely stop a running process that we consider suspicious or “interesting” issue the following command:

pskill 3968 \\10.xx.50.81 - note you can use either the PID # or the name of the process - however, you should use the PID if there are multiple instances of the application running

The results of the command, if successful, should look like:

PsKill v1.12 - Terminates processes on local or remote systems

Copyright (C) 1999-2005 Mark Russinovich

Sysinternals - www.sysinternals.com

Process 3968 on 10.xx.50.81 killed….

This process has become very useful when finding some rogue processes (malware) on some remote computers and there is no other way to disable the system or application. You can also issue a psshutdown command in a similar fashion, but the user may attempt to restart the machine again and then you will have to again shutdown the rogue application. There’s lots of ways to handle this situation including shutting down the LAN switch port if you have that access and privilege.

Let me know what processes you go through when managing remote systems where you may have limited physical access. Good luck out there and let’s be good network citizens!


Dec 10 2008   1:33PM GMT

Did you see this? - Latest Laura Chappell Newsletter



Posted by: Troy Tate
Networking, tools, Monitoring, reporting, IT education, performance monitoring, troubleshooting, howto, network analysis, Metrics, wireshark, packet capture, education, analysis

Newsletter 120908

Discount Codes - Nmap Book – Wireshark Certification Status – Global Knowledge – Movie Update - Virtual Conference Survey

 

Holiday/End-of-Year Specials at www.wiresharkU.com

  • 25% off on Wireshark University Self-Paced Courseware (code WSU1208)
  • $500 off already discounted price on Laura Chappell Master Library (code LCML1208)

Hot Links

 

Fyodor Releases Nmap Book

Gordon “Fyodor” Lyon, the creator of the must-have tool, Nmap, has released the long-awaited title “Nmap Network Scanning”. This 468-page book  nmap.org) is a required reading for anyone securing a network. I was thankful that Fyodor sent me a pre-release copy of the book, which was a blessing since the content was more in-depth than I’d hoped for. Chapters define scan variations, OS fingerprinting techniques, tips and tricks and the newly-developed ZenMap, the graphic front end for Nmap. “Nmap Network Scanning” should be front and center on your desk for months and years to come! Thanks, Fyodor!

Wireshark Certification Status

Final beta tests are underway for a planned January 2009 release of the long-awaited Wireshark Certification test. The Wireshark Certification Information Packet (WCIP) should be out at the beginning of the year (sign up to receive the document at www.wiresharktraining.com/certification).I know you’ve waited a long time for the certification and I appreciate your patience - it took me a lot longer to get the questions together and ensure we could deliver via the Internet.

Global Knowledge Signed as Wireshark Authorized Training Partner

We are thrilled to sign on Global Knowledge as our North American Wireshark University Authorized Training Partner. In Q1 2009, two new Wireshark courses release – the first course focuses on Wireshark basic through advanced functionality and in-depth review of TCP/IP communication patterns (CORE 1). The second course delves into troubleshooting and network forensics with the Wireshark Certification Vouchers included in the course price (CORE 2). More course information will be put on www.wiresharkU.com before end of year.Read the press release.

The Only Thing Slower than This Network is Hollywood!

Well, folks… after hearing about the ‘movie’ project for a few years now, you’re probably thinking the darn thing isn’t going to make it out there. You’re probably right, but one more step was checked off last week – the script was finalized. The writers, Joe D’Ambrosia and Tom Teves (Murray Hill 5 Productions) gave me the near-shocking news on Friday. If you don’t know these guys, check out the “Dead Rocker” (appropriate for kids) at www.youtube.com/watch?v=I5aD00UeE9g. I kinda figured out who the murderer was (at least I knew what the main clue was) after watching a second time. Can you/your kids figure it out?

Happy Holidays to All! [Oh... and if you checked out the movie link...no, I'm not a spy and no, my kids don't play soccer.]

Laura Chappell


Dec 10 2008   1:19PM GMT

The larger world of free technical support - Craigslist computer forum



Posted by: Troy Tate
Networking, tools, documentation, patching, web, IT education, malware, website, anti-virus, troubleshooting, howto, online identity, risk, awareness, education, professional

I recently came across the computer forums on Craigslist. I had heard of Craigslist previously but was not aware of the significant scope of what it offers besides classified ads. I’m not suggesting that folks leave ITKE to the Craigslist computer forums for support - far from that.

I have been watching some of the exchanges between posters on the forums. It amazes me what people will post when the environment provides seeming anonymity. ITKE does offer this also, but the moderators do a great job of keeping the Trolls away. The Craigslist posters do not behave in the same professional manner that ITKE users do. There are many writers on Craigslist that belittle computer user skills for those asking “noob” questions. There are also those who attempt to discredit or otherwise tear down answers from those who have real computer skills and knowledge.

For example, there was a recent posting thread reminding people that the Microsoft Tuesday patches had been released. One feature that Craigslist offers similar to ITKE is the ability to rate postings. Someone rated the patch Tuesday reminder as a “thumbs down” posting. This is really unprofessional behavior. The thread went on to describe that exploits were sure to follow the patches since hackers use the patches to reverse-engineer the vulnerability. Someone asked if the exploits could already exist. Of course they might, but the exploits would become more likely after the patches are released.

The thread also described how the patches are to protect users from themselves. Most users are tempted into doing something (downloading software, answering yes to some popup window, visiting that interesting website) that causes malicious software to do something on their system that is totally unintended by the users… so the patch is there to fix some things that might be otherwise used by these malware writers. Really patching is the only preventive mechanism. Antivirus is a detective method that detects when something is trying to do something it shouldn’t. Patching won’t let those things happen - unless the user makes a poor judgement call… we all do! I have even opened an infected PDF file thinking it was a legitimate document. Fortunately, AV was able to clean up after my mistake.

For some reason, some troll thought they would say that these postings were by a “know-it-all n00b”. It seems like this Craigslist forum user may be one of those miscreants who want people to remain ignorant and cannot handle someone else teaching others about safe computing and answering other users’ questions.

I would like to encourage ITKE readers and IT professionals to help make IT support forums professional and user-friendly. If you have time, watch the Craigslist computer forums, offer support to the users  who don’t have the same professional support available we have on ITKE. Make the trolls look even worse by treating the users with some respect. By sharing our knowledge and skills, we can help users use the computer in a productive manner. Thanks for reading this and hope you join me in sharing knowledge either here on ITKE and/or Craigslist.