Filed under: Performance Monitoring
As a pre-sales engineer, I see a lot of interesting network performance management scenarios while working with future customers on product trials. I’ve seen everything from a managed switch that had a rogue 10-meg port to a problematic WiFi access point, located in the basement of a hospital!
On a recent trial , I was working with a network engineer who works for a video conferencing services provider. Contrary to what I expected, they were not looking to solve a customer’s problem. This particular customer was concerned with their own internal Unified Communications platform. There were three core offices on the east coast, and a remote office in the UK. Once I heard we were dealing with UC over the WAN between remote offices, I thought “Jackpot! This is PathView Cloud’s forte.” This is going to be like a Shaquille O’Neal dunk at the Garden. However, in the words of Lee Corso, “Not so fast, my friend.”
Once we had the PathView Cloud microAppliances deployed in the four various offices, we configured the network paths in a full mesh manner. The spider web was starting to come together very nicely. But as I looked the PathView dashboard, I started to see some violations represented by red bubbles on the interface.
Looking more closely at the results of the hop by hop path analysis, it was evident that duplex conflicts everywhere. Every phone, video camera, even PC in this office was showing a duplex conflict. “Great!” I thought. We found the problem. Change the duplexes. Case closed, let’s go home.
If it were only that easy…
Once we changed the settings, PathView Cloud continued to detect errors. Some were cable errors, some were limiting errors. I started to scratch my head – what could this possibly be? I ran the results by a couple other engineers on my team. Adam Edwards, Director of Systems Engineers, thought it could be the switch which was bad.
After I shared the findings with the customer, he was a bit hesitant because his SNMP polling device was saying the switch was running as it is supposed to. To humour me, he swapped out the switch. As soon as that happened, the performance was drastically better. We still have not been able to definitively determine what caused the issue, though we are pretty sure it’s something tied to the settings that controlled the RTP/RTCP streams.
During the analysis phase of this, it felt like a twelve round bout with Mike Tyson. We found duplex conflict, rate limiting, and eventually uncovered that the whole switch was bummed out. All of this came right through the PathView Cloud interface within minutes of deploying the microAppliances. PathView Cloud performance was like David Ortiz batting in the bottom 9th. Only he will win.