Even the CMO Can Troubleshoot: The Day Salesforce Went Down
One morning last week, I came into the office to find that the folks in our Boston office couldn’t consistently access Salesforce.com. But of course I already knew this. I had gotten an alert from AppNeta Performance Manager that the app wasn’t functioning correctly, so I could quickly troubleshoot the issue. As Chief Marketing Officer (CMO) at AppNeta, it’s definitely not part of my job to make sure Salesforce is up and running. This kind of issue impacts the whole organization, though, so I dove in.
After getting the email alert, I immediately went to our usage analysis feature to see if there was another application affecting our Salesforce performance. AppNeta’s usage analysis automatically identifies all the applications in use across your organization, and can quickly see if there’s an application or process, such as a backup, impacting the performance of other applications. But as you can see in this image, there isn’t a single application sucking up the bandwidth. As you can imagine, a lot of our traffic is from our own solution.
Next, I went to our experience component. AppNeta’s experience module performs synthetic web transactions against a given website. What’s unique about AppNeta is that this synthetic traffic originates from our offices for a real-time view of the actual end-user experience for any given application. Since AppNeta uses Salesforce (along with a host of other web and SaaS-based web applications) a lot, our IT department had already set up tests to monitor Salesforce performance from our various offices.
I examined tests that we had set up from our three offices (Boston, Vancouver and San Francisco) as well as from a global monitoring point outside our offices and infrastructure. My first troubleshooting clue was that only the Boston office was having an issue. You can see that in the screenshot below where the gray boxes are—the transactions simply couldn’t be performed for Salesforce.
But the other offices and the global monitoring points weren’t showing the same issue. So I could answer the old “Is it the application or the network” question: The application is working. It’s just not working for our Boston office.
Having determined that the issue was network-related, I switched to our delivery analysis, shown below. Bingo. Packet loss is likely causing the issue. So now we can confirm it’s a network issue.
The next logical question is: Where is the network issue occurring and what’s the root problem? Is it with our WiFi? Behind our firewall? Or is it something happening on the internet or with the application provider? I can’t troubleshoot the network if I don’t know where the issue is.
This is where AppNeta takes troubleshooting a step further than all the other tools out there. Our patented TruPath technology can see network performance across any network, whether we own it or not. Contrast that with a classic SNMP-based tool like SolarWinds or Riverbed, which would be able to tell me that there was a network issue, but not where it’s occurring once the traffic hit the open internet.
AppNeta provides quick answers to network problems with our built-in, automatic diagnostics. Whenever a network issue is detected, we automatically kick off a diagnostic that examines every hop along the application delivery path.
Once I know the problem is packet loss, I’d like to know where it’s occurring. AppNeta had already performed a diagnostic, so I could see right away that the problem wasn’t on our network, but with a router owned by Comcast that was dropping packets like grocery bagger on roller skates. I also found packet loss at an edge router owned by Salesforce.
In less than five minutes, I figured out the issue and contacted Salesforce. Unlike all the other customers contacting Salesforce, we were able to tell them exactly where the network issue was so they could focus on getting it fixed. Alternately, we could have called Comcast to tell them about the issue, and provide our detailed diagnostics to back up what we’re seeing. (We’ve done it before, and they are amazed that we know the specific router!)
This is a great real-world example of how AppNeta Performance Manager provides visibility into the performance of applications and networks. Plus, it’s nice to see first-hand what our customer experience when they use AppNeta. The tool is so easy that even this marketing guy can find and fix problems fast.