Categories Industry Insights

An IT Nightmare – A true story from the IT trenches

Eight-alarm fire guts South Boston building

Firefighters battled an early morning blaze on Summer Street in Boston, just two doors down from AppNeta’s Office.  The blaze was discovered around 4:29 a.m., according to the Boston Fire department’s official Twitter account.

Wow… that was a little too close to home!  While we didn’t receive any physical damage to our building, we did see a dramatic increase in packet loss in two of our internet connections.  Within an hour and a half of the fire being discovered, PathViewCloud (PVC) started showing us the issues.

Further investigation in PVC reveals that the issue started three hops out from our firewall.

Our Boston office systems administrator quickly contacted our Internet provider to let them know about the issue.  As usual, they had no idea that there was a problem.  After investigating further, they discovered errors in the stats on the routers and dispatched a technician.

Nearly thirty two hours after the fire had first been reported, the issue was resolved and the packet loss had stopped.

The two charts below, show monitoring results from two remote locations, Vancouver and Texas.  Both locations show packet loss to the Boston office internet connections, all starting at the same time as the fire.

Once again PVC has come to the rescue. It shows us the issue and allows us to verify that everything is back to normal. What more could I ask for other than to have it phone the ISP?

Jason Hislop: