An IT Nightmare – A true story from the IT trenches
by December 11, 2013

Filed under: Industry Insights

Eight-alarm fire guts South Boston building

fire internet connectivity

Firefighters battled an early morning blaze on Summer Street in Boston, just two doors down from AppNeta’s Office.  The blaze was discovered around 4:29 a.m., according to the Boston Fire department’s official Twitter account.

Wow… that was a little too close to home!  While we didn’t receive any physical damage to our building, we did see a dramatic increase in packet loss in two of our internet connections.  Within an hour and a half of the fire being discovered, PathViewCloud (PVC) started showing us the issues.

fire internet connectivity

fire internet connectivity

Further investigation in PVC reveals that the issue started three hops out from our firewall.

fire internet connectivity

Our Boston office systems administrator quickly contacted our Internet provider to let them know about the issue.  As usual, they had no idea that there was a problem.  After investigating further, they discovered errors in the stats on the routers and dispatched a technician.

Nearly thirty two hours after the fire had first been reported, the issue was resolved and the packet loss had stopped.

fire internet connectivity

The two charts below, show monitoring results from two remote locations, Vancouver and Texas.  Both locations show packet loss to the Boston office internet connections, all starting at the same time as the fire.

fire internet connectivity

fire internet connectivity

Once again PVC has come to the rescue. It shows us the issue and allows us to verify that everything is back to normal. What more could I ask for other than to have it phone the ISP?