tag:status.deft.com,2005:/historyDeft Status - Incident History2024-03-29T00:23:09-05:00Defttag:status.deft.com,2005:Incident/108069482022-08-11T19:34:05-05:002022-08-19T09:41:24-05:00ORD IP Transit<p><small>Aug <var data-var='date'>11</var>, <var data-var='time'>19:34</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>11</var>, <var data-var='time'>16:45</var> CDT</small><br><strong>Update</strong> - Service continues to be stable at this time. Deft Network Engineering will re-enable the circuit in question after business hours and continue to monitor.</p><p><small>Aug <var data-var='date'>11</var>, <var data-var='time'>13:23</var> CDT</small><br><strong>Monitoring</strong> - Deft's 24/7 NOC has monitored an event with one of our upstream IP Transit circuits. Customers may have noticed a brief interruption in a portion of their internet traffic. We have disabled this circuit for the time being to mitigate any further impact.</p>tag:status.deft.com,2005:Incident/96871412022-04-11T13:32:47-05:002022-08-19T09:41:24-05:00Looking Glass - Undergoing Maintenance<p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>13:32</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>13:32</var> CDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>11:55</var> CDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>11:53</var> CDT</small><br><strong>Monitoring</strong> - Currently our Looking glass, https://lg.deft.com is down for maintenance</p>tag:status.deft.com,2005:Incident/49749902020-08-30T10:37:21-05:002022-08-19T09:41:24-05:00CenturyLink / Level3 Routing Incident<p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>10:37</var> CDT</small><br><strong>Resolved</strong> - CenturyLink has advised that they are taking corrective action and expect things to be clearing soon if not already. For our part, we are leaving them in a non-preferred state for the time being.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>09:52</var> CDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>09:50</var> CDT</small><br><strong>Update</strong> - CenturyLink have advised they have identified the issue and are taking corrective measures.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>08:16</var> CDT</small><br><strong>Identified</strong> - SCTG NOC has monitored trouble with the CenturyLink / Level 3 (AS3356) network. We have taken steps to mitigate but there are still issues remaining due to the size and scope of their network.</p>tag:status.deft.com,2005:Incident/47790622020-08-04T13:32:25-05:002022-08-19T09:41:25-05:00Chicago area dark fiber brief interruption<p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>13:32</var> CDT</small><br><strong>Resolved</strong> - Everything continues to be stable. We will follow up with vendor but expect no further issues.</p><p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>09:59</var> CDT</small><br><strong>Monitoring</strong> - SCTG Network Operations have identified a brief interruption to one of our redundant dark fiber paths between 9:29 AM CDT and 9:42 AM CDT. All services are restored. We are monitoring and pursuing this with the vendor.</p>tag:status.deft.com,2005:Incident/21784402019-01-29T18:03:59-06:002022-08-19T09:41:25-05:00Inbound Phone Issues<p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>18:03</var> CST</small><br><strong>Resolved</strong> - We've moved back to our primary provider.</p><p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>17:01</var> CST</small><br><strong>Identified</strong> - Calls to our main IVR and NOC have been rerouted to an alternate phone provider while we troubleshoot the issue with our primary vendor.</p>tag:status.deft.com,2005:Incident/21090262018-12-20T10:55:27-06:002022-08-19T09:41:25-05:00Increased Latency<p><small>Dec <var data-var='date'>20</var>, <var data-var='time'>10:55</var> CST</small><br><strong>Resolved</strong> - Storage latency has returned to normal levels. Enterprise Cloud is operating normally at this time.</p><p><small>Dec <var data-var='date'>20</var>, <var data-var='time'>10:43</var> CST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Dec <var data-var='date'>20</var>, <var data-var='time'>10:43</var> CST</small><br><strong>Investigating</strong> - ServerCentral Managed Services staff are investigating reports of increased storage latency affected Enterprise Cloud ORD.</p>tag:status.deft.com,2005:Incident/21000472018-12-14T22:00:15-06:002022-08-19T09:41:25-05:00Packet Loss on Enterprise Cloud ORD6<p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>22:00</var> CST</small><br><strong>Resolved</strong> - This issue was narrowed down to an incorrect configuration applied earlier this evening and has been resolved.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>21:39</var> CST</small><br><strong>Investigating</strong> - ServerCentral Turing Group is exploring a higher than usual amount of bandwidth entering our network. Some customers may experience slower than normal network response times. We are currently investigating this issue.</p>tag:status.deft.com,2005:Incident/19193172018-09-20T00:01:46-05:002022-08-19T09:41:25-05:00Veeam Cloud Connect - ORD6<p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>00:01</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>23:48</var> CDT</small><br><strong>Identified</strong> - We have identified the issue and are working to restore service.</p><p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>23:42</var> CDT</small><br><strong>Investigating</strong> - We are investigating reports of an issue relating to connecting to the Veeam Cloud Connect repository in our Elk Grove Village area data center.</p>tag:status.deft.com,2005:Incident/19053262018-09-13T07:34:07-05:002022-08-19T09:41:25-05:00Veeam Cloud Connect - ORD6<p><small>Sep <var data-var='date'>13</var>, <var data-var='time'>07:34</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>13</var>, <var data-var='time'>01:19</var> CDT</small><br><strong>Monitoring</strong> - Services have been restored and are resuming normal activity. We are continuing to monitor the platform to ensure stability.</p><p><small>Sep <var data-var='date'>13</var>, <var data-var='time'>01:13</var> CDT</small><br><strong>Identified</strong> - We have identified the issue and are working to restore service.</p><p><small>Sep <var data-var='date'>13</var>, <var data-var='time'>00:47</var> CDT</small><br><strong>Investigating</strong> - We are investigating reports of an issue relating to connecting to the Veeam Cloud Connect repository in our Elk Grove Village area data center.</p>tag:status.deft.com,2005:Incident/18957722018-09-06T12:22:21-05:002022-08-19T09:41:25-05:00Veeam Cloud Connect - ORD6<p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>12:22</var> CDT</small><br><strong>Resolved</strong> - This issue has been deemed resolved by our Managed Services team and all functionality is operating normally.</p><p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>11:40</var> CDT</small><br><strong>Update</strong> - Services have been restored and are resuming normal activity. We are continuing to monitor the platform to ensure stability.</p><p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>11:34</var> CDT</small><br><strong>Identified</strong> - We have identified the issue and are working to restore service.</p><p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>11:28</var> CDT</small><br><strong>Investigating</strong> - We are investigating reports of an issue relating to issues connecting to the Veeam Cloud Connect repository in our Elk Grove Village area datacenter.</p>tag:status.deft.com,2005:Incident/18722212018-08-21T22:17:24-05:002022-08-19T09:41:25-05:00VCC-ORD6<p><small>Aug <var data-var='date'>21</var>, <var data-var='time'>22:17</var> CDT</small><br><strong>Resolved</strong> - This issue has been resolved. All services are now operational.</p><p><small>Aug <var data-var='date'>21</var>, <var data-var='time'>22:03</var> CDT</small><br><strong>Monitoring</strong> - We have resolved the issue and all system functionality is being restored.</p><p><small>Aug <var data-var='date'>21</var>, <var data-var='time'>21:53</var> CDT</small><br><strong>Identified</strong> - We have Identified the issue and are actively working to resolve it.</p><p><small>Aug <var data-var='date'>21</var>, <var data-var='time'>21:45</var> CDT</small><br><strong>Investigating</strong> - ServerCentral Staff are currently examining an issue related to Veeam Cloud Connect and Backup Repositories located in our Elk Grove Village Datacenter. Further details will be posted to status.servercentral.com.</p>tag:status.deft.com,2005:Incident/17938482018-06-29T13:30:42-05:002022-08-19T09:41:25-05:00Comcast Backbone<p><small>Jun <var data-var='date'>29</var>, <var data-var='time'>13:30</var> CDT</small><br><strong>Resolved</strong> - Comcast has implemented a temporary solution as they deal with multiple fiber cuts on their backbone</p><p><small>Jun <var data-var='date'>29</var>, <var data-var='time'>12:11</var> CDT</small><br><strong>Investigating</strong> - The ServerCentral NOC monitoring systems have detected issues within Comcast's network. We are working to minimize the impact to our customers.</p>tag:status.deft.com,2005:Incident/16251552018-03-02T08:14:31-06:002022-08-19T09:41:25-05:00350 Cermak Fiber Riser, Loss of Redundancy<p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>08:14</var> CST</small><br><strong>Resolved</strong> - Repairs are complete on our DWDM system, and we are back to normal.</p><p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>06:08</var> CST</small><br><strong>Identified</strong> - ServerCentral has disabled a segment of our DWDM system on riser fiber between Floors 1 and 5 at 350 E Cermak. No customer capacity has been affected, but we have lost some redundancy within the building. We expect to have it repaired shortly.</p>tag:status.deft.com,2005:Incident/16125992018-02-22T09:39:04-06:002022-08-19T09:41:25-05:00Chicago 350 Cermak Core Router Process Bounced<p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>09:39</var> CST</small><br><strong>Resolved</strong> - Efforts continue in working with our vendor to determine the root cause of the incident.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>18:22</var> CST</small><br><strong>Update</strong> - Senior Network Engineering continues to monitor the situation closely.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>18:20</var> CST</small><br><strong>Monitoring</strong> - We are placing the affected devices under a configuration freeze until such time as our consultation with our vendor reaches a positive conclusion.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>18:08</var> CST</small><br><strong>Investigating</strong> - We saw the routing process on one of our two Core Routers in 350 Cermak bounce at 23:56 UTC today. We are investigating the event with our vendor.</p>tag:status.deft.com,2005:Incident/16062242018-02-16T11:05:51-06:002022-08-19T09:41:25-05:00Chicago 350 Cermak Core Router Process Bounced<p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>11:05</var> CST</small><br><strong>Resolved</strong> - We are continuing our follow up investigation with our vendor, but expect no further impact at this time.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>08:00</var> CST</small><br><strong>Monitoring</strong> - We saw the routing process on one of our two Core Routers in 350 Cermak bounce at 13:25 UTC today. We are investigating the event with our vendor.</p>tag:status.deft.com,2005:Incident/14814132017-11-06T13:24:40-06:002022-08-19T09:41:25-05:00Network Latency<p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>13:24</var> CST</small><br><strong>Resolved</strong> - Level3 has stopped leaking Comcast routes to NTT.</p><p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>13:02</var> CST</small><br><strong>Monitoring</strong> - ServerCentral is now blocking Comcast routes that were leaked from Level3 to NTT. Level3 is still leaking these routes, which may be causing congestion on Level3's network.</p><p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>12:53</var> CST</small><br><strong>Update</strong> - ServerCentral has removed the affected routes from this issue, and traffic should begin flowing normally to Comcast at this time.</p><p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>12:26</var> CST</small><br><strong>Identified</strong> - ServerCentral has identified an issue with Comcast, and is working with them to resolve. We are working to reroute the affected traffic.</p><p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>12:19</var> CST</small><br><strong>Investigating</strong> - We are investigating reports of latency on our Network</p>tag:status.deft.com,2005:Incident/13482012017-09-05T18:08:21-05:002022-08-19T09:41:25-05:00Enterprise Cloud ORD6<p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>18:08</var> CDT</small><br><strong>Resolved</strong> - The incident has been resolved at approximately 6:05PM CDT. We apologize for the inconvenience this may have caused.</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>17:59</var> CDT</small><br><strong>Identified</strong> - Our engineers have identified the issue and are working to bring the environment back online.</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>17:42</var> CDT</small><br><strong>Update</strong> - Our engineers are still currently investigating the interruption of service.</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>17:07</var> CDT</small><br><strong>Investigating</strong> - At approximately 4:50PM Central time, the Enterprise Cloud ORD6 availability zone experienced a partial outage. Engineers are investigating the cause now.</p>tag:status.deft.com,2005:Incident/13382202017-08-24T15:20:45-05:002022-08-19T09:41:25-05:00CR2.TK01<p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>15:20</var> CDT</small><br><strong>Resolved</strong> - The Network event is resolved. One router was affected resulting in an outage for single homed customers. We apologize for the inconvenience this may have caused.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>14:57</var> CDT</small><br><strong>Investigating</strong> - We are investigating a network event in Tokyo, Japan. Senior network engineering staff are currently working to isolate and resolve the issue.</p>tag:status.deft.com,2005:Incident/13342852017-08-21T14:27:53-05:002022-08-19T09:41:25-05:00We are investigating Network Software issue in SJC1 San Jose, CA affecting customers beginning at 1:53 PM CDT. Technicians are currently working to determine the problem. An update will be provided in 30 minutes.<p><small>Aug <var data-var='date'>21</var>, <var data-var='time'>14:27</var> CDT</small><br><strong>Resolved</strong> - The Network Software event is resolved. One router was affected resulting in an outage for single homed customers. We apologize for the inconvenience this may have caused.</p><p><small>Aug <var data-var='date'>21</var>, <var data-var='time'>13:55</var> CDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.deft.com,2005:Incident/13299942017-08-16T16:53:20-05:002022-08-19T09:41:25-05:00Amsterdam Routing Anomoly<p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>16:53</var> CDT</small><br><strong>Resolved</strong> - We are declaring this situation resolved.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>12:35</var> CDT</small><br><strong>Monitoring</strong> - A core router in Amsterdam has experienced an un-expected routing restart. It has recovered and we are working with out vendor to identify the cause and mitigate against future occurrences.</p>tag:status.deft.com,2005:Incident/13262602017-08-12T02:15:54-05:002022-08-19T09:41:25-05:00IAD Networking Issue Aug 12<p><small>Aug <var data-var='date'>12</var>, <var data-var='time'>02:15</var> CDT</small><br><strong>Resolved</strong> - One of our redundant core routers in Ashburn, VA experienced an issue between 01:24AM and 01:44AM CDT.
<br />Redundant services failed over expected. Single homed customers may have experienced a brief loss of connectivity as the router recovered.</p><p><small>Aug <var data-var='date'>12</var>, <var data-var='time'>01:50</var> CDT</small><br><strong>Investigating</strong> - ServerCentral personnel are investigating a network issues in Ashburn, VA. More details to be posted as they are available.</p>tag:status.deft.com,2005:Incident/13253512017-08-11T04:22:54-05:002022-08-19T09:41:26-05:00IAD network issues<p><small>Aug <var data-var='date'>11</var>, <var data-var='time'>04:22</var> CDT</small><br><strong>Resolved</strong> - A router in IAD experienced a router anomaly and recovered as expected. This resulted in a line card restart. Single threaded customers experienced a brief drop in connectivity as the line card restarted.</p><p><small>Aug <var data-var='date'>11</var>, <var data-var='time'>03:36</var> CDT</small><br><strong>Investigating</strong> - ServerCentral personnel are investigating possible network issues in the IAD network. More details to be posted as they are available.</p>tag:status.deft.com,2005:Incident/13077872017-07-25T20:10:54-05:002022-08-19T09:41:26-05:00European Latency<p><small>Jul <var data-var='date'>25</var>, <var data-var='time'>20:10</var> CDT</small><br><strong>Resolved</strong> - Services temporarily degraded in European region due to long haul Transit issue. Issue occurred between Jul 26 00:50 and Jul 26 00:56 UTC Service has since been restored.</p>tag:status.deft.com,2005:Incident/13054272017-07-22T22:28:47-05:002022-08-19T09:41:26-05:00routing reconvergence<p><small>Jul <var data-var='date'>22</var>, <var data-var='time'>22:28</var> CDT</small><br><strong>Resolved</strong> - At shortly after 22:00 CDT some routes from the Elk Grove Village Datacenter, mostly to AWS and Comcast were dropped. Routing has re-converged, and we are expecting no further issue</p>tag:status.deft.com,2005:Incident/12490312017-05-22T18:22:19-05:002022-08-19T09:41:26-05:00Degraded Hardware in Sydney, AU.<p><small>May <var data-var='date'>22</var>, <var data-var='time'>18:22</var> CDT</small><br><strong>Resolved</strong> - Degraded hardware has been restored to proper function. No more updates.</p><p><small>May <var data-var='date'>22</var>, <var data-var='time'>17:44</var> CDT</small><br><strong>Identified</strong> - We have identified degraded hardware and working to restore.</p>