Affecting Other - Network

• 03/05/2024 16:09 - 03/06/2024 03:19
• Last Updated 03/06/2024 03:19

We have been made aware of a partial network outage in our LNR1 data center that is affecting only a small segment of customers. Our NOC is currently working on the issue and we will provide updates as soon as we have them.

Incident: https://status.nodespace.com/incident/572007 

As of 10:10 PM (Eastern), this issue has been resolved

Affecting Other - LNR Network

• 02/07/2024 23:00 - 02/08/2024 01:00
• Last Updated 03/01/2024 03:33

Start Time: February 07, 2024, 23:00 EST
End Time: February 08, 2024, 01:00 EST
Service Call Reference Number: SM464887
Services Affected: Network Services Service
Location(s): 801 Main Street. Lenoir, NC
Expected Impact: Network outage

Details: The NodeSpace Network Engineering Team wishes to inform you of upcoming necessary software upgrades to our Transit router within the Data Center. To ensure continuous service, network traffic will be rerouted through our backup router during this period. While we anticipate minimal disruptions, you may experience brief, intermittent outages during the failover procedure. We aim to complete these critical enhancements swiftly, expecting a duration of approximately 60 minutes. We appreciate your understanding and patience as we enhance our network's reliability and performance.

https://status.nodespace.com/incident/464887 

Affecting Server - greywood.boltns.com

• 11/15/2023 18:21 - 11/15/2023 18:30
• Last Updated 11/15/2023 18:30

We are currently investigating an issue with our Greywood cPanel server.

Update: The issue appears to be resolved.

Affecting System - Server Manager

• 10/25/2023 19:35 - 10/25/2023 22:55
• Last Updated 10/25/2023 22:55

We are going to be performing emergency maintenance on our server manager (https://manage.nodespace.com) application. During this time, management of VPS servers and dedicated servers will temporarily be interrupted and unavailable. New server orders will temporarily be delayed. If you need assistance with your server during this outage, please open a support ticket or send an email to support@nodespace.com. 

This outage is expected to last less than 1 hour and will start around 4:00 PM Eastern Time and last until around 5:00 PM Eastern Time.

We apologize for any inconvenience caused.

Update:The maintenance is taking a little longer than expected. We apologize for any inconvenience.

Update: Our server manager is now back online - we are currently working to fix the routing for IPMI and this should be back online in the next hour or so. VPS functionality, however, is restored.

Update: IPMI access has been restored for all dedicated server customers. This maintenance is now complete.

Affecting Server - node862.nodespacehosting.com

• 10/25/2023 02:59 - 10/25/2023 03:30
• Last Updated 10/25/2023 03:29

We are currently investigating a major outage of cPanel NODE862.

Incident: https://nodespace.freshstatus.io/incident/397752 

UPDATE: The issue is now resolved.

Affecting Other - LNR Network Maintenance

• 09/23/2023 04:00 - 09/23/2023 05:05
• Last Updated 09/23/2023 05:04

We are currently performing network maintenance in our North Carolina data center to expand network capacity. There will be several brief periods of downtime between 10 to 15 minutes each between 12:00 AM EDT and 1:00 AM EDT.

Update: At this time the, network maintenance is complete.

Affecting Other - LNR1 Data Center Network

• 07/13/2023 12:33 - 07/20/2023 03:16
• Last Updated 07/20/2023 03:16

We are aware of a network latency issue that is affecting our LNR1 data center. Our NOC and network security teams are working on the issue and have put mitigations in place. We have an open incident on our status page you can follow for the latest updates as well as our Mastodon: https://nodespace.social/@nodespace 

Incident page: https://nodespace.freshstatus.io/incident/366083 

UPDATE: At this time our NOC has resolved the issue and there should be no further unexpected latency.

Affecting Other - LNR Data Center

• 07/12/2023 17:56 - 07/12/2023 18:20
• Last Updated 07/12/2023 18:20

We are aware of a possible issue affecting networking with our LNR data center. We will be updating the following incident with more details and updates:

https://nodespace.freshstatus.io/incident/365800 

UPDATE: We have noticed the network improve so we are going to close this incident out.

Affecting System - kvmvps01.lnr.nodespace.com

• 06/24/2023 16:59 - 06/25/2023 01:30
• Last Updated 06/25/2023 01:30

We are currently aware of, and working on, an issue with one of our legacy VPS hosts.

We will be regularly posting updates on our status site: https://nodespace.freshstatus.io/incident/362356 as well as here.

UPDATE: We are going to resolve this. We've been monitoring and we have been receiving some false-positive alerts for a while now so we're going to adjust our monitoring but we feel that this issue is resolved. If there are any issues, please contact support.

Affecting System - vps.nodespace.net

• 05/25/2023 02:05 - 05/25/2023 02:43
• Last Updated 05/25/2023 02:43

We are currently performing emergency maintenance on our legacy VPS system: https://vps.nodespace.net. During this time, management of legacy VPS systems will be impacted by slower than usual performance while our maintenance is underway. We're not expecting this to last long.

Update: Maintenance is complete.

Affecting System - manage.nodespace.net

• 05/09/2023 17:41 - 05/09/2023 23:46
• Last Updated 05/09/2023 23:46

We are changing our server manager domain name from manage.nodespace.net to manage.nodespace.com.

During this process, the server manager may be unavailable briefly while the switch is occurring and while we point our systems to the new URL.

This impacts some VPS customers and most dedicated server customers. If you need to perform system administration tasks and the server manager is unavailable, please submit a ticket with your request and we will waive our fee during the maintenance.

Update: This is complete. manage.nodespace.com is the primary domain, manage.nodespace.net will continue to work for the foreseeable future.

Affecting System - KVM01 - Lenoir NC

• 05/08/2023 01:55 - 05/08/2023 02:21
• Last Updated 05/08/2023 02:21

Our host node maintenance encountered a kernel panic and we are working to resolve the issue. Please follow the following incident:

https://nodespace.freshstatus.io/incident/346710

Update: This issue is now resolved.

Affecting System - kvm-uk2.vps.voxvm.com

• 04/28/2023 04:48 - 04/28/2023 05:07
• Last Updated 04/28/2023 05:07

We are currently investigating an issue with our legacy UK2 VPS node.

Update: The issue has been resolved.

Affecting Server - greywood.boltns.com

• 04/20/2023 03:25 - 04/20/2023 04:03
• Last Updated 04/20/2023 04:03

We are currently investigating issues with Greywood. There may be intermittent downtime as we troubleshoot.

Issue seems to be resolved.

Affecting Server - alfa.boltns.com

• 02/21/2023 13:00 - 02/21/2023 13:48
• Last Updated 02/21/2023 13:48

We are aware of an issue with alfa.boltns.com and we are working to resolve the problem. We will keep you updated with additional details on the incident page: https://nodespace.freshstatus.io/incident/313149 

The issue has been resolved.

Affecting Server - NODE02.NODESPACESERVERS.COM

• 05/19/2022 07:12 - 07/01/2022 07:31
• Last Updated 07/01/2022 11:49

We are aware that one of our shared hosting servers is down. We are investigating it.

Affecting Other - Rack LNR.R100

• 04/29/2022 21:00 - 05/11/2022 00:38
• Last Updated 04/27/2022 00:22

Maintenance Start: April 29, 2022 at 10:00 PM Eastern Time (UTC-4)
Maintenance End: April 29, 2022 at 11:59 PM Eastern Time (UTC-4)

Maintenance Effects: Lenoir, NC Data Center - Rack 100

Maintenance Details:

We have identified an issue with the PDU in rack LNR.R100. While the PDU is functional, we have lost all remote control of it, including the ability to remote reboot, as well as monitoring. Because of this, we need to replace the entire PDU with a new functional one. During this maintenance window, servers located within in this rack will be offline upwards of about 30 minutes. However, we are planning about 2 hours of downtime in case of any unforeseen circumstances. We apologize about any inconvenience this may cause.

Maintenance Notification: https://status.nodespace.net/incident/171772 

Affecting Other - Rack PDU

• 04/19/2022 16:08 - 04/27/2022 00:22
• Last Updated 04/19/2022 16:10

We are currently performing emergency maintenance on one of our rack PDUs in our Lenoir data center. No interruption in service is expected and we're posting this as an advisory.

Incident Link: https://status.nodespace.net/incident/170119 

Affecting System - Lenoir, NC UPS

• 04/21/2022 09:00 - 04/27/2022 00:18
• Last Updated 04/19/2022 02:14

Maintenance Window: April 21, 2022

Start Time: 09:00 ET
End Time: 16:00 ET
Services Affected: Facility UPS systems
Expected Impact: There should be no client impact.

Details: Please be advised that our facilities team will be performing a UPS maintenance. There is no expected impact during this maintenance. This is an advisory only.

Affecting System - All Linux Servers

• 01/26/2022 19:00 - 04/19/2022 02:15
• Last Updated 01/26/2022 19:02

We are patching our servers for the polkit vulnerability. There will be no downtime, no server reboots.

Full maintenance posted here: https://status.nodespace.net/incident/148610 

Affecting Other - Lenoir, NC Data Center

• 01/05/2022 10:28 - 01/05/2022 11:27
• Last Updated 01/06/2022 06:34

Our Lenoir, NC facility is currently experiencing an issue with the network. The following incident has been opened for tracking purposes:

https://status.nodespace.net/incident/143377

Update 1: We resolved the first incident prematurely. The network outages have continued on and we have opened a second incident: https://status.nodespace.net/incident/143382 

Update 2: 11:27 AM At this time the network has been stable for about 30 minutes so we feel confident resolving this issue. This issue also brought to our attention that we were hosting some relay infrastructure on our own infrastructure which would prevent customers from being able to access https://status.nodespace.net should our network be unavailable. We have put into place a permanent fix so that the status website is always available, even if our network is down. Apologies on the oversight! We fixed two issues in one.

Affecting Server - NODE02.NODESPACESERVERS.COM

• 11/24/2021 04:55 - 11/24/2021 03:58
• Last Updated 11/24/2021 03:57

Server NODE02 was relocated within the data center which caused a small outage. Our DCO team moved the server preemptively. Some services may have been interrupted during this move. We apologize about any inconvenience. 

Affecting Server - NODE02.NODESPACESERVERS.COM

• 10/08/2021 23:12 - 10/09/2021 01:29
• Last Updated 10/09/2021 01:29

We're noticing an issue with outbound email. We're currently working on the issue.

Update 1: We are now routing outbound emails through an external SMTP service for the time being. 

Affecting Other - Lenoir, NC Data Center

• 10/08/2021 18:18 - 10/08/2021 22:33
• Last Updated 10/08/2021 22:33

We are currently experiencing an outage at our North Carolina data center. 

Update 1: Most services should be restored to shared hosting & VPS customers.

Update 2: We believe that the issue is resolved. Customers still experiencing issues should open a support ticket for further troubleshooting.

Affecting Other - Lenoir, NC Networking

• 03/21/2021 00:00 - 04/14/2021 17:15
• Last Updated 03/18/2021 10:32

We will perform software upgrades and configuration changes to provider edge and core routers in order to facilitate the availability of new services. During this time, you may experience multiple routing reconvergence events or changes in routing. Every effort will be made to limit or eliminate the impact to clients.

Affecting Other - Network

• 10/21/2020 00:01 - 10/24/2020 23:42
• Last Updated 10/19/2020 15:16

Maintenance Window: October 21, 2020
Start Time: 00:01 EDT
End Time: 05:00 EDT

Services Affected:  Network Services
Service Location(s): 801 Main Street NW, Lenoir, North Carolina

Expected Impact: Internet Service interruption of up to 60 mins during the maintenance window

Details:
Please be advised that our network engineering team will be working with one of our carrier providers to provision a new redundant Internet link to add further capacity, diversity and reliability to our Lenoir DC. This change will unavoidably impact service to our customers with data services in our Lenoir NC data center. Our network engineers will ensure that your services will be restored as quickly as possible, however there will be a loss of services during this maintenance which can last up to 60 minutes due to the nature of work required.

Affecting Other - Network Maintenance

• 09/25/2020 00:01 - 10/05/2020 08:49
• Last Updated 09/23/2020 13:39

Maintenance Window:

Start Time: September 25th 2020 00:01 EDT
End Time: September 25th 2020 06:00 EDT

Services Affected: Internet transit Service

Location(s): Lenoir, NC, USA.

Expected Impact: Re-convergence time of up to 15 mins to redundant paths

Details: One of our carrier providers will be performing extensive work to fix a chronic problem with a long-haul fiber span that has been negatively impacting service to our customers in Lenoir NC. Our network engineers will ensure that the impact to your services will be as minimal as possible, however the reconvergence time will be between 3 minutes to 15 minutes due to the nature of this maintenance.

Affecting Other - Network Maintenance

• 08/25/2020 00:01 - 09/23/2020 13:38
• Last Updated 08/23/2020 17:02

Maintenance Window 1: Tuesday August 25th, 2020
Start Time: 00:01 EDT
End Time:  08:00 EDT

Maintenance Window 2: Friday August 28th, 2020
Start Time: 00:01 EDT
End Time:  08:00 EDT

 
Services Affected: Network Services
Service Location(s): 801 Main Street NW, Lenoir, North Carolina
Expected Impact: No impact

Details: One of our carriers will be performing required emergency fiber maintenance.  This impacted link is part of our diverse paths, and as such, we expect minimal to no impact. If there is impact, it would be limited to a brief network re-convergence.

Affecting Other - Network Maintenance

• 08/18/2020 00:00 - 08/23/2020 17:00
• Last Updated 08/17/2020 15:09

Maintenance Window 1: Tuesday August 18, 2020
Start Time: 00:01 EDT
End Time:  06:00 EDT

Maintenance Window 2: Wednesday August 19, 2020
Start Time: 00:01 EDT
End Time:  06:00 EDT

Maintenance Window 3: Thursday August 20, 2020
Start Time: 00:01 EDT
End Time:  06:00 EDT
 

Services Affected: Network Services
Service Location(s): 801 Main Street NW, Lenoir, North Carolina
Expected Impact: Brief re-convergence up to 15 minutes. 

Details: Our carrier partner, Spectrum, will be performing a standard network maintenance during the above listed windows. Our core network has redundant links that will support the changes necessary to minimize downtime and our Network Engineering team will be monitoring our Core throughout the window. Both pre and post checks will be completed to ensure the successful completion of the window.

Affecting Other - Lenoir, NC Data Center

• 08/07/2020 00:01 - 08/17/2020 15:09
• Last Updated 08/04/2020 17:58

Maintenance Window 1: Friday August 7th, 2020

Start Time: 00:01 EDT

End Time: 08:00 EDT

Maintenance Window 2: Tuesday August 11th, 2020

Start Time: 00:01 EDT

End Time: 08:00 EDT

Maintenance Window 3: Wednesday August 12th, 2020

Start Time: 00:01 EDT

End Time: 08:00 EDT

Maintenance Window 4: Thursday August 13th, 2020

Start Time: 00:01 EDT

End Time: 08:00 EDT

Services Affected: Network Services

Service Location(s): 801 Main Street NW, Lenoir, North Carolina

Expected Impact: Brief re-convergence up to 15 minutes. 

Details: Our carrier partner, Spectrum, will be performing a standard network maintenance during the above listed windows. Our core network has redundant links that will support the changes necessary to minimize downtime and our Network Engineering team will be monitoring our Core throughout the window. Both pre and post checks will be completed to ensure the successful completion of the window.

Affecting Other - Network Maintenance

• 04/11/2020 02:00 - 07/05/2020 22:21
• Last Updated 04/09/2020 08:22

Maintenance Window: April 11th, 2020

Start Time: 02:00 EDT
End Time: 05:00 EDT
Services Affected: Internet Transit
Service Location(s): 801 Main Street North West, Lenoir North Carolina
Expected Impact: Up to 15 minutes of re-convergence time to redundant paths

Details: Cogent will be performing network maintenance. During this maintenance window you may experience one or more brief interruptions in service as our Network completes re-convergence.

Affecting Other - UPS

• 04/06/2020 21:00 - 04/09/2020 08:22
• Last Updated 03/06/2020 12:54

Services Affected: UPS
Service Location(s): North Carolina Data Center
Expected Impact: No expected Impact
Details: We will be performing maintenance activities on our UPS(s) as part of our normal maintenance cycle. The activity will be split into two nights.

April 6th, 21:00 start time, engineers and our partner vendors will be replacing various components in UPS1 at our Lenoir, NC facility as part of our normal maintenance cycle. It will take approximately 8 hours to complete ending Tuesday, April 7th, 2020 by 05:00 EST.

April 7th, 21:00 start time, engineers and our partner vendors will be performing routine Preventative Maintenance on all UPS’s at our Lenoir, NC facility as part of our normal maintenance cycle. It will take approximately 8 hours to complete ending Wednesday, April 8th, 2020 by 05:00 EST.

There is no power loss expected.

Affecting Other - UPS

• 04/07/2020 21:00 - 04/09/2020 08:22
• Last Updated 03/06/2020 12:54

Services Affected: UPS
Service Location(s): North Carolina Data Center
Expected Impact: No expected Impact
Details: We will be performing maintenance activities on our UPS(s) as part of our normal maintenance cycle. The activity will be split into two nights.

April 6th, 21:00 start time, engineers and our partner vendors will be replacing various components in UPS1 at our Lenoir, NC facility as part of our normal maintenance cycle. It will take approximately 8 hours to complete ending Tuesday, April 7th, 2020 by 05:00 EST.

April 7th, 21:00 start time, engineers and our partner vendors will be performing routine Preventative Maintenance on all UPS’s at our Lenoir, NC facility as part of our normal maintenance cycle. It will take approximately 8 hours to complete ending Wednesday, April 8th, 2020 by 05:00 EST.

There is no power loss expected.

Affecting Server - NODE01.MCO.NODESPACE.US

• 09/01/2019 00:00 - 09/01/2019 00:00
• Last Updated 12/16/2019 11:49

We are going to be performing routine maintenance on NODE01.

Affecting Server - NODE01.MCO.NODESPACE.US

• 06/06/2019 23:20 - 06/06/2019 18:13
• Last Updated 06/06/2019 18:13

Recently, a severe bug was discovered in the email software installed on one of our web servers which is hosting your website (on NODE01). In order to resolve this issue, we have patched the software which will only take effect once we reboot the server. We have scheduled emergency maintenance tonight to reboot the server.

Please note: if you have other services with us such as dedicated or virtual servers, these are not affected.

Full details about the bug: https://nvd.nist.gov/vuln/detail/CVE-2019-10149


Thank you for your understanding and we apologize about the inconvenience.

UPDATES:

1. Server has been rebooted.

Affecting Server - NODE01.MCO.NODESPACE.US

• 01/19/2019 00:00 - 01/19/2019 00:05
• Last Updated 01/21/2019 23:50

As part of our commitment to provide the best possible infrastructure for your services, we're sending you this notice to inform you of some upcoming server maintenance in our Orlando Data Center. Please read the following details carefully:

Maintenance type:
Software Upgrade

Procedure:
PHP 5.6 is being officially deprecated as it will no longer receive security updates after January 19th. We will be setting the default PHP version on node01.mco.nodespace.us (the server your website is hosted on) to PHP 7.2.

Latest versions of most web software (such as WordPress, phpBB, MyBB, etc.) already has PHP 7.2 support, however we do recommend testing plugin and addon compatibility. 

If you are running any custom software, please use this time to test for PHP 7.2 compatibility. We've made it easy to switch your site to PHP 7.2 for testing. Please see the following support article on how to enable PHP 7.2 for your site (and how to switch it back): https://my.gonodespace.com/knowledgebase.php?id=174. If everything works as expected, you can reset your site to "inherited". If you encounter errors, we recommend setting the PHP option to version 5.6.

This email is regarding your hosting service with us as we are making a major change in January. As this is a major change, we are sending you this notification as a courtesy so you have time to prepare your website.

NodeSpace does not offer any sort of coding assistance, however our support team is more than happy to assist you with collecting any errors on your site. Please open a ticket through the My NodeSpace portal (https://my.gonodespace.com/submitticket.php) and we'll assist you as much as possible.

Maintenance Window:
Start - 01/19/2019 - 10:00 PM EST
End - 01/19/2019 - 11:59 PM EST

Estimated downtime:
There will be no downtime associated with this maintenance.

Thank you for taking the time to review this maintenance notice. If you do have any questions or concerns about this procedure and how you may be affected, please don't hesitate to contact our support team for more details.

Affecting Other - Orlando2 Data Center

• 11/29/2018 14:00 - 11/30/2018 19:49
• Last Updated 11/30/2018 22:11

Initial Report

As part of our commitment to provide the best infrastructure for your services, we're sending you this notice to inform you of emergency maintenance taking place in our Orlando based data center facility. We recommend reviewing the following details:

Maintenance type:
Electrical Infrastructure

Reason for Maintenance:
Our monitoring systems alerted our data center technicians to potential issues with electrical systems powering a portion of our Orlando data center facility. In order to proactively address the problems our team of engineers will be running tests to locate any potential flaws and replace any hardware found to be faulty.

Maintenance Window:
Current - ONGOING

Datacenter:
Orlando.DC2

Estimated amount of downtime:
We understand the importance in minimizing downtime while performing any type of maintenance and for that reason we will do everything we can to expedite this process. There are no guarantees, and while we have not encountered any actual power issues affecting your services, it is worth noting that due to the nature of the issue(s) at hand we are considering this an At-Risk maintenance and hence are alerting you to the incident. We do not expect to see any type of downtime but should issues arise power to various sectors of our facility may be affected. We will be sending out a follow up to this notice once the maintenance has been concluded.

Thank you so much for taking the time to review this notice and we do apologize for any inconvenience these unforeseen issues may pose to you and your clients. If you have any questions or concerns about this reboot procedure, please don't hesitate to contact our staff for more details.

Update 1 - 9:51 PM

We thank you very much for your extended patience and simply wanted to provide an update to our prior notice regarding emergency maintenance in our Orlando data center resulting in us placing it into maintenance mode.

At this time our engineers continue to monitor the status of our electrical systems and will continue to do so until our facility is out of maintenance mode. We were able to limit down the factors that triggered our monitoring systems and steps are already being taken to resolve those issues. We remain cautiously optimistic that we will finalize the maintenance with no impact to your services and are aiming to have everything completed by tomorrow afternoon.

As promised, a recap of the situation will be made available after the data center is out of maintenance mode. Lastly, in case an emergency should occur, we do urge you to follow our social media accounts, which are listed below, for updates:

Twitter: https://twitter.com/nodespace

Thank you once again for your patience and understanding during this time and we hope you have a great evening.

Resolution - 11/30 7:49 PM

First and foremost thank you very much for your extended patience and understanding while our team worked on the emergency maintenance over the last day and a half.

At this time we are happy to announce that our team has successfully concluded the maintenance with no reports of downtime and/or power issues and everything is functioning in an optimal state.

In order to be transparent, and as promised, we have included details provided by our engineers as to what prompted the at-risk emergency maintenance below.

At around 11 AM EST on November 29th, our monitors alerted our team to issues with one of our UPS systems as it was failing and eventually did. We were successfully able to transfer the load on the UPS to maintenance bypass and onto full commercial power resulting in the *At-Risk* notice. Upon further investigation the cause of the failure was diagnosed and attributed to a failed capacitor in the system which was successfully replaced. Once that was done we were able to run tests on UPS that proved to be successful and the critical load was transferred back with no issue. After further monitoring everything continues to function as it should and we are concluding our emergency maintenance.

We thank you, once again, for your patience and understanding and assure you that our goal is to always do what we can to ensure you get the services and support that you not only expect, but deserve. As always, if you have any further questions and/or concerns please do not hesitate to let us know. Have a great day!

Affecting System - Knowledgebase

• 08/07/2017 04:42 - 05/30/2018 23:10
• Last Updated 05/30/2018 23:10

We have identified an issue with our client portal that is causing some knowledgebase articles to not show up. This issue has been reported to our software vendor for further investigation.

Work Around: In the meantime, please check our Knowledgebase on our support portal site: https://support.gonodespace.com/knowledgebase
Update#1: We have started updating the links to our support site. You can access our support site with your My NodeSpace login.

Resolution: We have resolved the issue and knowledgebase articles are now showing in the customer portal.

Affecting System - Service Desk

• 05/01/2017 00:00 - 05/30/2018 23:09
• Last Updated 05/30/2018 23:10

We have identified an issue with our Service Desk software that isn't properly importing issues into our customer portal properly. Our software vendor is currently investigating this issue and will supply a patch when it is fixed.

Work Around: To view all important customer announcements, view them within our support portal here: https://support.gonodespace.com/announcements

Update#1: We have started updating the links to our support site. You can access our support site with your My NodeSpace login.

Resolution: We have resolved the issue and announcements now show in the customer portal.

Affecting Server - NODE01.MCO.NODESPACE.US

• 08/26/2017 11:24 - 08/26/2017 13:07
• Last Updated 08/26/2017 13:55

Reason for Outage (RFO)

Location:
NODE01.MCO.DC2

Date:
August 26, 2017

Events:
11:24 AM (GMT-4) During routine server maintenance, the NODE01.MCO server encountered an issue which caused Apache (HTTP) to stop responding to web requests. Our Linux Admins' attempts to bring Apache back online were unsuccessful so they issued a reboot of the server under emergency maintenance procedures.
11:34 AM (GMT-4) The server still had not come back up from reboot. Our Linux Administration Team alerted our NOC team to check the physical server out.
12:34 PM (GMT-4) Our NOC team reported that the server was fine hardware side and provided our Linux Team KVM access.
1:07 PM (GMT-4) Our Linux Admin Team was able to restore services.

Root cause of the issue:
Our Linux Administration team noticed that during server boot up, several of the critical services were not starting. Upon investigation, the root cause was SELinux becoming enabled in the "permissive" state instead of "disabled.

Resolution:
Our Linux Administration team booted the affected server into single user mode and disabled SELinux and rebooted the server normally.

Action plan:
Our Linux Administration team will be closely monitoring the server and reviewing any updates that were automatically installed and working with the upstream software vendors to ensure that this does not happen again.

Total downtime:
Customers on NODE01.MCO experienced 1 hour, 43 minutes of downtime. This downtime is able to be claimed for an SLA credit.

We sincerely apologize about the inconvenience and will do our best to reduce/eliminate future occurrences of the same problems.

Affecting Server - NODE01.MCO.NODESPACE.US

• 08/02/2017 23:00 - 08/03/2017 01:00
• Last Updated 08/06/2017 19:24

Maintenance Type:
Network Infrastructure Maintenance

Action to be taken:
In an effort to maximize the reliability of our services, we will be replacing a line card in one of our core routers at our Orlando.DC2 datacenter.

Maintenance Window:
Start - Wednesday, August 2nd @ 23:00 (11:00 PM EDT)
End - Thursday, August 3rd @ 01:00 (1:00 AM EDT)

Datacenter Location:
Orlando.DC2

Estimated Downtime:
We understand the importance of minimizing the downtime associated with any maintenance. We will take every reasonable precaution to ensure this process is completed safely and without downtime. Our fiber is redundant in our datacenter, so we do not expect any downtime will result from this maintenance. We are notifying you as a precaution. As traffic is re-routed throughout the network, you may experience up to 45 minutes of increased latency, minor packet loss, or possible service interruption during the two-hour maintenance window.

We do apologize for any inconvenience and ask that you please plan accordingly.

Affecting Server - NODE01.MCO.NODESPACE.US

• 07/23/2017 23:00 - 08/06/2017 19:22
• Last Updated 07/23/2017 02:49

Maintenance Type:
Network Maintenance

Action to be taken:
To ensure the reliability of our services, we will be replacing a line card in one of our core routers at our Orlando.DC2 datacenter.

Maintenance Window:
Start - Sunday, July 23rd @ 23:00 Local time (11:00 PM EDT)
End - Monday, July 24th @ 01:00 Local time (1:00 AM EDT)

Datacenter Location:
Orlando.DC2

Estimated Downtime:
We understand the importance of minimizing the downtime associated with any maintenance. Our team will take every reasonable precaution to ensure this process is completed safely and without downtime. Our fiber is redundant in our datacenter, so we do not expect any downtime will result from this maintenance; however, we are notifying you as a precaution. As traffic is re-routed throughout the network, you may experience up to 45 minutes of increased latency, minor packet loss, or possible service interruption during the two-hour maintenance window.

We do apologize for any inconvenience and ask that you please plan accordingly.

Thank you so much for taking the time to review this notice. If you have questions or concerns about this procedure, please don't hesitate to contact us.

Affecting Other - Service Desk

• 07/11/2017 04:41 - 07/12/2017 00:42
• Last Updated 07/12/2017 04:46

Our service desk was temporarily unavailable from the My NodeSpace customer portal. Our technical team found the issue and resolved it promptly. We have manually opened any opened tickets that were misdirected. Any tickets that were opened via Facebook, email, or Twitter were all opned correctly and you should now see these tickets opened within your customer portal. If you have any questions, please don't hesitate to reach out.

Affecting Other - Network

• 07/03/2017 00:00 - 07/03/2017 00:00
• Last Updated 07/05/2017 04:18

We had a brief outage lasting approximately 1 minute during the early morning on July 3, 2017. Our data center identified an issue with one of the core routers providing network connectivity to the data center. The issue has been mitigated and will be resolved during network maintenance.

If you have any questions, please do not hesitate to ask.

Thank you for your patience and understanding!

Affecting Server - NODE01.MCO.NODESPACE.US

• 12/07/2015 20:00 - 12/10/2015 01:55
• Last Updated 12/01/2015 01:39

Scheduled Date and Time: 
Start - December 7th, 2015 @ 8:00PM EST 
End - December 8th, 2015 @ 12:00AM EST 

Maintenance Window: 
4 Hours 

Datacenter Location: 
Orlando.DC2 

Maintenance Type: 
In order to ensure the most secure and reliable services possible, we will be updating firmware on the switch your server is connected to. To complete this process, we will need to power cycle the switch. This will cause a brief period of service unavailability for all servers using this switch. 

Estimated Downtime: 
We understand the importance of minimizing downtime while performing maintenance and for this reason we will do everything we can in hopes that we can expedite the process. As it stands we expect no more than 20 minutes of downtime during the maintenance window as the switch is updated and power cycled. Should there be any complications, we will have personel on-site to address these issues immediately. 

We do apologize for any inconvenience and ask that you please plan accordingly. 

If you have any questions or concerns, please feel free to submit a support ticket.