Multiple routing reconvergence events (Resolved)
  • Priority - Medium
  • Affecting Other - Lenoir, NC Networking
  • We will perform software upgrades and configuration changes to provider edge and core routers in order to facilitate the availability of new services. During this time, you may experience multiple routing reconvergence events or changes in routing. Every effort will be made to limit or eliminate the impact to clients.

  • Date - 2021-03-21 00:00 - 2021-04-14 17:15
  • Last Updated - 2021-03-18 10:32
North Carolina Data - Center Carrier Maintenance (Resolved)
  • Priority - Low
  • Affecting Other - Network
  • Maintenance Window: October 21, 2020
    Start Time: 00:01 EDT
    End Time: 05:00 EDT

    Services Affected:  Network Services
    Service Location(s): 801 Main Street NW, Lenoir, North Carolina

    Expected Impact: Internet Service interruption of up to 60 mins during the maintenance window

    Details:
    Please be advised that our network engineering team will be working with one of our carrier providers to provision a new redundant Internet link to add further capacity, diversity and reliability to our Lenoir DC. This change will unavoidably impact service to our customers with data services in our Lenoir NC data center. Our network engineers will ensure that your services will be restored as quickly as possible, however there will be a loss of services during this maintenance which can last up to 60 minutes due to the nature of work required.

  • Date - 2020-10-21 00:01 - 2020-10-24 23:42
  • Last Updated - 2020-10-19 15:16
North Carolina Data - Center Carrier Maintenance (Resolved)
  • Priority - Low
  • Affecting Other - Network Maintenance
  • Maintenance Window:

    Start Time: September 25th 2020 00:01 EDT
    End Time: September 25th 2020 06:00 EDT

    Services Affected: Internet transit Service

    Location(s): Lenoir, NC, USA.

    Expected Impact: Re-convergence time of up to 15 mins to redundant paths

    Details: One of our carrier providers will be performing extensive work to fix a chronic problem with a long-haul fiber span that has been negatively impacting service to our customers in Lenoir NC. Our network engineers will ensure that the impact to your services will be as minimal as possible, however the reconvergence time will be between 3 minutes to 15 minutes due to the nature of this maintenance.

  • Date - 2020-09-25 00:01 - 2020-10-05 08:49
  • Last Updated - 2020-09-23 13:39
North Carolina Data Center - Emergency Carrier Maintenance (Resolved)
  • Priority - High
  • Affecting Other - Network Maintenance
  • Maintenance Window 1: Tuesday August 25th, 2020
    Start Time: 00:01 EDT
    End Time:  08:00 EDT

    Maintenance Window 2: Friday August 28th, 2020
    Start Time: 00:01 EDT
    End Time:  08:00 EDT

     
    Services Affected: Network Services
    Service Location(s): 801 Main Street NW, Lenoir, North Carolina
    Expected Impact: No impact

    Details: One of our carriers will be performing required emergency fiber maintenance.  This impacted link is part of our diverse paths, and as such, we expect minimal to no impact. If there is impact, it would be limited to a brief network re-convergence.

  • Date - 2020-08-25 00:01 - 2020-09-23 13:38
  • Last Updated - 2020-08-23 17:02
North Carolina Data - Center Carrier Maintenance (Resolved)
  • Priority - Medium
  • Affecting Other - Network Maintenance
  • Maintenance Window 1: Tuesday August 18, 2020
    Start Time: 00:01 EDT
    End Time:  06:00 EDT

    Maintenance Window 2: Wednesday August 19, 2020
    Start Time: 00:01 EDT
    End Time:  06:00 EDT

    Maintenance Window 3: Thursday August 20, 2020
    Start Time: 00:01 EDT
    End Time:  06:00 EDT
     

    Services Affected: Network Services
    Service Location(s): 801 Main Street NW, Lenoir, North Carolina
    Expected Impact: Brief re-convergence up to 15 minutes. 

    Details: Our carrier partner, Spectrum, will be performing a standard network maintenance during the above listed windows. Our core network has redundant links that will support the changes necessary to minimize downtime and our Network Engineering team will be monitoring our Core throughout the window. Both pre and post checks will be completed to ensure the successful completion of the window.

  • Date - 2020-08-18 00:00 - 2020-08-23 17:00
  • Last Updated - 2020-08-17 15:09
North Carolina Data Center Carrier Maintenance (Resolved)
  • Priority - High
  • Affecting Other - Lenoir, NC Data Center
  • Maintenance Window 1: Friday August 7th, 2020

    Start Time: 00:01 EDT

    End Time: 08:00 EDT

     

    Maintenance Window 2: Tuesday August 11th, 2020

    Start Time: 00:01 EDT

    End Time: 08:00 EDT

     

    Maintenance Window 3: Wednesday August 12th, 2020

    Start Time: 00:01 EDT

    End Time: 08:00 EDT

     

    Maintenance Window 4: Thursday August 13th, 2020

    Start Time: 00:01 EDT

    End Time: 08:00 EDT

     

     

    Services Affected: Network Services

    Service Location(s): 801 Main Street NW, Lenoir, North Carolina

    Expected Impact: Brief re-convergence up to 15 minutes. 

     

    Details: Our carrier partner, Spectrum, will be performing a standard network maintenance during the above listed windows. Our core network has redundant links that will support the changes necessary to minimize downtime and our Network Engineering team will be monitoring our Core throughout the window. Both pre and post checks will be completed to ensure the successful completion of the window.

  • Date - 2020-08-07 00:01 - 2020-08-17 15:09
  • Last Updated - 2020-08-04 17:58
North Carolina Data Center Maintenance (Resolved)
  • Priority - Medium
  • Affecting Other - Network Maintenance
  • Maintenance Window: April 11th, 2020

    Start Time: 02:00 EDT
    End Time: 05:00 EDT
    Services Affected: Internet Transit
    Service Location(s): 801 Main Street North West, Lenoir North Carolina
    Expected Impact: Up to 15 minutes of re-convergence time to redundant paths

    Details: Cogent will be performing network maintenance. During this maintenance window you may experience one or more brief interruptions in service as our Network completes re-convergence.

  • Date - 2020-04-11 02:00 - 2020-07-05 22:21
  • Last Updated - 2020-04-09 08:22
North Carolina Data Center UPS Maintenance (Night 1) (Resolved)
  • Priority - Low
  • Affecting Other - UPS
  • Services Affected: UPS
    Service Location(s): North Carolina Data Center
    Expected Impact: No expected Impact
    Details: We will be performing maintenance activities on our UPS(s) as part of our normal maintenance cycle. The activity will be split into two nights.

    April 6th, 21:00 start time, engineers and our partner vendors will be replacing various components in UPS1 at our Lenoir, NC facility as part of our normal maintenance cycle. It will take approximately 8 hours to complete ending Tuesday, April 7th, 2020 by 05:00 EST.

    April 7th, 21:00 start time, engineers and our partner vendors will be performing routine Preventative Maintenance on all UPS’s at our Lenoir, NC facility as part of our normal maintenance cycle. It will take approximately 8 hours to complete ending Wednesday, April 8th, 2020 by 05:00 EST.

    There is no power loss expected.

  • Date - 2020-04-06 21:00 - 2020-04-09 08:22
  • Last Updated - 2020-03-06 12:54
North Carolina Data Center UPS Maintenance (Night 2) (Resolved)
  • Priority - Low
  • Affecting Other - UPS
  • Services Affected: UPS
    Service Location(s): North Carolina Data Center
    Expected Impact: No expected Impact
    Details: We will be performing maintenance activities on our UPS(s) as part of our normal maintenance cycle. The activity will be split into two nights.

    April 6th, 21:00 start time, engineers and our partner vendors will be replacing various components in UPS1 at our Lenoir, NC facility as part of our normal maintenance cycle. It will take approximately 8 hours to complete ending Tuesday, April 7th, 2020 by 05:00 EST.

    April 7th, 21:00 start time, engineers and our partner vendors will be performing routine Preventative Maintenance on all UPS’s at our Lenoir, NC facility as part of our normal maintenance cycle. It will take approximately 8 hours to complete ending Wednesday, April 8th, 2020 by 05:00 EST.

    There is no power loss expected.

  • Date - 2020-04-07 21:00 - 2020-04-09 08:22
  • Last Updated - 2020-03-06 12:54
NODE01 Scheduled Maintenance (Resolved)
  • Priority - Low
  • Affecting Server - NODE01.MCO.NODESPACE.US
  • We are going to be performing routine maintenance on NODE01.

  • Date - 2019-09-01 00:00 - 2019-09-01 00:00
  • Last Updated - 2019-12-16 11:49
Emergency Server Reboot: Node01 (Resolved)
  • Priority - Critical
  • Affecting Server - NODE01.MCO.NODESPACE.US
  • Recently, a severe bug was discovered in the email software installed on one of our web servers which is hosting your website (on NODE01). In order to resolve this issue, we have patched the software which will only take effect once we reboot the server. We have scheduled emergency maintenance tonight to reboot the server.

    Please note: if you have other services with us such as dedicated or virtual servers, these are not affected.

    Full details about the bug: https://nvd.nist.gov/vuln/detail/CVE-2019-10149


    Thank you for your understanding and we apologize about the inconvenience.

    UPDATES:

    1. Server has been rebooted.

  • Date - 2019-06-06 23:20 - 2019-06-06 18:13
  • Last Updated - 2019-06-06 18:13
PHP 7.2 Default (Resolved)
  • Priority - Medium
  • Affecting Server - NODE01.MCO.NODESPACE.US
  • As part of our commitment to provide the best possible infrastructure for your services, we're sending you this notice to inform you of some upcoming server maintenance in our Orlando Data Center. Please read the following details carefully:

    Maintenance type:
    Software Upgrade

    Procedure:
    PHP 5.6 is being officially deprecated as it will no longer receive security updates after January 19th. We will be setting the default PHP version on node01.mco.nodespace.us (the server your website is hosted on) to PHP 7.2.

    Latest versions of most web software (such as WordPress, phpBB, MyBB, etc.) already has PHP 7.2 support, however we do recommend testing plugin and addon compatibility. 

    If you are running any custom software, please use this time to test for PHP 7.2 compatibility. We've made it easy to switch your site to PHP 7.2 for testing. Please see the following support article on how to enable PHP 7.2 for your site (and how to switch it back): https://my.gonodespace.com/knowledgebase.php?id=174. If everything works as expected, you can reset your site to "inherited". If you encounter errors, we recommend setting the PHP option to version 5.6.

    This email is regarding your hosting service with us as we are making a major change in January. As this is a major change, we are sending you this notification as a courtesy so you have time to prepare your website.

    NodeSpace does not offer any sort of coding assistance, however our support team is more than happy to assist you with collecting any errors on your site. Please open a ticket through the My NodeSpace portal (https://my.gonodespace.com/submitticket.php) and we'll assist you as much as possible.

    Maintenance Window:
    Start - 01/19/2019 - 10:00 PM EST
    End - 01/19/2019 - 11:59 PM EST

    Estimated downtime:
    There will be no downtime associated with this maintenance.

    Thank you for taking the time to review this maintenance notice. If you do have any questions or concerns about this procedure and how you may be affected, please don't hesitate to contact our support team for more details.

  • Date - 2019-01-19 00:00 - 2019-01-19 00:05
  • Last Updated - 2019-01-21 23:50
Emergency Data Center Maintenance (Resolved)
  • Priority - High
  • Affecting Other - Orlando2 Data Center
  • Initial Report

    As part of our commitment to provide the best infrastructure for your services, we're sending you this notice to inform you of emergency maintenance taking place in our Orlando based data center facility. We recommend reviewing the following details:

    Maintenance type:
    Electrical Infrastructure

    Reason for Maintenance:
    Our monitoring systems alerted our data center technicians to potential issues with electrical systems powering a portion of our Orlando data center facility. In order to proactively address the problems our team of engineers will be running tests to locate any potential flaws and replace any hardware found to be faulty.

    Maintenance Window:
    Current - ONGOING

    Datacenter:
    Orlando.DC2

    Estimated amount of downtime:
    We understand the importance in minimizing downtime while performing any type of maintenance and for that reason we will do everything we can to expedite this process. There are no guarantees, and while we have not encountered any actual power issues affecting your services, it is worth noting that due to the nature of the issue(s) at hand we are considering this an At-Risk maintenance and hence are alerting you to the incident. We do not expect to see any type of downtime but should issues arise power to various sectors of our facility may be affected. We will be sending out a follow up to this notice once the maintenance has been concluded.

    Thank you so much for taking the time to review this notice and we do apologize for any inconvenience these unforeseen issues may pose to you and your clients. If you have any questions or concerns about this reboot procedure, please don't hesitate to contact our staff for more details.

    Update 1 - 9:51 PM

    We thank you very much for your extended patience and simply wanted to provide an update to our prior notice regarding emergency maintenance in our Orlando data center resulting in us placing it into maintenance mode.

    At this time our engineers continue to monitor the status of our electrical systems and will continue to do so until our facility is out of maintenance mode. We were able to limit down the factors that triggered our monitoring systems and steps are already being taken to resolve those issues. We remain cautiously optimistic that we will finalize the maintenance with no impact to your services and are aiming to have everything completed by tomorrow afternoon.

    As promised, a recap of the situation will be made available after the data center is out of maintenance mode. Lastly, in case an emergency should occur, we do urge you to follow our social media accounts, which are listed below, for updates:

    Twitter: https://twitter.com/nodespace

    Thank you once again for your patience and understanding during this time and we hope you have a great evening.

    Resolution - 11/30 7:49 PM

    First and foremost thank you very much for your extended patience and understanding while our team worked on the emergency maintenance over the last day and a half.

    At this time we are happy to announce that our team has successfully concluded the maintenance with no reports of downtime and/or power issues and everything is functioning in an optimal state.

    In order to be transparent, and as promised, we have included details provided by our engineers as to what prompted the at-risk emergency maintenance below.

    At around 11 AM EST on November 29th, our monitors alerted our team to issues with one of our UPS systems as it was failing and eventually did. We were successfully able to transfer the load on the UPS to maintenance bypass and onto full commercial power resulting in the *At-Risk* notice. Upon further investigation the cause of the failure was diagnosed and attributed to a failed capacitor in the system which was successfully replaced. Once that was done we were able to run tests on UPS that proved to be successful and the critical load was transferred back with no issue. After further monitoring everything continues to function as it should and we are concluding our emergency maintenance.

    We thank you, once again, for your patience and understanding and assure you that our goal is to always do what we can to ensure you get the services and support that you not only expect, but deserve. As always, if you have any further questions and/or concerns please do not hesitate to let us know. Have a great day!

  • Date - 2018-11-29 14:00 - 2018-11-30 19:49
  • Last Updated - 2018-11-30 22:11
Knowledgebase not showing all articles (Resolved)
  • Priority - Low
  • Affecting System - Knowledgebase
  • We have identified an issue with our client portal that is causing some knowledgebase articles to not show up. This issue has been reported to our software vendor for further investigation.

    Work Around: In the meantime, please check our Knowledgebase on our support portal site: https://support.gonodespace.com/knowledgebase
    Update#1: We have started updating the links to our support site. You can access our support site with your My NodeSpace login.

    Resolution: We have resolved the issue and knowledgebase articles are now showing in the customer portal.

  • Date - 2017-08-07 04:42 - 2018-05-30 23:10
  • Last Updated - 2018-05-30 23:10
Announcements not showing in Customer Portal (Resolved)
  • Priority - Low
  • Affecting System - Service Desk
  • We have identified an issue with our Service Desk software that isn't properly importing issues into our customer portal properly. Our software vendor is currently investigating this issue and will supply a patch when it is fixed.

    Work Around: To view all important customer announcements, view them within our support portal here: https://support.gonodespace.com/announcements

    Update#1: We have started updating the links to our support site. You can access our support site with your My NodeSpace login.

    Resolution: We have resolved the issue and announcements now show in the customer portal.

  • Date - 2017-05-01 00:00 - 2018-05-30 23:09
  • Last Updated - 2018-05-30 23:10
[RFO] NODE01.MCO down (Resolved)
  • Priority - Critical
  • Affecting Server - NODE01.MCO.NODESPACE.US
  • Reason for Outage (RFO)

    Location:
    NODE01.MCO.DC2

    Date:
    August 26, 2017

    Events:
    11:24 AM (GMT-4) During routine server maintenance, the NODE01.MCO server encountered an issue which caused Apache (HTTP) to stop responding to web requests. Our Linux Admins' attempts to bring Apache back online were unsuccessful so they issued a reboot of the server under emergency maintenance procedures.
    11:34 AM (GMT-4) The server still had not come back up from reboot. Our Linux Administration Team alerted our NOC team to check the physical server out.
    12:34 PM (GMT-4) Our NOC team reported that the server was fine hardware side and provided our Linux Team KVM access.
    1:07 PM (GMT-4) Our Linux Admin Team was able to restore services.

    Root cause of the issue:
    Our Linux Administration team noticed that during server boot up, several of the critical services were not starting. Upon investigation, the root cause was SELinux becoming enabled in the "permissive" state instead of "disabled.

    Resolution:
    Our Linux Administration team booted the affected server into single user mode and disabled SELinux and rebooted the server normally.

    Action plan:
    Our Linux Administration team will be closely monitoring the server and reviewing any updates that were automatically installed and working with the upstream software vendors to ensure that this does not happen again.

    Total downtime:
    Customers on NODE01.MCO experienced 1 hour, 43 minutes of downtime. This downtime is able to be claimed for an SLA credit.

    We sincerely apologize about the inconvenience and will do our best to reduce/eliminate future occurrences of the same problems.

  • Date - 2017-08-26 11:24 - 2017-08-26 13:07
  • Last Updated - 2017-08-26 13:55
Network Maintenance (Resolved)
  • Priority - High
  • Affecting Server - NODE01.MCO.NODESPACE.US
  • Maintenance Type:
    Network Infrastructure Maintenance

    Action to be taken:
    In an effort to maximize the reliability of our services, we will be replacing a line card in one of our core routers at our Orlando.DC2 datacenter.

    Maintenance Window:
    Start - Wednesday, August 2nd @ 23:00 (11:00 PM EDT)
    End - Thursday, August 3rd @ 01:00 (1:00 AM EDT)

    Datacenter Location:
    Orlando.DC2

    Estimated Downtime:
    We understand the importance of minimizing the downtime associated with any maintenance. We will take every reasonable precaution to ensure this process is completed safely and without downtime. Our fiber is redundant in our datacenter, so we do not expect any downtime will result from this maintenance. We are notifying you as a precaution. As traffic is re-routed throughout the network, you may experience up to 45 minutes of increased latency, minor packet loss, or possible service interruption during the two-hour maintenance window.

    We do apologize for any inconvenience and ask that you please plan accordingly.

  • Date - 2017-08-02 23:00 - 2017-08-03 01:00
  • Last Updated - 2017-08-06 19:24
Network Maintenance (Resolved)
  • Priority - High
  • Affecting Server - NODE01.MCO.NODESPACE.US
  • Maintenance Type:
    Network Maintenance

    Action to be taken:
    To ensure the reliability of our services, we will be replacing a line card in one of our core routers at our Orlando.DC2 datacenter.

    Maintenance Window:
    Start - Sunday, July 23rd @ 23:00 Local time (11:00 PM EDT)
    End - Monday, July 24th @ 01:00 Local time (1:00 AM EDT)

    Datacenter Location:
    Orlando.DC2

    Estimated Downtime:
    We understand the importance of minimizing the downtime associated with any maintenance. Our team will take every reasonable precaution to ensure this process is completed safely and without downtime. Our fiber is redundant in our datacenter, so we do not expect any downtime will result from this maintenance; however, we are notifying you as a precaution. As traffic is re-routed throughout the network, you may experience up to 45 minutes of increased latency, minor packet loss, or possible service interruption during the two-hour maintenance window.

    We do apologize for any inconvenience and ask that you please plan accordingly.

    Thank you so much for taking the time to review this notice. If you have questions or concerns about this procedure, please don't hesitate to contact us.

  • Date - 2017-07-23 23:00 - 2017-08-06 19:22
  • Last Updated - 2017-07-23 02:49
Brief Service Desk Outage from Customer Porta (Resolved)
  • Priority - Low
  • Affecting Other - Service Desk
  • Our service desk was temporarily unavailable from the My NodeSpace customer portal. Our technical team found the issue and resolved it promptly. We have manually opened any opened tickets that were misdirected. Any tickets that were opened via Facebook, email, or Twitter were all opned correctly and you should now see these tickets opened within your customer portal. If you have any questions, please don't hesitate to reach out.

  • Date - 2017-07-11 04:41 - 2017-07-12 00:42
  • Last Updated - 2017-07-12 04:46
Brief outage July 3 (Resolved)
  • Priority - Critical
  • Affecting Other - Network
  • We had a brief outage lasting approximately 1 minute during the early morning on July 3, 2017. Our data center identified an issue with one of the core routers providing network connectivity to the data center. The issue has been mitigated and will be resolved during network maintenance.

    If you have any questions, please do not hesitate to ask.

    Thank you for your patience and understanding!

  • Date - 2017-07-03 00:00 - 2017-07-03 00:00
  • Last Updated - 2017-07-05 04:18
Switch maintenance (Resolved)
  • Priority - Medium
  • Affecting Server - NODE01.MCO.NODESPACE.US
  • Scheduled Date and Time: 
    Start - December 7th, 2015 @ 8:00PM EST 
    End - December 8th, 2015 @ 12:00AM EST 

    Maintenance Window: 
    4 Hours 

    Datacenter Location: 
    Orlando.DC2 

    Maintenance Type: 
    In order to ensure the most secure and reliable services possible, we will be updating firmware on the switch your server is connected to. To complete this process, we will need to power cycle the switch. This will cause a brief period of service unavailability for all servers using this switch. 

    Estimated Downtime: 
    We understand the importance of minimizing downtime while performing maintenance and for this reason we will do everything we can in hopes that we can expedite the process. As it stands we expect no more than 20 minutes of downtime during the maintenance window as the switch is updated and power cycled. Should there be any complications, we will have personel on-site to address these issues immediately. 

    We do apologize for any inconvenience and ask that you please plan accordingly. 

    If you have any questions or concerns, please feel free to submit a support ticket. 

  • Date - 2015-12-07 20:00 - 2015-12-10 01:55
  • Last Updated - 2015-12-01 01:39