Scheduled Exchange Server Maintenance - May 3, 2014

Details

Change Title: Exchange 2010 SP2 upgrade/storage replacement
Change Type: Upgrade
Risk: Low
Service Affecting? Yes
Maintenance Date: Saturday, May 3, 2014
Maintenance Window: 8:00 pm - 3:00 am PDT
Expected Downtime (Min.): 420
Services Impacted: Microsoft Exchange
Expected Impact: We will be upgrading Exchange server 2010 to Service Pack 2 and moving mail databases to new storage. All other cloud resources will be available during this maintenance period except Exchange email. VirtualQube Cloud customers whose plans include either the Reflexion “RADAR” or Mimecast service will still be able to send/receive email messages via the respective Reflexion or Mimecast Web portal during the service outage. End user instructions for RADAR and for Mimecast are available for download from our Web site.

Please address any questions to our help desk by sending an email to tickets@VirtualQube.com.

Emergency Maintenance Notification 04-14-2014

*CLOSED*
This maintenance activity was completed successfully.

Tonight (April 14) we will be updating the firmware on our boundary firewalls to ensure that we are fully protected against the HeartBleed vulnerability. This maintenance effort will begin at 11 pm Pacific time this evening, and is not expected to take more than an hour. A reboot of the boundary firewalls will be required after the firmware is updated. Therefore, customers’ cloud sessions will appear to hang if they are logged in at this time. A reminder message will be sent to any users currently logged in 15 minutes prior to the beginning of maintenance. Details follow:

Change Title: Apply firewall firmware update to protect against the HeartBleed vulnerability.
Change Type: Change
Maintenance Date: Monday, April 14, 2014
Maintenance Window: 11:00 pm - 12:00 am PDT
Expected Downtime (min.): 30
Services Impacted: All VirtualQube cloud resources
Expected Impact: We will be upgrading the firmware on our firewalls. A reboot of the firewall will be required after the update is applied; therefore, if you are logged in and working at this time, your cloud session will appear to hang. A reminder message will be sent 15 minutes prior to the beginning of maintenance to any users logged in at that time.

HA Event in Data Center

Status: Closed
Severity: Minor impact, Not all customers were affected
Incident Start: 09/16/2013 7:55 PM
Incident Resolved: 09/16/2013 8:05 PM
Description: We experienced an HA event. An HA event is when a workload fails over to a redundant system.  The affected VM’s where automatically restarted on a standby cluster.
Resolution: Systems have been repaired.
Root Cause: Other
What/Who was affected: Some VirtualQube Customers
Actions: Performed permanent repairs

Loss of access to VirtualQube data centers

Status: Closed; Resolved
Severity: Full outage
Incident Start: 08/30/2013 2:55 PM
Start/End Time: 08/30/2013 2:55 PM - 3:58 PM
Incident Resolved: 08/30/2013 3:58 PM
Description: Our data center reports that there was a problem on a NetApp storage cluster that affected access to data and virtual machine storage. The issue was described as a significant degradation on the primary NetApp storage cluster.
Resolution: Our data center initiated a fail-over event to move VirtualQube resources to a stand-by NetApp storage system.
Root Cause: Hardware Failure
What/Who was affected: Virtual Desktop Access; Email Service; Data Access; All VirtualQube Customers
Actions: VirtualQube is currently designing new resources at our existing data center and is also expanding our data center operations to an additional data center with the intention of quickly improving our service delivery to eliminate these types of outages. We currently expect a number of new improvements to be implemented over the course of the next few days and continuing through the next few months.

Applications slow or not available in VirtualQube desktops

Status: Closed; Resolved
Severity: Significant degradation
Incident Start: 08/23/2013 8:00 AM
Start/End Time: 08/23/2013 8:00 AM - 11:30 AM
Incident Resolved: 08/23/2013 11:30 AM
Description: Our data center reported that a NetApp storage cluster was operating at peak utilization resulting in a slowing of application performance.
Resolution: Our data center provider took action to balance the workloads by adding storage system resources.
Root Cause: Datacenter Issue
What/Who was affected: Virtual Desktop Access; Some VirtualQube Customers
Actions: Our data center has added additional storage resources to the NetApp system and is continuing a rolling process to upgrade the storage system hardware and software.

Connectivity to VirtualQube Data Center problem

Print Close

Status: Closed; Resolved
Severity: Significant degradation
Incident Start: 08/23/2013 8:00 AM
Start/End Time: 08/23/2013 8:00 AM - 11:30 PM
Incident Resolved: 08/23/2013 11:30 AM
Description: We are currently investigating reports of connectivity problems to our data centers. Several customers and our own staff are reporting intermittent problems connecting to VirtualQube cloud resources. Some users are receiving “page not found” when trying to access the VirtualQube portal and others are reporting slow performance when using VirtualQube cloud applications. We have been tracking this problem since approximately 8am and have found that this is not just affecting our services. During our investigation we have found problems connecting to numerous websites and cloud resources. Some of the websites that have been unavailable this morning are www.whois.com and www.citrix.com.

At this time the issue appears to be outside of our data center, this is evidenced by loss of connectivity between various VirtualQube staff and sites outside of our data center. The issue seems to be related to recent problem with DNS Root servers on the internet.

Resolution: This issue was intermittent during the outage with pings to our data center and to other public website failing and then suddenly working again. When pings were failing it also appeared that internet DNS servers where not returning IP addresses.
What/Who was affected: Virtual Desktop Access; Email Service; Data Access; Some VirtualQube Customers
Actions: We will continue to monitor internet sources that report these issues.

Loss of access to VirtualQube Data Centers

Status: Closed; Resolved
Severity: Significant degredation
Incident Start: 03/05/2013 9:32 AM
Incident Resolved: 03/05/2013 9:42 AM
Description: We experienced a condition referred to as a Denial of Service attack. These attacks are relatively common on the internet and are typically intended to block services by sending large volumes of traffic to a destination with the malicious intent of blocking service.
Resolution: Our data center team took immediate action to block this attack and was able to resolve the situation within ten minutes.
Root Cause: Capacity/Traffic Issue
What/Who was affected: Virtual Desktop Access; Email Service; Data Access; All VirtualQube Customers
Actions: Implemented new policies on our network equipment to permanently block this activity.