******SERVICE IMPACTING UPS SYSTEM MAINTENANCE******

On Thursday, March 19th starting at 07:00 hrs PST and ending at 18:30 hrs PST, we will perform maintenance on our UPS systems. This maintenance will include replacing the batteries in UPS 1 and upgrading the capacitors in UPS 3.

******Impacted Services******

Our FileMaker 3, 4, 5, and 6, Lasso 3, 4, 5, and 6, and legacy Mac shared web hosting environments will be taken offline for 30 minutes in the morning at approximately 08:00 hrs PST and again for 30 minutes at approximately 18:00 hrs PST.

******************
For the duration of this maintenance our datacenter electrical load will be transferred to generator power.

This combined maintenance window has been selected based on factors to provide the greatest degree of protection for our clients. These factors include the availability of the most senior technicians from our UPS vendor and the ability to mobilize parts and additional service personnel in the very unlikely event of a component malfunction or failure during the maintenance. Additionally, by servicing both UPS systems during the same maintenance window, we eliminate an additional operation of our maintenance bypass switch thus limiting exposure to a potential voltage drop to datacenter critical equipment.

The schedule for this maintenance is as follows:

07:00 Datacenter Temperature Reduction: We will force the temperature of the datacenter toward the lower portion of the ASHRAE allowable envelope prior to transferring power to the back-up generator. When the transfer takes place our HVAC systems will power-cycle causing a small (2 to 4 degree Fahrenheit) thermal inclination in the datacenter. By lowering the overall space temperature we can be assured that equipment temperatures will not be adversely impacted by the 10-minute restart period of the HVAC systems.

07:00 to 07:30 Generator pre-flight evaluation: Check fluids, connections, air inlets/exhaust, fuel supply, fuel lines, filters and separators. Log results and announce startup to internal staff.

07:30 Generator start-up and warm-up: Start the generator and monitor performance stats, check for fluid leaks, supply artificial load and evaluate voltage, amperage, frequency and engine stats against established baselines.

07:45 FileMaker 3, 4, 5, and 6, Lasso 3, 4, 5, and 6, and legacy Mac shared web hosting environment shutdown: We will begin shutting down the shared hosting environment at this time. The shutdown will be complete by 08:00 hrs and server environment restarts will begin shortly after 08:00 hrs.

08:00 Transfer from grid power to generator power and wrap power around UPS: Manually operate the ATS and UPS maintenance bypass to force the datacenter electrical load to the generator. At this time our HVAC system will power cycle as described above.

08:00 to 12:00 Replace batteries in UPS 1: The UPS system must be completely powered off for life safety while the batteries are removed and replaced. During this operation UPS 3 will remain online and provide UPS power to clients with A+B power.

12:00 to 12:30 Power on UPS system and perform artificial load testing: Following the removal and replacement of components, the UPS systems will be powered on and connected to an artificial load (not datacenter equipment, servers or other infrastructure) for testing. With this equipment we will test the transfer process and mechanism as well as simulate a load of 80% of the UPS system capacity.

13:00 Transfer datacenter load back to UPS 1: Once the process and mechanism has been validated we will transfer the datacenter load back to UPS 1 though power will continue to be supplied to the UPS from the back-up generator.

13:30 to 17:30 Replace capacitors in UPS 3: The UPS system must be completely powered off for life safety while the capacitors are removed and replaced. During this operation UPS 1 will remain online and provide UPS power to clients with A+B power

17:30 to 18:00 Power on UPS 3 and transfer datacenter load: Once the capacitors have been replaced and charged we will transfer the datacenter load back onto UPS 3.

17:45 FileMaker 3, 4, 5, and 6, Lasso 3, 4, 5, and 6, and legacy Mac shared web hosting environment shutdown: We will begin shutting down the shared hosting environment at this time. The shutdown will be complete by 18:00 hrs and server environment restarts will begin shortly after 18:00 hrs

18:00 Transfer Datacenter live load back to grid power: Once all testing has been completed and both UPS systems are online and functioning within specifications, we will transfer the datacenter live load back to grid power.

18:00 to 18:30 Generator Cool-Down and Post-Flight Evaluation: Following the operation of our generator we will perform the same evaluations we performed during the pre-flight as well as allow the generator to cool prior to shutting down.

One to two weeks following this maintenance, we will perform a battery preventive maintenance on UPS system 1 to validate the condition of the new batteries and replace any questionable jars. This preventive maintenance will follow the same procedure as above but require significantly less time to complete. A notice will be posted to our support blog prior to this future maintenance.

digital.forest remains committed to providing our customers with the highest level of service, the greatest degree of protection, and the most transparent communications. If you have any questions or concerns about the above maintenance, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PST at 877-720-0483 Option 2.

******Update, Thursday March 5th, 08:11 hrs PST******

The scheduled maintenance on our UPS system has been cancelled due to complications with a vendor.

As always, protecting our clients remains our first priority. When a planned procedure cannot be carried out according to plan, our responsibility is to evaluate the situation and make a determination as to the safest course of action. In this situation the safest course is to reschedule the maintenance for a near-term future date.

Once rescheduled we will update the support blog with the new date and time and will follow essentially the same schedule as outlined below.

We apologize for any inconvenience this has caused and will continue to provide the most up-to-date information available.

If you have any questions or concerns about the above notice, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PST at 877-720-0483 Option 2.

******UPDATE******

The below schedule for "Tuesday, March 5th" should read "Thursday, March 5th"

The UPS system maintenance will occur on Thursday, March 5th.

******SERVICE IMPACTING UPS SYSTEM MAINTENANCE******

On Tuesday, March 5th starting at 06:00 hrs PST and ending at 18:30 hrs PST, we will perform maintenance on our UPS systems. This maintenance will include replacing the batteries in UPS 1 and upgrading the capacitors in UPS 3.

******Impacted Services******

Our FileMaker4, 5, and 6 hosting environment will be taken offline for 30 minutes in the morning at approximately 08:00 hrs PST and again for 30 minutes at approximately 18:00 hrs PST.

******************
For the duration of this maintenance our datacenter electrical load will be transferred to generator power.

This combined maintenance window has been selected based on factors to provide the greatest degree of protection for our clients. These factors include the availability of the most senior technicians from our UPS vendor and the ability to mobilize parts and additional service personnel in the very unlikely event of a component malfunction or failure during the maintenance. Additionally, by servicing both UPS systems during the same maintenance window, we eliminate an additional operation of our maintenance bypass switch thus limiting exposure to a potential voltage drop to datacenter critical equipment.

The schedule for this maintenance is as follows:

06:00 Datacenter Temperature Reduction: We will force the temperature of the datacenter toward the lower portion of the ASHRAE allowable envelope prior to transferring power to the back-up generator. When the transfer takes place our HVAC systems will power-cycle causing a small (3 to 5 degree Fahrenheit) thermal inclination in the datacenter. By lowering the overall space temperature we can be assured that equipment temperatures will not be adversely impacted by the 10-minute restart period of the HVAC systems.

07:00 to 07:30 Generator pre-flight evaluation: Check fluids, connections, air inlets/exhaust, fuel supply, fuel lines, filters and separators. Log results and announce startup to internal staff.

07:30 Generator start-up and warm-up: Start the generator and monitor performance stats, check for fluid leaks, supply artificial load and evaluate voltage, amperage, frequency and engine stats against established baselines.

07:45 FileMaker 4, 5, and 6 hosting environment shutdown: We will begin shutting down the FileMaker shared hosting environment at this time. The shutdown will be complete by 08:00 hrs and server environment restarts will begin shortly after 08:00 hrs.

08:00 Transfer from grid power to generator power and wrap power around UPS: Manually operate the ATS and UPS maintenance bypass to force the datacenter electrical load to the generator. At this time our HVAC system will power cycle as described above.

08:00 to 12:00 Replace batteries in UPS 1: The UPS system must be completely powered off for life safety while the batteries are removed and replaced. During this operation UPS 3 will remain online and provide UPS power to clients with A+B power.

12:00 to 12:30 Power on UPS system and perform artificial load testing: Following the removal and replacement of components, the UPS systems will be powered on and connected to an artificial load (not datacenter equipment, servers or other infrastructure) for testing. With this equipment we will test the transfer process and mechanism as well as simulate a load of 80% of the UPS system capacity.

13:00 Transfer datacenter load back to UPS 1: Once the process and mechanism has been validated we will transfer the datacenter load back to UPS 1 though power will continue to be supplied to the UPS from the back-up generator.

13:30 to 17:30 Replace capacitors in UPS 3: The UPS system must be completely powered off for life safety while the capacitors are removed and replaced. During this operation UPS 1 will remain online and provide UPS power to clients with A+B power

17:30 to 18:00 Power on UPS 3 and transfer datacenter load: Once the capacitors have been replaced and charged we will transfer the datacenter load back onto UPS 3.

18:00 Transfer Datacenter live load back to grid power: Once all testing has been completed and both UPS systems are online and functioning within specifications, we will transfer the datacenter live load back to grid power.

18:00 to 18:30 Generator Cool-Down and Post-Flight Evaluation: Following the operation of our generator we will perform the same evaluations we performed during the pre-flight as well as allow the generator to cool prior to shutting down.

One to two weeks following this maintenance, we will perform a battery preventive maintenance on UPS system 1 to validate the condition of the new batteries and replace any questionable jars. This preventive maintenance will follow the same procedure as above but require significantly less time to complete. A notice will be posted to our support blog prior to this future maintenance.

digital.forest remains committed to providing our customers with the highest level of service, the greatest degree of protection, and the most transparent communications. If you have any questions or concerns about the above maintenance, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PST at 877-720-0483 Option 2.

Just a reminder, tonight at 11 PM we will be performing an emergency maintenance on part of our electrical system. This will have an impact on a portion of our shared web and database servers, and eight dedicated and colocated servers. The shared hosting servers affected are:

acacia, acorn, alder, aralia, arrowwood, ash, aspen, avocado, balsa, balsam, bamboo, banana, banyan, bayberry, beech, bigleaf, blackberry, boojum, boxwood, bubinga, buckeye, cactus, cedar, cherry, chestnut, cholla, cinnamon, clover, columbia, commerce2, commerce3, cork, cottonwood, db, deerwood, dogwood, ebony, elm, evergreen, ficus, fig, filbert, fuji, gingko, grape, grapevine, hackberry, hazel, hemlock, hickory, ironbark, ivy, kentia, kola, kudzu, larch, laurel, lilac, lime, madrona, magnolia, mango, mangrove, maple, mesquite, mimosa, moringa, mulberry, myrtle, newninewire, ninewire2, olive, orchid, palmetto, papaya, pear, pecan, plum, poplar, privet, quince, redbud, sassafras, savin, sequoia, sherwood, snowberry, spiceberry, spruce, strawberry, sycamore, tamarack, teak, truffula, tutsan, walnut, woodpecker, yucca, yulan/fern

Every effort will be made to minimize the downtime. Our electricians have estimated the time required to complete the task at 2 hours.

Thank you for your patience and understanding while we strive to build a better facility.

Update 1:15 AM: The maintenance went very smoothly and was completed in under 90 minutes. We were able to supply alternate power to the few dedicated and colo servers involved in the maintenance. We also took the opportunity to replace some failed parts in a couple of servers (acorn for example whose fan inside the power supply had stopped working.)

Again, thank you so much for your patience during this critical maintenance interval.

Those who are hosted on hazel have noticed that there has been an issue with it and that it has been hanging a lot recently.

Diagnostics tools were run on it today, a clean copy of the operating system was installed (which included a FirmWare update).

No User files were changed.
In end effect, Hazel's uptime and stability should be a lot better from here on.

FTP access has been restored on hazel.forest.net.

It seems as though webstar has eaten its own data file for FTP users. This means that no one is able to log in via FTP and upload and download files. We are currently seeking the file off of our back up system which is taking longer than expected since the catalog file needs to be rebuilt. At the moment we are looking at having hazel.forest.net FTP back up and restored by the end of the day today.