digital.forest Technical Support
News archive: February 2009

Please Note: This posting to our Support Blog has no critical information concerning scheduled maintenance or datacenter expansion... it is just a bit of fun. Some geeky amusement for the technically-minded among our valued clients.

Uptime is an important metric. Here at digital.forest we know that uptime is the basis of our business. Everything is focused on uptime first, with all else coming next in line. When you run Internet-connected servers, as most of our clients do, the uptime of those servers can become an obsession, as this story will demonstrate. Our long-time clients Cheatcodes.com take their fun as seriously as they take their uptime. For example...

Through the normal course of business over the past several years since they moved their servers from our old facility in Bothell to our facility in Seattle, Cheatcodes.com has rebooted all of their servers except one. That server has been up continuously since that day in March 2005. Cheatcodes.com has outgrown their old rack and recently consolidated all of their servers into a new cabinet in digital.forest's Datacenter 1. But they REALLY wanted to maintain that unbroken streak of uptime for that server. Being the fun geeks that they are, Steve & Steve came prepared to move that server, while it stayed running. With dual power supplies, and four power cords, one of which was a long extension cord from from Steve J's garage, they carefully planned and executed moving their server "hot"...

Move complete, 100% Uptime maintained: 945 days and counting.

Congratulations to Steve & Steve of Cheatcodes.com, and thanks, as always, for the entertaining diversions!

posted by Chuck G. at 10:57 AM on Thursday, February 26, 2009
Categories: About digital.forest

******SERVICE IMPACTING NETWORK MAINTENANCE******

On Thursday, February 26th during our scheduled maintenance window we will be performing maintenance on one of our distribution switches. There will be a single 30 second outage caused by this maintenance. The maintenance will affect connection to shared hosting servers.

The maintenance will occur between 11:00 pm and 11:59 pm.

posted by Kyle at 06:04 PM on Tuesday, February 24, 2009
Categories: Network

******NON-SERVICE IMPACTING MAINTENANCE******

On Tuesday, February 24th at 09:00 hrs PST, our local fire department will be onsite conducting an inspection of the fire alarm and double pre-action system in our newest datacenter, DC 3. During this inspection the fire alarm system for that space will be placed in standby and the automatic fire suppression system deactivated. Fire department personnel will be conducting the tests and monitoring the space for any fire hazards. Our alarm strobes, sirens, and bells will all operate at different times and one of our HVAC systems will power cycle during the test.

At no time will there be any client impact or temperature inclination outside of the ASHRAE allowable envelope.

If you have any questions or concerns, please contact your account manager directly. Our account management staff is available Monday through Friday from 08:00 to 17:00 hrs PST and can be reached at 877-720-0483 Option 2.

posted by at 03:45 PM on Friday, February 20, 2009
Categories: Facility Maintenance

******Update, Thursday March 5th, 08:11 hrs PST******

The scheduled maintenance on our UPS system has been cancelled due to complications with a vendor.

As always, protecting our clients remains our first priority. When a planned procedure cannot be carried out according to plan, our responsibility is to evaluate the situation and make a determination as to the safest course of action. In this situation the safest course is to reschedule the maintenance for a near-term future date.

Once rescheduled we will update the support blog with the new date and time and will follow essentially the same schedule as outlined below.

We apologize for any inconvenience this has caused and will continue to provide the most up-to-date information available.

If you have any questions or concerns about the above notice, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PST at 877-720-0483 Option 2.

******UPDATE******

The below schedule for "Tuesday, March 5th" should read "Thursday, March 5th"

The UPS system maintenance will occur on Thursday, March 5th.


******SERVICE IMPACTING UPS SYSTEM MAINTENANCE******

On Tuesday, March 5th starting at 06:00 hrs PST and ending at 18:30 hrs PST, we will perform maintenance on our UPS systems. This maintenance will include replacing the batteries in UPS 1 and upgrading the capacitors in UPS 3.

******Impacted Services******

Our FileMaker4, 5, and 6 hosting environment will be taken offline for 30 minutes in the morning at approximately 08:00 hrs PST and again for 30 minutes at approximately 18:00 hrs PST.

******************
For the duration of this maintenance our datacenter electrical load will be transferred to generator power.

This combined maintenance window has been selected based on factors to provide the greatest degree of protection for our clients. These factors include the availability of the most senior technicians from our UPS vendor and the ability to mobilize parts and additional service personnel in the very unlikely event of a component malfunction or failure during the maintenance. Additionally, by servicing both UPS systems during the same maintenance window, we eliminate an additional operation of our maintenance bypass switch thus limiting exposure to a potential voltage drop to datacenter critical equipment.

The schedule for this maintenance is as follows:

06:00 Datacenter Temperature Reduction: We will force the temperature of the datacenter toward the lower portion of the ASHRAE allowable envelope prior to transferring power to the back-up generator. When the transfer takes place our HVAC systems will power-cycle causing a small (3 to 5 degree Fahrenheit) thermal inclination in the datacenter. By lowering the overall space temperature we can be assured that equipment temperatures will not be adversely impacted by the 10-minute restart period of the HVAC systems.

07:00 to 07:30 Generator pre-flight evaluation: Check fluids, connections, air inlets/exhaust, fuel supply, fuel lines, filters and separators. Log results and announce startup to internal staff.

07:30 Generator start-up and warm-up: Start the generator and monitor performance stats, check for fluid leaks, supply artificial load and evaluate voltage, amperage, frequency and engine stats against established baselines.

07:45 FileMaker 4, 5, and 6 hosting environment shutdown: We will begin shutting down the FileMaker shared hosting environment at this time. The shutdown will be complete by 08:00 hrs and server environment restarts will begin shortly after 08:00 hrs.

08:00 Transfer from grid power to generator power and wrap power around UPS: Manually operate the ATS and UPS maintenance bypass to force the datacenter electrical load to the generator. At this time our HVAC system will power cycle as described above.

08:00 to 12:00 Replace batteries in UPS 1: The UPS system must be completely powered off for life safety while the batteries are removed and replaced. During this operation UPS 3 will remain online and provide UPS power to clients with A+B power.

12:00 to 12:30 Power on UPS system and perform artificial load testing: Following the removal and replacement of components, the UPS systems will be powered on and connected to an artificial load (not datacenter equipment, servers or other infrastructure) for testing. With this equipment we will test the transfer process and mechanism as well as simulate a load of 80% of the UPS system capacity.

13:00 Transfer datacenter load back to UPS 1: Once the process and mechanism has been validated we will transfer the datacenter load back to UPS 1 though power will continue to be supplied to the UPS from the back-up generator.

13:30 to 17:30 Replace capacitors in UPS 3: The UPS system must be completely powered off for life safety while the capacitors are removed and replaced. During this operation UPS 1 will remain online and provide UPS power to clients with A+B power

17:30 to 18:00 Power on UPS 3 and transfer datacenter load: Once the capacitors have been replaced and charged we will transfer the datacenter load back onto UPS 3.

18:00 Transfer Datacenter live load back to grid power: Once all testing has been completed and both UPS systems are online and functioning within specifications, we will transfer the datacenter live load back to grid power.

18:00 to 18:30 Generator Cool-Down and Post-Flight Evaluation: Following the operation of our generator we will perform the same evaluations we performed during the pre-flight as well as allow the generator to cool prior to shutting down.

One to two weeks following this maintenance, we will perform a battery preventive maintenance on UPS system 1 to validate the condition of the new batteries and replace any questionable jars. This preventive maintenance will follow the same procedure as above but require significantly less time to complete. A notice will be posted to our support blog prior to this future maintenance.

digital.forest remains committed to providing our customers with the highest level of service, the greatest degree of protection, and the most transparent communications. If you have any questions or concerns about the above maintenance, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PST at 877-720-0483 Option 2.

Yesterday morning, Monday February 16th at 08:24 a network event occurred on a global scale. In short, a network in central Europe started feeding bad routing information to the entire Internet, which caused routers around the world to have problems. In our case, three of our four peer networks experienced "flapping", meaning their connections between their networks and ours going up and down. At no time was digital.forest "off the air" but as the entire Internet was unstable for a period of about 45 minutes to an hour, we may have been unreachable for some people, for some of that time.

We closely monitor the communications channels that enable the cooperative operations of the global Internet. Through these we quickly discovered the specific source and the proposed "fix" (filtering the source of the problem) and had it implemented not long afterwards.

posted by Chuck G. at 05:04 PM on Tuesday, February 17, 2009
Categories: Network

Emergency systems maintenance is required this morning on boysenberry.forest.net. The server experienced a hardware failure this morning at approximately 6:30AM PST. Evaluation of the hardware issue and the legacy nature of the server have led us to conclude that users on boysenberry will be best served by a migration to our newest MySQL server, honeysuckle.forest.net.

Boysenberry's IP will be added to honeysuckle, eliminating a need to make changes to database code.

We are working to migrate customer data to the new server as quickly as possible.

posted by digital.forest at 10:27 AM on Tuesday, February 17, 2009
Categories: Emergency Maintenance

******Service Impacting Maintenance Notice******

On Thursday, March 5th we will perform scheduled maintenance on our UPS systems. During this maintenance, our FileMaker 4, 5 and 6 hosting environments will be unavailable twice during the day for twenty minutes each time. No other services will be impacted during this period.

A complete schedule and description of the maintenance will be posted here on Monday, February 16th.

posted by at 01:39 PM on Friday, February 13, 2009
Categories: Scheduled Maintenance

******NON SERVICE IMPACTING NETWORK MAINTENANCE******

On Thursday, February 12th starting at 23:00 hrs PST and completing at approximately 01:00 hrs PST on February 13th, we will change the BGP configuration on our border routers as well as the physical configuration of two of our peers. These changes will increase available bandwidth, improve routing and better balance bandwidth across our peers.

During this maintenance persistent connections such as VPNs may become temporarily unavailable while new best routes are established.

Traffic will pass normally through our other upstream peers during this maintenance.

Following this maintenance, some traffic may use more efficient routes that were previously unavailable.

posted by at 01:31 PM on Monday, February 9, 2009
Categories: Network

******Service Impacting Maintenance******

Today at 14:45 HRS PST we will be performing an emergency reboot of our FileMaker 8 server jasmine.forest.net.

The server will be unavailable for approximately 10 minutes during the reboot.

posted by at 02:37 PM on Thursday, February 5, 2009
Categories: jasmine.forest.net

******Service Impacting Maintenance******

Tonight at 23:00 hrs PST we will be performing emergency maintenance on one of our email servers, treehouse.forest.net

The maintenance will require a server reboot which will cause it to be unreachable for 10 to 15 minutes. During the time the server is unreachable, mail will be delayed but not lost.

Once this service has been completed the server will become reachable and services will continue to function normally.

posted by at 01:46 PM on Thursday, February 5, 2009
Categories: Mail, treehouse.forest.net

On Tuesday, February 10th, starting at 07:00 hrs and ending at approximately 14:00 hrs, we will be conducting our semi-annual fire alarm inspection and test. During the test bells, alarms and strobes will be operated to verify functionality and the various components of the fire suppression system will be inspected.

The fire system control panel will be placed in bypass by the vendor during this inspection but all manual pull stations will remain active. Monitoring will be performed by both d.f and vendor staff during the inspection.

This entry will be updated once the inspection has been concluded.

posted by at 05:18 PM on Wednesday, February 4, 2009
Categories: Facility Maintenance