digital.forest Technical Support
Breaking News

The Sales, Billing, and Customer Service departments of digital.forest will be closed from 5:00 PM today, through the weekend, opening at 8:00 AM Monday, July 6th.

Technical Support will remain on-site throughout the holiday.

Our building and campus will be closed from 5 PM today through 8 am, Monday July 6th. During that time clients wishing access to the datacenter will need to contact digital.forest Technical Support for building access.

All of us here at digital.forest wish you and yours a very happy Independence Day!

posted by Shawn Hammer at 12:20 PM on Thursday, July 2, 2009
Categories:

******NON-SERVICE IMPACTING UPS SYSTEM MAINTENANCE******
On Tuesday, June 23rd starting at 07:00 hrs PST and ending at 18:30 hrs PST, we will perform maintenance on our UPS systems. This maintenance will include adding a graphic interface to UPS's 1 and 2 and adding additional monitoring devices to all UPS equipment.

******************

For the duration of this maintenance our datacenter electrical load will be transferred to generator power.

This maintenance window has been selected based on factors to provide the greatest degree of protection for our clients. These factors include the availability of the most senior technicians from our UPS vendor and the ability to mobilize parts and additional service personnel in the very unlikely event of a component malfunction or failure during the maintenance. Additionally, by servicing both UPS systems during the same maintenance window, we eliminate an additional operation of our maintenance bypass switch thus limiting exposure to a potential voltage drop to datacenter critical equipment.

The schedule for this maintenance is as follows:

07:00 to 07:30 Generator pre-flight evaluation: Check fluids, connections, air inlets/exhaust, fuel supply, fuel lines, filters and separators. Log results and announce startup to internal staff.

07:30 Generator start-up and warm-up: Start the generator and monitor performance stats, check for fluid leaks, supply artificial load and evaluate voltage, amperage, frequency and engine stats against established baselines.

08:00 Transfer from grid power to generator power and wrap power around UPS: Manually operate the ATS and UPS maintenance bypass to force the datacenter electrical load to the generator. At this time our HVAC system will power cycle as described above.

08:00 to 10:30 Add additional monitoring equipment to all UPS systems.

10:00 to 17:00 Add graphic interface to UPS's 1 and 2

17:30 to 18:00 Power on UPS system and perform artificial load testing: Following the completion of work, the UPS systems will be powered on and connected to an artificial load (not datacenter equipment, servers or other infrastructure) for testing. With this equipment we will test the transfer process and mechanism as well as simulate a load of 400% of the UPS system capacity.

18:00 Transfer datacenter load back to UPS's: Once the process and mechanism has been validated we will transfer the datacenter load back to the UPS's though power will continue to be supplied to the UPS from the back-up generator.

18:15 Transfer Datacenter live load back to grid power: Once all testing has been completed and both UPS systems are online and functioning within specifications, we will transfer the datacenter live load back to grid power.

18:15 to 18:30 Generator Cool-Down and Post-Flight Evaluation: Following the operation of our generator we will perform the same evaluations we performed during the pre-flight as well as allow the generator to cool prior to shutting down.

digital.forest remains committed to providing our customers with the highest level of service, the greatest degree of protection, and the most transparent communications. If you have any questions or concerns about the above maintenance, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PST at 877-720-0483 Option 2.

posted by Shawn Hammer at 01:03 PM on Wednesday, June 17, 2009
Categories: Facility Maintenance

****** SERVICE IMPACTING MAINTENANCE******

Tonight at 21:00 hrs PST we will be replacing a failing fan in one of the DC0 switches. There will be about 5 minutes of downtime for the FileMaker hosting servers connected to this switch.

digital.forest remains committed to providing our customers with the highest level of service, the greatest degree of protection, and the most transparent communications. If you have any questions or concerns about the above maintenance, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PST at 877-720-0483 Option 2.

posted by Kyle at 03:27 PM on Monday, June 15, 2009
Categories: Network

***** Incident Report ******

Between 15:20 hrs PST and 15:35 hrs PST on June 8 we experienced a network event.

The event was a UDP dDoS attack directed at one of our clients. The attack produced a very significant flow rate and caused some routing instability as 2 of our BGP sessions flapped (dropped and came back up) twice.

Unlike the *very* similar attack last week (both UDP, both 29 byte packets, both caused 13 minutes of BGP instability), this attack did not cause all of our BGP sessions to reset over the duration of the attack. Also different, today's attack was directed at a different IP address.

Our NOC staff identified the attack and took action to mitigate impact to the network immediately. Network impact was limited to degraded performance for approximately thirteen minutes.

For our clients privacy and security we are do not share syslog or netflow data from these events in this forum.

We monitor our network closely to identify any potential service impacting events as they develop in order to prevent them causing a network outage. Today's event caused only network instability and our network continued forwarding packets to the internet at all times.

digital.forest remains committed to providing our customers with the highest level of service, the greatest degree of protection, and the most transparent communications. If you have any questions or concerns about the above maintenance, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PST at 877-720-0483 Option 2.

posted by Shawn Hammer at 05:15 PM on Monday, June 8, 2009
Categories: Network