As part of the ongoing process to improve and enhance the statistical reporting package on our FreeBSD servers, one small change has been made to the error logs found in your home directory.
Previously, the current log was named "error_log", and all older logs were named "error_log-YYYYMMDD.bz2", reflecting the date of the log and the fact that it had been compressed.
Starting today, the current log will be named "error_log-YYYYMMDD", using the current day's date. The older logs remain unchanged.
If you have any questions or concerns about this change, please contact digtal.forest Technical Services via a trouble ticket at http://helpdesk.forest.net or via phone at 206-838-1630 option 3
************
digital.forest remains committed to providing our customers with the highest level of service, the greatest degree of protection, and the most transparent communications. If you have any questions or concerns about the above maintenance, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PDT at 877-720-0483 Option 2.
posted by Shawn Hammer at 09:57 AM on Tuesday, September 15, 2009 Categories:butternut.forest.net
Between approximately 01:00 HRS PDT and 04:00 HRS PDT on 9/10/09 digtal.forest experienced an intermittent network event that had noticeable impact on specific segments of our network. The event was caused by what our network manager has described as a broadcast storm that was initiated by some client equipment. A broadcast storm can be understood as the result of a malfunction of some computer device(s) (in this case a configuration error or driver failure in some client equipment) sending out a large number of broadcast packets or a continuous stream of broadcast packets. Those packets are then forwarded by every port on every network device on the network effectively amplifying the broadcast by the number of ports. This causes a storm of traffic that continues to grow as long as the malfunctioning equipment is connected to the network. It is normal for all network devices to send broadcast packets and at any given time we will have some amount of broadcast traffic on the network. During this event there were devices sending 200 times the normal amount of broadcast packets.
The duration of the event is a result of the amount of traffic and how that traffic impacts isolating the originating device(s). There is no easy way locate a small number of devices or single device in a broadcast storm because every device on the network is forwarding the broadcast traffic. The particular equipment that impacted the network has been isolated and is currently under evaluation to determine (if possible) the cause of the broadcasts. None of this equipment will be placed back on the network until it we can be certain there is no further risk of malfunction.
During the event our local network continued to pass traffic to the Internet without disruption. At several times though, traffic was passed very slowly from very specific segments of the local network out to the Internet. The reason some parts of the network were impacted and others were not is a result of our having multiple routers at our network core connecting to multiple discrete upstream providers. During this event one of the routers received the bulk of the broadcast traffic and was bogged down processing it while the other equipment continued to operate normally. Traffic departing our network on the normally operating equipment was not impacted by the event.
We at digital.forest apologize for any inconvenience this event may have caused you and greatly appreciate your business and patience.
************
digital.forest remains committed to providing our customers with the highest level of service, the greatest degree of protection, and the most transparent communications. If you have any questions or concerns about the above maintenance, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PDT at 877-720-0483 Option 2.
posted by Shawn Hammer at 10:33 AM on Friday, September 11, 2009 Categories:Network
In observance of Labor Day, our business offices (Sales & Billing) will be closed on Monday, Sept 7th. Regular hours will resume at 8 am on Tuesday, Sept 8th.
Technical Support will remain open all day Monday, and all through the preceding weekend. Colocation customers without card keys requiring access to the datacenter anytime between 6 pm Friday, Sept 4th and 8 am Tuesday Sept 8th will need to contact our NOC for access to the building and elevators as they will be secured throughout the holiday weekend.
posted by digital.forest at 12:01 AM on Saturday, September 5, 2009 Categories:Holiday Hours
We transferred back to grid power from our generator at 11:05 PM. The Secondary Generator never saw any use, but it did its job, namely providing us with a proper safety net. Now the task of packing up all the cables and putting everything back into normal operating conditions begins.
Our thanks go out to Sabey, for staying on Seattle City Light to get this maintenance completed tonight.
Thanks as well go to the digital.forest Technical Operations staff who have worked tirelessly over the past several weeks to prepare for this event. The fact that it went seamlessly is a testament to their professionalism.
******21:30 Update******
Grid power has been restored to the campus. We will be transferring back momentarily.
******20:00 Update******
Word from SCL is that grid power may be restored around 10 PM.
******19:00 Update******
Things are running well. SCL continues to do their work. We are still here keeping things running. Here is a quick video we shot showing a regular check of the backup power system as it is running. (Warning: It is very loud in the generator room, put on your ear protection.)
Above: Our building at 5 PM today.
******12:00 (Noon) Update******
The Good News: Everything is running fine.
The Bad News: Seattle City Light's original ETA of "around 3 PM" for completion of the work and restoration of grid power is now "around 6 AM tomorrow."
We are prepared. We have enough fuel on-site to run until Tuesday of next week, though we'll likely top off tanks later today.
Meanwhile Seaport Biofuels just stopped by to top off the tank of the generator running the building elevator and common-area lights:
******10:30 Update******
Above: digital.forest Facilities Manager Kevin Teker points to the fan relay he just replaced.
Why there is really no such thing as a "lights out datacenter" and why do we maintain all of our own facility systems? A short time ago Kevin Teker, our Facilities Manager performed a fan relay swap. On the right-hand side of the image above you see several large cooling fans. They supply air for our backup generator system and are driven from large electric motors. A short time ago they stopped working when their relay failed. Kevin noted this and opened up the box where the relay is located and replaced the failed relay with one of the spares he keeps right in that very box. We love Kevin, and you should too. Why? He always thinks of these things and like a Boy Scout, is always prepared. Had this happened in an unattended "lights out" facility the generator would have slowly overheated and perhaps eventually failed. Instead, because our lights are on and we're always here at digital.forest, the issue was noted, and resolved in a matter of minutes. It is never the big things that go wrong, it is the little things that eventually become big things. We keep an eye on the little things. Not the landlord. Not an outsourced facilities contractor. Not a real estate company. Our full-time staff here at digital.forest is always looking out for you.
Above: Kevin labels everything. The new relay is installed on the right, the new spare, replacing the old spare, is on the left side of the relay box.
Above: Kevin closes up the relay box after he's done with the repair.
******09:45 Update******
Grid power went offline around 09:00. All systems running normally.
Above: The reason for this event: Road construction on International Blvd. Seattle City Light is relocating the power feed for the campus as part of the road widening.
******08:45 Update******
Above: Generators.
The photo shows several generators. The large one on the left is the secondary generator for the digital.forest datacenter. The small green one in the middle is running backup power for the building, including one of the two elevators. The digital.forest offices are powered by our own generator but it is nice that the elevator is working today. In the distance you can see another portable generator in front of Building B. The entire campus will be off the grid during this maintenance event.
Above: More Generators.
The large generator on the left is another view of the secondary generator for our datacenter. The small one on the right is running the block heaters for the secondary generator. This minimizes the startup time for the secondary should we need to bring it online. In the background you can see the exhaust stack from our primary generator.
Above: The view of all the connections in the basement.
This is the scene at the convergence of all this activity in our building. Behind the sound baffles is our primary generator system. To the left of the door is our Bus Tie, where the "portable" generators tie into our power system. On the right is where the building backup generators to run lights, elevators, etc are tied in, as well as the other datacenter facility in our building.
******08:15 Update******
We transfered our facility load to our primary generator at 08:00. All services are operating normally and are expected to continue doing so. Seattle City Light crews will de-energize the campus feed sometime in the next hour or so. We will not notice any change at that time as we are already using our own generator power.
Above: The calm night before...
Today is the big day. Seattle City Light is relocating the power feed for our campus as part of the road construction ongoing on Tukwila International Blvd. We expect to be off the grid for several hours. In preparation we have brought in secondary backup power generators, as we always want to have at least two sources of electricity at all times. We expect to run the entire duration on our primary generator system, but have the secondary system in place. In the above photo you can see several "roll up" generators in place last night. We'll explain all of their purposes when we have time today. Check back often to read of status reports and more photos throughout the day Friday, September 4th.
On September 4th, as part of the widening of Tukwila International Boulevard, the electrical service to the Intergate.West campus will be shut down for several hours as it is relocated by Seattle City Light. Our facility will operate on generator power and will be further supported by secondary generator for the duration of the outage. We have taken every precaution available to ensure continuous power delivery to our customers, including pre-testing of secondary generator equipment. We will have a full compliment of staff and vendors on site, and will post updates here on the Support blog throughout the entire day. The following is a projected schedule of day's events:
************
0500 Hours PST - Veca Electric will begin relocating electrical circuits supplying some building common areas (Lobbies, hallways, restrooms, and elevators) to a backup generator to provide power through the duration of the maintenance. Anyone on site during this period may notice some dark areas in the building and a reduction to one operational elevator as the circuits are moved.
0600 Hours PST - Veca Electric will power the moved circuits with the backup generator and ensure life safety equipment, stairwell lights, and restroom lights are all functioning normally. digital.forest staff will prepare our secondary generator for standby for the duration of the day.
0700 Hours PST - digital.forest staff will preflight our main generator and verify all equipment is ready and operational for the day.
0730 Hours PST - digital.forest staff will start our main generator and begin run-up prior to transferring the data center load
0800 Hours PST - digtal.forest staff will manually transfer the data center load to the primary generator
0815 Hours PST - digtal.forest staff will notify project coordinator that load is successfully transfered and that we are ready for Seattle City Light to turn service off for the maintenance.
0815 to 1600 Hours PST - digital.forest will monitor generator status for signs of impending service degradation or failure and to initiate transfer to secondary generator if necessary.
Between 1400 and 1600 Hours PST - Seattle City Light will notify that maintenance has been completed and reinstate grid power to the campus.
30 minutes following stable grid power notification - digtal.forest staff will transfer data center load back to grid power and complete post flight procedures on the primary generator. Veca Electric will relocate electrical circuits supplying the building common areas (Lobbies, hallways, restrooms, and elevators) to their permanent panel positions to provide power from the grid. Anyone on site during this period may notice some dark areas in the building and a reduction to one operational elevator as the circuits are moved.
Following this maintenance we will post a notice here indicating that the data center is operating on grid power and that the maintenance is completed.
************
digital.forest remains committed to providing our customers with the highest level of service, the greatest degree of protection, and the most transparent communications. If you have any questions or concerns about the above maintenance, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PST at 877-720-0483 Option 2.
posted by Shawn Hammer at 01:58 PM on Thursday, September 3, 2009 Categories:Facility Maintenance
