Between the hours of 23:00 PST and 24:00 PST of November 28th, 2007, we will be performing some operating system upgrades on the following servers:

Columbia
Kudzu
Jasmine
Date
Helpdesk
Honeysuckle
Rosemary
Thyme

During the first 15 min of this hour the servers may respond a bit slower then normal and at some point after 23:15 PST they will be restarted and will be unavailable for up to 45 minutes. We expect all of these servers to be back up by 24:00 PST.

Thank you for your patience.

Between the hours of 12:01 AM and 1:00 AM Thursday, November 29th, and Friday, November 30th, we will be performing some hardware upgrades on portions of our network.

We will be replacing gigabit Ethernet modules on a few switches deployed in the colocation portions of our facility. Expect intermittent outages, likely less than a minute in duration, but possibly lasting up to five minutes while we install these improvements. Every effort will be made to minimize downtime.

Thank you for your patience.

In observance of the Thanksgiving Holiday, digital.forest will be closed Thursday November 22rd and Friday November 23th. We will resume regular business hours at 8am PST Monday November 26th.

Technical Support staff will remain on-site 24 hours a day throughout the holiday period. Please be aware that we'll have limited staff coverage for telephone tech support during the next few days. Please note that our building will be locked throughout the weekend and clients requiring access to the datacenter to work on colocated servers will have to call or email first to be allowed access to the building. We appreciate your patience in this regard.

All of us at digital.forest wish you a happy Thanksgiving and a wonderful holiday season!

Chuck Goolsbee
VP, Tech Ops
digital.forest

This morning we discovered a hardware failure with callisto.forest.net.

Thankfully all data has been recovered and we have already found replacement hardware. The new hardware is currently running some preliminary tests and checking its hardware, callisto should be back as soon as this is complete. We will update this blog when callisto has fully recovered. Sorry for any inconvenience, and thanks for you patience.

Update 11/21/2007 09:49 PST: The callisto server has returned to operational status and all services of that server should be working properly again.

UPDATE: 11-11-07 01:36 PDT Our upstream has resolved their routing issue by replacing a failed router processor card. We have turned up our interface and traffic is back to normal.

Tonight around 20:20 PDT we started seeing an issue with one of our network peers. Customers reported intermittent connectivity issues to our network. We contacted our peer and confirmed the issue and by 21:15 we determined the best course of action was to shut down our interface with them until the issue on their network was resolved. So for the time being one of our upstream connections is offline. This is not an issue really as we have more than enough bandwidth on our other connections to handle our full load and then some.

Our peer will notify us when they have corrected the issue on their network and we'll re-establish our connectivity with them at that time.

We'll update this post with more news as required.

Tomorrow at 9AM we will be doing major maintenance to rosemary.forest.net, our FileMaker 9 server.

I'm very happy to announce that the cooling phase of our current datacenter expansion is now complete. The two new Aaon 70 ton Air Handlers have been installed, and started up successfully. This is the largest project we've taken on since our move to the Seattle datacenter in 2005, and like that project this one finished on time, and with no major headaches or heartbreaks. Plenty of hard work, but everything went very well. The weather caused us some delays, however today was clear (though a bit cold to be having a party on the roof!)

We gathered on the roof to celebrate:

Above L-R: John Ford (Sabey), Connie Bradley (d.f), Shawn Hammer (d.f), Lee Pallat (d.f), Larry Harbison (MacDonald-Miller), Steve Flink (MacDonald-Miller), Andre Bearfield (d.f), Spike Graham (d.f), Tim Beyer (d.f), Paul Sampson (d.f), Rick Russell (d.f).

And of course launch our new machines in style:

Taking a bottle of champagne in hand, digital.forest CEO Kris Bourne on the southern HVAC system, and Connie Bradley from our Billing Department on the northern unit, we listened to Facilities Manager Kevin Teker give a toast and christened the units with their new names "Rainier" & "Baker" (named after the ice-covered mountains on either horizon) ... and the ladies let the bottles fly!

We then enjoyed champagne and some smoked salmon, basking in the hum of the running machines.

Huge thanks go out to our primary contractor, MacDonald-Miller for a job well done! Sub-contractors involved were VECA Electric, ATS Controls, Ness Crane, Aaon, & Centimark Roofing. Here at digital.forest my hat goes off to Shawn Hammer and Kevin Teker, who planned, budgeted, contracted and managed this project start to finish... all while carrying out day to day operations of our facility. Great job guys!

In the coming weeks we'll be massaging the controls to get the air balanced and the multiple systems all working together in perfect harmony. We also need to finish the hot-air return/recover ducting in DC2. The next phase of the expansion is beginning work on DC3. Stay tuned for more information about that.

Above: Always mindful of every detail about the facility, Kevin sweeps up after the celebration.

Today is the day we startup the two new HVAC systems. Stay tuned for updates and photos.

The electricians have finished the wiring and disconnect panels.

The controls vendor is adding the new HVAC to our control systems.

UPDATE: 10:30 PST One of the two units has been started up!

UPDATE: 11:45 PST Both of the units have been started up. We're working on some basic operating issues of the second unit that need to be ironed out.

Above: Techs install some dampers in the air intake. The Seattle skyline in the background.

Above: The brains of the new system.

Update: 14:30 PST Both systems are up and fully operational! All that is left to do is celebrate...

We will be taking the FileMaker 6 Server named boojum offline this morning to perform some emergency maintenance on the Userland Frontier agent. We do not currently have an ETA for when this work will be completed but we will update this entry as we have more information.

Thank You

11/07/2007 11:14: Boojum is now back online and the error in UserLand Frontier has been corrected.

Finally. After weeks of snapping off images at intervals ranging from 1 to 15 minutes the digital.forest ConstructionCam has produced the final product. Enjoy the two weeks condensed into 4 minutes. We'll put a link to a high-res version on one of our streaming servers soon.

Quite a bit of work remains, and in fact our contractors are up on the roof right now. As stated yesterday we're planning for a turn up of these units early next week. We'll post more info as it becomes available.

Our primary contractor MacDonald-Miller has done an excellent job, thanks guys!

We will be taking the FileMaker 9 Server named rosemary offline this morning to perform some emergency maintenance on the FileMaker Instant Web Publishing. We do not currently have an ETA for when this work will be completed but we will update this entry as we have more information.

Thank You

11/07/2007 10:25AM: Rosemary is now back online and the error in IWP has been corrected.

As we announced earlier, our new steel frames, and two HVAC units arrived today. It was a big event. I know our loyal clients expect me to provide them with timely and complete updates of our construction progress, but today I was late, sorry! The Puget Sound area was covered in thick fog, and despite a pre-dawn start to get here traffic delayed me and I arrived as the second steel frame was being craned up over the edge of the roof. I snapped this shot literally as I parked my car:

Running up to the roof with my MacDonald-Miller supplied hard hat on this is the scene I found:

Both steel frames already in place and workers from MacDonald-Miller installing the duct work between the openings created last week and the frames.

Top image above: A view down into the datacenter (sealed with plastic during the install!) via the hot air return duct.
Bottom image above: Kevin Teker inspects the steel frame before the HVAC units arrive.

Within an hour the HVAC units arrived on a flatbed truck. The rest you see captured on the video above. Craning the units up onto the roof and placing them on the steel frames. The crane itself was a very impressive unit, and the crew from Ness Crane were very professional and did an outstanding job swiftly, and accurately.

The steel frames fit perfectly onto the supports installed last week, and the HVAC units fit perfectly on the steel frames. Today was the first time these two components met, but since they were designed to go together, everything fit perfectly. The tolerances achieved were amazing, everything fit to within 1/64th of an inch. (That's 0.396875 millimeters... which accordion to my calculations is.... amazingly small!) We have to hand it to the guys at MacDonald-Miller, they did an fantastic job of design and fabrication. Today it all came together after months of planning and work. Good job guys!

Above: A nice view at the business end of the HVAC unit. On the left side is the cold air supply outlet, and on the right is the hot air return intake. These mated up to the ducts placed in the roof last week, after the unit was spun around in mid-air of course.

Above: Checking the fit and placement of the unit before it is lowered the final amount. Watching this huge unit being gingerly placed by a gigantic crane was very interesting to watch.

Afterwards the units were welded into place. Welding will continue tomorrow. The electricians will be here later in the week to run power to the units. Turn-up is scheduled for Tuesday, November 13th. Ironically snow is forecast for Wednesday, November 14th. This project began in the planning stages last Spring, and I always joked that we'd have our new air-conditioning units installed "just in time for Winter." Funny how it all works out.

Stay tuned for updates as work progresses.
(and YES, constructioncam will be updated tonight!)

Chuck Goolsbee
VP Technical Operations
digital.forest

All of us here at digital.forest are happy to announce the arrival of our newest family members, twin HVAC units. They arrived by crane (a member of the stork family) at 9:55 and 10:15 am today, both weighing in at 70 tons apiece. We're not sure of their gender, and haven't come up with any names for them yet, but we're all very proud and happy that the delivery went well. No complications and they have all their parts intact.

Above: The new arrivals, resting in their cradles after delivery.

The delivery crew are all enjoying coffee and cinnamon rolls as we speak and are in good spirits. More photos and videos will be posted soon!

Above: The proud parent, digital.forest Facilities Manager Kevin Teker (in red hard hat, center) flanked by the delivery crew. The MacDonald-Miller guys on the left, and Ness Crane crew on the right . L-R: Jared Sheeks, Steve Flink, Justin Skinner, Larry Harbison, Larry Mayrand, all from MacDonald-Miller. Kevin Teker, digital.forest. From Ness Crane: Bobby Aynes, Jim Burnett, & Mark Crawford.
Not pictured from Mac-Miller, but involved in the pick: Mike Klein, Andy Burlingham, and Steve Harbison.

Tomorrow is the big day. Steel frames and HVAC units arrive, and will be craned up onto the roof. Client access to DC2 will have to be restricted during times the crane is operational, which will be from just after dawn until around noon. We will update the support blog as work progresses.