As we've previously announced, we are performing biannual maintenance on our UPS system today, as well as performing some monitoring system upgrades on our older UPS systems. This involves taking the UPS system offline for portions of the work. While this is happening we'll have our backup power systems running to carry our load. We'll post more info as the work progresses.

Update - 12:10 PM PDT: The maintenance work is complete.

Tonight during our scheduled maintenance window, the servers palm.forest.net and tangerine.forest.net will undergo RAM upgrades. Each server will be down for approximately 5 to 15 minutes between the hours of 11:00 PM and 1:00 AM Pacific Standard Time. Every effort will be made to minimize the servers' offline time.

Please note: Palm is a mail server. Email will be unavailable while the server is offline. Inbound mail will be spooled on sending servers are delivered after the maintenance interval. Palm does not relay outbound mail itself, it works with one of our outbound mail hubs here so this event should have no impact on outbound email.

Tangerine is a database server running mySQLv4. Dynamic content on several of our other hosting servers may be affected during the maintenance interval.

Performance of both servers should be improved after their upgrades.

We finally turned up the new circuit over the weekend. Friday night to be specific. Close observation over the weekend proved that the circuit was performing exactly as we had anticipated. At this time we're happy to declare the installation a success and start focussing on our next project. Stay tuned for news on that very soon.

Thanks for your patience as we completed this installation.

UPDATE 02/01/08: As of 11:07 PM PST we are fully up and running on our newest BGP Peer.

Last week I announced that we were adding a new BGP peer. It was originally scheduled for last weekend, but ended up not happening on schedule.

After clearing a few technical and procedural hurdles this week, we're finally ready for this to actually happen, and it is now scheduled for Friday night. If you are extremely curious as to the nature of the delay, feel free to read on.

---

Please accept our apologies for the delay which unfortunately was completely beyond our control. To explain what happened I need to provide a bit of background on how the Internet works. Please remember that I am vastly simplifying a very complex system in order to condense this into a small blog post... books as heavy as boat anchors have been written on this subject but I really can't go into the minutiae here without everyone's eyes glazing over... so here is the Cliff Notes version:

* The Internet is a collection of autonomous networks, all interconnected.
* Networks are collections of hosts each given a unique address.
* The glue that holds the networks together is called BGP.
* BGP sees networks as aggregate collections of addresses called "prefixes"

So when we connect to another network, we announce our prefixes to them and they announce theirs to us. At either end of the connection are network devices called routers and they do filtering and weighting to decide what routes work best for your traffic. Filtering is important because it allows networks to send & receive the proper traffic and ignore improper traffic. For example if digital.forest has a connection with both "Network A" and "Network B". However we do not want to be a transit point BETWEEN "Network A" and "Network B" so we filter appropriately. We only want "our" traffic to go over these routes, not the whole world's traffic. Every network does this to a certain extent if they are connecting to multiple other autonomous networks.

Most large transit networks use routing databases to associate autonomous networks with their announced prefixes. This acts as a security & authentication layer, as well as a basis for filtering policies as the networks that query the databases. The databases are maintained (usually) by the entities that allocate the addresses, so they are a trusted source. The databases are then replicated and shared among the network operators. There are also "route servers" and "looking glasses" at various locations around the Internet for network operators to check to see how they fit into this big meshed network and verify that what they want to happen, is in fact happening.

Mind you all of the above is a vast simplification, so if you knew nothing about this until now, it is hopefully understandable. If you already know how all this works you know I left plenty of detail out, but you should hopefully recognize that it is all basically correct. Now on to what happened over the past week...

Here at digital.forest we announce several prefixes. A few of our own, and several on behalf of our customers who have been allocated specific IP address ranges different than ours. Last weekend we turned on our new circuit in the wee hours one night and from here it looked great - traffic flowed at a rate we expected it to. But before we went too far along in time we consulted the various route servers out there to see what the Internet saw: How did this new connection look from the outside looking in? What we saw was just one of our prefixes being carried by this new connection. Not wanting to risk weird routing issues we shut the new circuit down and got in contact with the provider's NOC to see why the all the prefixes we announced were not picked up by their network. This prompted a round of paperwork and approvals on their end, as we discovered that the do not rely on the routing databases to determine their route filtering policies. Instead they do it manually. I will not make any judgement calls as to that policy of theirs... I understand why some entities choose manual methods over automatic ones, after all I shift my own gears when I drive... sometimes manual systems are a better choice. In this case though it certainly slowed down the process. We submitted our full prefix list to them early in the week. It took them until yesterday to enter them in their systems. We are waiting a full 48 hours for the projected propagation time so that their entire network, and their BGP peers pick up the changes, then we will re enable the circuit. Kyle Murray, our Network Manager has been the man on point throughout this process and has done an excellent job making sure it all goes well.

Several of our clients are looking hopefully at this new circuit with some expected performance increases as it is a recognized "better" network than the circuit we are replacing. These clients are also some of the specific secondary prefixes that we announce. We wanted to make sure that this circuit turn up goes very well with no possibility for unusual behavior of our clients' traffic. Hence the delays to make sure everything was exactly as it should be. We are now very confident, but will go through the same process as last time: turn up, then check and see how it looks both from within and without. Trust, but verify.

My goal in these posts is to provide you with clarity as to what happens here at an operational level at digital.forest. We are blessed with excellent staff, and truly the best clients a company could hope for. I enjoy sharing this information and I hope it serves to boost your confidence in us as we care for your vital systems in our facility and on our network. I know that you look to us to "just make it work" but it can only help for us to communicate on an ongoing basis what is involved behind the scenes to accomplish that task.

Regards,
Chuck Goolsbee
VP Technical Operations
digital.forest, Inc

Our maintenance could not be completed this morning. We will be doing the work during our scheduled maintenance window this evening between 11:00 pm and midnight.

Our network maintenance originally scheduled for the early tomorrow morning has been pushed back a bit by the vendor. It is now scheduled for the early morning hours of Monday, January 28th.

We will be adding a new BGP Peer over the weekend. The actual cross-connect of the fiber circuit is happening today and BGP turn-up will happen sometime in the wee hours of Friday or Saturday. We'll be adding AS4323, also known as Time-Warner Telecom via a gigabit Ethernet connection. TWTC will be replacing our Fast Ethernet circuit we've had with AS2828, also known as XO Communications.

This should have no impact on service for any of our clients, just a change in routing at our network boundary. If anything we should see an improvement in performance overall. No changes to our other connections is scheduled.

In a few weeks we will be adding another Gigabit Ethernet circuit with AS3356, also known as Level(3). We'll post more news on that as it approaches.

On Wednesday, December 19th we will be performing some maintenance on our UPS system. While this maintenance is being performed we will put the UPS system and grid power in bypass and operate the datacenter on generator power. This is standard procedure for work such as this. While we do not expect any change in power delivery, any time power is switched from one source to another, a risk of interruption is possible. Every effort and precaution will be taken to minimize this risk.

We will also be turning up an additional 100kVA PDU in Datacenter 2 to facilitate growth in that area.

Thank You,

--Chuck Goolsbee
VP, Technical Operations
digital.forest, Inc.

Between the hours of 23:00 PST and 24:00 PST of November 28th, 2007, we will be performing some operating system upgrades on the following servers:

Columbia
Kudzu
Jasmine
Date
Helpdesk
Honeysuckle
Rosemary
Thyme

During the first 15 min of this hour the servers may respond a bit slower then normal and at some point after 23:15 PST they will be restarted and will be unavailable for up to 45 minutes. We expect all of these servers to be back up by 24:00 PST.

Thank you for your patience.

Tonight during our scheduled maintenance window (11:00pm-2:00am) we will be making some changes to our network configuration. This will momentarily interrupt service on one of our upstream network connections. There should no service impact as our other connections should carry all our traffic.

This is being done to address the Comcast routing issue posted yesterday.

We successfully completed our UPS upgrade work today. We were able to transfer to generator power, bypass the UPS, shut down the UPS, perform the work, the reverse the whole process and restore grid power. Everything worked as planned and as expected. We spent approximately 2.3 hours on backup power.

Above you can see the UPS Technician from MGE performing some circuit board replacements and firmware upgrades on our bypass switch.

Below you can see a gauge that shows the output from our backup power generator system. Running our entire facility's load, which includes all the servers and all of our mechanical systems (air conditioners, humidity controls, fans, etc) uses only 22% of our backup power system's capacity. Lots of room to grow.

Stay tuned for more news about our datacenter facility expansion.

Chuck Goolsbee
VP Technical Operations
digital.forest, Inc.

Between the hours of 9am and 12pm, Thursday, September 20th we will be performing maintenance on our UPS system, which will include upgrading some control boards and updating to the latest firmware. This is the final step in our recent UPS upgrade.

While this maintenance is being performed we will put the UPS system and grid power in bypass and operate the datacenter on generator power. This is standard procedure for work such as this.

While we do not expect any change in power delivery, any time power is switched from one source to another, a risk of interruption is possible. Every effort and precaution will be taken to minimize this risk.

Thank You,

--Chuck Goolsbee
VP, Technical Operations
digital.forest, Inc.

Tomorrow, Friday August 31st, between 8 AM and 10 AM our new UPS will be turned up. This procedure requires us to go into bypass mode on the entire UPS system. We will have the backup power generation system running. As always, any power transfer carries a risk of interruption, however we feel confident that this procedure will go smoothly and have no impact on service. We have tested the bypass system recently during our UPS preventative maintenance cycle, and the main power transfer switch is tested regularly and has also been triggered automatically by grid power failures a few times over the past year so we're confident in its operation.

We will post updates to this page prior to, and after the conclusion of the UPS turn-up.

Uptdate: 14:15 PDT

We have completed the UPS infrastructure work and are back to normal operations. Our power system performed as expected and there were no interuptions to any of our systems.

During this maintenance, our generator provided power for the datacenter for 3 hours and 44 minutes. This consumed approximately 150 gallons of the 5000 gallons of diesel fuel we have on site.

Our electrical feed is now backed-up by 4 independant strings of batteries and two parallel UPS modules. Output is 450 KVA or 360KW for 15 minutes (the generator supplies power to the UPS system in the first 15 to 20 seconds after a power failure from the public grid) of UPS runtime.

On Wednesday, September 5th we will be provisioning two new PDU's (Power Distribution Units). This activity has no potential for disruption and will simply increase the amount of power availabel to distribute in the datacenter.

During our scheduled maintenance window starting at 11:00 PM on Thursday, May 17th we will be resetting our connection with one of our network peers, NTT/America. As part of the post-mortem analysis of the last event on this circuit we discovered a way to improve the failover properties of the ethernet configuration. It will require up to 15 minutes of downtime while we reconfigure the router interfaces.

This should have no impact on uptime or network connectivity as we have other connections with other peers which will take the traffic during the maintenance window.

In observance of the Thanksgiving Holiday, digital.forest will be closed Thursday November 23rd and Friday November 24th. We will resume regular business hours at 8am PST Monday November 27th.

Technical Support staff will remain on-site 24 hours a day throughout the holiday period. Please be aware that we'll have limited staff coverage for telephone tech support during the next few days. Please note that our building will be locked throughout the weekend and clients requiring access to the datacenter to work on colocated servers will have to call or email first to be allowed access to the building. Additionally we will likely take advantage of the holiday lulls to perform maintenance and upgrades on core equipment. We will post notice of these beforehand. Finally, we are closing the phone support queue for a while on Wednesday, November 22nd in the afternoon for a Tech Department meeting. We appreciate your patience.

All of us at digital.forest wish you a happy Thanksgiving and a wonderful holiday season!

Chuck Goolsbee
VP, Tech Ops
digital.forest

During our scheduled maintenance window we will be applying Windows updates to our servers. These updates will require server reboots.

If your server is down, wait a few minutes and check again. If it is still down you may call our support line @ 1-206-838-1630 option 3 (toll free 1-877-720-0483)

Reboots will occur between 11:00 pm and 2:00 am

Update @ 2:04am on 05/06/2006: Both upgrades have been successfully completed

We will be performing a hardware upgrade (specifically a processor upgrade) on columbia (columbia.forest.net) and Ninewire's mail server (mail.ninewire.com) this coming Friday (05/05/2006). We anticipate the maintenance to last approximately 3 hours and it will begin at 10pm PDT and conclude at 1am PDT.

During this time streaming on Columbia will not be available & email service on mail.ninewire.com will also not be available.

This upgrade will increase the performance on Columbia & mail.Ninewire.com.

If you have any questions, feel free to open up a trouble ticket.

Update @ 6:07pm: thyme.forest.net has been updated successfully

This Wednesday, 05/03/2006, we will be updating FileMaker Server on thyme.forest.net from 8.0v2 to 8.0v3. All FileMaker Services, (IWP, XML, Remote) will be affected. The upgrade is scheduled to take place at 5:30 PM PDT and should last about 10 minutes.

FileMaker considers this a maintenance release and here are the change notes from 8.0v2:

1. New in FileMaker Server Advanced 8.0v3

1.1. Finds on an unstored calculation field

1.1.1. Fixed a problem that caused the entire record set to be returned if a client performed a find on an unstored calculation field if an 8.0v1 Web Publishing Engine was connected to an 8.0v2 Server.

1.1.2 Fixed a problem that could cause a FileMaker host to quit if a client performed a find on an unstored calculation field using either the Get(PrivilegeSetName) or Get(ExtendedPrivileges) functions.

1.1.3 Fixed a problem that caused unstored calculation finds to always be performed at the Web Publishing Engine which negatively affected performance. These finds are now performed appropriately at the Server or the WPE, which improves the performance of these types of finds.

Further information regarding this update is available on FileMaker's website.

Update @ 8:20am (05/03/2006): Update was delayed due to some minor problems, however it has now been completed and orchid is now on a new machine

Tonight, 05/02/2006, we will be moving all customers and databases on orchid.forest.net to new hardware. The scheduled maintenance will begin at 9:30pm and will conclude at 10pm PDT.

This maintenance will increase the performance for users on orchid.forest.net.

If you have any questions feel free to open up a trouble ticket.

Update @ 12:41am: Maintenance has been completed successfully on both banana & cactus.

Tomorrow night, Saturday April 8th, maintenance will be performed on banana.forest.net & cactus.forest.net. This will be the third, and final, maintenance performed on both of the machines. In the final maintenance we will be replacing the hard drives in both machines with faster and bigger drives.

The maintenance will begin at 9pm and will conclude at midnight.

If you have any questions or concerns feel free to open up a support ticket.

This has been completed.
________________________________
During the evening of the 29th we will be performing maintenance on a number of the core windows servers. Power, Radial, Chain, Skil, Jig, Sabre, Buzz, and Bard will each need to be rebooted and will unavailable for a few minutes. Thank you for your patience.

Tonight we will be upgrading memory on a few FileMaker 4/5/6 machines that are running Mac OS 9. The upgrades will take place between 12am and 2am.

List of machines:
dogwood.forest.net
yulan.forest.net
aspen.forest.net
sassafras.forest.net
plum.forest.net
savin.forest.net
boxwood.forest.net
hazel.forest.net
mimosa.forest.net

FileMaker service to the machines listed above will return to stable service after 2am PST.

Update @ 12:09am: Both machines are now online operating with the new processors (approximately 2.5 times faster then the old processor).

Tonight, Friday 03/24, Banana & Cactus will be taken offline at approximately 10PM PST. Both machines will be intermittently available for approximately 60-90 minutes while we update the processors in both machines.

We expect that the maintenance will be completed by midnight.

If you have any questions or concerns feel free to open up a trouble ticket.

Today we will be updating souari, butternut & silverpine's PHP to the most recent versions available in their respective version branch. The reason for this upgrade is to close a few known security vulnerabilities (primarily the XMLRPC code injection bug).

Butternut's PHP is being upgraded from 4.3.10 to 4.4.2
Souari's PHP is being upgraded from 4.3.10 to 4.4.2
Silverpine's PHP is being upgraded from 5.0.4 to 5.1.2

Feel free to open up a support ticket if you experience any problems related to PHP on your website.

Tuesday night, during our normal scheduled maintenance window, we will be performing a RAM upgrade on the QuickTime Streaming Server "nile.forest.net". The server will be down for about 10 minutes from approximately 23:00 PST. Streaming content will be unavailable during the maintenance period.

Update: Maintenance has been completed and tangerine is now running MySQL 4.1.18. If you experience any problems with your website and you are hosted on tangerine, either give us a call or open up a support ticket

Tonight we will be performing a software upgrade on tangerine.forest.net. We will be upgrading MySQL from version 4.0.24 to 4.1.18. A major compatbility problem between the two is the new password hash system, which we are disabling, thus this should only positively impact the customers using the MySQL software.

The maintenance will begin at 1 AM PST and will last until 2 AM PST. During this time the mysql service on tangerine.forest.net will spotty or completely unavailable.

Today I will be applying security updates (Apple Security Update 2006-001 & Security Update 2006-002) during the day today. The downtime for each machine will be less then 5 minutes. Below is a list of servers that will have the security patch installed.

alder.forest.net
banana.forest.net
cactus.forest.net
grapevine.forest.net
newninewire.forest.net
sage.forest.net
slime.forest.net
spruce.forest.net
tangerine.forest.net
date.forest.net
fungus.forest.net
helpdesk.forest.net
honeysuckle.forest.net
jasmine.forest.net
thyme.forest.net

Yvo

Tonight we will be performing memory upgrades on a few machines in our datacenter. The maintenance will begin promptly at 8:30 PM PST. The downtime will be 10 minutes for each machine.

Machines being serviced in order:

cactus.forest.net
cinnamon.forest.net
elm.forest.net
orchid.forest.net

If you have any questions, feel free to a support ticket.

Update: Upgrades on all three machines were successful

Tonight we will be performing a hardware maintenance on the machines listed below. The maintenance will begin at 8:30 PM PST and will last approximately 10 minutes per machine. The reason for this maintenance is to upgrade the memory on the machines to improve performance.

Machines being serviced:
banana.forest.net (Lasso 6)
jasmine.forest.net (FileMaker 7)
tangerine.foerst.net (MySQL 4)

Update: Upgrades on all three machines were successful

Tonight we will be performing a hardware maintenance on the machines listed below. The maintenance will begin at 8:30 PM PST and will last approximately 10 minutes per machine. The reason for this maintenance is to upgrade the memory on the machines to improve performance.

Machines being serviced:
banana.forest.net (Lasso 6)
jasmine.forest.net (FileMaker 7)
tangerine.foerst.net (MySQL 4)

During our scheduled maintenance window on Wednesday night, March 8th, we'll be upgrading the mail server software on the host "mail.ninewire.com". The updated software has better junk mail bounce handling and sending queue management features.

Downtime will be limited to 15 minutes or less. Work will begin shortly before midnight.

Update: Maintenance was successful and completed on time

Palmetto will be taken offline tonight at 8:30 PM PST for a quick memory upgrade. It should be back online by 8:50 PM PST.

Update: Maintenance was completely successfully at 8:50 PM PST

Spruce will be taken offline today at 8:30 PM PST for a scheduled maintenance. The maintenance will last approximately 30-45 minutes. Spruce should be back online at 9:30 PM PST.

Spruce will be receiving a massive CPU and processor upgrade, this will greatly improve the performance of Spruce.

Update: Maintenance was successful on Sage & Tangerine

Tonight Sage and Tangerine will be offline for approximately 15 minutes each starting at 8:30 PM PST. Sage will be the first to go down at 8:30 PM and then Tangerine shortly after. The entire maintenance should be concluded by 9:00 PM PST.

We will be upgrading the memory on both Sage & Tangerine in order to further improve performance on both machines.

On Friday (02/17/2006) we will be applying minor updates to the operating system on the machines below in order for our customers to receive the best possible service. The downtime will be approximately 15-20 minutes per machine, during this scheduled maintenance the service on the machine can be unstable and/or not available at all.

helpdesk.forest.net (Support system)
jasmine.forest.net (FileMaker Server 7)
date.forest.net (Lasso Professional 8.1)
honeysuckle.forest.net (Apache webhosting / MySQL 5)
grapevine.forest.net (Lasso Professional 7.1)
cactus.forest.net (Apache webhosting / NetCloak)

If you have any questions feel free to open up a support ticket.

Update: The maintenance will be pushed back to Tuesday, January 10th 2005 @ 10PM PST.

On Monday, January 9th 2005 at approx 10pm PST our Alder3 script located on alder.forest.net will be unavailable for use. During this time connectivity to alder.forest.net will be unavailable. Also connectivity to jasmine, sage and spruce maybe affected as well. The scheduled maintenance will take approximately one hour to complete and we anticipate that all services will be back to normal before midnight on Monday night.

Yvo V.
digital.forest technical support

Update:

We are currently experiencing issues with the Instant Web Publishing portion of FileMaker 7 Servers Sage.forest.net & Spruce.forest.net. Currently connecting via your FileMaker Pro client does not seem to show any problems. This in result of the upgrade. We estimate that we will have both servers fully up and running by the end of the morning.

YvoV
digital.forest technical support

Tonight we will be upgrading our Filemaker Server 7.0 solution to 7.0v4 from 7.0v3.

The servers affected are:
jasmine.forest.net
sage.forest.net
spruce.forest.net

The maintenance to our Filemaker 7.0 servers will take place between 1am and 2am PST. The anticipated downtime should be no more then 15 minutes for each server.

Tonight our online support system, helpdesk.forest.net, will be going offline at 10pm until midnight for scheduled maintenance. We will be working on improving the backend that powers Web Help Desk.

Please email support@forest.net or call us at 206-838-1630 during this scheduled downtime if you are in need of technical support assistance.

Tonight helpdesk.forest.net, our trouble ticketing system, will be unavailable from midnight to approximately 1am PST. It will be down for scheduled maintenance.

Thank you,

Yvo V.
digital.forest technical support.

Update: Maintenance was completed successfully at 12:36pm

During our scheduled maintenance window tonight we are performing some network configuration changes on various ethernet switches in the datacenter. There should be virtually no effect on network traffic beyond a few second pause as we update each switch.

Maintenance will take place between midnight and 1 AM PDT.

Update: this is complete as of 22.10hrs

Starting at 21.00 hours on Wednesday, September 21 these servers:

buzz.forest.net
power.forest.net

will be rebooted to apply current MicroSoft security patches. Two reboots will be needed, and downtime is expected to last about fime miuntes each.

Starting at 11:00 pm on Wednesday, September 14th we will be rebooting the following servers one at a time:

ads, findyourbet, calypso, wizard, mars, phantasy, celestial, callisto, titan, phobos, mimas, sol, mercury, neptune, gold, celtscript and dotcomdb.

Each server will be down for about 5 minutes.

Tonight around 10 pm we need to perform some maintenance on the mail server "treehouse.forest.net". It could be down for up to 90 minutes, though we will do what we can to minimize that.

This work completes the maintenance we started back on August 11, and should have some positive result on overall server performance.

Starting at 12:30 am on Saturday, August 20th we will be rebooting the following servers one at a time:

ads, findyourbet, calypso, wizard, mars, phantasy, celestial, callisto, titan, phobos, mimas and sol

Each server will be down for about 5 minutes.

At the urgent request of a number of important digital.forest clients in Europe we have been asked to move our mail server downtime to earlier in the night so as not to overlap their working day. It has been rescheduled to start at 10 PM PDT and be completed by midnight.

UPDATE: Work was begun around 10:10 PM PDT, and completed just after 11. Total downtime was just under one hour. Thanks for your patience.

Tonight at midnight PDT we will be shutting down one of our mail servers "treehouse.forest.net" for maintenance. Downtime could be up to 2.5 hours.

We are taking this step to mitigate or remove the issues that have been affecting performance of this mailserver over the past week (see post below). We have been working very closely with the mail server software vendor Communigate Systems to analyze the issues and investigate possibilities for solving them. Unfortunately doing these operations on a "live" production server during the business day is impossible. We believe that this downtime will be long enough to get them all done at once.

Thanks for your patience.

helpdesk.forest.net will once again be unavailable for the next 10-15 minutes while we perform a brief software update. Thank you!

helpdesk.forest.net will be unavailable for the next 10-15 minutes while we perform a brief software update. Thank you!

During this week's scheduled maintenance window we will be performing the following tasks:

1. Relocating power for Row 5, Server Room 2.
This row contains G3 and older systems in 8 racks. This move completes the electrical work we began last week. Server downtime will be limited to 5 minutes maximum. This work will start around 8 PM PST and be complete by 8:45 PM PST.

2. Replacing some ethernet switches in Server Room 2.
These will be located in the 216.168.61.0/24 and 216.168.63.0/24 colocation subnets. Offline time will be limited to a few seconds per server as we move ethernet cables from the old switch to the new one. This work will begin around 7 PM PST and complete about midnight.

Our electrician is installing some new wiring in the datacenter today for our new row of racks. This work will require the brief shutdown of the two rows immediately adjacent in Server Room 2. One row is home to our oldest MacOS-based computers ("BandW G3's and older "beige" units.) The other is home to the servers we acquired last summer from FMPHosting.net. At noon PST we will perform an orderly shutdown/reboot of the Mac servers to accommodate the work. We will delay the FMPHosting.net reboot until after 2PM PST (to match 5 PM EST)

NO OTHER servers will be affected.

Thanks for your patience.

helpdesk.forest.net is down as part of maintenance/updates that we are performing on the server. The help desk system will be available again when the work is complete.

We are taking advantage of the quiet time over the holiday to perform some network maintenance. Mostly this is in the form of cable cleanup and tie-down. It will have no affect on the network performance beyond an occasional reboot of a switch, which should take no longer than 5 seconds.

Our offices and technical support are closed for the day - as always, emergency support is available via normal channels.

Happy New Year.

Tonight during our scheduled maintenance window we will be performing an upgrade (adding disk space) to skil. Expect downtime of about 15 minutes, starting at 5:30 PM PST.

September 17th, 9pm through 12:00 AM

During our normal scheduled maintenance window (Wednesday night,) we will be migrating several hosting & colocation subnets to newer, better ethernet switches in Server Room1. The subnets affected are:

216.168.37.0/24
216.168.47.0/24
216.168.60.0/24

All servers in these subnets will experience very brief outages (less than 30 seconds) as they are patched into the new switches.

No configuration changes will be required.
No servers in Server Room 2 will be affected.

During the normal scheduled maintenance window tonight just after midnight (PDT) we will be performing the following procedures:

1. Swapping an Ethernet card on our boundary router.
This serves the 204.142.69.0/23 network that includes our newly acquired clients from Exceedia Inc. Users on this network will experience a brief service interruption, lasting approximately 15 seconds.

2. Installing a new switches in the 216.168.37.0/24, 216.168.58.0/24 & 216.168.62.0/24 networks. Users on these networks will experience a brief service interruption, lasting approximately 30 seconds, as each host is moved from the old switches to the new.

During our normal scheduled maintenance period this coming Wednesday night/Thursday morning we will be adding a memory upgrade to our boundary router. Clients will experience a downtime of approximately 3 to 5 minutes. The maintainance is scheduled for 12:01 AM (Midnight) Thursday March 20th.

Update: Upgrade completed.

One of our upstream physical layer (Fiber) providers will be performing a network equipment upgrade between Midnight and 6 AM on both January 24th & 25th. The expected duration of our downtime on that link will be a maximum of 20 seconds. Our other (OC-12 SONET) connection will remain up throughout, so connectivity will not cease completely, though we should see a short drop on two of our upstream connections.

Our Web Help Desk/Trouble Ticketing system will go down for maintenance today at 5:30 pm, and is expected to last up to twenty minutes. All tickets will be saved during the update.

cactus.forest.net will be temporarily offline for an operating system upgrade at 2:00 PM PT on November 10th, 2002. The upgrade will take approximately 1.5 hours, and will result in faster web serving and file access.

www.forest.net will be temporarily offline for operating system maintenance and upgrades. We expect a little under an hour of total downtime. This will affect the beta reboot system temporarily.

Please be advised that we are required to perform emergency maintenance on Monday, July 8th between 02:00-04:00 PDT. This is required to reload our Core Routers in Vancouver. You may notice up to three 45 second interruptions. These interruptions will occur sometime within the maintenance window. If you have received this email then you may be affected by this maintenance. We apologize for the short notice, and thank you for your understanding in this matter.

Due to a hardware failure on sabre.forest.net, we will need to take the server down between 7:30 and 7:45 pm, Pacific Time. A memory failure has been causing the machine to freeze at random intervals, requiring a manual reboot. We have replacement RAM available, and will be taking care of the replacement tonight. Thank you for your patience in this matter.

On Monday, June 10th at 6pm, we will be moving the filesystem of whitebeam.forest.net to a newer and faster machine. Expected downtime will be about 1 hour while the data is moved from the old to the new server. Clients will not be required to make any changes, as the server IP address and configuration will be identical.

On Sunday, April 14th at 6pm, we will be moving the filesystem of chain.forest.net to a newer and faster machine. Exected downtime will be about 1 hour while the data is moved from the old to the new server. Clients will not be required to make any changes, as the server IP address and configuration will be identical.

Logs on chain will not be availble for up to a day after this switch.

On Wednesday, April 3rd, we will be moving the www.infoasis.com web server to our data center in Bothell, WA. Due to the unique configuration of this server, this move will require physically shipping the box from the old location; therefore, the server will be down from late afternoon on Wednesday, April 3rd to late morning on Thursday, April 4th.

We realize this downtime will be inconvenient for you. However, once the move is complete, the server will be hosted on a much faster network with multiple high speed backbone connections; network performance should improve dramatically, and the bandwidth that server is currently consuming in the Marin County facility will become available to our Internet access customers, improving performance for them, as well. We will strive to complete this move as quickly as possible, and thank you for your patience.

The Datacenter in Bothell, WA will be getting a floor makeover on Wednesday, March 27th 2002. The floor will be washed and buffed and waxed. This will inhibit colocation customers from being able to get physical access to their machines, aswell as increased response times for any server reboot. Thanks for your patience in this matter.

Our network engineer will be working to segment the network into smaller subnets which will enhance the efficiency of our network as we move towards the lighting up our new OnFiber connection.

The 63 netblock will be physically separated onto its own VLAN.

Clients hosted on the 61 and 63 netblocks, ie. your ip address starts with 216.168.61.** or 216.168.63.**, will notice an increased amount of latency during the switchover. Network outages should not last more than a couple minutes.

No changes will need to be made on any customer hosting servers.

We will doing emergency maintenance on Wednesday, March 20th, 2002 at 4:30 pm Pacific Time. Users on this machine have experienced extended downtime and outages, much more than should be expected for this type of machine. We will be replacing the system hard drive and reorganizing the data drive for improved speed and reliability. This maintenance is imperative for the uptime of the machine. We are expecting a total downtime of 20-25 minutes. We apologize for this inconvenience, but we hope that this will keep this server up and running for a while to come.

We will doing emergency maintenance on Wednesday, March 20th, 2002 at 4:30 pm Pacific Time. Users on this machine have experienced extended downtime and outages, much more than should be expected for this type of machine. We will be replacing the system hard drive and reorganizing the data drive for improved speed and reliability. This maintenance is imperative for the uptime of the machine. We are expecting a total downtime of 20-25 minutes. We apologize for this inconvenience, but we hope that this will keep this server up and running for a while to come.

Our network engineer will be working to segment the network into smaller subnets which will enhance the efficiency of our network as we move towards the lighting up our new OnFiber connection.

The 47 netblock will be physically separated onto its own VLAN.

Clients hosted on the 47 netblock, ie. your ip address starts with 216.168.47.** will notice an increased amount of latency during the switchover. Network outages should not last more than a couple minutes.

No changes will need to be made on any customer hosting servers.

Our network engineer will be working to segment the network into smaller subnets which will enhance the efficiency of our network as we move towards the lighting up our new OnFiber connection. The first step in this process will be creating two new networks and assigning new subnet masks to specific network infrastructure components.

The new networks will be:
216.168.46.1-127 GW .1 SM 255.255.255.128
216.168.46.129-254 GW .129 Sm 255.255.255.128

Oak will have to have a new subnet mask in it's TCP/IP stack and will have to be reloaded.

Border001 will have to have a new subnet mask applied.

Branch will have to have a new subnet mask applied.

Overall traffic and customers will see a drop in packets for < 30 seconds.

When oak (Primary DNS Server) is rebooting, customers without willow as a secondary will see a loss of traffic until oak comes back up.

No changes will need to be made on any customer hosting servers.

A message from one of our upstream providers:

On Saturday, February 23, 2002, between 1am and 7am local time to the site, Genuity Back Bone Engineers will reload multiple routers across Genuity's backbone to the most current IOS. Customers homed to the following POPS may see up to 10 minutes of degraded service as backbone routers are reloaded:

atlnga1, chcgil1, cleveland1, crtntx1, denver, evrtwa1, iplvin1, lsanca2, nycmny1, oakland, ordcolo, paloalto, phlapa1, snjpca1, tamqfl1

digital.forest's Bothell DataCenter is on the evrtwa1 loop and thus we will see increased latency during this time frame.

At 11pm PST on Saturday, February 9, we will be moving the infoasis mail server to our network in Bothell, WA. No mail will be lost; the secondary mail server will hold the mail until the primary comes back up. However, during the move, your mail will be unavailable. Once the change is complete, your mail service should work as before; you will not need to make any changes to your settings. We expect the downtime to last 2-4 hours.