Wednesday, March 30, 2016

Planned maintenance April 16, 2016 two five-minute outages possible

Our network provider has informed us that they will be updating firmware on network devices that may result in one or two service interruptions of no more than five minutes duration for web sites hosted on Plesk. The maintenance window will be April 16, 2016 beginning at 11:00 p.m. ET and ending six hours later. Web sites hosted on "beta" servers are served by a different data center and are not impacted.

Friday, February 13, 2015

Transient networking issues

[09:46 a.m. ET]
We are seeing transient networking issues on our Plesk web server.   We are trying to isolate the problem and report back as soon as we know more.

[10:10 a.m. ET]
We dispatched a technician to the data center.  More information should be available in 30-40 minutes.

[10:10 a.m. ET]
Network now appears to be stable. 

Friday, January 23, 2015

Some web sites are not available


[1/23/2015 4:00 p.m. ET]
Due to a configuration error at Network Solutions, some igc.org web sites are not working.
We are working with Network Solutions to resolve the issue.

[1/23/2015 5:05 p.m. ET]
Network Solutions is working on the problem.  No ETA other than "this evening".

[1/23/2015 7:3o p.m. ET]
Network Solutions reports that the problem has been fixed and web sites should start working again in 15 minutes.

Thursday, January 22, 2015

Intermittent connectivity issues this morning

Some customers may have experienced intermittent connectivity issues this morning between 10:30 a.m. and noon EST. Our engineers have determined that an issue within the network of one of our upstream peers is likely the cause. At this time we believe the issue has been resolved.

Sunday, September 28, 2014

Bash Shellshock

We have updated all publicly facing servers for the bash vulnerability described below.  Our service providers have also confirmed that their servers have been updated.

 --Bash Shellshock Flaw
A serious flaw in a software component called Bash is said to be more
serious that the Heartbleed vulnerability that was disclosed earlier
this year. The flaw, which is being called Shellshock, can be exploited
to remotely take control of vulnerable systems. It affects an estimated
500 million UNIX and LINUX machines. Bash, or the GNU Bourne Again
Shell, is a command prompt on many Unix systems. The US Computer
Emergency Response Team (US-CERT) has issued a warning and is urging
admins to patch the flaw. Others have expressed concern that the patches
that have been made available are incomplete.

http://www.bbc.com/news/technology-29361794

Saturday, August 30, 2014

SQL errors

A few web sites running on the non-Plesk server are experiencing problems due to a denial of service attack aimed at the SQL server.  We are working to mitigate this attack now and expect to have that server back to normal shortly.

Update: 7:45 a.m. PST
All services on the non-Plesk server has been restored.

Thursday, June 5, 2014

Services are down

All services went off-line at 1:11 p.m. ET today.   A technician has been dispatched to the data center to investigate.  We hope to provide an update in 60 minutes.

Update: [Thu Jun  5 13:49:51 EDT 2014] Service has been restored as of 1:28 ET. 

Friday, April 11, 2014

Mail from IGC to Earthlink (igc.org) Blocked

Earthlink is currently blocking email from IGC.   This will impact you as an IGC user if you have an igc.org email account and if your domain and email is hosted at IGC.  In other words if we forward mail sent to your domain at IGC to any igc.org email account at Earthlink.
--
Update: Fri, 11 Apr 2014 07:31:51 -0400
Earthlink reports that they have removed the block on our mail server (208.40.195.230) and that mail should return to normal soon.

Wednesday, March 12, 2014

Two minute scheduled maintenance outage: 03-15-2014 11:00 PM and 03-16-2014 2:00 AM

Please find the details of a scheduled maintenance event below.

Change Control #: 40276
Description: Emergency Fiber Network Infrastructure Upgrade
Location: All Markets
Date / Time: 03-15-2014 11:00 PM - 03-16-2014 2:00 AM

Maintenance Details: This event will require an upgrade to the configuration of a core switch serving your location to improve redundancy on the fiber ring.

Customer Impact: Customers may experience up to two (2) service interruptions of no more than one (1) minute in duration during the scheduled time frame.

Thursday, February 20, 2014

Mail delivery issues

Update at 2/20/2014 8:56 ET
A few customers have reported problems with email delivery.  We are currently investigating.

Update at 2/20/2014 9:36 ET
We have determined that mail is being delayed due to a massive spam attack on our mail servers. We have increased the capacity of the mail server in question and are looking at other ways to mitigate the damage. Normally these attacks are aimed at a single domain and subside after all standard usernames have been attempted. We are unable to block these attacks by IP address as a botnet with thousands of unique IP addresses is being used to attempt the mail delivery. 

Sunday, January 19, 2014

Network Outage 3:20 a.m. to 4:08 a.m. ET

Our service provider had a 48 minute outage early this morning due to the failure of a core networking device. All services have been restored to normal. 

Thursday, January 9, 2014

Server outage at 10:10 a.m. ET 1/9/2014


We are currently experiencing an issue with servers, email, and mailing lists. The event started at 10:10 a.m. this morning January 9, 2014. Our technical staff is investigating.

Web sites came back on line at 12:39 p.m. ET.   Report on root cause of the failure to follow.

Update 2:43 p.m 1/9/2014: Outage was caused by a disk error on the disk storing mail files.  We are moving the mail backups to a new disk and will get mail back on line ASAP.  If you sent a message to support at IGC, we won't be able to answer until mail comes back on line.

Update 7:19 p.m. 1/9/2014: We have installed a new disk for the failed mail disk and we have restored all mail to the new disk.  Mail that was sent to us during the outage should have been stored on the sending system and is now being delivered. It may take 2-4 hours for our mail queue to get caught up.

Friday, December 27, 2013

Service Outage: Saturday, January 18, 2014 from 11 p.m. to 5:00 a.m. ET

Our Pittsburgh data center sends the following warning about a brief outage to do firmware upgrades.
This may impact customers with IGC standard hosting service and envirolink.org list service. It will not impact igc.org list service or web sites running on the new beta web server.


Change Control #: 37980
Description: Data Center Network Infrastructure Maintenance
Date / Time: Saturday, January 18, 2014 from 11 p.m. to 5:00 a.m. ET.

Maintenance Details: We will be updating the firmware on devices serving your data center services at the location listed above in support of future growth.

Customer Impact: Up to two (2) service interruptions of no more than five (5) minutes in duration, as well as one (1) service interruption of no more than ten (10) minutes in duration, are anticipated during this maintenance event. The actual impact should not exceed the scheduled time frame. 

Wednesday, November 13, 2013

Problems with Time Warner Cable network

Our hosting provider just provided the following alert regarding Time Warner Cable addresses:

We have received reports of problems experienced by multiple customers communicating with source addresses across the Time Warner Cable network. This may affect your staff and potentially end-users. We are in communication with Time Warner Cable to obtain additional information, however, the problem appears to be affecting a large segment of their network.

Monday, September 30, 2013

Brief outage on non-Plesk web server


We had a brief outage this morning on the web server that provides service for a handful of sites running Wordpress and Drupal.  It appears that an attacker caused a virtual machine to hang. We have reset the machine and are looking into how to defend against this type of issue going forward. 

Friday, August 2, 2013

apc.org under denial of service attack

 Our APC partner network in the UK, GreenNet,  has been under a denial of service attack for the last few days.  

This impacts the www.apc.org and other sites that use apc.org for domain name service.

This may delay mail addressed to support@igc.apc.org    If you wrote us for support and did not hear back, write again to support@igc.org

We are working with GreenNet to mitigate the attack and restore full service.  This may take a few days as GreenNet works with their providers to restore service.

Monday, March 11, 2013

Brief mail outage


We had a brief mailbox outage last night while we moved mailboxes off of a disk drive that was starting to report errors.   All mail was safely restored to disk and any mail that we refused during the brief outage was accepted immediately thereafter.

A few of you have experienced problems with FTP connections due to the load this bad disk was creating. That issue should be resolved now as well.   

Monday, October 29, 2012

Severe Weather Response - Hurricane Sandy


We are sharing the following message sent by the provider where we co-locate our main web servers.  Should the facility experience any damage that brings down our services, please watch here for updates. 

Charlie Rosenberg
IGC Supoprt
--

Dear Valued Customer,
We take our responsibility as your trusted service provider very seriously, and as part of that duty, we have contingency plans in place whenever severe weather events have the potential to impact our Data Center facilities. At this time, we are monitoring Hurricane Sandy, which currently has a projected path near our facilities located in Pittsburgh, PA.

We will of course keep you informed if, as a result of this severe storm, issues arise which may affect these facilities and/or your service. Many times during these events, availability of commercial utility power sources are interrupted or degraded for extended periods of time. We anticipate that our redundant, back up power systems will perform as expected and carry the critical power loads if interruptions occur. However, every situation is different and therefore, in advance of the storm, we are taking additional measures beyond our normal operating schedule:

Our staff is completing a full review of recent maintenance activities and system inspections of all Emergency Power Systems.

Our staff facility engineers will be on site during the storm and remain there until the storm has cleared and all commercial power systems are operating normally.

Additional Operations Support Personnel will be on site during the storm to aide in response if necessary. 

Engineers from our various facility vendors and partners will either be on site during the storm or in emergency response mode close by. 

Additional Operations Support Personnel and equipment from our non-storm-impacted markets will be available to aid in Service response if needed. 

Additional emergency materials have been purchased and will be on site; such as diesel fuel to replenish generators on long run time and food for our on site staff and guests.

Tuesday, September 18, 2012

Planned maintenance 10/6/2012


Date / Time: 10-06-2012 11:30 PM - 10-07-2012 1:30 AM ET


The facility where we keep our servers will be increasing network capacity in support of future growth.


They have reserved 10/20/2012 as an alternate date in the event that this maintenance would need to be postponed.


Customer Impact: Up to four (4) service interruptions of no more than three (3) minutes in duration, as well as one (1) service interruption of no more than twenty (20) minutes in duration are anticipated during this maintenance event. The actual impact should not exceed the scheduled time frame.

Wednesday, July 11, 2012

Plesk update

We have turned off the Plesk control panel in order to install security patches.  We had intended to have the patches installed by today but after testing, we have determined that the patches are not effective.  We are upgrading to a newer version of Plesk to resolve the issue. This is a significant update and it's not clear yet how many days it will take to complete. We apologize for the inconvenience.

Update (August 8):
We initially applied the patches to version 8 of Plesk and found that the patches did prevent the intrusions. We next installed Plesk 11 (the newest version) and tried to import the user data.  After several attempts and limited help from the vendor, we decided to install Plesk 10 instead. While not the latest version, it is thought to be secure. We are working on Plesk 10 install now and will then need some time to verify the imported user data. We will post again when we have more news to report.

Update (September 9):
We now have a work-around that provides access to the control panel.  Please contact support for details. 

Friday, October 28, 2011

Welcome to mailing list messages

A few folks have reported that they got a message welcoming them to a mailing list that they had been subscribed to for some time. It looks like this was caused by some errant code provided to us by a vendor that we were using to upgrade our mailing list server. Next time we will know to turn off mail notifications before we start the upgrade script. Sorry!

Wednesday, October 12, 2011

Web servers are down

Service has been restored. Downtime was limited to one hour and 48 minutes. A process that deletes log files died and allowed a disk to fill up. We have restarted the process and cleaned up the log files. We will now attempt to determine why the process died and how we might monitor it more closer in the future. We apologize for the inconvenience.

Web servers are down

On 9/12/2011 at 6:53 a.m., our web servers stopped responding. We are looking into this now and once we have more information, we will report back here.

Thursday, June 16, 2011

Slowness this morning between 10:10 a.m. and 10:50 a.m. ET

The following notice was sent to us today from our colocation provider in Pittsburgh.
If you noticed that mail or web service was slow this morning between 10:10 a.m. and 10:50 a.m. ET, it was probably due to this.

--
Some customers with traffic routed through Pittsburgh, PA may have noticed a period of network latency and/or packet loss between approximately 10:10am and 10:50am.

Engineers isolated the root cause to an inbound distributed denial of service (DDOS) attack targeted at a connected customer network in the area. The malicious traffic was blocked and network service returned to normal at 10:50am.

Expedient maintains multiple upstream peering relationships in all markets so you may not have been impacted by this event, but we apologize for the inconvenience if you were.

Thursday, October 7, 2010

Mail to Earthlink (igc.org) is delayed

IGC technicians held a conference call with Earthlink technicians today and Earthlink has acknowledged the issue and are working actively to resolve it.

Wednesday, October 6, 2010

Mail to Earthlink (igc.org) is delayed

For a brief period of time today, after email contact with Earthlink technicians, email began to flow again but was then cut off. We are waiting to hear back from Earthlink in regards to this matter. All email is being stored on IGC servers and we should be able to deliver it when Earthlink begins to accept mail from us again. We apologize for the inconvenience

Friday, October 1, 2010

Mail to Earthlink (igc.org) is delayed

Starting sometime early this morning, Earthlink began refusing email from IGC's mail server. This impacts anyone who has an igc.org account hosted at Earthlink and a web site and a domain hosted at IGC as we are unable to forward that mail to Earthlink.

This also impacts anyone with a mailing list on the lists.envirolink.org server or anyone trying to send mail to Earthlink from an IGC webmail account.

One of our users has already spoken to Earthlink about this problem but so far they have refused to acknowledge that they are blocking mail from us.

We are attempting to reach higher-level technical staff at Earthlink to respond to this problem.

Wednesday, August 4, 2010

All systems are operating normally

All systems are operating normally.

Friday, January 15, 2010

problems with lists.igc.org server

Lists hosted on the lists.igc.org server are not currently functioning. A power failure at the facility in San Francisico where this server is located seems to have damaged this server. We are working to get it back on line on new hardware and hope to have that work completed today. The server hosting Envirolink lists and servers hosting web sites are functioning normally.

Sunday, January 10, 2010

The control panel page was not answering. This has been fixed and all other systems are reporting normal.

Sunday, January 3, 2010

All systems are normal.