Update on Latest Hacker Attack
April 22, 2004

This month we have been under intruder attacks on our primary server - logos.  Last Sabbath at about 2 pm Mountain time, the hackers downed the server and corrupted the drive to the point we could not get it to reboot. (Data still appears to be okay.) When I have more time I may write a more detailed report on what happened and the events that led up to this.

One of the first things many ask when they become aware of our hacker problem is who are they? The answer is that we don’t know.  Right now our efforts have been concentrated on getting the server back up and online.. with all the services running (web, email, real server, online databases, etc.).

Next people want to know how this happens. We have been running the same version of the Operating System(OS) from when we put logos online in December of 1999 - Red Hat Linux version 7.1. This OS is old by computer standards.. and hackers have identified many ways to break into it. A couple of years ago we installed a High Point Technology RAID controller card so we could have redundant drives. This card has presented challenges in updating the operating system.  Also the fact that we’ve had so many sites and programs running on this server have detered us from taking the system off-line to upgrade. The other issues are of course the time and funding needed to do the upgrades. I am not a Linux technician.. and so CGCA has to hire tech staff to make these type fixes. Even with techs charging us reduced rates, these costs are expensive.

So what are we doing to resolve this? Forest Leonard, nephew of co-founder Allen Hirst, has donated a copy of Red Hat Enterprise LINUX Advanced Server version 3.0. In addition to providing the software, Forest has also been helping us get the server back up. Although not a member of a COG, Forest has been working at a much reduced rate and has spent nearly 40 hours this month in battling the hackers and rebuilding the server.

Last Sunday we began the process of installing the new software. We ran into compatiblility issues with the HT RAID card. It has been pulled from the system. We also had the power supply go out, a CPU fan fail, and later this week, the ethernet NIC card died. It seems we have an Adversary that would like to keep our server off-line!

The new OS has been installed on a separate drive. We currently have mounted all the old web page data and are in the process of getting the new web server configured. Immediately following the web server, we will be configuring the sendmail program. This site is the first to be online with others following soon. Even though we have the old configuration files to look at, this is taking time to re-do because the new versions (of all the programs) use new configuration files.

By upgrading to this new OS we hope to have plugged  the security holes that were allowing hackers to get into our system. Red Hat also will support this OS for 5 years.

What is next?

1.We have to re-install Real Server and get all congregations setup again for using it to do cybercasts this coming Sabbath. (We will be installing the new Helix version.)

2. We will need to re-enter all users and create new passwords.

3. We need to get the MySQL database reconfigured for all sites using it.

4. In moving data, ownership of files has been changed to root ownership- so we will need to change all ownership back to the correct users so they can update their pages, etc.

5. I am sure there will be many other issues to resolve as they arise.

How can you help?

Although Allen Hirst and I have been donating our time (for me nearly full time this entire week), we now have obligations to pay the technicians who have been helping us. We have also had to puchase a new 430 watt power supply, Network Interface Card, and CPU fan. These expenses are severely heavy on our fragile budget. (In addition to covering our regular hosting expeneses of $250 per month.) Your financial donations at this time would be most appreciated! 

We would also appreciate your prayers that God would continue to:
- give all of the individual techs working on this additional skills and insights to solve the problems as they arise.
- help us fully resolve these technical issues so we have a safe, secure, trouble free server.
- provide me with favor with my clients as I take time off to solve these issues.
- send us additional technical support volunteers to help us with maintaining the site and getting it back online now... Specifically we could use help in the following areas:
-- DNS issues
-- SendMail
-- Apache Webserver

We have a two young volunteers - one in general Linux admin, the other in PHP/MySQL admin, but their time availability is limited due to being in school and college. ..so we could use help in that area, too.

We appreciate your support, patience, and prayers.. and are working as hard and fast as we can to get all services fully restored as soon as possible.

Dan Deininger  (temporay email= dand@432lastchance.com)
Helena

PS - I also wish to publically thank my wife, Val, for her support during this week.. and the wives of other techs who have been working into the wee hours of the morning. Their support and patience has also been VERY much appreciated!!!

Update for individuals and congregations using our server:

On Friday, January 9th we dedected a hacker on our main web server. While trying to eliminate him/her the system went down* and we had to install a mirrored backup drive. (The backup was made December 25th, 2003.) Our security technician detected 3 hacker ?root kits? on the server and is working to remove them.

We apologize for the inconvenience this causes for our users who will need to re-upload files that have been updated since December 25th.

We encourage all users to maintain backups of ALL files and databases on our servers.

If you are experiencing problems with either your site or your email account please contact either Dan Deininger or Allen Hirst either via email or by phone at the following numbers at the bottom of this page.

Thank you for your patience and understanding in this matter.

Dan Deininger and Allen Hirst, Co Founders of CGCA.

*NOTE: We feel that the hacker had booby-trapped the system causing certain files to be eliminated and crashing our server in the event he was detected and we tried to remove him.
 
 
 
 Upgrade and Security Plans:

Since the hacker attacked our system the end of August we have been more diligent in our backups.. both on ?logos? and on ?storm?. We are currently working on developing a network archetecture that will provide us more redundancy and security.  During the UCG Northwest Weekend Allen and I were able to spend some time talking to a few computer professionals (Eric Johnson and Chip Chuprinko) to begin formulating a plan to upgrade our servers and the security. I am also going to be meeting with our techs here in Montana to get their input.

On the security side, we have been told to expect a donation of Symantec Enterprise Firewall version 7.0 - An extremely powerful security management, cluster, and load balancing tool with ?heart-beat? redundancy/fail safe features. We view this as a tremendous blessing! (Retail value is approximately $17,000). This will require at least one or preferably two dedicated firewall computers in our network.

 We will provide more details here as they become available. We know it is going to take both time and money - but are confident that if we make our needs known there are many individuals and organizations that will donate the funds to help us move forward.

NOTE: If you are a computer professional with networking experience and would like to provide input, please contact us! We would appreciate your comments, suggestions, etc!

We are also looking at upgrading from two basic REAL servers used for cybercasting to one larger one with a much increased capacity. (from 120 audio streams to 500) We will announce that here once we know if the funding comes through.

We certainly appreciate the support of all in our collective efforts to use the internet to help do the work of God!
 
 

Late on Saturday, August 30th one of our servers was compromised by a hacker. Index files (front page or top pages of the sites hosted on that server) were replaced with a different index.html or htm page.

To counteract this intrusion we have changed ALL passwords on that server. This will affect many of our client's email accounts and their ability to log into the server to update web sites. You will need to contact us for your new password.

In addition, we are in the process of installing Operating systems patches to prevent this happening again.

We apologize for the inconvenience this has caused.

If you are experienceing problems with either your site or your email account please contact either Dan Deininger or Allen Hirst either via email or by phone at the following numbers:

Thank you for your patience and understanding in this matter.

Dan Deininger and Allen Hirst, Co Founder of CGCA.
 

Contact information:

Dan Deininger   406-495-9291 or 406-410-0162 (cell) Email: dand@inovativ.com
Allen Hirst   509-489-7479 Email: rahirst@cgca.net

Financial contributions may be mailed to:
Churches of God Cyber Auxiliary
P.O. Box 10044
Spokane, WA 99209-0044