Update on Latest Hacker Attack
This month we have been under intruder attacks on our primary server - logos. Last Sabbath at about 2 pm Mountain time, the hackers downed the server and corrupted the drive to the point we could not get it to reboot. (Data still appears to be okay.) When I have more time I may write a more detailed report on what happened and the events that led up to this.
One of the first things many ask when they become aware of our hacker problem is who are they? The answer is that we don’t know. Right now our efforts have been concentrated on getting the server back up and online.. with all the services running (web, email, real server, online databases, etc.).
Next people want to know how this happens. We have been running the same version of the Operating System(OS) from when we put logos online in December of 1999 - Red Hat Linux version 7.1. This OS is old by computer standards.. and hackers have identified many ways to break into it. A couple of years ago we installed a High Point Technology RAID controller card so we could have redundant drives. This card has presented challenges in updating the operating system. Also the fact that we’ve had so many sites and programs running on this server have detered us from taking the system off-line to upgrade. The other issues are of course the time and funding needed to do the upgrades. I am not a Linux technician.. and so CGCA has to hire tech staff to make these type fixes. Even with techs charging us reduced rates, these costs are expensive.
So what are we doing to resolve this? Forest Leonard, nephew of co-founder Allen Hirst, has donated a copy of Red Hat Enterprise LINUX Advanced Server version 3.0. In addition to providing the software, Forest has also been helping us get the server back up. Although not a member of a COG, Forest has been working at a much reduced rate and has spent nearly 40 hours this month in battling the hackers and rebuilding the server.
Last Sunday we began the process of installing the new software. We ran into compatiblility issues with the HT RAID card. It has been pulled from the system. We also had the power supply go out, a CPU fan fail, and later this week, the ethernet NIC card died. It seems we have an Adversary that would like to keep our server off-line!
The new OS has been installed on a separate drive. We currently have mounted all the old web page data and are in the process of getting the new web server configured. Immediately following the web server, we will be configuring the sendmail program. This site is the first to be online with others following soon. Even though we have the old configuration files to look at, this is taking time to re-do because the new versions (of all the programs) use new configuration files.
By upgrading to this new OS we hope to have plugged the security holes that were allowing hackers to get into our system. Red Hat also will support this OS for 5 years.
What is next?
1.We have to re-install Real Server and get all congregations setup again for using it to do cybercasts this coming Sabbath. (We will be installing the new Helix version.)
2. We will need to re-enter all users and create new passwords.
3. We need to get the MySQL database reconfigured for all sites using it.
4. In moving data, ownership of files has been changed to root ownership- so we will need to change all ownership back to the correct users so they can update their pages, etc.
5. I am sure there will be many other issues to resolve as they arise.
How can you help?
Although Allen Hirst and I have been donating our time (for me nearly full time this entire week), we now have obligations to pay the technicians who have been helping us. We have also had to puchase a new 430 watt power supply, Network Interface Card, and CPU fan. These expenses are severely heavy on our fragile budget. (In addition to covering our regular hosting expeneses of $250 per month.) Your financial donations at this time would be most appreciated!
We would also appreciate your prayers that God would continue to:
We have a two young volunteers - one in general Linux admin, the other in PHP/MySQL admin, but their time availability is limited due to being in school and college. ..so we could use help in that area, too.
We appreciate your support, patience, and prayers.. and are working as hard and fast as we can to get all services fully restored as soon as possible.
Dan Deininger (temporay email= firstname.lastname@example.org)
PS - I also wish to publically thank my wife, Val, for her support during this week.. and the wives of other techs who have been working into the wee hours of the morning. Their support and patience has also been VERY much appreciated!!!
Update for individuals and congregations using our server:
Late on Saturday, August 30th one of our servers was compromised by a hacker. Index files (front page or top pages of the sites hosted on that server) were replaced with a different index.html or htm page.
In addition, we are in the process of installing Operating systems patches to prevent this happening again.
Financial contributions may be mailed to: