Today's Outage

[isdr]

I received an email from the hosting service telling me that the server was sending out over 6 megabytes per second. I checked and learned it had been on going for over 4 hours for a total of 85 or so gigabytes. The server is only allowed 200 gigabytes per month.

I did some detailed analysis and learned that the errant process was due to a copy of phpMyAdmin installed on the UCC website. I suspect this is the second time I've seen this process using all the bandwidth, though this is the first time I've been able to prove it.

If anyone knows anything about this application / process, please let me know via email at [email protected]. Due to the amount of bandwidth it consumes I may have to delete it Real Soon Now(tm) to avoid bandwidth overages that could dwarf the monthly subscription fee for the server. Potentially $2,380 in overage fees at $0.15 per GB. That is a worst case scenario, of course, but there is no reason for even a single GB of overages coming out of my pocket, much less 15,870.

Scott Dale Robison

[Snurd]

Ouch.
The only thing I have really noticed is yesterday and the day before, the site was down almost all day. I used tapatalk and also the main website and it gave me the "error 500, internal server error" message.

[isdr]

I got a notice from someone yesterday saying the forum was down, and by the time I got to it there was no problem. I doubt the recent problems you've seen are directly related to this particular problem, but I can't say for certain.

SDR

[gravedancer]

I received an email from the hosting service telling me that the server was sending out over 6 megabytes per second. I checked and learned it had been on going for over 4 hours for a total of 85 or so gigabytes. The server is only allowed 200 gigabytes per month.

I did some detailed analysis and learned that the errant process was due to a copy of phpMyAdmin installed on the UCC website. I suspect this is the second time I've seen this process using all the bandwidth, though this is the first time I've been able to prove it.

If anyone knows anything about this application / process, please let me know via email at [email protected]. Due to the amount of bandwidth it consumes I may have to delete it Real Soon Now(tm) to avoid bandwidth overages that could dwarf the monthly subscription fee for the server. Potentially $2,380 in overage fees at $0.15 per GB. That is a worst case scenario, of course, but there is no reason for even a single GB of overages coming out of my pocket, much less 15,870.

Scott Dale Robison

Not sure what else you have running on the server box, but you shoudlnt need phpmyadmin for anything on the forums, unless you ever needed to backup/restore the db. The funny thing is I cant see any reason why phpmyadmin would be using up bandwidth unless it was doing a backup or restore. My gut feeling is that someone got into phpmyadmin and was using it to take a data dump of the UCC forum database. How big is the db ? Id imagine its pretty good size by now. If it wasnt you who was into phpmyadmin then id make sure to change the credentials and stuff on it immediately.

[isdr]

It wasn't me, but I don't run / install software / maintain the forum, I just supply server space, hence my question and awaiting a response.

As for the size of the database, it certainly is no where near 85 GB. The raw data in the file system is under 400 MB in size (or under 0.4 GB to put things in a proper perspective).

My hope is that someone was trying to run a backup, but given the amount of data consumed vs the actual amount of data in MySQL, I fear it is something else.

SDR

[isdr]

Note: The server itself hosts a number of domains, of which www.utahconcealedcarry.com is far and away the winner on a popularity & resource consumption basis. While there are many other packages installed on the system, I'm only addressing things that are specific to UCC.

SDR

[IchBin]

phpMyAdmin has had several security issues over the last year IIRC. If you don't use it much, I'd certainly remove it.

[isdr]

I don't use it (or anything) for UCC. Hopefully Thomas or his minions will be able to shed some light on it.

SDR

[Shadehawk]

It's highly possible that we were a victim of a Denial Of Service (DOS) attack. These attacks are designed to bog down bandwidth and push servers by constantly pinging the server for upload/download processes. Given the size of the database, there could be no other way for us to have sustained that much usage over that span of time. Given that phpmyadmin was the source of the drain, I would assume that it has been compromised and we should secure it as soon as possible

[isdr]

I thought that at first, so I shut down the web server to all new requests. But the bandwidth consumption continued unabated, and it was traced back to a single process instance of php-cgi running a script from the UCC private copy of the phpMyAdmin application that had been started over 4 hours earlier.

I'm working to make things more reliable, but I still want to figure out who (if anyone here) was using phpMyAdmin.

SDR

[Shadehawk]

I thought that at first, so I shut down the web server to all new requests. But the bandwidth consumption continued unabated, and it was traced back to a single process instance of php-cgi running a script from the UCC private copy of the phpMyAdmin application that had been started over 4 hours earlier.

I'm working to make things more reliable, but I still want to figure out who (if anyone here) was using phpMyAdmin.

SDR

Well I'm sure either myself or divegeek can help with any digital forensics you might need

[isdr]

Okay, now that I'm home I did more digging. The IP address that last accessed the pma directory under UCC's vhost public directory was from Japan. So ... I've moved pma out of the public directory so that it cannot be abused. My apologies for any inconvenience it may cause, but I can't risk this kind of overage. If anyone needs access to phpMyAdmin or some other database access, please contact me at [email protected] and we'll work out something more secure.

SDR

[Shadehawk]

Okay, now that I'm home I did more digging. The IP address that last accessed the pma directory under UCC's vhost public directory was from Japan. So ... I've moved pma out of the public directory so that it cannot be abused. My apologies for any inconvenience it may cause, but I can't risk this kind of overage. If anyone needs access to phpMyAdmin or some other database access, please contact me at [email protected] and we'll work out something more secure.

SDR

Good call. I was going to suggest checking the logs.

[gravedancer]

phpMyAdmin has had several security issues over the last year IIRC. If you don't use it much, I'd certainly remove it.

You also wouldnt need a seperate copy of Phpmyadmin for each domain anyway. Typically its installed once on the box, and credentials granted to each domain to use it to manage databases on their domain.

[gravedancer]

I thought that at first, so I shut down the web server to all new requests. But the bandwidth consumption continued unabated, and it was traced back to a single process instance of php-cgi running a script from the UCC private copy of the phpMyAdmin application that had been started over 4 hours earlier.

I'm working to make things more reliable, but I still want to figure out who (if anyone here) was using phpMyAdmin.

SDR

Unless the web server is set up REALLY funky.,. there shouldnt be a seperate copy of phpmyadmin for the UCC domain. If there is one, you should be able to disable/remove it. PhpMyadmin is mostly used to create/manage .php databases. Other than backups/restores the only possible time you would have needed it for the forum was during installation, but the phpbb forum that were using has its own utilities built in for all of those functions.