Today's Outage

Any board related changes or announcements can be found here.

Today's Outage

Postby isdr » Thu 07 Jul 2011 10:54 am

I received an email from the hosting service telling me that the server was sending out over 6 megabytes per second. I checked and learned it had been on going for over 4 hours for a total of 85 or so gigabytes. The server is only allowed 200 gigabytes per month.

I did some detailed analysis and learned that the errant process was due to a copy of phpMyAdmin installed on the UCC website. I suspect this is the second time I've seen this process using all the bandwidth, though this is the first time I've been able to prove it.

If anyone knows anything about this application / process, please let me know via email at [email protected]. Due to the amount of bandwidth it consumes I may have to delete it Real Soon Now(tm) to avoid bandwidth overages that could dwarf the monthly subscription fee for the server. Potentially $2,380 in overage fees at $0.15 per GB. That is a worst case scenario, of course, but there is no reason for even a single GB of overages coming out of my pocket, much less 15,870.

Scott Dale Robison
isdr
Novice
 
Posts: 24
Joined: Tue 01 Sep 2009 3:35 pm

Re: Today's Outage

Postby Snurd » Thu 07 Jul 2011 11:36 am

Ouch.
The only thing I have really noticed is yesterday and the day before, the site was down almost all day. I used tapatalk and also the main website and it gave me the "error 500, internal server error" message.
Snurd
Posse
 
Posts: 9873
Joined: Sat 19 Jan 2008 10:24 am
Location: SLC

Re: Today's Outage

Postby isdr » Thu 07 Jul 2011 11:51 am

I got a notice from someone yesterday saying the forum was down, and by the time I got to it there was no problem. I doubt the recent problems you've seen are directly related to this particular problem, but I can't say for certain.

SDR
isdr
Novice
 
Posts: 24
Joined: Tue 01 Sep 2009 3:35 pm

Re: Today's Outage

Postby gravedancer » Thu 07 Jul 2011 12:15 pm

isdr wrote:I received an email from the hosting service telling me that the server was sending out over 6 megabytes per second. I checked and learned it had been on going for over 4 hours for a total of 85 or so gigabytes. The server is only allowed 200 gigabytes per month.

I did some detailed analysis and learned that the errant process was due to a copy of phpMyAdmin installed on the UCC website. I suspect this is the second time I've seen this process using all the bandwidth, though this is the first time I've been able to prove it.

If anyone knows anything about this application / process, please let me know via email at [email protected]. Due to the amount of bandwidth it consumes I may have to delete it Real Soon Now(tm) to avoid bandwidth overages that could dwarf the monthly subscription fee for the server. Potentially $2,380 in overage fees at $0.15 per GB. That is a worst case scenario, of course, but there is no reason for even a single GB of overages coming out of my pocket, much less 15,870.

Scott Dale Robison



Not sure what else you have running on the server box, but you shoudlnt need phpmyadmin for anything on the forums, unless you ever needed to backup/restore the db. The funny thing is I cant see any reason why phpmyadmin would be using up bandwidth unless it was doing a backup or restore. My gut feeling is that someone got into phpmyadmin and was using it to take a data dump of the UCC forum database. How big is the db ? Id imagine its pretty good size by now. If it wasnt you who was into phpmyadmin then id make sure to change the credentials and stuff on it immediately.
Image
http://bit.ly/2vHUiug will get you around the auto filtering of the domain on this site
gravedancer
Sniper
 
Posts: 1791
Joined: Mon 21 Mar 2011 1:46 pm

Re: Today's Outage

Postby isdr » Thu 07 Jul 2011 12:32 pm

It wasn't me, but I don't run / install software / maintain the forum, I just supply server space, hence my question and awaiting a response.

As for the size of the database, it certainly is no where near 85 GB. The raw data in the file system is under 400 MB in size (or under 0.4 GB to put things in a proper perspective).

My hope is that someone was trying to run a backup, but given the amount of data consumed vs the actual amount of data in MySQL, I fear it is something else.

SDR
isdr
Novice
 
Posts: 24
Joined: Tue 01 Sep 2009 3:35 pm

Re: Today's Outage

Postby isdr » Thu 07 Jul 2011 12:34 pm

Note: The server itself hosts a number of domains, of which www.utahconcealedcarry.com is far and away the winner on a popularity & resource consumption basis. While there are many other packages installed on the system, I'm only addressing things that are specific to UCC.

SDR
isdr
Novice
 
Posts: 24
Joined: Tue 01 Sep 2009 3:35 pm

Re: Today's Outage

Postby IchBin » Thu 07 Jul 2011 12:44 pm

phpMyAdmin has had several security issues over the last year IIRC. If you don't use it much, I'd certainly remove it.
~~ Dreams do not die with the Dreamer. ~~
IchBin
Marksman
 
Posts: 218
Joined: Mon 19 Apr 2010 11:05 am

Re: Today's Outage

Postby isdr » Thu 07 Jul 2011 1:27 pm

I don't use it (or anything) for UCC. Hopefully Thomas or his minions :D will be able to shed some light on it.

SDR
isdr
Novice
 
Posts: 24
Joined: Tue 01 Sep 2009 3:35 pm

Today's Outage

Postby Shadehawk » Thu 07 Jul 2011 1:48 pm

It's highly possible that we were a victim of a Denial Of Service (DOS) attack. These attacks are designed to bog down bandwidth and push servers by constantly pinging the server for upload/download processes. Given the size of the database, there could be no other way for us to have sustained that much usage over that span of time. Given that phpmyadmin was the source of the drain, I would assume that it has been compromised and we should secure it as soon as possible
"I finally found a cop light enough to carry around"
Springfield XD SC .40
Proudly Equipped With BLACKHAWK! Holsters and Accessories
NRA Member | Utah CFP Holder | OC Nut
User avatar
Shadehawk
Expert Marksman
 
Posts: 515
Joined: Sun 11 Jan 2009 5:41 pm
Location: Orem, ut

Re: Today's Outage

Postby isdr » Thu 07 Jul 2011 2:09 pm

I thought that at first, so I shut down the web server to all new requests. But the bandwidth consumption continued unabated, and it was traced back to a single process instance of php-cgi running a script from the UCC private copy of the phpMyAdmin application that had been started over 4 hours earlier.

I'm working to make things more reliable, but I still want to figure out who (if anyone here) was using phpMyAdmin.

SDR
isdr
Novice
 
Posts: 24
Joined: Tue 01 Sep 2009 3:35 pm

Today's Outage

Postby Shadehawk » Thu 07 Jul 2011 4:54 pm

isdr wrote:I thought that at first, so I shut down the web server to all new requests. But the bandwidth consumption continued unabated, and it was traced back to a single process instance of php-cgi running a script from the UCC private copy of the phpMyAdmin application that had been started over 4 hours earlier.

I'm working to make things more reliable, but I still want to figure out who (if anyone here) was using phpMyAdmin.

SDR


Well I'm sure either myself or divegeek can help with any digital forensics you might need
"I finally found a cop light enough to carry around"
Springfield XD SC .40
Proudly Equipped With BLACKHAWK! Holsters and Accessories
NRA Member | Utah CFP Holder | OC Nut
User avatar
Shadehawk
Expert Marksman
 
Posts: 515
Joined: Sun 11 Jan 2009 5:41 pm
Location: Orem, ut

Re: Today's Outage

Postby isdr » Thu 07 Jul 2011 7:44 pm

Okay, now that I'm home I did more digging. The IP address that last accessed the pma directory under UCC's vhost public directory was from Japan. So ... I've moved pma out of the public directory so that it cannot be abused. My apologies for any inconvenience it may cause, but I can't risk this kind of overage. If anyone needs access to phpMyAdmin or some other database access, please contact me at [email protected] and we'll work out something more secure.

SDR
isdr
Novice
 
Posts: 24
Joined: Tue 01 Sep 2009 3:35 pm

Today's Outage

Postby Shadehawk » Thu 07 Jul 2011 10:35 pm

isdr wrote:Okay, now that I'm home I did more digging. The IP address that last accessed the pma directory under UCC's vhost public directory was from Japan. So ... I've moved pma out of the public directory so that it cannot be abused. My apologies for any inconvenience it may cause, but I can't risk this kind of overage. If anyone needs access to phpMyAdmin or some other database access, please contact me at [email protected] and we'll work out something more secure.

SDR


Good call. I was going to suggest checking the logs.
"I finally found a cop light enough to carry around"
Springfield XD SC .40
Proudly Equipped With BLACKHAWK! Holsters and Accessories
NRA Member | Utah CFP Holder | OC Nut
User avatar
Shadehawk
Expert Marksman
 
Posts: 515
Joined: Sun 11 Jan 2009 5:41 pm
Location: Orem, ut

Re: Today's Outage

Postby gravedancer » Fri 08 Jul 2011 8:31 am

IchBin wrote:phpMyAdmin has had several security issues over the last year IIRC. If you don't use it much, I'd certainly remove it.


You also wouldnt need a seperate copy of Phpmyadmin for each domain anyway. Typically its installed once on the box, and credentials granted to each domain to use it to manage databases on their domain.
Image
http://bit.ly/2vHUiug will get you around the auto filtering of the domain on this site
gravedancer
Sniper
 
Posts: 1791
Joined: Mon 21 Mar 2011 1:46 pm

Re: Today's Outage

Postby gravedancer » Fri 08 Jul 2011 8:35 am

isdr wrote:I thought that at first, so I shut down the web server to all new requests. But the bandwidth consumption continued unabated, and it was traced back to a single process instance of php-cgi running a script from the UCC private copy of the phpMyAdmin application that had been started over 4 hours earlier.

I'm working to make things more reliable, but I still want to figure out who (if anyone here) was using phpMyAdmin.

SDR



Unless the web server is set up REALLY funky.,. there shouldnt be a seperate copy of phpmyadmin for the UCC domain. If there is one, you should be able to disable/remove it. PhpMyadmin is mostly used to create/manage .php databases. Other than backups/restores the only possible time you would have needed it for the forum was during installation, but the phpbb forum that were using has its own utilities built in for all of those functions.
Image
http://bit.ly/2vHUiug will get you around the auto filtering of the domain on this site
gravedancer
Sniper
 
Posts: 1791
Joined: Mon 21 Mar 2011 1:46 pm

Next

Return to Site Announcements

Who is online

Users browsing this forum: No registered users and 1 guest

cron