Downtime this morning

News updates about the Prohashing pool
Post Reply
User avatar
Chris Sokolowski
Site Admin
Posts: 937
Joined: Wed Aug 27, 2014 12:47 pm
Location: State College, PA

Downtime this morning

Post by Chris Sokolowski » Tue Jul 18, 2017 1:08 pm

The pool went offline this morning due to server overload. I was asleep at the time, and while I have alerts set to my phone to wake me when the server goes offline, the phone was accidentally muted preventing me from being woken to fix the issue.

The underlying issue is that we are CPU limited in the process of assigning coins to miners, and the server load is roughly proportional to the number of coins being assigned to miners. As we have more hashrate, we need to distribute miners over more coins to prevent each network from finding blocks too frequently, causing the mining server's CPU load to increase. This morning, the markets were all around the same profitability, causing lots of coins to be mined at once. At the same time, 700+ GH/s of X11 hashrate came online. The server reached 100% CPU usage and was no longer able to process shares frequently enough. Eventually the server crashed lacking the memory to store additional shares.

I am willing to compensate miners for part of the time that the server was online and accepting shares, but I do not have the exact data when the server was online and accepting shares but not recording them versus being completely offline. If someone could send me a private message with hashrate data from an external service, I would appreciate the information.

Steve is working on an architectural overhaul of the mining server that will infinitely parallelize the mining server, which is currently predominantly single-threaded. However, that will not be complete for weeks if not a few months, so I am currently trying to find some ways to reduce server load in the meantime.

I know these issues are frustrating to you, and they are equally frustrating to me. With the growth in mining, managing server load occupies a majority of Steve's time and a lot of mine as well. Please realize we are working to resolve this issue but can't provide an immediate solution. I appreciate your patience while we work to improve stability.
User avatar
vhmanu
Posts: 54
Joined: Fri Apr 21, 2017 1:43 pm

Re: Downtime this morning

Post by vhmanu » Tue Jul 18, 2017 1:22 pm

i can link you my MRR profile where you can see the downtime beginning.
What hurt was the fact that the pool was not really dead, instead it was "just" accepting no shares ----> no miner switched to a backup pool. I switched manually to another pool after ~1hour when i noticed this at work.
AvPro
Posts: 66
Joined: Tue May 23, 2017 6:13 am

Re: Downtime this morning

Post by AvPro » Tue Jul 18, 2017 1:30 pm

Chris, I PM'd you some statistics and times. Agree with vhmanu, it would be nice if it disconnected the miners rather than allow 100% rejects :)
coldstone
Posts: 9
Joined: Sun Jul 09, 2017 7:46 am

Re: Downtime this morning

Post by coldstone » Tue Jul 18, 2017 2:13 pm

downtime sucks, best thing to do is learn from all things causing downtime and performing tweaks and upgrades so everything runs and keeps running..
biscayne
Posts: 16
Joined: Mon Apr 17, 2017 4:29 pm

Re: Downtime this morning

Post by biscayne » Tue Jul 18, 2017 3:45 pm

Murphy's law is becoming daily business at PH, not amused to loose more money.
User avatar
Chris Sokolowski
Site Admin
Posts: 937
Joined: Wed Aug 27, 2014 12:47 pm
Location: State College, PA

Re: Downtime this morning

Post by Chris Sokolowski » Wed Jul 19, 2017 2:51 am

We increased all users' July 18 earnings by 10% to compensate for any missing shares yesterday. Changes are effective immediately.
mine_x
Posts: 9
Joined: Mon Jul 17, 2017 3:33 am

Re: Downtime this morning

Post by mine_x » Wed Jul 19, 2017 3:31 am

Chris, respects to you for this step, glad to see true fair play :!:
mine_x
Posts: 9
Joined: Mon Jul 17, 2017 3:33 am

Re: Downtime this morning

Post by mine_x » Wed Jul 19, 2017 5:07 am

.. unfortunately, the pool again became unstable.. 30 minutes 100% rejects at pool :(
mickeekung
Posts: 52
Joined: Mon Apr 10, 2017 4:25 am

Re: Downtime this morning

Post by mickeekung » Wed Jul 19, 2017 5:23 am

Problem occurs again at the same time as yesterday.
Wanna see some exciting and interesting cloud mining contract which is different to Hashflare or Genesis Mining? Here is the site I trust.

Image

https://www.eobot.com/new.aspx?referid=8945

https://www.cryptomining.farm/signup/?referrer=59E0E92B882D
User avatar
FRISKIE
Posts: 121
Joined: Sun Apr 16, 2017 12:51 pm

Re: Downtime this morning

Post by FRISKIE » Wed Jul 19, 2017 6:09 am

This provides another attack vector for those who have tried so hard to keep PH offline with DDoS.

If knocking PH offline is as simple as setting up some really big Nicehash orders , pointing them at Prohashing, and overloading the server, I'd say we have a serious vulnerability here.

@ Chris - your thoughts?

Maybe remove some of the lower value coins to reduce the load during periods when profitabilities even out?
Post Reply