Mining server memory issues

News updates about the Prohashing pool
Post Reply
User avatar
Steve Sokolowski
Posts: 4380
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Mining server memory issues

Post by Steve Sokolowski » Tue Jun 08, 2021 9:01 am

Good morning!

Recently, we've been receiving tickets from customers who state that they are "unable to connect" to the mining servers. We originally thought that these customers had Internet connectivity issues. While some do, we also discovered that the mining servers have been running out of memory and restarting, usually around every four hours. For five minutes after the restarts, some customers are unable to connect because the mining servers haven't yet received coin prices.

There has always been a memory leak in the mining servers, but we ignored it both because the servers would restart quickly, and because the restarts were occurring every few days. It's not clear why the problem has gotten worse recently.

We are addressing this issue in two ways. First, I'm working to reduce memory usage in the mining servers in the hopes of either fixing the leak, or if that is not possible to do in a reasonable timeframe, at least decreasing memory usage so that the servers restart less frequently. One task I already finished, and which will be active the next time each server restarts, is cleaning up old ethash DAGs sooner. Chris will also spend $500 to upgrade the memory in the mining servers from 32 to 128GB.

We anticipate that this problem will continue to improve over the course of this week, and hope that the new memory will arrive within two weeks. By June 21, we expect that at the least, we should be able to make the restarts occur 12 times less frequently.
-MaVerick-
Posts: 24
Joined: Wed May 12, 2021 3:12 pm

Re: Mining server memory issues

Post by -MaVerick- » Tue Jun 08, 2021 10:14 am

Can't go wrong with going 4 times the RAM!
User avatar
BMChris
Posts: 15
Joined: Fri May 14, 2021 8:10 pm

Re: Mining server memory issues

Post by BMChris » Wed Jun 09, 2021 5:23 am

I love how transparent the Dev team are at Prohashing, it's great and really helps to build trust. :)
User avatar
Steve Sokolowski
Posts: 4380
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Re: Mining server memory issues

Post by Steve Sokolowski » Wed Jun 09, 2021 7:55 am

There seems to be some confusion about the scope of this issue.

The memory issue only interrupts miners who are doing Proswitching FPPS mining for about 2 minutes. Solo or PPLNS miners may see longer delays, but only if they are mining a small coin network. The anchor coins, like bitcoin or litecoin, would only be unavailable for 2 minutes.

If you are encountering an issue where your connectivity is unavailable for 30 minutes or an hour, you are having an unrelated issue. We aren't aware of any widespread issues that would cause disconnects as lengthy as that. Please submit a support ticket in that case, and we'll be glad to help.
User avatar
Steve Sokolowski
Posts: 4380
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Re: Mining server memory issues

Post by Steve Sokolowski » Thu Jun 10, 2021 10:21 am

Update: yesterday's improvements appear to have mitigated this problem enough that it is now occurring about half as often - about every 5 hours for half of customers and every 20 hours for the other half. I plan to continue making improvements today, and the memory is in the mail for delivery next week.
User avatar
Steve Sokolowski
Posts: 4380
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Re: Mining server memory issues

Post by Steve Sokolowski » Fri Jun 11, 2021 8:01 am

Chris found that the memory will arrive on June 14.

We expect there to be 15 minutes of mining server downtime around midnight EDT on June 15 to install this memory. No other services will be affected.
User avatar
Steve Sokolowski
Posts: 4380
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Re: Mining server memory issues

Post by Steve Sokolowski » Sat Jun 12, 2021 12:43 pm

The shipment estimate has been delayed until June 15 now. Unfortunately, there is no vendor who will ship memory overnight, at any cost. However, we're looking at whether we can pay a delivery driver to drive 400 miles and deliver the parts.

At the same time, when a server runs out of memory and restarts, I'm working on a way to make the mining servers get prices from another mining server, rather than waiting for them to be provided by exchanges. That will minimize the interruptions when a restart occurs.
User avatar
Steve Sokolowski
Posts: 4380
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Re: Mining server memory issues

Post by Steve Sokolowski » Sat Jun 12, 2021 4:15 pm

I've reduced the time it takes for a mining server to start up from 10 minutes to 10 seconds. This change will be available tonight.
User avatar
Banished_Privateer
Posts: 43
Joined: Wed Feb 10, 2021 6:49 am

Re: Mining server memory issues

Post by Banished_Privateer » Sun Jun 13, 2021 11:35 am

That's like 60x improvement :shock:
Post Reply