Problems with share recording and live data

Steve Sokolowski avatar
Steve Sokolowski
Archived posts: 4559
Joined: Aug 27, 2014

Problems with share recording and live data

by Steve Sokolowski · · Post #20595

There are currently problems with share recording and live data. The cause of the problems is unknown, and they started occurring without any releases or action on our part, making them difficult to troubleshoot.

I'll continue to look into the issue. Right now, there is no ETA on when they will be resolved. Mining continues as normal and Chris will make share corrections after the issues have been resolved.
Steve Sokolowski avatar
Steve Sokolowski
Archived posts: 4559
Joined: Aug 27, 2014

Re: Problems with share recording and live data

by Steve Sokolowski · · Post #20597

CSZiggy wrote:Is WAMP down or just not showing on the main website?

Can bots still connect and mine the top coin off of it?
That data isn't available to the public and anyone trying to connect to it will find that it isn't useful for their purposes.

I'm having difficulty investigating because these things always tend to happen when we are the least equipped to fix them - for example, during the holidays when we do not have access to the development environment.
Steve Sokolowski avatar
Steve Sokolowski
Archived posts: 4559
Joined: Aug 27, 2014

Re: Problems with share recording and live data

by Steve Sokolowski · · Post #20600

OK, the issue appears to be resolved. The cause of the problem was that the system was waiting for daemons to respond to requests to obtain data about found blocks. I reduced the amount of wait time so that the system will continue inserting shares even if the blocks are not available. Unfortunately, I was not able to determine why the daemons were performing so poorly and why the problem happened suddenly without our having made any changes.

After the holiday weekend, I'll add code to, once the system is behind by a large amount, skip all the blocks and just insert shares as quickly as possible to get back to normal.

Chris will work on the share corrections later today. Yesterday's profitability is listed below litecoin, which indicates that the issue probably began yesterday for one of the share inserters and went unnoticed. Therefore, he will likely credit customers for both yesterday and today when hashrates are low. The corrections will be made in about 12 hours.
Steve Sokolowski avatar
Steve Sokolowski
Archived posts: 4559
Joined: Aug 27, 2014

Re: Problems with share recording and live data

by Steve Sokolowski · · Post #20602

OK, I think I figured this one out. The system is running out of memory and dropping shares as a result.

What probably happened is that while more miners have joined us over time, Chris never increased the amount of memory and there were too many shares to be processed. When he gets back from lunch, I'll ask him to increase the amount of memory available to the share inserters.
Steve Sokolowski avatar
Steve Sokolowski
Archived posts: 4559
Joined: Aug 27, 2014

Re: Problems with share recording and live data

by Steve Sokolowski · · Post #20606

For those who are interested, this is a case of unintended consequences. I determined that the cause of the out of memory issues is that we previously estimated the profitability from the master mining server's profitability. When we started making exact estimates from all the mining servers, the number of profitability updates climbed by a factor of 8.

Because the old profitability data is cleared after 15 minutes, and not by the number of records in memory, the updates were eight times as frequent, and the system ran out of memory despite all the code working great.