Hashrate and balance problems this morning

Hi,

There were some hashrate and balance calculation problems this morning that caused miners to have holes in the hashrate charts and lower balances than normal.

We still haven't been able to figure out why this issue occurs in the middle of the night on some nights. However, we added a new feature last week to automatically recover when this occurs by restarting the share inserter when the system gets too far behind, and that's what happened. Therefore, the mining server never went offline and no money was lost.

Since the problem happened after midnight, payouts for today were not affected. Chris will correct balances this afternoon, well in advance of payouts tomorrow.

We're continuing to try to figure out what is causing this problem, and we apologize of the inconvenience. In the meantime, we have new features completed that will allow WAMP subscribers to determine the age of published WAMP data.

Chris determined what was the cause of this issue.

He was running out of disk space on the database. It was about to crash because increased hashrate had been filling up the disk a lot recently, so he had to move the shares to a different disk. Copying the data caused shares to queue in memory on the mining server, until memory ran out and the database-operator restarted.

He won't have to do this again, so this particular cause is unlikely to recur. However, Chris is also going to add 64GB of "swap space" to the mining server, so that if the system runs out of memory again, the swap space can hold shares until they can be inserted.

Steve Sokolowski wrote:Hi,

There were some hashrate and balance calculation problems this morning that caused miners to have holes in the hashrate charts and lower balances than normal.

We still haven't been able to figure out why this issue occurs in the middle of the night on some nights. However, we added a new feature last week to automatically recover when this occurs by restarting the share inserter when the system gets too far behind, and that's what happened. Therefore, the mining server never went offline and no money was lost.

Since the problem happened after midnight, payouts for today were not affected. Chris will correct balances this afternoon, well in advance of payouts tomorrow.

We're continuing to try to figure out what is causing this problem, and we apologize of the inconvenience. In the meantime, we have new features completed that will allow WAMP subscribers to determine the age of published WAMP data.

I have hashrate holes in my chart today and the balance as of right now is quite a bit lower than it was this same time yesterday it appears.

Will this be corrected tonight or will it catch up?

holes? there is a black hole in 4.5hrs... still w8ing ltc for that time

Eyedol-X wrote:

Steve Sokolowski wrote:Hi,

There were some hashrate and balance calculation problems this morning that caused miners to have holes in the hashrate charts and lower balances than normal.

We still haven't been able to figure out why this issue occurs in the middle of the night on some nights. However, we added a new feature last week to automatically recover when this occurs by restarting the share inserter when the system gets too far behind, and that's what happened. Therefore, the mining server never went offline and no money was lost.

Since the problem happened after midnight, payouts for today were not affected. Chris will correct balances this afternoon, well in advance of payouts tomorrow.

We're continuing to try to figure out what is causing this problem, and we apologize of the inconvenience. In the meantime, we have new features completed that will allow WAMP subscribers to determine the age of published WAMP data.

I have hashrate holes in my chart today and the balance as of right now is quite a bit lower than it was this same time yesterday it appears.

Will this be corrected tonight or will it catch up?

Chris is correcting it now and expects to have it finished within 3 hours.

Steve Sokolowski wrote:
Chris is correcting it now and expects to have it finished within 3 hours.

Thank you Steve and Chris! All payments recieved