Some shares lost yesterday; downtime expected today

News updates about the Prohashing pool
Forum rules
The News forum is only for updates about the Prohashing pool.

Replies to posts in this forum should be related to the news being announced. If you need support on another issue, please post in the forum related to that topic or seek one of the official support options listed in the top right corner of the forums page or on prohashing.com/about.

For the full list of PROHASHING forums rules, please visit https://prohashing.com/help/prohashing- ... rms-forums.
Post Reply
User avatar
Steve Sokolowski
Posts: 4585
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Some shares lost yesterday; downtime expected today

Post by Steve Sokolowski » Tue Mar 26, 2019 8:20 am

Unfortunately, some shares were not recorded yesterday. The cause of the problem was determined to be that one miner was earning Stronghandcoins, which have an extremely low value. 100m of these coins are worth $17.

When we designed the system in 2013, there were no coins had a market capitalization of more than 82 million (the most was litecoin.) The database columns were sized so that the running earnings for a single day were recorded with eight digits before the decimal point, and 24 after. For the first time yesterday, someone earned more than 100m of a single type of coin in a single day. As designed, the share inserters attempted to insert as many shares as possible. Most batches of 2500 shares were inserted successfully, but batches including shares that would cause that customer's balance to exceed 100m Stronghandcoins were not inserted.

The daily balances tables are causing the problem, so the issue went away at midnight. It is likely to recur this evening when the daily balances get high again if nothing is done. It appears as if someone changed the all-time balance tables several years ago to mitigate this problem, so it must have been encountered in 2016 and at the time it must not have been thought that so many coins would be earned in a single day.

Resolving this issue will require us to modify the size of the database columns, an operation that causes Postgresql to rewrite all of the rows in the table. These tables are usually constantly locked for writes during normal operation, so we'll likely have to take the system offline to alter the tables. There are millions of rows in these tables, so the downtime could take a while.

After the columns have been modified, Chris will then determine if any data exists that would allow him to issue share corrections for shares that were not inserted yesterday evening. He'll post more in about six hours about the downtime.
srossman
Posts: 3
Joined: Thu Jul 06, 2017 4:08 pm

Re: Some shares lost yesterday; downtime expected today

Post by srossman » Tue Mar 26, 2019 11:38 am

Thank you for the update Steve. I appreciate your quick response as always!
User avatar
Steve Sokolowski
Posts: 4585
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Re: Some shares lost yesterday; downtime expected today

Post by Steve Sokolowski » Tue Mar 26, 2019 2:45 pm

srossman wrote:Thank you for the update Steve. I appreciate your quick response as always!
It turns out that the column was in status_daily_balance, which aggregates the coins that all miners have mined. As a result, it only has one row per coin per day, and we were able to rewrite all the rows in just one second. There was no service interruption and the bug was resolved.
srossman
Posts: 3
Joined: Thu Jul 06, 2017 4:08 pm

Re: Some shares lost yesterday; downtime expected today

Post by srossman » Tue Mar 26, 2019 4:41 pm

great news! Glad to hear it wasn't a complicated fix for you guys.
Thanks again and have a great night.
User avatar
Steve Sokolowski
Posts: 4585
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Re: Some shares lost yesterday; downtime expected today

Post by Steve Sokolowski » Wed Mar 27, 2019 7:02 am

srossman wrote:great news! Glad to hear it wasn't a complicated fix for you guys.
Thanks again and have a great night.
Chris completed the share corrections for this event, which credited about $973 to customers. As far as we're aware, everything relating to this issue is now closed.
Post Reply