How one incorrect constant could have caused a Bitcoin Cash crisis

User avatar
Steve Sokolowski
Posts: 3361
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA
Contact:

How one incorrect constant could have caused a Bitcoin Cash crisis

Postby Steve Sokolowski » Tue Dec 11, 2018 9:53 am

In Bitcoin Unlimited, on line 1185, in graphene.cpp, at https://github.com/BitcoinUnlimited/Bit ... 1511/files, there is a seemingly insignificant constant, nTimeToWait. On the morning of December 3, the value of this constant caused the Bitcoin Cash network to reorganize and discard a valid block - but upon further examination, the consequences could have been much worse.

The origin of this issue originates from a decision made by us in June 2015, when Michael Marquadt (/u/theymos), owner of the bitcointalk.org forums, many reddit "subreddits," and several cryptocurrency news websites, decided to split the community by banning discussion of larger blocksizes on his platforms. We were banned from his forums for ignoring his rules. When the Bitcoin Core developers failed to denounce his tactics, we feared that Bitcoin Core could allow code to be slipped in to do something like drop blocks with coinbase transactions containing "censored text," and looked for an alternate mining client. The first one to become widely available when we began implementing SHA-256 mining was Bitcoin Unlimited.

Over time, we developed software that is tested to work with Bitcoin Unlimited, and therefore it became a risk to change to any other client. For example, a geth upgrade caused us to be unable to mine Ethereum blocks for a few hours, so we had to revert to an older version and still don't understand what caused the problem. That's despite the geth developers claiming that the code was a "minor update." Since Bitcoin Unlimited has worked for years, we don't want to mess with it, and continue to mine several SHA-256 coins with it despite many Bitcoin Cash miners apparently mining with Bitcoin ABC.

As it turns out, Bitcoin Unlimited Cash does not support all of the "compact block" transfer protocols that other Bitcoin Cash clients do. To get around the problem, a hack was inserted: Bitcoin Unlimited will wait an average of 10s to see if a bandwidth-saving transfer protocol that it supports has a block available from another node. If not, then the original protocol that re-transmits all the transactions in the block is used instead. That 10s doubles because two nodes need to communicate.

On December 3, however, this simple line caused a major issue with Bitcoin Cash that could have affected almost every business that used the protocol. Another pool, btc.top, found Bitcoin Cash block 559307 about 8 seconds before we also found a block at the same height, as indicated at https://www.blocktrail.com/BCC/blocks/orphans/1. The download timer caused notification of that block to be received after our block was found, which triggered our "orphan mining" code. This code doubles the expected value of mining a coin after we identify that one of our blocks will be orphaned, because if we find the next block, we will confirm the previous block as well as the subsequent one. The function caused all SHA-256 hashrate to be diverted to mining Bitcoin Cash. In addition, once our published profitability rose, bots that monitor it noticed and diverted a large amount of hashrate to the pool, likely from cloud mining rental services. It was now profitable to pay even wildly inflated prices on places like NiceHash and earn profit by directing hashrate here. The increased hashrate succeeded in orphaning btc.top's block and reorganizing the Bitcoin Cash blockchain across the network.

In this case, the only loser was btc.top, because they ceased mining their chain at that point. Our "orphan mining" code ensures that the two blocks we mine include all of the transactions in the orphaned block, so exchanges depending on confirmation would simply see that their transactions possibly moved up in height by one block.

Originally, I believed that this issue was a one-off incident that would have been resolved with one pool or the other losing two blocks in the worst case. But after further examination, I discovered that there are many more bots monitoring our expected payouts than I had thought. Had a second block been found by some other pool following btc.top's chain and our now hugely expanded pool during the 20s window caused by the hardcoded delay, then the system would have tripled the expected profitability and continued on. It's likely that even more hashrate would have been sent to the pool to mine Bitcoin Cash. Earlier that week, 200PH/s of additional SHA-256 hashrate suddenly appeared when Bitcoin SV became extremely profitable, so it's reasonable to expect that all available rentable cloud miners in the world could have been pointed to the pool if we were paying 3x bitcoin profitability.

Given that nobody was present to monitor the system at that time, humans would not have known to resolve the issue. We have notifications to notify of downtime, but why would anyone think to write a notification for when profit was high? With 20s delays, the odds of the problem escalating to a two-block race between two networks with similar hashrate were as high as 3.3%, and the odds of a three-block crisis were 1 in 900. After three blocks, the debt owed to customers would have grown so high that it would no longer have been legally possible for us to simply "give up," just as it is not legally possible for us to decide to refund money accidentally sent with ridiculously high transaction fees (like the $100,000 Verge transaction fee last Christmas Day - viewtopic.php?f=4&t=2953) because that money is already owed to customers. A fork of Bitcoin Cash could have occurred as a result.

Bitcoin forked due to a different issue in early 2013, and the solution then was for miners to agree on one of the chains as valid. Even if some kind of agreement to compensate miners on one side of the Bitcoin Cash chain were reached, commerce on sites like Purse.io would have been disrupted until a solution was reached.

The surprising part is that if you take the math even further, the odds were just 1 in 27,000 that a three-block fork would have occurred due to this constant in Bitcoin Unlimited. Bitcoin Cash blocks occur every 10 minutes, so a critical situation was likely to occur twice per year. Given that Bitcoin Cash has been around for 1.3 years, the only reasons that the network did not fork is because of good luck, because pools have been willing to give up against their economic interests in situations like this, or because they never bothered to implement the code to improve their profitability by building upon their orphaned blocks.

Note that the odds are not affected by how many pools run Bitcoin Unlimited, because there is so much cloud mining hashrate available that it will all go to any pool that promises large payouts after the race begins. Furthermore, our algorithm ensures that it never reverses normal transactions; it only assigns the mining rewards to different people. It's much easier to implement code that ignores the transactions on both forks and just mines whatever - or, to purposely use the delay to engage in criminal activity.

While most people in cryptocurrency state that "competing implementations" are an important goal of client development, this occurrence should demonstrate that the consensus rules are not the only important code to adhere to in new clients. This delay was a far more serious problem than is widely recognized. Fortunately, user Jonathan Toomim contributed a pull request to Bitcoin Unlimited that reduced the timer to an average of 1s in the latest version, just 10% of what it was in the previous version, which will make this scenario far more unlikely in the future.
sickpig
Posts: 2
Joined: Thu Oct 13, 2016 9:39 am

Re: How one incorrect constant could have caused a Bitcoin Cash crisis

Postby sickpig » Tue Dec 11, 2018 11:27 am

Hi Steve

Fortunately, user Jonathan Toomim contributed a pull request to Bitcoin Unlimited that reduced the timer to an average of 1s in the latest version, just 10% of what it was in the previous version, which will make this scenario far more unlikely in the future.



Just wanted to point out that BUcash 1.5.1 (actually in release candidate 2) contains jtoomim's change. The final 1.5.1 release is planned to be released on Monday 17th (unless some issues with the current code arise).

The plan is to remove it completely once we have Compact Blocks implemented, Peter Tschipper already started to work on it.
User avatar
CSZiggy
Posts: 602
Joined: Wed Jan 31, 2018 2:44 pm

Re: How one incorrect constant could have caused a Bitcoin Cash crisis

Postby CSZiggy » Tue Dec 11, 2018 11:50 pm

Steve Sokolowski wrote:In addition, once our published profitability rose, bots that monitor it noticed and diverted a large amount of hashrate to the pool, likely from cloud mining rental services. It was now profitable to pay even wildly inflated prices on places like NiceHash and earn profit by directing hashrate here. The increased hashrate succeeded in orphaning btc.top's block and reorganizing the Bitcoin Cash blockchain across the network.


Shouldn't be too surpsrised, I've talked about the BOT miners and how they are killing the profits of your pool.

How if you were ever to take the time to analyze it, you might be able to bring profits up to make it profitable for everyone on the net to mine here. viewtopic.php?f=4&t=6043

I've brought this same topic up no less than 4 times over the past year-18 months. viewtopic.php?f=13&t=6023

By allowing the bot miners/static miners to profit from the pool, I feel you are lowering the payouts and profits of everyone else in the autopool.
The SMART people are robbing from the SHEEPLE. The SHEEPLE are using the pools formulas to mine, the SMART people are using their own and making more profit at all the SHEEPLES expenses. Once they figure that out, all the SHEEPLE have to do is move to any other pool that doesn't rob them to give their money to the SMART people running their own bot miners, and POOF an extra 10% profits.

I mined $55,000 this year. Would have been more if not for the November crash this year.
I did testing sporadically on this pol but as I stated I always found 10% more elsewhere, and didn't care what coin I got as my payout.
10% extra is $5000 that would have been lost to me had I mined here this year. I wasn't doing this as a hobby this was my full time business.
I really wanted to be able to mine here and make the max profit but was never able to support the pool due to loss of profits.

I have explained how the bot miners get overpaid and then pick coins the autopool miners are also picking, rising the prices charged them and ultimately delivering a smaller payout, in less coins earned per day. Resulting in a lower payout here versus other places.

I think 2 or 3 changes could fix a lot of issues and raise profits. Again I don't have access to the data, you tell us, maybe I have it all right but the percent is only .000001% and doesn't matter a hill of beans. On the other hand maybe the percent is 10%, and thats why the pool's payout has been lower all this time and could have been dead even with the rest of the world making less reason not to mine here.
User avatar
Steve Sokolowski
Posts: 3361
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA
Contact:

Re: How one incorrect constant could have caused a Bitcoin Cash crisis

Postby Steve Sokolowski » Wed Dec 12, 2018 7:37 am

CSZiggy wrote:
Steve Sokolowski wrote:In addition, once our published profitability rose, bots that monitor it noticed and diverted a large amount of hashrate to the pool, likely from cloud mining rental services. It was now profitable to pay even wildly inflated prices on places like NiceHash and earn profit by directing hashrate here. The increased hashrate succeeded in orphaning btc.top's block and reorganizing the Bitcoin Cash blockchain across the network.


Shouldn't be too surpsrised, I've talked about the BOT miners and how they are killing the profits of your pool.

How if you were ever to take the time to analyze it, you might be able to bring profits up to make it profitable for everyone on the net to mine here. viewtopic.php?f=4&t=6043

I've brought this same topic up no less than 4 times over the past year-18 months. viewtopic.php?f=13&t=6023

By allowing the bot miners/static miners to profit from the pool, I feel you are lowering the payouts and profits of everyone else in the autopool.
The SMART people are robbing from the SHEEPLE. The SHEEPLE are using the pools formulas to mine, the SMART people are using their own and making more profit at all the SHEEPLES expenses. Once they figure that out, all the SHEEPLE have to do is move to any other pool that doesn't rob them to give their money to the SMART people running their own bot miners, and POOF an extra 10% profits.

I mined $55,000 this year. Would have been more if not for the November crash this year.
I did testing sporadically on this pol but as I stated I always found 10% more elsewhere, and didn't care what coin I got as my payout.
10% extra is $5000 that would have been lost to me had I mined here this year. I wasn't doing this as a hobby this was my full time business.
I really wanted to be able to mine here and make the max profit but was never able to support the pool due to loss of profits.

I have explained how the bot miners get overpaid and then pick coins the autopool miners are also picking, rising the prices charged them and ultimately delivering a smaller payout, in less coins earned per day. Resulting in a lower payout here versus other places.

I think 2 or 3 changes could fix a lot of issues and raise profits. Again I don't have access to the data, you tell us, maybe I have it all right but the percent is only .000001% and doesn't matter a hill of beans. On the other hand maybe the percent is 10%, and thats why the pool's payout has been lower all this time and could have been dead even with the rest of the world making less reason not to mine here.


I understand your frustration, but I already replied in regards to this issue. I pointed out multiple times that the money that can be earned from constant static coin mining of very easy coins, like these bots are doing, is very minimal, about a few dollars per day. The profit potential is about 0.05%, it would take $5,000 in development costs to create measures that just anger customers, and the number of miners who would leave the pool due to thinking they are earning less due to having lower hashrates because of the work restarts would far offset any profit gains.

I don't know what else to tell you. We've run the math on this low-difficulty coin topic a number of times and have explained the reason why the system makes the choices it does in two different threads. There's not much more I can explain.
User avatar
TerryLewis
Posts: 2
Joined: Tue Dec 04, 2018 7:53 am

Re: How one incorrect constant could have caused a Bitcoin Cash crisis

Postby TerryLewis » Fri Jan 11, 2019 1:56 am

The surprising part is that if you take the math even further, the odds were just 1 in 27,000 that a three-block fork would have occurred due to this constant in Bitcoin Unlimited. Bitcoin Cash blocks occur every 10 minutes, so a critical situation was likely to occur twice per year. Given that Bitcoin Cash has been around for 1.3 years, the only reasons that the network did not fork is because of good luck, because pools have been willing to give up against official writemyessay and their economic interests in situations like this, or because they never bothered to implement the code to improve their profitability by building upon their orphaned blocks.

Note that the odds are not affected by how many pools run Bitcoin Unlimited, because there is so much cloud mining hashrate available that it will all go to any pool that promises large payouts after the race begins. Furthermore, our algorithm ensures that it never reverses normal transactions; it only assigns the mining rewards to different people. It's much easier to implement code that ignores the transactions on both forks and just mines whatever - or, to purposely use the delay to engage in criminal activity.

While most people in cryptocurrency state that "competing implementations" are an important goal of client development, this occurrence should demonstrate that the consensus rules are not the only important code to adhere to in new clients. This delay was a far more serious problem than is widely recognized. Fortunately, user Jonathan Toomim contributed a pull request to Bitcoin Unlimited that reduced the timer to an average of 1s in the latest version, just 10% of what it was in the previous version, which will make this scenario far more unlikely in the future.


Indeed, the development costs for creating measures would be too risky. And I agree that in this case, bots are not actually doing too loud damage to turn everything upside down and change the stability for the customers. However, in any situation, there are always chances for the possible crisis, like you mentioned.
Thanks for the deep analysis.

Terry Lewis

Who is online

Users browsing this forum: No registered users and 3 guests