Nicehash allowing block withholding and 51% attacks through its services

Read about Prohashing, mining, and coins in general!
Forum rules
Sign up to be notified of new posts to the Prohashing Blog at http://eepurl.com/gx1e0j
Post Reply
User avatar
Steve Sokolowski
Posts: 3780
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Nicehash allowing block withholding and 51% attacks through its services

Post by Steve Sokolowski » Fri Mar 22, 2019 2:09 pm

Whenever a bug has occurred over the past few years, it seems as if it always involves Nicehash rentals in some way or another. Until now, however, we never had any convincing evidence that Nicehash was negligent in any of these issues. Today, I'm finally able to detail a number of areas in which Nicehash is causing significant harm to the cryptocurrency industry.

It is extremely likely that Nicehash miners were involved in a block withholding attack that occurred against Ethereum from March 2 to 3.

When the Constantinople Ethereum fork occurred in late February, Prohashing upgraded to the latest version of the client. Unfortunately, unlike all other types of coin daemons, geth-based clients do not provide any API calls to determine the block reward of the next block. For ethash coins, we manually need to configure the system with the correct block reward. Chris forgot to reduce the block reward from 3 to 2, and apparently some bots following the profitability data detected the error and sent more hashrate to the pool during that period. The system performed as designed and configured, correctly finding Ethereum blocks on the new Constantinople system but paying out more than it should have.

On March 2, one customer rented about 1 TH/s of hashrate from Nicehash and directed it to the pool, statically mining Ethereum. Over the course of the next two days, this customer submitted enough hashrate that should have found about 13 blocks of Ethereum. In reality, zero blocks were submitted. The probability of such poor luck is about 0.000022, around the same as dying in a meteor strike or in a plane crash if you fly every day for your entire life.

We record a huge amount of data regarding shares, and this data is fed into an algorithm we discovered in 2016 that can identify these types of attacks. While we won't discuss what the algorithm showed, the data that is not proprietary is sufficient to show with high probability that Nicehash allowed this attack to occur. Share difficulties are different than the Ethereum network difficulty. The maximum submitted share difficulty of any share during this incident at 226078.78, about half the Ethereum share difficulty. During the five days preceding the incident, the maximum share difficulty submitted was 9508076.33, even though the amount of hashrate mining during that time was about 10 times less than the hashrate rented during the two day period.

In fact, had the hashrate between February 23 and 28 been mining Ethereum, three blocks would have been found when about 1.5 were expected. Not only were two additional shares higher than the Ethereum difficulty submitted, the highest share difficulty from these miners was 42 times higher than that performed by the Nicehash miners, despite their submitting 1/10 the number of shares as the Nicehash miners did.

Before contacting Nicehash, we examined the system to determine if bugs could have been responsible for the issue. First, we checked the Ethereum blockchain and did not find any blocks mined on March 2 or 3, so there was not a recording error in our database. We confirmed that the geth client was accepting blocks and sending them to the correct chain, because a block had been found by another miner on March 1 and no changes had been made since then. We confirmed that the database had not recorded any exceptions. Since no other miners, including other ethash miners, encountered a similar issue during that period, it is unlikely that a share checking issue affected just this one person.

Even though we are 95% certain that no bugs occurred, it is never possible to completely rule out the possibility of a bug, so the customer contacted Nicehash to present our findings. Nicehash repeatedly stated that nothing was wrong with their systems. I sent three E-mails to different representatives at the company asking them to provide details about how they knew that nothing was amiss. I showed them the analysis above, as well as other data, and asked them what algorithm they used, or what data they possessed, that led them to the conclusion that their systems were normal. I told them "running as normal" was so vague that simply having their website online could also be interpreted as "running as normal." I told them I was willing to consider that there was a bug in our system if they had anything that contradicted what I found. They provided no evidence whatsoever to support their conclusion, and therefore I concluded that Nicehash collects no evidence needed to detect nefarious activities and therefore allowed a block withholding attack to occur through their service.

Prohashing stands by its customers. Since Nicehash does not provide any tools to indicate what hashrate a person is renting, and actually couldn't provide these tools because they aren't recording the data necessary to provide them, there was nothing to indicate the customer was involved in criminal activity. We paid him in full. The portion lost due to the issue that wasn't due to misconfiguration was $3,000, a significant portion of the $17,000 I expect to earn this year as we try to stay alive in these difficult times for the cryptocurrency industry.

When I told Nicehash that we would be suing them over the incident, they directed me to their terms of service (https://www.nicehash.com/terms.) What is there is pretty shocking. Buried in that text, readers will find that Nicehash has no obligations to customers if "it would be unjust to enforce NiceHash’s obligations in the general opinion" or "NiceHash’s obligation becomes impossible." In other words, if they no longer have enough money to pay their customers, those impossible obligations go away.

They further state that they are not indebted for the December 6, 2017 hack on their servers, so if they issue any more refunds it's out of the goodness of their hearts. All of Nicehash's wallets are uninsured and customers agree that their funds might be lost again due to another hack. Nicehash offers no refunds for any reason, even if the hashrate isn't as ordered - "all shares forwarded to mining pool, selected on the Hashing power order are final and non-refundable."

Nicehash even acknowledges that renters of hashrate can change the configuration of their miners and that there is no guarantee that you're getting what you paid for. They can advertise that you rent 1 MH/s of hashrate and the renter submits 1 MH/s of shares but without the blocks, and if so, it's too bad for the buyer that he paid for something of no value. After all, "NiceHash does not guarantee that the hashing power will actually be delivered or verified and does not guarantee any quality of the NiceHash Services." But the best part of the entire document is hidden towards the end, where Nicehash's liability "shall in any event not exceed 100 EUR per user." It seemed pretty obvious after reading that that nobody should ever sue them for anything.

The block withholding attack is just the latest in a string of security incidents perpetrated with the use of Nicehash's services. Throughout 2018, reports indicated that Nicehash's services were used several times to conduct 51% attacks against small coin networks. Nicehash responded to these attacks in a blog post at https://www.nicehash.com/news/nicehash- ... -51-attack. Incredulously, rather than taking responsibility and announcing a plan to prevent attacks, Nicehash used the accusations as a publicity stunt to try to drum up interest in their services. If people want to attack your coin, you should rent just as much hashrate from Nicehash to defend yourself! The advertisement is like a protection scheme in the movies, where someone creates a problem that you can then hire them to solve.

In 2018, Nicehash miners mined Verge and other coins at the pool during the time the Verge network was being attacked to defraud an exchange, and we ended up having to pay the Nicehash miners a lot of miners for the worthless Verge blocks. In 2017, there was a race condition where easy blocks could pay out a few cents more per day than they were worth, and suddenly Nicehash miners appeared by the hundreds. Nicehash always seems to be involved in just about every issue that we have, regardless of whether the problem has anything to do with Nicehash or us.

At some times, Nicehash's rental prices have been higher than the profitabilities of all coin networks. Until now, it wasn't clear to me why that was. The answer seems clear - criminal activity is partially to blame. Block withholding attacks, 51% attacks, and exploitations of bugs at pools prop up Nicehash's rental prices above market value. Despite both accepting deposits from one source and allowing withdrawals to a different destination, Nicehash illegally provides this withdrawal to different address service to US customers by not performing KYC/AML verification. Whether intended that way or not, the service has become a platform where, whether Nicehash knows about such activity or not, criminals can anonymously provide bad hashrate, or rent hashrate, to exploit security vulnerabilities and steal money.

While the default argument in favor of Nicehash might be that protecting against abuse is difficult, Nicehash could likely eliminate almost all abuse for less than $50,000 - the salary of a developer for 6 months' work. Nicehash could prevent obvious block withholding attacks like the one discussed here by simply recording how many blocks were submitted. When luck becomes unreasonably poor, Nicehash could simply refuse to pay the seller until enough hashrate was rented to bring luck up to a minimal level - say, to maintain a probability greater than 1% that what is occurring is by chance. Even if Nicehash terminated the seller's account and still paid the seller after luck fell below a 1% threshold, then the losses in the case above would have been cut by 2/3. That's a 2/3 reduction by simple math, even before getting into the complex algorithm we use that can detect these attacks with near perfect accuracy.

Rather than asking people to E-Mail them after an attack occurred, when it's too late for anything to be done and when there are so many people potentially involved that it is impossible to prove which people knew about what, Nicehash could completely eliminate 51% attacks by simply not allowing more than 49% of a coin's network to be rented by Nicehash miners. This can be done with 100% effectiveness. All Nicehash has to do is to intercept the pool's messages, which it does already, and determine the difficulty of the network from the data it receives. Then, it can just refuse to send enough miners to mine that coin based on its difficulty. We already do this, preventing the pool from assigning more than 49% of a network's hashrate to a coin. If customers cannot rent more than 49% of a coin's difficulty from Nicehash, 51% attacks cannot occur using their system because the criminals' blockchains will eventually, by chance, be overtaken by the main chain.

In conclusion, Nicehash has a serious problem with abuse of its services, and as the latest incident shows, the abusers are becoming more brazen. Nicehash either is not aware of this abuse, or it has intentionally avoided taking steps to address it. Nicehash's terms of service attempt to disclaim any responsibility for its role in these activities. Now that this post has been published and forwarded to Nicehash's managment, it would be negligent for the company to knowingly allow criminal activity to continue to occur with its system. It will no longer be sufficient for Nicehash, for example, to claim innocence if criminals use their platform to commit fraud against an exchange with a 51% attack after I described above the simple steps that can be implemented to eliminate the 51% attacks. There are also ethical implications as people who rent miners to Nicehash need to decide if they are comfortable being associated with these type of activities.

Nicehash presents a unique issue for us because unlike them, we are not willing to pass responsibility onto the customer for fraudulent Nicehash rentals. It is not good customer service to tell a victim of a crime that we aren't going to help him or her and that (s)he lost all his or her money.

For now, we will be banning Nicehash miners from mining the coins with the highest difficulty and therefore the greatest probability of losses, of all algorithms at Prohashing. The changes will be implemented over the next few weeks. It is easy for us to detect these attacks on the other coins, but we still don't want to blame the customer when we do detect them. That leaves us with a decision to be made in the coming weeks about whether we should institute a complete ban on Nicehash or not. These sort of problems have not been found with direct miners, and if they were to occur, we would be able to point blame at the miner rather than not knowing who is responsible.

As to Nicehash, we are still willing to entertain the notion that a bug in our system did cause the most recent incident, and will modify this post if Nicehash provides any evidence to support their claims that their system was functioning normally. Until then, we will continue to place our customers' satisfaction first, and we will not accept future impacts from Nicehash allowing misuse of its services.
User avatar
CSZiggy
Posts: 691
Joined: Wed Jan 31, 2018 2:44 pm

Re: Nicehash allowing block withholding and 51% attacks through its services

Post by CSZiggy » Fri Mar 22, 2019 3:11 pm

So a bank lets a person take out a loan. They use the money from the loan to do bad things. And you want to blame the bank who lent them the money. Nicehash is a business. They provide a service, cloudmining, what the people do with that should not be held against that company--I am SOO not a fan of nicehash, seeing as they still owe me from Nov 2017 hack which I know I have as much chance to get back as Mt.Gox money, but blaming them for how people use their service is like sueing the bank who let the sniper take out a loan to buy a gun and shoot 15 people. So the pool forgot to decrease payout from 3 down to 2, the auto bots saw that the profits were up so they all swapped pools. It was soo good, they signed up for extra hashrate to take advantage of the increased profits being promised. I am not seeing how this is a nicehash issue.

https://cryptolinks.com/cryptocurrency-mining
Lists a whole bunch of cloud mining services. You want to ban Nicehash, have at it. You don't think they won't all just jump to the next one down the list? Or buy from nicehash, send it over to their miningrigrentals account, and then bounce that hash from there over to prohashing to still mine the 1/3 extra profit they saw on the profitability tables?

If the autopool worked as well as it should, it should have seen the 1/3 increased profit and jumped the whole pool onto that coin at the time it happened. Did it? If it did then it was not a nicehash problem it was a pool configuration issue based on the numbers the pool set. If the autopool didn't jump(why not if profits were higher) the whole pool onto that coin, then only the cherry-picking static miners were the ones benefiting. So banning nicehash cloudmining does what? Makes everyone sign up for the next pool on the list and continue as normal.

It is still my opinion that NON-SOLO static mining on this pool costs it and the autopool switching miners profit margins and money.
I stand by my opinion. So either ban ALL the cloud mining sites and lose a ton of potential business, or ban the static mining if they are not in SOLO mode. Otherwise I see a future in which your $17,000 salary continuing to drop the rest of the year and may end up costing you before its over.
User avatar
Steve Sokolowski
Posts: 3780
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Re: Nicehash allowing block withholding and 51% attacks through its services

Post by Steve Sokolowski » Fri Mar 22, 2019 4:02 pm

CSZiggy wrote:So a bank lets a person take out a loan. They use the money from the loan to do bad things. And you want to blame the bank who lent them the money. Nicehash is a business. They provide a service, cloudmining, what the people do with that should not be held against that company--I am SOO not a fan of nicehash, seeing as they still owe me from Nov 2017 hack which I know I have as much chance to get back as Mt.Gox money, but blaming them for how people use their service is like sueing the bank who let the sniper take out a loan to buy a gun and shoot 15 people. So the pool forgot to decrease payout from 3 down to 2, the auto bots saw that the profits were up so they all swapped pools. It was soo good, they signed up for extra hashrate to take advantage of the increased profits being promised. I am not seeing how this is a nicehash issue.

https://cryptolinks.com/cryptocurrency-mining
Lists a whole bunch of cloud mining services. You want to ban Nicehash, have at it. You don't think they won't all just jump to the next one down the list? Or buy from nicehash, send it over to their miningrigrentals account, and then bounce that hash from there over to prohashing to still mine the 1/3 extra profit they saw on the profitability tables?

If the autopool worked as well as it should, it should have seen the 1/3 increased profit and jumped the whole pool onto that coin at the time it happened. Did it? If it did then it was not a nicehash problem it was a pool configuration issue based on the numbers the pool set. If the autopool didn't jump(why not if profits were higher) the whole pool onto that coin, then only the cherry-picking static miners were the ones benefiting. So banning nicehash cloudmining does what? Makes everyone sign up for the next pool on the list and continue as normal.

It is still my opinion that NON-SOLO static mining on this pool costs it and the autopool switching miners profit margins and money.
I stand by my opinion. So either ban ALL the cloud mining sites and lose a ton of potential business, or ban the static mining if they are not in SOLO mode. Otherwise I see a future in which your $17,000 salary continuing to drop the rest of the year and may end up costing you before its over.
I think you misunderstood what the issue is here. You're confusing the configuration error, which was just a number that led to greater payouts in a database, with a block withholding attack. The core issue, which is the block withholding attack, was not caused by the database entry.

You may be going a little bit overboard here with the scope of the problem. There is no need to make wide-ranging bans. The problem is specific to Nicehash. No other cloud mining services have been identified as having low luck or exploiting bugs, and our experience is that the owners of other services have been more responsive to abuse. If we find that other services do have a problem, we'll take action at that time.
User avatar
CSZiggy
Posts: 691
Joined: Wed Jan 31, 2018 2:44 pm

Re: Nicehash allowing block withholding and 51% attacks through its services

Post by CSZiggy » Fri Mar 22, 2019 8:58 pm

So, the problem is with nicehash, but they are not going to stop it because they make money off of it. Assume they fess up, admit its all on them and they allow it and they have no plans to stop it. Before the CEO stepped down after the hack in 2017 it was a Slovenia-run business. I am not sure who now owns/runs it but if they are not based out of the U.S. what recourse would the pool have? Knowing the source of the issue lets you target that single issue, but isn't this a symptom of a larger problem? Regardless of who caused the issue or why, what are YOUR options to stop it from your side? I came up with 4 different things the pool could try, if you have any others in mind please let us know what they may be.


1.Ban nicehash: work-a-round: still buy hashrate at nicehash and send it to a VPN or another cloudmining service so it can then be dumped back onto the pool from a non-banned location.

2.Ban ALL cloudmining-since some use multiple IP and ranges or like nicehash have several locations around the globe may end up being a LARGE ban list and needing constant updating and monitoring as the users request the services to open other IPs to be used to still connect to the pool.

3.Ban incoming workers/accounts that go over a certain amount of hashrate all at once, which means they open multiple accounts or buy from multiple places to get around the limitations of a single account(not put all eggs in 1 basket)

4.Ban the NON-SOLO static mining, so if the autopool doesnt select or notices an error it can self-correct, and if there is a problem and they solo mine but never find any blocks then they just do not get paid any profit and the pool doesnt suffer a $3000 loss.
User avatar
Steve Sokolowski
Posts: 3780
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Re: Nicehash allowing block withholding and 51% attacks through its services

Post by Steve Sokolowski » Sat Mar 23, 2019 8:27 am

CSZiggy wrote:So, the problem is with nicehash, but they are not going to stop it because they make money off of it. Assume they fess up, admit its all on them and they allow it and they have no plans to stop it. Before the CEO stepped down after the hack in 2017 it was a Slovenia-run business. I am not sure who now owns/runs it but if they are not based out of the U.S. what recourse would the pool have? Knowing the source of the issue lets you target that single issue, but isn't this a symptom of a larger problem? Regardless of who caused the issue or why, what are YOUR options to stop it from your side? I came up with 4 different things the pool could try, if you have any others in mind please let us know what they may be.


1.Ban nicehash: work-a-round: still buy hashrate at nicehash and send it to a VPN or another cloudmining service so it can then be dumped back onto the pool from a non-banned location.

2.Ban ALL cloudmining-since some use multiple IP and ranges or like nicehash have several locations around the globe may end up being a LARGE ban list and needing constant updating and monitoring as the users request the services to open other IPs to be used to still connect to the pool.

3.Ban incoming workers/accounts that go over a certain amount of hashrate all at once, which means they open multiple accounts or buy from multiple places to get around the limitations of a single account(not put all eggs in 1 basket)

4.Ban the NON-SOLO static mining, so if the autopool doesnt select or notices an error it can self-correct, and if there is a problem and they solo mine but never find any blocks then they just do not get paid any profit and the pool doesnt suffer a $3000 loss.
It's telling, I think, that Nicehash has had 24 hours to issue a reply and they still have yet to put out a meaningful statement.

Fortunately, it is possible to ban Nicehash miners. They identify themselves through a string at authorization time, so simply rerouting traffic is unable to bypass the method we use to implement the bans. Even if this method was able to be used, the extra latency involved with a VPN would likely increase the number of rejected shares to be render their rentals unprofitable.

In regards to other services, I'll again repeat that the issue is solely with Nicehash. We have not detected any other cloud mining services with this problem and those services are responsive to support tickets. This is not a theoretical issue that could happen someday; it's an actual issue that is caused by one specific company.

The way for Nicehash to prevent 51% attacks works for all of their customers. They can easily track the total amount of hashrate being sent to any particular coin across all customers, and even reject additional rentals to mine that coin. The possibility that customers will go to other services to conduct attacks, if those services allow them, isn't an excuse for Nicehash's failure to prevent attacks using their system.

As to #4, that sounds like a great idea. Instead of what I mentioned earlier, we will start out by only allowing Nicehash miners to solo mine the anchor coins, rather than banning all static coin mining on those coins. That way, if someone wants to try his or her luck on the lottery, (s)he can. That said, only 50 cents per day is typically spent mining those anchor coins with static coin miners, so I think the end result will be no different. The only time Nicehash miners ever show up on those coins is when someone is involved in criminal activity, or there is a bug.
jde
Posts: 9
Joined: Mon May 21, 2018 5:27 am

Re: Nicehash allowing block withholding and 51% attacks through its services

Post by jde » Wed Apr 03, 2019 4:10 am

Has there been evidence of suspected block withholding attacks on other algos/coins here? I know of some other PPS pools that have banned NiceHash due to the poor luck that it tends to bring.
User avatar
Steve Sokolowski
Posts: 3780
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Re: Nicehash allowing block withholding and 51% attacks through its services

Post by Steve Sokolowski » Wed Apr 03, 2019 10:00 pm

jde wrote:Has there been evidence of suspected block withholding attacks on other algos/coins here? I know of some other PPS pools that have banned NiceHash due to the poor luck that it tends to bring.
I haven't gotten around to analyzing that yet, because I want to fix some more critical issues first, like the Litecoincash errors that some people are receiving.

Which pools have banned Nicehash?
apollo8
Posts: 1
Joined: Sat Jul 06, 2019 1:38 pm

Re: Nicehash allowing block withholding and 51% attacks through its services

Post by apollo8 » Sun Jul 07, 2019 8:05 pm

I realize this is an old thread, but all miners that mine on Nicehash will profit (in the short term) if they withhold blocks. Basically this is because block withholding makes the global apparent hashrate fall, which makes their hashes more valuable if they use a PPS pool. Nicehash is a PPS pool but Nicehash is set up to forward the cost of PPS on to the buyers. The buyer can dilute the effect of block withholding by forwarding the hash on to a pool. Which in the end causes the honest miners on the pool to lose (in PPLNS cases) or causes the pool to lose (in PPS cases).

The situation is described in this reddit thread:
https://www.reddit.com/r/NiceHash/comme ... ng_buyers/
Post Reply