Miners are centralizing into huge farms

Read about Prohashing, mining, and coins in general!
Forum rules
Sign up to be notified of new posts to the Prohashing Blog at http://eepurl.com/gx1e0j
User avatar
Steve Sokolowski
Posts: 3807
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Miners are centralizing into huge farms

Post by Steve Sokolowski » Tue Jul 09, 2019 5:48 pm

Earlier in the weekend, I posted a review of the July 3-4 meeting that we held to discuss Prohashing's path forward over the next few months. The purpose of the meeting was primarily to answer the following question: why, given that almost everything else is exactly the same, are there fewer customers now than there were a year and a half ago?

After a lot of discussion, we realized that we had enough evidence to conclude the answer is that miners are centralizing into huge farms, owned by fewer people. This development should be alarming to community members and coin developers who are constantly pursuing decentralization of hashrate.

The logic for this conclusion was simple: if all of the initial conditions of an experiment are the same, then the outcome will be repeatable. If the outcome is different, then it follows that one of the initial conditions must also be different. We researched an exhaustive list of initial conditions and found that they are all the same except for one. There is either a cause that have never appeared in any Google search results about mining, or mining has become increasingly centralized, making pools less necessary.

In this article, I'll present a list of all the conditions we identified and why we believe that none of them have changed. Maybe you'll be able to post a reply and point out a condition that we're missing. Here's what we think has not changed:


-----------------------------------

The system is not as capable

In December 2017, shortly before peak users and hashrate were hit, we closed the site to new registrations. We parallelized the mining servers so that the system's capacity increased from 12,000 simultaneous miners to about 240,000 miners. Then, on the day that NiceHash lost its money, we released the update that allowed us to accept all the customers who wanted to join us.

The number of workers increased to about 20,000 in December 2017 and January 2018, and then fell back to 12,000 now - exactly the same amount as there had been, despite the system having twenty times the current capacity.


The system has more bugs

This condition is perhaps the most weird. Two years ago, there were days where the system would go offline for hours at a time, or not record shares, so that Chris would have to perform share corrections on a daily basis. Share recording would fall behind by hours, and at one point there were hundreds of support tickets backlogged.

One change I'm 100% confident has not occurred is that the system has more bugs than it did two years ago. On Sunday night, the number of support tickets submitted was exactly zero, the first time that had ever occurred. The state of the system now, compared to two years ago, is not even comparable in terms of stability.


There are fewer features

If the system had removed features that customers used, then it would be understandable that those customers might decide to leave. However, there aren't any features that were removed, and we added a number of features like grouping. As demonstrated in many recent polls, there aren't any new features that are in great demand. litecoinpool.org, which has not changed in years and which offers very basic functionality, maintained or grew its hashrate at the same time ours remained stable.


The market has shrunk

Last week, hashrate for the bitcoin network finally rebounded to its all-time high that had been set during the last cycle, and litecoin hashrate is in a similar position. Despite that, hashrate in both algorithms at Prohashing is about half what it was at the time of peak hashrate the last time around. The market did not shrink, but the hashrate is elsewhere.


Profitability is too low

To eliminate the possibility that low profits were deterring customers from mining with us, we started scrypt and SHA-256 miners at several pools to compare profits earned by mining at the pools. We focused on litecoinpool because their hashrate was so high (at 40 TH/s), and determined that our miners earned 1% more mining here than they did mining at litecoinpool. Even if we were only able to equal their profitability, it would still be logical that miners would prefer the pool where where they can be paid in any coin.


The site has poor marketing

One of the conclusions we drew from the July 3-4 meeting was that Prohashing has poor marketing. We decided to budget $50,000 to attack this problem head-on, by producing a lot of content and positioning these forums as a widely-used and viable alternative to the censored bitcointalk.org forums. Someone is going to come onboard to produce "how to" YouTube videos for every type of miner. But this lack of marketing can't be a changed factor because we never performed much marketing in the first place. In fact, the charts page is largely the same as it appeared during the higher-visitation period, except recent work on performance meant that it took about twice as long to load back then.


Mining rigs have changed and become incompatible

An interesting idea that was brought up was that new mining rigs are incompatible with the system. We did receive some support tickets from customers about rigs that didn't perform as well, but those reported issues were addressed and there have been no additional reports since then. Additionally, testing performed by us with newer miners, like the Antminer E3, failed to reveal any incompatibilities. Furthermore, since the cycle is just getting started, many mining manufacturers do not have to funds, or have not yet had the time, to design and release new hardware.

-----------------------------------


Given that we eliminated all of the factors we could identify as having remained unchanged since the last cycle, we needed to search for things that were both reasonably probable and which we had not already considered. The only answer that remained was that mining is centralizing into large mining farms. If the price of electricity is rising or if margins are getting squeezed, it may be that locating miners in huge warehouses is now the only way to survive. These farms, of course, would be solo mining because they have enough hashrate to eliminate the possibility of never finding any blocks due to variance.

If correct, then the implications of this theory are disturbing. The theory implies that 60% of the hashrate owned by independent miners during the last cycle has now gone offline, replaced by huge mining farms. In the recent Bitcoin Cash fork, we saw how two large pools colluded to execute a 51% attack against the network, which caused mega-exchange Coinbase to lock its wallet while it performed an investigation into why transactions has rolled back. In that case, it was obvious who had executed the attack. Now, the evidence demonstrates that hashrate has moved to private mining farms that anonymously submit blocks to the networks, and which for many coins may be more powerful than all of the known pools.

Fortunately, we have more than enough income to prosper even at current levels, and will be able to continue our push to improve our services. However, others should be as concerned as we are about the worrisome trend of miners moving to independent, anonymous farms, where they are stripped of any real accountability while gathering more hashrate every day.
RCMco
Posts: 15
Joined: Mon Feb 13, 2017 5:50 pm

Re: Miners are centralizing into huge farms

Post by RCMco » Tue Jul 09, 2019 8:12 pm

Did you look at any other pools besides litecoinpool ? I am wondering if pools in general lost miners do to a lack of profitability and some went to litecoinpool giving the appearance that they have remained stable when in fact they too had a drop.

Or, maybe the 5% fee? Litecoinpool.org does not have a fee since merged mining began.

I am mining with a skeleton farm right now with 13 of my 23 Antminer L3+ units and 8 of my 12 Antminer S9 units sitting idle due to heat. The AC system I have for the farm cannot handle everything this time of year.
User avatar
Steve Sokolowski
Posts: 3807
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Re: Miners are centralizing into huge farms

Post by Steve Sokolowski » Wed Jul 10, 2019 7:22 am

RCMco wrote:
Tue Jul 09, 2019 8:12 pm
Did you look at any other pools besides litecoinpool ? I am wondering if pools in general lost miners do to a lack of profitability and some went to litecoinpool giving the appearance that they have remained stable when in fact they too had a drop.

Or, maybe the 5% fee? Litecoinpool.org does not have a fee since merged mining began.

I am mining with a skeleton farm right now with 13 of my 23 Antminer L3+ units and 8 of my 12 Antminer S9 units sitting idle due to heat. The AC system I have for the farm cannot handle everything this time of year.
I reviewed a number of pools and found that, in general, the ones I had checked previously had fewer customers. Of course, I can't make any statements about pools that didn't exist when I checked last year.
mike0204
Posts: 15
Joined: Sat May 19, 2018 5:09 pm

Re: Miners are centralizing into huge farms

Post by mike0204 » Wed Jul 10, 2019 8:33 pm

Steve,

Steve,

I think that your conclusion is 100% correct. It is concerning, but I think that the cause is economics and an "After the Gold Rush" effect.

The economics are pretty simple. In Q1, I sold most of my miners for $2,000 and bought several mining contracts with a big centralized miner. Last month, I made ~$40 per day in profit. Had I kept my old miners, I would have made perhaps $12 per day. Even if I had been lucky and bought a new S17 Pro @56Thz/s, I would have only made about $15/day profit. Granted, my contracts only last for 1 year, but as you have stated many times, technology moves fast. Plus, my contracts will end before the halving. Who knows what will happen to mining profits after that.

The "After the Gold Rush" effect is pretty simple too. Prior to 2018, mining was a niche hobby that attracted little corporate interest. There were booms and busts. The Bitcoin rise to $20K was the event that attracted interest from both corporations and governments. Since then, most of the profits have arbitraged away, effectively ending the gold rush. Profitability is now determined only by your cost of electricity and the value of bitcoin when you entered the market. If you are in an area of even reasonable electricity costs, you can't compete with your own miner. You can also pretty much forget about solo mining any coin of value.

I do think that your site provides value, particularly from the aspect of being able to receive payment in lots of different coins. If I were you, I would consider getting on the platform of one of the big miners. Better to ride the wave than get crushed under it.

If it's of any value, I'd be happy to send you the spreadsheet that I use to track my small operation. Just let me know.

Best regards,

Mike
dra2120
Posts: 24
Joined: Thu Aug 16, 2018 9:47 am

Re: Miners are centralizing into huge farms

Post by dra2120 » Thu Jul 11, 2019 9:58 am

I don't think it is quite as nefarious as it sounds, but you are correct. When the market dropped, many if not most miners stopped mining because it was costing more to run the machine than they were making. When I was at the Boston Blockchain Conference, many of the people who ran big mining farms (30K+ machines) said that this was great for them, because they have positioned their operations where electricity is so cheap that it doesn't matter what machine they are using they will still make a profit. So they said that they are snatching up tons of "older" machines for next to nothing, cleaning them up, and then putting them to work in their mega farms. While this makes perfect business sense on their end, on the side of "hobby" miner. (what I would classify as 50 machines or less.) this moves the majority of hashing power and cheap equipment to the mega miner, forcing the hobby miner to purchase the newest and most expensive equipment. This I think is why there has been a down turn in smaller operations. Everyone sold their machines, and now the cost to get back in is too high, especially with the volatility of the market and no-one knowing if they will spend 5,6, 10K on equipment only to be in an upside-down market again.

On another note, I think the fee scares a lot of people away. I think you guys should get your fair share, and I think 5% is reasonable, but everyone I've told about the pool has asked what the fee is as their first question, and when I tell them, they say no thanks...It doesn't matter to them that they can be paid in any coin, or that they will likely make more money with coin switching. they hear the fee and are immediately out...Maybe there is some way to integrate the fee in a different way, that still gets you paid, but isn't as scary to established miners who are used to a much lower pool fee.
User avatar
Steve Sokolowski
Posts: 3807
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Re: Miners are centralizing into huge farms

Post by Steve Sokolowski » Thu Jul 11, 2019 10:07 am

dra2120 wrote:
Thu Jul 11, 2019 9:58 am
I don't think it is quite as nefarious as it sounds, but you are correct. When the market dropped, many if not most miners stopped mining because it was costing more to run the machine than they were making. When I was at the Boston Blockchain Conference, many of the people who ran big mining farms (30K+ machines) said that this was great for them, because they have positioned their operations where electricity is so cheap that it doesn't matter what machine they are using they will still make a profit. So they said that they are snatching up tons of "older" machines for next to nothing, cleaning them up, and then putting them to work in their mega farms. While this makes perfect business sense on their end, on the side of "hobby" miner. (what I would classify as 50 machines or less.) this moves the majority of hashing power and cheap equipment to the mega miner, forcing the hobby miner to purchase the newest and most expensive equipment. This I think is why there has been a down turn in smaller operations. Everyone sold their machines, and now the cost to get back in is too high, especially with the volatility of the market and no-one knowing if they will spend 5,6, 10K on equipment only to be in an upside-down market again.

On another note, I think the fee scares a lot of people away. I think you guys should get your fair share, and I think 5% is reasonable, but everyone I've told about the pool has asked what the fee is as their first question, and when I tell them, they say no thanks...It doesn't matter to them that they can be paid in any coin, or that they will likely make more money with coin switching. they hear the fee and are immediately out...Maybe there is some way to integrate the fee in a different way, that still gets you paid, but isn't as scary to established miners who are used to a much lower pool fee.
The fees are not the cause of the decline. The fees were the same one year ago and now, so that can't be the cause. It does make sense, as you said, that some people are definitely deterred by fees, and they were neither customers before nor are they now because of the fees.

One of the biggest problems with fees is that I never get replies to the answer of what the fees should be, so they stay the same. In the past month, there have been four support tickets asking for special deals or lower fees, and when I asked each one what fee they wanted, there was no response. A lack of responses has been a problem for months. The exact fee level makes a huge difference because then we can figure out what services we should discontinue.

Until we can get some answers as to what fees people want, we unfortunately can't make any changes. Are you willing to suggest what you think the fee should be?
Last edited by Steve Sokolowski on Thu Jul 11, 2019 6:12 pm, edited 1 time in total.
XXXJAYXXX
Posts: 14
Joined: Fri Feb 01, 2019 3:54 am

Re: Miners are centralizing into huge farms

Post by XXXJAYXXX » Thu Jul 11, 2019 3:02 pm

I agree to some extent.. however.... I think the coming rise in price will change the tides considerably. I'm sure there are plenty of people like me out there who have a stack of old mining gear just waiting on the sidelines for the prices to once again out-pace the hash rate. I'm pretty sure BTC hash rate is only increasing due to the new antminers hitting the market.. lots of commercial farms are switching over to these new machines that are still priced out of reach of the average hobby miner.

I think the part you have wrong here is that bigger farms are not actually buying up older machines... they are buying brand new machines that are 4x faster than what the average hobby miner can afford/justify.. Smart hobby miners are either sitting on their older machines.. or even buying them up at significant discount just waiting for the price to come up high enough so they can turn them back on. Basically it's a waiting game at this point.. once the prices/profitability come back you should see an increase in miners on the pool again... I remember at the start of 2017.. the S7 became unprofitable for most people at one point as the S9 came online... then a few weeks later it was back in the green due to the price spike and it remained profitable for most of 2017... I think we will see the same thing happen again on a larger scale this time. I actually see a squeeze coming on the mining equipment market... if the price goes up fast enough.. all of the older machines will start getting bought up.. and there are not currently enough manufacturers to supply the coming demand... A lot of ASIC manufactures went bust in the last market cycle.. and I have not seen many new startup's since then.. The "once bitten, twice shy" scenario I think will come into play here.. people will be far less likely to go "all in" on a new hardware manufacturing venture which will just add fuel to the hardware shortage fire... I have a stack of A4 Dominators just waiting to hit ebay when that time comes ;-)

As for the plan of action for prohashing.. My best suggestion would be to tighten up the coding/features of the website.. get rid of features that are not commonly used or are redundant... clean up the coin list so you don't have coins offered that you are not able to pay out (Im guessing this will alleviate the amount of support tickets you receive as well). Get rid of redundant links.. basically simplify and streamline the whole operation so when the flood of people does arrive.. you won't be slammed with glitches and support tickets. The #1 thing that causes me to find a new pool is glitches and downtime... I'm sure most hobby miners (which is your main market I'm guessing) feel the same way!

I do think your idea of advertising is a good one.. but the "entry" for this campaign should be carefully timed as to draw the most attention... Wait for profitability to become favorable.. and for the market to start the frenzy for hardware... then hit em with an ad blitz as the new crowd of miners comes online...

Just my 2 cents :-)
XXXJAYXXX
Posts: 14
Joined: Fri Feb 01, 2019 3:54 am

Re: Miners are centralizing into huge farms

Post by XXXJAYXXX » Thu Jul 11, 2019 3:20 pm

Steve Sokolowski wrote:
Thu Jul 11, 2019 10:07 am
dra2120 wrote:
Thu Jul 11, 2019 9:58 am
I don't think it is quite as nefarious as it sounds, but you are correct. When the market dropped, many if not most miners stopped mining because it was costing more to run the machine than they were making. When I was at the Boston Blockchain Conference, many of the people who ran big mining farms (30K+ machines) said that this was great for them, because they have positioned their operations where electricity is so cheap that it doesn't matter what machine they are using they will still make a profit. So they said that they are snatching up tons of "older" machines for next to nothing, cleaning them up, and then putting them to work in their mega farms. While this makes perfect business sense on their end, on the side of "hobby" miner. (what I would classify as 50 machines or less.) this moves the majority of hashing power and cheap equipment to the mega miner, forcing the hobby miner to purchase the newest and most expensive equipment. This I think is why there has been a down turn in smaller operations. Everyone sold their machines, and now the cost to get back in is too high, especially with the volatility of the market and no-one knowing if they will spend 5,6, 10K on equipment only to be in an upside-down market again.

On another note, I think the fee scares a lot of people away. I think you guys should get your fair share, and I think 5% is reasonable, but everyone I've told about the pool has asked what the fee is as their first question, and when I tell them, they say no thanks...It doesn't matter to them that they can be paid in any coin, or that they will likely make more money with coin switching. they hear the fee and are immediately out...Maybe there is some way to integrate the fee in a different way, that still gets you paid, but isn't as scary to established miners who are used to a much lower pool fee.
The fees are the cause of the decline. The fees were the same one year ago and now, so that can't be the cause. It does make sense, as you said, that some people are definitely deterred by fees, and they were neither customers before nor are they now because of the fees.

One of the biggest problems with fees is that I never get replies to the answer of what the fees should be, so they stay the same. In the past month, there have been four support tickets asking for special deals or lower fees, and when I asked each one what fee they wanted, there was no response. A lack of responses has been a problem for months. The exact fee level makes a huge difference because then we can figure out what services we should discontinue.

Until we can get some answers as to what fees people want, we unfortunately can't make any changes. Are you willing to suggest what you think the fee should be?
I think your fees should reflect the competitive nature of the market... if you can trim the fat on the website to minimize the amount of time your team needs to spend maintaining the website.. it should... in theory.. reduce the operating costs.. I think the question you need to ask yourself is "what can we afford to drop our fees to?"... Pretty much everyone on the site will tell you that a 0% fee would be most preferable.. but obviously this does not make sense from a business standpoint... I'm guessing the reason you have not received a response to your question.. is there are a lot of metrics at play that we do not have access to... What are your current margins? If you decrease the amount of time/work required to maintain the website... can you justify a lower fee? Considering there are other websites that are not quite as "pretty" as Prohashing.. but also offer auto-conversion and profit switching for only 0.75-1.15% fee(Zpool comes to mind) I can see why the 5% scares a lot of people away... It doesn't matter if your pool brings a miner 1% more than another pool... 5% fee comes off as "Greedy" when compared to your competitors.. and its a turn off regardless of how much nicer and more user friendly your interface is, or if the end result happens to be 1% higher earning for the miner. It's a psychological barrier.. most people make decisions on their emotions.. and if they feel 5% is a rip-off.. they won't care about the finer details. The question automatically becomes "If other pools can do it for 1-2%... why can't you?"

Just imagine the appeal it would bring if you could advertise the same 0.75-1.15% fee as your competitor.. but offer 3-5% higher earnings AND a more user friendly interface.... would that increase your traffic? I think it would....

Business 101 - Taking a short term 75% pay cut in order to increase your customer base 300-500% is not a bad move... you will make your money back in the long run if your company can support the move...
XXXJAYXXX
Posts: 14
Joined: Fri Feb 01, 2019 3:54 am

Re: Miners are centralizing into huge farms

Post by XXXJAYXXX » Thu Jul 11, 2019 3:53 pm

I also think KYC might have something to do with it... You are the only pool I have ever come across that requires this for payouts... Most are anonymous, or at most require only email validation. Crypto miners by nature are not favorable to government control... just food for thought..
jkndt
Posts: 9
Joined: Sat Nov 17, 2018 6:50 am

Re: Miners are centralizing into huge farms

Post by jkndt » Thu Jul 11, 2019 5:02 pm

Perhaps something else to consider is the variety of algorithms that are available.
I haven't added any more ASICs to Prohashing because my 9 most recent acquisitions aren't supported. I'm also near the point of power consumption saturation and will be selling off my least profitable per kW when I get more machines. It all depends on ROI as to the algorithm that I happen to buy for. My next planned purchase will go to prohashing, but if something else comes along with a better deal before, it may not be supported.

Regarding your fees, it is a difficult thing to decide upon. Obviously you need to earn money to pay yourselves and provide the time/equipment/service. As your service user, I want to maximise my earnings. I think 5% is OK, that's why you are still the primary pool on the miners I have that you support.
Post Reply