Update on the pool downtime

[Steve Sokolowski]

Hey Steve - thanks for addressing the payout issue clearly, it's going to become a top concern as the day goes forward.

You guys do what you have to do, I'll still be here, as will most of the rest of the community. These are the growing pains that plague online businesses that managed to catch attention of these cowards who don't have the balls to resolve grievances through discussion.

I don't think these people have any purpose at all, and they're actually pretty dumb. Would you:

1. Spend your own money
2. Not be able to get anything in return because you can't steal any money or hack any systems
3. Provide more evidence for us to add to the police reports so that you can live in fear of arrest

That doesn't seem to be very smart to me. If I were to do something like this, I would at least want to have some potential for gain in exchange for my money and the chance of spending years in prison.

[shyrwall]

Yes cloudflare does only http and https.

It's pretty simple what you have to do. Put the website on cloudflare like you already tried. Create a subdomain like pool.prohashing.com that goes to some backend ip at verizon. Then let verizon filter all traffic except tcp port 3333. If they have a stateful firewall then also tell them to enable syncookies and some rate limiting.

The prohashing setup was flawed from the beginning anyway since you only have a pool-server in the US which is too high latency for the rest of the world.

Done.

[FRISKIE]

By the way, I kinda like any approach that simply lets the attacker(s) run out of money

[shyrwall]

If I understood correctly prohashing has under 1gbit of bw to the servers (since steve said he might upgrade to 1gb). If so any kid with a decent home connection can be the one taking down prohashing atm. So $0 cost. Or the attacker could rent a VPS with 1gbit from zetservers for $10/month. So money is not really an issue heh.

[Steve Sokolowski]

Yes cloudflare does only http and https.

It's pretty simple what you have to do. Put the website on cloudflare like you already tried. Create a subdomain like pool.prohashing.com that goes to some backend ip at verizon. Then let verizon filter all traffic except tcp port 3333. If they have a stateful firewall then also tell them to enable syncookies and some rate limiting.

The prohashing setup was flawed from the beginning anyway since you only have a pool-server in the US which is too high latency for the rest of the world.

Done.

The lengthiest part of this whole process isn't actually setting anything up, but simply performing research on which service provider to use and what options are available.

Unfortunately, because of the way the pool works, there is no way to have servers across the world. Coin selection requires that there be a single core service that has the data in memory tracking what coins each person is mining, because that affects what other customers should mine. If we set up a second server elsewhere, it would not reduce ping times, because the second server would have to constantly communicate with the first server to find out what coins to assign. The latency would be longer than a direct connection from the miner.

Multiple servers would also not protect against an attack for the same reason - because there is still a single server that needs to track all the data in one place.

[Steve Sokolowski]

If I understood correctly prohashing has under 1gbit of bw to the servers (since steve said he might upgrade to 1gb). If so any kid with a decent home connection can be the one taking down prohashing atm. So $0 cost. Or the attacker could rent a VPS with 1gbit from zetservers for $10/month. So money is not really an issue heh.

That's the problem with these attacks and why it takes so long to fix them.

The pool can't be profitable by getting a 10Gbps connection because such connections are extraordinarily expensive. The fine print says that this is an instantaneous burst rate. The services that advertise these connections online for $10/month cut off your access as soon as you start to use the entire pipe for more than a few minutes. Lunarpages did that to me with a website in the past, and I even had to enter a dispute with DISCOVER over their charging my credit card.

The other problem is that these 10Gbps+ services require that you host your site on their servers, which is not an option when we're dealing with so much money.

I kept looking I think that it might actually be better to buy an Amazon server instance, which apparently has built-in DDoS protection. That way, we can install a proxy server ourselves and have full control over it, the cost is only $900 for 10TB of legitimate data, and I doubt that anyone could possibly take down Amazon's servers.

[shyrwall]

Yes cloudflare does only http and https.

It's pretty simple what you have to do. Put the website on cloudflare like you already tried. Create a subdomain like pool.prohashing.com that goes to some backend ip at verizon. Then let verizon filter all traffic except tcp port 3333. If they have a stateful firewall then also tell them to enable syncookies and some rate limiting.

The prohashing setup was flawed from the beginning anyway since you only have a pool-server in the US which is too high latency for the rest of the world.

Done.

The lengthiest part of this whole process isn't actually setting anything up, but simply performing research on which service provider to use and what options are available.

Unfortunately, because of the way the pool works, there is no way to have servers across the world. Coin selection requires that there be a single core service that has the data in memory tracking what coins each person is mining, because that affects what other customers should mine. If we set up a second server elsewhere, it would not reduce ping times, because the second server would have to constantly communicate with the first server to find out what coins to assign. The latency would be longer than a direct connection from the miner.

Multiple servers would also not protect against an attack for the same reason - because there is still a single server that needs to track all the data in one place.

So just get a few well places VPS servers that just port forwards 3333 to the backend. And never expose the backend ip. You said vps servers to proxy but that would increase the latency. Just simple port forward is better.If the VPS servers are choosen wisely then latency should not be an issue. It can even decrease it for some people due to shorter transit paths. And you can ratelimit etc.


Regardless of solution. An exposed single point of failure should never have been setup.

[shyrwall]

If needed I can supply you with some 10G VPSes for free to forward through until you have stuff sorted out. If you manage to get access to the backend again.

[jaybizz]

Thanks for updating us. I've only very recently started using the service, and it obviously sucks having the downtime given how profitable my miners have been here, but making sure to get it right this time is more important in the long run. It's not shocking that some moron gets a chuckle out of messing shit up for other people.. it's just the sad state of the world right now.

[Steve Sokolowski]

Here's an update on what Chris is doing:

He contacted Javapipe and was able to set up a DoS protection account. He started with the minimum account, which provides 10Gbps of protection, and can increase it if it turns out that the attackers are willing to spend more money than that. The account provides a "protected IP" and forwards filtered traffic through to the servers.

He modified the new Debian router that he set up for the first attack to rewrite packets with Javapipe's traffic.

Then he spent almost two hours on the phone with Verizon, after having done so earlier in the morning as well. Whatever happened in this attack caused their routers to malfunction and they still can't figure out why traffic to those two IP addresses isn't making it through their network. In the end, they had to escalate a ticket to the highest level of support. The support probably won't be available until 5:00pm on Monday.

The system requires four IP addresses to work, but we only have three. Chris is going to try taking the proxy.prohashing.com system offline and using its IP address for more essential services until Verizon figures out what happened to its network. After that, he'll configure the "protected IP" to redirect traffic to one of our remaining addresses to bring the system back online. We don't know whether that IP address has ever been exposed to the attackers before, as it has only ever been used to execute trades at exchanges. If an insider at the exchanges is responsible for the attacks, then they will continue.

Chris hopes to restore services sometime this afternoon. However, if the Javapipe protection is insufficient, or the attacker somehow figured out that IP address, or (because he didn't know anything about how to prevent these attacks before today) makes a mistake in setting things up, then it's likely we won't be able to return until Monday evening.

Chris will execute payouts after the system gets back online. In addition to lost revenue, the attacks caused a huge amount of lost profit because the price of everything rose across the board, and the trader was not able to buy coins before prices rose. That left the pool in an unusual "negative reserve" situation. He'll have enough money for payouts when either the system mines a few more blocks and builds up a reserve. If the system can't be brought online soon, it would take an ACH transfer clears to convert dollars from his personal account into the payout coins. ACH transfers are so slow that it's almost certain that we'll be able to get the system online and mine up a reserve before it would be able to complete.

Thanks for your patience! Chris will update you in a few hours when we make the first attempt.