Update on the pool downtime

[agrip666]

Когда будете работать снова ???

Они сами пока не знают. 60%, что сегодня. 80%, что завтра.

[Steve Sokolowski]

I don't think these people have any purpose at all, and they're actually pretty dumb. Would you:
1. Spend your own money
2. Not be able to get anything in return because you can't steal any money or hack any systems
3. Provide more evidence for us to add to the police reports so that you can live in fear of arrest

That doesn't seem to be very smart to me. If I were to do something like this, I would at least want to have some potential for gain in exchange for my money and the chance of spending years in prison.

If they're another Pool operator that directly benefits from PH being down, then there's a strong economic motivation to keep DDoSing.
If they do it for long enough their strategy may be to cause you sufficient loss of revenue and cost to remediate to ultimately drive you out of business. I hope this doesn't happen of course but there are plenty of precedents for competitor DDoS attacks.

DDoS attacks are actually pretty cheap. Just google "web site stressor" and you'll find services (sic) for a few bucks per hour.
Javapipe sounds promising, my only concern would be increased latency, but we'll just have to wait and see what the impact of that might be.

I'm back. The latency appears to be about 20ms, or a 0.2% reduction in profit.

I found out today that Verizon has an "enterprise" level, which supposedly has support on the weekends. Apparently "business" is only for beginners. The problem is that "enterprise" costs $5000/month to start, a ridiculous price that would make it difficult to make any profit.

Chris will be getting up closer to 8, the first time he can get Verizon on the phone. He wasn't able to do anything since last night because three of our five IP addresses are now affected by whatever happened, and we need a minimum of three for the system to function.

[Steve Sokolowski]

I don't think these people have any purpose at all, and they're actually pretty dumb. Would you:
1. Spend your own money
2. Not be able to get anything in return because you can't steal any money or hack any systems
3. Provide more evidence for us to add to the police reports so that you can live in fear of arrest

That doesn't seem to be very smart to me. If I were to do something like this, I would at least want to have some potential for gain in exchange for my money and the chance of spending years in prison.

If they're another Pool operator that directly benefits from PH being down, then there's a strong economic motivation to keep DDoSing.
If they do it for long enough their strategy may be to cause you sufficient loss of revenue and cost to remediate to ultimately drive you out of business. I hope this doesn't happen of course but there are plenty of precedents for competitor DDoS attacks.

DDoS attacks are actually pretty cheap. Just google "web site stressor" and you'll find services (sic) for a few bucks per hour.
Javapipe sounds promising, my only concern would be increased latency, but we'll just have to wait and see what the impact of that might be.

One of the interesting parts here that I'll write about in a post later is that there's a difference between us and places like NiceHash.

I'm told these attacks are pretty common, so once in four years is a pretty low frequency to be attacked. I suspect that part of the reason is that these other pools that don't publish their real identities and have no phone number or E-Mail address have no legal recourse. Many of them are probably committing tax fraud, so they can't go to the police to take action against attackers.

On the other hand, because we do pay our taxes, we can involve the police. These attacks pushed the total damages over $10,000, which means that the FBI will now take a report. I'm going to task Chris next weekend with filing a report with the FBI so that this is on record with them. I would assume that such a small amount of money is not enough for them to care about us, but when other sites are attacked, this does allow a "correlation attack" against the perpetrator that could allow evidence to be gathered. For example, they may be perfectly secure in not revealing their source IPs during this attack, but it could be found that one of these IPs logged into some other site they are investigating and leaked information there, and they didn't know what the information meant until they had the context that they wanted to try to get into a mining pool.

After we've fixed this, we need to make it clear that we don't fool around with criminal activity. Even if no arrests are made, the criminals know that the stakes are higher because cops will be looking for them when they attack us. When they attack NiceHash, they know the cops aren't after them because NiceHash isn't willing to reveal who actually owns the site. This is probably what has for four years, and should in the future, encourage criminals to attack other sites instead.

[Steve Sokolowski]

The forums will be going offline for an hour while Chris changes their IP address. See you when they return!

[FRISKIE]

It's interesting that NH hasn't been attacked this whole time - just sayin'

[Steve Sokolowski]

Chris called Verizon and they said that they were going to change the IP addresses at 10:00am, but they haven't done that yet. He called Comcast Enterprise Solutions and they are going to have someone call him back to discuss services.

[Chris Sokolowski]

I am finishing the configuration now. Forums are back online. You may need to flush your DNS cache. Mining will be back in an hour or two.

[JoeTheMiner]

I am finishing the configuration now. Forums are back online. You may need to flush your DNS cache. Mining will be back in an hour or two.

Awesome thanks for the update!

[Steve Sokolowski]

The system is coming back online now.

Verizon finally changed the IP address block yesterday. Chris signed up for Javapipe's VPS service and configured a VPN so that all traffic now goes through the VPN to the mining system. The VPS server at the site's IP addresses has a firewall in front of it that drops bad packets before they get to our system. The VPN adds about 20ms of latency, so profitability will be reduced by 0.1% or so due to increased orphan rates for the time being. All of the packets inside the VPN appear as if they were coming from the original source, reducing the probability of bugs that would occur if there were a "proxy server" setup.

If the attackers pay more to increase their bandwidth, we have the ability to spend a few hundred dollars and get more bandwidth to compensate. It's possible that the attackers may return and exceed the amount of bandwidth we have now, in which case the site might go offline again for a few minutes while we call Javapipe and have them increase their bandwidth.

The VPN setup took Chris about 28 hours of work over two days. He hasn't slept or eaten much during that time. He did learn a lot about VPNs and Linux networking, though, so it was time well spent. After the system is stable, Chris has scheduled a site inspection with Comcast Enterprise Solutions, which should be able to reduce latency by eliminating the need for a VPN. They said they have support on the weekends, too.

Services will come back online one at a time. The forums came back first to test that a simple website would work. E-mail is coming online now, and the other sites hosted on this server also just came back. The coins are going to be online soon to download blocks, and the trader will rebalance our wallets to make sure we have the right coins. We can't resume mining until most coins have downloaded the latest blocks.

The website should be online in about 90 minutes, and Chris expects to resume mining in two hours.

[FRISKIE]

@ Everyone - be sure to set Chris a tip for this work marathon!