Page 1 of 1

Status as of Monday, September 25, 2017

Posted: Mon Sep 25, 2017 6:45 pm
by Steve Sokolowski
There are two issues of note this evening: a new release, and DDoS attacks.
  • Chris will issue a new release tonight. It addresses two issues that he thinks are responsible for a lot of the customer support tickets with vague complaints.
  • First, he found that some mining software has a "worker name" textbox, which actually appends the worker name to the username, like "username.worker name" without the user's knowledge. This caused failures to authenticate. Since periods are already prohibited in usernames, it is simple to split the usernames on periods and use the part after the "." as a worker name. However, "n=" names will still override any mining software that provides usernames like this, and we won't document this feature to reduce confusion.
  • Second, I modified the behavior when the server detects a duplicate worker name. We still need to enforce the unique worker name problem, because I don't know what will happen on the UI if we allow names to be duplicated. Now, instead of disconnecting the user for violating uniqueness, an error message is displayed instead and the miner's name is replaced with a UUID. The error remains until the miner replaces the name.
Another issue that Chris is investigating is DDoS attacks. Apparently, someone's website is being DDoS attacked, and it turns out that our system is one of many performing these attacks. Further investigation reveals that while we paid a lot of attention to making sure nobody could steal any money, we never considered that someone could put code in a coin daemon to send SYN floods to a site. It is possible that the network connectivity issues that have plagued the system for months may be due to this attack. It also explains why it cost an additional $210 for bandwidth last month - the floods seem to be programmed to occur at specific times and we've never been looking at those times.

Chris is trying to track down which coin is causing the problem. Exchanges that are going offline may be having troubles with this coin too. Once he figures it out, he'll pay out balances for that coin, delete it, and then we should see a dramatic improvement in network connectivity. Our goal will be to 51% attack this coin to destroy the network and inform exchanges to delist it, so that we can make a statement to people that this sort of behavior is unacceptable.

Re: Status as of Monday, September 25, 2017

Posted: Mon Sep 25, 2017 6:50 pm
by simonjbcmm
You and Chris are truly very talented individuals. Thank you for your hard work and attention to detail. It is a privilege to have an account with your prohashing service.

With appreciation,
Jonathon

Re: Status as of Monday, September 25, 2017

Posted: Mon Sep 25, 2017 6:51 pm
by Alex
You mentioned "programmed to occur at specific times" would it be helpful if you knew when our miners switched to their respective backup pools or I am going down the wrong path?

Re: Status as of Monday, September 25, 2017

Posted: Tue Sep 26, 2017 12:37 am
by vinylwasp
Steve Sokolowski wrote:There are two issues of note this evening: a new release, and DDoS attacks.

Another issue that Chris is investigating is DDoS attacks. Apparently, someone's website is being DDoS attacked, and it turns out that our system is one of many performing these attacks. Further investigation reveals that while we paid a lot of attention to making sure nobody could steal any money, we never considered that someone could put code in a coin daemon to send SYN floods to a site. It is possible that the network connectivity issues that have plagued the system for months may be due to this attack. It also explains why it cost an additional $210 for bandwidth last month - the floods seem to be programmed to occur at specific times and we've never been looking at those times.

Chris is trying to track down which coin is causing the problem. Exchanges that are going offline may be having troubles with this coin too. Once he figures it out, he'll pay out balances for that coin, delete it, and then we should see a dramatic improvement in network connectivity. Our goal will be to 51% attack this coin to destroy the network and inform exchanges to delist it, so that we can make a statement to people that this sort of behavior is unacceptable.
Steve, the first thing to do is to stop allowing any of your coin daemon servers to initiate direct connections to unknown hosts on the internet. That's a fairly standard egress rule for most enterprises. If they need connectivity use an internal web proxy/email relay/dns relay.

Syn attacks are very old school, if they're attacking a website, they've probably hardcoded one of the many versions of LOIC, HOIC or Slowloris available on GitHub into the source code.

The big question is whether you're DoSing a single website or even worse, you're part of a bot-net which is being used (sold) to DDoS multiple websites. If you're sure it's just one, the domain name or IP address is probably hardcoded into the binary or in a config file somewhere (assuming they're not too sophisticated and haven't encrypted it) so you might try using a suitable binary search tool to look for it.

Happpy hunting.

Re: Status as of Monday, September 25, 2017

Posted: Tue Sep 26, 2017 8:08 am
by Steve Sokolowski
vinylwasp wrote:
Steve Sokolowski wrote:There are two issues of note this evening: a new release, and DDoS attacks.

Another issue that Chris is investigating is DDoS attacks. Apparently, someone's website is being DDoS attacked, and it turns out that our system is one of many performing these attacks. Further investigation reveals that while we paid a lot of attention to making sure nobody could steal any money, we never considered that someone could put code in a coin daemon to send SYN floods to a site. It is possible that the network connectivity issues that have plagued the system for months may be due to this attack. It also explains why it cost an additional $210 for bandwidth last month - the floods seem to be programmed to occur at specific times and we've never been looking at those times.

Chris is trying to track down which coin is causing the problem. Exchanges that are going offline may be having troubles with this coin too. Once he figures it out, he'll pay out balances for that coin, delete it, and then we should see a dramatic improvement in network connectivity. Our goal will be to 51% attack this coin to destroy the network and inform exchanges to delist it, so that we can make a statement to people that this sort of behavior is unacceptable.
Steve, the first thing to do is to stop allowing any of your coin daemon servers to initiate direct connections to unknown hosts on the internet. That's a fairly standard egress rule for most enterprises. If they need connectivity use an internal web proxy/email relay/dns relay.

Syn attacks are very old school, if they're attacking a website, they've probably hardcoded one of the many versions of LOIC, HOIC or Slowloris available on GitHub into the source code.

The big question is whether you're DoSing a single website or even worse, you're part of a bot-net which is being used (sold) to DDoS multiple websites. If you're sure it's just one, the domain name or IP address is probably hardcoded into the binary or in a config file somewhere (assuming they're not too sophisticated and haven't encrypted it) so you might try using a suitable binary search tool to look for it.

Happpy hunting.
I think the better solution here is to make permanent what Chris did already. We'll simply count SYN packets, and if there are an unreasonable number coming from a username, then an E-Mail is sent to Chris notifying him of the problem. We're still waiting for that E-Mail, so no attacks have happened yet.

One of our goals is that every time we encounter a problem, we want to permanently fix it so that the system becomes more stable in the future. Since this guy pointed out the attacks on his website, it will now likely never be possible for a coin network to conduct SYN attacks because exchanges will also detect and discontinue the coin causing the attacks.

Unfortunately, we can't block outgoing connections because the way that coins work is that they discover new peers by asking existing peers for addresses of those peers. The only way we could block outgoing connections is if we were able to obtain a large list of initial peers, and guarantee that those peers would never go offline. Since we can't do that, we'll have to settle for the second best choice, which is detecting each type of attack as we encounter it and making sure that nobody will be able to use it again.

Re: Status as of Monday, September 25, 2017

Posted: Tue Sep 26, 2017 10:17 am
by tomos81
Steve thanks for this update, now nicehash accepting higher hashrate to rent, i dont care of error with duplicate names in the PH website earnings page :)

Re: Status as of Monday, September 25, 2017

Posted: Tue Sep 26, 2017 10:37 am
by zafar177
Reading these posts make realise my lack of knowledge and your impressive expertise. Great work guys.

Re: Status as of Monday, September 25, 2017

Posted: Tue Sep 26, 2017 4:49 pm
by Kfedorek
Great work guys. That is very crazy about the ddos. What a crazy way to perform a ddos but very scary for you guys and us as well! Best of luck. Thanks again for the updates and tremendous work you guys put in.

Re: Status as of Monday, September 25, 2017

Posted: Wed Sep 27, 2017 5:08 am
by pumaro
I point my miners through miningrigrentals which uses 3 possible locations to hash ; East, Central, and West is it possible that the duplicate names are coming from two locations at once, maybe load sharing or fail over causing this issue?

us-west01.miningrigrentals.com:3333
us-east01.miningrigrentals.com:3333
us-central01.miningrigrentals.com.3333

- Ron

Re: Status as of Monday, September 25, 2017

Posted: Thu Sep 28, 2017 3:30 pm
by Denis
Dear Dev Team,

I am trying to get total hashpower suscribing "f_all_miner_updates" RPC. On first call I get all miners, more than 10. But next call return just 1 miner data. So mostly my hashpower value is wrong.

Best,
Den