Status as of Friday, June 16, 2017

[Steve Sokolowski]

Hi. Here's today's status:

• I apologize for the downtime last night. As soon as we fix one problem, it seems like another comes up. In this case, the software had been running great since the database optimizations, and then what seems to be some sort of attack occurred. Chris investigated the "attack" and replaced the hardware router by installing a new copy of Debian and using fail2ban to kill connections from the "attackers."
• "Attack" is in quotation marks because it wasn't a particularly successful attack, and it actually helped us out immensely. These sort of attacks are what criminals use when they realize they aren't smart enough to find vulnerabilities that can deal any actual damage. The attacker wasn't able to steal any money or log into any systems, and what I infer is that the criminals ended up directing a huge number of packets at Verizon's edge routers, which must have some sort of automatic attack detection and dropped almost all of them before they entered Verizon's network. The remaining packets overloaded the memory of the cheap ASUS router behind our servers, so Chris simply installed a Debian virtual machine and configured it as a router to be able to drop connections before they caused CPU load on any servers. The end result was that someone spent their own money to help us find that this router would not have been able to handle the load in the future when the pool grew and needed more connections. Thanks for that.
• This weekend, my main goal will be to reduce the amount of data the system stores in the database. During the last round of optimizations, we realized that there is a huge amount of data, like block hashes of shares, that is never used and can be deleted. By deleting data, we can improve database insert times, reduce memory usage, reduce the CPU usage necessary to copy all these strings around, reduce bandwidth usage for transferring backups, reduce the cost of disks, and a lot more. I'll post an update on how much data we were able to delete, and we will release a fix at the end of the weekend to not store it at all. For now, we are ahead of performance requirements, and this should put us in an even better position for SHA-256 mining.

[GregoryGHarding]

with the exception of the downtime trade off... id call that a win win

[GregoryGHarding]

just a side note, with the ip changes last night along with dns and firewall changes, i did not have to re-auth my login after the site came back this morning. i figured id either have to relogin or at least reauth my 2FA. is this a security oversight or just me overthinking it?

[mrgoldy]

Nice! is there an ETA on the SHA-256?

[nostec]

Hi. Here's today's status:

• I apologize for the downtime last night. As soon as we fix one problem, it seems like another comes up. In this case, the software had been running great since the database optimizations, and then what seems to be some sort of attack occurred. Chris investigated the "attack" and replaced the hardware router by installing a new copy of Debian and using fail2ban to kill connections from the "attackers."
• "Attack" is in quotation marks because it wasn't a particularly successful attack, and it actually helped us out immensely. These sort of attacks are what criminals use when they realize they aren't smart enough to find vulnerabilities that can deal any actual damage. The attacker wasn't able to steal any money or log into any systems, and what I infer is that the criminals ended up directing a huge number of packets at Verizon's edge routers, which must have some sort of automatic attack detection and dropped almost all of them before they entered Verizon's network. The remaining packets overloaded the memory of the cheap ASUS router behind our servers, so Chris simply installed a Debian virtual machine and configured it as a router to be able to drop connections before they caused CPU load on any servers. The end result was that someone spent their own money to help us find that this router would not have been able to handle the load in the future when the pool grew and needed more connections. Thanks for that.
• This weekend, my main goal will be to reduce the amount of data the system stores in the database. During the last round of optimizations, we realized that there is a huge amount of data, like block hashes of shares, that is never used and can be deleted. By deleting data, we can improve database insert times, reduce memory usage, reduce the CPU usage necessary to copy all these strings around, reduce bandwidth usage for transferring backups, reduce the cost of disks, and a lot more. I'll post an update on how much data we were able to delete, and we will release a fix at the end of the weekend to not store it at all. For now, we are ahead of performance requirements, and this should put us in an even better position for SHA-256 mining.

chief dead?

[Steve Sokolowski]

just a side note, with the ip changes last night along with dns and firewall changes, i did not have to re-auth my login after the site came back this morning. i figured id either have to relogin or at least reauth my 2FA. is this a security oversight or just me overthinking it?

It's not a security oversight - the webserver was kept running, and it just changed IP addresses.

If you had changed IP addresses, that would have been different.

[GregoryGHarding]

just a side note, with the ip changes last night along with dns and firewall changes, i did not have to re-auth my login after the site came back this morning. i figured id either have to relogin or at least reauth my 2FA. is this a security oversight or just me overthinking it?

It's not a security oversight - the webserver was kept running, and it just changed IP addresses.

If you had changed IP addresses, that would have been different.

that makes sense. thanks