An overview of the DDoS attacks against us, and what they teach about customer service

User avatar
Steve Sokolowski
Posts: 3253
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA
Contact:

An overview of the DDoS attacks against us, and what they teach about customer service

Postby Steve Sokolowski » Sun Jun 25, 2017 8:50 am

Last weekend, just after we had expanded from another round of performance constraints, Prohashing was targeted by denial-of-service (DoS) attacks. This post will focus on two areas: how we eliminated the attacks, and how the success or failure of a business often comes down to customer service.

In the past, I wrote that DoS attacks, particularly distributed DoS attacks (DDoS), are one of the critical problems that will face society in the next decade. Currently, attacks mostly cause websites to go down. In a decade, attacks will probably cause your augmented reality goggles to cut out while you are in the middle of a party with people across the globe, or cause you to think more slowly when your neural interface can't obtain the data it needs. In 20 years, the attacks will cut off the communications between your nanobots while they are in the process of repairing your heart attack, causing you to die.

The major issue with DDoS attacks is that the cost of mitigating the attacks is greater than that of executing them. Businesses can afford to mitigate attacks, but the common person will have little defense when life without the Internet becomes unimaginable.

The attacks on our system began on Thursday, June 16, when a huge amount of traffic was directed at Verizon's network. Some sort of automatic protection must have kicked in after 10 minutes, allowing no traffic to reach the system. Chris called Verizon's support line, and during the interactive voice response (IVR) system's checks, he noticed that the system was accessible again. Everything returned to normal, and at the time, it was not clear that an attack had occurred. The connectivity issues continued for several hours, and because Chris didn't know that an attack was occurring, he spent time investigating other potential problems with the system. Every time the connection cut out, he found that the Verizon system's automated checks would resolve the problem without his needing to talk to a customer service representative.

On Friday night at the same time, the same condition recurred, but this time Verizon's routers at the edge of their network blocked traffic to the system and did not restore access when Chris called customer service. It was at this point he was able to determine that an attack might have occurred, when Chris came up with the theory that a large amount of traffic had caused the networking equipment at Verizon to fail. He was able to get a customer service representative on the line, but the representative was surprisingly incompetent about the primary service they provide (i.e. Internet networking.) Chris found, however, that the other IP addresses had not been deactivated, so he switched one of the servers and its DNS record to that IP address and the system returned to normal.

On Saturday night, the attacks occurred again, and this time they took out another IP address. Chris ran out of energy and fell asleep, simply being unable to continue working due to exhaustion. When he woke up, he called Verizon and was able to talk to an agent who actually got in contact with a technician. They troubleshot the blocked IP addresses for two hours and could not identify the cause of the problem. Chris was told that they would have to escalate the case to their highest level of support and that it would take two days to resolve.

Chris then asked whether he could simply purchase additional IP addresses so that he could figure out a mitigation technique, and ignore this problem for now. Verizon said that the department that handled sales could only take requests on Mondays through Fridays from 8:30 to 5:00. It was at this point that a third address failed due to whatever flaw was in Verizon's equipment, and Chris took the system offline because it could not function without at least three of the five IP addresses we possessed being available.

Chris then contacted Cloudflare, an attack protection provider. Their website was extremely confusing and written in "business speak," with few technical details about how what services they actually provide. He called them to try to get someone on the phone to actually learn about what they did, and they took two days to return his call. While he gave up with Cloudflare long before then, he eventually found out that Cloudflare is only able to protect websites because its limited number of IP addresses relies on HTTP request hostnames to forward the requests.

Later that Sunday he settled upon Javapipe, a provider that actually detailed what services they provided on their site. Our first thought was that we needed their standard DDoS protection server, which simply retransmits good packets it receives with the same source and destination IP addresses. We would directly reply as if the packets coming from us actually originated from them. However, we found after modifying our servers that some router upstream was blocking the packets we rewrote to appear to be coming from the Javapipe server. A support ticket to Javapipe suggested that we set up a virtual private network (VPN), which would send traffic in both directions through their servers so that the packets coming from us weren't blocked by the upstream router.

Having talked with Javapipe, Chris ordered a virtual private server (VPS) and configured it with OpenVPN to route all traffic to our servers and back out again. When Verizon's business office returned on Monday, he was finally able to get their sales department to allocate new IP addresses (they never did determine why those other IPs still don't work.) But when he started the system and as miners started to connect, he found that OpenVPN was singlethreaded and the system's core was overloaded. The core were overloaded because Javapipe's "latest processors" were actually 1st generation Core machines, and hardware encryption didn't become available until the 2nd generation machines were released.

Therefore, he had to order three more cores and set up three more OpenVPN tunnels, and then balance the load through all of them on Tuesday. Once he accomplished that, CPU load on each core dropped to just 25%, eliminating the random dropouts that had occurred as packets were unable to be processed.

Next, we found that the WAMP server that provides live profitability data was overloading the now CPU-limited connection because a small number of customers were making many connections. Additionally, we discovered that the messages being sent were inefficient, with it being possible to reduce their size by 95% through sending differences since the last message, rather than the entire message again. I began work on a "difference engine" to only send the differences between the last piece of data and the current piece, but in the meantime Chris discovered that Crossbar has an option for websocket compression. When he enabled it, traffic coming out of the Crossbar WAMP server was reduced from 60Mbps to 2Mbps, a 30x decrease. I scrapped the "difference engine" because it was believed that the risk of introducing new bugs would outweigh a further reduction in bandwidth.

The attacks have continued over the past few nights, but they are now ineffective. After about 2s of downtime, Javapipe's services begin to filter the traffic and everything returns to normal. As with the x11 mining release, we found that when one goes looking for bugs, one tends to find them. The bugs that one finds are often issues that have been around for a while that nobody had known about. The WAMP server bandwidth overload explained many issues that previously made little sense.

In the end, it took about one week to get the system back to normal operation. The downtime was about 4 days in total. The opportunity cost in lost revenue from money not earned during the downtime, customers who will take a long time to return, customers who will never return, the cost of all the protection services, the errors and bugs introduced by hasty introduction of untested code into a production system, the delay of SHA-256 mining by two weeks to deal with the attacks, thefts due to the delay in implementing E-Mail notification on payout address changes on the website, and the increased number of orphaned blocks caused by the VPN latency is estimated to be about half a million dollars. The attacks are easy to execute because the attackers likely spent a trivial amount in comparison, perhaps a thousand dollars. The problems could have been worse, however: the attacks happened at night on a weekend, when we were available to respond for as long as necessary to solve the problem. Even though Verizon business would have been open if the attacks had occurred at, say, 10:00am on a Monday when Chris was sleeping and I was unavailable, they might have been missed and we would still be thinking that the only problem was an upstream router issue.

While Chris learned a lot about how to prevent and respond to attacks, the most interesting lessons are those relating to business.

First, DDoS attacks are particularly troubling to cryptocurrency businesses, because they can't host their services on high-bandwidth shared servers. We have 370 coin wallets with a hundred thousand dollars in them at times, and there are too many horror stories of insider jobs to take any risk of someone at a hosting facility pulling a drive and copying keys, or even looking into the memory of a virtual machine. Additionally, social engineering is ineffective when you are the only person who has access to the computer. Our only choice is to have every physical server in our possession, where nobody else can modify configurations or trick someone else into modifying configurations.

But because our systems are on our own servers, and the only thing on our servers is our systems, they need to be hosted nearby. Not only do we need to be able to make sure they are still there, we need to be able to go to the location so that they can be upgraded when necessary. Last-mile bandwidth is expensive. Thus, a VPN is necessary to connect from a high-bandwidth "DDoS protected" location to a low-bandwidth one. While the arrangement drives up orphan rates (a loss of about $80 per day at current estimates), every pool should suffer from this same problem, leaving an equal playing field. If your pool is using a shared server in Romania to store its data, then the increased profits from the reduced orphan rates are being paid for in the expected value of your money being lost due to poor security.

A additional lesson is that companies that try to hide what they do behind nice pictures and simple catchphrases on their websites lose business. Chris and I have taken a simple approach to our dealings lately: if we can't understand exactly how what somebody does works, then we aren't going to buy it from them. We shouldn't have to look through hidden pages of simple text to find basic numbers and metrics. Cloudflare and Coinbase's GDAX are examples of companies that do this. Never require a customer to click "view demo" to see what services are actually offered!

More importantly, Chris and I are more convinced than ever that the quality of customer service is the most important factor in a business's success or failure. We specifically avoided Cloudflare because we figured that if their customer service representatives know nothing more than what they read from a script, then their hiring practices are likely to be to pay just enough to get a minimally qualified person in every position. We knew that it would be impossible to contact someone knowledgeable when an attack actually occurs. Furthermore, a company in their field should know that attacks need to be mitigated quickly, not by a sales agent returning a Saturday evening call on Monday night.

Verizon's support was also unacceptable for a company that wants to retain customers. Telecommunications is a field where Sprint will buy out an AT&T phone contract just to get a customer to switch at a loss, because they know that customers are likely to remain with a provider for years. In our case, Verizon dug up the ground to the office where the servers are located and paid technicians to run equipment through a concrete wall. This investment will be lost when we cancel our service with them, as Chris has already called Comcast Enterprise Solutions, who promised him that they actually have people available on the weekends who will respond within 15 minutes. Comcast has scheduled a conference call on the afternoon of June 26 to negotiate a price. We figure that if we can avoid just one day of downtime over the next year, it will be worth paying a thousand dollars per month more. If you want to put a number to performing 5 minutes of maintenance on a Sunday, it's worth $12,000 per year.

The amount of money these companies are losing is what strikes me most. If a large customer that was willing to pay thousands of dollars to get back online immediately called me, I would be doing whatever is necessary to help him. But when Chris called a nearby datacenter knowing that they charged that much and asking what services they provided, he never received a call back.

Javapipe's customer service is unbelievable, and the fact that it is always possible to contact them within 15 minutes through their ticket system means that the only reason we would ever consider switching from them would be if their services are insufficient for potential larger attacks in the future. At one point, Chris said that his business was going to go offline if they did not get to his ticket within the hour, and they actually escalated the ticket and activated the new services he requested.

I believe we are still in a better position than many of the companies in the cryptocurrency industry. Besides the obvious benefits of having resolved yet another reliability issue with our system, we discovered how easy it is to gain customers in general. Even if Prohashing becomes too unprofitable to continue when the current cryptocurrency bubble crashes, all you have to do in any business is to answer the phone, know what you're talking about, and be honest. The number of companies who can't do these basic things is astounding. If you can respond to tickets quickly, your product can be inferior to what the competition provides and you will get a lot of customers. Javapipe provides less bandwidth than Cloudflare, but we would never consider Cloudflare because of our experience with them.

Looking around the cryptocurrency industry, one can see businesses and industries that are succeeding, and those that aren't, and it's obvious why. Ethereum is succeeding because its founder treats people respectfully, is knowledgeable about what he's doing, and tells the truth. Bitcoin is failing because most Core members lie and attack people who would otherwise contribute. Likewise, Novaexchange is growing exponentially and we are their largest customer. Meanwhile, I closed my Coinbase account because Coinbase doesn't reply to support tickets - plain and simple. Clevermining never listed their names or contact information their website, and even threatened us if we didn't pay 42 bitcoins to acquire them, and they ended up folding because people respect those who are willing to show their faces and stand behind their actions.

In conclusion, we successfully mitigated a number of denial of service attacks. In the process, we learned that companies live and die not only by the quality of their products, but also by the quality of their customer service. We plan to set up a ticketing system to better track customer service requests, and will be hiring somebody to respond to tickets more quickly during the Monday-Friday, 8-4 timeframe. If you get hit with attacks, we highly recommend Javapipe. If you prefer to use a different provider, then whether you can get in touch with someone when something goes wrong is more important than the raw numbers. Finally, if you're buying into ICOs or investing in companies, pay close attention to how the owners treat people, as organizations that ignore or belittle their customers fail quickly.
User avatar
rootdude
Posts: 79
Joined: Wed Jan 07, 2015 3:14 pm

Re: An overview of the DDoS attacks against us, and what they teach about customer service

Postby rootdude » Sun Jun 25, 2017 2:36 pm

One definitive way of eliminating DDOS, is to register users (should they choose to do so). This can be accomplished pretty easily...

The settings UI/UX for a registered user would have the name of the rig(s) and the IP the rig was connecting from with a 'Register IP" button. Once pressed, the IP address of the miner(s) would be put in a table which would 'whitelist' that IP or list of IP's that the rigs use to connect to prohashing. This would initiate a script that would write those IP's to IPTABLES or some similar firewall - ensuring that those IPs, regardless of other activities (DDOS) would have access to the mining stratum. So, even in the event of a DDOS, the miners would continue unabated, while making it easier to administratively shut down the crap by blacklisting any non-registered IPs from the admin UI/UX on the backend.

Of course, this would preclude new or unregistered miners from joining the fun, but I assume we are more concerned with registered, veteran miners getting work done. Once the DDOS abates (because it's a waste of time and money for the attacker, since no mining would stop despite their efforts) - the blacklist could be dispensed with until needed again.

Thanks for listening,
rootdude
ProDDoSMitigation technique
derat
Posts: 3
Joined: Fri May 13, 2016 2:55 pm

Re: An overview of the DDoS attacks against us, and what they teach about customer service

Postby derat » Sun Jun 25, 2017 3:10 pm

First i must congratulate you both on the great service you provide.

Chris and I are more convinced than ever that the quality of customer service is the most important factor in a business's success or failure.


An opinion from a 62 year old that also had a small successful business, the above quote is the most important in any business, big or small, but as businesses grow they tend to forget or ignore this to concentrate purely on profits at any cost, thereby loosing many customers along the way.

It seems to be a common problem with big enterprises, specially in the communications area.

Here in Portugal these companies act like they are dictators or tyrants.

Keep on...

Greetings from a tiny but happy and satisfied client.

António (Derat)
User avatar
GregoryGHarding
Posts: 634
Joined: Sun Apr 16, 2017 3:01 pm
Contact:

Re: An overview of the DDoS attacks against us, and what they teach about customer service

Postby GregoryGHarding » Sun Jun 25, 2017 5:03 pm

rootdude wrote:One definitive way of eliminating DDOS, is to register users (should they choose to do so). This can be accomplished pretty easily...

The settings UI/UX for a registered user would have the name of the rig(s) and the IP the rig was connecting from with a 'Register IP" button. Once pressed, the IP address of the miner(s) would be put in a table which would 'whitelist' that IP or list of IP's that the rigs use to connect to prohashing. This would initiate a script that would write those IP's to IPTABLES or some similar firewall - ensuring that those IPs, regardless of other activities (DDOS) would have access to the mining stratum. So, even in the event of a DDOS, the miners would continue unabated, while making it easier to administratively shut down the crap by blacklisting any non-registered IPs from the admin UI/UX on the backend.

Of course, this would preclude new or unregistered miners from joining the fun, but I assume we are more concerned with registered, veteran miners getting work done. Once the DDOS abates (because it's a waste of time and money for the attacker, since no mining would stop despite their efforts) - the blacklist could be dispensed with until needed again.

Thanks for listening,
rootdude
ProDDoSMitigation technique

the issue with that is most of nort american ISPs issue dynamic ip addresses, nothing like coming back after a weekend out of town, and find out you made no profit because 5 hours after you left on friday your ISP issued a new IP Lease.
User avatar
rootdude
Posts: 79
Joined: Wed Jan 07, 2015 3:14 pm

Re: An overview of the DDoS attacks against us, and what they teach about customer service

Postby rootdude » Sun Jun 25, 2017 6:14 pm

GregoryGHarding wrote:
rootdude wrote:One definitive way of eliminating DDOS, is to register users (should they choose to do so). This can be accomplished pretty easily...

The settings UI/UX for a registered user would have the name of the rig(s) and the IP the rig was connecting from with a 'Register IP" button. Once pressed, the IP address of the miner(s) would be put in a table which would 'whitelist' that IP or list of IP's that the rigs use to connect to prohashing. This would initiate a script that would write those IP's to IPTABLES or some similar firewall - ensuring that those IPs, regardless of other activities (DDOS) would have access to the mining stratum. So, even in the event of a DDOS, the miners would continue unabated, while making it easier to administratively shut down the crap by blacklisting any non-registered IPs from the admin UI/UX on the backend.

Of course, this would preclude new or unregistered miners from joining the fun, but I assume we are more concerned with registered, veteran miners getting work done. Once the DDOS abates (because it's a waste of time and money for the attacker, since no mining would stop despite their efforts) - the blacklist could be dispensed with until needed again.

Thanks for listening,
rootdude
ProDDoSMitigation technique

the issue with that is most of nort american ISPs issue dynamic ip addresses, nothing like coming back after a weekend out of town, and find out you made no profit because 5 hours after you left on friday your ISP issued a new IP Lease.


I'd suggest that you wouldn't be a good candidate for using this - but I suspect a LARGE percentage of miners these days colocate their equipment. Further, the dynamic IP's issued by ISP's these days have such a long lease time that they are, in effect, static IPs. YMMV of course.
JKDReaper
Posts: 101
Joined: Fri Mar 31, 2017 11:17 am

Re: An overview of the DDoS attacks against us, and what they teach about customer service

Postby JKDReaper » Sun Jun 25, 2017 6:45 pm

derat wrote:First i must congratulate you both on the great service you provide.

Chris and I are more convinced than ever that the quality of customer service is the most important factor in a business's success or failure.


An opinion from a 62 year old that also had a small successful business, the above quote is the most important in any business, big or small, but as businesses grow they tend to forget or ignore this to concentrate purely on profits at any cost, thereby loosing many customers along the way.

It seems to be a common problem with big enterprises, specially in the communications area.

Here in Portugal these companies act like they are dictators or tyrants.

Keep on...

Greetings from a tiny but happy and satisfied client.

António (Derat)


Own my own business as well (Tax Prep and Accounting) and pretty much every year I have to hire 1 new person because of new customers. 90% of these new customers are coming from H&R Block or Jackson Hewitt. I'd say a about 85% of those all tell me the same thing..."we know someone who comes here and they tell us you sit and talk to them and don't rush them out the door, you're people are friendly, and you actually try to help them"...and I preach this to m employees every year before tax season starts...treat them with respect and regardless of how tired or sick or how wore out you are, make them KNOW you are here to help them, don't treat them like a commodity, treat them like you've known them all your life.

I'm 41 years old and have preached 2 things in business since I opened my first Custom Car Stereo shop when I was 19...Location, and Customer Service! You make a customer happy...and they will come back, even if they may have to pay more! (I refuse to walk into Wal-Mart in our town...I'll drive 35 miles and pay more at another store just because I can't stand the way ours act toward customers)

The questions and issues I've had here at PH have been address quickly (as possible given what may have been happening at the time here) and they've went out of their way to make sure that, even though I'm probably in the lowest 10% of people's hash rate here, that they wanted me to be as satisified as the top hasher...was told that almost exactly a while back...and this is the primary reason I still point 90% of my hash here. Doesn't hurt that it's the most profitable of course lol, but honestly, I would point it elsewhere if they didn't do the things they do.
vinylwasp
Posts: 97
Joined: Mon Oct 31, 2016 3:42 am
Location: New Zealand

Re: An overview of the DDoS attacks against us, and what they teach about customer service

Postby vinylwasp » Tue Jun 27, 2017 4:04 pm

rootdude wrote:One definitive way of eliminating DDOS, is to register users (should they choose to do so). This can be accomplished pretty easily...

The settings UI/UX for a registered user would have the name of the rig(s) and the IP the rig was connecting from with a 'Register IP" button. Once pressed, the IP address of the miner(s) would be put in a table which would 'whitelist' that IP or list of IP's that the rigs use to connect to prohashing. This would initiate a script that would write those IP's to IPTABLES or some similar firewall - ensuring that those IPs, regardless of other activities (DDOS) would have access to the mining stratum.
Thanks for listening,
rootdude
ProDDoSMitigation technique


Given the support issues this would raise with DHCP as noted by GH it might be easier just to auto-provision a VPN connection per registration and restrict access that way so that when a user's ISP connection bounces, their router re-establishes the VPN; but actually neither of these last mile solutions will work under anything but the smallest volumetric DoS attack as the upstream pipe back to Verizon will be saturated before legitimate traffic even reaches the edge router (assuming 1GB or less). With a large carrier like Verizon you could probably push the VPN termination points out geographically to the edges of their global network via some kind of managed VPN service and then safely come across their 'clean' network as long as they can offer some guarantee that the internal network will be protected effectively. This way you don't publish the pool ip publicly, only the VPN pops.

It sounds like Verizon's response to this incident was to simply Black hole all traffic bound for PH (both good and bad) as opposed to redirecting it to a scrubbing center for cleaning, but then they'd probably want you to sign up for a Platinum SLA before they'd do that and you generally need to have you own AS so they can swing the routing to the scrubbing center or have an Attack Mitigator like a TopLayer that does it for you.

With respect to Customer service, it's always great when the people you're dealing with are the owners as is the case with Chris and Steve, but that simply doesn't scale beyond the first employee you hire who isn't as invested in the success of your business as either of you are. Then you become a people manager and not a doer - oh happy days. Cloudflare grew out of the Honeypot project and is technically a very smart platform. To be fair to them it does say it's a CDN/WWW security service on the Under Attack page amongst other places but it's easy to miss when you're in a hurry:

I deployed it as the lead security architect for a $14Trillion Exchange for their web site and it worked great, but then the tech support team was about 500 yards down the road and we didn't do it in the heat of a full blown DDoS so we had time to talk and understand all the pros and cons.

Speaking of that, it's horses for courses, carriers can do layer 1-4 very well and stumble with 5-7 where you may need more specialist help though I doubt anyone can do much with stratum other that a circuit proxy of some sort. I haven't looked at Javapipe so maybe they do both, tech moves fast.

Steve/Chris, do you have any information on exactly what kind of DDoS attack it was and how large it really was? This may help you plan for the next one.

On the topic of locating your services, I've worked for very large banks, govt and the worlds largest payment processor and they all had mission critical systems holding $Bs in 3rd party data centers. Building and operating them is a specialist function and the big established pleyers bring scale and security you can't afford to build yourself. Amazing physical security, biometric access controls down to the rack level, 24/7 surveillance, network scale with carrier and geographic diversity are just a few of the advantages you get with a Tier 3 or 4 DC.

It costs, but then it's probably time to brush off that business plan and think about where you're going to be in 3 years. If you need investors, I'd be in, and happy to help with security if I can.
User avatar
Steve Sokolowski
Posts: 3253
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA
Contact:

Re: An overview of the DDoS attacks against us, and what they teach about customer service

Postby Steve Sokolowski » Tue Jun 27, 2017 4:44 pm

vinylwasp wrote:
rootdude wrote:One definitive way of eliminating DDOS, is to register users (should they choose to do so). This can be accomplished pretty easily...

The settings UI/UX for a registered user would have the name of the rig(s) and the IP the rig was connecting from with a 'Register IP" button. Once pressed, the IP address of the miner(s) would be put in a table which would 'whitelist' that IP or list of IP's that the rigs use to connect to prohashing. This would initiate a script that would write those IP's to IPTABLES or some similar firewall - ensuring that those IPs, regardless of other activities (DDOS) would have access to the mining stratum.
Thanks for listening,
rootdude
ProDDoSMitigation technique


Given the support issues this would raise with DHCP as noted by GH it might be easier just to auto-provision a VPN connection per registration and restrict access that way so that when a user's ISP connection bounces, their router re-establishes the VPN; but actually neither of these last mile solutions will work under anything but the smallest volumetric DoS attack as the upstream pipe back to Verizon will be saturated before legitimate traffic even reaches the edge router (assuming 1GB or less). With a large carrier like Verizon you could probably push the VPN termination points out geographically to the edges of their global network via some kind of managed VPN service and then safely come across their 'clean' network as long as they can offer some guarantee that the internal network will be protected effectively. This way you don't publish the pool ip publicly, only the VPN pops.

It sounds like Verizon's response to this incident was to simply Black hole all traffic bound for PH (both good and bad) as opposed to redirecting it to a scrubbing center for cleaning, but then they'd probably want you to sign up for a Platinum SLA before they'd do that and you generally need to have you own AS so they can swing the routing to the scrubbing center or have an Attack Mitigator like a TopLayer that does it for you.

With respect to Customer service, it's always great when the people you're dealing with are the owners as is the case with Chris and Steve, but that simply doesn't scale beyond the first employee you hire who isn't as invested in the success of your business as either of you are. Then you become a people manager and not a doer - oh happy days. Cloudflare grew out of the Honeypot project and is technically a very smart platform. To be fair to them it does say it's a CDN/WWW security service on the Under Attack page amongst other places but it's easy to miss when you're in a hurry:

I deployed it as the lead security architect for a $14Trillion Exchange for their web site and it worked great, but then the tech support team was about 500 yards down the road and we didn't do it in the heat of a full blown DDoS so we had time to talk and understand all the pros and cons.

Speaking of that, it's horses for courses, carriers can do layer 1-4 very well and stumble with 5-7 where you may need more specialist help though I doubt anyone can do much with stratum other that a circuit proxy of some sort. I haven't looked at Javapipe so maybe they do both, tech moves fast.

Steve/Chris, do you have any information on exactly what kind of DDoS attack it was and how large it really was? This may help you plan for the next one.

On the topic of locating your services, I've worked for very large banks, govt and the worlds largest payment processor and they all had mission critical systems holding $Bs in 3rd party data centers. Building and operating them is a specialist function and the big established pleyers bring scale and security you can't afford to build yourself. Amazing physical security, biometric access controls down to the rack level, 24/7 surveillance, network scale with carrier and geographic diversity are just a few of the advantages you get with a Tier 3 or 4 DC.

It costs, but then it's probably time to brush off that business plan and think about where you're going to be in 3 years. If you need investors, I'd be in, and happy to help with security if I can.


We haven't been able to get information about the types of attacks. Verizon was very incompetent and didn't know anything, and I think that Javapipe's policy is to simply mitigate attacks so as not to reveal how they do that. I'm fine with that as long as they do a good job.

For now, there are a few things we want to deal with, as they seem to be obvious problems. First, we need to get someone to handle easy customer service requests, like "why did the site go down today?" so that Chris isn't spending a lot of time answering these questions. Second, I need to continue to work 30 hours a week solely on performance improvements. The sole miscalculation that was made at the beginning of the project was how it would take someone doing nothing but performance optimizations just to keep the system online, and even that isn't enough at present.

After that, it becomes more difficult because of the costs. We have to be careful in making investments in the business until the bubble crashes and we get a better idea of what profitability will end up at when everything flushes out. I suspect that we will make 1/10 of the profit we are now when the bottom of the cycle arrives. I would be surprised if scrypt profitability holds above 0.5 cents at the end of the year, for example.

I understand about the high-tier datacenters, but the truth is that the margins in mining are very low. These datacenters cost thousands of dollars, but interestingly there is another problem - they don't seem to want customers, either. Chris hasn't been able to get any datacenters to call him back. That's why we decided to investigate Comcast Enterprise. If they are reasonable, then we can install a multi-port network card in one of the servers, downgrade verizon to low bandwidth, and use them as a backup. At the office, we already have a generator, UPSs, and a security system.

I completely agree with what you've said, but I think more caution is needed than what you're saying. I'm not ready yet to quit a good job and spend a huge amount of money given the current uncertainty of costly attacks that could make the site unprofitable to run someday, the fact that the bubble is going to crash at some point, and (especially) that Republicans might eliminate the preexisting conditions clause in the Affordable Care Act.

Is it really that bad to slowly grow, moving up to higher levels of service as required and as conditions become more certain?

We'd be happy to have help with security, but one thing we did decide is that we don't want to give away shares in the business. Once that happens, it can never be taken back.
vinylwasp
Posts: 97
Joined: Mon Oct 31, 2016 3:42 am
Location: New Zealand

Re: An overview of the DDoS attacks against us, and what they teach about customer service

Postby vinylwasp » Tue Jun 27, 2017 5:30 pm

Thanks Steve, I understand your perspective. From an outsiders perspective looking at the US economy and political situation, I think you're very wise to be cautious. It's also one of the reasons I'm bullish on crypto though, when the markets and $ collapses, traditional refuges like gold and newer stores of wealth like crypto will be seen as a safe haven.
User avatar
allinone
Posts: 1
Joined: Mon Jun 26, 2017 9:30 am

Re: An overview of the DDoS attacks against us, and what they teach about customer service

Postby allinone » Wed Jun 28, 2017 2:15 am

Yes Steve n all, I don't know much about computing and things, but I too do believe that those companies that are putting their highest priority on human values rather than gain are those that will win the game, both in terms of happiness and wealth of all those involved. Nobody likes to be ignored, cheated or bullied, everybody likes to be respected, well treated, and appreciate trustfulness. As simple as that... even greediness of the majority can't change that. I mean you know, people are now used to be badly treated in most of their day to day experiences, so finding oasis where they experience the exact opposite acts like a powerful relieving magnet on them.

and Steve, you spoke about a cryptobubble crash, I'm fairly ignorant in trading so I'd tend to hear your words here, but looking at the mid/long term picture, don't you think that crypto has the potential and qualities to bring about the fairness most needed in the financial department of humanity, and as such might have a very bright future? And Prohashing is certainly the kind of smart tool that we need to built this 'everybody wins' model in place of the present 'I keep it all for me cos I'm the one ruling' one :D

Who is online

Users browsing this forum: No registered users and 4 guests