Improved website performance

Over the past few weeks, website performance has become a problem. At first, we simply increased the number of cores on the server to 6, but CPU usage then increased to a steady 1200%. After some investigation, I determined that, for some reason, the url /user/checkPassword was being called 570 times per second. This URL is used to determine if a user has two-factor authentication enabled. Even though users are banned after 25 invalid password attempts, there would be thousands of calls from the same IP addresses despite that IP address never being able to log in again, even if valid credentials were provided.

We changed the system so that on the 26th attempt, the IP address is now added to hardcoded list. That way, these requests will no longer load the entire Grails framework and query the database just to reject the request. If the IP address is on the list, all packets received from it are simply dropped, saving CPU load. About 17,500 addresses have already been added to the list.

We were able to reduce the CPU load on the website from 1200% to 130% after two hours of recording these addresses, and the number of checkPassword requests dropped from 1 million per hour to 80,000. Google Analytics reports that the change improved page load time from 7 seconds to 4.2 seconds. Feel free to share your observations on performance below.

So, if someone typed in the wrong and then saved those credentials, and the page reloaded, and tried to apply the credentials 26 times and got the IP banned...how would that user go about getting their IP unbanned if possible. Will there be some type of warning when they try to go to login "your IP is on the blocked list" or anything to let them know they need to get unblocked? Is there a decay rate on the IPs, after 30 days the IP is released to be tried again?

Is this something that when people put in a ticket with --cannot loggin to website-- will be one of the first things checked for? if their IP is on the blocked list?

AppleMiner wrote:So, if someone typed in the wrong and then saved those credentials, and the page reloaded, and tried to apply the credentials 26 times and got the IP banned...how would that user go about getting their IP unbanned if possible. Will there be some type of warning when they try to go to login "your IP is on the blocked list" or anything to let them know they need to get unblocked? Is there a decay rate on the IPs, after 30 days the IP is released to be tried again?

Is this something that when people put in a ticket with --cannot loggin to website-- will be one of the first things checked for? if their IP is on the blocked list?

The user is automatically unbanned the next day. We've only gotten one ticket about this issue before, and our response was simply to wait until the ban expired.

Well, everything worked fine until this morning. Now I can't get on the prohashing website. The site can't be reached... If I try on my phone and I put it on LTE (not my home wifi)... then I can get to the site. So, It seems like an IP issue. But I never put any credentials in wrong... Ill put in a ticket tomorrow if it still has issues. L3+ seems to be connected fine...? odd.

I had issues also connecting into the prohashing.com website starting around 4PM EST , from both verizon and comcast in 2 different areas. Phone on ATT worked fine to connect and see live data so site wasnt offline at time I checked. At first all 3 connected in, and after a while stopped getting website to come up, but I could still ping and traceroute completely to http://www.prohashing.com with 13 hops and 54 ms time. Plus...live data on the phone so the site was up, all 3 PC networks could ping and see it, none would pickup or display the website. Different browsers tested, systems rebooted, caches flushed. Made new user account on a PC so it had all default settings, still no joy.

After a while I had 1 of them start to work, but the other 2 were still down, not sure if they were same ISP in 2 differnt places or not, then later all were down then all 3 had a working page after a refresh or two.

Seems to be up and working now across all my devices, if that changes I will post an update.

AppleMiner wrote:I had issues also connecting into the prohashing.com website starting around 4PM EST , from both verizon and comcast in 2 different areas. Phone on ATT worked fine to connect and see live data so site wasnt offline at time I checked. At first all 3 connected in, and after a while stopped getting website to come up, but I could still ping and traceroute completely to http://www.prohashing.com with 13 hops and 54 ms time. Plus...live data on the phone so the site was up, all 3 PC networks could ping and see it, none would pickup or display the website. Different browsers tested, systems rebooted, caches flushed. Made new user account on a PC so it had all default settings, still no joy.

After a while I had 1 of them start to work, but the other 2 were still down, not sure if they were same ISP in 2 differnt places or not, then later all were down then all 3 had a working page after a refresh or two.

Seems to be up and working now across all my devices, if that changes I will post an update.

This issue is definitely not caused by the new changes to reduce CPU load by banning those IP addresses. The bans last for 86400 seconds, and that amount of time hasn't yet elapsed since the feature was released. Therefore, if the system started connecting again for you recently, whatever was causing the problem could not have been this feature.