Upcoming performance improvements maintenance

News updates about the Prohashing pool
Post Reply
User avatar
Steve Sokolowski
Posts: 4510
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Upcoming performance improvements maintenance

Post by Steve Sokolowski » Tue Oct 12, 2021 4:28 pm

Over the past week, we've spent a significant amount of effort improving the system's capacity. Even though we increased the capacity of the system in 2018 to support 250,000 users, it wasn't until Salad joined that it was discovered that the reliability of the live data is not at the same level as that of the mining servers and money auditing. The system has proven itself to work as designed by not losing any money or data, but simply recovering later when there aren't enough CPU cycles available isn't the level of service we want to deliver.

One of the immediate changes you should notice after these upgrades have been completed is that the data at the top of the forums pages will load almost instantly, instead after the delay caused by the current 100% CPU load on the server. Miner dashboard updates and balance updates should occur at more regular intervals, instead of appearing to occur in batches where there are few updates and then the system catches up later.

Here are the upcoming releases:
  • On October 13, we will release the Crossbar server upgrade that will enable parallel proxy workers. We need to both change the configuration and also upgrade the version of Crossbar from 19.1 to 21.3. Miners won't be disconnected, and shares will be queued in memory if necessary, but there could be an interruption to live data and a delay before earnings are recorded while this upgrade is performed.
  • On October 14, we will restart all mining servers to apply changes that will reduce orphan rates and increase profitability. Currently, the mining servers run queries like "select username, api_key from person" every minute. These queries worked well in 2013, but now can take many seconds to compute the results. The 100% CPU time prevents found blocks from being submitted to the network. The upgraded servers will only query for changes, reducing the rows processed every minute from 500,000 to 0 or 1. This upgrade is expected to disconnect miners for less than 10 seconds, unless a revert is necessary.
  • On the weekend of October 16-17, we may upgrade Crossbar a second time to add another type of worker to parallelize additional types of requests, should the first upgrade on October 13 prove that we only added enough capacity for the some of the request types.
  • On October 21, Comcast will be upgrading our dedicated bandwidth to give us 8 times more capacity, from 500Mbps to 2Gbps. This procedure requires multiple people in different locations to adjust the ISP's routers, fiber wavelengths, and other configuration at the same time Chris manually changes cables. Chris is working on a plan to try to minimize downtime, but it's difficult to foresee a circumstance where everything goes so smoothly that it can be done with less than 15 minutes of downtime.
We have already upgraded the share inserters to reduce load and those have had CPU usage decline by 30% compared to Sunday.

Chris and I are continuing to work 12-16 hours per day, 7 days per week, until the live data has at least four times the capacity needed to serve today's customers. We apologize for these frequent upgrades all of a sudden; however, we are spending as much money as possible right now on increasing capacity to make Prohashing an even better experience for all of you and for Salad's customers!
Sumoblei
Posts: 34
Joined: Sat May 29, 2021 8:34 am

Re: Upcoming performance improvements maintenance

Post by Sumoblei » Tue Oct 12, 2021 4:44 pm

Absolutely outstanding!
User avatar
Mastermind007
Posts: 57
Joined: Wed Aug 25, 2021 2:02 pm

Re: Upcoming performance improvements maintenance

Post by Mastermind007 » Tue Oct 12, 2021 6:31 pm

Very nice and thank you for doing all this.
User avatar
-MaVerick-
Posts: 85
Joined: Wed May 12, 2021 3:12 pm
Location: Germany

Re: Upcoming performance improvements maintenance

Post by -MaVerick- » Tue Oct 12, 2021 6:38 pm

Sounds like we need a fund for providing our Prohashing overlords with enough coffee to survive the upcoming days or weeks. We are with you and we are grateful for your hard work!
Alynn
Posts: 89
Joined: Wed Mar 03, 2021 12:40 pm

Re: Upcoming performance improvements maintenance

Post by Alynn » Wed Oct 13, 2021 8:50 am

I mean you can tip via your payout page heh
User avatar
Steve Sokolowski
Posts: 4510
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Re: Upcoming performance improvements maintenance

Post by Steve Sokolowski » Wed Oct 13, 2021 10:13 am

The Crossbar router has been successfully upgraded. You should now see an improvement in latency for live data.

We are going to move up the mining server release to today, to try to get everything done as soon as possible. There will be a brief mining outage shortly.
User avatar
Steve Sokolowski
Posts: 4510
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Re: Upcoming performance improvements maintenance

Post by Steve Sokolowski » Wed Oct 13, 2021 11:06 am

The mining servers have been deployed, and all upgrades have been completed. Enjoy the performance improvements!
User avatar
Mastermind007
Posts: 57
Joined: Wed Aug 25, 2021 2:02 pm

Re: Upcoming performance improvements maintenance

Post by Mastermind007 » Wed Oct 13, 2021 12:06 pm

Image
Post Reply