Relocating the servers
One of the reasons for the upgrade was the relocation of the servers from Philadelphia to a datacenter near State College. The entire upgrade was set off by the discovery last October that the landlord in the previous location was insolvent. We didn't want to suddenly find out that the company was bankrupt, that the building had been foreclosed upon, and that the new owner was converting it into an office. Our two-year contract with Comcast was up for renewal on April 18, and we decided that we should renew the contract elsewhere where the building's owner was in a better financial situation.
After researching locations, we decided that the best option was to bring the servers to a location closer to everyone's homes in the center of the state. There have been times in the past when physical hardware was failing and Chris needed to install software to work around the problem until he could travel to Philly to do several things at once. With the servers closer, Chris can fix them anytime if something fails. Prohashing does not rent hosting on cloud computing - not only is cloud hosting more expensive, but having physical control of the servers eliminates the possibility of employees at the cloud mining company being tricked into granting access.
Chris left State College at 7:30am on Saturday, drove to Philly on the empty roads, and arrived there at around 10:50. After a bit of prep work, he disassembled the server rack and loaded everything into the back of an SUV. He disconnected our backup Verizon FiOS connection, took the Comcast switch, and started driving in the opposite direction about 80 minutes later, returning by 5:30pm EDT. We were worried about asking Chris to drive east towards the cities, into a dangerous area with many more cases. But as I mentioned on twitter, the world is a ghost town and Chris didn't come across any people. Even when Chris was driving through Centre county where he lives, which is the only county in his trip that was in the "yellow" initial phase of reopening (the others were red), there were times when he would drive 10 or 15 miles without seeing any cars. Every sign warned people to stay home, and cops were everywhere, likely looking for minor excuses like taillight violations to stop cars on nonessential business.
Adding TRIM support to the solid state disks
When Chris set up the servers on December 23, 2013, Chris was unaware of the TRIM command for solid state disks. Solid state disks wear out over time, and have a set number of writes available for each sector. TRIM marks deleted space on a disk so that it can be reused evenly, and when TRIM is not used, overwriting the same data can cause some areas of the disk to be written many times, while others are never written. That can cause a disk to slow down over time, and prematurely fail as most of the disk's areas are never written, while one area is written thousands of times.
Normal computers, like the typical Windows desktop you're probably using now, detect SSDs and support trim natively. Virtual machines are a different story because the virtual machine doesn't always know what type of disk is being used, so they can't use TRIM without proper configuration and the right drivers.
The database and daemon servers run on virtual machines. To the hypervisor, their disks appear as one huge file on the underlying filesystem, so unless support is enabled, the hypervisor just writes and rewrites this huge file that is too big to move elsewhere on the disk in the same location. A program that writes once and reads data over and over, like a webserver, is unaffected by this problem. Our servers write a lot of data, and most of it is never read at all. Examples of this data include blocks that daemons receive but which don't contain transactions we care about; W8-BEN forms, which are copied to disconnected disks to satisfy a legal requirement but have never actually been looked at; and most of the data dealing with money earned from raw shares, which is only queried if we need to audit earnings. Because of the high number of writes, over time the database had slowed down dangerously and eventually would have caused unplanned downtime where we would have had to enable TRIM on short notice.
To enable TRIM, Chris had to copy 16TB of data to two hard drives, which could handle only 600MB/s of writes, recreate the virtual machines' disks, copy the data back onto the new disks, and then wait for the server to rewrite the RAID 5 parity data to restore the critically degraded array back to normal status. The data from daemons and the database can't be copied while the servers are online, because it is constantly being changed.
The wait for the disks to copy was the bottleneck in the maintenance, taking hours longer than expected. It is clear that disks are now the slowest, most expensive, and most limiting part of any computer system. The 500Mbps of bandwidth the servers have is about 10,000 times the transfer rate of the 56k modem the first time I used the Internet in 1998, but the speed of disks has only improved by 100 times - if that - during that period. For the first time, when I downloaded Call of Duty: Modern Warfare to a different system last month, the download speed was limited by the array of disks onto which the data was being written, not by the Internet connection.
Changing datatypes in columns
Once the database was online, Chris had to change the datatypes of several columns in tables that had 1TB of data. A change in datatype is a blocking operation, so it could only be done when the servers were offline.
Since the data was already on solid state disks, this write didn't take as long as the hard drives took to copy, but it was still a long process. The data to be changed focused on worker statistics and API calls. The "difficulty," "submitted difficulty," and similar columns had been created with a type of INTEGER as an oversight, and the psycopg2 Python PostgreSQL library will not reject a query that attempts to insert a decimal number into an INTEGER column. Instead, a query such as
Code: Select all
INSERT INTO table(column) VALUES(3.1);
Upgrading the network to 10Gbps links
Until last week, all of the servers were connected with gigabit Ethernet cables. Chris had assumed when the servers were created that gigabit Ethernet would be sufficient, because the outbound connection was only 300Mbps. Indeed, gigabit Ethernet was fine for the first few years of operation.
We had wondered for some time why orphan rates on some coins were so high, because we have a set of comprehensive and well-tested quality of service rules that limits the upload bandwidth of payout-only daemons, and which will delay all other traffic in favor of a found block. Recently, however, he discovered that the internal network traffic between the daemon servers was being throttled by gigabit links. The cause was that in 2018 we invested hundreds of thousands of dollars to expand the capacity of the system to meet a demand of 250,000 miners - a capacity that is 35 times greater than current usage and which never arrived.
Nevertheless, it may arrive someday, and one of the keys to that expansion is that we can run many parallel mining servers that all receive block template data at the same time. The mining servers communicate with each other constantly - for example, they tell other servers that a miner found a block even before the daemon acknowledges it, so that all future shares for that block on all servers are rejected until the next block's template is available. Daemon servers also publish block data to all mining servers at the same time.
To add to this problem, backups are performed frequently, copying wallet files and database data across the internal network to two external hard drives in case the primary disks fail.
The amount of data transferring between the daemon servers and mining servers became a serious issue that was unnoticed until February. As the number of coins increased over the years, the single gigabit link coming out of each server would throttle found blocks going to the Internet along with the traffic going to the other daemon servers too. Chris replaced the gigabit links with 40Gbps network cards, which have 10Gbps transceivers attached to them to save money. About 1.5Gbps of data is being transferred between the servers now at peak times, ending the throttling problem.
Upgrading operating systems
The operating systems of many of the servers had become obsolete. In general, Prohashing takes a "if it ain't broke, don't fix it" approach, which is why we still run the obsolete Python 2 for most services programmed in 2014. There is too much money at stake and too little benefit to be gained (at least until Python 2 support starts to be removed from things) to risk introducing bugs into Python software.
Unfortunately, the operating systems are one of the things that need to be fixed, even if they aren't broken. One issue is that many coin developers like to use bleeding-edge development tools, so new that packages sometimes aren't even in "sid" for the stable Debian version. But we found that we could eliminate most issues by upgrading from Debian 7 and 8 to Debian 10, which Chris did on many servers.
The upgrade process was delayed, ironically, by the unavailability of working drivers for the 40Gbps network cards on the older Ubuntu 16 LTS. After trying to fix compilation errors, Chris decided the best solution was to simply get rid of Ubuntu and to go to Debian 10. Rebuilding those servers took several hours, but he decided that he knew he could complete the task in a set amount of time, whereas if he tried to figure out what packages he needed for the older version of Ubuntu, he could end up spending several hours and having to go with Debian anyway.
One of the largest time-sinks we have been encountering recently has been upgrading software that provides no benefit to us directly, but which the developers have ceased maintaining and which continuing to use would expose us to security vulnerabilities. There isn't enough attention in the software industry to whether new features are actually needed, or whether they are just imposing costs on businesses unnecessarily.
Upgrading to PostgreSQL 12
Chris also upgraded the database from PostgreSQL 10 to 12. This upgrade went according to plan and was simple, a testament to the competency of the PostgreSQL developers. There is even a tool that can upgrade a database in-place with significantly less time required, but Chris didn't use it because we wanted to VACUUM FULL all the tables anyway. We needed to reclaim disk space that was being wasted by coins that had been discontinued and deleted from the block explorers.
The PostgreSQL developers, unlike those of many packages, understand that what their users want is reliability and performance. Relational databases are a "mature technology," similar to cell phones, where there aren't many new features that can be added. Many of the changes in PostgreSQL 12 involve parallelizing tasks that were previously limited by singlethreaded operations. If PostgreSQL 12 had been available in 2013, we would be six months further along in development, because much of our early efforts were spent denormalizing and summarizing table data to reach an acceptable level of performance.
The final reason for moving the servers was for cost-cutting. We were able to negotiate a contract with Comcast that increased the amount of bandwidth from 300Mbps to 500Mbps, for the same price, for only 12 months compared to the 24 month commitment of the previous contract. The new contract cancels some services we don't need, saving money.
Because we can visit the physical site in only a few minutes if the servers crash, there is no need to pay Verizon $140/month for a backup connection. There is no need to pay Javapipe $100/year to provide a VPN to protect that backup connection from DDoS attacks, like the ones that happened in 2017, and there is no need to expose services through that connection that could theoretically have vendor security vulnerabilities in them.
A significant amount of time during this process was spent on the phone with companies to cancel and set up services. While Comcast is notorious for poor customer service, perhaps from the 1980s when the company focused primarily on cable TV, Comcast's metro ethernet support is the best of any company I have ever seen. Chris called Comcast metro ethernet at 11pm on Thursday, and a person answered the phone immediately. He was called back by a technician who understood how the Internet works, and he was able to log into and configure their routers. A few years ago, Comcast had misconfigured its routers so that we only had connectivity within Comcast's network, and Chris called at 2:30am, was connected to a customer service agent, and they called back in 15 minutes to say it was fixed.
In the coming days, Chris will be completing some minor tasks that will bring online some parts of the system that are known to still have issues. The monitoring system that will send Alexa "notifications" when critical errors are detected isn't running yet, and there is an API connectivity issue that should be resolved later today. We also need to modify the quality of service rules, to relax restrictions in the face of increased bandwidth. Feel free to submit a support ticket if you see something that needs to be resolved.
Now that this upgrade is behind us, we plan to assign even more effort to the new version of the website, which is nearing feature completion. Look for images of the site soon, and for the two sites to start running in parallel in the middle of the summer.