Page 1 of 1

A look at our daemon servers

Posted: Thu Jan 07, 2016 5:54 pm
by Steve Sokolowski
I thought I'd take some time today to write a bit about how our architecture works. I thought an overview might be interesting to non-miners who know little about the field, and would also be informative to knowledgable users who have been reading about the bitcoin industry frequently. What happens in the real world is often quite different from what happens in theory, so I thought a description of actual real-world operations would be useful. This article will focus on the coin daemons; I may write about the mining server, trading algorithms, database, and website in future posts.

Among other hardware, we own three servers that contribute to our mining operations. One is an enterprise-grade 24-core system that hosts multiple virtual machines, including the database. It has an Avago RAID card that operates four 1TB SSDs in a RAID 1+0 configuration. These SSDs can handle about 200,000 IOPS, but share insertions only stress them to about 3% of capacity because of concurrency issues. We plan to use the additional capacity to add new features like live updates to the website. This server also hosts the mining server, trading server, website, and a legacy daemon coin virtual machine for a few daemons that won't compile on Debian 8.

Other two are desktop machines that were built using inexpensive components purchased at Microcenter and assembled by Chris himself to save money. They cost about $700 each at the time of purchase in May 2015. The cases were salvaged from Chris's old gaming and media PCs and repainted; the Ethernet cables were made using old Cat-5e cable that he had in his garage. The only unrecoverable information on these servers is the wallet.dat files, so Chris uploads those to a remote site every night and the only component other than the motherboard in these servers is a single 256GB SSD. Since there are two servers, we can continue mining half the coins at reduced profitability if Chris needs to rebuild one of them. There is nothing other than the pool installed on these servers, and the connection is dedicated, and we physically own these servers, so that they cannot be hacked through social engineering or security vulnerabilities in a host's hypervisor.

The following list summarizes what component is the performance limiter in each virtual machine:
  • Website: CPU
  • Database: Memory latency
  • Mining server: CPU
  • Trader: Artificial API usage limits from exchanges
  • Daemon servers: CPU
As you can see, the primary limitation to our systems is CPU usage, not memory, disk performance, or bandwidth. CPUs are by far the most expensive components in these servers, and the most difficult to upgrade. In the case where CPu is not the limiting factor (the database), we haven't upgraded the memory because we don't have a need to do so.

While many people worry about bitcoin scaling out of reach of the general consumer, bitcoin is one of the 165 daemons that run on the two daemon servers, all at the same time. The daemon servers are built with cheap, midrange consumer grade hardware. On each server, all the daemons combined use less than 150GB of disk space. The largest daemon consumer of CPU (the limiting resource) is Ethereum, not Bitcoin.

When daemons receive new blocks, most of them have a feature that can execute a shell command. Those daemons initiate the shell command, which connects to the mining server and informs the mining server that a new block is available. The mining server then calls getblocktemplate, getwork, or getauxblock. It checks to see whether the new block either has a higher block number than the previous block, or whether time-based retargeting algorithms changed the difficulty of the current block. The mining server then executes a procedure to determine which miners should be reassigned based on profitability, their work restart times, whether the block is auxiliary, and other factors.

We are able to manage the number of daemons on these servers because almost all of the time, it is not critical that they receive access to resources immediately. For example, when another node has requested to get blocks way back in the past, most of the time it is serviced quickly. When something more important is going on, the response to this blockchain download request can be delayed by 20ms without any major issue. Full CPU availability can be restored to that coin milliseconds later. While it might seem at first that uplink would be saturated by having all these daemons online at once, in reality the upload bandwidth of all the daemons averages 5Mbps.

We wrote a tool called daemon-network-management that assigns priorities to the daemons. By default, daemons are limited by iptables to a total available bandwidth for all of them of 35Mbps, and are assigned a CPU "niceness" of 5. When a block is found for a daemon, it is raised into a "priority mode" for 10s. During the priority mode, a rule is added to iptables to grant unlimited bandwidth to the daemon and to have its "niceness" set to 4. These daemons then have as much CPU power and bandwidth as they need to submit the block to the network. Since multiple coins are always being mined simultaneously, there are usually several daemons in this state at any given time. When the 10s priority mode expires, bandwidth and CPU usage is returned to normal for that daemon until the next block is discovered.

Each of the servers also has a cron job that runs a bash script once per two hours. It queries a table in the database and determines which coin was most recently restarted, and then restarts the next coin in the list. Given that there are 80 daemons on each server, this script ensures that every daemon is restarted once per week. Restarting daemons is necessary to deal with coins that contain memory leaks.

The interesting part about the priority mode implementation is that we could probably run even more daemons on these servers, some of which have many more blocks than bitcoin does. The problem with daemons is not that they take up too many resources; it's that they take up resources at the wrong time, and they don't have the privileges to determine themselves when to delay their work. Because they don't have root access, coins like bitcoin can't see what else is happening to decide which tasks are important (like publishing blocks) and which can wait (like uploading old blocks to a new client). But what is clear is that it makes no sense to state that bitcoin and other networks can't handle 8MB blocks.

With 300 lines of code, consumer grade hardware can run 85 networks with no increase in orphan rates or lost blocks. Unlike bitcoin, many of these networks are developed by inexperienced coders who create memory leaks and waste CPU. In addition, all of these daemons are running with txindex=1, and every single block and transaction received by any daemon results in an API call to index it in the block explorer at https://prohashing.com/explorer.html.

Taking into account the depreciation, electricity, and bandwidth used since July, the cost of running all these services for all these networks for six months has been a grand total of $2.50 each.