- My main focus today is the memory leak that is causing the mining servers to need to be restarted every few hours. You might have noticed your own miners disconnecting, or seen another mining server go offline and drop down the hashrate for a few minutes. While this only causes a brief interruption to mining, we've found that stability is the #1 draw to mining pools. I spent the day yesterday trying to set up a remote debugger and failed, so today I'm going to write a lot of debug code and deploy it to one mining server to get some data.
- There are several other known issues that I want to work on next. Some customers are reporting that payout rows in the "Earnings" page specify "Invalid Date," but after submitting the tickets, they fail to reply to the questions I ask to troubleshoot the issue. If you see this problem, please submit a ticket and continue the conversation so that we can figure out how to reproduce it and fix the issue. We usually can't resolve issues submitted in tickets without being provided the requested information.
- Another known issue is that miners stop reporting to "live coin status" periodically. Since this problem does not cause a loss of money, it's a lower priority, and you can continue mining even if that happens to you.
- Chris added 12 new coins and discontinued 90 coins last night, now that he is starting to be able to get caught up on maintenance. While the focus has been on the demise of Bitconnectcoin, a lot of the recent drop in profitability has resulted from the demise of Novaexchange, which made 1/4 of scrypt coins worthless overnight because they have no exchanges to trade on.
- We now are down to 23 tickets in the old support system, and Chris is assigned 21 in the new system. He next needs to fix the DKIM mismatch in our E-Mail server, and then will be moving full-time onto resolving the support tickets. Once he resolves all the tickets, then he'll be able to work to increase profitability by adding new exchanges and even more coins.
The key issue is that when people are hired, they reduce productivity for a month or two, not increase it. First, it takes a lot of time to find candidates and select one. Then, the person needs to be trained. The training not only means that the new hire is unproductive, but it sucks up another employee's time for a week or two. We saw this two weeks ago when Constance came onboard. She is an exceptional employee and figured out most cryptocurrency concepts on the first day, but it still took days to configure a development environment. Our response to support tickets suffered and the tickets went up to 220, before falling to the current 30 or 40.
Of course, we want to hire people as quickly as possible, but there are a number of critical issues coming up that are going to need immediate attention. We need to finish training people so that those people can then supervise and train the next people, leaving me free to continue with normal development. My goal is that we can get Constance trained and then have her hire someone, and then the two of them can hire two more, resulting in exponential growth while still allowing some progress to continue.
In some industries, like government contracting, the "solution" to a schedule slippage is often to double the staff and expect double the work to get done. The government is then surprised to find that the problem got worse, instead of better. The reality is that work proceeds at half-pace for the first two months, and then after six months about 1.9 times the work gets accomplished, because there is 10% more overhead in supervision and communication. People who like the XKCD comics will probably find a treasure trove of them referencing this idea.
I think it's really important for people to understand that when an exchange goes down, the problem usually isn't incompetence or lack of planning. The technology exists to solve the capacity problems. It's the limitation of humans of being able to learn quickly enough and train others to do the same thing. I wonder if there has ever been an industry in the history of the world that has grown so quickly. This may be the first industry where growth is not limited by the demand for a product, but by the ability to hire enough people to keep up with the demand.
