System offline

The system is currently offline. Chris tried to restart the mining servers to resolve an issue where some customers reported that "Live miner status" was not displaying their miners. However, when he attempted to restart the servers, a "no module named Cryptography" error occurred and the servers will not start. There were no code changes to cause the issue.

Chris is trying to determine what caused the problem and will provide more information as soon as he has some. He suspects the cause is that a Debian automatic update changed a package configuration.

This issue is fixed. I had to force downgrade a package.

it seems rather, um, bold, to allow automatic updates on a production system.

dnprod wrote:it seems rather, um, bold, to allow automatic updates on a production system.

I'm not sure I agree. Testing every single update before release would cost so much that we would have to raise our fees. We'd have to hire someone solely to install packages on development.

The current process is that Chris reviews the updates that are available, and then sets calls aptitude upgrade to automatically upgrade the packages once he confirms there is nothing obviously wrong about them. This is the only time in two years that an upgrade using this method has caused any downtime, so I think it's fine.

The procedure saved a lot of potential problems because Chris was available when the servers were restarted to deal with the consequences of the update. One way of doing this that I would not recommend is to just allow the updates to install without Chris or anyone else having any input, because then the failure could have occurred when nobody was around to notice or resolve it.

ah, so not fully automatic