System offline

News updates about the Prohashing pool
Post Reply
User avatar
Steve Sokolowski
Posts: 3827
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

System offline

Post by Steve Sokolowski » Fri Jun 07, 2019 5:50 pm

The system is currently offline. Chris tried to restart the mining servers to resolve an issue where some customers reported that "Live miner status" was not displaying their miners. However, when he attempted to restart the servers, a "no module named Cryptography" error occurred and the servers will not start. There were no code changes to cause the issue.

Chris is trying to determine what caused the problem and will provide more information as soon as he has some. He suspects the cause is that a Debian automatic update changed a package configuration.
User avatar
Chris Sokolowski
Site Admin
Posts: 934
Joined: Wed Aug 27, 2014 12:47 pm
Location: State College, PA

Re: System offline

Post by Chris Sokolowski » Fri Jun 07, 2019 6:29 pm

This issue is fixed. I had to force downgrade a package.
User avatar
dnprod
Posts: 81
Joined: Tue Feb 21, 2017 3:19 pm
Location: Ontario, Canada

Re: System offline

Post by dnprod » Sat Jun 08, 2019 7:28 am

it seems rather, um, bold, to allow automatic updates on a production system.
User avatar
Steve Sokolowski
Posts: 3827
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Re: System offline

Post by Steve Sokolowski » Sat Jun 08, 2019 8:45 am

dnprod wrote:it seems rather, um, bold, to allow automatic updates on a production system.
I'm not sure I agree. Testing every single update before release would cost so much that we would have to raise our fees. We'd have to hire someone solely to install packages on development.

The current process is that Chris reviews the updates that are available, and then sets calls aptitude upgrade to automatically upgrade the packages once he confirms there is nothing obviously wrong about them. This is the only time in two years that an upgrade using this method has caused any downtime, so I think it's fine.

The procedure saved a lot of potential problems because Chris was available when the servers were restarted to deal with the consequences of the update. One way of doing this that I would not recommend is to just allow the updates to install without Chris or anyone else having any input, because then the failure could have occurred when nobody was around to notice or resolve it.
User avatar
dnprod
Posts: 81
Joined: Tue Feb 21, 2017 3:19 pm
Location: Ontario, Canada

Re: System offline

Post by dnprod » Sat Jun 08, 2019 1:57 pm

ah, so not fully automatic
Post Reply