Share processing delayed

Steve Sokolowski avatar
Steve Sokolowski
Archived posts: 4559
Joined: Aug 27, 2014

Re: Share processing delayed

by Steve Sokolowski · · Post #30355

We determined the cause of this problem.

It turns out that there is a bug in the PostgreSQL 12 to 13 upgrader, which is maintained by the Postgres developers. PostgreSQL 13 provides support for a faster index format, and part of the upgrade process is to create those new indexes.

For an unknown reason, the upgrader failed to create the newer indexes on some tables. Chris assumed, as was written in the documentation, that failure would simply mean that the old indexes would be used. Instead, what actually happened is that there are now multiple indexes that are all updated on each INSERT ON CONFLICT statement, while only one of them is actually needed for reads, as the bug in the upgrader did not delete the corrupt indexes. That causes four disk writes for some tables that only needed two before, halving the database's performance.

Chris is too tired to continue working on the issue, and it looks like the database has almost caught up. Tonight, he will contact the Postgres developers and attempt to figure out how to correct this problem in a live system that has 2TB of data.