Status as of Wednesday, June 17 (ATTN: rootdude)

Here's today's status:

• Chris's health is poor this week, complicating dealing with the performance issues. We are all hands on deck trying to eliminate problems.
• Yesterday, we transferred 600m rows from many tables across the Internet to permanent archival and then deleted them. An autovacuum is running now to clean up the dead tuples.
• I looked at the website logs and noticed that there were some slow queries, so I fixed them. Performance of the website seems to have improved dramatically.
• Chris has moved 20 of the 150 or so running daemons to the new servers. These are cheap, desktop-grade computers, but because they were bought a year after the original server, they are much faster. I want Chris to get 80 more daemons off the large server that hosts the database before we evaluate performance further.
• One issue I was hoping rootdude could comment on is that we discovered that 8TB array is 76% non-contiguous. There do not appear to be any Linux defragmentation utilities, so I'm not sure what to do about that. The obvious solution would appear to be deleting files to free up contiguous space, which we are doing as much as possible, but it doesn't move existing pieces around. rootdude, do you have any suggestions for resolving this problem?
• I'll continue to update this post throughout the day as more performance issues are addressed. We're working as hard as possible on these issues and hope to continue to make progress today.

Steve Sokolowski wrote:Here's today's status:

• One issue I was hoping rootdude could comment on is that we discovered that 8TB array is 76% non-contiguous. There do not appear to be any Linux defragmentation utilities, so I'm not sure what to do about that. The obvious solution would appear to be deleting files to free up contiguous space, which we are doing as much as possible, but it doesn't move existing pieces around. rootdude, do you have any suggestions for resolving this problem?

It's really very simple. Assuming the partition the Linux server is using is formatted as ext2,3 or 4, 'fragmentation' is quite normal. Those ext formats leave space between files when writing them:

QuoteHow Linux File Systems Work
Linux’s ext2, ext3, and ext4 file systems – ext4 being the file system used by Ubuntu and most other current Linux distributions – allocates files in a more intelligent way. Instead of placing multiple files near each other on the hard disk, Linux file systems scatter different files all over the disk, leaving a large amount of free space between them. When a file is edited and needs to grow, there’s usually plenty of free space for the file to grow into. If fragmentation does occur, the file system will attempt to move the files around to reduce fragmentation in normal use, without the need for a defragmentation utility.

image

Because of the way this approach works, you will start to see fragmentation if your file system fills up. If it’s 95% (or even 80%) full, you’ll start to see some fragmentation. However, the file system is designed to avoid fragmentation in normal use.

If you do have problems with fragmentation on Linux, you probably need a larger hard disk. If you actually need to defragment a file system, the simplest way is probably the most reliable: Copy all the files off the partition, erase the files from the partition, then copy the files back onto the partition. The file system will intelligently allocate the files as you copy them back onto the disk.

As the article suggests, if you want to see a smaller fragmentation figure, you can copy the files off the partition, erase the partition, then copy the files back... but the benefit will be minimal at best. I wouldn't worry about that fragmentation figure at all. Carry on guys.

Thanks - that helps.

In that case, I think Chris's current course of action is best - moving the daemons to the other servers, so that the database is the primary or (if we can) the only user of these disks. As you can see by today's stability, I think that even the little Chris has done so far has significantly improved the disks.

Performance will go up even more once enough daemons are gone that the CacheCade can flush its writes to disk, and the autovacuum daemon can run. There are over 300m dead rows in the database right now, which are making joins take much longer than they should.

The best part about this system is that it is nearly infinitely parallelizable. There are so many daemons, and so many scripts, and postgres is multithreaded, that it's always possible to buy more computers and spread it all out.

By the way, in case anyone is interested, our wallet.dat files were over 3GB from receiving so much change. We've found 0.53m blocks during the past 32 days.

Steve Sokolowski wrote:
By the way, in case anyone is interested, our wallet.dat files were over 3GB from receiving so much change. We've found 0.53m blocks during the past 32 days.

This is crazy - never even considered the possibility of bloat like that.

Suggestion: -salvagewallet is the one command that might help a bit, at least it does exactly this - grabs key pairs it can find in wallet.dat, create a new wallet.dat with these key pairs and rescan blockchain to get balance right. However it will also fill wallet.dat with some additional info about txes/addresses that your key pairs belongs to so size might be almost the same as you have now. In my experience, however, it should help the wallet performance quite a bit.

NOTE: Make a backup before messing up with your wallet!

P.S. Tell Chris I hope he recovers quickly...

Maybe (probably) a stupid idea, but here goes. I wonder how many daemons could run happily on a RasPi or the like?

kires wrote:Maybe (probably) a stupid idea, but here goes. I wonder how many daemons could run happily on a RasPi or the like?

The one thing a RPi doesn't do well is I/O operations because of the storage card limitations (though iSCSI might work)... they'd be butt slow and the SD cards would wear out in a matter of days given the sizes of the wallet.dat files, much less the blockchain reads and writes. Also, the RAM is so limited, the RPi's would be swapping like no one's business...

Sorry man, but you called it right - not a good idea