maybe getting the access point from 100 Mbit to 1000 will fix some flakiness
For reference, my hardware is an 8700K, 1TB 970 Pro NVMe, and 64 GB DDR4 RAM. I ran these tests on Ubuntu Desktop 21.04 with Postgres 13.3, and osm2pgsql 1.4.1.
I would also like to look into optimizations for the tile serving process, which will require different Postgres settings for a very read-heavy workload.
I also tried a full planet file import, just to see how long it would take. In 2014 at work we used 2010 server hardware to do a planet import in about 5 days. Back then the planet was about 24 GB; now it is 62 GB.
I choose a 40 GB node cache for the import and disabled autovacuum in PG. The main import and indexing took 14 hours, and an additional 6 hours for a manual "VACUUM FULL VERBOSE" afterwards.
That's pretty good! The postgres dataset in ZFS is 395 GB. For the entire planet import.
Optimizing RAM usage will provide better import performance gains than your choice of filesystem.
I was surprised to find that ZFS compression didn't help, as I recall it used to help about 5 years ago on a server at work.
The OSM planet file is available by a torrent since Nov 2020, btw.
I've been doing some testing of the OpenStreetMap database import process and trying to find what "tweaks" actually work to make osm2pgsql faster. After some testing with the Canada extract region (2.6G), I have a short list.
* Use SSDs, preferably NVMe drives
* Use a good node cache size (I found 2x extract size to be good)
* Use a flat node cache
* Optimize postgresql memory settings
* Set your CPU governor to performance mode
* ZFS needs tweaking to match performance of XFS and ext4
Poor quality glue on this steel series mouse, the thumb pads have been loose for over a year. After using superglue to put them back on, they are sticking but the side buttons are “weird” now.
Going to recycle this junk and go back to the G5 I kept in storage. I’ll keep looking for an ambidextrous mouse that is properly made.
Calgary. I still like Ruby
Server run by the main developers of the project It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!