Importing OpenStreetMap™ into YAAC

YAAC has been deliberately designed to not use external map servers. Considering the amateur radio operator's legal and moral obligation to provide emergency communications (in the United States, it's FCC regulations Part 97, paragraph §1(a)), a good APRS application should be able to function solely over amateur radio communications, not requiring the Internet or any commercial services only available over the Internet; though Internet-based services can be used, they cannot be mandatory for the functioning of a good APRS client application.

So, YAAC is designed to use locally-loaded map data. YAAC currently supports the OpenStreetMap database, downloadable from https://planet.openstreetmap.org and its mirror websites and bit torrents, both in total and in segments. For efficiency in rendering the map data, YAAC compiles the bulky and inefficient OpenStreetMap XML data (or slightly less bulky protocol buffers [PBF] format) into a compact and edited-down format specific to YAAC and optimized for real-time local map rendering. This compilation is done with the File->OpenStreetMap->Import RAW OSM Map File menu choice.

Because compiling large portions of the world map data to YAAC binary format can take considerable time (possibly literally days of processing time), a short-cut has been provided to copy already-compiled map data from the YAAC author's website. This option is available through the File->OpenStreetMap->Download Pre-Imported Tiles menu choice, and selects downloading the map data for a specified area around the area you are currently viewing.

Prerequisites:

If you are going to import the OpenStreetMap data yourself, first you must obtain a copy of the OpenStreetMap dataset in bzip2'd XML format (filetype ".osm.bz2") or Google protocol buffers format (filetype ".osm.pbf") from the OpenStreetMap website or one of its mirrors or torrents. Note that the complete planetary dataset (as of October 2023) is over 132GB of size in bzipped XML format or 72GB in protocol buffer (PBF) format, so expect it to take a long time to download; alternatively, use a subset datafile (pre-segmented by one of the mirror websites) appropriate to the geographical area you are interested in. To assist you in finding the websites with OpenStreetMap data, the File->OpenStreetMap->Download OpenStreetMap Extract Dataset menu choice is available to steer your web browser to some of the OpenStreetMap website mirrors.

Secondly, you need a system with YAAC installed and at least 156GB of free disk space after downloading the OSM dataset (assuming the full planetary dataset; smaller datasets will still need at least 92GB because of the temporary indexing files), to buffer the temporary files used to geographically segment the OSM dataset into tiled subsets that are efficient to scan through for rendering. At the present OSM dataset size, 114GB of the full planetary size will be recyclable after the import completes. Also, you will need to run YAAC with a non-default larger memory heap size of at least 2048MB (so your computer will need at least 3GB of RAM to perform well); 3GB just of heap or more is preferred. Due to the heap thrashing caused by the more efficient protocol buffers file format, you may need to set the Java initial heap size (-Xms) to the same value as the maximum heap size (-Xmx); some releases of the Java garbage collector have a problem with the particular memory utilization patterns of the PBF file importer and may cause spurious failures due to excessive garbage collection overhead.

Some Linux system kernels (particularly those in the 5.10.x to 5.15.x version range) have a problem with how their filesystem caching interacts with YAAC's huge amount of semi-random-access to gigabytes of data involved in the OpenStreetMap import. The author of YAAC has found that setting the kernel parameter vm.swappiness as low as 1 slightly relieves these symptoms, but not sufficiently. Also, if you are using a swap file, you may want to lower the overcommit ratio (how much virtual memory above actual physical memory that can be allotted to processes). You may need to run a script as root to force dropping the filesystem caches when they fill up enough to cause a memory crisis on your system; thankfully, the most recent kernel versions (5.16.11 and up) seem to have resolved this problem.

Note that you can copy the compiled map data (or tiles thereof) to other smaller computers after the compilation is completed, as long as they are running the same version of YAAC. Note that different releases of YAAC may change the compiled format in incompatible ways; to correct such problems, simply re-import the map data.

Thirdly, if you wish proper representation of oceans and major salt-water seas on the map, you will want to use the OSMcoastline tool to extract coastline boundaries from the OSM dataset and produce closed ocean boundaries suitable for reprocessing by the YAAC importer. Download the tool from https://github.com/osmcode/osmcoastline along with its dependencies and build it, then, prior to importing the OSM dataset into YAAC, execute osmcoastline on the downloaded OSM datafile to extract the water polygons into a database, using a command something like

The ESRI shapefiles files you then want are named water_polygons.dbf (and related filetypes .prj, .shp, and .shx), and should be copied into your configured tile directory before starting the import. If a water_polygons.shp file is found in the tile directory when node import processing is completed, the coastlines database will be processed to extract all segments of the ocean bodies and added to the ways files of the imported dataset.

Compiling the map data

Ensure that your installation of YAAC is configured to place the tile directory in a disk partition that meets the above prerequisites for free space. Then select the File menu. OpenStreetMap submenu, and the Import Raw OSM Map File menu choice. This will display a file selection dialog:

Select the OpenStreetMap dataset file (in compressed OSM XML format or protocol buffer format) that you wish to import. Then, adjust the import bounding box settings for the rectangular portion of the planet you wish to extract from the OSM file; this allows you to not waste disk space and considerable processing time compiling parts of the OSM data that you will never use. Then click the Open button to begin the import.

A progress dialog will now be displayed, reporting the statistics of importing the file, including counts of OpenStreetMap changesets, nodes, ways, and relations read from the input file and written to tile files.

You may continue to use YAAC normally while the import is proceeding (although the strain the importer puts on the computer for large datasets may impact responsiveness). Note that the import will take several hours (at least 2 hours, possibly as long as 2 days for the entire planet, depending on your CPU and disk drive performance, how much of the planet you are reading, and whether you are using compressed XML or protocol buffer file format), so do not turn off your computer, click the Cancel button on the progress dialog, or stop running YAAC while the import is in progress. We suggest that you leave it running overnight, so that you will have your map data ready when you return to your computer in the morning. When the import is completed, the progress dialog will automatically close. At this point, any pan or zoom of the map display will cause rendering of the street map data for display in the background (assuming map display has not been disabled by the Select Geographical Map Layers menu choice).