public abstract class OsmSegmenter
extends java.lang.Object
Modifier and Type | Class and Description |
---|---|
protected static class |
OsmSegmenter.AcceptTileSubdirsOrPlanetWays
This class provides a FileFilter for identifying the relevant temporary files for sorting
the results of an OSM import.
|
protected static class |
OsmSegmenter.BoundsToNameTag
Class of bounding box to name of an area.
|
protected static class |
OsmSegmenter.OpenRandomAccessFileCache
LIFO cache for random access files, so as to minimize file opens and reloads while
not exceeding the maximum number of open file descriptors.
|
static interface |
OsmSegmenter.ReportUpdates
Callback interface for OsmSegmenter instances reporting updates to a progress indicator.
|
protected class |
OsmSegmenter.WayFileSorter
This class handles the logic of sorting a temporary .ways file into the proper order for
correct rendering.
|
Modifier and Type | Field and Description |
---|---|
static long |
ESTIMATED_EXTRA_TILE_SPACE
Kilobytes of free disk space needed on the tile directory's partition to handle importing a new OSM file,
over and above the size of the generated new tile way and node files.
|
static long |
ESTIMATED_TEMP_TILE_SPACE
Kilobytes of free disk space needed on the temporary directory's partition to handle importing a new OSM file.
|
protected static java.lang.String[] |
FILETYPE_SUFFIXES
Filetypes for different types of Way files.
|
protected static int |
FILETYPE_SWAY
Flag indicating OSM Ways not properly labeled for rendering.
|
protected static int |
FILETYPE_WAY
Flag indicating OSM Ways properly labeled for rendering.
|
protected boolean |
flushUntiled |
protected int |
fullPostalQueueDelay
Total number of milliseconds the OSMImporter thread was blocked because the
postal_address writing queue was full.
|
protected StatusListener |
listener |
protected static int |
MAX_LATITUDE
Maximum latitude in millionths of degrees.
|
protected static int |
MAX_LONGITUDE
Maximum longitude in millionths of degrees.
|
protected static int |
MAX_WAY_READ_FILES
Maximum number of Way files being read back to build Relations (to avoid exceeding
operating system open file limits).
|
protected static int |
MIN_LATITUDE
Minimum latitude in millionths of degrees.
|
protected static int |
MIN_LONGITUDE
Minimum longitude in millionths of degrees.
|
protected int |
minWaysToWrite
Smallest backlog of Ways to consider opening a .ways file for writing.
|
protected long |
numAddressesSaved
Number of postal_address records written to database.
|
protected int |
numDupWaysDeleted
Counter of number of duplicate Ways deleted from the collection of Way files.
|
protected int[] |
numInvalidPostal
Number of invalid conditions for postal_address records not written to database.
|
protected int |
numNodeFiles
Counter of processed Node files (used by common reporting code).
|
protected int |
numPatchedAddresses
Total number of address records modified by Relations with additional address data,
|
protected int |
numPostalQueueEmpty
Number of times the postal_address writing queue ran empty.
|
protected int |
numPostalQueueFull
Number of times the postal_address writing queue filled up.
|
protected int |
numRelations |
protected int |
numSortFailures
Counter of failed sorts of Way and Node files.
|
protected int |
numTileWays |
protected int |
numWayFiles
Counter of processed Way files (used by common reporting code).
|
protected int |
numWayReaderOpens |
protected int |
numWayReads |
protected java.io.PrintStream |
out
PrintStream used to write the OSM import log file.
|
protected NonshareableBufferedDataOutputStream |
planetDos
Shared area for stream to planet.ways.tmp file.
|
protected static org.ka2ddo.yaac.osm.OsmSegmenter.RelationsStatsMapEntryComparator |
RELATIONS_STATS_MAP_ENTRY_COMPARATOR
Comparator to sort the per-type statistics for OSM Relations in descending order of appearances.
|
static java.lang.String |
S_LOG_TIMESTAMP_FMT
Format string for SimpleDateFormat
sdfLogTimestamp . |
protected java.text.SimpleDateFormat |
sdfLogTimestamp
Non-thread-safe date formatter for logging.
|
protected static java.io.File |
tileRootDirectory
File identifying the directory used by YAAC to store map tile file latitude directories.
|
protected java.io.File |
tileTmpDirectory
File identifying the directory used by OpenStreetMap importers for high-data-access-rate temporary files.
|
protected java.util.ArrayList<java.io.File> |
unsavedWayFileCache |
protected int |
unsavedWaysFileCounter |
protected NonshareableBufferedDataOutputStream |
untiledDos |
protected static byte[] |
WAY_INDEX_PREFILL
Buffer initialization pattern for non-existent way index records.
|
protected static int |
WAY_READBACK_BUF_SIZE
Size of buffers for
MultiBufferedRandomAccessFile s used to read back Ways from ways files. |
protected static int |
WAY_READBACK_MAX_BUFFERS
Quantity of buffers for
MultiBufferedRandomAccessFile s used to read back Ways from ways files. |
protected HugeBitSet |
wayIdUsedSet
BitSet keeping track of Way ID numbers that have been used.
|
protected MultiBufferedRandomAccessFile |
wayIndex
Disk-buffered index of where all the complete Ways (indexed by their ID's) are stored on
disk.
|
protected boolean |
wayQueuesAreEmpty |
protected GenericTaggedQueue[] |
wayWriterQueues |
protected java.lang.Thread |
wayWriterThread |
protected java.util.HashSet<java.lang.Number> |
zeroLengthWayIds |
Modifier | Constructor and Description |
---|---|
protected |
OsmSegmenter(java.lang.String rootTilePath)
Abstract pre-constructor for any OSM file segmenter,
|
Modifier and Type | Method and Description |
---|---|
java.lang.String |
checkDiskSpace(long filesize)
Check if sufficient disk space is available to import the OSM file.
|
protected void |
closePostalAddresses()
Finish any final batch of inserts and close the connection to the DB
(if it was opened).
|
protected static int |
computeActiveQueueCount(GenericTaggedQueue[] queues)
Helper method to compute the current count of non-empty queues.
|
protected static int |
computeQueuesBacklog(GenericTaggedQueue[] queues)
Helper method to compute the current backlog count of queued objects.
|
protected void |
copyAddressFromNodeToOther(Relation relation)
For relation type=associated_street, copy the address information from the role=address node
to all the other elements of the relation.
|
static void |
ensurePostalTablesExist(StatusListener listener,
java.sql.Connection dbConn,
java.io.PrintStream out)
Verify that the database tables for postal addresses have been created,
create them if not, and empty the tables if they already exist so a new set
of records can be inserted.
|
protected static GenericTaggedQueue |
findBiggestQueue(GenericTaggedQueue[] writerQueues,
int upperLimitQueueSize)
Helper method to find the queue with the largest backlog in an array of queues.
|
protected void |
flushPostalAddressesToDB()
Flush any incomplete batches of PostalAddress record inserts.
|
protected org.ka2ddo.yaac.osm.OsmSegmenter.WayIndexRecord |
getFileForIndexedWay(long wayId) |
protected abstract long |
getMinHeapRequired()
Report the minimum total size of data structures that will be simultaneously
allocated during import.
|
protected int |
getPostalAddressQueueSize()
Get the number of entries in the postal address writing queue.
|
protected Way |
getWayForRelation(org.ka2ddo.yaac.osm.OsmSegmenter.WayIndexRecord wayIndexRecord) |
protected boolean |
isPostalDbInUse()
Test if the OsmSegmenter is using the postal DB.
|
abstract void |
parse(java.io.FileInputStream fileInputStream,
NonshareableCountingBufferedDataInputStream inputStream,
java.io.File inputFile,
boolean useFixMe,
StatusListener listener)
Parse a OpenStreetMap snapshot file into YAAC's version of OpenStreetMap tiles.
|
protected static java.lang.String |
processNonStdAmenityTypes(GenericTaggedNode<?> gtn,
java.lang.Object[] decodedTagArray,
java.lang.String origAmenityTypeName,
java.lang.String amenityTypeName)
Process non-standard amenity types that have had prefixes or suffixes attached
to the type name value.
|
protected java.lang.String |
produceSortStatusMessage(java.io.File latDir)
Generate a string reporting the current state of sorting the tile files.
|
static void |
refreshMapWindows()
Tell all map windows to redraw their maps based upon the updated map tile files.
|
protected void |
tryToSetUpPostalDB(StatusListener listener)
Attempt to set up access to the postal database.
|
protected void |
updateAddressesWithStreet(Relation relation,
java.lang.String relationType)
Patch PostalAddress DB records that are missing their street names but have
them provided by a Relation of type "associatedStreet" or "street".
|
protected void |
writePostalAddressToDb(PostalAddress postalAddress,
int latitude,
int longitude)
If the postal DB is in use, insert a PostalAddress record into the database.
|
public static final long ESTIMATED_EXTRA_TILE_SPACE
public static final long ESTIMATED_TEMP_TILE_SPACE
ESTIMATED_EXTRA_TILE_SPACE
,
Constant Field Valuesprotected static final int MAX_LATITUDE
protected static final int MIN_LATITUDE
protected static final int MAX_LONGITUDE
protected static final int MIN_LONGITUDE
protected static final java.lang.String[] FILETYPE_SUFFIXES
FILETYPE_WAY
,
FILETYPE_SWAY
protected static final int MAX_WAY_READ_FILES
protected static final int WAY_READBACK_BUF_SIZE
MultiBufferedRandomAccessFile
s used to read back Ways from ways files.protected static final int WAY_READBACK_MAX_BUFFERS
MultiBufferedRandomAccessFile
s used to read back Ways from ways files.protected static final byte[] WAY_INDEX_PREFILL
protected static final int FILETYPE_WAY
protected static final int FILETYPE_SWAY
public static final java.lang.String S_LOG_TIMESTAMP_FMT
sdfLogTimestamp
.protected final java.text.SimpleDateFormat sdfLogTimestamp
protected int numRelations
protected transient int numWayReaderOpens
protected transient int numWayReads
protected StatusListener listener
protected int numTileWays
protected transient java.util.HashSet<java.lang.Number> zeroLengthWayIds
protected transient java.lang.Thread wayWriterThread
protected transient GenericTaggedQueue[] wayWriterQueues
protected transient boolean wayQueuesAreEmpty
protected transient int unsavedWaysFileCounter
protected transient NonshareableBufferedDataOutputStream untiledDos
protected transient boolean flushUntiled
protected transient MultiBufferedRandomAccessFile wayIndex
protected transient HugeBitSet wayIdUsedSet
protected transient java.util.ArrayList<java.io.File> unsavedWayFileCache
protected int minWaysToWrite
protected static final org.ka2ddo.yaac.osm.OsmSegmenter.RelationsStatsMapEntryComparator RELATIONS_STATS_MAP_ENTRY_COMPARATOR
protected static java.io.File tileRootDirectory
protected final java.io.File tileTmpDirectory
tileRootDirectory
protected transient NonshareableBufferedDataOutputStream planetDos
protected transient int numWayFiles
protected transient int numNodeFiles
protected transient int numSortFailures
protected transient int numDupWaysDeleted
protected transient java.io.PrintStream out
protected transient long numAddressesSaved
protected transient int[] numInvalidPostal
protected transient int numPostalQueueEmpty
numPostalQueueFull
protected transient int numPostalQueueFull
fullPostalQueueDelay
,
writePostalAddressToDb(PostalAddress, int, int)
protected transient int fullPostalQueueDelay
numPostalQueueFull
protected transient int numPatchedAddresses
updateAddressesWithStreet(Relation, String)
protected OsmSegmenter(java.lang.String rootTilePath) throws java.io.IOException
rootTilePath
- String pathname of the configured tile directoryjava.io.IOException
- if tile directory cannot be createdpublic abstract void parse(java.io.FileInputStream fileInputStream, NonshareableCountingBufferedDataInputStream inputStream, java.io.File inputFile, boolean useFixMe, StatusListener listener) throws java.io.IOException, javax.xml.parsers.ParserConfigurationException, org.xml.sax.SAXException
fileInputStream
- FileInputStream underlying the input stream being usedinputStream
- NonshareableCountingBufferedDataInputStream reading the snapshot file of whatever format the implementing class supportsinputFile
- File of the OpenStreetMap snapshot file being readuseFixMe
- boolean true if FIXME-tagged elements of the OSM data are to be includedlistener
- StatusListener to receive progress report of the importjava.io.IOException
- if the input file cannot be read or any of the output files cannot be writtenjavax.xml.parsers.ParserConfigurationException
- if the Java runtime's XML parser cannot handle its configuration parametersorg.xml.sax.SAXException
- if the XML parser cannot decode the fileprotected abstract long getMinHeapRequired()
public java.lang.String checkDiskSpace(long filesize) throws java.io.IOException
filesize
- long byte-count size of the OSM file to read (to estimate resulting tile size)java.io.IOException
- if FileStores (partitions) for the two directories could not be obtainedprotected static GenericTaggedQueue findBiggestQueue(GenericTaggedQueue[] writerQueues, int upperLimitQueueSize)
writerQueues
- array of GenericTaggedQueuesupperLimitQueueSize
- int threshold of largest queue capacity, so scan can stop as soon as the
fullest possible queue is foundprotected static int computeQueuesBacklog(GenericTaggedQueue[] queues)
queues
- array of GenericTaggedQueuesprotected static int computeActiveQueueCount(GenericTaggedQueue[] queues)
queues
- array of GenericTaggedQueuesprotected java.lang.String produceSortStatusMessage(java.io.File latDir)
latDir
- File of the latitude diectory currently being processedprotected static java.lang.String processNonStdAmenityTypes(GenericTaggedNode<?> gtn, java.lang.Object[] decodedTagArray, java.lang.String origAmenityTypeName, java.lang.String amenityTypeName)
gtn
- GenericTaggedNode with the amenityTypeName valuedecodedTagArray
- GenericTaggedNode.decodedtagArray
of the gtn parameterorigAmenityTypeName
- the String value of amenityTypeName before case-normalization and tokenizingamenityTypeName
- String of the current normalized amenityTypeName token to processprotected Way getWayForRelation(org.ka2ddo.yaac.osm.OsmSegmenter.WayIndexRecord wayIndexRecord) throws java.io.IOException
java.io.IOException
protected org.ka2ddo.yaac.osm.OsmSegmenter.WayIndexRecord getFileForIndexedWay(long wayId) throws java.io.IOException
java.io.IOException
public static void refreshMapWindows()
protected void tryToSetUpPostalDB(StatusListener listener)
out
log file has been set up by the subclass's implementation
of parse(FileInputStream, NonshareableCountingBufferedDataInputStream, File, boolean, StatusListener)
.listener
- StatusListener to receive DB action statusespublic static void ensurePostalTablesExist(StatusListener listener, java.sql.Connection dbConn, java.io.PrintStream out) throws java.sql.SQLException
listener
- StatusListener to receive updates, or null to not report to the UIdbConn
- java.sql.Connection to use to access the databaseout
- PrintStream to receive log messagesjava.sql.SQLException
- if DB cannot be queried or tables cannot be created or emptiedprotected boolean isPostalDbInUse()
protected int getPostalAddressQueueSize()
protected void writePostalAddressToDb(PostalAddress postalAddress, int latitude, int longitude)
postalAddress
- PostalAddress
record to insertlatitude
- int latitude in millionths of degrees Northlongitude
- int longitude in millionths of degrees Eastprotected void flushPostalAddressesToDB() throws java.io.IOException
java.io.IOException
- if inserts fail for any reasonprotected void updateAddressesWithStreet(Relation relation, java.lang.String relationType)
relation
- Relation proving the street namerelationType
- relation typenumPatchedAddresses
protected void copyAddressFromNodeToOther(Relation relation) throws java.io.IOException
relation
- Relation to analyzejava.io.IOException
protected void closePostalAddresses() throws java.io.IOException
java.io.IOException
- if final inserts or close failed