How do I divide a very large OpenStreetMap file into smaller files in R without running out of memory?

Max Candocia Published at Dev

Max Candocia

I am currently looking to have map files that are no larger than the sizes of municipalities in Mexico (at largest, about 3 degrees longitude/latitude across). However, I have been running into memory issues (at the very least) when trying to do so. The file size of the OSM XML object is 1.9 GB, for reference.

library(osmar)
get.map.for.municipality<-function(province,municipality){
  base.map.filename = 'OpenStreetMap/mexico-latest.osm'
  #bounds.list is a list that contains the boundaries
  bounds = bounds.list[[paste0(province,'*',municipality)]]
  my.bbox = corner_bbox(bounds[1],bounds[2],bounds[3],bounds[4])
  my.map.source = osmsource_file(base.map.filename)
  my.map = get_osm(my.bbox,my.map.source)
  return(my.map)
}

I am running this inside of a loop, but it can't even get past the first one. When I tried running it, my computer froze and I was only able to take a screenshot with my phone. The memory steadily inclined over the course of a few minutes, and then it shot up really quickly, and I was unable to react before my computer froze.

What is a better way of doing this? I expect to have to run this loop about 100-150 times, so any way that is more efficient in terms of memory would help. I would prefer not to download smaller files from an API service. If necessary, I would be willing to use another programming language (preferably Python or C++), but I prefer to keep this in R.

enter image description here

defvol

I'd suggest not use R for that.

There are better tools for that job. Many ways to split, filter stuff from the command line or using a DBMS.

Here are some alternatives extracted from the OSM Wiki http://wiki.openstreetmap.org:

Filter your osm files using osmfilter: "osmfilter is used to filter OpenStreetMap data files for specific tags. You can define different kinds of filters to get OSM objects (i.e. nodes, ways, relations), including their dependent objects, e.g. nodes of ways, ways of relations, relations of other relations."

Clipping based on Polygons or borders using osmconvert: http://wiki.openstreetmap.org/wiki/Osmconvert#Applying_Geographical_Borders

You can write bash scripts for both osmfilter and osmconvert, but I'd recommend using a DBMS. Just import into PostGIS using osm2pgsql, and connect your R code with any Postgresql driver. This will optimize your read/write ops.

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at2021-02-18

Comments

0 comments

From Dev

How to append several large data.table objects into a single data.table and export to csv quickly without running out of memory?

From Dev

Related Related

Article