USGS Historical Maps Via Topoview

In the days before Google Earth (and similar) one of my favorite pastimes was to pour over printed maps. And my favorite maps were the large scale topographic maps produced by the the USGS, the United States Geological Survey. Back in the day we actually went to a store and bought the maps we needed and a collection of large-scale maps covering the Adirondacks was a non-trivial investment.

Going back to the mid-90s the USGS has also provided digital access to maps from its' vast library. That access has evolved considerably over the last 20 years culminating in the recently unveiled Internet application called TopoView. TopoView provides public access to (nearly) the entire library of historical topographic maps created by the agency over the past 135 years; over 170,000 map sheets from different editions (year and scale). If you want to dive right in the link goes to the TopoView home page where you'll find a short video that explains how to use the application. Or just go straight to the application and start poking around. The TopoView interface is well designed and well executed. Best of all, it's fun to use; at least I think so.

A screen capture of the TopoView interface. The box in the lower left provides details map sheets available for the selected grid cell. Clicking the map image in the box opens a high resolution image of the map. Links in the box allow maps to be downloaded in several different formats.  

TopoView is a remarkable application and, for the Adirondacks, it provides a unquie source of historical data. The Adirondack region has undergone two great transformations over the last 200 years. The first was the destruction of vast swaths of virgin forest by logging and fire starting around 1850 and lasting into the early 20th century. The second transformation has proceeded more slowly but is well underway. It is the regeneration and succession of the Adirondack forests following the  protections put in place starting in the 1890s. The maps available via TopoView provide a timeline and geographic snapshot of  that second transformation.

Let's look at some maps that illustrate what I'm talking about. The four maps that follow are:

• 1898 - USGS 1:62500-scale Quadrangle for Thirteenth Lake. The map is overlaid with two current hydrologic data sets (also available from the USGS). The red lines represent streams and rivers and the dark blue areas are represent water bodies (lake and ponds).
• 1954 - USGS 1:62500-scale Quadrangle for Thirteenth Lake, NY - with hydro overlays
• 1954 - USGS 1:62500-scale Quadrangle for Thirteenth Lake, NY - water boundary overlay is not shown to make it easier to see the representation on the topographic map
• 1997 - USGS 1:25000-scale Quadrangle for Bakers Mills, NY - a higher resolution recent map

Map 1: 1898 - USGS 1:62500-scale Quadrangle for Thirteenth Lake. The map is overlaid with current hydrological data. The red lines are streams and rivers. The dark blue areas represent water bodies (lake and ponds). The mis-alignment between this 120 year old topographic map and the modern GIS layer is clearly visible. But this map was made in the days before aerial surveys and it met the accuracy standards of the day. Quite remarkable actually, this is not easy country to move around in.

Map 2: 1954 - USGS 1:62500-scale Quadrangle for Thirteenth Lake, NY - with overlays. The modern boundary layer for the Mud Ponds aligns closely with the outlines shown on the topographic map.

Map 3: 1954 - USGS 1:62500-scale Quadrangle for Thirteenth Lake, NY - water boundary overlay not shown. Note the addition of the trail that reaches the ponds.

Map 4: 1997 - USGS 1:25000-scale Quadrangle for Bakers Mills, NY - a higher resolution recent map with water boundary and stream overlays. The current hydrological overlays align closely with the topographic map. 

I download the topographic maps using the TopoView application and loaded them into qGIS (Quantum GIS, open source GIS) so I could add the high resolution water layers that provide an up-to-date geographic context. Map one, from a USGS map printed in 1898, shows an approximation of the size, shape and location of the two ponds (Mud Ponds, in the Siamese Lakes Wilderness). But that's not to say that the map did not meet the accuracy standards of that time. Given the survey methods available then the map was very likely up to standards and accurate enough. If you wanted to visit these ponds that map could serve as your guide. While it is ultimately true for all data, geographic data is always an imperfect and approximate representation of the real world. It's easy to forget that when reading a well produced map from a trusted source. USGS topographic maps are produced to meet an accuracy standard that defines acceptable levels for accuracy and errors. You can learn more about this here.

Map two shows the same water layers overlaid on the USGS map from 1954 and, as you can see, the alignment between the USGS map and the modern water layers is significantly better. This map also shows a trail. That trail was not shown on earlier coverages of the area. Map three is the same configuration but with the display of water boundaries turned off. This makes it easier to see the ponds as they are represented on the topographic map.

Map four was made using the current base map from 1997. The newest map is higher resolution with a scale of 1:24,000. The earlier maps represented a scale of 1:62,500. The alignment of the water layers with the latest topographic map is very good. An interesting side point is that the trail visible in the 1954 map is no longer present on the map. See my post on a recent hike to these ponds to learn more about why this point was important to me.

The maps I downloaded from TopoView to create the above images were retrieved in the GeoTiff format. That format provides the highest quality option for maps retrieved from TopoView. The KML option is nearly as good and works best with Google Earth. The GeoPDF format is best if you want to print a copy of a map that you download. And the JPG option is good for viewing maps directly on your computer, especially in a web browser.

In upcoming posts I'll discuss additional opportunities and issues associated with this incredible source of historical geographic data.

Google Earth and KML - Part Two (GETECH)

In the previous post I introduced the topic of working with geographic data in the KML format. As mentioned there, KML is the native data format used by Google Earth (GE) and it is widely supported by both GPS devices and software. I won't attempt to cover all the gory details of working with and editing KML in this series of posts (resource links at the end), but there a few practical things I've learned about working with GPS data in the KML format that I want to share.

As a basis for this discussion I'm looking at two GPS tracks I recorded on a short Kayak trip on the Kayaderosseras Creek near Saratoga Springs. One was recorded using my trusty Garmin GPS and the other was recorded using the GPS receiver in my camera; a Canon Sx260. I'll discuss and compare the accuracy of the two tracks and touch on some other issues you might encounter while using GPS to collect geo-referenced data for use in your projects.

The first issue is evident in the screen shot below (Image 1). The red line represents the track captured by the GPS receiver in the camera. The purple line shows the route recorded by the Garmin. The obvious difference is that the purple line extends beyond the point where the red line stops. That's because the red line stops at the place where I put the kayak into the creek and later took it out (at the Spa State Park Canoe Launch Site on Driscoll Road). I paddled upstream for a mile or so and then floated back down. I had both GPS units turned on and both units recorded the path. The purple line continues because when I got back to the launch point I forgot to turn off the Garmin GPS. So it recorded a track for the short carry from the creek up to the parking area and for my drive home. That data is not useful so the first thing I want to do is remove it.

Image 1: Screen capture from Google Earth. The two lines (red and purple) are GPS tracks recorded simultaneously on Kayak trip on the Kayaderosseras Creek near Saratoga Springs (NY)

I could do that by editing the track directly in Google Earth. That would be easy but it would be extremely tedious. There are several mouse clicks required to delete a point and there are a lot of points to get rid of to clean up this problem. You can see this in image two. For this screen capture I turned off the display of the track recorded by the camera (the red line) and selected the purple track for editing This makes visible the points recorded by the GPS. Each recorded point is represented by a red dot in the image and as you can see, there are a lot of points in just this short section of the route from the creek to my house. To remove this part of the track by editing it in GE you'd click on each points, select Delete, confirm the deletion and repeat. Lots of repeats.

Image 2: Screen capture from Google Earth. With the display of the track recorded by the camera (the red line) off, the track represented by the purple line was selected for editing. This allows us to see the individual points recorded by the GPS receiver. 

A better alternative is to open the KML file containing the track in a text editor and remove the unwanted points from the file. Once you have the hang of it you can edit the data in the file very quickly. There are few steps involved in this process which goes something like this:

1. Export the track to a file. In Google Earth right click on the item in the Places pane, select Save Place As. Be sure to select KML from the format drop down so the file is saved in the KML format.
2. Open the file in a text editor or an XML editor (I use Notepad++; a free and open source editor that is popular with programmers). You may need to configure your editor to recognize that the KML file is an XML file. That should allow the editor display the contents with nice formatting.
3. In this case, removing the points I don't want comes down to removing all the points where the latitude is greater than the latitude of the put-in point. I'm in the northern hemisphere --so the latitude coordinates increase as you go north-- and all the points I want to keep are south of the put-in. Image three shows the map zoomed in on that location with the properties dialog for the point displayed. The properties dialog shows the latitude and longitude of the marker (and the location). This is how I got the latitude I will use as the basis for my edits; any point with a latitude greater than 43.034293 can be removed from the file. A potential complication is that that the coordinates in the KML file are stored using the decimal degree format. That's as you see it in the dialog seen in image three, but that representation is not the default for coordinates in Google Earth. The default is to represent coordinates in the Degree/Minute/Second format; like this: 43°02'03.4548" You can change the representation used by Google Earth using the Options dialog on the Tools menu.

Image 3: Screen capture from Google Earth. Viewing the properties of a marker added at the location of the start/end point of the trip. The properties dialog shows the coordinates of that point.

Once you have a KML file containing your track, and you have a way to identify the point(s) to remove (in this case, latitude > 43.034293) you can remove the unwanted points from the file. To do this you do need to know a little bit about the XML data format and how to edit data stored in XML, but there are many introductions to XML available on-line and I've included references at the end of this post. Also, if you've done any work with HTML this will all look familiar. XML data elements are placed inside of tag pairs and you must delete the entire tag pair or you wil break the formatting of the file (and GE will complain when you try to reload it). You'll also need to recognize how the coordinates are represented. An example from my file looks like this:

<gx:coord>-73.79277999999999 43.034036 102.52</gx:coord>

The coordinate of the location is represented by three values with a space placed between each value (it might be hard to see the space; -73.79277999999999<space>43.034036<space>102.52). The values are:

-73.79277999999999	longitude	a "west" longitude represented as a negative number
43.034336	latitude	a "north" latitude represented as a positive number
102.52	elevation	meters above sea level

As discussed in the previous section, I want to delete any point where the latitude is greater than 43.034293, so this point can be removed from the file. The points are stored in the file in order, so once you find the place break between what you want and what you want to remove you can select all the bad rows and remove them. If editing data in this way is familiar to you then this should get you started. If this is entirely new then you may need more help. Look at the references or ask a friend who knows about this stuff.

With unwanted data removed from your file there is one final step; opening the file in Google Earth so you can see the results of your work. In Google Earth select "Open" from the File menu and find your edited file on disk. If you get warnings about errors check the file over for mis-matched tag pairs. That's it, you can save the file in your Places folder so that Google Earth reopens it automatically or you can reopen the file when you need it in the future.

This post is already rather long so I'm going to cover the other topics I raised at the start in a separate "part two" posting.

Using Google Earth for Geographic Exploration (GETECH)

Readers of this blog know that I use Google Earth (GE) to plan outings and to manage data that I collect using various GPS devices. I use Google Earth as a sort of entry-level Geographic Information System (GIS) and in the following series of posts I plan to share what I've learned about using Google Earth as part of a citizen/community science data management system. I'll probably move these posts out to a separate blog in the future but to get it started I'm putting them here. If you aren't interested in these more technically oriented posts you can skip them. I''ll stick (GETECH) on the title to make them easy to recognize.

On most of my outings I collect geo-referenced data using one or more GPS devices. I have a Garmin hand-held GPS, GPS in my phone and even a camera with GPS built-in. I collect "tracks", the GPS receiver stores a point every few seconds allowing you to trace your route, and I also save WayPoints. These are specific locations of interest and the GPS saves the coordinates along with a name you enter. The Garmin GPS can grab a bunch of coordinate pairs in quick succession and average them to improve the accuracy of the fix. This is useful if you want to get the most accurate location that your device can record, usually around ten feet. I'll do a future post on GPS accuracy to explain this limitation.

After a trip I upload the data from my GPS devices into GE. This allows me to see the tracks (my route) and waypoints in the geographic context provided by GE. If the data was collected for a specific use I move it into whatever storage system I'm using for that project (more on this in a future post). My Garmin GPS is directly supported by GE so I simply plug the GPS unit into a USB port and use the GPS option on the Tools Menu. There are options to allow you select what you want to import. By default GPS data is added into a sub-folder of the "Temporary Places" folder in your GE Places. I typically tweak things a bit and move the imported data into the "MyPlaces" folder. This allows me to save each GPS data set for ready access in GE. You could also right click on your newly uploaded dataset and save it to your computer using the Save Place As option.

It's even easier to grab the GPS data using the phone. I have several GPS related apps on my Android phone (a Google Nexus 5) but I most often use GPS Essentials. As with the dedicated Garmin, GPS Essentials allows me to record tracks and save waypoints (it has other capabilities that I'll discuss in future posts). To get the data into GE I use the Export feature of GPS Essentials and save the data directly to Google Drive (Internet-based storage). It takes a few minutes but once the data shows up in Google Drive I save it to my local computer as a KML file. KML is the native file format of Google Earth and most software that uses geographically referenced data can use data in the KML format.

This has been my basic working process for several years but I've recently become concerned that it was breaking down. The data I collect is best categorized as "ecological inventory". I note where species are found, when I saw them and I record additional information about the circumstances. The "where" and "when" are essential to the data having value and I was not managing this information in a consistent way. It's there --embedded in the KML files-- but I didn't have an easy way to find everything related to a location; or everything for a specific time span. Some of data was stored in raw files and some was in databases that I've created. To maximize the value the data needs to be consolidated and I need a more robust process for recording and managing the metadata. Also, with over 200 top-level sub-folders in my MyPlaces folder, Google Earth was running noticeably slower.

Screen capture of Google Earth after I cleaned up and organized my MyPlaces data. The lines are tracks captured using GPS and the markers are points of interest.

So I looked into how GE actually manages this data and found that all of the data you see in GE in the "MyPlaces" folder is actually stored as a single file on your computer named myplaces.kml. The location of this file varies based on the operating system of your computer and the version of GE you are using. On my computer, running Windows 7 and using Google Earth Pro, the file is saved to:
C:\Users\<your-windows-name>\AppData\LocalLow\Google\GoogleEarth

Seeing the MyPlaces.kml file made it clear why GE was straining a bit. The MyPlaces.kml file on my computer was just under 50 megabytes in size and using a text editor to open it I found that it contained over 1.1 million lines. For context, if we assume that a page of text averages around 60 lines, the MyPlaces file on my computer contained over 18,000 pages. That's a bit much.

I had let the situation get out of hand and now it was going to require a lot of work to straighten it out. One approach would be use the GE user interface. In GE you can save each folder or item in the MyPlaces folder to a separate file on your computer.  You can then delete that item from MyPlaces and open the file when you want to use that data. If the item you save is a folder GE creates a single archive file with the (.KMZ extension) containing the entire contents of the folder. This approach would work but it was going to be a tedious process to say the least.

A second alternative is to edit a copy of the MyPlaces.kml file using a text editor (note that I said "edit a copy" - don't edit the file used by GE). You can use any text editor to open a KML file. Unfortunately, given the size of the file on my computer this also was going to be a slow and error prone process. Using NotePad++ --an editor design for working with lots of text-- opening and navigating the file was extremely sluggish (recall the 1.1 millions lines). I could have copied out sections and worked on pieces of the file but the data is structured (it's an XML file if you are familiar with that sort of thing) and there are references among different sections in the file. It would be very easy to mess this up.

The solution I chose was to write a script to run through the file (a copy of course) and save each folder, document and placemark to a new, separate, KML file. I could then remove (making a copy) the MyPlaces.kml file used by GE (make sure that GE is not running if you do this). The next time GE is run it creates a new and empty MyPlaces.kml file and you are back to a "clean" install of GE. The separate KML files created by the script can then be opened using GE when I need them.

I'm making this script available for anyone who wants to run it and you can access here:
https://gist.github.com/kentstanton/3441cc368d3c52621b19

Please note that you must have PowerShell 5.0 installed on your computer to run the script and you need to know how to run PowerShell scripts. If you are familiar with PowerShell programming you can alter the script as needed. And it would not be too difficult to port the code to a different language if you do not have a Windows computer to run it on. The script requires Windows PowerShell 5.0; the very latest version (as of this writing). I'm working on a larger project that will incorporate this functionality and remove this requirement but I wanted to go ahead and make this available now because it might be useful to some people as is.

Moreau Lake State Park Boundary (KML) for Google Earth

This is a follow up to my recent post on the addition of lands formerly held by the McGregor Correctional facility to Moreau Lake State Park. That post included a map image with a rough outline of the newly added parcel. I've refined that boundary and saved it to a KMZ format file that can be opened in Google Earth/Maps.

Link to the Moreau Lake State Park Boundary File
https://drive.google.com/file/d/0B3QHBL0PwDqWZFpxSG9nYU1BY0E/view?usp=sharing

Clicking the link will open a new browser window (or tab) showing Google Maps with the boundary layer displayed as an overlay. You can also download the boundary from Google Maps by clicking the Download Button (down arrow icon) on the Google Maps toolbar. The file is in the KML/KMZ format native to Google Earth/Maps. If you have Google Earth installed on your computer you can double click the downloaded file it to open it.

Image 1: Moreau Lake State Park Boundary.
Note: These boundaries are approximate and not official in any way. (Base map source: Google Earth)

The red outline represents the boundary of the main section of Moreau Lake State Park. The newly added "Lake Bonita" section is shown in blue (lower left). The area in brown (near the top) is the section along the Hudson River and north of Spier Falls Road, The section with the green outline is the area east of the Hudson (previous post on this here).

Image 2: The same boundary outlines viewed as an overlay in Google Maps. 

Viewing the boundary layer in Google Maps you might wonder why the outlines I've provided don't match the park boundary shown tinted light green. Most striking is that Maps only shows the main section of the Park. Probably because the data that Google used as the basis for the parks layer was created before the section east of the river was added. That question of "when was the data created" also applies to the new Lake Bonita section. The boundary outline I've provided was digitized from images provided with the press release announcing the land transfer (so again, it is only approximate).

Perhaps the most interesting discrepancy is the area outside of the red outline seen just below and left of Moreau Lake. That's not newly added so, is it in the park or outside of the park? It's probably a data error in Google Maps. No other source that I've seen includes that acreage in the Park and if you zoom in, as seen in Image 3, we see that there are streets located there. I know from having been there that there are houses on those streets so that area is probably not inside the State Park.

Image 3: The section in green, but outside of the red outline, is probably a data error. (Source Google Maps)

The outlines I've shared were digitized from base map layers obtained from several sources. My starting point was the official Park map and I edited and added detail using a base map layer from ArcGIS Online (Esri). My point in discussing these discrepancies is that geographic data usually represents a moment in time. If our subject of interest is represented by data that changes continuously then we can consider that data to be out-of-date the second we save it to a database. Even something far less dynamic --a State Park boundary for example-- will change over time and representations provided by varied sources will not necessarily be in agreement. And then there are errors of all kinds and the question of accuracy. But that's another whole story.

As a representation of the real world geographic data is uncertain. And uncertain is not the opposite of certain. It's a lot more complicated than that.