A day in the life of a petro-data manager - Part I - Shorthand

[Update: a simpler workflow that uses  for-fee & for-free software is posted here]

After intoducing the process to extract, transform & load  (ETL) www.boem.gov well data into a www.ppdm.org database, here is the short version expanded over on my sister blog.

Web Maps on Steroids, Part II

I showed a couple of months ago ways to post mega-datasets online without choking the system. This month OSGeoUK gave me the opportunity to present the same on PostGIS Day 2014, as postGIS was the backbone of two of these examples on GIScloud and GeoCloud2. Thanks to OSGeoUK and British Computer Society for hosting this.

A day in the life of a petro-data manager - intro

[Update: Talend made the early version complex, so simpler one was posted later]

Have you ever been given plain text geodata and wondered how to database and map it? And has this happened to you lately with tens of thousands of lines of data? Well help is at hand! Here is an ETL  workflow (extract, transform, load) useful to any data manager in or out of petroleum using free tools:

Big data and maps reloaded

Social media really work! +Vicky Gallardo posted on Google+ that the map is not the territory, with a wonderful big data map by Ben Allen of Denton TX (near where I used to live). That prompted a post by the same title on my companion map blog. The distinction between data and maps has long kept me up at night (see my blog tagline), and many others to judging by exhibits and media only in London (near where I live now).

Web maps on steroids

The last 6 blog posts over the last 3 months chronicled the use of dynamic maps using time attribute - years for historic ship tracks and wind data from same 150-350 yrs ago - not to animate maps but to filter them by decade and manage data fetches on ArcGIS Online. A parallel series of posts showed mega data sets on Amazon Web Services, as Mapcentia assured me postGIS handled giga datasets...

On joining and merging historic multi-lingual geodata

Earlier posts chronicled the history even the beauty of historic shipping and climate data from CLIWOC. British, Dutch, French and Spanish maritime agencies transferred paper logs to digital records. In doing so look-up tables allowed to convert multi-lingual records into quantifiable attributes. Something odd (to me) happened in the process of mapping these: over 1/4M records doubled to almost 1/2M when look-ups were joined and then wind and direction tables merged to create maps symbolised by wind force and orientation.

The happenstance art of maps

I showed recently how CLIWOC weather data from ships captains logs dating 1662 to 1885 totalled almost 1/2M points. It started with a 1/4M ships tracks, and combining look-up tables from four maritime agencies they yield numeric wind force and direction...

Dynamic maps final (for now)

[Update1: longhand version posted on my map catalog blog
Update 2: companion Slideshare delves into technical issues]

This is the end installment of progressing from static to dynamic maps online. A few lessons learned along the way on posting a quarter million point dataset, which ballooned to almost half a million after links & joins...

Lessons from 'static to dynamic maps'

Last month related the stumbling blocks in posting too much data on arcgis.com - time animation pushed the limits of stock web service even when limits are set above 250K points - and other services don't offer animation as yet.

From static to dynamic maps, continued

Last month I reported on posting on arcgis.com some time-stamped, time-slider or time-aware maps - I showed my WhereIsAndrew map - I also mentioned how time-sliders are a great way to roll-up diverse datasets that are time dependent. CLIWOC Captain's ships logs was the other example cited, and I proceeded to post it as a service from ArcMap on arcgis.com.

From static to dynamic maps, my travel so far

I tell people "I know just enough java to be dangerous", and it has served my well in my prior attempts logged in my old web page. These were all Google Maps API v.2 I built about 5 years ago. This blog as well as my new map catalog showed how I built maps in QGIS then ArcGIS, and then posted them on giscloud.com and AWS via Mapcentia GeoCloud2. I recently posted maps on arcgis.com on desktop and smartphone, static results of 'traveling salesman' geoprocessing on the desktop or online.

Handy maps

A 'handy' is what Germans call a mobile or cell phone. I uploaded free ArcGIS for Android on my  smartphone, which now has a decent screen to read maps on - love my Navigon Europe on it, and Google Maps too - but here is a quick&easy application of arcgis.com for the rest of us. 

... now HOW open is open?

[Update: Part II is on my Medium professional channel 2½ year later: Almost open data]

A lot of (virtual) ink has flowed around opening up data, as in this blog, GISuser crowdsourcing open data (below left) etc. etc. And everyone is getting into the act, from White House (below center) and Whitehall (UK Cabinet Office) to the number of open data hits (below right).

Releasing data really works, Part VI

Three months into posting data on my web service - Part V - I created a short list of free data for Great Britain on my Map Catalog called GB Freebie.

Global Sailings (1662 - 1856, English, Spanish, Dutch, French) revisited

[Update: climate data (wind speed & direction) have now been added in short & long posts]

I originally extracted CLIWOC (CLImatological database for the World's OCeans) ship captains' logs ships locations over a decade ago, to demonstrate the processing of 250K+ points in ArcGIS desktop using then new File Geodatabase. Five years later I posted this on my old website with instructions how to use it in  old ArcGIS Explorer and KML, and then I put a layer package on arcgis.com - both related historic details like de laPerouse's demise below, the importance of data standards and metadata, and the interst it generated elsewhere - more recently I posted a time-based variation of same, where using a time slider helps clarifiy complex data on desktop GIS.

Maps are forever (Part III)

My first post on this theme traced beautiful medieval maps from the British library in London to the Harvard Libray online, to land on a 1610 beauty.

Map catalog page 4

My map catalog also works well to post new projects: added British Geological Survey web mapping services to East Anglia web map, in order to compare historic and current geology. The Snapshot format also give a variety of viewing options of the catalog.

Map catalog continued

In my ongoing suite of posting webmaps in my new and fresh Mind the Map blog, here is another take on loading global vectore megadatasets but this time on Amazon Web Services direct via Mapcentia's GeoCloud.

Standards & Metadata, Part VIII

My previous post on this topic stated how careful documentation and appropriate metadata high-grades any information that is shared online by giving origin, context and other information. It helps build bridges and I quipped a well-known tear down this wall that also closed my second last post on free data and apps.

More map catalog

Last week was the inaugural post of my catalog, highlighting maps from this blog in (very) roughly reverse chronological order. The second entry make the display options more evident, as this is meant to be a light and flexible version of this more complete blog. I am not about to end its success quite yet, and have no fear I will post items of substance as well as cross-links to the catalog until it has its own followers.

Andrew Zolnai map catalog

Happy New Year! Don't you think that 175 blog posts and almost 150,000 page views over 7.5 years merit a catalog of my maps? This also coincides with my first end-to-end project:

1. collect on the desktop
2. disseminate and receive feedback via social media
3. post as a webmap on private AWS account