Web maps on steroids

The last 6 blog posts over the last 3 months chronicled the use of dynamic maps using time attribute - years for historic ship tracks and wind data from same 150-350 yrs ago - not to animate maps but to filter them by decade and manage data fetches on ArcGIS Online. A parallel series of posts showed mega data sets on Amazon Web Services, as Mapcentia assured me postGIS handled giga datasets...

click image to enlarge or go to map directly

Well this turned out to be quite true, as detailed on map catalog blog: almost half a million points  totalling almost a gigabyte of data post at a blistering speed... faster than the desktop in fact! This is  due to tile caching in Leaflet on the Mapcentia GeoCloud stack reading AmazonWeb Services data on postGIS.

Why post so may points, one may ask? Because CLIWOC chronicles over 180 weather and shipping attributes almost daily along sailings over 100 years (plus a sample 100 years older) from British, Dutch, French, and Spanish maritime agencies! This resulted in over 230,000 points that doubled to 470,000 when multilingual lookup tables were joined to make wind force and direction computable attributes. I queried this on my blog too, in case I lost my way somewhere... In fact my friend Hussein Nassr, who wrote two books on ArcGIS Server and Geodatabases, pointed out that:

It is not recommended to use relational model for such huge data. This is the era of Big data, in technical words, De-normalize, minimize the number of tables, eliminate unnecessary look ups and domains, it is ok to have duplicated data, we don't have a memory or space problem anymore.

So I did just that, and posted  a redux version of same that maintained the features but cleaned the attributes, and resulted in a 181Mb file geodatabase (33Mb compressed). These can be found on ArcGIS Online and ShareGeo Open as shapes.

And lest we forget, this is just the geo part - climatologists and historians who know this data are in the process of collecting historic weather data, to study climate change history in multidisciplinary and multinational efforts such as ACRE...