Releasing data really works, Part V

It took five days (after hours) to stand up, learn, tweak and display my East Anglia Fenlands project on Mapcentia's web service. It started with a GISuser group post on LinkedIn on Monday, I used my Amazon Web Service free EC2 trial and GeoCloud2 under beta, and by Friday I had it working and styled. No small thanks to Martin Hogh's original work and help, the result is a simple yet modern and pleasing web map.

Web maps for the rest of us

 The previous post on Community Engagement updated the rebranding of cambridgeshire.ai, with accompanying use of "AI for the rest of us" (work backward from here) this blogpost title came from. One of the mentioned changes were working with Wikimedia, OpenStreetMap and Climate Central - my Esri Developper or Non-Profit stacks are free and frozen, respectively - I had a whole lot of work put on ice, the same time I relinquished my original website www.zolnai.ca. This blog is OK however.

To Geo or not to Geo, that is the question...

... at least in oil&gas. I always wondered why petroleum was only 5% Esri's market (unofficial from my tenure as petroleum manager there, they publish no figures as a private company). A current rationalization project at an oil major hinted why - I've 'pushed Geo' for 25 yrs. so I saw that in my previous tenure at Halliburton, but that only crystallized later - whilst a large percentage of data has a spatial component in oil&gas, only a small part of it is stored in spatial databases. GIS are generally for surface infrastructure like geology, plants and pipelines, rather than for subsurface exploration and production. Surface data can actually be seen and measured directly on or near the ground, whereas subsurface are interpolated data from drilling and seismic deep in the subsurface. Indeed the challenges in oil exploration in the news of late revolve around this frontier.

DaaS BooT (Data as an Asset, Part II)

[Note: this is a re-post from LinkedIn Pulse that requires a user login]

After Das Boot - German U-boat disruptive technology of WWII - it stands here for Data-as-a-Service, Business Opportunity of Things - rather than Internet of Things (IoT); it underscores that without business outcome, we might as well stay home: that is the tenor of GEO Business 2015, in its third year, as it draws together Surveying, BIM, GIS and business.

Releasing data really works, Part VII

And now for something completely different - the original posts until Part VI are listed below - I ran across a nice map of Steve Feldman's: He also tried out free data and software to map UK flood maps, an up-scaled version not for professional re-use.

A brief web history of mine

As I wind down my blog posts and I quit socials, perhaps this may be a good time to reflect on my Web journey that started almost 40 years ago (a recap of my IT journey is here)...

But wait... Tim Berners-Lee didn't invent it til just over 35 years ago (Wikipedia)! That's because I had a site dubbed HTTP in 1986, see inset below:  "world wide web" hadn't been coined yet, but I knew as a student at late 70's University of Calgary of Ted Nelson's failed Project Xanadu (Wikipedia), to try and link all forms of knowledge via hypertext; it failed partly due to Nelson's eccentricity,  partly because there was no global network to carry it. 

A brief history of mine

Update 2: see my journey on the web here.

Update 1: a duo of posts on my Medium professional channel here relates my early computing.

As I go through a 'hard reset' in my life and plan to exit social media, this may be a good time to pause and reflect on my IT journey.

Calgary 1986 see here

Although I started on mainframes at Univ. of Calgary 1979 (here), I transitioned to desktop at Queen's Univ. (here). That post's intro lists articles illustrating the progression from geology to computer mapping and GIS. The reason I was able to do that, is that my eight years of high-school Latin allowed me to read, fix and re-run Unix scripts running petrodata management software in the oil industry, where I spent the bulk of my career.  

That then prepared me for HTML coding in the early pre-Internet days... This was 1986, 3 yrs. before Tim Berners-Lee ushered in the World Wide Web! The City of Calgary in W CAN was optically wired by the then government telephone co. AGT (now Telus) for free, betting on the fact they'd recoup the money - and they did so handsomely - via the traffic of all the oil companies jammed downtown. This was against a backdrop of the Province of Alberta having forward-looking Land Information System based in the capital Edmonton a couple hours drive north (here), but that was the heyday of early data transmission protocols (rpc)  and open binary data (EBCDIC) that opened the data flow cross province on said optical network.

Anecdote 1: as an undergrad at University of Calgary, I learned of Ted Nelson's Xanadu project (here), to catalog the world's information via what later became Hypertext Markup Language (HTML) - few people realise Tim Berners-Lee didn't invent anything, he built on exisiting systems... as we all do - but Nelson was a propeller-head who couldn't  market his idea (here). John Walker, co-founder of Autodesk, reportedly bought the technolgy but it never got anywhere... I heard of this as I was cooling my heels awaiting to meet him, with Carol Bartz his then PA - the same who eventually lead Autodesk then Yahoo, a true pioneering woman in IT - when a surveyor friend and I tried to sell them the idea of a mapping front-end to AutoCAD ... in 1986! Remember MS Windows was launched just the year before (see Side Story here and illustration here incl. photo above). Did you know I received a copy of  Windows v.1.O as part of a hardware upgrade to boost my computer's memory from 640Kb to 1Mb?! Sidebar: Was Bill Gates not famously quoted that no personal computer would ever need over 640Kb RAM  (here)?

CD-ROMs

Note that was the early to mid eighties and the web had yet ot be invented! In fact I helped launch a CD-ROM publishing firm, that put all of the Alberta well data on one CD and petroleum production info on another... using a Texas school board hashing algorithm. CDs were so rare in fact, that we leased the CD readers together with the discs themselves. Aside from the hashing algorithm, CD-ROM publishing conferences at the USGS with Gerry McFaul encouraged us to publish CD data. Another effect of pioneering CDs was that we shipped data to Dublin OH to have the CDs pressed... and US customs did not know what Exabyte data tapes were, so we shipped data as videos!

Anecdote 2: Microsoft was largely credited with popularizing CD for software distribution. While that is true, the real initiative was the US Dept. of Defense - the same people whose DARPA developed the precursor to the internet linking US military to government contractors, chiefly universities in the beginning - the reason? In the Cold War era, it was feared the Soviets might detonate an atom bomb at 5-10 mi. altitude that would fry the entire electronic infrastructure of mainland US, thus disabling the military - oh! and the rest of the country is collateral - I learned via the USGS mentioned above, that CDs were immune to resulting electromagnetic pulses.

Anecdote 3: as I had been on contract to the Geological Survey of Canada Calgary office across from the university, we visited the geological library and offered to scan the index cards to electronically catalog them on CD-ROM - after all our software came from a school library application - do you know what the librarian said then? "If we made the catalog that easily accessible, can you imagine the amount of work that might create?" We obviously had work to do, convincing a traditional business of the advantages of electronic access, taken for granted now!

 

Pre-WWW

Notwithstanding work on CD-ROMs, we benefited form the optical network to start building HTTP networks as we called them, as internet never mind intranet or the WWW hadn't been invented yet, not outside DARPA. My blog posts the first screenshot of an early 'website' (term as yet n/a) under 1986 at top left of the banner menu:

Progressively larger firms in the petroleum service sector took me to Texas, where we did lots of things on the nascent internet. That's when I built my first website, playing with HTML formats on my life and work stories by year, patterned after a glossary I co-authored with another counsellor. You'll recognise the website structure inherited or inspired by my first site a decade earlier:

Standards & Metadata

Parallel to this, I got involved in standards and metadata via CNC/CODATA to try and rationalize semantics of geological terms, which are quite complex. At the time the mining geological community and the USGS made some inroads, and the North American Geological Data Model emerged from that. But that was a long and arduous process that didn't come to fruition until after I switched to GIS in California in 2000. That is in turn written up in the manuscript splitting data into concepts (data taxonomy that evolve over time) and occurrence (geographic location that are fixed). In fact I took one of the geological map sheets and formatted it accordingly as an example: 

There followed a series of standards&metadata papers given at Esri and followed up on my blog and Medium channel (here, here, here, here, here, here and here) as I moved from US to UK. That put me in touch with AGI and eventually the Geospatial Commission, who have big plans supporting digital twins to enhance and enable UK infrastructure.

Volunteered Geographic Information

After I left "da awl bidness" (the oil business Texas-style) in the economic downturn, I became full time Volunteered Geographic Information as this blog is titled. I started a Community Interest Company to address East Anglia rural isolation affected by climate change and the pandemic via local community engagement. I also bird-dogged COVID polling NHS and ONS data. This is on the back of using Ordnance Survey parish data released in public domain with HC Derby Medieval Fendlands and Drainage of the Fens data to build up  a series of agro-history maps here... since Domesday in 1067! Sharing that was followed by a series on interacting with data agencies and improving their data ending here. My highlight was running a YouTube livestream on dsiplaying & tracking COVID data here.

Building from that I drew a series of sea level rise maps and training materials branching into environmental issues for East Anglia re: climate change ending here. I also engaged in the new medium of story maps key along the top level menu, a professional portfolio through the Community Interest Company as well as some of personal interest. In order to help others build the same, I shared here a series of "build your own" resources. This topographic map has sea level rise calculated  from NOAA:

I also posted atop the blog my professional Medium channel on various geodata topics here. Feel free also to peruse my YouTube channel and other resources along the top level menu bar of this blog: It is only visible on the desktop version, so on mobile make sure you scroll to the bottom and select "View web version".

A day in the life of a petro-data manager - Part I - Shorthand

[Update: a simpler workflow that uses  for-fee & for-free software is posted here]

After intoducing the process to extract, transform & load  (ETL) www.boem.gov well data into a www.ppdm.org database, here is the short version expanded over on my sister blog.

"With a little help from my friends"

[ Update:  Part II re-uses this and improves the manual for a London area action ]

Part of our mandate at cottenham.info is to raise awareness around climate change issues in East Anglia. A key part is to quantify risks around flooding from land during increasingly variable weather, as well as to predict what sea level rise would look like over time from melting polar ice caps. That combines respectively excellent ground work by DEFRA - see their Future Fens twitter feed - and modelling against topography by Ordnance Survey and DEFRA. And timing of sea level rises is an emotional issue: to balance the reality of the risk with questions around time scales (see comment), will help raise awareness without unduly raising alarm.

Historic Fenlands Mashup

[See updates at bottom, and predecessors Medieval Fenlands GIS and Post-medieval Fenlands GIS]

Here is a mashup on giscloud.com of the geographic history of land cover and surface geology of East Anglia since Domesday based on:

Historic climate data revisited - 1 - circum-Antarctic

[Update 2: this series has full CC BY-SA 3.0 class notes here
Update 1: please see a mirror project for the Arctic in Part 2.]

With ongoing debates whether Antarctic ice in increasing or not, and its effect on climate change, we must avail ourselves of as much data as we can. If historic climate data is at hand, not only do they get scarcer going farther back, but 1880 also marks a time prior to which their reliability falls off.

So having mapped climate data off tall ships captains logs from 1750 to 1850, I wondered how far south they sailed, and how much they augmented historic climate data around the Antarctic?

Welcome to new friends

[Update: read here my  new occupation following this]

A fond farewell to two old friends explained my transition to open source platforms. As announced in LinkedIn You can get Andrew out of the geo... (... but you can't get the geo out of Andrew) "Terry Jackson pulled me back in to the publishing business as a data wrangler". What does that mean?

Baaack in the arcgis.com

[rhymes with: Baaack in the USSR] After a brief hiatus trying my hand outside ESRI, I'm baaack... posting maps on arcgis.com. Daniel Schobler from ESRI(DE) Schools Program kindly reposted and time-enabled my Global sailings, captains ships logs 1750 - 1850 into his Explorations and voyages 1662-1855 (time-enabled)

"East is east and west is west...", or is it?

Geo-meta-data news flashes:

quickly access web resources regardless of resource location via ESRI's geoportal extension

free metadata tools for the EU INPSIRE website using ESRI Irelands Be-Inspired site

quickly add data anywhere in the world, crowdsourcing debut on Google Map Maker

geocode data into the recently increased Google palette in the US at least announced

"Horses for courses", part II

Update 2: see two much later updates here (this spot) and here (Northwest Passage)

Update 1: a fascinating firsthand report from the discoverers posted on Russell Potter's excellent blog.

A previous post here contrasted full professional workflows for petroleum geology students, with very simple analytical tools for a businessman looking to ascertain population density. As my subsequent posts show, I have an interest in Antarctic and Arctic maps, history and climate as for example in this Story Map.

Show me the money

That was my response to Peter Batty's call for comment on his GITA panel this week:

Esri, Google and if the shoe fits...

[Update: a Google business partner's view of things, 3 mo. after the fracas]

[Here is a further update based on input from other people on what is surely a timely topic. Originally posted on LinkedIn Pulse as The (Geo) Internet of Things, it's posted there for a wider audience.]

London Thames Barrier revisited

Update 2: see renewed Sea-level Rise extents according to new information here 

Update 1: see addition at bottom... thanks to our indefatigable London climate activists!

Further to my original blog post 3½ yrs. ago here, I was asked to share maps of the area surrounding the Thames Barrier (Wikipedia): A WhatsApp group considered the necessity for a second barrier under Climate Change that increases both flooding and sea level seasonal elevations.