tag:blogger.com,1999:blog-249908662024-03-12T22:07:26.397-02:30Critical TechnologyCurrently focused on the importance of a data lab for the digitization of oceans and in building year-round greenhouses for Newfoundland.Peterhttp://www.blogger.com/profile/01204894304194832459noreply@blogger.comBlogger468125tag:blogger.com,1999:blog-24990866.post-69955541660018343062021-04-09T12:38:00.000-02:302021-04-09T12:38:06.106-02:30The Leanstack Way<p>The <b>Oceans of Data Lab</b> is honored to be a part of <a href="https://www.propelict.com/apply">PropelICT's startup accelerator</a>. We had our kickoff meeting a few days ago and the current focus is on learning the <a href="https://leanstack.com/">Leanstack</a> methodology and using <a href="https://leanstack.com/leancanvas?">lean canvas</a> to tease out ALL the important details for success. The inline supporting learning modules that are available through <a href="https://leanstack.com/">leanstack</a> are very helpful. </p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAGXNIM6UJCoqMJOQO1AS9mkYJzgKPGHSgVz-coctihgZvn62nMSNAw597um08GBhOFYm-Ptduk4Uf9nUFdI7kYszHvo3NZ0myNnZr_DuL_aRUDY0xSnqIq5-qRcB77EF5NlnoRA/s800/LeanCanvas.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="556" data-original-width="800" height="278" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAGXNIM6UJCoqMJOQO1AS9mkYJzgKPGHSgVz-coctihgZvn62nMSNAw597um08GBhOFYm-Ptduk4Uf9nUFdI7kYszHvo3NZ0myNnZr_DuL_aRUDY0xSnqIq5-qRcB77EF5NlnoRA/w400-h278/LeanCanvas.png" width="400" /></a><br /><b>The Lean Canvas</b> - numbers indicate the order of completion</div><p>I feel fortunate that I have been familiar with Agile and Lean approaches for over 20 years. I've got two favorite sayings I use when running software teams, and I like to think I run many aspects of my life with an Agile / Lean mindset.</p><p></p><ol style="text-align: left;"><li><b>Ship and ship often</b> (deliver new releases as often and frequently as possible)</li><li><b>Fail and fail often</b> (take risks, innovate, don't apologize, keep moving), success comes from failure.</li></ol><p></p><p>For me the use of Lean in startups all began with Eric Ries when I watched a YouTube interview of Eric conducted a decade ago. This interview became a part of <a href="https://criticaltechnology.blogspot.com/2011/03/director-of-information-technology.html">a 2011 blog post where I describe lean approaches within the Director of Technology</a> role. Since this time I have revisited the works of Eric Ries every few years, he has a lot of useful insights to lean startups. One of my all time favorites in the talk he gave at Google 10 years back.</p><p></p><div class="separator" style="clear: both; text-align: center;"><iframe allowfullscreen="" class="BLOG_video_class" height="266" src="https://www.youtube.com/embed/fEvKo90qBns" width="320" youtube-src-id="fEvKo90qBns"></iframe></div><br /><div style="text-align: center;">Google Talk: Eric Ries and the Lean Startup</div><p></p><p><br /></p>Peterhttp://www.blogger.com/profile/01204894304194832459noreply@blogger.comtag:blogger.com,1999:blog-24990866.post-1866693595223259752021-04-08T09:27:00.005-02:302021-04-08T10:37:27.056-02:30ODL Newsletter - March 2021<p>The Oceans of Data Lab (ODL) monthly newsletter is also finding its footing. It is still going to include monthly updates to the progress we make AND it will start with a few articles of interest within the data labs technology world. I am discovering so many interesting technologies and approaches within the data realm. I'm going to fold my 30 years of data experience into why I believe these are of interest to those working with large amounts of data.</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEio6cFd_PhRRQpPT-1Cn7GQMTtsXJkHvR6rHqlxwTI0epvjUubtUAyD9ZMTbbww4GH6ch-gcIaDeSAfz-x4qdcc78axp6z_GyxuiHT9ifzsiQTafd162bVCmZ2Cgqni-YJbkK6aXA/s272/apachedatalab.jpg" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="186" data-original-width="272" height="137" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEio6cFd_PhRRQpPT-1Cn7GQMTtsXJkHvR6rHqlxwTI0epvjUubtUAyD9ZMTbbww4GH6ch-gcIaDeSAfz-x4qdcc78axp6z_GyxuiHT9ifzsiQTafd162bVCmZ2Cgqni-YJbkK6aXA/w200-h137/apachedatalab.jpg" width="200" /></a></div><b><span style="font-size: medium;"><a href="https://datalab.apache.org/">Apache Data Lab</a></span></b><p></p><p>The Apache data lab that comes from the same organization that <a href="https://projects.apache.org/projects.html">has brought us so many of the important technologies</a> over the years. And specifically, to think of all the big data technologies they have delivered in recent years... there are just to many to list. What I like most about the data lab is its ability to be deployed to the big three cloud hosting environments. Super smart given the storage and compute requirements for data projects shouldn't be the responsibility of Apache.</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg2VsP8PqCyRwBt6xpXvgwvJcKNn0CvBxfck3GFe0AzpnnCSb3EDvIEJZF4HI_g-q8ubLZmOiDxXlWYIRjG34gEeryLF_BC5yQm6qBv8gJjp4aHOosM4UY6bawAPbOhqmDj9oNXlw/s368/DataKitchenLogo.jpg" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="368" data-original-width="227" height="100" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg2VsP8PqCyRwBt6xpXvgwvJcKNn0CvBxfck3GFe0AzpnnCSb3EDvIEJZF4HI_g-q8ubLZmOiDxXlWYIRjG34gEeryLF_BC5yQm6qBv8gJjp4aHOosM4UY6bawAPbOhqmDj9oNXlw/w123-h200/DataKitchenLogo.jpg" width="60" /></a></div><b><span style="font-size: medium;"><a href="https://en.wikipedia.org/wiki/DataOps">DataOps</a> and the <a href="https://datakitchen.io/">DataKitchen</a></span></b><p></p><p>DataOps is a fairly recent concept / term that is about seven years old... and it makes sense that it becomes a discipline in itself as it is not <a href="http://criticaltechnology.blogspot.com/2021/03/an-important-difference-between-devops.html">DevOps for Data, it is so much more</a>. The DataKitchen looks to be doing some amazing work in this capacity and <a href="https://datakitchen.io/the-dataops-cookbook/">have published a good read</a> to help get your head into this important and emerging technology space.</p><p>I'm another 3 people into working towards my 100 conversations. It is said that you need to have 100 conversations as you solidify your business / startup idea. So I managed to get another three conversations in. I know this isn't that many, but that's ok as this month was more about setting up technology and thinking about risk, revenue, and the escalator pitch for the startup. I still need to talk with people, and I need peoples help, always. If you know anyone who works with analyzing data or works for a business that has a growing interest in their data, I'd love to talk with them.</p><p><span style="font-size: medium;"><b>What has changed this month?</b></span></p><p>Over this month my thinking has broadened and become more focused on the needs of organizations and their data. No real pivot, but clarifying what the business will be. The changes fell into three main themes;</p><p></p><ul style="text-align: left;"><li><b>A broader interest in helping people with their data.</b> The backstory to my career has always been the information technology around the data. For 30 years I have focused on managing, moving, and building software for the data. This will continue with the data lab. We are still interested in ocean data and a reference architecture for the digitization of oceans, these subjects will become part of the bigger data lab.</li><li><b>It's a Data Lab</b>. It became very clear this month that what I was wanting to do is stand up and run a data lab. I had a great conversation with Graham Truax at <a href="https://innovationisland.ca/">Innovation Island</a> and this identified the alignment with my accelerator pitch and the data lab concept. After I re-read my proposal (and subsequent acceptance) to the PropelICT accelerator I confirmed... the startup is focused on creating a data lab with related products and services.</li><li><b>Start with a services focus, rather than product.</b> We need revenue and the data lab is not a small product with a near MVP that can generate revenue. There are a number of MVP's that could bring business value for our customers, but nothing with significant revenue possibility. So our focus needs to be on services where we can leverage the skills and knowledge of the founder and identify projects that align well with the overall vision for <u><b>Oceans of Data Lab</b></u>.</li></ul><p></p><p><b>It's been a business and technology focused month</b></p><p>This really was a more technology focused month. It was getting all the infrastructure in place to have the lab, fetch some data, and display a basic analytics dashboard. So while setting things up, we weren't that focused on reaching out to potential customers.</p><p></p><ul style="text-align: left;"><li>Setting up servers</li><li><a href="http://criticaltechnology.blogspot.com/2021/03/its-alive-elastic-stack-as-our-data-lab.html">Installing, configuring, and securing analytics software</a> (ELK stack)</li><li>Identifying and registering domain names, setting up websites.</li></ul><p></p><p><b>What are the risks and assumptions?</b></p><p>We also thought about what are the business risks and what assumptions are we making that could work against our success. We are not going to get into these in detail, writing them down and publishing them helps attract attention and hopefully getting the feedback we need to reduce the risk and prove or disprove the assumptions. We are also focused on what can be a product rather than what is a service.</p><p><b>Assumptions</b></p><p></p><ol style="text-align: left;"><li>Companies / Organizations will participate in a publish - subscribe business model for data sets</li><li>The data lab concept for preparing data sets for publishing will become accepted by SMB </li></ol><p></p><p><b>Risks</b></p><p></p><ol style="text-align: left;"><li>MVP doesn't generate enough revenue or provide business value</li><li>Primary founder having knowledge, energy, or bandwidth to keep up the pace</li><li>Finding skilled employees with deep understanding of data engineering</li><li>High cost of cloud based infrastructure </li></ol><p></p><p><b>Where is the revenue?</b></p><p></p><ul style="text-align: left;"><li>The transactional costs in the publish and subscribe (every data set transaction earns money)</li><ul><li>this is definitely my riskiest assumption</li></ul><li>SMB pay for services in preparing the data sets for themselves and the marketplace.</li><ul><li>does the rise of the data engineer role show a willingness to pay for data preparation</li></ul></ul><p></p><p><b>What do we consider our Escalator Pitch?</b></p><p>These are early times and we don't yet have a story to tell. Gak! The escalator pitch is hard, and we really don't know what we are doing when it comes to an escalator pitch.</p><p></p><ul style="text-align: left;"><li>We help SMB realize new revenue possibilities from their existing data.</li><li>We reduce the cost of data preparation for their internal analysis and business intelligence.</li><li>We provide the services and technology to help you make sense of all your data. </li><li>We make it easy for you to see the value and opportunities based upon your unique business data.</li></ul><p></p><p><b><span style="font-size: medium;">Next Steps:</span></b></p><p></p><ul style="text-align: left;"><li>We need to focus on the customer. We need to find the customers and talk to them.</li><li>We need to reduce our risk and prove, or disprove, our assumptions.</li><li>We need a technical platform to host an Minimum Viable Product (MVP). I need to identify and prioritize a few MVP's.</li></ul><p></p><p><b>If you find the Data Lab an interesting idea or have the need to bring greater value to your existing data, please feel free to contact me. We are building a business and we want to help you bring greater value from your data.</b></p>Peterhttp://www.blogger.com/profile/01204894304194832459noreply@blogger.comtag:blogger.com,1999:blog-24990866.post-65179251501451118712021-03-23T20:40:00.003-02:302021-03-23T21:08:27.126-02:30It's Alive! The Elastic Stack as our Data Lab<p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhkTZk3R5MO5alJRAfPZRgOYXdteEW4Io4j3MjuJRMpV_eEOjdOzVWU20onXOqVn4ZBcyGqE15xdOb3HpxXBUUgoIzbFshyphenhyphen8XCIaFsnoqgOHIURlO0ncq9ZhIcG2yoCtiWL1ygZgA/s471/Welcome.jpg" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="471" data-original-width="408" height="265" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhkTZk3R5MO5alJRAfPZRgOYXdteEW4Io4j3MjuJRMpV_eEOjdOzVWU20onXOqVn4ZBcyGqE15xdOb3HpxXBUUgoIzbFshyphenhyphen8XCIaFsnoqgOHIURlO0ncq9ZhIcG2yoCtiWL1ygZgA/w277-h320/Welcome.jpg" width="230" /></a></div>So much technical work, so little time! I finished my first three sprints toward standing up the data lab. Standing up infrastructure from scratch so you have clean new compute power is fun, and also a lot of work. Particularly when you include; doing it right, taking no short cuts, and making sure it is secure.<p><u><b>Sprint 0</b>: Setup Ubuntu 20.04 Server with ELK stack.</u></p><p>This was mostly rehydrating virtual server infrastructure I hadn't used in 8.5 years. It needed an upgrade from all perspectives and had a completely new OS. I implemented the ELK stack and made a couple of security changes to lock it all down. I ran a few tests by setting up a couple of websites, getting the JSON confirmation from ElasticSearch, and called up the Kibana dashboard. Oooo... sweet success!</p><p><u><b>Sprint 1:</b> Vulnerability Assessment. Security changes if required.</u></p><p>This evening I spent some time poking at the overall vulnerability of the server and with the ElasticSearch and Kibana services. I made a few additions and changes for further locking down the services and believe they are as secure as they can be for this first release. Very happy to feel reasonably confident about it's being locked down. Maybe, I'll get lucky and get some free PEN testing. ha.</p><p><u><b>Sprint </b>2: Identify and register some well aligned domain names.</u></p><p>I registered the following domain names, even considered buying one... it would have been too expensive. I'll implement the data lab on the oceansofdatalab.com site when it becomes closer to being a minimal viable product (MVP).</p><p></p><ul style="text-align: left;"><li>oceansofdatalab.com</li><li>oceansofdatalab.org</li><li>oceansofdatalab.net</li><li>oceansofdata.net</li><li>sevenseasofdata.com</li><li>sevenseasofdata.org</li></ul><p></p><p><br /></p>Peterhttp://www.blogger.com/profile/01204894304194832459noreply@blogger.comtag:blogger.com,1999:blog-24990866.post-39789955623577773002021-03-18T21:56:00.004-02:302021-03-19T07:43:01.260-02:30ODE Data Lab has its technology footing<p>This month has become more about standing up technology than it has been talking to people about their ocean data needs. That's ok... if you are building a technology company, you need to build technology. Ocean of Data Endeavours (ODE) is about building and utilizing software towards making it super easy to work with data, large amounts of data.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBbvg_TRWpUL6pfAXq-fXfdQLAHU2yztr_auek8UhEXvUxxryRM2vupkzZh2Qw82ge28kOHGgM-xkD4Om_DricRAA59-RN9rcubGLVwtMpb3xr3vv1fURtbUfDt2K43WHPSvkIhA/s830/ubuntu_apache2.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="433" data-original-width="830" height="209" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBbvg_TRWpUL6pfAXq-fXfdQLAHU2yztr_auek8UhEXvUxxryRM2vupkzZh2Qw82ge28kOHGgM-xkD4Om_DricRAA59-RN9rcubGLVwtMpb3xr3vv1fURtbUfDt2K43WHPSvkIhA/w400-h209/ubuntu_apache2.jpg" width="400" /></a></div><br /><div style="margin-left: 1em; margin-right: 1em; text-align: center;"><br /></div><span style="text-align: left;">The last 10 days have been about refreshing a server infrastructure I stood up 12 years ago for a number of other projects. What was left was a couple of simple websites, some domain hosting, and all the related mail server infrastructure. All of this needed a complete refresh to be brought up to date;</span><p></p><ol style="text-align: left;"><li>Rebuild the server infrastructure to have more horsepower. - DONE</li><li>Upgrade the Ubuntu OS from 10.04 (Lucid Lynx) to 20.04 (Focal Fossa). - DONE</li><li>Rework all the domain aliases to remove dependency on a domain I no longer owned. - DONE</li><li>Do some basic security work to the server. Mostly SSH focused. - DONE</li><li>Create a new mail server, and do some mailbox maintenance. - DONE</li><li>Install Apache2 httpd host. - DONE</li><li>Configure Apache2 for a couple of web sites. - DONE</li><li>Code some basic HTML to confirm the sites are working. - DONE</li><li><b>Celebrate!</b> <a href="http://endeavours.com/">http://endeavours.com/</a></li></ol><p></p><p>So good to have all this done. The server will provide a strong foundation and is well prepared for the ELK stack and the first load of ocean data. So excited! </p><p><br /></p>Peterhttp://www.blogger.com/profile/01204894304194832459noreply@blogger.comtag:blogger.com,1999:blog-24990866.post-10204083866408260802021-03-16T08:42:00.000-02:302021-03-16T08:42:24.631-02:30An Important difference between DevOps and DataOps<p>Where DevOps is automation, technology, and delivery focused; <b>DataOps is more customer focused</b>. I like these descriptions from Wikipedia for DevOps and DataOps;</p><p></p><ul style="text-align: left;"><li><b>DevOps </b>is a set of practices that combines software development (Dev) and IT operations (Ops). It aims to shorten the systems development life cycle and provide continuous delivery with high software quality. <a href="https://en.wikipedia.org/wiki/DevOps">https://en.wikipedia.org/wiki/DevOps</a></li></ul><ul style="text-align: left;"><li><b>DataOps </b>is an automated, process-oriented methodology, used by analytic and data teams, to improve the quality and reduce the cycle time of data analytics. While DataOps began as a set of best practices, it has now matured to become a new and independent approach to data analytics. DataOps applies to the entire data lifecycle from data preparation to reporting, and recognizes the interconnected nature of the data analytics team and information technology operations. <a href="https://en.wikipedia.org/wiki/DataOps">https://en.wikipedia.org/wiki/DataOps</a></li></ul><div>The similarities between these two are many, particularly from a process and automation perspective. I see DevOps really focused on delivering quality software, and DataOps focused on delivering visualized data analytics to the customer.</div><div><br /></div><div><b><span style="font-size: medium;">Customer focused DataOps assists with Agility</span></b></div><div><br /></div><div>Having a customer focused data analytics team fits well with an Agile approach. The data analytics team needs very involved customer analysts (or product owners). The customer analyst identifies the KPI's, models, or intelligences that need to be fulfilled. These become part of the backlog, and as new sprints are defined they become focused on the item(s) of analytic. A sprint can be built around a few analytics, then iterate around the items for a DataOps sprint;</div><div><ol style="text-align: left;"><li>Where is the data? How do we get at it?</li><li>How do we best move it? How often? What are the security or privacy issues?</li><li>What needs to be cleansed or transformed? Is the data at the correct granularity?</li><li>Do we already have any related data to improve the intelligence? Is this a new build or do we use / alter an existing pipeline?</li><li>What models or analytics do we apply?</li><li>How do we best visualize the data?</li></ol><div>Not to say that DevOps can't fit well within Agile approaches, it can.... the backlog is more technically focused and fits into the sprint more from a continuous perspective than a customer perspective. (What DevOps features go into a sprint are often negotiated with the product owner). The focus of DataOps is in shipping features that fulfill a visualized analytic or more... The focus of DevOps is in <a href="https://en.wikipedia.org/wiki/Continuous_delivery">CD</a> / <a href="https://en.wikipedia.org/wiki/Continuous_integration">CI</a>...</div></div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhQk5ewS2k5qXHlnrDgcLbHaOfueJyYAYD8N55tLQ0dd_mSAh1MWhqpkKk93Wz8K2R4rcxL2nG9G16KpDUrlYcKSre3_iKKVotmJn1mYE-Hxtdd04IeloIylhC8T6Y2_6SLyRWgRg/s774/Pipleines.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="318" data-original-width="774" height="164" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhQk5ewS2k5qXHlnrDgcLbHaOfueJyYAYD8N55tLQ0dd_mSAh1MWhqpkKk93Wz8K2R4rcxL2nG9G16KpDUrlYcKSre3_iKKVotmJn1mYE-Hxtdd04IeloIylhC8T6Y2_6SLyRWgRg/w400-h164/Pipleines.jpg" width="400" /></a></div><div style="text-align: center;"><a href="https://medium.com/data-ops/dataops-is-not-just-devops-for-data-6e03083157b7">https://medium.com/data-ops/dataops-is-not-just-devops-for-data-6e03083157b7</a></div><div><br /></div><div>This approach worked well for us when working on a Business Intelligence project and our nine week sprints usually focused around 4 to 9 KPI's. The organization was in aerospace, they had many legacy data sources with new data sources coming online. As with many organizations, they were in a state of improvement and transformation. Fitting new cubes, representing KPI's, into sprints allowed us to show progress and success. The biggest challenge wasn't in the technical or delivery side of getting the data to the customer. The challenge came in developing a data team where every team member understood the process end-to-end and the efforts required during each step of the DataOps pipeline. Acquiring, cleansing, and transforming data takes as much effort and understanding as visualizing the data for the customer.</div><p></p>Peterhttp://www.blogger.com/profile/01204894304194832459noreply@blogger.comtag:blogger.com,1999:blog-24990866.post-92189883244912447822021-03-13T11:34:00.007-03:302021-03-14T09:35:23.516-02:30For Contract Database Administrator<p><b>Do you require contracted database administration?</b> Medium to small organizations using database technology to store corporate data definitely need database administration to care and feed for there database technologies. This care and feeding includes, and is not limited to;</p><p></p><ul style="text-align: left;"><li><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyjDyO7JyRwcByc1I7mUyDv367m7WBzs45_fT-WYk-ddcWrJtC-kr9E2v4aV_v1PjL8vzJzlnpDq56Q-8M6nPbtNAzoCKFukylbavHG_R53RJztl28186nu8TvrnG8DVQ-9yn3Pg/s335/DBA.jpg" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="335" data-original-width="289" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyjDyO7JyRwcByc1I7mUyDv367m7WBzs45_fT-WYk-ddcWrJtC-kr9E2v4aV_v1PjL8vzJzlnpDq56Q-8M6nPbtNAzoCKFukylbavHG_R53RJztl28186nu8TvrnG8DVQ-9yn3Pg/w173-h200/DBA.jpg" width="173" /></a></div>Install and maintain database servers.</li><li>Optimize database security.</li><li>Build and maintain ETL pipelines.</li><li>Performance tuning of databases.</li><li>Storage optimization for databases.</li><li>Implement DataOps for up-to-date business analytics and its need for continuous data.</li><li>Install, upgrade, and manage database applications.</li><li>Create automation, and schedule, repeating database tasks.</li><li>Ensure recoverability of database systems.</li></ul><div>If you require any, or all, of these database administration task we can help. With over 30 years of database experience complimented with a technology degree in database management we can work remotely to keep your databases healthy and reduce business risk. Part-time or full-time, reasonable rates.</div><p></p>Peterhttp://www.blogger.com/profile/01204894304194832459noreply@blogger.comtag:blogger.com,1999:blog-24990866.post-83503911167626591432021-03-12T16:39:00.005-03:302021-03-12T16:42:22.124-03:30Data Challenge Panel Session: The Power of data<p><span style="background-color: #f9f9f9; color: #030303; font-size: 14px; white-space: pre-wrap;">I listened in on this presentation about the power of data. In particular, having Susan Hunt from Canada's Ocean Supercluster as one of the presenters. All the presenters had very valuable insights. It was an hour well spent!</span></p><p><span style="background-color: #f9f9f9; color: #030303; font-size: medium; white-space: pre-wrap;"><b>The Session: The Power of data</b></span></p><p><span style="background-color: #f9f9f9; color: #030303; font-size: 14px; white-space: pre-wrap;"><b>Description:</b> </span><a href="https://youtu.be/2lAtE9CKmr4" style="background-color: #f9f9f9; font-family: inherit; font-size: 14px; white-space: pre-wrap;">Come learn with us.</a><span style="background-color: #f9f9f9; color: #030303; font-family: inherit; font-size: 14px; white-space: pre-wrap;"> Our panel guests are engaged in transformational data products and projects. Together, we’ll learn what opportunities they are creating when using the newest technology to exploit the power of data. And we’ll talk about careers. The opportunities are limitless.</span></p><p><span style="background-color: #f9f9f9; font-size: 14px; white-space: pre-wrap;"><span style="color: #030303; font-family: inherit;"><b>Moderator:</b>
Cathy Simpson | Chief Executive Officer, TechImpact
<b>Panelists:</b>
Susan Hunt | Chief Technical Officer, Canada’s Ocean Supercluster
Justin Kamerman | Chief Product Officer, Instnt Inc.
Jason Lee | Partner, MNP Technology Solutions</span></span></p><p><span style="font-family: inherit; font-size: medium;"><b>Items for my follow-up:</b></span></p><p>A number of subjects sparked my interest from the presenters discussions. I believe these are the three that need follow-up from the current ODE perspective:</p><p></p><ul style="text-align: left;"><li><span style="font-family: inherit;">Building Models for Data, or transforming Data for Models</span></li><li><span style="font-family: inherit;">What is the business case for the datalab / data workbench?</span></li><li><span style="font-family: inherit;">What is the current state of DataOps?</span></li></ul><p></p><p><br /></p>Peterhttp://www.blogger.com/profile/01204894304194832459noreply@blogger.comtag:blogger.com,1999:blog-24990866.post-35399316482301314582021-03-12T11:25:00.008-03:302021-03-12T13:02:14.992-03:30Peter, we need to integrate data more easily.<p>I liked <a href="https://www.inc.com/jeff-haden/why-innovators-like-elon-musk-jeff-bezos-embrace-this-ancient-problem-solving-technique.html">this article on Fundamental truths</a> when it comes to innovators. I do think Elon Musk and Jeff Bezos have figured out how to be crazy successful in business. I do wish they were more philanthropic with the monies they have accumulated from their success. I digress...</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzNunBVB3d760PmBjFRyo5r0LnI_ykJ-n17mlILOSdOvAcPq9z2uWGWtnJWNNu5bRxAxc9quJXLG70QB0kedGuG3s5sAVX3kjAxtxRxxfPXzz5NH-G5LF8noJK2jjxGxQ7XLr5HQ/s334/Fundamental-Truths.png" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="230" data-original-width="334" height="138" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzNunBVB3d760PmBjFRyo5r0LnI_ykJ-n17mlILOSdOvAcPq9z2uWGWtnJWNNu5bRxAxc9quJXLG70QB0kedGuG3s5sAVX3kjAxtxRxxfPXzz5NH-G5LF8noJK2jjxGxQ7XLr5HQ/w200-h138/Fundamental-Truths.png" width="200" /></a></div>I like the idea of a <b><u>fundamental truth</u></b> as a foundation for your business that doesn't change through time. And when I think about my commitment to build a data services business, I have now started to think about what would be the fundamental truths?<p></p><p></p><ol style="text-align: left;"><li>We need to integrate data more easily.</li><li>We need better ways to visualize, communicate, and understand, the data.</li><li>We would like to reduce the compute and storage costs of large amounts of data.</li></ol><div>These are what I came up with through my review of my initial thinking of fundamental truths. I know there will be a few more and these three will be edited as my idea grows and gets greater footing.</div><p></p>Peterhttp://www.blogger.com/profile/01204894304194832459noreply@blogger.comtag:blogger.com,1999:blog-24990866.post-44448017031702469842021-03-08T13:37:00.003-03:302021-03-08T16:47:00.488-03:30Building the Data Lab Technology Stack<p>The idea of building a data lab is emerging from my ocean data conversations and how to best utilize my knowledge and skillset within this opportunity. In my mind, the service offering would be twofold;</p><p></p><ol style="text-align: left;"><li><b> Data Engineering / Software Development consulting and services</b> with focus on ocean data. We will do the heavy lifting of extracting, cleansing, transforming, and loading your data. And then we will help with analysis and visualizing the data. We are comfortable working in both the open source and Microsoft technology stacks.</li><li>Standing up (and data loading) <b>the technology stack for the data lab</b>. You are going to need to host all this compute power and storage somewhere. It could be on-premise. Most likely, it will be in the cloud. We can help with this also. We could build it in Azure, using the Microsoft technology stack. Or we could build it using an Open Source stack on top of Linux in any of the hosted environments of Azure, AWS, or Rackspace.</li></ol><p></p><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgseASeUzVJ3u_CWqPJLlakR0SvmjzTe0wSYfY1i93qhwXyGeZw4gwzX-VwcMLG9fon81hGY27ipm6IjntNqMDw2q-Y8EXCTYjqSBS2Ynz3LdiPp8wH3HEWnPpG9r01E8u6hUuWg/s499/AdobeDataLab.jpg" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="305" data-original-width="499" height="122" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgseASeUzVJ3u_CWqPJLlakR0SvmjzTe0wSYfY1i93qhwXyGeZw4gwzX-VwcMLG9fon81hGY27ipm6IjntNqMDw2q-Y8EXCTYjqSBS2Ynz3LdiPp8wH3HEWnPpG9r01E8u6hUuWg/w200-h122/AdobeDataLab.jpg" title="https://stock.adobe.com/ca/images/data-lab-vector-illustration/73663389" width="200" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">https://stock.adobe.com/</td></tr></tbody></table>How do you build a low priced, large compute, technology stack to support data engineering efforts, implement a data lab, and showcase these new services capabilities. The low price is the key factor given the current startup state of this ocean data endeavour. Particularly, when you think of the cost of compute for processing and storing large amounts of data. I believe the the best way forward is as follows;<p></p><p></p><ol style="text-align: left;"><li><b>Use open source</b> where you can. Fortunately, many of the infrastructures, tools, frameworks, and programming languages for the data lab are open source.</li><li><b>Automate the build</b> so it can be built and torn down with ease. This would eliminate the need for the stack to always be running.</li><li><b>Store the data at it's source</b>, if possible. Fetch, and load, the data when you automatically rebuild the stack. Keep in mind this limits the amount of big data you can store locally, and loading large amounts of data can cumulatively take days. Be mindful of this. </li></ol><p></p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"><i>Note: this stack is to showcase the services capabilities. A full data lab would also need the ability to both persist and fetch data. It's going to take some time to build the data lab!</i></p></blockquote><p><span style="font-size: medium;"><b>The Data Lab Technology Stack</b></span></p><p>The deployment of this technology stack will use open source wherever possible running on a Linux (Ubuntu) Server hosted at <a href="https://www.rackspace.com/" target="_blank">Rackspace</a>. The rational for these decisions are;</p><p></p><ul style="text-align: left;"><li>Little to No licensing costs </li><li>Strong familiarity with Rackspace as hosting company</li><li>Existing domain name (endeavours.com) hosted with Rackspace</li><li>Extensive experience with Ubuntu Linux in a hosted environment</li><li>Familiarity with deploying data intensive solutions using the ELK stack</li><li>Experience programming in Python</li></ul><p></p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"><i>Note: The deployment of this technology stack will happen in phases, where each phase will complete with some basic tests to ensure the stack behaves as desired.</i></p></blockquote><p><b>Phase 0</b></p><p>Phase 0 will be a basic <a href="https://www.elastic.co/elastic-stack">ELK stack</a> running on an Ubuntu Linux server hosted at Rackspace with access via the endeavours.com web domain. The use case for where the data comes from, how we transform it, the analysis, and visualization is still to be determined. This use case will be used for testing this first iteration of the newly stood up data lab. Exciting times!</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRJLZDYfXTr_XB1l1JRaYEysfc3pAepmZpcU5v2Y-JQpbbvVidNcyZUMpFRViMdLw_n9MO7tLvGjARZzk0VAnkQbi_n-aTnm2dwst4yP5wWS1xDdCc8oVuiTiUfOPPZJWrrsC8XQ/s788/Phase0.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="238" data-original-width="788" height="121" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRJLZDYfXTr_XB1l1JRaYEysfc3pAepmZpcU5v2Y-JQpbbvVidNcyZUMpFRViMdLw_n9MO7tLvGjARZzk0VAnkQbi_n-aTnm2dwst4yP5wWS1xDdCc8oVuiTiUfOPPZJWrrsC8XQ/w400-h121/Phase0.jpg" width="400" /></a></div><p><b>Phase 1</b></p><p>During phase 1 we will add the Python programming language to the technology stack and use it for two purposes;</p><p></p><ul style="text-align: left;"><li>Apply a model to the data using Python.</li><li>Present the processed data to a web page for display.</li></ul><p></p><p><b>Phase 2</b></p><p>During phase 2 we will add Kafka as an infrastructure resource, identify some additional data sources, and pre-process the data before it gets loaded into ElasticSearch.</p><p><b>Phase 3 and beyond</b></p><p>Investigate the Apache Data lab stack, add Spark to our lab, add a data workbench...</p>Peterhttp://www.blogger.com/profile/01204894304194832459noreply@blogger.comtag:blogger.com,1999:blog-24990866.post-31951267454782956332021-03-05T14:56:00.055-03:302021-03-06T08:23:28.955-03:30ODE Newsletter - February 2021<p>I'm 7 people into working towards my 100 conversations. It is said that you need to have 100 conversations as you solidify your business / startup idea. So this is where I am, seven conversation in. If you know anyone who works with ocean data or works for a business that has an interest in oceans, I'd love to talk with them.</p><p>Given the time restraints of being deep into a large data / database migration project, I consider February has been a good month for conversations. It provided me a good view into the horizon of ocean data. I followed the conversations that were presented to me without me directing the focus. For this is the first month, and I have yet to gain clarity of the gaps of where I need more information. This makes sense given I am at the beginning and don't know what I don't know. Now that it is the end of February I have identified the need to talk with customers of ocean data. This could become a focus for March. The conversations for February unfolded in the following order, with the following summaries and highlights;</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPZSng56k9a0uz85RCzZ0toQCQWOT0DlqMCTiQlY2g6KI6A1AjJSV6mXUhRk94i18PLXu9N_3_yB8sAsrTpqvfiyr15AaFAyoNx9c3RLJc_s4EsO1cMB7Tc9NpeoCCoI4PkXLq-g/s275/PropelICT.jpg" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="183" data-original-width="275" height="65" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPZSng56k9a0uz85RCzZ0toQCQWOT0DlqMCTiQlY2g6KI6A1AjJSV6mXUhRk94i18PLXu9N_3_yB8sAsrTpqvfiyr15AaFAyoNx9c3RLJc_s4EsO1cMB7Tc9NpeoCCoI4PkXLq-g/w200-h133/PropelICT.jpg" width="100" /></a></div><span style="font-size: medium;"><b>PropelICT (</b><a href="https://www.propelict.com/">https://www.propelict.com/</a><b>)<br /></b></span><p></p><p>I reached out to a past co-worker in a leadership position within PropelICT. PropelICT is an Atlantic Canada e-accelerator for tech startups. The conversation was very encouraging and initiated my application to their April cohort. Looking forward to their support in the coming months (and years).</p><p><b>Highlights:</b> </p><p></p><ul style="text-align: left;"><li>The idea of 100 conversations.</li><li>My first suggested conversation contact. </li><li>Being a candidate for their e-accelerator.</li></ul><p></p><p><span style="font-size: medium;"><b>eOceans (</b><a href="https://www.eoceans.co/">https://www.eoceans.co/</a><b>)</b></span></p><div class="separator" style="clear: both; text-align: center;"><span style="font-size: medium;"><b><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5Uwvc0jfxjgzGXMSn7R9isGgwMr9VgZs_X8NHrGGC0XA6PGEYuGB6kQONuvODMYA98vqShoOpnVnCCE9WJK2H5MKQ37r09_Gp0mzKoAVuBDpZI9vPdBGS8Uy_i-lgUpcwlpkPjQ/s320/eOceans.png" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="320" data-original-width="320" height="100" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5Uwvc0jfxjgzGXMSn7R9isGgwMr9VgZs_X8NHrGGC0XA6PGEYuGB6kQONuvODMYA98vqShoOpnVnCCE9WJK2H5MKQ37r09_Gp0mzKoAVuBDpZI9vPdBGS8Uy_i-lgUpcwlpkPjQ/w200-h200/eOceans.png" width="100" /></a></b></span></div><p>I spoke with one of the principals of eOceans. Time very well spent, Thank-you! So many details to be digested from this conversation. This organization clearly understands ocean data and where it intersects with social media! A bulleted list seems the best to call out the highlights;</p><p></p><ul style="text-align: left;"><li>There are many open standards and organizations working in this space. The data standards seem to be "standardizing" and there are many organizations working toward bringing the data standards together. More open organizations are contributing than the closed proprietary types. <a href="https://cioos.ca/">CIOOS</a> is the standout for Canada. EU and US are much further down the standards and open data path than Canada.</li><li>Both ends [(data storage and end-points (IoT)] of the data collection are well serviced with lots of business and startup activity. It's the middle were the greater opportunity exists. It's with the data integration with consideration for all the standards and granularity. "It would be nice to dust off a 10 year old data set and be able to easily use it".</li><li>Working with ocean data initiatives is very project based and finding the revenue sources / the business model for an open reference architecture for the digitization of oceans could prove difficult.</li></ul><p></p><p><b>Highlights:</b> </p><p></p><ul style="text-align: left;"><li>Many open organizations already working in the ocean data space. </li><li>The business side of what you are exploring (reference architecture) may be difficult, so much work is project based and gov't funded. A reference architecture seems like an NGO or consortium kind of thing.</li><li>Middle ground of software and data integration could be a big need given my skillset.</li></ul><p></p><p><b><span style="font-size: medium;">Mentorship</span></b></p><p>Super fortunate to reconnect with an older friend who has loads of experience; small devices, programming, data, startups to a favorable exit, machine learning, etc... many skills that align well with what I am doing. And on top of all this, I really enjoy the meandering conversations we share!</p><p>The one area where there is a strong overlap towards my ocean data focus and the mentors previous experience with the integration of data. And yes he confirmed, integrating data from different devices to a common standard is a lot of work for creating a single view into a broad data realm.</p><p><b>Highlight:</b> He agreed to provide me mentorship within this endeavour. So great!</p><p><a href="https://oceansupercluster.ca/event/february-16-2021-events/"><b><span style="font-size: medium;">New Brunswick Ocean Strategy: Our Opportunities in the Blue Economy</span></b></a></p><p>This was an excellent online conference put together by the <a href="https://oceansupercluster.ca/">Ocean Supercluster</a>. What I did most was listen, and a good thing too... I have so much to learn. I really liked the breakout sessions where there was more individual participation. Some names, and acronyms are becoming more familiar too me. </p><p><b>Highlight: </b>A small list of contacts I could reach out to. All good!</p><p><span style="font-size: medium;"><b>TechNL (</b><a href="https://www.technl.ca/">https://www.technl.ca/</a><b>)</b></span></p><div class="separator" style="clear: both; text-align: center;"><span style="font-size: medium;"><b><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAMxMr4IfxkANRgDNeCPsKab1pFTjZDDFMga_hyphenhyphencFxMX7hp0BxchJX2LI_eq5pwZLMWpo3mTmEoaSo6TF0CckJf988GNXh0vNOmJ0ak-22bt8WypUyT-qUwQOaAKfnFWUS1TfA7w/s200/TechNL.jpg" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="200" data-original-width="200" height="100" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAMxMr4IfxkANRgDNeCPsKab1pFTjZDDFMga_hyphenhyphencFxMX7hp0BxchJX2LI_eq5pwZLMWpo3mTmEoaSo6TF0CckJf988GNXh0vNOmJ0ak-22bt8WypUyT-qUwQOaAKfnFWUS1TfA7w/w200-h200/TechNL.jpg" width="100" /></a></b></span></div><p></p><p>I spoke with one of the leaders in TechNL and we talked about what I am wanting to do with data, in particular, ocean data. The conversation pointed towards two relevant contacts;</p><p></p><ul style="text-align: left;"><li><a href="https://petroleumresearch.ca/">Petroleum Research of Newfoundland Labrador (PRNL)</a> and how they are always focused on innovation within the Oil and Gas.</li><li><a href="https://oceansadvance.net/">OceansAdvance</a> which is focused on the Newfoundland and Labrador Ocean Technology sector.</li></ul><p></p><p><b>Highlight: </b>That if I am going to be successful in this endeavour I am going to need partners. The time required for setting up an organization isn't the best place for me to be focusing my time at this stage of the startup. And given the nature of this startup needing to work in the open, the partnership route may be the best way to go...</p><p><span style="font-size: medium;"><b>Canadian Integrated Ocean Observing System (</b><a href="https://cioos.ca/">https://cioos.ca/</a><b>)</b></span></p><p>So fortunate to have the attention of two CIOOS employees! They were so gracious a provided a broad and deep amount of information regarding the state of ocean data. Super helpful! CIOOS clearly knows the data. The best way to summarize my conversation is by including the important questions and there answers;</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p><b>With ocean data where is the greatest pain?</b></p><p>Resources as in financial and skills / knowledge.</p><p>At the more general project level; governance and the people who know how to organize and stewardship data through its lifecycle. This is more a reference to the industry in general... it's a project issue. And having the ability to integrate with a project that happened years ago...</p><p><b>Do open data standards have an influence?</b></p><p>Absolutely! There are many references to open data. Most of what we deal with are open.</p><p><b>How easily integrated are the existing data sets?</b></p><p>It’s getting better. It can be difficult to get an older data set and want to integrate it. These older sets often lack the granularity or metadata that makes it easier to ingest. There is a definite need here at a project level. Developing an expertise here could become a strong business.</p><p>Most initiatives within this space are project based. Which makes it difficult for longer initiatives that have some data sustainability. Rarely are there long term funding initiatives.</p></blockquote><p><b>Highlights:</b> </p><p></p><ul style="text-align: left;"><li>So many acronyms, references and URLs. The CIOOS folks provided me many references all pointing in the right direction. Reference to some of the ISO standards. </li><li>The need for better stewardship of data so as data ages it still has usefulness.</li></ul><p></p><p><span style="font-size: medium;"><b>Pisces Research Project Management (</b><a href="https://piscesrpm.com/">https://piscesrpm.com/</a><b>)</b></span></p><div class="separator" style="clear: both; text-align: center;"><span style="font-size: medium;"><b><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzK4CTqBYW6IFVgaYCakhii7wzqsrnCFgx1MBvIptPcKsWgCHcGa0LbqlScjfMllbpw_ALL15vFr8XoRJF18wpABxXxkKEhAOD4bPW_4C66uMMyFh6_9Ppl0SGeAG-g8ElJLLW_Q/s2048/PiscesRPMlogofish.jpg" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="1313" data-original-width="2048" height="64" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzK4CTqBYW6IFVgaYCakhii7wzqsrnCFgx1MBvIptPcKsWgCHcGa0LbqlScjfMllbpw_ALL15vFr8XoRJF18wpABxXxkKEhAOD4bPW_4C66uMMyFh6_9Ppl0SGeAG-g8ElJLLW_Q/w200-h128/PiscesRPMlogofish.jpg" width="100" /></a></b></span></div><p>Another fortunate conversation with a person deep into ocean data and with the added bonus of being very technical. This was a contact I harvested from the New Brunswick Ocean Strategy Conference. There are may topics I could summarize from this outstanding conversation, much of the information confirmed things I discovered from the previous conversations described above. This is good!</p><p>I did pitch my idea about mooring buoys as a fixed points of data collection, and having these buoys like the personalized weather stations that have become so popular. This employee loved the idea.</p><p>The exciting part of this conversation was the discussion of the technical stack used within the open data within the oceans sector. It was good to add this to the knowledge I had of the proprietary technical stack used when I was managing the software engineering dept. at Provincial Aerospace.</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p><b>What is the most common tech stack for Ocean Data?</b></p><p>This person has extensive experience working with Government Organizations and Academics. From what they have seen the most common, and emerging, technology stack includes;</p></blockquote><p style="text-align: left;"></p><ul style="text-align: left;"><ul><li>Python</li><li>Assorted data storage approaches. Often NOT an RDBMS.</li><li>QGIS is common.</li></ul></ul><p></p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p>These are the tools he finds most effective and common. Using QGIS pushes you into the geo representation of data. Much ocean data requires different kinds of models, more 3d, more oceans… not necessarily geographic, etc.</p><p>The ability to prove models with real data is the biggest need from a technical perspective. This is why python has such good traction. It is easy for non-programmers and also rich enough for programmers. A good language for data, and useful across the technical skills working with data.</p><p>NetCDF is the most common data-store. Also CSV and proprietary data storage. Remember data people are mostly not programmers or overly technical.</p><p>Also take a look at CKAN (https://ckan.org/)</p><p><b>What are people looking for from a technical perspective?</b></p></blockquote><p style="text-align: left;"></p><ul style="text-align: left;"><ul><li>Proving models with real data.</li><li>Integrating data</li></ul></ul><p></p><p><b>Highlights:</b> </p><p></p><ul style="text-align: left;"><li>A deep discussion about the technical stack. The preferred programming languages, data storage, integration approaches, and technical issues.</li><li>Confirmation that integrating data and proving models is an area of software development opportunity.</li></ul><p></p><p><span style="font-size: large;">Lessons Learned</span></p><div class="separator" style="clear: both; text-align: center;"><span style="font-size: large;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgT9RHhSkz5cb5ERq8YzFRZyz7FKaqdiKrdeVw8bg20J9pwocw7Neivnm3_rdkpKvg5sop2gF0uFaAUUXywh5s7j9PU6W2Z3MknxsBYPvK9B_wm4AMvfOdkbZiWeSCu28YyoaQaHQ/s600/davis-instruments-3000199.jpg" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="600" data-original-width="600" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgT9RHhSkz5cb5ERq8YzFRZyz7FKaqdiKrdeVw8bg20J9pwocw7Neivnm3_rdkpKvg5sop2gF0uFaAUUXywh5s7j9PU6W2Z3MknxsBYPvK9B_wm4AMvfOdkbZiWeSCu28YyoaQaHQ/w200-h200/davis-instruments-3000199.jpg" width="200" /></a></span></div><p></p><ol style="text-align: left;"><li><b>A reference architecture for the digitization of oceans</b> is not enough to hang a startup or business upon at this time! Where I do believe it is still a good idea that will form through time. There is so much work already going on for a common open architecture that another doesn't need to be started. I truly believe a reference architecture will emerge, it is a; when it will happen, not if it will happen.</li><li>There is a big need for technical and software development skills and knowledge in the data engineering space of ocean data. I believe the opportunity exists for a <b>software development / data engineering consulting firm</b> with the specialty of ocean data.</li><li>The idea of an anchored (or fixed) buoy for ocean data collection is very compelling too me. Kind of like the personal weather station but as a fixed mooring buoy. Anyone who has a mooring buoy could replace it with <b>the data buoy</b>, and have real-time data about the conditions at the buoy in preparation for mooring.</li></ol><p></p><p><span style="font-size: large;">Next Steps</span></p><p></p><ol style="text-align: left;"><li>March will be the month of <b>broadening my reach</b>. I need to talk with a broader section of people working in the oceans space. I need to find potential customers for the processing and software development in, and around, ocean data. </li><li>I need to start building software tools for the processing of ocean data. I need a reference <b>technology stack showcasing</b> our abilities to work with data.</li><li>I need to start developing an elevator pitch for the<b> ocean data software consulting</b> firm. I need customers and revenue to get the real feedback to focus the business mission.</li></ol><p></p>Peterhttp://www.blogger.com/profile/01204894304194832459noreply@blogger.comtag:blogger.com,1999:blog-24990866.post-81514532134120466302021-02-21T17:21:00.006-03:302021-02-22T11:43:39.493-03:30The Beginning of Ocean Data Endeavours (ODE)<p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWEYcAMO52h8b8aU0LZJ1XJOwF5akt0JDO2E0ZnNi7vAcgtzjJVvqvD8S6w1IFHuIwrsxO5CHceEi7hsJ-91MnebYzYCz_5AkPMpNzatBrNT6hHC5lOvjezksoL5jtJiFK-T-nDg/s327/dataWorkbench.jpg" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="257" data-original-width="327" height="157" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWEYcAMO52h8b8aU0LZJ1XJOwF5akt0JDO2E0ZnNi7vAcgtzjJVvqvD8S6w1IFHuIwrsxO5CHceEi7hsJ-91MnebYzYCz_5AkPMpNzatBrNT6hHC5lOvjezksoL5jtJiFK-T-nDg/w200-h157/dataWorkbench.jpg" width="200" /></a></div>Thirty-nine months ago I started on a deep dive into developing <a href="http://criticaltechnology.blogspot.com/2017/11/digitization-of-oceans-reference.html" target="_blank">a reference architecture for the digitization of oceans</a>. The idea of developing this reference architecture was initiated by the Canadian Government awarding Atlantic Canada with the <a href="https://oceansupercluster.ca/" target="_blank">Ocean Super Cluster</a> initiative and all my recent work with leading <a href="https://palaerospace.com/en/our-services/data-collection-and-management" target="_blank">the software engineering group at Provincial Aerospace</a>. My writing and research into ocean data took me to the point where I needed to deepen my understanding of a number of subjects, and I needed this deeper understanding before I could continue the writing and research (even though you could consider deepening understanding as research). I needed to have an intermediate understanding of what had come before and the current state of things with a reference architecture for the digitization of oceans. In particular, I needed to work directly with ocean data and the standards that influence its structure.<p></p><p>Over the last three years I have been lucky to work with <a href="https://triware.ca/" target="_blank">Triware Technologies Inc.</a>, and together we have found projects that align with this need to deepen my understanding of all things digital and all things ocean. My recent project successes include;</p><p><b><span style="font-size: medium;">OCIO Digital by Design</span></b> - I was fortunate to be awarded the opportunity to be the data architect for the initial phase in digitizing the <a href="https://login.my.gov.nl.ca/login" target="_blank">NL governments citizen facing portal</a>. I remained on the project for the first 12 months through to the portals launch. Being on the design team to create the data tier and integrate with legacy data was a great achievement. And I deeply enjoyed using a scrum / jira approach with a multi-vendor, multi-disciplined team. We achieved a lot in a short period of time.</p><p><b>Lessons Learned</b> - Agile, Scrum and Jira can scale well to a government organization with multiple scrum teams working toward an integrated solution.</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiam6Jw2w9swQ-OB5-iXVfTOuIuCGuKCb_f8dpAZvzmGOAPN2_yaCZHAcM-nBDxwpzZImgVcyStJdi0sYDRu8N4d15k8e567P6JGhb1_ZYWbtRpw_0yWRI5aDtwb6c7mxj4bhy6Xw/s520/ocean-technologies.jpg" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="200" data-original-width="520" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiam6Jw2w9swQ-OB5-iXVfTOuIuCGuKCb_f8dpAZvzmGOAPN2_yaCZHAcM-nBDxwpzZImgVcyStJdi0sYDRu8N4d15k8e567P6JGhb1_ZYWbtRpw_0yWRI5aDtwb6c7mxj4bhy6Xw/s320/ocean-technologies.jpg" width="320" /></a></div><b><span style="font-size: medium;">Ocean Sector Search</span></b> - We needed a way to index the Canadian Ocean Sector. <a href="https://canadasoceanassets.ca/ecosystemsearch/" target="_blank">So we built a search engine</a> seeded by as many oceans related URLs as our analysts could gather. The <a href="http://criticaltechnology.blogspot.com/2019/11/ocean-sector-specific-search.html" target="_blank">technical architecture of this ocean specific search engine</a> can be found in this previous post.<p></p><p><b>Lessons Learned</b> - With reasonable technical effort <a href="http://nutch.apache.org/">Nutch</a> can be configured and seeded to crawl a specific industry sector (in this case Canada Oceans Sector). The Nutch crawl harvested a significant number of pages (> 32000) that were then loaded into the <a href="https://www.elastic.co/elastic-stack" target="_blank">ElasticSearch (ELK) stack</a> while relevancy scoring each page along the way.</p><p><b><span style="font-size: medium;">NLCHI</span></b> - My work with the Newfoundland and Labrador Centre for Health Information (NLCHI) was a quick engagement to focus their requirements backlog into a few manageable sprints. I was super fortunate to help get an important project underway and gain insights into the concept of a customer focused data workbench for a specific subject domain.</p><p><b>Lessons Learned</b> - The idea of a personal data workbench is very compelling when you consider the number of data sets already available in the oceans sector. And if we could fold in open and proprietary data sets, while honoring security and privacy we may be onto something...</p><p><b><span style="font-size: medium;">Nalcor Energy Database Consolidation</span></b> - So many databases, so little time. One of my favorite enterprise type projects is when the project pays for itself, over time, by the savings created by the projects downstream accomplishments. Not revenue generation, but operational expense savings. I believe one of the best KPI's for IT is not new systems implemented, but old systems retired.</p><p><b>Lessons Learned</b> - an amazing amount of data can be moved with the correct use of tools, well built and managed ETL (pipelines), and a mindset of automation.</p><p><b><span style="font-size: medium;"></span></b></p><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhtylW9c_-af1OVfFFPE_ZjeTOiHYw2gkS9O5yan5qn-FKjsMAIqj0yU-yNKxUwmgaXj4xzZGUVP6TvmICxENXhgV0scczR926OK5FHytYQ6UFAxR9mEP4ktZHJx239eFoHhPkm3A/s568/ArgoFloats2018.png" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="291" data-original-width="568" height="164" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhtylW9c_-af1OVfFFPE_ZjeTOiHYw2gkS9O5yan5qn-FKjsMAIqj0yU-yNKxUwmgaXj4xzZGUVP6TvmICxENXhgV0scczR926OK5FHytYQ6UFAxR9mEP4ktZHJx239eFoHhPkm3A/w320-h164/ArgoFloats2018.png" title="Argo Floats 2018" width="320" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Argo Floats - 2018</td></tr></tbody></table><b><span style="font-size: medium;"><br />NEXT STEPS</span></b><p></p><p>Over the last month I have revisited how to best develop my intermediate understanding of oceans data. After a number of conversations, with experts of oceans data, I believe my next steps are twofold; I need to focus on the existing standards for oceans data and I need to write some code to integrate some open oceans data sets.</p><p><b>I need to find opportunities to work directly with oceans data. If you are in the oceans sector, in any way, and you have the need of a very experienced data engineer, then I would love to help with your project. If you know of an oceans data project in needs of a data engineer, please forward on my credentials. Thanks to everyone for reading this far. And thank-you Triware for your ongoing career support!</b></p>Peterhttp://www.blogger.com/profile/01204894304194832459noreply@blogger.comtag:blogger.com,1999:blog-24990866.post-15116863418388556942020-03-19T16:33:00.010-02:302021-02-22T12:39:10.631-03:30Digital by Design, Agility and Data Architecture<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmbQ5zih8OX5x5KPj1r4iuubhUkoruFdig0n4P0iKqhPSR4wN_tzn71P-O7N152nt4C_VvDXHVLm0XHeLLAp-hp3dmIRx_F__hqNJCcAhJ-5cF-AZzzlj0-zxMpHyY0fNA5wiXMA/s1600/DbdWayForward.jpg" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1200" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmbQ5zih8OX5x5KPj1r4iuubhUkoruFdig0n4P0iKqhPSR4wN_tzn71P-O7N152nt4C_VvDXHVLm0XHeLLAp-hp3dmIRx_F__hqNJCcAhJ-5cF-AZzzlj0-zxMpHyY0fNA5wiXMA/s200/DbdWayForward.jpg" width="200" /></a></div>
For 12 months, starting the summer of 2018, I was very fortunate to fill the data architect role for the <a href="https://www.gov.nl.ca/releases/2018/exec/0424n03/">Government of Newfoundland and Labrador's digital by design citizen facing web portal</a>. An amazing team was brought together and we accomplished an amazing amount of work given the complexity of the environment we were all working. Kudos to the leadership team for seeding the ground and pulling together a diverse and effective group of people.<br />
<br />
Being the oldest team member, with 35 years as a technology professional, I noticed a number of items and approaches that I consider the highlights of the project. I call out my 35 years experience because I know success doesn't always happen in a large group of people (with a team larger than 35). A group of strangers doesn't always come together when tasked to ship software on schedule and on budget. The cool part of this project is that the highlights were both technical and project management. In a nutshell, we came together using a scrum model of project management (hosted within JIRA) and architected a microservices technology stack using predominantly Microsoft technologies. The user experience design was exemplary and the software approach stayed aligned with the best of agile practices. We also used a scrum of scrums approach to manage the three distinct scrum teams.<div><br /></div><div><b>What made this first year of a new project so effective?</b><br />
<br />
<b>The Agile Practices</b><br />The team was encouraged to use Agile approaches to successfully ship software. Thankfully, the commitment came from the most senior level and agile workshops were used to align the teams understanding and approach to agile. I consider these three agile practices what kept us all well aligned;<br /><div><ol style="text-align: left;"><li><b>We rigorously stayed with 3 week sprints.</b> This was facilitated by the scrum of scrums group and kept us all focused on shipping working software.</li><li><b>
We embraced jira and stayed true to moving cards.</b> It took a few sprints, as a whole we ended up having all the team members updating and moving cards. This, combined with morning standups, kept the team transparency high and important issues in the open.</li><li><b>We always had demo days and retrospectives.</b> This went a long way to keeping us focused and successful. All team members were encouraged to attend the other scrums demo days, this built excitement and kept us focused and moving.</li></ol>
<b>Software Engineering Discipline</b><br />Developing software is as much art as it is science. Our team included many accomplished software engineers and this helped us implement features quickly and completely. Kudos are deserved by many on this project team, in particular, one of our technical leads (this is you, Phil) was hellbent and lead through example with two attributes of software engineering that are super important and sometimes missed;</div><div><ol style="text-align: left;"><li><b>We refactored always, no excuses</b>. As a group we were always learning, as implementing features is a relentless teacher. Improving upon our code base through refactoring kept the quality improving, and the bugs low. Even from a data architecture perspective, at the beginning of each sprint we refactored the data tier with the required data changes from the previous sprint. Data tiers often have different heart beats that the middle and user tiers as they are dependent on the legacy systems, which often have legacy heart beats. This is a blog post in itself...</li><li><b>
Automated testing.</b> We automated whenever we could, we aspired to have automated tests with coverage to all our code. We got close by using frameworks and having a test first mind set. And don't underestimate how effective existing testing frameworks can be applied to the data tier.</li></ol>
<b>Architecture was collaborative </b><br />
All architects were encouraged to contribute and discuss, we were always white-boarding and soliciting feedback. This kept the architecture strong and well understood throughout the team. And because we had a shared understanding of architecture the refactoring was reduced. All good...<br />
<br /></div></div>Peterhttp://www.blogger.com/profile/01204894304194832459noreply@blogger.comtag:blogger.com,1999:blog-24990866.post-88485639281056602502019-11-26T17:31:00.000-03:302020-02-16T22:50:15.824-03:30Ocean Sector Specific Search<br />
<div style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;">
<img border="0" data-original-height="116" data-original-width="179" height="55" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQPbJU3VGJ8p0pzLIPuhzSGw_J0gLAIL6IIDB64hXtxMg1c5RJdhlqvC8cs7iKviVdl2RzN5bnew3PjJLcJy4dOTJG9J96INSg3P3txngaEJzZweVYB1qbUNq-8l-zsSUdzqbcig/s200/COSC.png" style="text-align: center;" width="80" /></div>
<br />
I've recently finished building <a href="http://canadasoceanassets.ca/ecosystem-search/" target="_blank">an industry specific search engine</a>. The primary use case is to drive international and domestic business traffic to the Canadian websites doing business within the oceans technology and innovation sectors.<br />
<br />
From a technology architecture perspective we built a search engine for the Canadian oceans super cluster initiative where all components run, and are based, upon Canadian assets hosted in Canada. We seeded the search engine using the URLs for all the organizations identified as participants within this economic sector. The indexing process analysed each URL and followed all links up to two hops deep. All the identified URLs were scored using a web graph and the top pages were indexed.<br />
<br />
<b>The architecture decisions<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijv27osSOIIy7J71UMxr1lUqjVMlHDMEhqOThQRO2q_4gblKmMOTtcru_gi9-Ug1yWQ_X7n5uHYLjoeaN0ST_pBvi434ULnHC6uovhxo5TCNUmlwcwevNChUPTRapPRG0fhLVymQ/s1600/ELK+Stack.jpg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="524" data-original-width="462" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijv27osSOIIy7J71UMxr1lUqjVMlHDMEhqOThQRO2q_4gblKmMOTtcru_gi9-Ug1yWQ_X7n5uHYLjoeaN0ST_pBvi434ULnHC6uovhxo5TCNUmlwcwevNChUPTRapPRG0fhLVymQ/s320/ELK+Stack.jpg" width="281" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The NELK stack became our back-end infrastructure.</td></tr>
</tbody></table>
</b><br />
A number of important architecture decisions were made early on as the project was detailed. Mostly decisions were made to support the technologies that the small team was already familiar. And if the team wasn't familiar, we chose technologies that had the most industry support and local resources in our personal networks so we could help out if we needed. We ended up having Nutch feeding the ELK stack using Wordpress for the UX. In the project it became known as the NELK stack.<br />
<ul>
<li><a href="https://nutch.apache.org/"><b>N</b>utch</a> - for web crawling and first round of web page extraction and cleanse.</li>
<li><a href="https://www.elastic.co/"><b>E</b>lasticSearch</a> (ES) - as the search engine / data manager</li>
<li><a href="https://www.elastic.co/logstash"><b>L</b>ogstash</a> - as the data transform and load.</li>
<li><a href="https://www.elastic.co/kibana"><b>K</b>ibana</a> - as the administration / developer console</li>
</ul>
<b>Crawling the web with Nutch</b><br />
We ended up using Nutch to crawl the internet for ocean sector specific web pages. We also needed to integrate with ElasticPress so the broader ecosystem search included the contents of our websites Wordpress database. To do all this we ended up using Nutch 1.15 for it integrated best across our technology stack. We used the Nutch recommended approach seeding, ingesting, fetching, and duplicate removing as we prepared the data for export to ElasticSearch. Due to versioning issues we exported the Nutch database to CSV before importing the data. For the first load of data our use of Nutch created the following page loading metrics;<br />
<ul>
<li>seeded with 2612 domain names</li>
<li>removed 709 duplicate or in error domain names</li>
<li>identified 86872 candidate webpages </li>
<li>fetched the 29323 most relevant web pages (based upon web graph algorithms)</li>
<li>indexed 29270 pages into ElasticSearch</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOc_JZIakwb4Bn4u1YnpUFeBv3tbeHGLjHW5O__fJiCPeXSn5hU6oGgJZj01ifW6-_MgXdhShZEkk-wD-7Mn9wulNCRhwRGy5DgnQe1s62nbvLy6QpXkvrFOoOBJ8EkN18Eol03g/s1600/logstash.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="493" data-original-width="762" height="129" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOc_JZIakwb4Bn4u1YnpUFeBv3tbeHGLjHW5O__fJiCPeXSn5hU6oGgJZj01ifW6-_MgXdhShZEkk-wD-7Mn9wulNCRhwRGy5DgnQe1s62nbvLy6QpXkvrFOoOBJ8EkN18Eol03g/s200/logstash.png" width="200" /></a></div>
<b>Loading data with Logstash, inspecting the results in Kibana</b><br />
We used Logstash to bring the Nutch exported CSV data into ElasticSearch. Coding up the logstash job was fairly easy, the most important aspect was choosing the correct logstash filter. It was better to use the dissect filter rather than the csv filter. More on this in a later post. In the end, I was amazed with how quickly Logstash loaded, and ElasticSearch indexed, all the data.<br />
<br />
Once all the data was loaded into ElasticSearch I used Kibana to confirm data was correctly loaded into the data repository. Kibana has a very intuitive interface and creating filters and running queries to confirm the successful loading of data was straight forward. I look forward to using Kibana to manage the repository and create meaningful dashboards.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<b>Integrating ElasticSearch with WordPress PhP</b><br />
<div class="separator" style="clear: both; text-align: center;">
<b></b></div>
<br />
<div style="text-align: right;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<b></b></div>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuOgDP4S5zexlmkGn01b6C8Xh-OcbUO5az1W2D1Pk75t5AhANjp50uDKjZPJPNWCtY2MhWrS_KZ5Ps83HO8jGGSvipREhxF37yScT_REjIFd3sYjc_YTXhz8diXoMVX3aL50v0Bw/s1600/WP-ESIntegration.jpg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="365" data-original-width="278" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuOgDP4S5zexlmkGn01b6C8Xh-OcbUO5az1W2D1Pk75t5AhANjp50uDKjZPJPNWCtY2MhWrS_KZ5Ps83HO8jGGSvipREhxF37yScT_REjIFd3sYjc_YTXhz8diXoMVX3aL50v0Bw/s320/WP-ESIntegration.jpg" width="241" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Integrating with Wordpress</td></tr>
</tbody></table>
Once we had the back-end built and loaded with industry specific web pages we still needed to find the correct tool-set to provide a query interface within a Wordpress template and to integrate with the organizations identified in the Wordpress database. We wanted the ecosystem search to include both what we had indexed from the internet and the organizations listed in our directories database. The solution ended up using two solutions;<br />
<ul>
<li><b><a href="https://www.elastic.co/guide/en/elasticsearch/client/php-api/current/index.html">The ElasticSearch (ES) PhP library</a></b> which provides a mature (and easy to use) set of features to build your own interface into ES using PhP.</li>
<li><b><a href="https://www.elasticpress.io/"><span id="goog_240793797"></span>ElasticPress</a><span id="goog_240793798"></span></b> which allows automated ElasticSearch integration with a wordpress database.</li>
</ul>
The Wordpress / PhP tools for integrating with ES are very effective. ElasticPress has automation that keeps it up to date as changes are made within the Wordpress database. The ES PhP library has a robust set of features that makes for easy coding and kept search performance very high. Even with large query results the ability to traverse the result set forward and back was easily handles with features available in the PhP library.<br />
<br />
In conclusion, using Nutch with the ELK stack provides for a very strong search engine that integrates easily with Wordpress on the front-end. The learning curve for this approach was not overwhelming and whenever challenges presented themselves the online groups help us resolve issues within days.<br />
<br />
Special thanks to the team put together by Triware Technologies. Without all the other technical people, analysts, business people, data entry, project managers, Oceans Advance, ACOA, Ocean Super Cluster, ElasticSearch support, Azure support, and those clearing the way... none of this would have been possible. Thank-you!<br />
<br />
<ul>
</ul>
<br />
<br />
<br />Peterhttp://www.blogger.com/profile/01204894304194832459noreply@blogger.comSt. John's, NL, Canada47.5615096 -52.71257679999996547.2181116 -53.358023799999962 47.9049076 -52.067129799999968tag:blogger.com,1999:blog-24990866.post-54336841941051204562018-07-05T10:00:00.000-02:302018-07-09T10:17:04.562-02:30Seeking New Opportunities<span style="font-size: large;"><a href="https://www.urbandictionary.com/define.php?term=TLDR">TLDR;</a></span><br />
<b>Veteran technology professional seeking new opportunities.</b> If you are requiring 35 years of career success in the software technology realm that spans startups through to large enterprise environments, then I'm your guy!<br />
<br />
<a href="https://www.linkedin.com/in/prawsthorne/">I have tonnes of experience</a> and I can hit the ground running. I can work as a senior project manager, an enterprise solution architect, as a scrum master and build a team focused with agile approaches, I can design databases and related structures for your business analytic projects, and I can own your Business Intelligence initiative. I can provide strong, and proven, leadership. And if need be I can occupy the technical director level. You will find I provide great business benefit and good value, my experience provides the ability to save you more than I would cost. <br />
<br />
<span style="font-size: large;">A history of project success</span><br />
I'm back to consulting in the technology realm. Being a technology consultant has provided my clients, and myself, many project successes. These successes have been both with technology startups and with medium to large business environments. For more details describing some of my project successes please read these two posts describing projects I have managed and provided technical leadership over the last decade;<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjp7b829y9jgFjXDSTbXHMmh4WbPd1z6BYZNfAE7ekZmqA3oaveW_Nk6UsILKzV_2uU7uOI-jU7Drc-oumAkZ2HWX_nbNFN0gVnlW2ek-mPapdHFlDfW9GByvX5VtFeSlHQvqJLVA/s1600/CLEBC-logo-with-name.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="121" data-original-width="595" height="64" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjp7b829y9jgFjXDSTbXHMmh4WbPd1z6BYZNfAE7ekZmqA3oaveW_Nk6UsILKzV_2uU7uOI-jU7Drc-oumAkZ2HWX_nbNFN0gVnlW2ek-mPapdHFlDfW9GByvX5VtFeSlHQvqJLVA/s320/CLEBC-logo-with-name.png" width="320" /></a></div>
<a href="https://criticaltechnology.blogspot.com/2011/05/increasing-access-to-education.html">Increasing Access to Education</a> - this collection of eight projects helped CLEBC become one of the globes premier legal educators.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOL0G0ktdGS0L-O_K9MpzqbcxF5nhRYnQ8tszIsGY_PAByvyGwYAagUL9tx8tBz3dahqCl0zTACgf2rXpe82XGKLUf35tPJq-F7RhaCcDTXeTTSR-Eys0q3dElAbK4yOYGRAzqUg/s1600/PAL-Aerospace-Logo-and-PLane.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="231" data-original-width="797" height="92" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOL0G0ktdGS0L-O_K9MpzqbcxF5nhRYnQ8tszIsGY_PAByvyGwYAagUL9tx8tBz3dahqCl0zTACgf2rXpe82XGKLUf35tPJq-F7RhaCcDTXeTTSR-Eys0q3dElAbK4yOYGRAzqUg/s320/PAL-Aerospace-Logo-and-PLane.jpg" width="320" /></a></div>
<a href="https://criticaltechnology.blogspot.com/2018/01/career-success-in-three-year-cycles.html">Career success in three year cycles</a> - These four major corporate initiatives leveraged all my abilities to help PAL into their next level of technology capability and realized business value.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div>
<span style="font-size: large;">Where I can help the most</span></div>
<div>
I can have an immediate positive impact to any technology initiative within large and small organizations. This success can come with startups and enterprise environments, or something new that can leverage my skills, knowledge, and experience.<br />
<br />
There are five roles to where I can bring the greatest immediate business value;<br />
<ol>
<li><b>Enterprise and Solution Architecture</b> - The solution side of designing technical architecture has most often been my preferred role within a software development initiative. I've done this for small startups and large enterprises. Good solution architecture creates a better technical solution that saves development costs, improves quality, and ensures alignment with business needs and the other technologies within the business ecosystem.</li>
<li><b>Senior Project Management and/or Managing a Software Development Teams</b> - Through time I have managed many projects with software development teams varying in size from 3 to 30. I have proven track record of bringing projects to completion on-time and on-budget. I seriously enjoy assisting a team of technical people to ship software.</li>
<li><b>Data Analytics Project (Architecture and Team Management)</b> - The B.Tech undergrad degree I completed in the 80's was focused on database management. And since that time my career has had data as its foundation. I do very well with data management [including the design of data structures and related communications (read API's)]. I can also work well within big data projects as my experience with big data goes back to the 1980's. Read one of my big data posts from a few years back to get a sense of this capability; <a href="https://criticaltechnology.blogspot.com/2014/05/big-data-similarities-and-differences.html">Big Data; Similarities and Differences</a>.</li>
<li><b>Technical Mentorship</b> - I'm an educator and have experienced many startups during my time in Vancouver. If you want to discuss the technical side, the team management side, and/or the development methodology to an innovation project (startup or otherwise) I can help here.</li>
<li><b>Innovation</b> - Many times in my career I have taken business strategy and made it happen. If you have a business strategy that requires software innovation I can make it tactical and implement. </li>
</ol>
<span style="font-size: large;">I'm willing to travel</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3x-sV02mOzwdG2n9ZpSG_FYiFcfAYZ1mri6hyphenhyphenJA8YIA-XBkHYhkc19s_zc3kIvzm7e937ihU_IpzXef72zLlraQMbk2GgJui-BxT9s_9XClefBnjMmgjjhWjPj_q-h3jyZX0zkw/s1600/SJIAA-Route-Map-Jan-2018-edit.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="1056" data-original-width="1600" height="131" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3x-sV02mOzwdG2n9ZpSG_FYiFcfAYZ1mri6hyphenhyphenJA8YIA-XBkHYhkc19s_zc3kIvzm7e937ihU_IpzXef72zLlraQMbk2GgJui-BxT9s_9XClefBnjMmgjjhWjPj_q-h3jyZX0zkw/s200/SJIAA-Route-Map-Jan-2018-edit.jpg" width="200" /></a></div>
A single flight from St. John's Newfoundland is my preference. I would also be very interested in consulting work in Vancouver (my home town). I have family there and could easily have all the amenities I require for extended working stays. In a previous blog post I describe how<a href="https://criticaltechnology.blogspot.com/2014/05/the-st-johns-nl-40-million-population.html"> I consider the size of the St. John's marketplace to be 40 million people</a>. I also need to consider the St. John's market to be a single flight away as I seek out new challenges and opportunities. My list of cities I will immediately consider consulting opportunities, include;<br />
<ol>
<li><b>Halifax:</b> it's a morning flight away, and can be there and back including a full days work. I'd also stay for a few days if required.</li>
<li><b>Toronto, Ottawa, Montreal:</b> single 3 hour flight, couple of days then return home for remote working</li>
<li><b>Vancouver:</b> longer stays and longer aways... I have family in Vancouver, so it would be a very nice approach to working.</li>
<li><b>London, England:</b> An outlier, but... It would be nice to work in the EU and I have a British passport so it would ease my travel and work permitting.</li>
</ol>
<span style="font-size: large;">One personal constraint</span><br />
To be succinct, <a href="https://criticaltechnology.blogspot.com/2018/06/work-life-balance-for-sandwiched-cto.html">I require work-life balance...</a> mostly it is about having time to care for middle-school age children and aging parents. This means I am available part-time or full-time with loads of schedule flexibility.<br />
<br />
<br /></div>
Peterhttp://www.blogger.com/profile/01204894304194832459noreply@blogger.comtag:blogger.com,1999:blog-24990866.post-55496705740371582512018-06-27T16:03:00.000-02:302018-06-27T16:03:48.338-02:30Work Life Balance for the Sandwiched CTO<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgm2U1DCnAwGXvuAs-y6f-ISBRgO0PLPv10neSJOGQMyc-UEl2Lkb84pvIDbcixcwSDUOavzLYYRUaBAt0gi_8xo0jFchGeGh_gEtv3092ACKyzAxgBx7v7O-gTEymb7lkb3oHGCw/s1600/KaiBFSGradutaion2018.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="387" data-original-width="847" height="144" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgm2U1DCnAwGXvuAs-y6f-ISBRgO0PLPv10neSJOGQMyc-UEl2Lkb84pvIDbcixcwSDUOavzLYYRUaBAt0gi_8xo0jFchGeGh_gEtv3092ACKyzAxgBx7v7O-gTEymb7lkb3oHGCw/s320/KaiBFSGradutaion2018.jpg" width="320" /></a></div>
Never more in my life have I required work-life balance and understand what it means at a personal level. This all started when I accepted a CTO position with <a href="http://www.bluedroplearningnetworks.com/">Bluedrop Learning Networks</a> ( a great organization with a good alignment to my career journey ). The interview process was extensive and I had the opportunity to meet all corners of the team I would be working with. Overall a good and thorough process. Kudos to everyone involved! The interview process included multiple discussions about work-life balance, and how I (in my mid-50's) have four aging parents over 80 years, two middle-school age children, one young adult age child, a bunch of volunteer activities, personal interests that I wont let go, and three residences to maintain. Oh yes, and a spouse who is an family doctor who also works in palliative care... needless to say I am often the go-to person when a short notice family issue arises. All this said, and after a few important conversations I had to let go of my new CTO role after only six months. I needed to focus on what I consider most important for my family. Though I am not an entrepreneur ( - though, I do think a good CTO needs to have the entrepreneur mindset - ) I completely agree with Randi Zuckerberg about approaches to managing the workload of a senior professional, or also known as the entrepreneur's dilemma.<br />
<blockquote class="twitter-tweet" data-lang="en">
<div dir="ltr" lang="en">
The entrepreneur's dilemma:<br />
<br />
Maintaining friendships. Building a great company. Spending time w/family. Staying fit. Getting sleep.<br />
<br />
Pick 3.</div>
— Randi Zuckerberg (@randizuckerberg) <a href="https://twitter.com/randizuckerberg/status/145030699966136320?ref_src=twsrc%5Etfw">December 9, 2011</a></blockquote>
<script async="" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>
<b>For me, it's that simple.</b> It's the implementation that will be the hard part. I need to have spending time with family, staying fit (in good health), and getting sleep to be my priorities. And I need to organize my life around these three and the others come in between or when priorities change. Something important to consider is that these five items need to be put into the larger context of a life well lived through a whole lifetime. That means you pick three on a daily, weekly, or yearly basis. Set your priorities for the day and/or week and follow through. This allows for flexibility within the entrepreneur's dilemma. In other words;<br />
<blockquote class="twitter-tweet" data-lang="en">
<div dir="ltr" lang="en">
Peter's dilemma:<br />
<br />
Maintain friendships. Have meaningful work. Spend time w/family. Stay healthy. Get sleep. Be creative.<br />
<br />
Iterate around 3 at a time.</div>
— Peter Rawsthorne (@prawsthorne) <a href="https://twitter.com/prawsthorne/status/1012036516229664778?ref_src=twsrc%5Etfw">June 27, 2018</a></blockquote>
<script async="" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>As you can see I also added one, be creative... my thinking around this aligns with having the time to think and create, free the brain to focus on the abstract and artistic and to wonder aimlessly... it is an important part, and keeps your edge in a world where innovation is becoming increasingly important. Peterhttp://www.blogger.com/profile/01204894304194832459noreply@blogger.comtag:blogger.com,1999:blog-24990866.post-15888060518284788312018-01-19T13:14:00.002-03:302018-01-20T09:33:49.696-03:30Career success in three year cyclesMy career seems to go in three year cycles. I've given up trying to determine if this is my unconscious doing or my skills and knowledge lend themselves very well to successful three year project life-cycles. Either way, I'm 35 years into a successful technology career and another three year cycle is coming to an end. And I'm very excited with what is coming! Before I get into the details of the most recent cycle I went back over my past 12 years of blogging to get a sense of my working ebbs and flows, and to confirm my three year cycle theory. The past four, three year cycles also seem to swing from being an employee to working on startups and as a consultant;<br />
<ul>
<li><b>2015 - 2018</b>: Provincial Aerospace </li>
<li><b>2012 - 2014</b>: Startups & Consulting (ICBC, Mozilla, UNESCO, zedIT)</li>
<li><b>2008 - 2011</b>: CLEBC - <a href="https://criticaltechnology.blogspot.ca/2011/05/increasing-access-to-education.html">increasing access to professional development</a></li>
<li><b>2005 - 2007</b>: Startups & Consulting (MUN, COL, ICBC)</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiaTJCPeyClKqHumAiMoN0hUwKMWnPxgsNdKK9PwPm_IzuymnjM_kvaxAg5r5jyAD28-2hVJI6VxCCK2p7cJ9FKzZCBx_vdUyCkECESU0mwUHQC7zGdBprjoLA_gxAD7AU1XHDQXw/s1600/whales_main_full.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="261" data-original-width="598" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiaTJCPeyClKqHumAiMoN0hUwKMWnPxgsNdKK9PwPm_IzuymnjM_kvaxAg5r5jyAD28-2hVJI6VxCCK2p7cJ9FKzZCBx_vdUyCkECESU0mwUHQC7zGdBprjoLA_gxAD7AU1XHDQXw/s1600/whales_main_full.jpg" /></a></div>
<br />
My most recent three year cycle has been with Provincial Aerospace (PAL) in St. John's Newfoundland. It has been a fantastic and exhausting three years. If I was to pick a few themes, they would be; <b><i>team building, shipping software, quality management, and an acquisition.</i></b> So what do I consider my achievements over the past three years and where do I best give thanks. In giving thanks, I will not be calling people out by name - they know who they are.<br />
<ol>
<li><b>Team Alignment</b></li>
When I joined PAL there was so much raw software engineering power with good team trust and camaraderie that my focus immediately moved from team building and skills development to team alignment and clearing the way for their success. And successful they were! We got to the point where every customer was very satisfied and we had solved some lingering and potentially expensive issues.<br /><b style="background-color: #f1c232;">Special Thanks:</b> I want to give thanks to PAL senior management for being patient, keeping the team together, and trusting me to start shipping software. It took some time, we had to clean up some minor releases and complete a significantly complicated major release with a very broad deployment scope.<ul>
<li><b><u>SED Team</u></b></li>
What is most important here is the team was solid within itself as a software engineering team. They had very good end-to-end practices and audit proof trace-ability. The organization as a whole struggled with getting software out to the production environments due to many factors. Keeping in mind deployment is beyond the responsibility of the software engineering team and we had to deploy into highly secure customer networks in very rugged environments. Once we had a realistic schedule the team held the vision for success and delivered.<br /><b style="background-color: #f1c232;">Special Thanks:</b> I want to give thanks to our project manager who is insanely detailed in the best of ways and the team for not only building quality code, but taking on the quality assurance roles for each others work.
<li><b><u>Business Intelligence Team</u></b></li>
Innovation within a strong corporate culture can be hard and an organization will struggle to understand a team with a different rhythm and dance step. And the pressure for the team to align itself back with the corporate rhythm will be ever present and an effort to avoid. If you want innovation you need to allow teams to dance a different step, and allow them to master their new dance. We built a scrum - agile data analytics / business intelligence team from an exceptional group of transactional developers used to a more waterfall type development approach. It took us 12 months (which I consider a short time) to have a performing group of analytics and intelligence developers who ship new data dimensions in a sprint cycle time of 4 - 6 weeks.<br /><b style="background-color: #f1c232;">Special Thanks:</b> I want to give thanks to the whole team for deeply embracing the learning toward becoming knowledgeable analytics developers. With the ability to analyse, design, develop, and deploy for multidimensional data from a plethora of different data sources. I want to thank the scrum master for holding the vision toward the value scrum would bring. I want to give thanks to the business intelligence consultant we brought in for the first six months of the project, they did an excellent job of leading by example and transferring the knowledge required to get the team more than started. I again want to give thanks to PAL senior management for being patient, keeping the team together, and trusting that the data dimensions will come and the team would exceed the targeted number of KPI's.</ul>
<br />
<li><b>Shipping Software</b></li>
Shipping software can be really hard or it can be simple and repeatable. Shipping and deploying software has become an understood and well managed process as the practices have built toward Agile approaches and DevOps over the last 10 - 15 years. The time of failed non-deployed projects should be a thing of the past. If its not, you should either; clear house at the senior leadership level or send everyone to upgrade their literacy of how to provide leadership to software and technology development.<br /><b style="background-color: #f1c232;">Special Thanks:</b> I want to give thanks to the team leadership who came before me. The use of Microsoft TFS within the SDLC made <b>team alignment</b> really easy. I also want to give thanks to the team for adapting so well to adopting more agile and devops type approaches. I may have cleared the way, but the team did the real work!<ul>
<li><u><b>IDNS 2.x (ADAM8 and SIS)</b></u></li>
This was an eighteen month project that included many major features within its release scope. The main considerations for the project were;
<ol>
<li>the schedule restraints [we could NOT go-live to production during iceberg-season (Feb - June)]</li>
<li>the breadth of the feature set we were delivering included;<ul>
<li>Increased security constraints</li>
<li>Service Orientation</li>
<li>Directory Integration</li>
<li>Content Management</li>
<li>Geo-referencing with advanced search</li>
<li>Off-shore client deployment</li>
<li>Legacy application support</li>
</ul>
</li>
<li>deploying to a new data center and multi-server architecture.</li>
<li>Having two versions run in parallel for an ice-season to allow for comprehensive testing and customer network / security integration.</li>
</ol>
<b style="background-color: #f1c232;">Special Thanks:</b> I want to give thanks to everyone involved with this project to hold the vision and getting to finished. We exceeded expectations. I'd also like to give thanks to the IT department for building out the infrastructure, owning the active directory deployment, and being sure we are secure to the levels required by our customers.
<li><b><u>Business Intelligence - Data Analytics</u></b></li>
We started to ship software as releases of new data dimensions (or clusters of key performance indicators). Once we had the basic infrastructure in place with the beginnings of the Extract, Cleanse, Transform, and Load (ECTL), data marts and warehouse we moved into using a scrum type approach to ship new dimensions out of each sprint. This worked well as it gave a reasonable velocity to complete a set of deliverables and then have retrospectives for learning. The team shipped 12 cubes with consistent and interval data refresh which took data from three distinct and distributed data sources. The challenge going forward for the team is in assisting the product owners and subject matter experts in leveraging all the cubes into dashboards and analytics. <br /><b style="background-color: #f1c232;">Special Thanks:</b> I want to give thanks to the team for putting in the extra effort from being on a steep learning curve and focusing on the delivery of data cubes as our measure for success. I want to recognize the commitment of our senior developer in being a learning machine of the business intelligence and data analytics subject domain. I want to give thanks for our DBA taking on all the system and database level tasks to make the project a success.</ul>
<b><br />
</b>
<li><b>Quality Management</b></li>
I'm a strong believer that good software quality management (QMS) brings significant business value. The value comes from; being able to ship software on schedule and on budget, more easily integrating new and unexpected features, ease in adding and training new team members into well known process', and the ability to easily address the scrutiny from customers, auditors, and potential investors. Our team regularly exceeded the procedures and practices within our quality management system and much of our success was due to this rigor.<br /><b style="background-color: #f1c232;">Special Thanks:</b> I want to give thanks to the PAL director of quality, she saw beyond the business of quality certification and her enthusiasm for quality made us all better. I want to again give thanks to our project manager; her commitment to process, record keeping, and traceability is a thing to behold.<ul>
<li><b>Rock solid processes</b></li>
We were fortunate to have developed an end-to-end <a href="https://en.wikipedia.org/wiki/Systems_development_life_cycle">SDLC</a> process that was reflected in our use of Microsoft <a href="https://en.wikipedia.org/wiki/Team_Foundation_Server">TFS</a>. The team was committed to following the process' and we passed all our insanely rigorous ISO audits. We were continuously improving our process' to better suit a more effective and efficient SDLC. We adjusted our QMS to our practices, rather than adjust our practices to our QMS. And our adjustment followed solid change management practices.<br /><b style="background-color: #f1c232;">Special Thanks:</b> I want to give thanks to the software engineering groups leadership team for being willing to continuously improve our SDLC as it was reflected within our QMS, and vice-versa. I want to give thanks to the team for following our practices. I want to give thanks to the auditor for providing candid feedback and encouragement to make change within our ISO certification.
<li><b>Traceability</b></li>
Having the ability to see an idea through to working software is an accomplishment. Having evidence of how the idea became working software, from an auditors perspective, is a thing of beauty.
<li><b>Multiple Successful ISO audits </b></li>
<b style="background-color: #f1c232;">Special Thanks:</b> I want to give thanks to all those who participated in our ISO audits. They can be stressful, but for us we embraced them and used them for our continuous improvement.
</ul>
<br />
<li><b>An Acquisition</b></li>
I feel very fortunate to have had an amazing 35 years working within the technology realm. I've written a lot of code, tuned my share of databases, managed a number of talented teams, architected working solutions that are used daily across Canada and around the world, and held leadership roles with amazing peers. I have had my share of working with startups as an employee, shareholder, and outsider performing technical due diligence. My 30 years in the Vancouver technology scene included working with angel investors and VC's to perform technical due diligence to provide them much needed information. During my time at Provincial Aerospace I was asked to again perform technical due diligence for <a href="https://www.bcms.com/se/en-se/article/pal-and-cartenav-well-equipped-meet-market-demands-following-strategic-acquisition">the PAL acquisition of CarteNav</a>. What made this different is it the first time I did it as an employee of the acquiring company. And fortunately I became a member of CarteNav senior team after the acquisition. This allowed me to see firsthand some of the success and challenges that happen during the first 18 months after an acquisition. Such a great professional experience.<br /><b style="background-color: #f1c232;">Special Thanks:</b> I want to give thanks to senior management for giving me the opportunity to see an acquisition from the other side. It confirms that acquisitions success are about cultural integration during the months that follow. I also want to give thanks to the top tier consulting firm that confirmed my due diligence findings, and were very focused and gracious. I want to give thanks to the CarteNav leadership team (and all the CarteNav employees) for being so welcoming and working so hard through the challenges that come with any acquisition.</ol>
So there you have it, a reasonable summary of the standout themes from my last three years working with Provincial Aerospace. My next adventure begins... more on this to come. Be well...Peterhttp://www.blogger.com/profile/01204894304194832459noreply@blogger.comtag:blogger.com,1999:blog-24990866.post-7856329803819330972018-01-03T03:14:00.004-03:302018-07-05T08:33:50.196-02:30 The oceans data collection platformsMy last post from the end of 2017 spoke to <a href="https://criticaltechnology.blogspot.ca/2017/12/a-plethora-of-end-points.html">the number of sensors and devices</a> that are currently, and becoming easily, available for data collection. This previous post focused more on the generic types of end-points and sensor technology. In this post we focus on the oceans and look at the variety of end-points used in the maritime environment.<br />
<br />
<b>Underwater Sensors</b><br />
As previously identified there are a growing number of sensors and devices available for collecting data in all types of environments. This is also true for the growing number of "platforms" in which these sensors can be deployed. When we extend our view of sensors to include those used in oceans the list of data capture technologies grows. What is important, beyond the details of the data being emitted by these sensors, is how many are becoming a part of the digitization of oceans data "ecosystem". And with so many sensors, and therefore, so much data... a standardized approach and reference architecture defining the approaches to coalesce all this data will become increasingly important.<br />
<b><br /></b>
<b>The oceans data collection "platforms"</b><br />
<br />
<b><a href="https://en.wikipedia.org/wiki/Weather_buoy">Weather buoys</a></b><br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWTlVCRd3AAVMqGxoCObVXyLhKLO0HHa6C5wXsvk6q3sccZqTFw8Z94pcJi7ruDyDhgq9clqlSbDqFdWfKFyWO0PqZVlUMXuJvmtHPWEnXJB810ysp_gkb7SBt7_hxU_S5gm8f6A/s1600/oceanBuoy.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="458" data-original-width="389" height="100" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWTlVCRd3AAVMqGxoCObVXyLhKLO0HHa6C5wXsvk6q3sccZqTFw8Z94pcJi7ruDyDhgq9clqlSbDqFdWfKFyWO0PqZVlUMXuJvmtHPWEnXJB810ysp_gkb7SBt7_hxU_S5gm8f6A/s200/oceanBuoy.png" width="85" /></a>Weather buoys are instruments which collect weather and ocean data within the world's oceans, as well as aid during emergency response to chemical spills, legal proceedings, and engineering design. Moored buoys have been in use since 1951, while drifting buoys have been used since 1979.<br />
<br />
<b><br /></b>
<b>An Integrated Ocean Data System </b><br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiKVGtl2HqCQCeHI1n3wePCWskZBn3bo87IgB-Q5jlZ-tTtihYUlGPd5_JWyJLH7Ex_tPEdii6ioVcJwVYjxK-PEgel82WeHdSoYXT4ylBVHpG1edKqmfeAZzuBNaoyHEw4JA508g/s1600/spoondrift.png" imageanchor="1" style="clear: right; display: inline; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;"><img border="0" data-original-height="210" data-original-width="210" height="100" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiKVGtl2HqCQCeHI1n3wePCWskZBn3bo87IgB-Q5jlZ-tTtihYUlGPd5_JWyJLH7Ex_tPEdii6ioVcJwVYjxK-PEgel82WeHdSoYXT4ylBVHpG1edKqmfeAZzuBNaoyHEw4JA508g/s200/spoondrift.png" width="100" /></a><a href="https://spoondrift.co/pages/our-solution">Spotter</a> is a web-integrated solution for collecting ocean wave and surface current data, designed from the ground up to be easy to use, intuitive, and extremely low cost. The Spotter Device is a compact, solar-powered, surface-follower, which measures surface waves and currents. Through our online Dashboard you can remotely configure your Spotters, access your data in real time, visualize wave data and position tracks. Your Spotter is already connected, so all you do is turn it on and focus on collecting the data you need.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjG2CeaVuyP2Dsb8Fu2aMAK6gUaN3jHlHSD3IshQYXzacSjfZgiNxR5LCPKu8KYA2BwyVFeYP225AxGSx8RErEGg6ybZYpoVqXVI4jNHiQdSlbqoeCjPYwK9pEaiKu1qKPuI6HkIQ/s1600/Underwater-Robots.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="370" data-original-width="530" height="70" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjG2CeaVuyP2Dsb8Fu2aMAK6gUaN3jHlHSD3IshQYXzacSjfZgiNxR5LCPKu8KYA2BwyVFeYP225AxGSx8RErEGg6ybZYpoVqXVI4jNHiQdSlbqoeCjPYwK9pEaiKu1qKPuI6HkIQ/s200/Underwater-Robots.jpg" width="100" /></a></div>
<b>Autonomous Underwater Vehicle</b> (AUV, or unmanned robot submarine)<br />
AUV are increasingly non-proprietary and made up of “plug-and-play” AUV modules which can be brought together and configured in the field. When assembled with a set of survey-grade sonar modules, a <a href="http://www.teledynemarine.com/autonomous-underwater-vehicles-auvs/">Gavia AUV</a> becomes a self-contained survey solution with a low logistics footprint that is capable of carrying out a wide range of missions for commercial, defense, and scientific applications.<br />
<br />
<b>Autonomous Underwater Observatories</b><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgthmKTuZycXfYGteFlECnZ5JLqvIb2qcTb1UIk6XtQjU3-QIadP0JjFamd-4TbtaxM2oB88nfyFIFdN_rkmcUzmILDZhm3Jdk2UbC4vTvQUSQG3VAhV3nSxOlk-tZECX52fJ_y8w/s1600/SmartSub.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="240" data-original-width="268" height="90" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgthmKTuZycXfYGteFlECnZ5JLqvIb2qcTb1UIk6XtQjU3-QIadP0JjFamd-4TbtaxM2oB88nfyFIFdN_rkmcUzmILDZhm3Jdk2UbC4vTvQUSQG3VAhV3nSxOlk-tZECX52fJ_y8w/s200/SmartSub.jpg" width="100" /></a></div>
A submersible platform enabling tailor-made solutions with sensors from a variety of sources. They can be used off-the-shelf or customized to better cover application requirements. They have the ability for long-term deployments and the ability to collect and transmit data through a variety of methods. As an example, review the <a href="https://oceantoday.noaa.gov/deepargo/">deep argo</a> a submersible device for collecting data at extreme depths.<br />
<br />
<b>Others...</b><br />
The list doesn't stop at what is described here. Reading and research on ocean sensors and the digitization of oceans will direct the reader toward the large number of devices collecting data in and around the ocean. Honestly, I'm amazed with the number of devices and the amount of data currently being collected about the oceans. <i>I am increasingly of the belief the we need a reference architecture for the digitization of oceans... Actually, I'm surprised there isn't one already...</i> for it would certainly help to bring together all the oceans data collection on a global scale for the good of us all. If you have any knowledge of an open reference architecture for the digitization of oceans please forward this information along. Thank-you!<br />
<br />
<span style="background-color: white; font-family: "trebuchet ms" , "trebuchet" , "verdana" , sans-serif; font-size: 13.2px;">Over the next few months I will be publishing a series of blog posts describing, in more detail, all the aspects for building a successful digitization of oceans reference architecture. Next up is; "communications" with focus on the data communications available to oceans technology. Please follow along and make comment. For a table of contents of these coming posts please review a companion post; </span><a href="https://criticaltechnology.blogspot.ca/2017/11/digitization-of-oceans-reference.html" style="background-color: white; color: #674ea7; font-family: "Trebuchet MS", Trebuchet, Verdana, sans-serif; font-size: 13.2px; text-decoration-line: none;">Digitization of Oceans Reference Architecture TOC</a><br />
<br />Peterhttp://www.blogger.com/profile/01204894304194832459noreply@blogger.comtag:blogger.com,1999:blog-24990866.post-17041637684108372342017-12-07T14:59:00.003-03:302018-07-05T08:35:41.140-02:30A plethora of end pointsThere is a growing number of data collection devices available to the digitization of everything (including oceans). The variety of devices and sensors includes everything from temperature through chemicals to acceleration. Combine the number of different sensors with the ability to transfer data over great distances and the ability to monitor even the most remote places for obscure data points is increasingly easy and affordable. The following list provides an overview of the types of devices and sensors available.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-bEaTIBuESEPnWKhMiPQAM9zKTRq1WwrTgmG5FwiWU4By3-wAYWlgOIYByKSu2DJXktWyg4vCCD1ta_C7uPMRoZzU70a893HgTvirjIW11ta-X0Gp_5ZCGf4Oq1X_e161lst_GQ/s1600/PlethoraSensors.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="490" data-original-width="633" height="440" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-bEaTIBuESEPnWKhMiPQAM9zKTRq1WwrTgmG5FwiWU4By3-wAYWlgOIYByKSu2DJXktWyg4vCCD1ta_C7uPMRoZzU70a893HgTvirjIW11ta-X0Gp_5ZCGf4Oq1X_e161lst_GQ/s400/PlethoraSensors.jpg" width="550" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://blackboxparadox.com/2016/12/19/iot-sensor-classification/">Internet of Things (IoT) Sensor Classification</a> from Black Box Paradox.</td></tr>
</tbody></table>
<br />
<ul>
<li><b>Position / Presence / Proximity<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibROUQK658dKqFOhBQ1HfkoHymTmBmQT5PfKUGDyf7bi4yCYlmFc27gouw7uQIR8JKwpUFdUWEZey9sWowtfnfWnuY09RrpFcrYU2e8-Ne-dy9SJt67FOOPZPZcSuj8v7oRKATxQ/s1600/rapir_1.jpg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="390" data-original-width="390" height="100" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibROUQK658dKqFOhBQ1HfkoHymTmBmQT5PfKUGDyf7bi4yCYlmFc27gouw7uQIR8JKwpUFdUWEZey9sWowtfnfWnuY09RrpFcrYU2e8-Ne-dy9SJt67FOOPZPZcSuj8v7oRKATxQ/s200/rapir_1.jpg" width="100" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Presence Sensor</i></td></tr>
</tbody></table>
</b></li>
</ul>
<ol>
<li><a href="https://en.wikipedia.org/wiki/Proximity_sensor">Proximity Sensor</a> - A proximity sensor is a sensor able to detect the presence of nearby objects without any physical contact.</li>
<li><a href="https://en.wikipedia.org/wiki/Position_sensor">Position Sensor</a> - A position sensor is any device that permits position measurement.</li>
<li><a href="https://en.wikipedia.org/wiki/Occupancy_sensor">Presence (or Occupancy) Sensor</a> - An occupancy sensor is a motion detecting devices used to detect the presence of a person or object.</li>
</ol>
<ul>
<li><b>Motion / Velocity / Displacement<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgV8BUqopt-RgnqN6zH05WeXFi2eoFxeHjBP07I40AyYs0ES3P_QGKfClEOQm6DhUN-_WOJttnyVMiHpaYKG_Xn6HsT5TqPnxQqjqyrp44JrNs6d8mVqUqYIEPtEVddFNSmKz7t9Q/s1600/Displacement-Sensor.jpg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="250" data-original-width="250" height="100" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgV8BUqopt-RgnqN6zH05WeXFi2eoFxeHjBP07I40AyYs0ES3P_QGKfClEOQm6DhUN-_WOJttnyVMiHpaYKG_Xn6HsT5TqPnxQqjqyrp44JrNs6d8mVqUqYIEPtEVddFNSmKz7t9Q/s200/Displacement-Sensor.jpg" width="100" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Displacement Sensor</i></td></tr>
</tbody></table>
</b></li>
</ul>
<div>
<div>
<ol>
<li><a href="https://en.wikipedia.org/wiki/Motion_detector">Motion Sensor </a>- A motion detector is a device that detects moving objects. Such a device is often integrated as a component of a system that automatically performs a task or alerts a user of motion in an area.</li>
<li><a href="https://en.wikipedia.org/wiki/Velocity_receiver">Velocity Sensor</a> - A velocity receiver (velocity sensor) is a sensor that responds to velocity rather than absolute position.</li>
<li><a href="https://en.wikipedia.org/wiki/Capacitive_displacement_sensor">Displacement Sensor</a> - A displacement sensor (displacement gauge) is used to measure travel range between where an object is and a reference position. Displacement sensors can be used for dimension measurement to determine an object's height, thickness, and width in addition to travel range.</li>
</ol>
</div>
</div>
<ul>
<li><b>Temperature<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJDffJHVRxAJqA4JZxKl5a-b78c6qt2yGMUHc26wZNqGpwoG-3dXleZ2SefI4Px6hSDRz5J_lVd5bUIfvyQaU1QyEgu-_73GY5td6uiHMuVtBNlSZQ9Vug4ZKnalmtaRMBAwU4JQ/s1600/temperature.jpg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="200" data-original-width="200" height="100" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJDffJHVRxAJqA4JZxKl5a-b78c6qt2yGMUHc26wZNqGpwoG-3dXleZ2SefI4Px6hSDRz5J_lVd5bUIfvyQaU1QyEgu-_73GY5td6uiHMuVtBNlSZQ9Vug4ZKnalmtaRMBAwU4JQ/s200/temperature.jpg" width="100" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Temperature Sensor</i></td></tr>
</tbody></table>
</b></li>
</ul>
<blockquote class="tr_bq">
The temperature sensor detects the current temperature or changes in temperature. There is a large number of temperature sensors available and <a href="https://en.wikipedia.org/wiki/List_of_temperature_sensors">a comprehensive list</a> is available on Wikipedia.</blockquote>
<br />
<br />
<ul>
<li><b>Humidity / Moisture<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVwlp1O5_FPPup8uvCjw32jpMzsUDPvxj4rTMQEWoq4YrkYE7BpzDEm2uR-0iuntLDqSfUxSJfQ6kEFl0pxLdMFWtg7LFnc53gwyMveAFan4JOmjcyFrxI1Z4d_Vg7EqWeXBHW7Q/s1600/humidity.jpg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="200" data-original-width="200" height="100" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVwlp1O5_FPPup8uvCjw32jpMzsUDPvxj4rTMQEWoq4YrkYE7BpzDEm2uR-0iuntLDqSfUxSJfQ6kEFl0pxLdMFWtg7LFnc53gwyMveAFan4JOmjcyFrxI1Z4d_Vg7EqWeXBHW7Q/s200/humidity.jpg" width="100" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Humidity Sensor</td></tr>
</tbody></table>
</b></li>
</ul>
<div>
<div>
<ol>
<li><a href="http://www.futureelectronics.com/en/sensors/humidity-dew.aspx">Humidity Sensor</a> - A humidity sensor (or hygrometer) senses, measures and reports the relative humidity in the air. It therefore measures both moisture and air temperature. </li>
<li><a href="https://en.wikipedia.org/wiki/Hygrometer">Moisture sensor</a> - A moisture sensor is an instrument used for measuring the water vapor in the atmosphere. Sometime considered same device as humidity sensor.</li>
</ol>
</div>
</div>
<ul>
<li><b>Acoustic / Sound / Vibration<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0rKt7cazUbtrNGKGEtnQIf075M7XEPlGCc3WZRp7Ovusjqy8eHXxCS1RhLw96_LGcHm3mI3oTOM6Rp4sGVQQB-m9QuC3ld2LVj-sbX1j5SQiS5rW_CIhOKGxg4VXvd-L2ik6BRg/s1600/acoustic.jpg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="169" data-original-width="240" height="70" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0rKt7cazUbtrNGKGEtnQIf075M7XEPlGCc3WZRp7Ovusjqy8eHXxCS1RhLw96_LGcHm3mI3oTOM6Rp4sGVQQB-m9QuC3ld2LVj-sbX1j5SQiS5rW_CIhOKGxg4VXvd-L2ik6BRg/s200/acoustic.jpg" width="100" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Acoustic Sensor</td></tr>
</tbody></table>
</b></li>
</ul>
<div>
<div>
<ol>
<li><a href="https://en.wikipedia.org/wiki/Surface_acoustic_wave_sensor">Acoustic Sensor</a> - Surface acoustic wave sensors are a class of microelectromechanical systems (MEMS) which rely on the modulation of surface acoustic waves to sense a physical phenomenon.</li>
<li>Sound sensor - Sound Sensor can detect the sound intensity of the environment. The Sound Detector is a small board that combines a microphone and some processing circuitry. It provides not only an audio output, but also a binary indication of the presence of sound, and an analog representation of it's amplitude.</li>
<li><a href="http://www.dictionaryofengineering.com/definition/vibration-sensor.html">Vibration sensor</a> - a vibration sensor can detect vibrations. A <a href="http://www.dictionaryofengineering.com/definition/transducer.html">transducer</a>, such as that incorporating a laser or a piezoelectric crystal, which converts vibrations into an electrical equivalent such as a voltage. Also called vibration transducer, or vibration pickup.</li>
</ol>
</div>
</div>
<ul>
<li><b>Chemical / Gas<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZ4WZNDO9Nqu82J7vQNyOQPxRD_C-G93K4JM0_QP8MPn_ODVBQqmWI3Bv3wyqqmJHPVYMK6QuQcy72hYjY6CfCz9wlkf6ftq3ZVJvD_IoX4pwM1wfFQ_NKUzePChfxM0hdko3IKg/s1600/gas.jpg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="600" data-original-width="600" height="100" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZ4WZNDO9Nqu82J7vQNyOQPxRD_C-G93K4JM0_QP8MPn_ODVBQqmWI3Bv3wyqqmJHPVYMK6QuQcy72hYjY6CfCz9wlkf6ftq3ZVJvD_IoX4pwM1wfFQ_NKUzePChfxM0hdko3IKg/s200/gas.jpg" width="100" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Gas Sensor</td></tr>
</tbody></table>
</b></li>
</ul>
<div>
<div>
<ol>
<li><a href="https://en.wikipedia.org/wiki/Sensor#Chemical_sensor">Chemical Sensor</a> - A chemical sensor is a self-contained analytical device that can provide information about the chemical composition of its environment, that is, a liquid or a gas phase. </li>
<li><a href="https://en.wikipedia.org/wiki/Gas_detector">Gas sensor</a> - A gas detector is a device that detects the presence of gases in an area, often as part of a safety system. This type of equipment is used to detect a gas leak or other emissions and can interface with a control system so a process can be automatically shut down.</li>
</ol>
</div>
</div>
<ul>
<li><b>Flow<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCjbVecA1b1mlWMCCQ3QwZUfebnfkvVtnvU0B1IRVDSV2CVryyChVIgKCa8EXHvEhUj9GeaxfnN1MqFVREp3mBFT3zRwiktbmfvFoNERx71kdzt_bwJcyiI62iuTt20dORxHPy_w/s1600/FlowSensor.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="165" data-original-width="200" height="82" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCjbVecA1b1mlWMCCQ3QwZUfebnfkvVtnvU0B1IRVDSV2CVryyChVIgKCa8EXHvEhUj9GeaxfnN1MqFVREp3mBFT3zRwiktbmfvFoNERx71kdzt_bwJcyiI62iuTt20dORxHPy_w/s200/FlowSensor.jpg" width="100" /></a></div>
</b></li>
</ul>
<blockquote class="tr_bq">
Flow Sensors monitor liquid flow rates and accumulated flow volume. Flow measurement is the quantification of bulk fluid movement and <a href="https://en.wikipedia.org/wiki/Flow_measurement">can be measured in a variety of ways</a>.</blockquote>
<ul>
<li><b>Force / Load / Torque / Strain / Pressure</b></li>
</ul>
<div>
<div>
<ol>
<li><a href="https://en.wikipedia.org/wiki/Force-sensing_resistor">Force Sensor</a> - A Force Sensor is defined as a transducer that converts an input mechanical force into an electrical output signal. Force Sensors are also commonly known as Force Transducers.<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5qdYw2IFqo8ubqa9PinzBInbRvjCl-7hQ5R_k_1mBYXk-oU-B3Xh6ePSICpPbBy5MUEtbLy-l3_kZSjFtizUd_N_B49eqRiGRW-hZI4guMaGuK3_ljN03t5eDfAWXMDA0Ql7sdw/s1600/torque_sensor.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="203" data-original-width="200" height="100" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5qdYw2IFqo8ubqa9PinzBInbRvjCl-7hQ5R_k_1mBYXk-oU-B3Xh6ePSICpPbBy5MUEtbLy-l3_kZSjFtizUd_N_B49eqRiGRW-hZI4guMaGuK3_ljN03t5eDfAWXMDA0Ql7sdw/s200/torque_sensor.png" width="99" /></a></div>
</li>
<li><a href="https://en.wikipedia.org/wiki/Load_cell">Load Sensor</a> - A Load Sensor is defined as a transducer that converts an input mechanical force into an electrical output signal. Load Sensors are also commonly known as Load Transducers or Load Cells.</li>
<li><a href="https://en.wikipedia.org/wiki/Torque_sensor">Torque Sensor</a> - A torque sensor, torque transducer or torque meter is a device for measuring and recording the torque on a rotating system, such as an engine, crankshaft, gearbox, transmission, rotor, a bicycle crank or cap torque tester. Static torque is relatively easy to measure.</li>
<li><a href="https://en.wikipedia.org/wiki/Strain_gauge">Strain Sensor</a> - A Strain gage (sometimes referred to as a Strain Gauge) is a sensor whose resistance varies with applied force; It converts force, pressure, tension, weight, etc., into a change in electrical resistance which can then be measured.</li>
<li><a href="https://en.wikipedia.org/wiki/Pressure_sensor">Pressure</a> - A pressure sensor is a device for pressure measurement of gases or liquids. Pressure is an expression of the force required to stop a fluid from expanding, and is usually stated in terms of force per unit area. A pressure sensor usually acts as a transducer; it generates a signal as a function of the pressure imposed.</li>
</ol>
</div>
</div>
<ul>
<li><b>Leaks / Levels<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuPCbFzkbGhiObJeyXMok_XCLhMM760URTqH0E5RVGB5h-Y1F2AYsFYgRtkUSzztZQEYTOGSf-mb9Ew6GaBkLs3SQFmavs1B6iet033KTVla8Vr0wJ3I4sa5JEQ3kPut9FXq_JVg/s1600/level-sensor.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="200" data-original-width="200" height="100" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuPCbFzkbGhiObJeyXMok_XCLhMM760URTqH0E5RVGB5h-Y1F2AYsFYgRtkUSzztZQEYTOGSf-mb9Ew6GaBkLs3SQFmavs1B6iet033KTVla8Vr0wJ3I4sa5JEQ3kPut9FXq_JVg/s200/level-sensor.jpg" width="100" /></a></div>
</b></li>
</ul>
<div>
<div>
<ol>
<li><a href="https://en.wikipedia.org/wiki/Leak_detector">Leak Sensor</a> - leak detection is used to determine if and in some cases where a leak has occurred in systems which contain liquids and gases.</li>
<li><a href="https://en.wikipedia.org/wiki/Level_sensor">Level Sensor</a> - Level sensors detect the level of liquids and other fluids and fluidized solids, including slurries, granular materials, and powders that exhibit an upper free surface.</li>
</ol>
</div>
</div>
<ul>
<li><b>Electric / Magnetic<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqbAqdZ_ZrhyphenhyphenItMRpy6ulDBytX2FInA1VAqeOtSbypFB4iuu1594lC1iInhmbzCrGbk7p23qWICp8rBig0ALejUSaOEIqCEf2JO4uaa_m_76xAsOv7EJ6YHisqNq3A21w-VmHQ_w/s1600/magnetic-sensor.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="200" data-original-width="200" height="100" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqbAqdZ_ZrhyphenhyphenItMRpy6ulDBytX2FInA1VAqeOtSbypFB4iuu1594lC1iInhmbzCrGbk7p23qWICp8rBig0ALejUSaOEIqCEf2JO4uaa_m_76xAsOv7EJ6YHisqNq3A21w-VmHQ_w/s200/magnetic-sensor.jpg" width="100" /></a></div>
</b></li>
</ul>
<div>
<div>
<ol>
<li><a href="https://en.wikipedia.org/wiki/Current_sensor">Electric Sensor</a> - A current sensor is a device that detects electric current in a wire, and generates a signal proportional to that current. The generated signal could be analog voltage or current or even a digital output. The generated signal can be then used to display the measured current in an ammeter, or can be stored for further analysis in a data acquisition system, or can be used for the purpose of control.</li>
<li><a href="https://en.wikipedia.org/wiki/MEMS_magnetic_field_sensor">Magnetic Sensor</a> - A MEMS magnetic field sensor is a small-scale microelectromechanical systems (MEMS) device for detecting and measuring magnetic fields (Magnetometer).</li>
</ol>
</div>
</div>
<ul>
<li><b>Acceleration / Tilt<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZ7Hv8wiQKupTiDgggYTn-MEX2vQ7KriLaEmeTkBnQnfqZpS9HvbbZsn6DgtZuWSQzTKORvvclGgpUatezY2YM3Had1cZyi1tMJBrK8ZpMmACXj8HXAMfZF413x2DGbUHtOMjVdg/s1600/acceleration-sensor.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="200" data-original-width="200" height="100" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZ7Hv8wiQKupTiDgggYTn-MEX2vQ7KriLaEmeTkBnQnfqZpS9HvbbZsn6DgtZuWSQzTKORvvclGgpUatezY2YM3Had1cZyi1tMJBrK8ZpMmACXj8HXAMfZF413x2DGbUHtOMjVdg/s200/acceleration-sensor.jpg" width="100" /></a></div>
</b></li>
</ul>
<div>
<div>
<ol>
<li><a href="https://en.wikipedia.org/wiki/Accelerometer">Acceleration Sensor</a> - An accelerometer is a device that measures proper acceleration. Proper acceleration, being the acceleration (or rate of change of velocity) of a body in its own instantaneous rest frame, is not the same as coordinate acceleration, being the acceleration in a fixed coordinate system.</li>
<li><a href="https://en.wikipedia.org/wiki/Inclinometer">Tilt Sensor</a> - A clinometer or inclinometer is an instrument for measuring angles of slope (or tilt), elevation or depression of an object with respect to gravity. It is also known as a tilt indicator, tilt sensor, tilt meter, slope alert, slope gauge, gradient meter, gradiometer, level gauge, level meter, declinometer, and pitch & roll indicator.</li>
</ol>
</div>
</div>
<ul>
<li><b>Machine Vision / Optical / Ambient Light<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-4S3gAjDzL7ymX6sDOG5J-xYYYj7s9ZXYf3oK1G5vfIs1BXi41GcddmFgF130euEeo2HX4FcxII2B6AzRFxLdHZ3dak3aBAvoYX315GDbTLXtO6PkXnNx8ybzaJygPCO_j001Nw/s1600/LightSensors.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="254" data-original-width="254" height="100" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-4S3gAjDzL7ymX6sDOG5J-xYYYj7s9ZXYf3oK1G5vfIs1BXi41GcddmFgF130euEeo2HX4FcxII2B6AzRFxLdHZ3dak3aBAvoYX315GDbTLXtO6PkXnNx8ybzaJygPCO_j001Nw/s200/LightSensors.png" width="100" /></a></div>
</b></li>
</ul>
<div>
<ol>
<li><a href="http://www.globalspec.com/learnmore/video_imaging_equipment/machine_vision_inspection_equipment/vision_sensors">Machine Vision Sensor</a> - As a simple concept, machine vision is the use of devices for optical non-contact sensing to automatically receive and interpret an image of a real scene in order to obtain information and/or control machines or processes. Machine vision (MV) is the technology and methods used to provide imaging-based automatic inspection and analysis for such applications as automatic inspection, process control, and robot guidance, usually in industry. Machine vision is a term encompassing a large number of technologies, software and hardware products, integrated systems, actions, methods and expertise. Machine vision as a systems engineering discipline can be considered distinct from computer vision, a form of computer science.</li>
<li><a href="https://en.wikipedia.org/wiki/Electro-optical_sensor">Optical Sensor</a> - Electro-optical sensors are electronic detectors that convert light, or a change in light, into an electronic signal.</li>
<li>Ambient Light Sensor - A device that detects the amount of light in the vicinity. An ambient light sensor may be built into a smartphone or tablet to adjust the screen brightness based on the available light.</li>
</ol>
</div>
<div>
<span style="font-size: large;">Security matters!</span><br />
It is also very important to note that as sensors become increasingly available and controlled over the network, <u>security should be of huge concern</u>. Industrial controllers are becoming increasingly targeted for security vulnerabilities and if an oceans sensor is available from a remote location over a network it is potentially open to attack. Being aware of the data available from the sensor and any control features the sensor (activator, controller) may have.<br />
<br /></div>
<div>
<span style="font-size: large;">The other end</span><br />
When considering the digitization of oceans reference architecture and what is considered an end-point we need to also look to the other end. And by the other end, I mean the data storage end. In this post we have discussed all the end points that <a href="https://www.google.ca/search?q=define+emit">emit</a> the data, and at some point the data will need to be "at rest" stored in some storage device. The topic of data storage will be discussed in a later post.</div>
<div>
<br /></div>
<div>
<span style="font-size: large;">Some examples of end-points</span><br />
<ul>
<li><b>Nomad</b> - The AXYS NOMAD is a unique aluminum environmental monitoring buoy designed for deployments in extreme conditions. The NOMAD (Navy Oceanographic Meteorological Automatic Device) is a modified version of the 6m hull originally designed in the 1940’s for the U.S. Navy’s offshore data collection program. It has been operating in Canada’s Weather Buoy network for over 25 years and commonly experiences winter storms and hurricanes with wave heights approaching 20m.</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjwLp4sCEE1eWwSzjQqVsZ3XsGvWtpKl2NH7TOmnDbSIezG-7mpS5VkRLrF65fM1hWDlGviCFSNQ_JAEegxncEyX5gs-ipYZMahpLQWHgCkz-K1iuJlNkk0hFhFVG13rN9s1ddztQ/s1600/Nomad.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="400" data-original-width="698" height="227" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjwLp4sCEE1eWwSzjQqVsZ3XsGvWtpKl2NH7TOmnDbSIezG-7mpS5VkRLrF65fM1hWDlGviCFSNQ_JAEegxncEyX5gs-ipYZMahpLQWHgCkz-K1iuJlNkk0hFhFVG13rN9s1ddztQ/s400/Nomad.jpg" width="400" /></a></div>
<div>
<br /></div>
<ul>
<li><b>Coast Guard Canada</b> - it is very hard to imagine this autonomous vessel will not be loaded with sensors to collect data. Portsmouth, UK, based ASV Global has converted a 26ft hydrographic survey launch to enable it to operate autonomously using the ASView control system, while maintaining its ability to operate in a conventional manned mode. The launch, which is part of the Canadian Coast Guard’s fleet dedicated to the survey operations of the Canadian Hydrographic Service, will be used as a test platform for unmanned survey work.</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4J7nQfsFVPWlUBP7ayEkqh5milTLbivPc9tL-5XXEY-Ycl5iFtNDdTcr-elbjdlNiq9bLoTx61TT_D1FIH1Q1e3cVoNJaCywTYEr10vCUUhQ8ZipYOJGIIfR7yNfR06ju3DQ56w/s1600/CCC.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="281" data-original-width="500" height="223" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4J7nQfsFVPWlUBP7ayEkqh5milTLbivPc9tL-5XXEY-Ycl5iFtNDdTcr-elbjdlNiq9bLoTx61TT_D1FIH1Q1e3cVoNJaCywTYEr10vCUUhQ8ZipYOJGIIfR7yNfR06ju3DQ56w/s400/CCC.png" width="400" /></a></div>
<div>
<br /></div>
<ul>
<li><b>Personal Weather Stations</b> - A personal weather station is a set of weather measuring instruments operated by a private individual, club, association, or even business (where obtaining and distributing weather data is not a part of the entity's business operation). Personal weather stations have become more advanced and can include many different sensors to measure weather conditions. These sensors can vary between models but most measure wind speed, wind direction, outdoor and indoor temperatures, outdoor and indoor humidity, barometric pressure, rainfall, and finally UV or solar radiation.</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiabAfG1Kx5RJPAxgC8DgDksj8AaWi5NxAGjbhmq2HcmZRS8NefvDbJ2baWYTJNMw1d6c3gVY4l60XUHt7jhLsg3ARYtonvjnGYuM3xB1KDR-eEe4PLgGno0x4TfaWHZX7hEQ2pxA/s1600/PersonalWeatherStation.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="375" data-original-width="500" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiabAfG1Kx5RJPAxgC8DgDksj8AaWi5NxAGjbhmq2HcmZRS8NefvDbJ2baWYTJNMw1d6c3gVY4l60XUHt7jhLsg3ARYtonvjnGYuM3xB1KDR-eEe4PLgGno0x4TfaWHZX7hEQ2pxA/s320/PersonalWeatherStation.jpg" width="320" /></a></div>
<div>
<span style="background-color: white; font-family: "trebuchet ms" , "trebuchet" , "verdana" , sans-serif; font-size: 13.2px;"><br /></span>
<span style="background-color: white; font-family: "trebuchet ms" , "trebuchet" , "verdana" , sans-serif; font-size: 13.2px;"><br /></span>
<span style="background-color: white; font-family: "trebuchet ms" , "trebuchet" , "verdana" , sans-serif; font-size: 13.2px;">Over the next few months I will be publishing a series of blog posts describing, in more detail, all the aspects for building a successful digitization of oceans reference architecture. Next up is; "communications" with focus on the data communications available to oceans technology. Please follow along and make comment. For a table of contents of these coming posts please review a companion post; </span><a href="https://criticaltechnology.blogspot.ca/2017/11/digitization-of-oceans-reference.html" style="background-color: white; color: #674ea7; font-family: "Trebuchet MS", Trebuchet, Verdana, sans-serif; font-size: 13.2px; text-decoration-line: none;">Digitization of Oceans Reference Architecture TOC</a></div>
</div>
Peterhttp://www.blogger.com/profile/01204894304194832459noreply@blogger.comtag:blogger.com,1999:blog-24990866.post-45053988038441223232017-11-26T09:13:00.000-03:302017-11-26T09:19:32.846-03:30three posts for digitization of oceans reference architectureThe next three posts within my series describing the need for a digitization of oceans reference architecture will be focused on the three technology domains of; end-points, communications, and data stores. This separation is to allow a deeper look into each domain as they have different considerations in relation to technology architecture and attributes important to the digitization of oceans.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEihY_QKTuoqfZt8GFMyavEIs_E_uAPcczVbwZmWO-UMaBBP5Nr2QrJmwDOyQ7-2yzDIU2hNRIASlwxgHvA4p48DwVxR7oHT2EYUILiBcFKNY3ZJq7G0SqV4q_zX4crR9BuRf-SDfA/s1600/DigitizationOceans.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="155" data-original-width="471" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEihY_QKTuoqfZt8GFMyavEIs_E_uAPcczVbwZmWO-UMaBBP5Nr2QrJmwDOyQ7-2yzDIU2hNRIASlwxgHvA4p48DwVxR7oHT2EYUILiBcFKNY3ZJq7G0SqV4q_zX4crR9BuRf-SDfA/s1600/DigitizationOceans.jpg" /></a></div>
<b><br /></b>
<b>End Points:</b> the sensors and devices which collect and emit data. Consider this the Internet of Things (IoT) that can be located anywhere within and around oceans, airborne, surface, and submersible.<br />
<br />
<a href="https://en.wikipedia.org/wiki/User:Peterrawsthorne/Digitization_of_Oceans#End_Points">https://en.wikipedia.org/wiki/User:Peterrawsthorne/Digitization_of_Oceans#End_Points</a><br />
<br />
<b>Communications: </b>the communications technologies available to transfer data from one place to another. A lot to explore within this domain; as underwater data transmission is an emerging technology, and the structure of the data messages will become the foundation of the reference architecture.<br />
<br />
<a href="https://en.wikipedia.org/wiki/User:Peterrawsthorne/Digitization_of_Oceans#Communications">https://en.wikipedia.org/wiki/User:Peterrawsthorne/Digitization_of_Oceans#Communications</a><br />
<br />
<b>Data Stores:</b> there are many existing data storage approaches, locations, and structures that can be used to store the oceans data. Databases and database designs are already available for many of the subjects within the digitization of oceans. Better to use exiting methods to store the structured and unstructured data and use a federated approach to bring them together.<br />
<br />
<a href="https://en.wikipedia.org/wiki/User:Peterrawsthorne/Digitization_of_Oceans#Data_Stores">https://en.wikipedia.org/wiki/User:Peterrawsthorne/Digitization_of_Oceans#Data_Stores</a><br />
<br />
<b>Creating an inventory</b><br />
The number of technologies, vendors, standards, and approaches within these three domains will be large and forever growing.<a href="https://en.wikipedia.org/wiki/User:Peterrawsthorne/Digitization_of_Oceans#Lists"> To start documenting this inventory I have created a Wikipedia page under my Wikipedia account.</a> Once this page describing the Digitization of Oceans and its Reference Architecture is more complete I will submit it as a published article otherwise consider it a <u>work in progress</u>. Feel free to join in and edit the work in progress wiki page;<br />
<br />
<a href="https://en.wikipedia.org/wiki/User:Peterrawsthorne/Digitization_of_Oceans#Lists">https://en.wikipedia.org/wiki/User:Peterrawsthorne/Digitization_of_Oceans#Lists</a>Peterhttp://www.blogger.com/profile/01204894304194832459noreply@blogger.comtag:blogger.com,1999:blog-24990866.post-26609162837433186842017-11-21T21:30:00.000-03:302018-07-05T08:34:58.336-02:30What is a reference architecture?<div dir="ltr">
There are many descriptions of reference architectures available on the web. Here is a list of some I consider do a good job of describing the subject while supporting the description I am working toward in the digitization of oceans;</div>
<div dir="ltr">
<ol>
<li><a href="https://www.ibm.com/developerworks/rational/library/2774.html">Reference Architecture: The best of best practices</a> - given its age (published 2002) it is still relevant and pragmatic. Though I do consider the description too dependent upon <a href="https://en.wikipedia.org/wiki/Rational_Unified_Process">RUP</a>, which introduces many weighty practices and misses some of the more agile and emergent approaches. Still the description gives good detail to the importance, breadth, and depth of the reference architecture. The later sections of this description provide information on creating, using, updating, and working with a reference architecture. These sections are particularly useful in developing the digitization of oceans reference architecture. I strongly believe the oceans reference architecture will be emergent as many new technologies and stakeholders contribute and become a part of developing the architecture.<br /><blockquote class="tr_bq">
<i><a href="https://simplicable.com/new/emergent-architecture">Emergent architecture</a> is when organizational structures such as business processes and technologies are designed incrementally by many designers.</i></blockquote>
</li>
<li><a href="https://en.wikipedia.org/wiki/Reference_architecture">Wikipedia: Reference Architecture</a> - very short for a complex topic, but it is too the point in defining the reference architecture as templates within a subject, industry, or domain. It stresses the importance of a common vocabulary and in drawing upon successful projects within the domain. It aligns with the use of <a href="https://en.wikipedia.org/wiki/Application_programming_interface">APIs</a> which I believe will become an important part of a strong digitization of oceans reference architecture. It also calls out a number of the benefits derived from the reference architecture.<br /><br />
</li>
<li><a href="https://www.cio.com/article/2372571/enterprise-architecture/the-essential-ea-toolkit-part-2---a-reference-architecture-and-standards-rep.html">CIO Online Magazine</a> - describes where the reference architecture fits within the <a href="https://en.wikipedia.org/wiki/Enterprise_architecture">EA</a> toolkit, and looks to all the relationships among <a href="https://timreview.ca/article/919">business, systems, and technology</a>. It describes how the reference architecture can greatly assist in defining specific technical deliverables within these complex systems. Having a proven, standards based, and shared toolkit for developing the oceans reference architecture will assist in keeping the architecture team distributed throughout Atlantic Canada well aligned when creating and maintaining the reference architecture..</li>
</ol>
<b>Some example reference architectures:</b><br />
<ol>
<li><a href="http://microsoft%20industry%20reference%20architecture%20for%20banking%20%28mira-b%29/">Microsoft Industry Reference Architecture for Banking (MIRA-B)</a></li>
<li><a href="https://hortonworks.com/blog/reference-architecture-open-banking-standard/">A Reference Architecture for The Open Banking Standard</a></li>
<li><a href="https://www.ibm.com/support/knowledgecenter/en/SSCNZQ_7.0.0/com.ibm.ws.icp.insp_c.doc/ins/pc/pc_ra/concept/int/c_hcarepaypack.html">IBM Insurance Reference Architecture</a></li>
<li><a href="https://www.ibm.com/support/knowledgecenter/en/SSFPJS_7.5.0/com.ibm.ws.icp.hccpay.doc/hc/payor/payor_ra/concept/int/c_hcarepaypack.html">Healthcare Reference Architecture</a></li>
</ol>
These four serve as examples of reference architectures from established industries where the patterns, technologies, and architectures have developed through time. As you read through these you can get a sense of the value, industry collaboration, growth, and innovation that can be facilitated by having a comprehensive industry reference architecture.<br />
<br />
<b>What is unique about a digitization of oceans reference architecture?</b><br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4pYFg02c9Ws4dBL8h6DunxAoAox2cmyjeCSese1ectAG5b3tDPlVRpCdYyiEr7pLqbbh_nZVyJ3KIEpDoOXDYpc8xBhDagSS5cw9MjyZl_YHKhrUOzcHnk5Gm3HlNw8s5zDi6BQ/s1600/Digital-Ocean-A-System-of-Systems-Liquid-Robotics.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="436" data-original-width="768" height="290" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4pYFg02c9Ws4dBL8h6DunxAoAox2cmyjeCSese1ectAG5b3tDPlVRpCdYyiEr7pLqbbh_nZVyJ3KIEpDoOXDYpc8xBhDagSS5cw9MjyZl_YHKhrUOzcHnk5Gm3HlNw8s5zDi6BQ/s640/Digital-Ocean-A-System-of-Systems-Liquid-Robotics.png" width="512" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The infrastructure of the Digital Ocean. (Courtesy of Liquid Robotics, a Boeing Company)</td></tr>
</tbody></table>
It's too early in this discussion to be specific about the oceans reference architecture, it is important to note that it is both broad and deep. It is broad in that it includes many ground based systems, processes, and infrastructure similar in complexity to the previously mentioned examples. In addition to this broadness we need to add the theater in which the digitization of oceans operates; we have vessels of many types (airborne, surface, and submersible), we have a growing collection of sensors and protocols, we have cross industry collaborations (fisheries, environment, oil and gas, shipping, researchers, academia, defense, etc.). I believe it is safe to say the reference architecture for the digitization of oceans will be very broad due to the number of data collection points and the number of intersections (technical and otherwise). The oceans reference architecture will also be very deep in that the data will be coming from many sources above and below the ocean surface. And the data that is being collected will both be very specific and detailed, while also being general at a more meta level. I believe it is the breadth and depth of the digitization of oceans that make its reference architecture unique. And its creation is a large, important, and emerging challenge... more on this to come.<br />
<br />
Over the next few months I will be publishing a series of blog posts describing, in more detail, all the aspects for building a successful digitization of oceans reference architecture. Next up is;<b> "a plethora of end points"</b> with focus on oceans technology and the <a href="https://en.wikipedia.org/wiki/Internet_of_things">Internet of Things (IoT)</a>. Please follow along and make comment. For a table of contents of these coming posts please review a companion post; <a href="https://criticaltechnology.blogspot.ca/2017/11/digitization-of-oceans-reference.html">Digitization of Oceans Reference Architecture TOC</a><br />
<div>
<br /></div>
</div>
Peterhttp://www.blogger.com/profile/01204894304194832459noreply@blogger.comtag:blogger.com,1999:blog-24990866.post-2858514853175760742017-11-13T19:10:00.000-03:302018-01-06T09:30:20.427-03:30Digitization of Oceans Reference Architecture TOC<div dir="ltr">
</div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3DJMg0mx_0DR0t_LMu-uGHxEmmSkRyakDU5crywaHJF9fNbVUagzcOXFv1MuSHkqn5oDQ3A8WUD1gKRhTfqgWDfeBSGctNq4V6e063yrYsTWZyLw2SbYeVMW7-gnhmxQ4ma4iSQ/s1600/LobsterPots.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="525" data-original-width="288" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3DJMg0mx_0DR0t_LMu-uGHxEmmSkRyakDU5crywaHJF9fNbVUagzcOXFv1MuSHkqn5oDQ3A8WUD1gKRhTfqgWDfeBSGctNq4V6e063yrYsTWZyLw2SbYeVMW7-gnhmxQ4ma4iSQ/s400/LobsterPots.jpg" width="218" /></a></div>
For a sense of where I am going with this series of posts describing a reference architecture for the digitization of oceans, please consider this "table of contents". As I complete items in the list, I will update this TOC;</div>
<ol>
<li><b><a href="https://criticaltechnology.blogspot.ca/2017/11/reference-architecture-for-digitization.html">Introduction</a></b> - summary of why a series of posts describing a digitization of oceans reference architecture.</li>
<li><b><a href="https://criticaltechnology.blogspot.ca/2017/11/what-is-reference-architecture.html">What is a reference architecture?</a></b> - summary of the existing online descriptions of reference architecture and why it is important to building a strong technology ecosystem.</li>
<li><b><a href="https://criticaltechnology.blogspot.ca/2017/12/a-plethora-of-end-points.html">A plethora of end points</a></b> - with new Internet of Things (IoT) end points coming available with increasing frequency we look to how many sensor types are available to an oceans reference architecture and some examples of how they are being used. I've included a couple more posts describing end points, as I deepened my research I felt the descriptions needs to be expanded to include what is happening as end-points in the oceans and in the air.</li>
<ul>
<li><a href="https://criticaltechnology.blogspot.ca/2018/01/the-oceans-data-collection-platforms.html">Oceans end-points</a></li>
<li>Airborne end-points</li>
</ul>
<li><b>Communications</b> - description of the current state of data communications above and below the oceans surface. And why it matters to the reference architecture.</li>
<li><b>Messaging Standards - </b>the structure of the data packages (or messages) between the endpoints is a very important attribute of a successful reference architecture.</li>
<li><b>What is the digitization of oceans? </b>- a high level description of the digitization of oceans. This will detail the breadth and depth of what is considered the digitization of oceans. This description should also consider the intersection of the <a href="http://criticaltechnology.blogspot.ca/2017/04/a-focused-economic-sector.html">different ecosystems</a> of; business, innovation, and knowledge.</li>
<li><b>What is a digitization of oceans reference architecture? </b>- comprehensive diagram of the entities within the oceans reference architecture with detailed description of each item and their connections (digital or otherwise).</li>
<li><b>The importance of good governance - </b>the dynamic nature of innovation within the digitization of oceans will cause many elements of the reference architecture to be changing. To encourage interoperability at all levels (technical and otherwise) having good governance will be paramount for success.</li>
<li><b>The economic value to be found in the oceans reference architecture</b> - why is a reference architecture valuable for community, business, innovation, etc. And why Atlantic Canada should be a major contributor or primary steward of the reference architecture.</li>
<li><b>How to create the digitization of oceans reference architecture</b> - what is the road map in completing the <u>first release</u> of a digitization of oceans reference architecture. I purposely say first release as this reference architecture will need constant tending as new technologies and capabilities come available.</li>
</ol>
Keep in mind this is meant to kick off a conversation about creating a reference architecture for the digitization of oceans. This is NOT something I want to do on my own, or believe I could do effectively on my own without contributions from others and a few years of focused effort. I really want engagement across Atlantic Canada to discuss the idea of creating this reference architecture and to become stewards of the reference architecture as it is used globally for the benefit of everybody.<br />
<i><br /></i>
<i>Disclaimer - All views expressed on this site are my own and do not represent the opinions of any entity whatsoever with which I have been, am now, or will be affiliated. They are views created by my many years as an IT professional and, more importantly, an enterprise architect responsible for building large and distributed systems.</i>Peterhttp://www.blogger.com/profile/01204894304194832459noreply@blogger.comtag:blogger.com,1999:blog-24990866.post-32488192619757469142017-11-12T11:50:00.000-03:302018-07-05T08:33:13.030-02:30Reference architecture for the digitization of oceans<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiO_Ttw0uyFPh6oRYJkzW8qGk8ik8cbhieKWUHpm_vjvBVudV-9-EhiaC6CqQcNqns0TXtK8-3kfarpdGnqvlXwnb-Zf41QUyGcdz8PbQocGruLMhHt_NF-xWC18ThOYTAjZc3kRQ/s1600/DigitalOcean1.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="427" data-original-width="274" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiO_Ttw0uyFPh6oRYJkzW8qGk8ik8cbhieKWUHpm_vjvBVudV-9-EhiaC6CqQcNqns0TXtK8-3kfarpdGnqvlXwnb-Zf41QUyGcdz8PbQocGruLMhHt_NF-xWC18ThOYTAjZc3kRQ/s320/DigitalOcean1.png" width="205" /></a></div>
I strongly believe one of the cornerstones for the successful digitization of oceans is a <a href="https://en.wikipedia.org/wiki/Reference_architecture">reference architecture</a>. I believe this also holds true for the digitization of oil and gas, but I see this digitization as a subset of the oceans. Or more specifically, I see the digitization of maritime oil and gas as a subset of the digitization of oceans. I regress... One of the most important aspects of the reference architecture is its openness (as opposed to proprietary). If we are wanting to spur innovation in Atlantic Canada we need every small, medium, and large organization to realize benefit from shared digital resources. We need a way for all these organizations to openly communicate and build this digital ecosystem. The digitization of oceans reference architecture will define (or utilize existing approaches) the "language" that all oceans technologies communicate with one another and remember their collective history.<br />
<br />
An example; the reference architecture would specify the digital messaging structure for an ocean temperature event. Therefore, when a small startup (that specializes in ocean temperature sensors) needs to publish their data they need only comply with messaging structures for ocean temperature. This would allow everyone in the ecosystem to get at their temperature event data as soon as it is available. Also important, is the startups market for ocean temperature sensors customers includes everyone who is aligned with the oceans reference architecture and it's messaging structures.<br />
<div>
<br />
Another example; the reference architecture would specify the <a href="https://www.wired.com/2013/10/undersea/">underwater wireless</a> communications approaches and suggested protocols and practices. All allowing the temperature event data to be broadcast and communicated to the historical repository for archiving.<br />
<br />
Another example; the reference architecture would specify all the messaging structures and the data storage approaches so the data could be archived and be available through time. The historical archive would allow for retrieval, searching, research, planning, and analysis.<br />
<br />
Over the next few months I will be publishing a series of blog posts describing, in more detail, all the aspects for building a successful digitization of oceans reference architecture. Next up is; "what is a reference architecture" with focus on oceans technology. Please follow along and make comment. For a table of contents of these coming posts please review a companion post; <a href="https://criticaltechnology.blogspot.ca/2017/11/digitization-of-oceans-reference.html">Digitization of Oceans Reference Architecture TOC</a><br />
<br />
<i>Disclaimer - All views expressed on this site are my own and do not represent the opinions of any entity whatsoever with which I have been, am now, or will be affiliated. They are views created by my many years as an IT professional and, more importantly, an enterprise architect responsible for building large and distributed systems.</i></div>
Peterhttp://www.blogger.com/profile/01204894304194832459noreply@blogger.comtag:blogger.com,1999:blog-24990866.post-80492051243275208142017-05-12T13:49:00.001-02:302017-05-12T13:49:34.027-02:30ACAITA Inaugural Meeting St. John's<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgHNvBlfFmiEI_f-Z2sZ4jMTv3BbrqHKGIiskBHJS-3-acGi_GqksON-HL8WM_wjSKjJ7ab5kWvZQG3MFA9E-zNkAlNMfhpUPivhSx4e8zCPv2W8XNJZrzKL0WDPCfXTIyFceOxWw/s1600/the-ship-pub.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgHNvBlfFmiEI_f-Z2sZ4jMTv3BbrqHKGIiskBHJS-3-acGi_GqksON-HL8WM_wjSKjJ7ab5kWvZQG3MFA9E-zNkAlNMfhpUPivhSx4e8zCPv2W8XNJZrzKL0WDPCfXTIyFceOxWw/s1600/the-ship-pub.jpg" /></a></div>
Three architecturally minded technical professionals got together at the Ship Public House to share a couple of beers, tell a few technical project yarns, and enjoy some traditional music. The conversation was also very candid and at times could have been considered cynical. But... from all this emerged some optimism and some healthy and solid work we can do to support and encourage the growth of technology architecture practices in Newfoundland and the whole of Atlantic Canada.<br />
<br />
We all agreed it's a challenging time in Newfoundland. The economy is in decline, the population is aging, the young and up-n-comers are leaving, and the current government is overly focused on the immediate need to get the provincial house in order. So when a few architects and senior software developers get together our pragmatism gets the better of us and we trend toward being pessimistic about the senior technical opportunities available in Newfoundland. We also identified a handful of activities to be optimistic about;<br />
<ul>
<li>St. John's has a growing startup ecosystem - this didn't exist 5 - 7 years ago and now it has a physical space (<a href="http://www.workatcommonground.com/">common ground</a>), a few small and growing technology companies, and s small group of committed people who want to see technology startups thrive in Newfoundland.</li>
<li>A technology industry association (<a href="http://www.nati.net/">NATI</a>) which is committed, effective, and strongly resourced toward building success for the province (and Atlantic Canada as a whole).</li>
<li>Strongly positioned for the <a href="https://en.wikipedia.org/wiki/Digital_transformation">digitization</a> of two or three industry sectors (<a href="http://www.connectedfuturesmag.com/Research_Analysis/docs/OilGasDigitalTransformationWhitePaper.pdf">Oil and Gas</a>, Fisheries, Arctic Environments). Newfoundland, if it put its mind too it, could be at the table in a big way for the digitization of any of these three industry sectors.</li>
<li>Our current federal government is <a href="https://www.ic.gc.ca/eic/site/062.nsf/eng/h_00009.html">committed to encouraging innovation</a> and in supporting collaborative initiatives that fall well within the digitization of oceans and ocean technology.</li>
</ul>
<div>
We all agreed that participating in the local technology community is important, and we get enjoyment from sharing our technical experiences and understanding other technology project current within St. John's. Having a strong and growing technology architecture community would help in building the local technology economy and opportunities.</div>
<br />
<br />Peterhttp://www.blogger.com/profile/01204894304194832459noreply@blogger.comtag:blogger.com,1999:blog-24990866.post-11787485176204083622017-04-02T15:00:00.000-02:302017-04-02T17:56:55.668-02:30A focused economic sectorRecently I have been working toward growing the <a href="https://www.linkedin.com/groups/8593739">Atlantic Canada Association of Information Technology Architects</a> (ACAITA). The idea of creating this group got a lot of support and quickly had 90 members representing all four Atlantic Canadian provinces. A few weeks back a group of us got together in Halifax to discuss <a href="http://criticaltechnology.blogspot.ca/2017/03/acaita-inaugural-meeting-halifax.html">the association, its purpose, its road ahead, and other things architectural</a>. I believe for this association to be successful it needs to bring value to its members and to the business communities in which it exists.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigPs2dbzn5gqPQHTkuZv-JkpmCRgSufZa2vamDu2yrrx_ZkXtQgzF4dHCkJTkbXXC6rJXwQriFQ7VYuWgV9weLWxGAXFx5a4inKOjU63yc0-abYePtddOuHFX1gdfNgR3KbzuKiw/s1600/EcosystemModeling.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="336" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigPs2dbzn5gqPQHTkuZv-JkpmCRgSufZa2vamDu2yrrx_ZkXtQgzF4dHCkJTkbXXC6rJXwQriFQ7VYuWgV9weLWxGAXFx5a4inKOjU63yc0-abYePtddOuHFX1gdfNgR3KbzuKiw/s400/EcosystemModeling.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Differing <a href="https://timreview.ca/article/919">ecosystems should work together</a> to progress a focused economic sector.</td></tr>
</tbody></table>
Over the last few weeks I have had informal conversations with a number of intelligent and supportive people who work for <a href="http://www.acoa-apeca.gc.ca/Eng/Pages/Home.aspx">ACOA</a>, <a href="http://www.nati.net/">NATI</a>, and Industry.<i> I want to provide many thanks to these organizations and their representatives for taking the time and providing insight into how best to build the ACAITA</i>. The subjects we discussed are focused upon growing the association and how best to engage the business community. I also believe that my knowledge as an EA combined with recent research activities around <a href="https://en.wikipedia.org/wiki/Business_ecosystem">business ecosystem</a> modeling and <a href="https://en.wikipedia.org/wiki/Reference_architecture">reference architectures</a> had an influence over the directions these conversations took. Described below are the highlights to these conversations;<br />
<ol>
<li><b>ACOA - Atlantic Canada Opportunities Agency</b><br />
<ul>
<li>The ACAITA needs to define itself and have a comprehensive set of demographic data. Not only we need to know how many members we have and which province they are located, we need to know the total number of architects in the Atlantic provinces, where their focus is, what industries they work, etc.</li>
<li>Reach out to industry / business and find out exactly what their architectural needs are and where they see gaps in the workforce or capability.</li>
<li>The Oil and Gas sector remains strong for Newfoundland and Labrador and providing architectural support here should be considered a pillar for ACAITA. Becoming champions of the Oil and Gas Reference Architecture could be one of the associations cornerstones.</li>
<li>Keeping things Atlantic would support the ACAITA mission. Identify all the main business ecosystems and reference architectures where Atlantic Canada could become internationally recognized would be a solid approach. </li>
</ul>
</li>
<li><b>NATI - Newfoundland and Labrador Association of Technology Industries</b><br />
<ul>
<li>Having ACAITA as a pan-Atlantic organization remains as a very good idea. And looking for NATI equivalents in all the other three provinces would help in getting ACAITA support.</li>
<li>From a Technology Industries perspective the Digitization of Oil and Gas is capturing increasing attention and the investments made here should have derived benefit for Atlantic Canada outside Oil and Gas for the near and long terms.</li>
<li>NL already has many technology companies within the Oil and Gas sector and identifying all the organizational ecosystems and intrinsic reference architectures would help support ACAITA success and growth.</li>
<li>Its important that NL grows technology and digitization capabilities outside of Oil and Gas. And an Association like ACAITA would fit well with growing this derived capabilities perspective.</li>
</ul>
</li>
<li><b>Industry - conversations with assorted industry professionals</b><br />
<ul>
<li>Confirmed the focus of Oil and Gas for NL and that growing the architectural capabilities supporting these industries in the province of NL would assist greatly.</li>
<li>Mapping out the ecosystems (business, innovation, and knowledge) supported by the intrinsic reference architectures could further pull everything together, It could provide a technology foundation in which to build the <a href="http://reports.weforum.org/digital-transformation/wp-content/blogs.dir/94/mp/files/pages/files/dti-oil-and-gas-industry-white-paper.pdf">Digitization of Oil and Gas</a>. An expertise which NL should own internationally.</li>
</ul>
</li>
</ol>
<span style="font-size: large;">The Conclusion</span><br />
Identify the main industry sectors with Atlantic Canada and begin ecosystem mapping with an eye to defining the intrinsic reference architectures. Engage both IT Architects and organizations (business, innovation, and knowledge) to define and publish documents describing ecosystems and reference architectures.<br />
<br />Peterhttp://www.blogger.com/profile/01204894304194832459noreply@blogger.comtag:blogger.com,1999:blog-24990866.post-11966742194762016112017-03-22T06:26:00.001-02:302017-03-22T06:26:22.604-02:30ACAITA Partnership RecommendationThere are a number of groups, professional associations, organizations involved with developing and forwarding the practice of IT architecture. The ones I see relevant to developing an IT Architecture association are as follows;<br />
<ol>
<li><a href="http://www.iasaglobal.org/">IASA</a> - An association for all IT Architects</li>
<li><a href="http://www.opengroup.org/">OpenGroup</a> - Vendor neutral IT standards and certifications</li>
<li><a href="https://www.globalaea.org/">AEA</a> - Association of Enterprise Architects</li>
<li><a href="http://www.eacoe.org/">EACOE</a> - Enterprise Architecture Centre of Excellence</li>
<li><a href="https://feacinstitute.org/index.php">FEAC</a> - Training and Certification Institution for Enterprise Architects</li>
<li><a href="https://www.isc2.org/">ISC</a> - Vendor-neutral education products, career services, and Gold Standard credentials to professionals.</li>
<li><a href="http://www.cips.ca/">CIPS</a> - Canada's association of Information Technology (IT) professionals</li>
<li><a href="http://businessarchitectureguild.site-ym.com/">BAG</a> - Business Architecture Guild </li>
</ol>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhkDnbJPy5JKK-wwyV5nTlH7lRxR2XUi0vpW4cI4JOboLxdgOT7D7SBrGVyus8Lg5CatYYCEWJfGyHWVH3sKQgbsZGAdbYym0NAtQOshWSCT9M8NGpeOlvz0fMVVjJxPHTJ_6pJnA/s1600/Iasa_Logo_with_text.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="128" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhkDnbJPy5JKK-wwyV5nTlH7lRxR2XUi0vpW4cI4JOboLxdgOT7D7SBrGVyus8Lg5CatYYCEWJfGyHWVH3sKQgbsZGAdbYym0NAtQOshWSCT9M8NGpeOlvz0fMVVjJxPHTJ_6pJnA/s400/Iasa_Logo_with_text.jpg" width="400" /></a></div>
<div>
<br /></div>
<div>
I believe any of these have valuable resources that could support, and be useful to, the ACAITA membership. I believe the <a href="http://iasaglobal.org/">IASA </a>provides the broadest view into architecture, where the other organizations are more focused on an area of architecture, such as; security, business, the enterprise, or IT in general.</div>
<div>
<br /></div>
<div>
I recommend the ACAITA aligns itself with two or three of these groups / associations as determined by the resources they can make available toward growing the architectural capabilities of Atlantic Canadians. To start, I believe a few of us should consider becoming full members of the IASA Canada chapter with the intention to form the Atlantic Canada chapter. I also recommend we align ourselves with one or two of the training and certification groups... I believe this can wait until we have our association alignment.</div>
Peterhttp://www.blogger.com/profile/01204894304194832459noreply@blogger.com