Tuesday, November 26, 2019

Ocean Sector Specific Search

I've recently finished building an industry specific search engine. The primary use case is to drive international and domestic business traffic to the Canadian websites doing business within the oceans technology and innovation sectors.

From a technology architecture perspective we built a search engine for the Canadian oceans super cluster initiative where all components run, and are based, upon Canadian assets hosted in Canada. We seeded the search engine using the URLs for all the organizations identified as participants within this economic sector. The indexing process analysed each URL and followed all links up to two hops deep. All the identified URLs were scored using a web graph and the top pages were indexed.

The architecture decisions
The NELK stack became our back-end infrastructure.

A number of important architecture decisions were made early on as the project was detailed. Mostly decisions were made to support the technologies that the small team was already familiar. And if the team wasn't familiar, we chose technologies that had the most industry support and local resources in our personal networks so we could help out if we needed. We ended up having Nutch feeding the ELK stack using Wordpress for the UX. In the project it became known as the NELK stack.
  • Nutch - for web crawling and first round of web page extraction and cleanse.
  • ElasticSearch (ES) - as the search engine / data manager
  • Logstash - as the data transform and load.
  • Kibana - as the administration / developer console
Crawling the web with Nutch
We ended up using Nutch to crawl the internet for ocean sector specific web pages. We also needed to integrate with ElasticPress so the broader ecosystem search included the contents of our websites Wordpress database. To do all this we ended up using Nutch 1.15 for it integrated best across our technology stack. We used the Nutch recommended approach seeding, ingesting, fetching, and duplicate removing as we prepared the data for export to ElasticSearch. Due to versioning issues we exported the Nutch database to CSV before importing the data. For the first load of data our use of Nutch created the following page loading metrics;
  • seeded with 2612 domain names
  • removed 709 duplicate or in error domain names
  • identified 86872 candidate webpages 
  • fetched the 29323 most relevant web pages (based upon web graph algorithms)
  • indexed 29270 pages into ElasticSearch
Loading data with Logstash, inspecting the results in Kibana
We used Logstash to bring the Nutch exported CSV data into ElasticSearch. Coding up the logstash job was fairly easy, the most important aspect was choosing the correct logstash filter. It was better to use the dissect filter rather than the csv filter. More on this in a later post. In the end, I was amazed with how quickly Logstash loaded, and ElasticSearch indexed, all the data.

Once all the data was loaded into ElasticSearch I used Kibana to confirm data was correctly loaded into the data repository. Kibana has a very intuitive interface and creating filters and running queries to confirm the successful loading of data was straight forward. I look forward to using Kibana to manage the repository and create meaningful dashboards.

Integrating ElasticSearch with WordPress PhP

Integrating with Wordpress
Once we had the back-end built and loaded with industry specific web pages we still needed to find the correct tool-set to provide a query interface within a Wordpress template and to integrate with the organizations identified in the Wordpress database. We wanted the ecosystem search to include both what we had indexed from the internet and the organizations listed in our directories database. The solution ended up using two solutions;
  • The ElasticSearch (ES) PhP library which provides a mature (and easy to use) set of features to build your own interface into ES using PhP.
  • ElasticPress which allows automated ElasticSearch integration with a wordpress database.
The Wordpress / PhP tools for integrating with ES are very effective. ElasticPress has automation that keeps it up to date as changes are made within the Wordpress database. The ES PhP library has a robust set of features that makes for easy coding and kept search performance very high. Even with large query results the ability to traverse the result set forward and back was easily handles with features available in the PhP library.

In conclusion, using Nutch with the ELK stack provides for a very strong search engine that integrates easily with Wordpress on the front-end. The learning curve for this approach was not overwhelming and whenever challenges presented themselves the online groups help us resolve issues within days.

Special thanks to the team put together by Triware Technologies. Without all the other technical people, analysts, business people, data entry, project managers, Oceans Advance, ACOA, Ocean Super Cluster, ElasticSearch support, Azure support, and those clearing the way... none of this would have been possible. Thank-you!

Thursday, July 05, 2018

Seeking New Opportunities

Veteran technology professional seeking new opportunities. If you are requiring 35 years of career success in the software technology realm that spans startups through to large enterprise environments, then I'm your guy!

I have tonnes of experience and I can hit the ground running. I can work as a senior project manager, an enterprise solution architect, as a scrum master and build a team focused with agile approaches, I can design databases and related structures for your business analytic projects, and I can own your Business Intelligence initiative. I can provide strong, and proven, leadership. And if need be I can occupy the technical director level. You will find I provide great business benefit and good value, my experience provides the ability to save you more than I would cost.

A history of project success
I'm back to consulting in the technology realm. Being a technology consultant has provided my clients, and myself, many project successes. These successes have been both with technology startups and with medium to large business environments. For more details describing some of my project successes please read these two posts describing projects I have managed and provided technical leadership over the last decade;

Increasing Access to Education - this collection of eight projects helped CLEBC become one of the globes premier legal educators.
Career success in three year cycles - These four major corporate initiatives leveraged all my abilities to help PAL into their next level of technology capability and realized business value.
Where I can help the most
I can have an immediate positive impact to any technology initiative within large and small organizations. This success can come with startups and enterprise environments, or something new that can leverage my skills, knowledge, and experience.

There are five roles to where I can bring the greatest immediate business value;
  1. Enterprise and Solution Architecture - The solution side of designing technical architecture has most often been my preferred role within a software development initiative. I've done this for small startups and large enterprises. Good solution architecture creates a better technical solution that saves development costs, improves quality, and ensures alignment with business needs and the other technologies within the business ecosystem.
  2. Senior Project Management and/or Managing a Software Development Teams - Through time I have managed many projects with software development teams varying in size from 3 to 30. I have proven track record of bringing projects to completion on-time and on-budget. I seriously enjoy assisting a team of technical people to ship software.
  3. Data Analytics Project (Architecture and Team Management) - The B.Tech undergrad degree I completed in the 80's was focused on database management. And since that time my career has had data as its foundation. I do very well with data management [including the design of data structures and related communications (read API's)]. I can also work well within big data projects as my experience with big data goes back to the 1980's. Read one of my big data posts from a few years back to get a sense of this capability; Big Data; Similarities and Differences.
  4. Technical Mentorship - I'm an educator and have experienced many startups during my time in Vancouver. If you want to discuss the technical side, the team management side, and/or the development methodology to an innovation project (startup or otherwise) I can help here.
  5. Innovation - Many times in my career I have taken business strategy and made it happen. If you have a business strategy that requires software innovation I can make it tactical and implement. 
I'm willing to travel
A single flight from St. John's Newfoundland is my preference. I would also be very interested in consulting work in Vancouver (my home town). I have family there and could easily have all the amenities I require for extended working stays. In a previous blog post I describe how I consider the size of the St. John's marketplace to be 40 million people. I also need to consider the St. John's market to be a single flight away as I seek out new challenges and opportunities. My list of cities I will immediately consider consulting opportunities, include;
  1. Halifax: it's a morning flight away, and can be there and back including a full days work. I'd also stay for a few days if required.
  2. Toronto, Ottawa, Montreal: single 3 hour flight, couple of days then return home for remote working
  3. Vancouver: longer stays and longer aways... I have family in Vancouver, so it would be a very nice approach to working.
  4. London, England: An outlier, but... It would be nice to work in the EU and I have a British passport so it would ease my travel and work permitting.
One personal constraint
To be succinct, I require work-life balance... mostly it is about having time to care for middle-school age children and aging parents. This means I am available part-time or full-time with loads of schedule flexibility.

Wednesday, June 27, 2018

Work Life Balance for the Sandwiched CTO

Never more in my life have I required work-life balance and understand what it means at a personal level. This all started when I accepted a CTO position with Bluedrop Learning Networks ( a great organization with a good alignment to my career journey ). The interview process was extensive and I had the opportunity to meet all corners of the team I would be working with. Overall a good and thorough process. Kudos to everyone involved! The interview process included multiple discussions about work-life balance, and how I (in my mid-50's) have four aging parents over 80 years, two middle-school age children, one young adult age child, a bunch of volunteer activities, personal interests that I wont let go, and three residences to maintain. Oh yes, and a spouse who is an family doctor who also works in palliative care... needless to say I am often the go-to person when a short notice family issue arises. All this said, and after a few important conversations I had to let go of my new CTO role after only six months. I needed to focus on what I consider most important for my family. Though I am not an entrepreneur ( - though, I do think a good CTO needs to have the entrepreneur mindset - ) I completely agree with Randi Zuckerberg about approaches to managing the workload of a senior professional, or also known as the entrepreneur's dilemma.
For me, it's that simple. It's the implementation that will be the hard part. I need to have spending time with family, staying fit (in good health), and getting sleep to be my priorities. And I need to organize my life around these three and the others come in between or when priorities change. Something important to consider is that these five items need to be put into the larger context of a life well lived through a whole lifetime. That means you pick three on a daily, weekly, or yearly basis. Set your priorities for the day and/or week and follow through. This allows for flexibility within the entrepreneur's dilemma. In other words;
As you can see I also added one, be creative... my thinking around this aligns with having the time to think and create, free the brain to focus on the abstract and artistic and to wonder aimlessly... it is an important part, and keeps your edge in a world where innovation is becoming increasingly important.

Friday, January 19, 2018

Career success in three year cycles

My career seems to go in three year cycles. I've given up trying to determine if this is my unconscious doing or my skills and knowledge lend themselves very well to successful three year project life-cycles. Either way, I'm 35 years into a successful technology career and another three year cycle is coming to an end. And I'm very excited with what is coming! Before I get into the details of the most recent cycle I went back over my past 12 years of blogging to get a sense of my working ebbs and flows, and to confirm my three year cycle theory. The past four, three year cycles also seem to swing from being an employee to working on startups and as a consultant;

My most recent three year cycle has been with Provincial Aerospace (PAL) in St. John's Newfoundland. It has been a fantastic and exhausting three years. If I was to pick a few themes, they would be; team building, shipping software, quality management, and an acquisition. So what do I consider my achievements over the past three years and where do I best give thanks. In giving thanks, I will not be calling people out by name - they know who they are.
  1. Team Alignment
  2. When I joined PAL there was so much raw software engineering power with good team trust and camaraderie that my focus immediately moved from team building and skills development to team alignment and clearing the way for their success. And successful they were! We got to the point where every customer was very satisfied and we had solved some lingering and potentially expensive issues.
    Special Thanks: I want to give thanks to PAL senior management for being patient, keeping the team together, and trusting me to start shipping software. It took some time, we had to clean up some minor releases and complete a significantly complicated major release with a very broad deployment scope.
    • SED Team
    • What is most important here is the team was solid within itself as a software engineering team. They had very good end-to-end practices and audit proof trace-ability. The organization as a whole struggled with getting software out to the production environments due to many factors. Keeping in mind deployment is beyond the responsibility of the software engineering team and we had to deploy into highly secure customer networks in very rugged environments. Once we had a realistic schedule the team held the vision for success and delivered.
      Special Thanks: I want to give thanks to our project manager who is insanely detailed in the best of ways and the team for not only building quality code, but taking on the quality assurance roles for each others work. 
    • Business Intelligence Team
    • Innovation within a strong corporate culture can be hard and an organization will struggle to understand a team with a different rhythm and dance step. And the pressure for the team to align itself back with the corporate rhythm will be ever present and an effort to avoid. If you want innovation you need to allow teams to dance a different step, and allow them to master their new dance. We built a scrum - agile data analytics / business intelligence team from an exceptional group of transactional developers used to a more waterfall type development approach. It took us 12 months (which I consider a short time) to have a performing group of analytics and intelligence developers who ship new data dimensions in a sprint cycle time of 4 - 6 weeks.
      Special Thanks: I want to give thanks to the whole team for deeply embracing the learning toward becoming knowledgeable analytics developers. With the ability to analyse, design, develop, and deploy for multidimensional data from a plethora of different data sources. I want to thank the scrum master for holding the vision toward the value scrum would bring. I want to give thanks to the business intelligence consultant we brought in for the first six months of the project, they did an excellent job of leading by example and transferring the knowledge required to get the team more than started. I again want to give thanks to PAL senior management for being patient, keeping the team together, and trusting that the data dimensions will come and the team would exceed the targeted number of KPI's.

  3. Shipping Software
  4. Shipping software can be really hard or it can be simple and repeatable. Shipping and deploying software has become an understood and well managed process as the practices have built toward Agile approaches and DevOps over the last 10 - 15 years. The time of failed non-deployed projects should be a thing of the past. If its not, you should either; clear house at the senior leadership level or send everyone to upgrade their literacy of how to provide leadership to software and technology development.
    Special Thanks: I want to give thanks to the team leadership who came before me. The use of Microsoft TFS within the SDLC made team alignment really easy. I also want to give thanks to the team for adapting so well to adopting more agile and devops type approaches. I may have cleared the way, but the team did the real work!
    • IDNS 2.x (ADAM8 and SIS)
    • This was an eighteen month project that included many major features within its release scope. The main considerations for the project were;
      1. the schedule restraints [we could NOT go-live to production during iceberg-season (Feb - June)]
      2. the breadth of the feature set we were delivering included;
        • Increased security constraints
        • Service Orientation
        • Directory Integration
        • Content Management
        • Geo-referencing with advanced search
        • Off-shore client deployment
        • Legacy application support
      3. deploying to a new data center and multi-server architecture.
      4. Having two versions run in parallel for an ice-season to allow for comprehensive testing and customer network / security integration.
      Special Thanks: I want to give thanks to everyone involved with this project to hold the vision and getting to finished. We exceeded expectations. I'd also like to give thanks to the IT department for building out the infrastructure, owning the active directory deployment, and being sure we are secure to the levels required by our customers.
    • Business Intelligence - Data Analytics
    • We started to ship software as releases of new data dimensions (or clusters of key performance indicators). Once we had the basic infrastructure in place with the beginnings of the Extract, Cleanse, Transform, and Load (ECTL), data marts and warehouse we moved into using a scrum type approach to ship new dimensions out of each sprint. This worked well as it gave a reasonable velocity to complete a set of deliverables and then have retrospectives for learning. The team shipped 12 cubes with consistent and interval data refresh which took data from three distinct and distributed data sources. The challenge going forward for the team is in assisting the product owners and subject matter experts in leveraging all the cubes into dashboards and analytics.
      Special Thanks: I want to give thanks to the team for putting in the extra effort from being on a steep learning curve and focusing on the delivery of data cubes as our measure for success. I want to recognize the commitment of our senior developer in being a learning machine of the business intelligence and  data analytics subject domain. I want to give thanks for our DBA taking on all the system and database level tasks to make the project a success.

  5. Quality Management
  6. I'm a strong believer that good software quality management (QMS) brings significant business value. The value comes from; being able to ship software on schedule and on budget, more easily integrating new and unexpected features, ease in adding and training new team members into well known process', and the ability to easily address the scrutiny from customers, auditors, and potential investors. Our team regularly exceeded the procedures and practices within our quality management system and much of our success was due to this rigor.
    Special Thanks: I want to give thanks to the PAL director of quality, she saw beyond the business of quality certification and her enthusiasm for quality made us all better. I want to again give thanks to our project manager; her commitment to process, record keeping, and traceability is a thing to behold.
    • Rock solid processes
    • We were fortunate to have developed an end-to-end SDLC process that was reflected in our use of Microsoft TFS. The team was committed to following the process' and we passed all our insanely rigorous ISO audits. We were continuously improving our process' to better suit a more effective and efficient SDLC. We adjusted our QMS to our practices, rather than adjust our practices to our QMS. And our adjustment followed solid change management practices.
      Special Thanks: I want to give thanks to the software engineering groups leadership team for being willing to continuously improve our SDLC as it was reflected within our QMS, and vice-versa. I want to give thanks to the team for following our practices. I want to give thanks to the auditor for providing candid feedback and encouragement to make change within our ISO certification.
    • Traceability
    • Having the ability to see an idea through to working software is an accomplishment. Having evidence of how the idea became working software, from an auditors perspective, is a thing of beauty.
    • Multiple Successful ISO audits 
    • Special Thanks: I want to give thanks to all those who participated in our ISO audits. They can be stressful, but for us we embraced them and used them for our continuous improvement.

  7. An Acquisition
  8. I feel very fortunate to have had an amazing 35 years working within the technology realm. I've written a lot of code, tuned my share of databases, managed a number of talented teams, architected working solutions that are used daily across Canada and around the world, and held leadership roles with amazing peers. I have had my share of working with startups as an employee, shareholder, and outsider performing technical due diligence. My 30 years in the Vancouver technology scene included working with angel investors and VC's to perform technical due diligence to provide them much needed information. During my time at Provincial Aerospace I was asked to again perform technical due diligence for the PAL acquisition of CarteNav. What made this different is it the first time I did it as an employee of the acquiring company. And fortunately I became a member of CarteNav senior team after the acquisition. This allowed me to see firsthand some of the success and challenges that happen during the first 18 months after an acquisition. Such a great professional experience.
    Special Thanks: I want to give thanks to senior management for giving me the opportunity to see an acquisition from the other side. It confirms that acquisitions success are about cultural integration during the months that follow. I also want to give thanks to the top tier consulting firm that confirmed my due diligence findings, and were very focused and gracious. I want to give thanks to the CarteNav leadership team (and all the CarteNav employees) for being so welcoming and working so hard through the challenges that come with any acquisition.
So there you have it, a reasonable summary of the standout themes from my last three years working with Provincial Aerospace. My next adventure begins... more on this to come. Be well...

Tuesday, January 02, 2018

The oceans data collection platforms

My last post from the end of 2017 spoke to the number of sensors and devices that are currently, and becoming easily, available for data collection. This previous post focused more on the generic types of end-points and sensor technology. In this post we focus on the oceans and look at the variety of end-points used in the maritime environment.

Underwater Sensors
As previously identified there are a growing number of sensors and devices available for collecting data in all types of environments. This is also true for the growing number of "platforms" in which these sensors can be deployed. When we extend our view of sensors to include those used in oceans the list of data capture technologies grows. What is important, beyond the details of the data being emitted by these sensors, is how many are becoming a part of the digitization of oceans data "ecosystem". And with so many sensors, and therefore, so much data... a standardized approach and reference architecture defining the approaches to coalesce all this data will become increasingly important.

The oceans data collection "platforms"

Weather buoys
Weather buoys are instruments which collect weather and ocean data within the world's oceans, as well as aid during emergency response to chemical spills, legal proceedings, and engineering design. Moored buoys have been in use since 1951, while drifting buoys have been used since 1979.

An Integrated Ocean Data System 
Spotter is a web-integrated solution for collecting ocean wave and surface current data, designed from the ground up to be easy to use, intuitive, and extremely low cost. The Spotter Device is a compact, solar-powered, surface-follower, which measures surface waves and currents. Through our online Dashboard you can remotely configure your Spotters, access your data in real time, visualize wave data and position tracks. Your Spotter is already connected, so all you do is turn it on and focus on collecting the data you need.

Autonomous Underwater Vehicle (AUV, or unmanned robot submarine)
AUV are increasingly non-proprietary and made up of “plug-and-play” AUV modules which can be brought together and configured in the field. When assembled with a set of survey-grade sonar modules, a Gavia AUV becomes a self-contained survey solution with a low logistics footprint that is capable of carrying out a wide range of missions for commercial, defense, and scientific applications.

Autonomous Underwater Observatories
A submersible platform enabling tailor-made solutions with sensors from a variety of sources. They can be used off-the-shelf or customized to better cover application requirements. They have the ability for long-term deployments and the ability to collect and transmit data through a variety of methods. As an example, review the deep argo a submersible device for collecting data at extreme depths.

The list doesn't stop at what is described here. Reading and research on ocean sensors and the digitization of oceans will direct the reader toward the large number of devices collecting data in and around the ocean. Honestly, I'm amazed with the number of devices and the amount of data currently being collected about the oceans. I am increasingly of the belief the we need a reference architecture for the digitization of oceans... Actually, I'm surprised there isn't one already... for it would certainly help to bring together all the oceans data collection on a global scale for the good of us all. If you have any knowledge of an open reference architecture for the digitization of oceans please forward this information along. Thank-you!

Over the next few months I will be publishing a series of blog posts describing, in more detail, all the aspects for building a successful digitization of oceans reference architecture. Next up is; "communications" with focus on the data communications available to oceans technology. Please follow along and make comment. For a table of contents of these coming posts please review a companion post; Digitization of Oceans Reference Architecture TOC

Thursday, December 07, 2017

A plethora of end points

There is a growing number of data collection devices available to the digitization of everything (including oceans). The variety of devices and sensors includes everything from temperature through chemicals to acceleration. Combine the number of different sensors with the ability to transfer data over great distances and the ability to monitor even the most remote places for obscure data points is increasingly easy and affordable. The following list provides an overview of the types of devices and sensors available.

Internet of Things (IoT) Sensor Classification from Black Box Paradox.

  • Position / Presence / Proximity
    Presence Sensor
  1. Proximity Sensor - A proximity sensor is a sensor able to detect the presence of nearby objects without any physical contact.
  2. Position Sensor - A position sensor is any device that permits position measurement.
  3. Presence (or Occupancy) Sensor - An occupancy sensor is a motion detecting devices used to detect the presence of a person or object.
  • Motion / Velocity / Displacement
    Displacement Sensor
  1. Motion Sensor - A motion detector is a device that detects moving objects. Such a device is often integrated as a component of a system that automatically performs a task or alerts a user of motion in an area.
  2. Velocity Sensor - A velocity receiver (velocity sensor) is a sensor that responds to velocity rather than absolute position.
  3. Displacement Sensor - A displacement sensor (displacement gauge) is used to measure travel range between where an object is and a reference position. Displacement sensors can be used for dimension measurement to determine an object's height, thickness, and width in addition to travel range.
  • Temperature
    Temperature Sensor
The temperature sensor detects the current temperature or changes in temperature.  There is a large number of temperature sensors available and a comprehensive list is available on Wikipedia.

  • Humidity / Moisture
    Humidity Sensor
  1. Humidity Sensor - A humidity sensor (or hygrometer) senses, measures and reports the relative humidity in the air. It therefore measures both moisture and air temperature. 
  2. Moisture sensor - A moisture sensor is an instrument used for measuring the water vapor in the atmosphere. Sometime considered same device as humidity sensor.
  • Acoustic / Sound / Vibration
    Acoustic Sensor
  1. Acoustic Sensor - Surface acoustic wave sensors are a class of microelectromechanical systems (MEMS) which rely on the modulation of surface acoustic waves to sense a physical phenomenon.
  2. Sound sensor - Sound Sensor can detect the sound intensity of the environment. The Sound Detector is a small board that combines a microphone and some processing circuitry. It provides not only an audio output, but also a binary indication of the presence of sound, and an analog representation of it's amplitude.
  3. Vibration sensor - a vibration sensor can detect vibrations. A transducer, such as that incorporating a laser or a piezoelectric crystal, which converts vibrations into an electrical equivalent such as a voltage. Also called vibration transducer, or vibration pickup.
  • Chemical / Gas
    Gas Sensor
  1. Chemical Sensor - A chemical sensor is a self-contained analytical device that can provide information about the chemical composition of its environment, that is, a liquid or a gas phase. 
  2. Gas sensor - A gas detector is a device that detects the presence of gases in an area, often as part of a safety system. This type of equipment is used to detect a gas leak or other emissions and can interface with a control system so a process can be automatically shut down.
  • Flow
Flow Sensors monitor liquid flow rates and accumulated flow volume. Flow measurement is the quantification of bulk fluid movement and can be measured in a variety of ways.
  • Force / Load / Torque / Strain / Pressure
  1. Force Sensor - A Force Sensor is defined as a transducer that converts an input mechanical force into an electrical output signal. Force Sensors are also commonly known as Force Transducers.
  2. Load Sensor - A Load Sensor is defined as a transducer that converts an input mechanical force into an electrical output signal. Load Sensors are also commonly known as Load Transducers or Load Cells.
  3. Torque Sensor - A torque sensor, torque transducer or torque meter is a device for measuring and recording the torque on a rotating system, such as an engine, crankshaft, gearbox, transmission, rotor, a bicycle crank or cap torque tester. Static torque is relatively easy to measure.
  4. Strain Sensor - A Strain gage (sometimes referred to as a Strain Gauge) is a sensor whose resistance varies with applied force; It converts force, pressure, tension, weight, etc., into a change in electrical resistance which can then be measured.
  5. Pressure - A pressure sensor is a device for pressure measurement of gases or liquids. Pressure is an expression of the force required to stop a fluid from expanding, and is usually stated in terms of force per unit area. A pressure sensor usually acts as a transducer; it generates a signal as a function of the pressure imposed.
  • Leaks / Levels
  1. Leak Sensor - leak detection is used to determine if and in some cases where a leak has occurred in systems which contain liquids and gases.
  2. Level Sensor - Level sensors detect the level of liquids and other fluids and fluidized solids, including slurries, granular materials, and powders that exhibit an upper free surface.
  • Electric / Magnetic
  1. Electric Sensor - A current sensor is a device that detects electric current in a wire, and generates a signal proportional to that current. The generated signal could be analog voltage or current or even a digital output. The generated signal can be then used to display the measured current in an ammeter, or can be stored for further analysis in a data acquisition system, or can be used for the purpose of control.
  2. Magnetic Sensor - A MEMS magnetic field sensor is a small-scale microelectromechanical systems (MEMS) device for detecting and measuring magnetic fields (Magnetometer).
  • Acceleration / Tilt
  1. Acceleration Sensor - An accelerometer is a device that measures proper acceleration. Proper acceleration, being the acceleration (or rate of change of velocity) of a body in its own instantaneous rest frame, is not the same as coordinate acceleration, being the acceleration in a fixed coordinate system.
  2. Tilt Sensor - A clinometer or inclinometer is an instrument for measuring angles of slope (or tilt), elevation or depression of an object with respect to gravity. It is also known as a tilt indicator, tilt sensor, tilt meter, slope alert, slope gauge, gradient meter, gradiometer, level gauge, level meter, declinometer, and pitch & roll indicator.
  • Machine Vision / Optical / Ambient Light
  1. Machine Vision Sensor - As a simple concept, machine vision is the use of devices for optical non-contact sensing to automatically receive and interpret an image of a real scene in order to obtain information and/or control machines or processes. Machine vision (MV) is the technology and methods used to provide imaging-based automatic inspection and analysis for such applications as automatic inspection, process control, and robot guidance, usually in industry. Machine vision is a term encompassing a large number of technologies, software and hardware products, integrated systems, actions, methods and expertise. Machine vision as a systems engineering discipline can be considered distinct from computer vision, a form of computer science.
  2. Optical Sensor - Electro-optical sensors are electronic detectors that convert light, or a change in light, into an electronic signal.
  3. Ambient Light Sensor - A device that detects the amount of light in the vicinity. An ambient light sensor may be built into a smartphone or tablet to adjust the screen brightness based on the available light.
Security matters!
It is also very important to note that as sensors become increasingly available and controlled over the network, security should be of huge concern. Industrial controllers are becoming increasingly targeted for security vulnerabilities and if an oceans sensor is available from a remote location over a network it is potentially open to attack. Being aware of the data available from the sensor and any control features the sensor (activator, controller) may have.

The other end
When considering the digitization of oceans reference architecture and what is considered an end-point we need to also look to the other end. And by the other end, I mean the data storage end. In this post we have discussed all the end points that emit the data, and at some point the data will need to be "at rest" stored in some storage device. The topic of data storage will be discussed in a later post.

Some examples of end-points
  • Nomad - The AXYS NOMAD is a unique aluminum environmental monitoring buoy designed for deployments in extreme conditions. The NOMAD (Navy Oceanographic Meteorological Automatic Device) is a modified version of the 6m hull originally designed in the 1940’s for the U.S. Navy’s offshore data collection program. It has been operating in Canada’s Weather Buoy network for over 25 years and commonly experiences winter storms and hurricanes with wave heights approaching 20m.

  • Coast Guard Canada - it is very hard to imagine this autonomous vessel will not be loaded with sensors to collect data. Portsmouth, UK, based ASV Global has converted a 26ft hydrographic survey launch to enable it to operate autonomously using the ASView control system, while maintaining its ability to operate in a conventional manned mode. The launch, which is part of the Canadian Coast Guard’s fleet dedicated to the survey operations of the Canadian Hydrographic Service, will be used as a test platform for unmanned survey work.

  • Personal Weather Stations - A personal weather station is a set of weather measuring instruments operated by a private individual, club, association, or even business (where obtaining and distributing weather data is not a part of the entity's business operation). Personal weather stations have become more advanced and can include many different sensors to measure weather conditions. These sensors can vary between models but most measure wind speed, wind direction, outdoor and indoor temperatures, outdoor and indoor humidity, barometric pressure, rainfall, and finally UV or solar radiation.

Over the next few months I will be publishing a series of blog posts describing, in more detail, all the aspects for building a successful digitization of oceans reference architecture. Next up is; "communications" with focus on the data communications available to oceans technology. Please follow along and make comment. For a table of contents of these coming posts please review a companion post; Digitization of Oceans Reference Architecture TOC

Sunday, November 26, 2017

three posts for digitization of oceans reference architecture

The next three posts within my series describing the need for a digitization of oceans reference architecture will be focused on the three technology domains of; end-points, communications, and data stores. This separation is to allow a deeper look into each domain as they have different considerations in relation to technology architecture and attributes important to the digitization of oceans.

End Points: the sensors and devices which collect and emit data. Consider this the Internet of Things (IoT) that can be located anywhere within and around oceans, airborne, surface, and submersible.


Communications: the communications technologies available to transfer data from one place to another. A lot to explore within this domain; as underwater data transmission is an emerging technology, and the structure of the data messages will become the foundation of the reference architecture.


Data Stores: there are many existing data storage approaches, locations, and structures that can be used to store the oceans data. Databases and database designs are already available for many of the subjects within the digitization of oceans. Better to use exiting methods to store the structured and unstructured data and use a federated approach to bring them together.


Creating an inventory
The number of technologies, vendors, standards, and approaches within these three domains will be large and forever growing. To start documenting this inventory I have created a Wikipedia page under my Wikipedia account. Once this page describing the Digitization of Oceans and its Reference Architecture is more complete I will submit it as a published article otherwise consider it a work in progress. Feel free to join in and edit the work in progress wiki page;