Ocean Sector Specific Search

I've recently finished building an industry specific search engine. The primary use case is to drive international and domestic business traffic to the Canadian websites doing business within the oceans technology and innovation sectors.

From a technology architecture perspective we built a search engine for the Canadian oceans super cluster initiative where all components run, and are based, upon Canadian assets hosted in Canada. We seeded the search engine using the URLs for all the organizations identified as participants within this economic sector. The indexing process analysed each URL and followed all links up to two hops deep. All the identified URLs were scored using a web graph and the top pages were indexed.

The architecture decisions

The NELK stack became our back-end infrastructure.

A number of important architecture decisions were made early on as the project was detailed. Mostly decisions were made to support the technologies that the small team was already familiar. And if the team wasn't familiar, we chose technologies that had the most industry support and local resources in our personal networks so we could help out if we needed. We ended up having Nutch feeding the ELK stack using Wordpress for the UX. In the project it became known as the NELK stack.

• Nutch - for web crawling and first round of web page extraction and cleanse.
• ElasticSearch (ES) - as the search engine / data manager
• Logstash - as the data transform and load.
• Kibana - as the administration / developer console

Crawling the web with Nutch
We ended up using Nutch to crawl the internet for ocean sector specific web pages. We also needed to integrate with ElasticPress so the broader ecosystem search included the contents of our websites Wordpress database. To do all this we ended up using Nutch 1.15 for it integrated best across our technology stack. We used the Nutch recommended approach seeding, ingesting, fetching, and duplicate removing as we prepared the data for export to ElasticSearch. Due to versioning issues we exported the Nutch database to CSV before importing the data. For the first load of data our use of Nutch created the following page loading metrics;

• seeded with 2612 domain names
• removed 709 duplicate or in error domain names
• identified 86872 candidate webpages 
• fetched the 29323 most relevant web pages (based upon web graph algorithms)
• indexed 29270 pages into ElasticSearch

Loading data with Logstash, inspecting the results in Kibana
We used Logstash to bring the Nutch exported CSV data into ElasticSearch. Coding up the logstash job was fairly easy, the most important aspect was choosing the correct logstash filter. It was better to use the dissect filter rather than the csv filter. More on this in a later post. In the end, I was amazed with how quickly Logstash loaded, and ElasticSearch indexed, all the data.

Once all the data was loaded into ElasticSearch I used Kibana to confirm data was correctly loaded into the data repository. Kibana has a very intuitive interface and creating filters and running queries to confirm the successful loading of data was straight forward. I look forward to using Kibana to manage the repository and create meaningful dashboards.

Integrating ElasticSearch with WordPress PhP

Integrating with Wordpress

Once we had the back-end built and loaded with industry specific web pages we still needed to find the correct tool-set to provide a query interface within a Wordpress template and to integrate with the organizations identified in the Wordpress database. We wanted the ecosystem search to include both what we had indexed from the internet and the organizations listed in our directories database. The solution ended up using two solutions;

• The ElasticSearch (ES) PhP library which provides a mature (and easy to use) set of features to build your own interface into ES using PhP.
• ElasticPress which allows automated ElasticSearch integration with a wordpress database.

The Wordpress / PhP tools for integrating with ES are very effective. ElasticPress has automation that keeps it up to date as changes are made within the Wordpress database. The ES PhP library has a robust set of features that makes for easy coding and kept search performance very high. Even with large query results the ability to traverse the result set forward and back was easily handles with features available in the PhP library.

In conclusion, using Nutch with the ELK stack provides for a very strong search engine that integrates easily with Wordpress on the front-end. The learning curve for this approach was not overwhelming and whenever challenges presented themselves the online groups help us resolve issues within days.

Special thanks to the team put together by Triware Technologies. Without all the other technical people, analysts, business people, data entry, project managers, Oceans Advance, ACOA, Ocean Super Cluster, ElasticSearch support, Azure support, and those clearing the way... none of this would have been possible. Thank-you!

A plethora of end points

There is a growing number of data collection devices available to the digitization of everything (including oceans). The variety of devices and sensors includes everything from temperature through chemicals to acceleration. Combine the number of different sensors with the ability to transfer data over great distances and the ability to monitor even the most remote places for obscure data points is increasingly easy and affordable. The following list provides an overview of the types of devices and sensors available.

Internet of Things (IoT) Sensor Classification from Black Box Paradox.

• Position / Presence / Proximity

  

  Presence Sensor

1. Proximity Sensor - A proximity sensor is a sensor able to detect the presence of nearby objects without any physical contact.
2. Position Sensor - A position sensor is any device that permits position measurement.
3. Presence (or Occupancy) Sensor - An occupancy sensor is a motion detecting devices used to detect the presence of a person or object.

• Motion / Velocity / Displacement

  

  Displacement Sensor

1. Motion Sensor - A motion detector is a device that detects moving objects. Such a device is often integrated as a component of a system that automatically performs a task or alerts a user of motion in an area.
2. Velocity Sensor - A velocity receiver (velocity sensor) is a sensor that responds to velocity rather than absolute position.
3. Displacement Sensor - A displacement sensor (displacement gauge) is used to measure travel range between where an object is and a reference position. Displacement sensors can be used for dimension measurement to determine an object's height, thickness, and width in addition to travel range.

• Temperature

  

  Temperature Sensor

The temperature sensor detects the current temperature or changes in temperature.  There is a large number of temperature sensors available and a comprehensive list is available on Wikipedia.

• Humidity / Moisture

  

  Humidity Sensor

1. Humidity Sensor - A humidity sensor (or hygrometer) senses, measures and reports the relative humidity in the air. It therefore measures both moisture and air temperature. 
2. Moisture sensor - A moisture sensor is an instrument used for measuring the water vapor in the atmosphere. Sometime considered same device as humidity sensor.

• Acoustic / Sound / Vibration

  

  Acoustic Sensor

1. Acoustic Sensor - Surface acoustic wave sensors are a class of microelectromechanical systems (MEMS) which rely on the modulation of surface acoustic waves to sense a physical phenomenon.
2. Sound sensor - Sound Sensor can detect the sound intensity of the environment. The Sound Detector is a small board that combines a microphone and some processing circuitry. It provides not only an audio output, but also a binary indication of the presence of sound, and an analog representation of it's amplitude.
3. Vibration sensor - a vibration sensor can detect vibrations. A transducer, such as that incorporating a laser or a piezoelectric crystal, which converts vibrations into an electrical equivalent such as a voltage. Also called vibration transducer, or vibration pickup.

• Chemical / Gas

  

  Gas Sensor

1. Chemical Sensor - A chemical sensor is a self-contained analytical device that can provide information about the chemical composition of its environment, that is, a liquid or a gas phase. 
2. Gas sensor - A gas detector is a device that detects the presence of gases in an area, often as part of a safety system. This type of equipment is used to detect a gas leak or other emissions and can interface with a control system so a process can be automatically shut down.

• Flow
  

Flow Sensors monitor liquid flow rates and accumulated flow volume. Flow measurement is the quantification of bulk fluid movement and can be measured in a variety of ways.

• Force / Load / Torque / Strain / Pressure

1. Force Sensor - A Force Sensor is defined as a transducer that converts an input mechanical force into an electrical output signal. Force Sensors are also commonly known as Force Transducers.
   
2. Load Sensor - A Load Sensor is defined as a transducer that converts an input mechanical force into an electrical output signal. Load Sensors are also commonly known as Load Transducers or Load Cells.
3. Torque Sensor - A torque sensor, torque transducer or torque meter is a device for measuring and recording the torque on a rotating system, such as an engine, crankshaft, gearbox, transmission, rotor, a bicycle crank or cap torque tester. Static torque is relatively easy to measure.
4. Strain Sensor - A Strain gage (sometimes referred to as a Strain Gauge) is a sensor whose resistance varies with applied force; It converts force, pressure, tension, weight, etc., into a change in electrical resistance which can then be measured.
5. Pressure - A pressure sensor is a device for pressure measurement of gases or liquids. Pressure is an expression of the force required to stop a fluid from expanding, and is usually stated in terms of force per unit area. A pressure sensor usually acts as a transducer; it generates a signal as a function of the pressure imposed.

• Leaks / Levels
  

1. Leak Sensor - leak detection is used to determine if and in some cases where a leak has occurred in systems which contain liquids and gases.
2. Level Sensor - Level sensors detect the level of liquids and other fluids and fluidized solids, including slurries, granular materials, and powders that exhibit an upper free surface.

• Electric / Magnetic
  

1. Electric Sensor - A current sensor is a device that detects electric current in a wire, and generates a signal proportional to that current. The generated signal could be analog voltage or current or even a digital output. The generated signal can be then used to display the measured current in an ammeter, or can be stored for further analysis in a data acquisition system, or can be used for the purpose of control.
2. Magnetic Sensor - A MEMS magnetic field sensor is a small-scale microelectromechanical systems (MEMS) device for detecting and measuring magnetic fields (Magnetometer).

• Acceleration / Tilt
  

1. Acceleration Sensor - An accelerometer is a device that measures proper acceleration. Proper acceleration, being the acceleration (or rate of change of velocity) of a body in its own instantaneous rest frame, is not the same as coordinate acceleration, being the acceleration in a fixed coordinate system.
2. Tilt Sensor - A clinometer or inclinometer is an instrument for measuring angles of slope (or tilt), elevation or depression of an object with respect to gravity. It is also known as a tilt indicator, tilt sensor, tilt meter, slope alert, slope gauge, gradient meter, gradiometer, level gauge, level meter, declinometer, and pitch & roll indicator.

• Machine Vision / Optical / Ambient Light
  

1. Machine Vision Sensor - As a simple concept, machine vision is the use of devices for optical non-contact sensing to automatically receive and interpret an image of a real scene in order to obtain information and/or control machines or processes. Machine vision (MV) is the technology and methods used to provide imaging-based automatic inspection and analysis for such applications as automatic inspection, process control, and robot guidance, usually in industry. Machine vision is a term encompassing a large number of technologies, software and hardware products, integrated systems, actions, methods and expertise. Machine vision as a systems engineering discipline can be considered distinct from computer vision, a form of computer science.
2. Optical Sensor - Electro-optical sensors are electronic detectors that convert light, or a change in light, into an electronic signal.
3. Ambient Light Sensor - A device that detects the amount of light in the vicinity. An ambient light sensor may be built into a smartphone or tablet to adjust the screen brightness based on the available light.

Security matters!
It is also very important to note that as sensors become increasingly available and controlled over the network, security should be of huge concern. Industrial controllers are becoming increasingly targeted for security vulnerabilities and if an oceans sensor is available from a remote location over a network it is potentially open to attack. Being aware of the data available from the sensor and any control features the sensor (activator, controller) may have.

The other end
When considering the digitization of oceans reference architecture and what is considered an end-point we need to also look to the other end. And by the other end, I mean the data storage end. In this post we have discussed all the end points that emit the data, and at some point the data will need to be "at rest" stored in some storage device. The topic of data storage will be discussed in a later post.

Some examples of end-points

• Nomad - The AXYS NOMAD is a unique aluminum environmental monitoring buoy designed for deployments in extreme conditions. The NOMAD (Navy Oceanographic Meteorological Automatic Device) is a modified version of the 6m hull originally designed in the 1940’s for the U.S. Navy’s offshore data collection program. It has been operating in Canada’s Weather Buoy network for over 25 years and commonly experiences winter storms and hurricanes with wave heights approaching 20m.

• Coast Guard Canada - it is very hard to imagine this autonomous vessel will not be loaded with sensors to collect data. Portsmouth, UK, based ASV Global has converted a 26ft hydrographic survey launch to enable it to operate autonomously using the ASView control system, while maintaining its ability to operate in a conventional manned mode. The launch, which is part of the Canadian Coast Guard’s fleet dedicated to the survey operations of the Canadian Hydrographic Service, will be used as a test platform for unmanned survey work.

• Personal Weather Stations - A personal weather station is a set of weather measuring instruments operated by a private individual, club, association, or even business (where obtaining and distributing weather data is not a part of the entity's business operation). Personal weather stations have become more advanced and can include many different sensors to measure weather conditions. These sensors can vary between models but most measure wind speed, wind direction, outdoor and indoor temperatures, outdoor and indoor humidity, barometric pressure, rainfall, and finally UV or solar radiation.

Over the next few months I will be publishing a series of blog posts describing, in more detail, all the aspects for building a successful digitization of oceans reference architecture. Next up is; "communications" with focus on the data communications available to oceans technology. Please follow along and make comment. For a table of contents of these coming posts please review a companion post; Digitization of Oceans Reference Architecture TOC

three posts for digitization of oceans reference architecture

The next three posts within my series describing the need for a digitization of oceans reference architecture will be focused on the three technology domains of; end-points, communications, and data stores. This separation is to allow a deeper look into each domain as they have different considerations in relation to technology architecture and attributes important to the digitization of oceans.

End Points: the sensors and devices which collect and emit data. Consider this the Internet of Things (IoT) that can be located anywhere within and around oceans, airborne, surface, and submersible.

https://en.wikipedia.org/wiki/User:Peterrawsthorne/Digitization_of_Oceans#End_Points

Communications: the communications technologies available to transfer data from one place to another. A lot to explore within this domain; as underwater data transmission is an emerging technology, and the structure of the data messages will become the foundation of the reference architecture.

https://en.wikipedia.org/wiki/User:Peterrawsthorne/Digitization_of_Oceans#Communications

Data Stores: there are many existing data storage approaches, locations, and structures that can be used to store the oceans data. Databases and database designs are already available for many of the subjects within the digitization of oceans. Better to use exiting methods to store the structured and unstructured data and use a federated approach to bring them together.

https://en.wikipedia.org/wiki/User:Peterrawsthorne/Digitization_of_Oceans#Data_Stores

Creating an inventory
The number of technologies, vendors, standards, and approaches within these three domains will be large and forever growing. To start documenting this inventory I have created a Wikipedia page under my Wikipedia account. Once this page describing the Digitization of Oceans and its Reference Architecture is more complete I will submit it as a published article otherwise consider it a work in progress. Feel free to join in and edit the work in progress wiki page;

https://en.wikipedia.org/wiki/User:Peterrawsthorne/Digitization_of_Oceans#Lists

A focused economic sector

Recently I have been working toward growing the Atlantic Canada Association of Information Technology Architects (ACAITA). The idea of creating this group got a lot of support and quickly had 90 members representing all four Atlantic Canadian provinces. A few weeks back a group of us got together in Halifax to discuss the association, its purpose, its road ahead, and other things architectural. I believe for this association to be successful it needs to bring value to its members and to the business communities in which it exists.

Differing ecosystems should work together to progress a focused economic sector.

Over the last few weeks I have had informal conversations with a number of intelligent and supportive people who work for ACOA, NATI, and Industry. I want to provide many thanks to these organizations and their representatives for taking the time and providing insight into how best to build the ACAITA. The subjects we discussed are focused upon growing the association and how best to engage the business community. I also believe that my knowledge as an EA combined with recent research activities around business ecosystem modeling and reference architectures had an influence over the directions these conversations took. Described below are the highlights to these conversations;

1. ACOA - Atlantic Canada Opportunities Agency
   
   • The ACAITA needs to define itself and have a comprehensive set of demographic data. Not only we need to know how many members we have and which province they are located, we need to know the total number of architects in the Atlantic provinces, where their focus is, what industries they work, etc.
   • Reach out to industry / business and find out exactly what their architectural needs are and where they see gaps in the workforce or capability.
   • The Oil and Gas sector remains strong for Newfoundland and Labrador and providing architectural support here should be considered a pillar for ACAITA. Becoming champions of the Oil and Gas Reference Architecture could be one of the associations cornerstones.
   • Keeping things Atlantic would support the ACAITA mission. Identify all the main business ecosystems and reference architectures where Atlantic Canada could become internationally recognized would be a solid approach. 
2. NATI - Newfoundland and Labrador Association of Technology Industries
   
   • Having ACAITA as a pan-Atlantic organization remains as a very good idea. And looking for NATI equivalents in all the other three provinces would help in getting ACAITA support.
   • From a Technology Industries perspective the Digitization of Oil and Gas is capturing increasing attention and the investments made here should have derived benefit for Atlantic Canada outside Oil and Gas for the near and long terms.
   • NL already has many technology companies within the Oil and Gas sector and identifying all the organizational ecosystems and intrinsic reference architectures would help support ACAITA success and growth.
   • Its important that NL grows technology and digitization capabilities outside of Oil and Gas. And an Association like ACAITA would fit well with growing this derived capabilities perspective.
3. Industry - conversations with assorted industry professionals
   
   • Confirmed the focus of Oil and Gas for NL and that growing the architectural capabilities supporting these industries in the province of NL would assist greatly.
   • Mapping out the ecosystems (business, innovation, and knowledge) supported by the intrinsic reference architectures could further pull everything together, It could provide a technology foundation in which to build the Digitization of Oil and Gas. An expertise which NL should own internationally.

The Conclusion
Identify the main industry sectors with Atlantic Canada and begin ecosystem mapping with an eye to defining the intrinsic reference architectures. Engage both IT Architects and organizations (business, innovation, and knowledge) to define and publish documents describing ecosystems and reference architectures.