ODE Data Lab has its technology footing

This month has become more about standing up technology than it has been talking to people about their ocean data needs. That's ok... if you are building a technology company, you need to build technology. Ocean of Data Endeavours (ODE) is about building and utilizing software towards making it super easy to work with data, large amounts of data.

The last 10 days have been about refreshing a server infrastructure I stood up 12 years ago for a number of other projects. What was left was a couple of simple websites, some domain hosting, and all the related mail server infrastructure. All of this needed a complete refresh to be brought up to date;

1. Rebuild the server infrastructure to have more horsepower. - DONE
2. Upgrade the Ubuntu OS from 10.04 (Lucid Lynx) to 20.04 (Focal Fossa). - DONE
3. Rework all the domain aliases to remove dependency on a domain I no longer owned. - DONE
4. Do some basic security work to the server. Mostly SSH focused. - DONE
5. Create a new mail server, and do some mailbox maintenance. - DONE
6. Install Apache2 httpd host. - DONE
7. Configure Apache2 for a couple of web sites. - DONE
8. Code some basic HTML to confirm the sites are working. - DONE
9. Celebrate! http://endeavours.com/

So good to have all this done. The server will provide a strong foundation and is well prepared for the ELK stack and the first load of ocean data. So excited! 

Target Technology Stack

To support my work in completing my Open and Networked PhD one of my major tasks will be developing software. My current thinking is I will be developing an internet portal to allow people to bring together their digital badges. I want to provide people a way to bring together all their validated learning accomplishments into one place for display and supporting analytics.

For software development I need to choose a technology stack, or the server software and programming language(s) I will use for development. This will be a LAMP stack for the following reasons;

1. I have already been developing on LAMP for a number of years. So any additional learning curve for this project will be shallow.
2. I see LAMP to be the technologies of;
   • Linux (or Ubuntu)
   • Apache (Web Server)
   • MongoDB (Database) - this replaces MySQL as it is now owned by Oracle and can no longer be considered non-proprietary.
   • PhP - used with HTML5 and CSS3 in an MVC pattern
3. I want to choose an open and popular platform currently used within the education technology space.This stack, with the exception of MongoDB, is used by WikiMedia, Moodle, Drupal, and Wordpress. Pretty much all the major players in the Open Education space.
4. I will host all this on rackspace... because I've used them for a number of years, for a number of clients, and they totally rock!!!
5. I seriously considered node.js and it may become a part of the project as I more deeply explore the openbadges infrastructure... this would add to my learning curve.
6. I will most likely use github for source code librarianship... but I wonder about the best approach and schedule to releasing my work into the open. Mostly, I think about working for free and releasing my work at the correct time.

Mobile First

I have now moved my personal server(s) over to rackspace utilizing a cloud server for hosting my sites. Given this change of hosting I can spin-up new sites and servers with ease. I've been a subscriber to rackspace for a while now and have deployed a number of clients onto this excellent service. As I move my sites over I have also decided it is time to revisit the whole structure, coding and target device(s). Therefore, I've decided to take a mobile first approach. What I mean by this is the website content is made available for mobile devices at the same time as it is made available for desktop web browsers. The information architecture and user experience design consider the mobile device before the website. Where the website becomes more of the users "administrative console" to follow up upon what they do first on their mobile device. More on this in subsequent posts...

I can say with confidence that most of the sites I build from now on will use a mobile first approach. This due to the adoption of mobile devices has started to outpace the adoption of desktops and laptops. Thinking about how a solution should be deployed to mobile first is a shift in thinking from a usability and architecture perspective. And how the browser-based "administrator console" works with the mobile experience creates the complete solution. Again, more on this in subsequent posts...

So as I build my new rackspace solutions with a mobile first approach feel free to follow along. I'll be building the sites step-by-step with companion learning resources and with a more non-technical bent. Even though the implementation will be quite technical and follow a MVC Three-Tier approach. If you want some early insight into the technologies I'll be using, this is what I am currently thinking;

• Presentation Tier (MVC) - HTML5, CSS3, JavaScript and some PhP
• Business Tier - PhP (with object orientation and some RESTful approaches where appropriate)
• Data Tier - PhP, MySQL (with some separation of reads from writes)

Rackspace Step 4.2: I used rackspace email instead

As I begun my task of setting up the postfix server I needed to answer a few questions. The list of questions included;

1. What is the dependency between my DNS and MX records and what do I need to be aware of as I install and configure the mail server. Is there a preferred order in doing these?
2. Does rackspace have fanatical support on the care and feeding of my postfix mail server?
3. If something goes wrong with my mail server, who is going to help and how long would it take to fix?
4. How much time would I spend in administering this postfix mail server on a monthly basis?
5. How would I know if the mail server was hacked and I became a spam server? Being black listed is NOT a good idea.

Answering these questions really drove me to use the Rackspace Hosted Email. I really don't want to spend time setting-up and administering a mail server. I don't believe it is a good use of my time. I build internet solutions and email is a service to me. I really have no need to develop or administer email services. Mostly what I am after is the ability to host my domains email boxes as close to my domain as possible at a reasonable price. And have someone else responsible for its uptime, security and integrity. Rackspace email provides me all this for a really good price.

Rackspace Step 6: Configuring Apache

I'm going to be hosting multiple domains on this single cloud server and through time I am hoping to have a fair bit of traffic on these sites so I'm going to need to deepen my understanding of the Apache Server. Along with the recently purchased book on postfix, I also purchased the book Apache Cookbook, 2nd Ed. By Ken Coar and Rich Bowen.

After getting everything done DNS and hosting wise to point your domain names at your rackspace server then you can start to set up virtual hosting. The process of setting up multiple virtual hosts is very straight forward. It does take a little reading to get your head into it and I suggest a few searches using the terms "virtual hosting apache ubuntu". It is also a good idea to include your OS in the search terms, for each Linux OS has different idiosyncrasies. And in the end I found the rackspace knowledge center post to be the best.

The last step of configuring the apache server to host multiple sites under one ip address is to run "sudo /etc/init.d/apache2 restart". This command restarts apache and provides you errors and warnings if your configuration is incorrect. I would strongly suggest these get cleaned away before you consider yourself finished. Most often the warnings don't stop apache from successfully running your sites. But they could be an indication of a performance issue. From a security, stability and performance perspective it is good to get all errors and warnings cleared away. And searches on the warnings in your favorite search engine should quickly find a solution.

Rackspace Step 5: Updating the DNS

It's been a while since I posted on my work with moving all my sites over the rackspace, it's been summer and the start of the school year for my kids. The task I intermittently focused on through the summer was to move my domain name hosting over to rackspace. Its great that rackspace also provides a DNS based cloud service, and I like the management console available to manage your DNS.

 
Moving DNS servers may not be so simple
Usually you would think that changing DNS servers would be a simple, and it should be. Depending on where you start and who "controls" the ability to update, things may not go as smoothly as you would like. I mention this because without a good move of your DNS your site may disappear from the internet for a period of time. What I want to say is, "When moving your DNS it is important that you monitor the move closely". This is what happened to me and a similar series of events could happen to you;

1. I logged into my previous providers domain hosting console and changed the domain name server for the domain I was moving. I was prompted the save was successful.
2. I went back to the console to see what name servers were assigned to the domain, it was still the old names. I figured this was OK because name server changes need to be updated through-out the internet to truly complete.
3. A couple days later I logged into the domain hosting console to check the name associated with the name server of the domain. It was still set to the old name server. Naturally, I tried again to update it myself. And again I got a confirmation of the change.
4. I got busy and a few days later I checked the names again and my DNS was still pointing at the old name server. I wrote an email to tech support, sent it off and waited.
5. Almost immediately, I got confirmation of my query and was assigned a tracking number for the issue. A few days later nothing, so I phoned... I did speak to someone and they confirmed they had made the change, to the correct domain name. I was adamant about this and they confirmed the correct domain name.
6. The next morning I logged into my domain hosting console and discovered they had made the name server changes to the incorrect domain. 

There is really no point in going any further with this description, and eventually I got it all cleared up. Needless to say, all this was only confirming I was doing the right thing to be moving away from netnation as my hosting provider. Don't get me wrong, netnation has provided me with many years of very stable hosting. Its just my needs have changed and the cost savings provided by cloud based services are too strong to ignore. The main lesson learned is when making changes to things DNS related you need to monitor it very closely, particularly when their are intermediaries involved...

Suggested Reading
http://en.wikipedia.org/wiki/Domain_Name_System
http://www.rackspace.com/cloud/blog/2009/06/04/dns-the-overlooked-cloud-service/
http://www.rackspace.com/knowledge_center/index.php/Managing_DNS

Raskspace Step 4.1: Email Forwarding Server

My email configuring research took me down the path of purchasing the Postfix: The Definitive Guide by Kyle Dent. Fortunately, I purchased to book in digital form so I can get started my reading right away. As I mentioned in my previous post I am concerned about setting up a spam server, if I set up a mail server incorrectly it could become a relay server and I could get my domain blacklisted. I have learned a lot over the last few days with all the reading that is available regarding setting up a mail server. And the conversations I have had with rackspace and one of my clients system administrators has really helped. What stands out for me most is that installing a mail server on Linux requires a lot of pieces. These pieces help with the following services;

High-level postfix and supporting services architecure

• the mail server
• data storage
• forwarding to POP and IMAP
• security
• spam filtering
• other optional services

Given my architectural background I find images very useful in prompting questions, deepening my understanding and putting together an architecture. I have also come to the realization that what I am looking for is more of an email forwarding server. I am wanting to host half a dozen domain names with two to five email boxes per domain. Each email box will have a primary email and a couple of email aliases. All the email coming to these mail boxes will be forwarded to peoples respective gmail accounts. Given this forwarding I should be able to remove all of the services responsible for the POP and IMAP integration. This means I do not need the Courier half of the above diagram. I believe the architecture I am going to end up with is going to look like a simplified version of the above diagram.

Implementation without having Courier.

Fortunately, the postfix book I have purchased covers all the components required to install a mail server. So, a couple of days reading the book and a few quick reviews of the online installation guides and I should be ready to go. Given my simplified architecture, I believe I am only going to require the following services;

• the mail server
• data storage
• security
• spam filtering

One thing to remember is that by having a backup copy of the server configured so far makes it very easy to restart the mail server installation again if something doesn't work out.

Rackspace Step 4.0: The Mail Server

Setting up the mail server is the most important of steps. Not to say that securing the server and installing Apache, MySQL and PhP isn't important, it's just that a mail server set up incorrectly can become a spam server. A spam server can have a negative impact on other people; where the other servers (Apache, MySQL and PhP), incorrectly setup, mostly have a negative impact on yourself. This is why I believe correct setup of the mail server is so important. Doing a lot of reading around this is important. The following is a good list of sites and pages to read as you become more familiar with setting up a Mail Server. After having read through all these sites and a few others, I would recommend the Mail Server - Overview from rackspace.

1. http://cloudservers.rackspacecloud.com/index.php/Mail_Server_-_Overview
2. https://help.ubuntu.com/community/MailServer
3. http://articles.slicehost.com/2010/3/1/barebones-postfix-install-for-ubuntu 
4. https://help.ubuntu.com/community/PostfixBasicSetupHowto
5. https://help.ubuntu.com/community/Postfix

I see all this reading, research and private study as step 4.0 of the mail server setup, step 4.1 describes the architecture of my mail server configuration due to it only needing to forward mail and not provide client access, step 4.2 will get into the actual process of installing, configuring and securing the mail server. I had a chat with rackspace technical support last night as I was considering using their email and apps service. After describing my need of just a few email addresses from a few of my domains all redirected to respective gmail accounts it still seems like the best idea is to setup postfix (and related mail services) on my own server. Designing / Architecting the best mail server solution is the thrust of this 4.0 step.

Rackspace Step 3: Creating the LAMP Server

In the past there was usually a number of steps to set up an Ubuntu cloud based LAMP server. This can also be done using a single command. However you install the AMP on your Ubuntu (Linux) Server one important thing to remember is to install the php-mysql package after mysql. Usually installation takes the following steps;

1. Install Apache
2. Install MySQL
3. Install PhP

All of these steps can also be completed with a single command of;

# sudo apt-get install lamp-server^

After running the apt-get command you will need to reboot the server to finish the install and preform a couple of tests to check everything is working. First test would be to point your browser at the ip address of your new server; if apache was installed correctly you should get the following web page;

 

Once you have confirmation of the apache web server working you should then check the php and mysql features are also working. Creating a simple html / php script will help with this;

<html>
<body>
The Apache2 server works!</br>
<?php
print "PhP5 is working!</br>";
$usr = "username";
$pwd = "password";
$hst = "localhost";
$dbms = mysql_connect($hst, $usr, $pwd) or die("Unable to connect to MySQL");
print "Connected to MySQL!</br>";
mysql_close($dbms);
?>
</body>
</html>

Save a similar section of code with your MySQL username and password to a file with the .php extension in the root directory of your web server (should be /var/www) and point your browser at this new file. If all goes well you should get the following lines displayed in your browser. Congratulations, you have a running LAMP server (In a coming post I will talk about securing PhP and mySQL).

The Apache2 server works!
PhP5 is working!
Connected to MySQL!

Again, this would be an opportune time to do a complete backup of your LAMP server so you have an image to restore from as you continue in setting up your new server.

Rackspace Step 2: Securing the new server

The first step in setting up the new server should always be securing the server the best you can before you continue. An overview of this process can be found in the following video by Chad Keck.



A step-by-step description of this process can also be found on the Rackspace Cloud Server Knowledge Base. In summary the process of securing the new Ubuntu server included the following main steps (I've included a few links for your reading, particularly for configuring ssh and the firewall);

1. Give the root account a new password
2. Create a new administrator account and give it the correct permissions
3. Update Ubuntu
4. Lock down SSH connectivity
• http://cloudservers.rackspacecloud.com/index.php/Ubuntu_-_Setup#Basic_Firewall
• http://cloudservers.rackspacecloud.com/index.php/Introduction_to_iptables
  
5. Firewall configuration
• http://blog.jtclark.ca/tag/rackspace-cloud/
6. Configure Ubuntu to reload these new configuration when it reboots
• http://cloudservers.rackspacecloud.com/index.php/Ubuntu_-_Setup#Saving_your_rules
7. Set the local timezone
• http://www.linuxsa.org.au/tips/time.html

Now is a good time to test out all this new configuration. Reboot your server from within the rackspace management console and then login with your ssh client and test only the new account gains access. Also, once logged in check the timezone is correct.

If all is good it would also be a good time to backup the server so you have a secured image to restore from as you continue down setting up the server.

Rackspace Step 1: Creating an account

I started to migrate all my sites to rackspace today. I've been a netnation user for many years and it has come to the point where they couldn't support want I wanted to do. I need to build some RESTful APIs and needed access to some of the server configuration files... after a number of support calls they said they couldn't do what I wanted.

I already have a number of clients who I've moved over to rackspace so it was time to do the same for myself. This process took a while for I needed to move a couple more clients off my netnation instances and clean up some old blogs that I wanted to keep around. All that is done, so I begin the move. 
A great way to get started with creating a cloud server is to watch this introductory video; 



The features that stood out for me from this introductory video is how you can do the following with a rackspace cloud server;

1. create a new server instance from a backup, allowing you to create a "template" image that you can use for subsequent cloud servers.
2. the ability to rescue a server from another server instance by mounting the "damaged" servers file system.
3. rebuild a server by restoring it from a backup without losing the servers existing IP address.

Rackspace android app

An important issue came up right off the start. I had set up a client using my preferred username and after contacting rackspace support told me I couldn't change the primary username for a rackspace account. I should have known better. A lesson well learned, never use a personal login name for a client. In the end its all for the better for I am going to use another preferred username and make it more robust by putting leets in both my username and password. A more secure solution.

After completing my account setup I created an API Key so I could access my rackspace instance from my android phone. This is a nice feature that would allow me to either soft or hard reboot the server, allow me to resize a server (most likely due to performance needs) or delete the server completely. I'm looking forward to more features becoming available with the android app; the ability to backup and recover. After that a nice ssh app for the android and all would be good...

Building my home base on rackspace

I'm moving all my domains over to rackspace and will firmly establish them as my home base. The sites include the following;

• rawsthorne.org - this is my personal site and will be used to host my personal profile and all published content and links for my Open PhD
• endeavours.com - eventually this will host the site for the massively collaborative assessment system (or some other domain name, TBD)
• bit.bc.ca - Bowen Institute of Technology, this will be the instance pointing at the blended learning / community of practice site where people can work toward of becoming a Solutions Architect / Learning Systems Architect.

Why the move away from netnation for my hosting? For a few reasons;

1. Cost. Cumulatively netnation costs me about $400 per year, I believe I will have my dedicated cloud server for about $100 per year to start.
2. Features. I was building some PhP/MySQL software a while back and I ran into challenges in altering the .hosts file to support a RESTful API I was building. Even after contacting netnations technical support they couldn't help me. I ended up using rackspace, their support was awesome.
3. Cloud based servers. I firmly believe in cloud based approaches and if I can host multiple sites with the ability to expand for close to $100 per year, why wouldn't I move.

Honestly, I really don't think netnation will mind that I move. I exhausted the discussion with the features I required. The features I required are to enable a RESTful API with a specific domain name. It requires some customizations to configuration files at the server level. Netnation were not willing to support the customizations I was requesting.