It's Alive! The Elastic Stack as our Data Lab

So much technical work, so little time! I finished my first three sprints toward standing up the data lab. Standing up infrastructure from scratch so you have clean new compute power is fun, and also a lot of work. Particularly when you include; doing it right, taking no short cuts, and making sure it is secure.

Sprint 0: Setup Ubuntu 20.04 Server with ELK stack.

This was mostly rehydrating virtual server infrastructure I hadn't used in 8.5 years. It needed an upgrade from all perspectives and had a completely new OS. I implemented the ELK stack and made a couple of security changes to lock it all down. I ran a few tests by setting up a couple of websites, getting the JSON confirmation from ElasticSearch, and called up the Kibana dashboard. Oooo... sweet success!

Sprint 1: Vulnerability Assessment. Security changes if required.

This evening I spent some time poking at the overall vulnerability of the server and with the ElasticSearch and Kibana services. I made a few additions and changes for further locking down the services and believe they are as secure as they can be for this first release. Very happy to feel reasonably confident about it's being locked down. Maybe, I'll get lucky and get some free PEN testing. ha.

Sprint 2: Identify and register some well aligned domain names.

I registered the following domain names, even considered buying one... it would have been too expensive. I'll implement the data lab on the oceansofdatalab.com site when it becomes closer to being a minimal viable product (MVP).

• oceansofdatalab.com
• oceansofdatalab.org
• oceansofdatalab.net
• oceansofdata.net
• sevenseasofdata.com
• sevenseasofdata.org

Integrating with Open Badges

I have been thinking about how best a project or organization can integrate with open badges. This isn't the bits and bytes type of thinking, but the technical project management thinking required for the near-term (6 -12 months) success of the OBI and all projects that integrate with the OBI. Keep in mind this is a list of what I would focus upon as you work toward success. It is my approach as a technical architect and senior project manager and not Mozilla's or the OBI teams. I see these tasks as a benefit to the Open Badges project, as they will bring further insight into the design of the project, provide further requirements, and put the OBI through is paces before it is considered a production release.

1. Figure out the data model of your badge system design
This may require some technical skills and knowledge (and I suggest you find them at this time) to understand the data requirements of your badge system design. Do this without making reference to the existing OBI metadata specification, for that may compromise the vision of your badge system design. And you should stay true to your badge system design without restraining it by the existing Mozilla OBI specification. Keep in mind the OBI is in pre-release and really needs rubber-hits-the-road insights into how people want to implement badges. Once you have this data model try and map it to the OBI metadata specification for issuing a badge. If it doesn't fit, let the OBI team know via the google group, this could kick off an interesting discussion.

2. Start loading (and deleting) data
I see two kinds of badging systems. Those that already exist; like girl guide badges, khan academy badges, stackexchange badges or military badges. And the badges systems that don't yet exist. These existing badge systems have a huge number of badges already stored within them. If these badges are to make it into Mozilla Open Badges, we need to start loading them into the OBI now. This attempt to mass load badges into the OBI should come as a gift to the OBI team. Better they start dealing with the issue of thousands of badges showing up while still in pre-release than having them show up once Open Badges is officially released and bringing the whole OBI to a halt. If you are going to load data, you should also be able to delete the same data. The ability to load data should be a feature available within the OBI. A test environment should also be available, so partners can test their data loads, and delete the data that was loaded. The testing of data loading is a blog post in itself and beyond the scope of this particular tome.

Keep in mind that manually entering badges (one by one) is also a good idea as it will utilize the same process as the earners will use for adding badges to their backpack.

3. Build an automated testing approach
If you have already started working with the Mozilla Open Badges Infrastructure (OBI), or are considering it in the next short while, you should consider yourself an early adopter. And given this type of role expect things to change within the OBI, as it is currently a pre-release version. Given this early release things can change that may break the code at your end of the integration. Building an automated testing approach will yield great benefits as it will reduce the downtime of your system, increase the quality of your code, and provide early warning if the system elements beyond your control have changed. These automated tests could be built for both your internal system and the API calls you make to the OBI. Once these automated tests are built, it would also be beneficial to run them against the OBI at scheduled times to ensure the programatic assumptions made by your system are still valid against the OBI.

4. Determine where and how you want to encourage the display of your badges
Once you have badges "loaded" into the OBI and have viewed these badges in the earners backpack it becomes important to display the badges in a variety of locations. There is a growing number of widgets built for badge display and with the variety of methods for display you never know where your badges will show up. It is important to test these different display approaches and as a part of your issuing site encourage people to use specific displayers. You may also want to build or integrate your own displayer. What is important here is that you have displayed your badges in a number of different locations for testing purposes.

5. Advocate for other attributes to be added to the OBI metadata specification
Learning rarely occurs in isolation and learning is always related to other or previous learning in some way. The current metadata specification has a badge standing on its own. There is currently no attribute within the badges metadata that allow for it to link-to, be the detail of, or cluster-within another badge or set of badges. The OBIs' criteria attribute is being extended to allow blobs of data, so you could store this badge inter-relationship and linkage data in this criteria attribute. I believe this approach will store this information in the wrong place, for issuing systems would then have to store and maintain the data about these relationships of interest. A badge should know for itself if it exists in relationship to other badges.Erin Knight has created a great blog post describing the webmaker badge system design.

WebMaker Badges show a system design where badges are linked, clustered and the detail of a cumulative badge.

6. Encourage the completion of badge validation
Over the last year there has been much discussion regarding badge assertion or validation. The idea being that it can be determined if a badge is still valid through time. A number of approaches have been tabled, and agreement to the value and approach to implementation of this feature has yet to be agreed upon. I believe it is an important feature, for it would be nice to click on a badge from within the backpack or a displayer site and be notified if the badge is still valid.

If you have further questions or require assistance please do not hesitate to contact me. I will be very happy to assist. 

Rackspace Step 4.2: I used rackspace email instead

As I begun my task of setting up the postfix server I needed to answer a few questions. The list of questions included;

1. What is the dependency between my DNS and MX records and what do I need to be aware of as I install and configure the mail server. Is there a preferred order in doing these?
2. Does rackspace have fanatical support on the care and feeding of my postfix mail server?
3. If something goes wrong with my mail server, who is going to help and how long would it take to fix?
4. How much time would I spend in administering this postfix mail server on a monthly basis?
5. How would I know if the mail server was hacked and I became a spam server? Being black listed is NOT a good idea.

Answering these questions really drove me to use the Rackspace Hosted Email. I really don't want to spend time setting-up and administering a mail server. I don't believe it is a good use of my time. I build internet solutions and email is a service to me. I really have no need to develop or administer email services. Mostly what I am after is the ability to host my domains email boxes as close to my domain as possible at a reasonable price. And have someone else responsible for its uptime, security and integrity. Rackspace email provides me all this for a really good price.

Rackspace Step 6: Configuring Apache

I'm going to be hosting multiple domains on this single cloud server and through time I am hoping to have a fair bit of traffic on these sites so I'm going to need to deepen my understanding of the Apache Server. Along with the recently purchased book on postfix, I also purchased the book Apache Cookbook, 2nd Ed. By Ken Coar and Rich Bowen.

After getting everything done DNS and hosting wise to point your domain names at your rackspace server then you can start to set up virtual hosting. The process of setting up multiple virtual hosts is very straight forward. It does take a little reading to get your head into it and I suggest a few searches using the terms "virtual hosting apache ubuntu". It is also a good idea to include your OS in the search terms, for each Linux OS has different idiosyncrasies. And in the end I found the rackspace knowledge center post to be the best.

The last step of configuring the apache server to host multiple sites under one ip address is to run "sudo /etc/init.d/apache2 restart". This command restarts apache and provides you errors and warnings if your configuration is incorrect. I would strongly suggest these get cleaned away before you consider yourself finished. Most often the warnings don't stop apache from successfully running your sites. But they could be an indication of a performance issue. From a security, stability and performance perspective it is good to get all errors and warnings cleared away. And searches on the warnings in your favorite search engine should quickly find a solution.

Rackspace Step 5: Updating the DNS

It's been a while since I posted on my work with moving all my sites over the rackspace, it's been summer and the start of the school year for my kids. The task I intermittently focused on through the summer was to move my domain name hosting over to rackspace. Its great that rackspace also provides a DNS based cloud service, and I like the management console available to manage your DNS.

 
Moving DNS servers may not be so simple
Usually you would think that changing DNS servers would be a simple, and it should be. Depending on where you start and who "controls" the ability to update, things may not go as smoothly as you would like. I mention this because without a good move of your DNS your site may disappear from the internet for a period of time. What I want to say is, "When moving your DNS it is important that you monitor the move closely". This is what happened to me and a similar series of events could happen to you;

1. I logged into my previous providers domain hosting console and changed the domain name server for the domain I was moving. I was prompted the save was successful.
2. I went back to the console to see what name servers were assigned to the domain, it was still the old names. I figured this was OK because name server changes need to be updated through-out the internet to truly complete.
3. A couple days later I logged into the domain hosting console to check the name associated with the name server of the domain. It was still set to the old name server. Naturally, I tried again to update it myself. And again I got a confirmation of the change.
4. I got busy and a few days later I checked the names again and my DNS was still pointing at the old name server. I wrote an email to tech support, sent it off and waited.
5. Almost immediately, I got confirmation of my query and was assigned a tracking number for the issue. A few days later nothing, so I phoned... I did speak to someone and they confirmed they had made the change, to the correct domain name. I was adamant about this and they confirmed the correct domain name.
6. The next morning I logged into my domain hosting console and discovered they had made the name server changes to the incorrect domain. 

There is really no point in going any further with this description, and eventually I got it all cleared up. Needless to say, all this was only confirming I was doing the right thing to be moving away from netnation as my hosting provider. Don't get me wrong, netnation has provided me with many years of very stable hosting. Its just my needs have changed and the cost savings provided by cloud based services are too strong to ignore. The main lesson learned is when making changes to things DNS related you need to monitor it very closely, particularly when their are intermediaries involved...

Suggested Reading
http://en.wikipedia.org/wiki/Domain_Name_System
http://www.rackspace.com/cloud/blog/2009/06/04/dns-the-overlooked-cloud-service/
http://www.rackspace.com/knowledge_center/index.php/Managing_DNS

Raskspace Step 4.1: Email Forwarding Server

My email configuring research took me down the path of purchasing the Postfix: The Definitive Guide by Kyle Dent. Fortunately, I purchased to book in digital form so I can get started my reading right away. As I mentioned in my previous post I am concerned about setting up a spam server, if I set up a mail server incorrectly it could become a relay server and I could get my domain blacklisted. I have learned a lot over the last few days with all the reading that is available regarding setting up a mail server. And the conversations I have had with rackspace and one of my clients system administrators has really helped. What stands out for me most is that installing a mail server on Linux requires a lot of pieces. These pieces help with the following services;

High-level postfix and supporting services architecure

• the mail server
• data storage
• forwarding to POP and IMAP
• security
• spam filtering
• other optional services

Given my architectural background I find images very useful in prompting questions, deepening my understanding and putting together an architecture. I have also come to the realization that what I am looking for is more of an email forwarding server. I am wanting to host half a dozen domain names with two to five email boxes per domain. Each email box will have a primary email and a couple of email aliases. All the email coming to these mail boxes will be forwarded to peoples respective gmail accounts. Given this forwarding I should be able to remove all of the services responsible for the POP and IMAP integration. This means I do not need the Courier half of the above diagram. I believe the architecture I am going to end up with is going to look like a simplified version of the above diagram.

Implementation without having Courier.

Fortunately, the postfix book I have purchased covers all the components required to install a mail server. So, a couple of days reading the book and a few quick reviews of the online installation guides and I should be ready to go. Given my simplified architecture, I believe I am only going to require the following services;

• the mail server
• data storage
• security
• spam filtering

One thing to remember is that by having a backup copy of the server configured so far makes it very easy to restart the mail server installation again if something doesn't work out.

Rackspace Step 4.0: The Mail Server

Setting up the mail server is the most important of steps. Not to say that securing the server and installing Apache, MySQL and PhP isn't important, it's just that a mail server set up incorrectly can become a spam server. A spam server can have a negative impact on other people; where the other servers (Apache, MySQL and PhP), incorrectly setup, mostly have a negative impact on yourself. This is why I believe correct setup of the mail server is so important. Doing a lot of reading around this is important. The following is a good list of sites and pages to read as you become more familiar with setting up a Mail Server. After having read through all these sites and a few others, I would recommend the Mail Server - Overview from rackspace.

1. http://cloudservers.rackspacecloud.com/index.php/Mail_Server_-_Overview
2. https://help.ubuntu.com/community/MailServer
3. http://articles.slicehost.com/2010/3/1/barebones-postfix-install-for-ubuntu 
4. https://help.ubuntu.com/community/PostfixBasicSetupHowto
5. https://help.ubuntu.com/community/Postfix

I see all this reading, research and private study as step 4.0 of the mail server setup, step 4.1 describes the architecture of my mail server configuration due to it only needing to forward mail and not provide client access, step 4.2 will get into the actual process of installing, configuring and securing the mail server. I had a chat with rackspace technical support last night as I was considering using their email and apps service. After describing my need of just a few email addresses from a few of my domains all redirected to respective gmail accounts it still seems like the best idea is to setup postfix (and related mail services) on my own server. Designing / Architecting the best mail server solution is the thrust of this 4.0 step.

Rackspace Step 3: Creating the LAMP Server

In the past there was usually a number of steps to set up an Ubuntu cloud based LAMP server. This can also be done using a single command. However you install the AMP on your Ubuntu (Linux) Server one important thing to remember is to install the php-mysql package after mysql. Usually installation takes the following steps;

1. Install Apache
2. Install MySQL
3. Install PhP

All of these steps can also be completed with a single command of;

# sudo apt-get install lamp-server^

After running the apt-get command you will need to reboot the server to finish the install and preform a couple of tests to check everything is working. First test would be to point your browser at the ip address of your new server; if apache was installed correctly you should get the following web page;

 

Once you have confirmation of the apache web server working you should then check the php and mysql features are also working. Creating a simple html / php script will help with this;

<html>
<body>
The Apache2 server works!</br>
<?php
print "PhP5 is working!</br>";
$usr = "username";
$pwd = "password";
$hst = "localhost";
$dbms = mysql_connect($hst, $usr, $pwd) or die("Unable to connect to MySQL");
print "Connected to MySQL!</br>";
mysql_close($dbms);
?>
</body>
</html>

Save a similar section of code with your MySQL username and password to a file with the .php extension in the root directory of your web server (should be /var/www) and point your browser at this new file. If all goes well you should get the following lines displayed in your browser. Congratulations, you have a running LAMP server (In a coming post I will talk about securing PhP and mySQL).

The Apache2 server works!
PhP5 is working!
Connected to MySQL!

Again, this would be an opportune time to do a complete backup of your LAMP server so you have an image to restore from as you continue in setting up your new server.

Rackspace Step 2: Securing the new server

The first step in setting up the new server should always be securing the server the best you can before you continue. An overview of this process can be found in the following video by Chad Keck.



A step-by-step description of this process can also be found on the Rackspace Cloud Server Knowledge Base. In summary the process of securing the new Ubuntu server included the following main steps (I've included a few links for your reading, particularly for configuring ssh and the firewall);

1. Give the root account a new password
2. Create a new administrator account and give it the correct permissions
3. Update Ubuntu
4. Lock down SSH connectivity
• http://cloudservers.rackspacecloud.com/index.php/Ubuntu_-_Setup#Basic_Firewall
• http://cloudservers.rackspacecloud.com/index.php/Introduction_to_iptables
  
5. Firewall configuration
• http://blog.jtclark.ca/tag/rackspace-cloud/
6. Configure Ubuntu to reload these new configuration when it reboots
• http://cloudservers.rackspacecloud.com/index.php/Ubuntu_-_Setup#Saving_your_rules
7. Set the local timezone
• http://www.linuxsa.org.au/tips/time.html

Now is a good time to test out all this new configuration. Reboot your server from within the rackspace management console and then login with your ssh client and test only the new account gains access. Also, once logged in check the timezone is correct.

If all is good it would also be a good time to backup the server so you have a secured image to restore from as you continue down setting up the server.

Rackspace Step 1: Creating an account

I started to migrate all my sites to rackspace today. I've been a netnation user for many years and it has come to the point where they couldn't support want I wanted to do. I need to build some RESTful APIs and needed access to some of the server configuration files... after a number of support calls they said they couldn't do what I wanted.

I already have a number of clients who I've moved over to rackspace so it was time to do the same for myself. This process took a while for I needed to move a couple more clients off my netnation instances and clean up some old blogs that I wanted to keep around. All that is done, so I begin the move. 
A great way to get started with creating a cloud server is to watch this introductory video; 



The features that stood out for me from this introductory video is how you can do the following with a rackspace cloud server;

1. create a new server instance from a backup, allowing you to create a "template" image that you can use for subsequent cloud servers.
2. the ability to rescue a server from another server instance by mounting the "damaged" servers file system.
3. rebuild a server by restoring it from a backup without losing the servers existing IP address.

Rackspace android app

An important issue came up right off the start. I had set up a client using my preferred username and after contacting rackspace support told me I couldn't change the primary username for a rackspace account. I should have known better. A lesson well learned, never use a personal login name for a client. In the end its all for the better for I am going to use another preferred username and make it more robust by putting leets in both my username and password. A more secure solution.

After completing my account setup I created an API Key so I could access my rackspace instance from my android phone. This is a nice feature that would allow me to either soft or hard reboot the server, allow me to resize a server (most likely due to performance needs) or delete the server completely. I'm looking forward to more features becoming available with the android app; the ability to backup and recover. After that a nice ssh app for the android and all would be good...

OpenStack is some serious shit!

Being backed by Rackspace, NASA, Dell, Citrix, Cisco, Canonical and over 50 other organizations make this a force to be reckoned with... and two of my current favorites of Rackspace and Canonical... and given I run nothing but Dell (servers, desktops and laptops) at home with a mix of XP, Vista and Ubuntu, I'd have to say Dell could be considered a favourite as well. I digress. OpenStack is going to provide the freedom to move your applications around with relative ease, if your an IT shop with Internet facing applications / servers I'd strongly suggest you start becoming aware of OpenStack and where available, playing in this architecture for building apps and sites.
 

 

 

 

Building my home base on rackspace

I'm moving all my domains over to rackspace and will firmly establish them as my home base. The sites include the following;

• rawsthorne.org - this is my personal site and will be used to host my personal profile and all published content and links for my Open PhD
• endeavours.com - eventually this will host the site for the massively collaborative assessment system (or some other domain name, TBD)
• bit.bc.ca - Bowen Institute of Technology, this will be the instance pointing at the blended learning / community of practice site where people can work toward of becoming a Solutions Architect / Learning Systems Architect.

Why the move away from netnation for my hosting? For a few reasons;

1. Cost. Cumulatively netnation costs me about $400 per year, I believe I will have my dedicated cloud server for about $100 per year to start.
2. Features. I was building some PhP/MySQL software a while back and I ran into challenges in altering the .hosts file to support a RESTful API I was building. Even after contacting netnations technical support they couldn't help me. I ended up using rackspace, their support was awesome.
3. Cloud based servers. I firmly believe in cloud based approaches and if I can host multiple sites with the ability to expand for close to $100 per year, why wouldn't I move.

Honestly, I really don't think netnation will mind that I move. I exhausted the discussion with the features I required. The features I required are to enable a RESTful API with a specific domain name. It requires some customizations to configuration files at the server level. Netnation were not willing to support the customizations I was requesting.