{"id":380,"date":"2017-07-20T11:32:35","date_gmt":"2017-07-20T15:32:35","guid":{"rendered":"http:\/\/blog.kortar.org\/?p=380"},"modified":"2017-08-07T16:25:39","modified_gmt":"2017-08-07T20:25:39","slug":"dirty-clouds-done-cheap","status":"publish","type":"post","link":"https:\/\/blog.kortar.org\/?p=380","title":{"rendered":"Dirty Clouds Done Cheap"},"content":{"rendered":"

I recently gave a presentations at the OpenStack Summit in Boston with the same title.\u00a0You can find a video of the talk here: https:\/\/www.openstack.org\/videos\/boston-2017\/dirty-clouds-done-dirt-cheap<\/a>. This blog post will cover the same project, but it will go into a lot more detail, which I couldn’t cover during the presentation.<\/p>\n

Just a heads up this post is long!\u00a0 I try to cover every step of the project with all the details I could remember. It probably would have made sense to split things up into multiple posts, but I wrote it in a single sitting and doing that felt weird. If you’re just looking for a quicker overview, I recommend watching the video of my talk instead.<\/p>\n

\n

The Project Scope<\/h1>\n

When I was in college I had a part time job as a sysadmin at a HPC research lab<\/a> in the aerospace engineering department. In that role I was responsible for all aspects of the IT in the lab, from the workstations and servers to the HPC clusters. In that role I\u00a0 often had to deploy new software with no prior knowledge about it. I managed to muddle through most of the time by reading the official docs and frantically google searching when I encountered issues.<\/p>\n

Since I started working on OpenStack I often think back to my work in college and wonder if I had been tasked with deploying an OpenStack cloud back then would I have been able to? As a naive college student who had no knowledge of OpenStack would I have been successful in trying to deploy OpenStack by myself? Since I had no knowledge\u00a0 of configuration management (like puppet or chef) back then I would have gone about it by installing everything by hand. Basically the open question from that idea is how hard is it actually to install OpenStack by hand\u00a0 using the documentation and google searches?<\/p>\n

Aside from the interesting thought exercise I also have wanted a small cloud at home for a couple of reasons. I maintain a number of servers at home that run a bunch of critical infrastructure. For some time I’ve wanted to virtualize my home infrastructure mainly just for the increased flexibility and potential reliability improvements.\u00a0 Running things off a residential ISP and power isn’t the best way to run a server with a decent uptime.\u00a0 Besides virtualizing some of my servers it would be nice to have the extra resources for my upstream OpenStack development, I often do not have the resources available to me for running devstack or integration tests locally and have to rely on upstream testing.<\/p>\n

So after the Ocata release I decided to combine these 2 ideas and build myself a small cloud at home. I would do it by hand (ie no automation or config management) to test out how hard it would be. I set myself a strict budget of $1500 USD (the rough cost of my first desktop computer in middle school, an IBM Netvista A30p that I bought with my Bar Mitzvah money) to acquire hardware. This was mostly just a fun project for me so I didn’t want to spend an obscene amount of money. $1500 USD is still a lot of money, but it seemed like a reasonable amount for the project.<\/p>\n

However, I decided to take things a step further than I originally planned and build the cloud using the release tarballs from http:\/\/tarballs.openstack.org\/<\/a>. My reasoning behind this was to test out how hard it would be to take the raw code we release as a community and turn that into a working cloud. It basically invalidated the project as a test for my thought exercise of deploying the cloud if I was back in college (since I definitely would have just used my Linux distro’s packages back then) but it made the exercise more relevant for me personally as an upstream OpenStack developer. It would give me insight as to where what we’re there are gaps in our released code and how we could start to fix them.
\n<\/p>\n

Building the Cloud<\/h1>\n

Acquiring the Hardware<\/h2>\n

The first step for building the cloud was acquiring the hardware. I had a very tight budget and it basically precluded buying anything new. The cheapest servers you can buy from a major vendor would pretty much eat up my budget for a single machine. I also considered building a bunch of cheap desktops for the project and putting those together as a cloud. (I didn’t actually need server class hardware for this cloud) But for the cost the capacity was still limited. Since I was primarily building a compute cloud to provide me with a pool of servers to allocate My first priority was the number of CPU cores in the cloud. This would give me the flexibility to scale any applications I was running on it. With that in mind I decided on the priority list for the hardware of:<\/p>\n

    \n
  1. Number of Cores<\/li>\n
  2. Amount of RAM<\/li>\n
  3. Speed of CPU<\/li>\n<\/ol>\n

    The problem with building with desktop CPUs is (at the time I was assembling pieces) the core count \/ USD was not really that high for any of the desktop processors. Another popular choice for home clouds is the Intel NUCs<\/a>, but these suffer from the same problem. The NUCs use laptop processors and while reasonably priced you’re still only getting a dual or quad core CPU for a few hundred dollars.<\/p>\n

    It turns out the best option I found for my somewhat bizarre requirements was to buy used hardware. A search of eBay shows a ton of servers from 8 or 9 years ago that are dirt cheap. After searching through my various options I settled on old Dell PowerEdge R610, which was a dual socket machine. The one I ordered came with 2 Intel Xeon E5540<\/a> CPUs in it. This gave me a total of 8 physical cores (or 16 virtual cores if you count HyperThreading\/SMT) The machines also came with 32 GB of RAM and 2x 149GB SAS hard drives. The best part though was that each machine was only $215.56 USD. This gave me plenty of room in the budget, so I bought 5 of them. After shipping this ended up costing only $1,230.75<\/span>. That gave me enough wiggle room for the other parts I’d need to make everything working. The full hardware specs from the eBay listing was:<\/p>\n

    \"\"<\/a><\/p>\n

    Although, the best part about these servers were that I actually had a rack full of basically the same exact servers at the lab in college. The ones I had back in 2010\u00a0 were a little bit slower and had half the RAM, but were otherwise the same. I configured those servers as a small\u00a0 HPC cluster my last year at college, so I was very familiar with them. Although back then those servers were over 10x the cost as what I was paying for them on eBay now.<\/p>\n

    The only problem with this choice was the hardware, the Xeon E5540 is incredibly slow by today’s standards. But, because of my limited budget speed was something I couldn’t really afford.<\/p>\n

    Assembling the Hardware<\/h2>\n

    After waiting a few days the servers were delivered.\u00a0 That was a fun day, the FedEx delivery person didn’t bother to ring the door bell. Instead I heard big thud outside and found that they had left all the\u00a0 boxes in front of my apartment door. Fortunately I was home and heard them throw the boxes on the ground, because it was raining that day. Leaving my “new” servers out in the rain all day would have been less than an ideal way to start the project . It also made quite the commotion and several of my neighbors came out to see what was going on and watched me as I took the boxes inside.<\/p>\n

    After getting the boxes inside my apartment and unboxed, I stacked them on my living room table:<\/p>\n

    \"\"<\/a>
    \nMy next problem with this was where to put the servers and how to run them. I looked at buying a traditional rack, however they were a bit too pricey. (even on eBay) Just a 10U rack looked like it would cost over $100 USD and after shipping that wouldn’t leave me too much room if I needed something else. So I decided not to go that route. Then I remembered hearing about something called a     LackRack<\/a> a few years ago. It turns out the IKEA Lack<\/a> table has a 19 inch width between the legs which is the same as a rack. They also only cost $9.99 which made it a much more economical choice compared to a more traditional rack. However, while I could just put the table on the floor and be done with it, I was planning to put the servers in my “data closet” (which is just a colorful term for my bedroom closet where I store servers and clothing) but I didn’t want to deal with having to pick up the “rack” every time I needed to move it. So I decided to get some casters and mount them to the table so I could just push the server around.<\/p>\n

    Once I got the table delivered, which took a surprisingly long time, I was able to mount he casters and rack the servers. As I put each server on the table I was able to test each of them out. (I only had a single power cable at the time, so I went one at a time) It turns out that each server was slightly different from the description and had several issues:<\/p>\n