Using VirtualBox for Development
Polycademy has been improving it's development workflow. Especially the deployment of web server stack.
One of the things that I realised in running the courses, is that the work to setup web server stacks which means the HTTP server, Application Server, Database Server and various other daemons and applications is a very time consuming process. This is mainly devops work and configuration management, it's not really development. What is required is an automatic way to create development and production servers that is isolated from each other, and has the exact same configuration every time you destroy or create one. The development server should match the environment of the production servers so that you reduce the number configuration problems during deployment of your application to the production server. In setting up a workflow like this, you can get a file that specifies how these virtual servers are setup, and just copy paste the file to whichever computer you are currently developing with. In some cases you can just straight up copy the virtual machine image which is the entire operating system and associated installed applications and load this into wherever you want to develop as well.
So I've been spending time researching on deployment automation. And I came across services like Vagrant, Docker, VirtualBox, Chef, Puppet, Ansible, SaltStack and a whole lot more. Enterprises tend to use devops personnel to write up configuration scripts that essentially deploy hundreds or thousands of servers in an automated mannger. This saves time and cost. However these methods are perfectly applicable to the individual developer or startup.
My research has led me to conclude that a combination of VirtualBox, Vagrant, Docker, and Ansible is the best development and deployment workflow for individual or startup projects. VirtualBox is for setting up virtual operating systems on your computer, this is only for development, in production your virtual operating systems are provided to you when you rent a VPS from a hoster such as Digital Ocean. Vagrant is a command line tool that allows you auto configure VirtualBox. Docker is for setting up lightweight virtual machines inside the virtual operating system, and these will run the applications you're developing. It's like a virtual machine inside a virtual machine. I don't yet understand the specifics of how Docker works, but when I do I'll probably write a blog post about setting up your own private cloud host. Finally Ansible the configuration management/IT automation tool, that will basically setup Docker and other services inside the Virtual Machines, and perhaps setup your application inside Docker as well. Note that another service called Packer allows you to quickly replicate your virtual machine images so it's good for the development of virtual appliances.
So in order to learn how to do all of this, I had start at the beginning with VirtualBox. I've managed to setup VirtualBox as a guest operating system that not only has a synchronised development folder with my host operating system (Windows, Linux and Mac) but is also accessible over the network.
What does this all mean? This means I can open up my Sublime Text Editor to my "www" directory. This is where all my projects reside. I can edit a file in this directory. This file change is automatically propagated to the virtual machine instance. Then in my host machine I can open up Firefox to "http://localhost:8000" or "http://dev.test.com:8000" and get to see my updated change in my application. Then in my host or guest machine, can I commit and synchronise to github. All of this can be done without installing a web server stack (HTTP servers, Application Runtimes..etc) on my host machine, thus keeping my host machine clean. I'm not talking about production deployment at this moment, because I haven't got that far with Docker..etc. Therefore any deployment routines to production servers will still stay the same. However I'm still quite happy with what I have here, it is definitely an improvement over running WAMP or MAMP. As I can easily use NGINX and other Unix based tools. Furthermore, the instructions below can easily be adapted to setting up SSH access from your host machine too if you want to command your guest operating system from your host's terminal.
Now comes the tutorial of how to set this all up. This part does require you understand Ubuntu Linux, basic bash scripting, CLI usage, modifying the hosts file and setting up NGINX.
First download VirtualBox: https://www.virtualbox.org/wiki/Downloads. Then download Ubuntu Desktop: http://www.ubuntu.com/download/desktop
Run VirtualBox and create a new virtual machine instance. A wizard dialog will pop up. Make sure to select the correct settings for your virtual machine. The perfomance of the virtual machine will be better if you give it a static size hard drive and 2048 mb ram, 2 CPUs and 32 mb of Video Memory along with 3D acceleration. These settings can be changed later inside the GUI of VirtualBox, they make your VM more expensive to use, but it also makes your VM a lot more faster and responsive. Once your virtual machine instance is set, you need to start it up. Then load the ISO of Ubuntu image you just downloaded. Follow the instructions to install Ubuntu in your new virtual operating system.
Once it's all been installed, you need to remove the ISO CD from your virtual operating system. This can be done by shutting down your VM (virtual machine), then going into the settings of your VM inside VirtualBox and removing the CD in the storage settings. Start it up again and this time install the VBoxAdditions ISO. The VM would have prompted you to do this. This VBoxAdditions allows your VM to setup shared folders and other good things. We'll need shared folders to allow your host machine to edit files and have your guest machine use those updated files. Once that is done, you'll need to restart your VM and remove the VBoxAdditions ISO CD.
Setting up shared folders is quite easy. Shutdown your VM. Go into the VirtualBox settings for the VM. Go into shared folders. Add a new shared folder record. The first folder path is the path your development folder. For me this is "D:www". The folder name would be "www". Make sure Read-only is off, switch on Auto-mount and Make Permanent. Hit Ok. Start your VM. Using your terminal in root user assuming that your folder is "www", run:
cd /media/sf_www/
ls -l
This should show you the contents of your "www" folder on your host machine. If your folder was called something else, it will always have the prefix of "sf_". Furthermore we can see the owner and group the files and folders are associated with. This folder will be grouped under "vboxsf". Right now we can only access it with the root user. In order to add extra users to this group we can use this bash command (fill "username" with the name of the user):
sudo adduser username vboxsf
We should add the default user that you log in with, and also the "www-data" user. The "www-data" user will be the user that your web server stack will use. Your webserver such as NGINX, PHP, Cron scripts should not be using root or your default user, but a custom user for web stuff. It doesn't need to be called "www-data" but that is the default name used by PHP-FPM.
Once you do that, the setting does not take effect immediately. You need to restart your VM. After which you no longer have to be a root user to access the "sf_www" directory inside your VM.
Now the /media/sf_www is not really convenient directory path. So let's try to access it from something like /www. To do so, setup a symbolic link from /media/sf_www to /www.
ln -s /media/sf_www /www
Now test it out. Go into the "www" directory from your host machine, and create or edit a file. Now in your VM, check the /www directory and see if the file is there or edited. We should now have synchronised development folders!
Now let's do some port forwarding. First we need an HTTP server on the VM. So we can install NGINX which by default listens on port 80 and shows a default NGINX page.
sudo apt-get install nginx
Test it out by launching Firefox inside the VM and heading to "http://localhost". If it shows the page, then NGINX is working. While you're at it, try out Polycademy's NGINX configuration best practices: https://github.com/Polycademy/WebserverConfiguration
Shutdown the VM. Inside VirtualBox's VM settings, go into Network. From Network, the Network Adapter should be set to NAT. Click on Port Forwarding. Add a new record. In this record, set the Host Port to 8000 and the Guest Port to 80. The Protocol should be TCP. The name and everything else can be left as default. You can change the name to anything you like. Restart the VM.
Now in your host machine, launch your browser and head to "http://localhost:8000". This should show the same page as the page you saw when you visited with Firefox inside the VM. In the future you can also port forward HTTPS connections, and you'll need to go from a random host port such as 8001 to the port 443 on the guest machine, this would result in URLs like https://localhost:8001.
You can change the hosts file inside the host machine independently of the guest machine. This means the hosts file inside the guest machine affects the guest. So if you want a development URL such as "http://dev.example.com" inside the guest machine, just set that up inside /etc/hosts in the guest machine. This will work for browsers inside the guest machine. But if you want that for the host machine, you'll need to add that record to the hosts file in the host machine.
Now you can install all your web server stack into your VM and not worry about the host machine or incompatibilities between operating systems especially when you develop on Windows/Mac but deploy on Linux.
If you meet any problems, check the NGINX logs which should be specified inside the "nginx.conf" file inside /etc/nginx/nginx.conf. There could be issues with file caching with NGINX, which means that NGINX is caching the files it's serving. This is not good for development, and should be switched off. You should then restart NGINX with sudo service nginx restart. Other problems could include permissions like "vboxsf" but this should work great.
The next step would be learning Vagrant & Ansible to automate the VM constructions as it builds on top VirtualBox (and other various VM provisioiners). Currently all you have is a VM image you could share, but sharing a 1 gig file is a bit hard, a configuration file would be smaller and easier to work with.