Polycademy has been improving it's development workflow. Especially the deployment of web server stack.

One of the things that I realised in running the courses, is that the work to setup web server stacks which means the HTTP server, Application Server, Database Server and various other daemons and applications is a very time consuming process. This is mainly devops work and configuration management, it's not really development. What is required is an automatic way to create development and production servers that is isolated from each other, and has the exact same configuration every time you destroy or create one. The development server should match the environment of the production servers so that you reduce the number configuration problems during deployment of your application to the production server. In setting up a workflow like this, you can get a file that specifies how these virtual servers are setup, and just copy paste the file to whichever computer you are currently developing with. In some cases you can just straight up copy the virtual machine image which is the entire operating system and associated installed applications and load this into wherever you want to develop as well.

So I've been spending time researching on deployment automation. And I came across services like Vagrant, Docker, VirtualBox, Chef, Puppet, Ansible, SaltStack and a whole lot more. Enterprises tend to use devops personnel to write up configuration scripts that essentially deploy hundreds or thousands of servers in an automated mannger. This saves time and cost. However these methods are perfectly applicable to the individual developer or startup.

My research has led me to conclude that a combination of VirtualBox, Vagrant, Docker, and Ansible is the best development and deployment workflow for individual or startup projects. VirtualBox is for setting up virtual operating systems on your computer, this is only for development, in production your virtual operating systems are provided to you when you rent a VPS from a hoster such as Digital Ocean. Vagrant is a command line tool that allows you auto configure VirtualBox. Docker is for setting up lightweight virtual machines inside the virtual operating system, and these will run the applications you're developing. It's like a virtual machine inside a virtual machine. I don't yet understand the specifics of how Docker works, but when I do I'll probably write a blog post about setting up your own private cloud host. Finally Ansible the configuration management/IT automation tool, that will basically setup Docker and other services inside the Virtual Machines, and perhaps setup your application inside Docker as well. Note that another service called Packer allows you to quickly replicate your virtual machine images so it's good for the development of virtual appliances.

So in order to learn how to do all of this, I had start at the beginning with VirtualBox. I've managed to setup VirtualBox as a guest operating system that not only has a synchronised development folder with my host operating system (Windows, Linux and Mac) but is also accessible over the network.

What does this all mean? This means I can open up my Sublime Text Editor to my "www" directory. This is where all my projects reside. I can edit a file in this directory. This file change is automatically propagated to the virtual machine instance. Then in my host machine I can open up Firefox to "http://localhost:8000" or "http://dev.test.com:8000" and get to see my updated change in my application. Then in my host or guest machine, can I commit and synchronise to github. All of this can be done without installing a web server stack (HTTP servers, Application Runtimes..etc) on my host machine, thus keeping my host machine clean. I'm not talking about production deployment at this moment, because I haven't got that far with Docker..etc. Therefore any deployment routines to production servers will still stay the same. However I'm still quite happy with what I have here, it is definitely an improvement over running WAMP or MAMP. As I can easily use NGINX and other Unix based tools. Furthermore, the instructions below can easily be adapted to setting up SSH access from your host machine too if you want to command your guest operating system from your host's terminal.

Now comes the tutorial of how to set this all up. This part does require you understand Ubuntu Linux, basic bash scripting, CLI usage, modifying the hosts file and setting up NGINX.

First download VirtualBox: https://www.virtualbox.org/wiki/Downloads. Then download Ubuntu Desktop: http://www.ubuntu.com/download/desktop

Run VirtualBox and create a new virtual machine instance. A wizard dialog will pop up. Make sure to select the correct settings for your virtual machine. The perfomance of the virtual machine will be better if you give it a static size hard drive and 2048 mb ram, 2 CPUs and 32 mb of Video Memory along with 3D acceleration. These settings can be changed later inside the GUI of VirtualBox, they make your VM more expensive to use, but it also makes your VM a lot more faster and responsive. Once your virtual machine instance is set, you need to start it up. Then load the ISO of Ubuntu image you just downloaded. Follow the instructions to install Ubuntu in your new virtual operating system.

Once it's all been installed, you need to remove the ISO CD from your virtual operating system. This can be done by shutting down your VM (virtual machine), then going into the settings of your VM inside VirtualBox and removing the CD in the storage settings. Start it up again and this time install the VBoxAdditions ISO. The VM would have prompted you to do this. This VBoxAdditions allows your VM to setup shared folders and other good things. We'll need shared folders to allow your host machine to edit files and have your guest machine use those updated files. Once that is done, you'll need to restart your VM and remove the VBoxAdditions ISO CD.

Setting up shared folders is quite easy. Shutdown your VM. Go into the VirtualBox settings for the VM. Go into shared folders. Add a new shared folder record. The first folder path is the path your development folder. For me this is "D:www". The folder name would be "www". Make sure Read-only is off, switch on Auto-mount and Make Permanent. Hit Ok. Start your VM. Using your terminal in root user assuming that your folder is "www", run:

cd /media/sf_www/
ls -l

This should show you the contents of your "www" folder on your host machine. If your folder was called something else, it will always have the prefix of "sf_". Furthermore we can see the owner and group the files and folders are associated with. This folder will be grouped under "vboxsf". Right now we can only access it with the root user. In order to add extra users to this group we can use this bash command (fill "username" with the name of the user):

sudo adduser username vboxsf

We should add the default user that you log in with, and also the "www-data" user. The "www-data" user will be the user that your web server stack will use. Your webserver such as NGINX, PHP, Cron scripts should not be using root or your default user, but a custom user for web stuff. It doesn't need to be called "www-data" but that is the default name used by PHP-FPM.

Once you do that, the setting does not take effect immediately. You need to restart your VM. After which you no longer have to be a root user to access the "sf_www" directory inside your VM.

Now the /media/sf_www is not really convenient directory path. So let's try to access it from something like /www. To do so, setup a symbolic link from /media/sf_www to /www.

ln -s /media/sf_www /www

Now test it out. Go into the "www" directory from your host machine, and create or edit a file. Now in your VM, check the /www directory and see if the file is there or edited. We should now have synchronised development folders!

Now let's do some port forwarding. First we need an HTTP server on the VM. So we can install NGINX which by default listens on port 80 and shows a default NGINX page.

sudo apt-get install nginx

Test it out by launching Firefox inside the VM and heading to "http://localhost". If it shows the page, then NGINX is working. While you're at it, try out Polycademy's NGINX configuration best practices: https://github.com/Polycademy/WebserverConfiguration

Shutdown the VM. Inside VirtualBox's VM settings, go into Network. From Network, the Network Adapter should be set to NAT. Click on Port Forwarding. Add a new record. In this record, set the Host Port to 8000 and the Guest Port to 80. The Protocol should be TCP. The name and everything else can be left as default. You can change the name to anything you like. Restart the VM.

Now in your host machine, launch your browser and head to "http://localhost:8000". This should show the same page as the page you saw when you visited with Firefox inside the VM. In the future you can also port forward HTTPS connections, and you'll need to go from a random host port such as 8001 to the port 443 on the guest machine, this would result in URLs like https://localhost:8001.

You can change the hosts file inside the host machine independently of the guest machine. This means the hosts file inside the guest machine affects the guest. So if you want a development URL such as "http://dev.example.com" inside the guest machine, just set that up inside /etc/hosts in the guest machine. This will work for browsers inside the guest machine. But if you want that for the host machine, you'll need to add that record to the hosts file in the host machine.

Now you can install all your web server stack into your VM and not worry about the host machine or incompatibilities between operating systems especially when you develop on Windows/Mac but deploy on Linux.

If you meet any problems, check the NGINX logs which should be specified inside the "nginx.conf" file inside /etc/nginx/nginx.conf. There could be issues with file caching with NGINX, which means that NGINX is caching the files it's serving. This is not good for development, and should be switched off. You should then restart NGINX with sudo service nginx restart. Other problems could include permissions like "vboxsf" but this should work great.

The next step would be learning Vagrant & Ansible to automate the VM constructions as it builds on top VirtualBox (and other various VM provisioiners). Currently all you have is a VM image you could share, but sharing a 1 gig file is a bit hard, a configuration file would be smaller and easier to work with.

I've recently began on a small client project involving a Wordpress site for Auto Timor Leste - Toyota East Timor. I don't usually work on Wordpress, but the last time I worked with it, I realised it was a clunky blogging platform that is quite difficult to work with when you want to do more than just blogging. Nowadays there are better alternatives like Docpad, Octopress and Jekyll but those are still in the early adopters stage with developers. So Wordpress does have the advantage of being widely known and easily supported by developers. It also has a whole set of plugins. So you can say it's made for the non-developer. With that in mind, I suggested to use Wordpress as Toyota's official website framework.

When I was working Wordpress a while ago, the deployment process involved downloading the official Wordpress installation via zip file. Unzipping it into your web root directory, running the initial configuration, hardcoding any configuration required in the wp-config.php file and then finally FTPing into an online server usually purchased from Lunarpages.

Such a workflow has several disadvantages. The worst is FTP. FTP is very slow. It transfers each file individually one by one, and each time it does this, it needs to reestablish the connection. This can be a real hassle when you have thousands of files, and the Wordpress 3.7.1 framework has as of now 1176 files. And if you hit any interruption in the connection, you may need to restart the whole process since you don't know which file was half uploaded and corrupted. The other problem is that server differences will require reconfiguration on the remote server. This generally means saving the config file separately, modifying it to match the necessary remote server configuration which could involve database settings and database migration, uploading it via FTP to the remote server, access the site from your web browser and debug things manually. Oh and you have do this all over again if you need to install plugins or fiddle with the theme. This leads us to our second problem: dependency management and version control.

Storing the wordpress project as files on your work computer makes it difficult to work flexibly. What if your computer dies? What if you want to work on a different computer? What if you need to share code with a team and each needs to contribute in different ways to the overall project? So you might decide to copy paste your working folder in a backup harddisk, and copy it to a USB thumbdrive so you can work on the go. Now you have the problem of version control. How do you reconcile different versions of the project elegantly. If you changed one file on the USB thumbdrive, then this becomes the new master version, which has to be manually copied to all the other locations where you stored the code. You might meet merge conflicts, where 2 files might have been updated to different code at the same time, which version is the right one to use, and what if both versions solve different problems. And this problem becomes compounded if you're working in a team. Furthermore, this constant copying and pasting results in a lot of duplication, especially if the majority of the project are external dependencies, that is plugins or themes or the Wordpress framework itself, which is code written by people outside of your current scope. There should be some way of isolating your application's code from dependencies so that there's less duplication of the source code, and you only need to bring in the dependency when it's actually being executed.

Now I am digressing, of course there has been solutions to this distributed source control via SVN and friends for large software projects, but if you're a single developer, there are still many lessons to learn from large software development workflow.

So I began looking a different, more modern way. Nowadays we have PAAS (platform as a service) businesses that offer a more streamlined way of deploying web applications. They usually operate via a customised deployment routine similar to Capistrano, Dokku, Git based deployment, or some form proprietary continuous integration. All of these systems provide deployment hooks, which is essentially automated code that is ran after the code has be transported to the remote server. This solves the FTP problem because they package up the directory you want to send into a compressed archive, and once it's uploaded, it's extracted to the web root location. This speeds up the deployment process immensely, and it's more secure as well.

But there is still the problem of configuration and dependency management. So I researched a bit and found articles regarding Wordpress and Composer and Git. Here are my sources: Using Composer with Wordpress and Deploying Wordpress using Git and Capistrano. You should read those sources before going further.

Those sites are really good sources, but they left a bit of detail out. So I'm going to fill in some gaps:

Start by getting Composer (dependency management tool) to install Wordpress as a dependency. Now Wordpress has not been registered as a package on Packagist (Composer's department store). But this is not a problem. We can use a custom installer which will bring in Wordpress as a zip file and unload it into the project directory. Here is the composer.json

{
	"repositories": [
		{
			"type": "package",
			"package": {
				"name": "wordpress",
				"type": "webroot",
				"version": "3.7.1",
				"dist": {
					"type": "zip",
					"url": "https://github.com/WordPress/WordPress/archive/3.7.1.zip"
				},
				"require": {
					"fancyguy/webroot-installer": "1.0.0"
				}
			}
		}
	],
	"require":{
		"php": ">=5.3.0",
		"wordpress": "3.7.1"
	},
	"config": {
		"bin-dir": "bin/",
		"process-timeout": "1000"
	},
	"extra": {
		"webroot-dir": "wp",
		"webroot-package": "wordpress"
	}
}

When running composer install, this will download https://github.com/WordPress/WordPress/archive/3.7.1.zip and utilise this "fancyguy/webroot-installer": "1.0.0" custom installer to extract the archive into your root directory. The "extra": { "webroot-dir": "wp", "webroot-package": "wordpress" } becomes important. The "webroot-dir" specifies what the wordpress directory's name, while the webroot-package specifies what package we want the custom installer to use. To get different versions of Wordpress, just change the "3.7.1" version to desired Wordpress version. There are three locations to change, the "version": "3.7.1", "url": "https://github.com/WordPress/WordPress/archive/3.7.1.zip" and "wordpress": "3.7.1". The Wordpress framework is downloaded from the Github mirror. Because Wordpress is a rather large dependency, I set the "process-timeout": "1000" to 1000 seconds, since a slow internet connection may cause your installation to timeout. Composer uses 300s by default. Your directory layout should look like this:

vendor
wp
composer.json
composer.lock

Now let's configure Wordpress so it understands that it's in its own directory. Follow the Using a pre-existing subdirectory install instructions here Giving Wordpress Its Own Directory. The General panel means going to http://pathtoprojectdir/wp/wp-admin, then going to Settings > General and change the Site Address Url but not the Wordpress Address Url. When you first get there, it'll ask you about the wp-config.php file, just say yes. Make sure you've got the database settings setup already, this is just like a normal Wordpress installation. After you've copied the "index.php" (note that the .htaccess file probably does not exist anymore) to your project root, you need to make the index.php's require command point to the Wordpress directory. Which going to be like this snippet:

/** Loads the WordPress Environment and Template */
require( dirname( __FILE__ ) . '/wp/wp-blog-header.php' );

You should now copy the wp-config.php or wp-config-sample.php into your project root and rename it to wp-config.php. Next copy the whole wp-content directory into your project root as well. Your project directory should look like this:

vendor
wp
wp-content
composer.json
composer.lock
index.php
wp-config.php

Now we have to configure wp-config.php so that Wordpress understands that our application code is in wp-content and the wordpress installation is in wp. Here's what my wp-config.php file looks like:

<?php

define('ENVIRONMENT', 'development');

/**
 * Automatic Url + Content Dir/Url Detection for Wordpress
 */
$document_root = rtrim(str_replace(array('/', '\'), '/', $_SERVER['DOCUMENT_ROOT']), '/');

$root_dir = str_replace(array('/', '\'), '/', __DIR__);
$wp_dir = str_replace(array('/', '\'), '/', __DIR__ . '/wp');
$wp_content_dir = str_replace(array('/', '\'), '/', __DIR__ . '/wp-content');

$root_url = substr_replace($root_dir, '', stripos($root_dir, $document_root), strlen($document_root));
$wp_url = substr_replace($wp_dir, '', stripos($wp_dir, $document_root), strlen($document_root));
$wp_content_url = substr_replace($wp_content_dir, '', stripos($wp_content_dir, $document_root), strlen($document_root));

$scheme = (isset($_SERVER['HTTPS']) AND $_SERVER['HTTPS'] != 'off' AND !$_SERVER['HTTPS']) ? 'https://' : 'http://';
$host = rtrim($_SERVER['SERVER_NAME'], '/');
$port = (isset($_SERVER['SERVER_PORT']) AND $_SERVER['SERVER_PORT'] != '80' AND $_SERVER['SERVER_PORT'] != '443') ? ':' . $_SERVER['SERVER_PORT'] : '';

$root_url = $scheme . $host . $port . $root_url;
$wp_url = $scheme . $host . $port . $wp_url;
$wp_content_url = $scheme . $host . $port . $wp_content_url;

define('WP_HOME', $root_url); //url to index.php
define('WP_SITEURL', $wp_url); //url to wordpress installation
define('WP_CONTENT_DIR', $wp_content_dir); //wp-content dir
define('WP_CONTENT_URL', $wp_content_url); //wp-content url

/**
 * Secrets
 */
require_once('Secrets.php');
Secrets::load();

/**
 * MySQL settings
 */
if(ENVIRONMENT == 'production'){

	define('DB_NAME', $_ENV['secrets']['database_name']);
	define('DB_USER', $_ENV['secrets']['database_user']);
	define('DB_PASSWORD', $_ENV['secrets']['database_pass']);
	define('DB_HOST', $_ENV['secrets']['database_host']);
	define('DB_CHARSET', 'utf8');
	define('DB_COLLATE', '');

}else{

	define('DB_NAME', 'autotimorleste');
	define('DB_USER', 'root');
	define('DB_PASSWORD', '');
	define('DB_HOST', 'localhost');
	define('DB_CHARSET', 'utf8');
	define('DB_COLLATE', '');

}

/**#@+
 * Authentication Unique Keys and Salts.
 *
 * Change these to different unique phrases!
 * You can generate these using the {@link https://api.wordpress.org/secret-key/1.1/salt/ WordPress.org secret-key service}
 * You can change these at any point in time to invalidate all existing cookies. This will force all users to have to log in again.
 *
 * @since 2.6.0
 */
define('AUTH_KEY',         $_ENV['secrets']['auth_key']);
define('SECURE_AUTH_KEY',  $_ENV['secrets']['secure_auth_key']);
define('LOGGED_IN_KEY',    $_ENV['secrets']['logged_in_key']);
define('NONCE_KEY',        $_ENV['secrets']['nonce_key']);
define('AUTH_SALT',        $_ENV['secrets']['auth_salt']);
define('SECURE_AUTH_SALT', $_ENV['secrets']['secure_auth_salt']);
define('LOGGED_IN_SALT',   $_ENV['secrets']['logged_in_salt']);
define('NONCE_SALT',       $_ENV['secrets']['nonce_salt']);

/**#@-*/

/**
 * WordPress Database Table prefix.
 *
 * You can have multiple installations in one database if you give each a unique
 * prefix. Only numbers, letters, and underscores please!
 */
$table_prefix  = 'wp_';

/**
 * WordPress Localized Language, defaults to English.
 *
 * Change this to localize WordPress. A corresponding MO file for the chosen
 * language must be installed to wp-content/languages. For example, install
 * de_DE.mo to wp-content/languages and set WPLANG to 'de_DE' to enable German
 * language support.
 */
define('WPLANG', '');

/**
 * For developers: WordPress debugging mode.
 *
 * Change this to true to enable the display of notices during development.
 * It is strongly recommended that plugin and theme developers use WP_DEBUG
 * in their development environments.
 */
define('WP_DEBUG', false);

/* That's all, stop editing! Happy blogging. */

/** Absolute path to the WordPress directory. */
if ( !defined('ABSPATH') )
	define('ABSPATH', __DIR__ . '/wp/');

/** Sets up WordPress vars and included files. */
require_once(ABSPATH . 'wp-settings.php');

It probably looks quite different from yours. There's a quite a bit of important logic working here. Let's go through it. The first part defines the ENVIRONMENT constant. This will be useful for our branching off our database configuration. The next we do some magic URL detection. Remember the part in the General panel where you had to set the URL to the Wordpress installation and the site URL. It's probably along the lines of "http://localhost/pathtoyoursite". This will be a bit of a problem if you upload it to a remote server, and the URL will probably be your production URL having a real domain. We don't want to do so much configuration work, so I wrote some code to automatically detect the URL to the project root and other relevant areas such as:

define('WP_HOME', $root_url); //url to index.php
define('WP_SITEURL', $wp_url); //url to wordpress installation
define('WP_CONTENT_DIR', $wp_content_dir); //wp-content dir
define('WP_CONTENT_URL', $wp_content_url); //wp-content url

This allows our Wordpress project to work with any URL on any server, whether it's your development server or production server. I saved this portion of the code as a gist:

The next portion loads the Secrets class and loads secrets. What are these secrets? Well they are the production database configuration and authentication keys. These are required since it is not a good practice to save production configuration keys to version control. I will get back to this and the database/auth configuration in a moment. At the bottom, we made sure that the require command required from the "wp" directory.

/** Absolute path to the WordPress directory. */
if ( !defined('ABSPATH') )
	define('ABSPATH', __DIR__ . '/wp/');

/** Sets up WordPress vars and included files. */
require_once(ABSPATH . 'wp-settings.php');

Let's get the Secrets working. First we need a secrets directory which will hold keys.php. Something like this:

<?php

//encryption

$secrets['auth_key'] = '';
$secrets['secure_auth_key'] = '';
$secrets['logged_in_key'] = '';
$secrets['nonce_key'] = '';
$secrets['auth_salt'] = '';
$secrets['secure_auth_salt'] = '';
$secrets['logged_in_salt'] = '';
$secrets['nonce_salt'] = '';

//production database details

$secrets['database_host'] = '';
$secrets['database_name'] = '';
$secrets['database_user'] = '';
$secrets['database_pass'] = '';

The above will be inside "secrets/keys.php". This file will be ignored in version control. This is a good practice. Next we'll get the Secrets class which I wrote to facilitate the loading of secrets. It will include any php file inside the secrets folder, and pass all properties that is part of the $secrets[] array into the $_ENV['secrets'][] array. Here is a gist of the Secrets class:

The Secrets.php should be committed to version control and should be at the project root. You project structure should look like:

vendor
secrets
wp
wp-content
composer.json
composer.lock
index.php
wp-config.php
Secrets.php

Now you can see how the secrets can be utilised in production settings. On the production side, you would write a deploy hook that downloads your keys.php into the secrets folder which would be kept in a separate location that is of course secret. You can use Dropbox, private git then curl.

Now that we have separated the framework from our application code, which mostly kept in the wp-content directory, because that's where our themes and plugins go, we should go a step further and use Composer to manage any plugin dependencies. Now most Wordpress plugins are not Composer compatible. But the guys at http://wpackagist.org/ as mirrored every single Wordpress plugin as a composer installable package. All of which use a custom installer to install into the "wp-content/plugins" directory. I installed the WP-Migrate-DB plugin via Composer. You can do this by adding their location as a repository location as seen in this snippet:

	"repositories": [
		{
			"type": "composer",
			"url": "http://wpackagist.org"
		},
		{
			"type": "package",
			"package": {
				"name": "wordpress",
				"type": "webroot",
				"version": "3.7.1",
				"dist": {
					"type": "zip",
					"url": "https://github.com/WordPress/WordPress/archive/3.7.1.zip"
				},
				"require": {
					"fancyguy/webroot-installer": "1.0.0"
				}
			}
		}
	],

Then you can install the plugins in the require position:

	"require":{
		"php": ">=5.3.0",
		"wordpress": "3.7.1",
		"wpackagist/wp-migrate-db": "0.5"
	},

Now how do we find out the exact name and version of the Wordpress plugin that we want? Well you need to go into the plugin SVN directory: http://plugins.svn.wordpress.org/ and do a Ctrl+F to the plugin name, and that's the correct name to use. Click on the plugin, and you'll find the version you will want. Run composer update and this will install the plugin into "wp-content/plugins/pluginname/".

Let's now commit this to Git version control. However before we do so, we need to add some directories to the .gitignore. These ones to be specific:

#################
## Custom
#################

vendor/*
bin/*
secrets/*
!secrets/*.example
!secrets/*.gitkeep
wp/*
wp-content/backup/*
wp-content/cache/*
wp-content/plugins/*
wp-content/upgrade/*
wp-content/uploads/*

We have to ignore those files and folders because they don't belong in the version control. The secrets are self-explanatory. The wp folder is a dependency which can be brought in via Composer, there's no need to duplicate third party source code in our source control. The wp-content folders are ignored for the same reason. The uploads and backup and cache directory are actually meant to be managed outside the source control because they do not constitute the source code of the software application. Of course when you want to migrate to another server, creating a server snapshot or a database dump. For best case practices, your file uploads should go to a completely separate file sever such as Amazon S3, so the storage, backup and serving of the files can be abstracted from the logical component which is your software application source code.

So this blog post was a bit longer than I thought it would take, and it is more catered to those who already understand bits of Wordpress and how Composer works. But I hope it helps you to make your Wordpress workflow better.