In 2009, Ryan Dahl created a thing called Node.js. Fast forward 3 years and it's taken the web development world by storm. It's Github repository is the most watched repository just behind Twitter Bootstrap as of 2012 December. For those who aren't technical, Github is like an online library of (almost) all coding projects, it is used by developers to co-ordinate and collaborate with their teams, and to show off their work to potential employers and partners.
This is the one I'm most excited about. Web applications have a number of advantages over desktop applications, one of which is the ability to write once and run everywhere. One major disadvantage is that desktop applications that are downloaded and executed on your own computer, has always seemed more snappier and more events can happen at the same time. This is because the web as it stands runs on HTTP request - response model.
There is no state that is kept between requests and responses. Your computer and browser (client) sends a request to the server (my website), the server processes and outputs some content, in which it responds to your computer and then forgets you ever existed, until of course you send in another request. The connection between my server and your computer is not persistent, there is no constant communication, and hence the web is not real time and is "stateless".
Web developers innovated around this limitation by using sessions/cookies, iframes, AJAX and other interesting hacks in order to make applications seem instant and realtime. That's what you see in Facebook, where your friend's status updates are constantly pushed to your browser and you can instantly chat with your friends. 6 years ago that would have been impossible at Facebook's 1 billion user scale!
For the last few years several innovations came together in order to deal and ultimately remove this disadvantage from web applications:
Thirdly, the former omnipresent Flash died. Some may blame Apple, but I think it was just a matter of time before this proprietary multimedia platform was replaced by standards from the W3C (HTML 5). Ok I jest, Flash is not yet dead. But any company looking to the future of web applications won't be building it on Flash. Flash is still used on Youtube, but that's mainly because HTML 5 video hasn't caught up yet, but in order to satisfy iOS users, Youtube provides those videos on HTML 5 or through Apple's app. HTML 5 is not just some new markup syntax, but it is a set of protocols and application programming interfaces that gets integrated into modern browsers, (Firefox/Chrome/Opera) that allows us to create complex web applications and cross platform mobile applications like PhoneGap. The two APIs that are quite exciting and relevant to real time web is the release of Web Sockets, and the development of Web RTC.
Web Sockets allow full duplex persistent connections between the browser and the server. Previously in order to attain a semblance of real time applications, AJAX RIAs would long poll the server in order to receive updates. This kind of "comet programming" was still based on the HTTP request response model. The server could not push updates to the browser, so what they did is get the browser to constantly send requests to the server to ask if there is any new updates, and when there is, the browser retrieves the new update. This was understandably inefficient and complex to maintain, when there were many users on the application, it was quite easy to overload the server. There were other methods mentioned here. The point is, these methods were not truly realtime, and since it was still using the request response model, it introduces unnecessary latency and useless data being transferred with the HTTP Headers information. If only we could have a single persistent duplex (two way between client and server) protocol available... Well we do now. It's called web sockets, which is used to provide true real time communication between the server and the client. This allows us to build chat applications very easily and can allow multiplayer games, or collaborative environments such as Google Docs or Cloud 9.
A similar technology is being developed as we speak which is web RTC. This stands for real time communication. The RTC is designed for audio and video content, whereas web sockets is used for messages. When this is released in modern browsers, we'll be able to do peer to peer (your browser to another person's browser) real time communication. For example, you will no longer need Skype. Your browser could just hook up to web cam and then connect to another person's browser. Other applications involve intensive graphic streaming like videos or instant live streaming without going through a third party service. Your radio could replaced with just a browser. You could share files with your friend just by launching your browser. Now why can't we use web sockets for this? Well web sockets operate over TCP, and it emphasises reliability over timeliness. When you're streaming video or audio, you won't notice a missing pixel, but you will notice lag, so web RTC operates over the RTP while using the UDP protocol, this tolerates packet losses, so your information won't be corrupted just because a pixel disappeared in the vast internet while streaming your video. The web RTC is still very experimental and early stages, perhaps one day it may superseded web sockets, but it shows the direction of where the web is going, more distributed and more interconnected.
Welcome to the real time web. The advantages of using client heavy architecture is that your entire server is just an API, and you can create clients to suit different platforms, yet still use the same server APIs. Your mobile app, your web app, your ipad app, your desktop app, your mashup all could be designed differently and have different styles, but your server doesn't care. It doesn't need to worry about all of that. It just needs to perform CRUD on the RESTful requests or socket requests. The end user, your customer doesn't care about the magic being performed and the back end, the user experience on the front end, in his/her browser is what matters. The philosophy with single page apps is to make the front end beautiful and awesome, and keep the data and any heavy CPU processing on the back end and keep it invisible and flexible.
So coming back after that whirlwind tour of the development of the web, what does this all mean for Node.js and why is it relevant?
Wait! Let's analyse why Node.js is so suited to the real time web. Node.js works on event driven asynchronous programming. Most programming languages are procedural and imperative. This means code is evaluated in a synchronous manner, that is one by one, line by line. In order to handle concurrent connections, you need to be able to write asynchronous code. Code is evaluated simultaneously. For example, imagine a web application that is connected to 5 different separate independent web services. Perhaps a social mashup pulling in updates from multiple Facebook and Twitter clones. In synchronous programming, code is evaluated one by one, and you would need to wait for each service to return the request before moving onto the next. What happens if one of the services lag? Well your entire application hangs! Asynchronous programming allows for all the queries to be executed at the same time. It doesn't care when they return data, but they do, they run a callback which executes the event. So you can skip slow services and keep moving on, when that slow service finally returns, it catches up with a callback. Your response time for the web app mashup is reduced to slowest query rather than the combined time of all queries. This blog post explains it a bit better. When this is connected with web sockets or long polling, the end user receives the formatted data from the other 4 web services even if the 5th one is slow, whereas in synchronous he would have to wait for all of them to finish before receiving the final output, and there goes your customer!
The web is becoming more and more real time. For this to happen it has to become more interconnected and social. I don't mean social in terms of human social, but social machines. Machines and applications will become more social and more distributed. Web applications and programming languages cannot be siloed in their respective communities, they have to communicate and be social with each other. Perhaps this will lead to the rise of the "Polyglot" programmer (see the resemblance?), or perhaps it will lead Skynet. Who knows? All I can say is that this field is going to get really interesting, and software is definitely eating the world.
Why am I blogging about this?
Mainly because I'm planning a major application called Polyhack, a destination for those wanting to organise hackathons, designathons and gamathons (lan parties) or any live event that enjoys participation from the audience. I was originally just going to build something simple to list all of Polycademy's events, but the more I thought about it, the more I realised that Polyhack would be useful to other people.