blogrium

There once was a command line. It was a powerful thing. Not only could you navigate your filesystem and launch applications, you could program shell scripts to automate tasks and make convenient shortcuts.

One of the major sources of power there was the simple construct of input and output. A program could read from STDIN and could write to STDOUT, and you had a fair amount of control over rerouting them to do what you want. The most popular use of this was chaining commands together, piping output of one to the input of another.

This was infrastructure. Infrastructure that allowed simple, independent programs to be made almost exclusively for the purpose of chaining with other commands. These are commands like grep, uniq, wc, sort, etc… Very simple pieces that when used together with more “useful” programs would give you great power. They were made possible by this simple idea of input and output.

That was 1977. Twenty years later Jon Udell expressed a vision of web sites as data sources that could be reused and remixed, and a new programming paradigm that took the whole Internet as its platform. This lead Tim OReilly to ask in 2000:

What is the equivalent of the pipe in the age of the web?

There seems to be a resounding consensus that the answer is feeds. The name sounds very promising, it sort of make you imagine feeds like in the telecom world. Data coming directly to you. But this is very misleading because feeds aren’t about data coming to you. They’re about you getting that data yourself. A lot.

Polling is just dumb when you don’t have to. Especially when you have to write a polling system anytime you want to work with a feed, which means messing with the crontab or writing a daemon. Then you have to worry about caching and parsing. Client side mashups get by, but it’s a different story if you want it to be asynchronous and persistent. Then it just becomes too much work to do what could be done with these feeds and APIs. And if you want to use feeds for real-time notification, good luck.

There’s also a problem with the command line metaphor. The web is not linear. Web apps aren’t synchronous. A better metaphor would be a daemon process. How do they communicate? IPC? Actually IPC sucks. I learned that from AjaxWar. IPC is usually done with a shared queue. Amazon provides a pretty great, lesser known web service that provides just that for web apps.

Unfortunately, this just defers the polling to another node. It’s basically a fancier RSS. What we need is something more efficient, more direct and more real-time. We need to push.

This is where web hooks come in. A web hook is essentially a user specified callback via HTTP POST. This means to support web hooks, you have to somehow allow the user to specify where your application will post to and on what events. However, your application can then push data out there onto the Internet wherever your users want it, very much like rerouting STDOUT from the command line.

We’re already doing this with pingbacks on blogs. However, the difference with web hooks is that the payload is varied and the targets are obscure user specified web scripts. From there, the users can do whatever they want.

The idea here is to create new infrastructure. New opportunities. I’ve thought a lot about the possibilities if most web applications provided web hooks, and it makes me really excited. But I’m sure the possibilities extend well beyond what I can think of. Especially when they work with existing API’s and feeds.

Web hooks are easy to implement for application vendors. The hardest part is making a UI to let the user specify targets, but even that’s optional if you have an API. Otherwise, the main mechanism is making a standard HTTP POST, which is fairly trivial in most environments.

Being HTTP POST means it’s very accessible to work with for the user. Every web programmer knows how to work with POST variables. PHP hosting is practically free and widely available. I think PHP would become the most popular language for writing hook scripts, similar to commands in YubNub.

Just think about it. Writing all the boilerplate polling and parsing infrastructure just to use a feed? Or writing a little PHP script that has all the incoming data in predefined variables. And it’s real-time. And nicer on the Internet. Which do you think is going to be more likely taken advantage of when somebody gets a bright idea on how to use this data we’ve “opened up” to the user?

The Unix pipe is simple because it’s about input and output in linear order that generally happens quick enough to stare at the screen while it works. The web is very different. I think web hooks would achieve the same simplicity, and when coupled with our existing infrastructure of feeds and API’s, we’ll have even more power than what pipes gave Unix.

It will turn the web into a whole new platform. Remember, this is a simple solution that will have a huge impact. I think web hooks will be the tipping point to what’s really possible with the web.

This entry was posted on Thursday, May 3rd, 2007 at 2:39 am and is filed under Web, Ideas, Web hooks. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.