Friday, December 2, 2011

ql.io is interesting.

ql.io provides the same function as the ThumbWhere web service composition layer and was written to solve the same problem with the same philosophy of putting more control in the hands of developers.

Where it differs is that like YQL it is read-only so you can't compose everything.


What is a Web Service Composition Layer?

The aim is to push the back-and-forth traffic closer to the services and to avoid the return trip latency.

As developers one big constraint is the speed of light. To send a request, get the result, send another request based on the result, all this back and forth can add many seconds to the start up time of an application.

We can't make light go any faster, but we can make the total trip shorter.

If you need to make 6 calls to different systems to complete authentication, catalogue, entitlements and analytics, you are adding a lot to your client applications start up time.


Another aim to make sure that developers are not overly constrained by workflows imposed by services. You can be more granular with your services and know that the client can compose the workflow that suits the application or platform they are using. eg. PC vs Mobile can have radically different workflows for the same services.

  • As a service designer, you don't need to second guess every case that developers may come up with.
  • As a developer and service consumer, you can construct a perfect single web service call, every time.


ql.io combines JSON and SQL


I can appreciate that it is compact, but i'm concerned that it's hard to make existing web services look like tables in a database, and that for many front end developers SQL would be a new skill they would have to acquire,.


ThumbWhere's implementaion combines JavaScript and XPATH. This was carefully chosen to take advantage of the native XPATH engines that are available on the server side and in modern web browsers, XML/XPATH is at least a familiar technology to most front end developers.

In the environment where ThumbWhere was developed JavaScript and XSLT/XPATH was and is still a heavily used language for client side development, this was a consideration of the design.

(Here is the Composition layer being called from JavaScript).


Here it is being called from C#



Some features in ThumbWhere has that it's not obvious are in ql.io (site is down at the time of writing this article).

Asynchronous call backs: As the result streamed back from the server multiple call-backs can be triggered.

Generic feed integration (you can pull in any XML feed or existing REST web service call and manipulate it, use content as parameters for other calls in the composition).

Flow control (if/while) that allows a call to for example make decisions based on the return values of other.

Response Mutation: ThumbWhere lets you move, rename and delete elements of the feed or
API responses before they are sent back to the client.

Request Parallelisation: ThumbWhere can detect or lets you mark requests that can be run
in parallel.

Write & Execute: ql.io only has has a select statement (at this point). It's read only. Thumbwhere allows you to compose read and write operations.


What ql.io does provide is a wonderful join syntax and ability to perform complex reductions based on relations - which can be done using XPATH in ThumbWhere - but is not as nice as JOIN :)

If anything this is vindication for our approach.

Its a bit hard to try and induce a new way of thinking so I'm really glad that ql.io has put a stake in the ground for saner web services. With ql.io and YQL out there now, my elevator pitch will get less blank looks :)

I'm going to strongly consider rolling this into our notification/feed/edge stack as it's such a close fit with what we already have.


Wednesday, November 23, 2011

First release of 1.1 is Live!

Greetings ThumbWhere clients and spectators.

Over the last 2 weeks we have rolled out some new services as well as a whole new API layer to support one of our longest running trial members.

The old public facing API Twitter integration demo site has been switched off. It was using the legacy (v0.0) API and it needs to be rewritten to use our latest JavaScript Service library. For now it redirects to this blog. Of note I believe it was the first tool ever that let you upload audio, images and video to Twitter from your mobile phone. Fun times :)

We will eventually replace it with our newly planned service sign-up/administration portal.

That's right folks, we are slowly on the way to going public and allowing anyone to sign up and start using our services.



Over the last 4 years we have run 8 campaigns with 6 different organisations to validate and test our product.

Just these word of mouth campaigns and other IPTV development has been keeping us busy for the last year - but we have continued to plod on with our ThumbWhere product development.

This most recent update (v1.1) represents a lot of the learnings and results of R&D from the last year.

Notable changes:

  • The v0.0 and v1.0 API's were versioned, the v1.1 API is versioned and modular using SOA principles.
  • FTPS and FTPES are now supported for first line storage as well as with CDN storage with Amazon S3 and more to come.
  • Campaign members can now email media and have it ingested into ThumbWhere with their own email address or a pre-generated ThumbWhere specific email address.
  • We have changes under the hood to improve the reporting of ingestion and transcoding workflow exceptions and new operations in the primary distributed workflow engine.
  • Auto-generated Web Service interface libraries for .NET and JavaScript. PHP and Java are on the way.
  • Auto-generated windows tools . PowerShell 'Snap-ins' for each of the new services to allow easy scripting and tool writing. Windows workflow 'Activities' are on the way.
  • Media can be FTPd directly to ThumbWhere on a per member or campaign basis.
  • A lot of work has gone into making our internal development process more agile with a lot more code generation. Now we can design, generate and stand-up a new SOA service and supporting repositories in a matter of hours.
Over the next month expect to see a lot more news about the latest v1.1 Content and Social APIs

Cheers.