Friday, December 2, 2011

ql.io is interesting.

ql.io provides the same function as the ThumbWhere web service composition layer and was written to solve the same problem with the same philosophy of putting more control in the hands of developers.

Where it differs is that like YQL it is read-only so you can't compose everything.


What is a Web Service Composition Layer?

The aim is to push the back-and-forth traffic closer to the services and to avoid the return trip latency.

As developers one big constraint is the speed of light. To send a request, get the result, send another request based on the result, all this back and forth can add many seconds to the start up time of an application.

We can't make light go any faster, but we can make the total trip shorter.

If you need to make 6 calls to different systems to complete authentication, catalogue, entitlements and analytics, you are adding a lot to your client applications start up time.


Another aim to make sure that developers are not overly constrained by workflows imposed by services. You can be more granular with your services and know that the client can compose the workflow that suits the application or platform they are using. eg. PC vs Mobile can have radically different workflows for the same services.

  • As a service designer, you don't need to second guess every case that developers may come up with.
  • As a developer and service consumer, you can construct a perfect single web service call, every time.


ql.io combines JSON and SQL


I can appreciate that it is compact, but i'm concerned that it's hard to make existing web services look like tables in a database, and that for many front end developers SQL would be a new skill they would have to acquire,.


ThumbWhere's implementaion combines JavaScript and XPATH. This was carefully chosen to take advantage of the native XPATH engines that are available on the server side and in modern web browsers, XML/XPATH is at least a familiar technology to most front end developers.

In the environment where ThumbWhere was developed JavaScript and XSLT/XPATH was and is still a heavily used language for client side development, this was a consideration of the design.

(Here is the Composition layer being called from JavaScript).


Here it is being called from C#



Some features in ThumbWhere has that it's not obvious are in ql.io (site is down at the time of writing this article).

Asynchronous call backs: As the result streamed back from the server multiple call-backs can be triggered.

Generic feed integration (you can pull in any XML feed or existing REST web service call and manipulate it, use content as parameters for other calls in the composition).

Flow control (if/while) that allows a call to for example make decisions based on the return values of other.

Response Mutation: ThumbWhere lets you move, rename and delete elements of the feed or
API responses before they are sent back to the client.

Request Parallelisation: ThumbWhere can detect or lets you mark requests that can be run
in parallel.

Write & Execute: ql.io only has has a select statement (at this point). It's read only. Thumbwhere allows you to compose read and write operations.


What ql.io does provide is a wonderful join syntax and ability to perform complex reductions based on relations - which can be done using XPATH in ThumbWhere - but is not as nice as JOIN :)

If anything this is vindication for our approach.

Its a bit hard to try and induce a new way of thinking so I'm really glad that ql.io has put a stake in the ground for saner web services. With ql.io and YQL out there now, my elevator pitch will get less blank looks :)

I'm going to strongly consider rolling this into our notification/feed/edge stack as it's such a close fit with what we already have.


Wednesday, November 23, 2011

First release of 1.1 is Live!

Greetings ThumbWhere clients and spectators.

Over the last 2 weeks we have rolled out some new services as well as a whole new API layer to support one of our longest running trial members.

The old public facing API Twitter integration demo site has been switched off. It was using the legacy (v0.0) API and it needs to be rewritten to use our latest JavaScript Service library. For now it redirects to this blog. Of note I believe it was the first tool ever that let you upload audio, images and video to Twitter from your mobile phone. Fun times :)

We will eventually replace it with our newly planned service sign-up/administration portal.

That's right folks, we are slowly on the way to going public and allowing anyone to sign up and start using our services.



Over the last 4 years we have run 8 campaigns with 6 different organisations to validate and test our product.

Just these word of mouth campaigns and other IPTV development has been keeping us busy for the last year - but we have continued to plod on with our ThumbWhere product development.

This most recent update (v1.1) represents a lot of the learnings and results of R&D from the last year.

Notable changes:

  • The v0.0 and v1.0 API's were versioned, the v1.1 API is versioned and modular using SOA principles.
  • FTPS and FTPES are now supported for first line storage as well as with CDN storage with Amazon S3 and more to come.
  • Campaign members can now email media and have it ingested into ThumbWhere with their own email address or a pre-generated ThumbWhere specific email address.
  • We have changes under the hood to improve the reporting of ingestion and transcoding workflow exceptions and new operations in the primary distributed workflow engine.
  • Auto-generated Web Service interface libraries for .NET and JavaScript. PHP and Java are on the way.
  • Auto-generated windows tools . PowerShell 'Snap-ins' for each of the new services to allow easy scripting and tool writing. Windows workflow 'Activities' are on the way.
  • Media can be FTPd directly to ThumbWhere on a per member or campaign basis.
  • A lot of work has gone into making our internal development process more agile with a lot more code generation. Now we can design, generate and stand-up a new SOA service and supporting repositories in a matter of hours.
Over the next month expect to see a lot more news about the latest v1.1 Content and Social APIs

Cheers.

Sunday, August 22, 2010

SFTP Added To ThumbWhere Storage

Checked in SFTP support.

We need to test with more SFTP servers than we have in the development environment, but this is all ready for testing.

Now for FTPS and plain old FTP.

One side effect of the AmazonS3 Storage integration is that now every time we run automated my integration tests it costs us a few fractions of a cent. :)

22-08-2010 7-45-48 AM

But it is well worth it :)

Sunday, July 4, 2010

ThumbWhere AmazonS3 Support

I just checked in the test candidate for external storage which supports Amazon S3 and FTP.

When this is deployed, API licensees can provide one or more Amazon S3 or FTP accounts and ThumbWhere will use that to store your media.

You specify an account and then a number of selectors which are used to select media for inclusion (or exclusion) with respect to a designated storage provider.

For example. You might want your videos stored in one S3 account and your thumbnails in another. In fact you might want to deploy certain thumbnails to a different type of storage altogether.

So by configuration of selectors and storage endpoints you can obtain a high degree of control over where your content is actually stored.

The system is architected to support generic storage providers via a plug-in model so I’m now investigating what others I can support in the first release. If I don’t see any easy wins then S3 and FTP will be the only ones in the first release.

I spent a good portion of Sunday writing automated tests to ensure that the core flow processor was able to handle the cases of first time deploy, revocation and update with the minimum amount of API calls and traffic. If multiple deploy instructions for the same media gets into the queue the engine is able to avoid double handling. If some of the media is modified and re-transcoded but all the media is marked for redeployment, only the media that has actually changed will be deployed.

You put a bit more thought into these things when you get a running bill from Amazon per API call and byte shipped. This weekends efforts added up to 3c :)

I’m also considering personalised storage, so actual end users of the system (members of each social network) could in theory provide their own S3 account info and have their own media shipped off to their own storage.

The media in external storage is represented by extra URL elements in the raw feed XML. The default XSLT for the account specific feeds will make this transparent by selecting a single URL and favouring external storage over files hosted on internal ThumbWhere storage.

SNAGHTML3246c319

We will start to factor this into our pricing model but the immediate effect is that this will allow you to ship most of your media traffic to cheaper storage and it will allow us to control our own storage costs.

Friday, July 2, 2010

Update



We've been rather busy for the last year and a half working in the emerging interactive television market so unfortunately there has been no time or resources to devote to the public API as I've spent most of my time either head down coding or out of the country working with some amazing organisations.

The workload has recently started to get reasonable so I'm now looking again at deploying updates to the white site social networking site to take advantage of the last year or so of work in the commercial API.

This means that finally some of the features requested in http://feedback.thumbwhere.com that have already been implemented in the API will start to see the light of day.

Also we are taking on some pro-bono clients who have expressed an interest in using the API to make the world a batter and more interesting place. These will act as example implementations for integrating with the API. At the moment as we have been doing all of the integration, public facing documentation is a little sparse. This exercise is an attempt to do some good and get some good example code. The long term aim is to start opening up the API to a general developer community.

It's amazing to see what people have been building with the API.





Thanks everyone for your ongoing support and encouragement.

Sunday, September 6, 2009

ThumbWhere Image upload now supported by MahTweets

“MahTweets is a rich, awesome, twitter client, feature inline media, filtering, tracking (saved searches) and much more”

 image

MahTweets can be downloaded from http://theleagueofpaul.com/mahtweets/

The ThumbWhere plugin was developed by @WillHughes

It supports Image uploading and uses ThumbWhere’s URL shrinking service ( tny.tw ).

You can get a ThumbWhere account at http://thumbwhere.com

Also you don’t have an account you can always MMS images, video and audio to +61-447-100-293 and ThumbWhere will publish it anonymously to the public feed.

Wednesday, April 29, 2009

ThumbWhere Launched

For some reason blogger had a massive password fail with my google account linked to this blog. That now seems to have been resolved.

ThumbWhere launched last month - 1 day early :)

Getting heaps of submissions and suggestions in http://feedback.thumbwhere.com

First update will be submission via email to allow all the iPhone users to use ThumbWhere.