Sunday, August 22, 2010

SFTP Added To ThumbWhere Storage

Checked in SFTP support.

We need to test with more SFTP servers than we have in the development environment, but this is all ready for testing.

Now for FTPS and plain old FTP.

One side effect of the AmazonS3 Storage integration is that now every time we run automated my integration tests it costs us a few fractions of a cent. :)

22-08-2010 7-45-48 AM

But it is well worth it :)

Sunday, July 4, 2010

ThumbWhere AmazonS3 Support

I just checked in the test candidate for external storage which supports Amazon S3 and FTP.

When this is deployed, API licensees can provide one or more Amazon S3 or FTP accounts and ThumbWhere will use that to store your media.

You specify an account and then a number of selectors which are used to select media for inclusion (or exclusion) with respect to a designated storage provider.

For example. You might want your videos stored in one S3 account and your thumbnails in another. In fact you might want to deploy certain thumbnails to a different type of storage altogether.

So by configuration of selectors and storage endpoints you can obtain a high degree of control over where your content is actually stored.

The system is architected to support generic storage providers via a plug-in model so I’m now investigating what others I can support in the first release. If I don’t see any easy wins then S3 and FTP will be the only ones in the first release.

I spent a good portion of Sunday writing automated tests to ensure that the core flow processor was able to handle the cases of first time deploy, revocation and update with the minimum amount of API calls and traffic. If multiple deploy instructions for the same media gets into the queue the engine is able to avoid double handling. If some of the media is modified and re-transcoded but all the media is marked for redeployment, only the media that has actually changed will be deployed.

You put a bit more thought into these things when you get a running bill from Amazon per API call and byte shipped. This weekends efforts added up to 3c :)

I’m also considering personalised storage, so actual end users of the system (members of each social network) could in theory provide their own S3 account info and have their own media shipped off to their own storage.

The media in external storage is represented by extra URL elements in the raw feed XML. The default XSLT for the account specific feeds will make this transparent by selecting a single URL and favouring external storage over files hosted on internal ThumbWhere storage.


We will start to factor this into our pricing model but the immediate effect is that this will allow you to ship most of your media traffic to cheaper storage and it will allow us to control our own storage costs.

Friday, July 2, 2010


We've been rather busy for the last year and a half working in the emerging interactive television market so unfortunately there has been no time or resources to devote to the public API as I've spent most of my time either head down coding or out of the country working with some amazing organisations.

The workload has recently started to get reasonable so I'm now looking again at deploying updates to the white site social networking site to take advantage of the last year or so of work in the commercial API.

This means that finally some of the features requested in that have already been implemented in the API will start to see the light of day.

Also we are taking on some pro-bono clients who have expressed an interest in using the API to make the world a batter and more interesting place. These will act as example implementations for integrating with the API. At the moment as we have been doing all of the integration, public facing documentation is a little sparse. This exercise is an attempt to do some good and get some good example code. The long term aim is to start opening up the API to a general developer community.

It's amazing to see what people have been building with the API.

Thanks everyone for your ongoing support and encouragement.