Sunday, July 4, 2010
ThumbWhere AmazonS3 Support
I just checked in the test candidate for external storage which supports Amazon S3 and FTP.
When this is deployed, API licensees can provide one or more Amazon S3 or FTP accounts and ThumbWhere will use that to store your media.
You specify an account and then a number of selectors which are used to select media for inclusion (or exclusion) with respect to a designated storage provider.
For example. You might want your videos stored in one S3 account and your thumbnails in another. In fact you might want to deploy certain thumbnails to a different type of storage altogether.
So by configuration of selectors and storage endpoints you can obtain a high degree of control over where your content is actually stored.
The system is architected to support generic storage providers via a plug-in model so I’m now investigating what others I can support in the first release. If I don’t see any easy wins then S3 and FTP will be the only ones in the first release.
I spent a good portion of Sunday writing automated tests to ensure that the core flow processor was able to handle the cases of first time deploy, revocation and update with the minimum amount of API calls and traffic. If multiple deploy instructions for the same media gets into the queue the engine is able to avoid double handling. If all some of the media is modified and re-transcoded but all the media is marked to redeployment, only those thumbnails that have actually changed will be deployed.
You put a bit more thought into these things when you get a running bill from Amazon per API call and byte shipped. This weekends efforts added up to 3c :)
I’m also considering personalised storage, so actual end users of the system (members of each social network) could in theory provide their own S3 account info and have their own media shipped off to their own storage.
The media in external storage is represented by extra URL elements in the raw feed XML. The default XSLT for the account specific feeds will make this transparent by selecting a single URL and favouring external storage over files hosted on internal ThumbWhere storage.
We will start to factor this into our pricing model but the immediate effect is that this will allow you to ship most of your media traffic to cheaper storage and it will allow us to control our own storage costs.