Saturday, July 19, 2008

Challenges with scaling MySql on Amazon EC2 and S3

During the past few months I've talked to many Amazon EC2 and S3 users. They all face the same interesting challenge: how do I develop a scalable, transactional and data-intensive application in the EC2-S3 cloud environment?

They are experiencing first-hand the painful gap between scalable hardware (what Amazon provides) and a scalable application. Some of them are already using the GigaSpaces EC2 solution to bridge that gap.

When it comes to persistence, without exception, they are all looking at our partners at MySQL as their solution. Kudos to the MySQL team. You guys really are the scale-out database of choice!

But persistency in a cloud environment, even with a great database like MySQL, poses a challenge. A comment on a recent post on Jonathan Schwartz's (Sun CEO) blog - describes this challenge well:

MySQL is actually quite hard to host on the Amazon cloud right now because you need to somehow schedule syncronization to the Amazon S3 storage system to prevent your elastic compute servers losing all that valuable data.

The issue is creating a fast and reliable solution to persist data from EC2 to the S3 storage system. Due the problematic connectivity between the two, an asynchronous solution, which provides a "buffer" layer between the in-memory data cloud and the MySQL database, is required.

The solution involves writing to the database in the background (async), while decoupling the runtime transaction from the EC2-S3 connection overhead.

We, at GigaSpaces, believe our Enterprise Data Grid - EDG, deployed as a front-end to MySQL provides EC2-S3 users with an elegant solution to this problem.

Nati Shalom's recent post, Scaling Out MySQL, provides a detailed description of the how to leverage GigaSpaces to scale MySQL.

Friday, July 18, 2008

CloudCamp London

I'm on my way back from CloudCamp London. The event was very impressive and I had lot of fun (not only because of the beer and the pizza…)
To be honest, I did not imagine that there will be over 250 people in the room, I guess cloud is one of the hottest topics out there today.

My humbled contribution was a short session about challenges when scaling data-intensive transactional/statefull applications in the cloud (the video is available here).
When dealing with this class of applications the bottleneck is not access to hardware and memory on-demand, nor the need for scalable deployment tools, but rather the architecture itself.

Traditional tier-based middleware simply does not allow scalability on-demand in a linear manner. This present a barrier for applications to truly leverage the power of the cloud.
Sound complex to solve? it is :) , but GigaSpaces scale-out application server, packaged for the cloud comes to the rescue.

We were able to demonstrate the above as well the powerful notion of cloud portability using a live demo with our good partners from CohesiveFT whom we've just announced a partnership with. Thanks Ben! we all know how live demos in events is a risky business :)

Thanks for Adam Vile and his team from Excelian for demonstrating how simple it was to develop a GigaSpaces-based Mote Carlo Simulation on EC2. "We've just followed your Wiki" as they told us.

I've learned that the data-intensive transactional class of application no longer relates to “those fortune 500 enterprise”, this becomes a common challenge for small startups as well.
The cloud is actually one of the important reasons for that as it provides a cost-effective scalable infrastructure which enables small and medium vendors to offer reacher and more performance demanding functionality to their users.

I also learned that the idea of cloud portability makes a lot of sense. People would like to have the freedom of using their cloud vendor of choice according to their specific needs, without changing their application when moving between clouds.

This was music to my ears, as I believe Gigaspaces cloud Application Server offers an elegant solution to both of these issues.

Looking forward for the next event, Reuven and Alexis, thanks for inviting me.