New App Engine Pricing policy, the good the bad and the ugly.

Not many good news from the cloud recently, Amazon’s AWS had a long downtime 3 weeks ago and Google announced a much controversial new pricing model for App Engine during Google I/O 2011. I will concentrate on the later since a lot of developers keep asking what this really  means to them.
To start with, the announcement was immature and got developers by surprise. Up to now many details are sketchy and a lot of things remain to be defined (remember - devil hides in the details). I understand it was made in a hurry in order to catch up with I/O 2011, but this is not a good enough excuse. Google could just announce the basics and wait for an official announcement when they were really ready to present a well defined pricing policy, preferably after some more consultation with developers and other platform stakeholders. Yes there was a survey last February but no results were published and if I judge from users comments at the time it seems their concerns are not resolved by the new pricing policy.


Good things first :
Sure there are some good things announced so let me name a few:

• App Engine is leaving Preview status so it becomes a mainstream product thus offsetting some of the worries that Google would possibly discontinue the platform.
• It will come soon with a 99.95% uptime service level agreement for paying customers, which means it is mature enough for enterprise level applications.
• “Go” language is added to the stack along with the python and java.
• Back end (always on) servers are available now.
• High Replication Datastore prices got a haircut, probably as an incentive for developers to move their applications from Master-Slave Datastore.
• Blobstore is available now to free applications as well as back ends and other APIs. This is great since it allows new comers to experiment with all available tools and APIs before they commit to the platform.
• New interesting features are added to road-map as well as promises for some badly needed tools (sockets etc..)

Bad things :

• Free application usage is much more restricted with new limits applicable to Datastore API (max 50k operations per day), email recipients per day drastically reduced, and XMPP and channel API quotas are trimmed down drastically.
• On-demand Frontend Instances (max 24 Instance Hours). This although on paper looks good compared to 6.50 CPU hours of current quota, until we take into consideration that the new unit is instances with a minimum charge of 15 minutes per instance per use as was disclosed in the forums. This combine with the new datastore quotas makes absolutely prohibitive for any free (especially python which lacks a multi threading capability) applications to serve reliably 24 hours a day even for minimum amount of traffic and defies the promised 5000000 requests / month free (well ... that was the promise made 3+ years ago when app engine came to life) .
• Datastore departs from actual CPU cycles used model and joins a not yet defined model that charges per varius datastore operations. Although I understand the motives here (more transparency they claim and they are right, cpu usage it is hard to be understood by enterprise accountants) I do not see how they can make it measurable and transparent given the many different types of datastore operations (reads, writes, key only fetches, deletes etc. etc.). For example how will they charge for a normal fetch of 1000 entities vs an enumerated fetch where a coder trades memory for execution speed. This and many more questions remain answered by the released Pricing and Features preview table.
• Pricing based in CPU usage is over, new billing will be on a per live instance per hour basis with minimum 15 minutes (reasons given: again more transparency and inability to charge for memory used by an instance while serving). Well this is the issue that created a lot of backlash among the developer community and with very good reason. How in the world we are moving from ms pricing granularity to 15 minutes, it is beyond my imagination. This is against the long standing App Engine motto “pay as you go”. Now you got to pay going or not going, if you want to secure an instance on standby in case needed or if 2 requests happen to come at same time in a python application, your application can serve those consuming just 100ms of cpu time still you got to pay for 15 minutes of instance time while the instance will probably sit there idling for 899900 ms. This is not green computing.
• Google’s answer is that the new scheduler will take care of those things to some extend, which I really doubt, but even if this comes true still why should we have to be charged acording to schedulers efficiency which we do not control ?
• Reserved instances with a reduced pricing (an idea borrowed from AWS ?) is a new toy, we have to see how it works out but still it makes application utilization planing and billing much more complicating.

Ugly things :

1. The way the new scheme was introduced created a lot of confusion to app Engine advocates, while helped its enemies to spread a lot of FUD around. All in all it was close to a PR disaster. Some app engine engineers writing in the forums and talking in I/O 2011 helped with calming down the crowds for the time been but, I am not sure for the end result at the end of the day.
2. The tactics used left the developer community with the impression that GAE is concentrating on enterprise and abandoning developers and small business. This may be unfounded but if you take a look at the url pointing to the new pricing list you can see it written - loud and clear : “enterprise/appengine/appengine_pricing”.
3. It looks like after abandoning App Engine for business, Google tried to accommodate that project into existing App Engine platform by squeezing some of the breathing space used by existing developers.
4. New pricing model looks more like IaaS than a PaaS service which GAE claims to be.

So how really the Years Ahead for Google App engine look like ?
I have no simple answer to that and I do not want to jump to premature conclusions until the dust raised by the latest announcements settles down and more concrete pricing policy details emerge. Unfortunately this is going to take sometime, meanwhile I feel that unless there are some changes to the policies just announced and some pleasant surprises by GAE team when all this gets finalized GAE’s future looks grim.
Do not read me wrong I am an early adopter and advocate of App Engine and I want it to succeed, but I am an grown up man and can’t be turned into a fun boy applauding everything  that comes out of Googleplex.
I am disappointed but I hope things will turn better than what look like now and I do see some signs on the horizon that tell me this is happening already as engineers are trying to take back their baby from the poor sighted accountants and GAE4B group who hijacked the plane.
I fully understand that accountants do have a place in managing this business and help make it sustainable and profitable, something that will benefit Google’s shareholders and developers alike. My objection is that they do not really understand the product, its strengths and virtues, so it is up to the engineers to communicate those to them and only then finalize a pricing policy. From what I read in the groups and social media developers are ready to support the product and willing to pay double or even triple the price they are paying now, what they really do not like are new policies that ruin their work and time they spend trying hard to optimize their code.
App engines main attractions are:

• Automatic unlimited scalability. Do not spoil this by introducing policies that fight that. I am referring to reserved instances and passing responsibility of fine adjusting the scheduler to developers.
• Pay as you go. Billing by the instance especially at this high granularity is against this principle, further more it violently repositions App engine as more of VS kind of thing and closer to an IaaS rather than a PaaS service. I believe this is also a bad marketing policy because App engine can’t never compete with IaaS offers like AWS and the vast ecosystem that exists around these products. Of course App engine’s people argue that a managed environment can’t be compared to an unmanaged one but their actions make this differentiation a very difficult thing.
• Start up and small business friendly. That used to mean a smooth gradual transition path between free and per use paying system. New policies destroy this by drastically lowering the quota on free package and steeply raising the entrance fee of a paying account. This gap has to be bridged somehow. I am not talking about the $9 per month fee which is reasonable but the accumulating costs of instances running, datastore operations quota etc.. Perhaps a step to this direction that can be considered is the introduction of an intermediate pricing level between free and fully paying applications for developers who are not really ready for prime time and do not need an SSL certificate neither an SLA contract.

Steering GAE’s ship to the enterprise is not inherently wrong for most of developers, since it provides opportunities for them. But putting most of the effort to the enterprise while individual developers feel - rightly or wrongly does not really matter - abandoned is wrong and is not going to work in the long run. Not many fortune 500 type customers will join unless they know there exists a healthy and growing ecosystem around the product. I do not have the numbers but Google says there are around 100k active developers, although this is not a small number still can not be considered a game changer. So even if product development strategy is looking to big enterprise customers the timing is wrong, priority at this point in time should be given to developing the ecosystem.

I want to believe all this is a nightmare that will pass soon as GAE’s team start to understand what is happening to their ecosystem and steer clear of trouble and  my plane keeps on "flying into the clouds".

---

P.S. 
Update May 18, 2011

Google's Gregory D'alesandre has posted a "FAQ for out of preview pricing changes" where he tries to answer some of the questions, and clear up some of the mesh new policy has created. Also there are some definitions there of what consists what in new App Engine speak, looks like we got to study a new science and a brand new terminology before we can proceed.

IMHO this is a sisyphean task, new policy has opened Pandora's box with questions popping up from it in a much faster rate that can be answered.
Update May 19, 2011
Lots of talk and fighting in the forums with developers comparing App Engine vs AWS vs Rackspace vs any_other_VPS_service_on_earth.
There was not such talk before, coz App Engine looked different from those products both in terms of pricing as well as features.
Now thanks to latest news it managed to be transformed to Yet_An_Other_VPS overnight and I can not blame developers for those comparisons.
To tell you the truth I have seen that coming ever since I show SQL databases on the roadmap.
IMHO these are signs that we are on the wrong path, but .... then again who am I to give advice ?
or ... if I can quote @saidimu : "A true mark of a dysfunctional platform: in-fighting among developers who formerly only sang praises of the platform. #AppEngine"
Update May 20, 2011
Plenty of new questions waiting for replies.
A real interesting one by Raymond C :"Is MapReduce still a flexible solution on AppEngine under the new pricing model ?"
My answer : probably not, new pricing model makes mapreduce operations a no - no. Price will be prohibitive for such operation especially ones that depend  on many instances to run a job fast, unless those used to take hours rather than minutes to complete. So I guess the team can drop the "reduce" part and query based mapreduce things from roadmap, new model renders  those irrelevant for most use cases. Also drawing a "danger - high $$$" icon as a precaution next to copy/delete model buttons on control panel would be a good idea.
You can read a great, summary of what new changes bring to App engine by johnP here