Cloud Computing with Google Application Engine

New App Engine Pricing policy, the good the bad and the ugly.

2011-05-18T03:35:00.011+03:00

Not many good news from the cloud recently, Amazon’s AWS had a long downtime 3 weeks ago and Google announced a much controversial new pricing model for App Engine during Google I/O 2011. I will concentrate on the later since a lot of developers keep asking what this really means to them.
To start with, the announcement was immature and got developers by surprise. Up to now many details are sketchy and a lot of things remain to be defined (remember - devil hides in the details). I understand it was made in a hurry in order to catch up with I/O 2011, but this is not a good enough excuse. Google could just announce the basics and wait for an official announcement when they were really ready to present a well defined pricing policy, preferably after some more consultation with developers and other platform stakeholders. Yes there was a survey last February but no results were published and if I judge from users comments at the time it seems their concerns are not resolved by the new pricing policy.

Good things first :
Sure there are some good things announced so let me name a few:

App Engine is leaving Preview status so it becomes a mainstream product thus offsetting some of the worries that Google would possibly discontinue the platform.
It will come soon with a 99.95% uptime service level agreement for paying customers, which means it is mature enough for enterprise level applications.
“Go” language is added to the stack along with the python and java.
Back end (always on) servers are available now.
High Replication Datastore prices got a haircut, probably as an incentive for developers to move their applications from Master-Slave Datastore.
Blobstore is available now to free applications as well as back ends and other APIs. This is great since it allows new comers to experiment with all available tools and APIs before they commit to the platform.
New interesting features are added to road-map as well as promises for some badly needed tools (sockets etc..)

Bad things :

Free application usage is much more restricted with new limits applicable to Datastore API (max 50k operations per day), email recipients per day drastically reduced, and XMPP and channel API quotas are trimmed down drastically.
On-demand Frontend Instances (max 24 Instance Hours). This although on paper looks good compared to 6.50 CPU hours of current quota, until we take into consideration that the new unit is instances with a minimum charge of 15 minutes per instance per use as was disclosed in the forums. This combine with the new datastore quotas makes absolutely prohibitive for any free (especially python which lacks a multi threading capability) applications to serve reliably 24 hours a day even for minimum amount of traffic and defies the promised 5000000 requests / month free (well ... that was the promise made 3+ years ago when app engine came to life) .
Datastore departs from actual CPU cycles used model and joins a not yet defined model that charges per varius datastore operations. Although I understand the motives here (more transparency they claim and they are right, cpu usage it is hard to be understood by enterprise accountants) I do not see how they can make it measurable and transparent given the many different types of datastore operations (reads, writes, key only fetches, deletes etc. etc.). For example how will they charge for a normal fetch of 1000 entities vs an enumerated fetch where a coder trades memory for execution speed. This and many more questions remain answered by the released Pricing and Features preview table.
Pricing based in CPU usage is over, new billing will be on a per live instance per hour basis with minimum 15 minutes (reasons given: again more transparency and inability to charge for memory used by an instance while serving). Well this is the issue that created a lot of backlash among the developer community and with very good reason. How in the world we are moving from ms pricing granularity to 15 minutes, it is beyond my imagination. This is against the long standing App Engine motto “pay as you go”. Now you got to pay going or not going, if you want to secure an instance on standby in case needed or if 2 requests happen to come at same time in a python application, your application can serve those consuming just 100ms of cpu time still you got to pay for 15 minutes of instance time while the instance will probably sit there idling for 899900 ms. This is not green computing.
Google’s answer is that the new scheduler will take care of those things to some extend, which I really doubt, but even if this comes true still why should we have to be charged acording to schedulers efficiency which we do not control ?
Reserved instances with a reduced pricing (an idea borrowed from AWS ?) is a new toy, we have to see how it works out but still it makes application utilization planing and billing much more complicating.

Ugly things :

The way the new scheme was introduced created a lot of confusion to app Engine advocates, while helped its enemies to spread a lot of FUD around. All in all it was close to a PR disaster. Some app engine engineers writing in the forums and talking in I/O 2011 helped with calming down the crowds for the time been but, I am not sure for the end result at the end of the day.
The tactics used left the developer community with the impression that GAE is concentrating on enterprise and abandoning developers and small business. This may be unfounded but if you take a look at the url pointing to the new pricing list you can see it written - loud and clear : “enterprise/appengine/appengine_pricing”.
It looks like after abandoning App Engine for business, Google tried to accommodate that project into existing App Engine platform by squeezing some of the breathing space used by existing developers.
New pricing model looks more like IaaS than a PaaS service which GAE claims to be.

So how really the Years Ahead for Google App engine look like ?
I have no simple answer to that and I do not want to jump to premature conclusions until the dust raised by the latest announcements settles down and more concrete pricing policy details emerge. Unfortunately this is going to take sometime, meanwhile I feel that unless there are some changes to the policies just announced and some pleasant surprises by GAE team when all this gets finalized GAE’s future looks grim.
Do not read me wrong I am an early adopter and advocate of App Engine and I want it to succeed, but I am an grown up man and can’t be turned into a fun boy applauding everything that comes out of Googleplex.
I am disappointed but I hope things will turn better than what look like now and I do see some signs on the horizon that tell me this is happening already as engineers are trying to take back their baby from the poor sighted accountants and GAE4B group who hijacked the plane.
I fully understand that accountants do have a place in managing this business and help make it sustainable and profitable, something that will benefit Google’s shareholders and developers alike. My objection is that they do not really understand the product, its strengths and virtues, so it is up to the engineers to communicate those to them and only then finalize a pricing policy. From what I read in the groups and social media developers are ready to support the product and willing to pay double or even triple the price they are paying now, what they really do not like are new policies that ruin their work and time they spend trying hard to optimize their code.
App engines main attractions are:

Automatic unlimited scalability. Do not spoil this by introducing policies that fight that. I am referring to reserved instances and passing responsibility of fine adjusting the scheduler to developers.
Pay as you go. Billing by the instance especially at this high granularity is against this principle, further more it violently repositions App engine as more of VS kind of thing and closer to an IaaS rather than a PaaS service. I believe this is also a bad marketing policy because App engine can’t never compete with IaaS offers like AWS and the vast ecosystem that exists around these products. Of course App engine’s people argue that a managed environment can’t be compared to an unmanaged one but their actions make this differentiation a very difficult thing.
Start up and small business friendly. That used to mean a smooth gradual transition path between free and per use paying system. New policies destroy this by drastically lowering the quota on free package and steeply raising the entrance fee of a paying account. This gap has to be bridged somehow. I am not talking about the $9 per month fee which is reasonable but the accumulating costs of instances running, datastore operations quota etc.. Perhaps a step to this direction that can be considered is the introduction of an intermediate pricing level between free and fully paying applications for developers who are not really ready for prime time and do not need an SSL certificate neither an SLA contract.

Steering GAE’s ship to the enterprise is not inherently wrong for most of developers, since it provides opportunities for them. But putting most of the effort to the enterprise while individual developers feel - rightly or wrongly does not really matter - abandoned is wrong and is not going to work in the long run. Not many fortune 500 type customers will join unless they know there exists a healthy and growing ecosystem around the product. I do not have the numbers but Google says there are around 100k active developers, although this is not a small number still can not be considered a game changer. So even if product development strategy is looking to big enterprise customers the timing is wrong, priority at this point in time should be given to developing the ecosystem.

I want to believe all this is a nightmare that will pass soon as GAE’s team start to understand what is happening to their ecosystem and steer clear of trouble and my plane keeps on "flying into the clouds".

P.S.
Update May 18, 2011

Google's Gregory D'alesandre has posted a "FAQ for out of preview pricing changes" where he tries to answer some of the questions, and clear up some of the mesh new policy has created. Also there are some definitions there of what consists what in new App Engine speak, looks like we got to study a new science and a brand new terminology before we can proceed.

IMHO this is a sisyphean task, new policy has opened Pandora's box with questions popping up from it in a much faster rate that can be answered.
Update May 19, 2011
Lots of talk and fighting in the forums with developers comparing App Engine vs AWS vs Rackspace vs any_other_VPS_service_on_earth.
There was not such talk before, coz App Engine looked different from those products both in terms of pricing as well as features.
Now thanks to latest news it managed to be transformed to Yet_An_Other_VPS overnight and I can not blame developers for those comparisons.
To tell you the truth I have seen that coming ever since I show SQL databases on the roadmap.
IMHO these are signs that we are on the wrong path, but .... then again who am I to give advice ?
or ... if I can quote @saidimu : "A true mark of a dysfunctional platform: in-fighting among developers who formerly only sang praises of the platform. #AppEngine"
Update May 20, 2011
Plenty of new questions waiting for replies.
A real interesting one by Raymond C :"Is MapReduce still a flexible solution on AppEngine under the new pricing model ?"
My answer : probably not, new pricing model makes mapreduce operations a no - no. Price will be prohibitive for such operation especially ones that depend on many instances to run a job fast, unless those used to take hours rather than minutes to complete. So I guess the team can drop the "reduce" part and query based mapreduce things from roadmap, new model renders those irrelevant for most use cases. Also drawing a "danger - high $$$" icon as a precaution next to copy/delete model buttons on control panel would be a good idea.
You can read a great, summary of what new changes bring to App engine by johnP here

Google Maps API rate limiting and Google App Engine again

2011-04-09T01:29:00.000+03:00

@bFlood : Maps Premium is an expensive service usually used by not public-facing, password protected Web sites, while what we are talking about here is the free Google Maps service, where G is penalizing all Apps running on top of GAE since they have to share the ip addresses pool of GAE ip adresses.
@Ikai : what you write applies to old maps (V2) API where you can obtain an application authorization key and applications are rate limited based on this key, although I feel that some kind of ip based limitations exists also.
Maps V3 which is the way to go especially for mobile appls, do not require an application key, instead there are rate limitations based just on originating IP addresses, this puts GAE based appls on a disadvantage since we have to share this with all other GAE based appls using the service.
(see : post publised in App Engine group)

App Engine's Email service future

2011-03-25T02:51:00.003+02:00

From time to time I have seen posts in app engine's groups of people complaining for Email service glitches. Some of those real some not so. Sometimes coming from people who did not bother to read the documentation or just ignorant of what an Email service is all about.
App Engines team response alarmed me a little.
IMHO it is not a good policy for GAE to abandon(?) services middle way
instead of improving - enhancing those.
so I wrote the following post and now I am waiting to see how the conversation evolves.

@Icai
I agree on most of what you write above, and I understand that you prefer to focus on more important things, also having run Email services for enterprises in the past I do know it is not trivial.
But ....
still I believe Email service is a major asset for GAE and dropping it (or anything to that effect) will constitute a major blow to App Engine.
Gae offers a limited subset of services compared to what a LAMP box or a IaaS box can offer but been a PaaS provides trouble free operation and automatic scalability.
Email service usually is part of any web operation so by dropping it out you farther limit the number of potential applications that fit well into what GAE offers.
Of course, developers can look into alternative options but this makes our life difficult since we have to integrate several other third party services in order to make a working web solution i.e. setting up multiple accounts, feed traffic back and forth to other services, having to monitor and deal with one more possible point of failure. All this defies to some extend the benefits of GAE as a PaaS.
Also, dropping a service in a time when competition is adding services, will sent the wrong signal to App Engine's developers/users echo system and having in mind that G is associated with the best email service can possibly turn into a PR disaster.

Further more, been a regular reader of the groups and following App Engine since the very early days I do not see that Email service has raised a lot of issues. I believe for most people who know what they are doing and do not abuse the service it works quite smoothly. Some of the issues raised (mainly spam flagging)
a) happen to the best of Email services b) are addressed by well known techniques and practices described by others here and elsewhere.
In conclusion:
I would welcome any measure taken to fight service abuse like using GAE primarily as a mail server - we all understand that this is not what GAE is all about.
Instead of dropping the service I would prefer to consider:
a) put false positive spam flagging issues under the responsibility of developers.
b) exclude the service or part of it (like delivery assurances) from the future SLA offer.
c) think about the technical possibility to integrate it to gmail which is the *most* reliable email service in town.

Selecting distinct entities across a large table

2011-02-15T01:53:00.003+02:00

I have faced this kind of problem some time ago.
I tried some of the solutions suggested below (in memory sort and filtering, encoding things into keys etc. and I have benchmarked those for both latency and cpu cycles using some test data around 100K entities)
An other approach I have taken is encoding the date as an integer (day since start of epoch or day since start of year, same for hour of day or month depending on how much detail you need in your output) and saving this into a property. This way you turn your date query filter into an equality only filter which does not even needs to specify an index) then you can sort or filter on other properties.
Benchmarking the latest solution I have found that when the filtered result set is a small fraction of the unfiltered original set, is 1+ order of magnitude faster and cpu-eficient. Worst case when no reduction of the result set due to filtering the latency and cpu usage was comparable to the previous solutions)

Hope this helps, or did I missed something ?
Happy coding-:)

On Feb 14, 11:51 pm, Stephen Johnson wrote:
> Okay I think I got something that might work. Reverse the StartDate and
> CarId for the key from what I said above so the key would look like this:
> 2011:02:14:17:13:33:123 and the KEYS ONLY query then is:
>
> select __key__ where __key__ >= MAKEKEY(StartDate + CarId) && __key__ <=> MAKEKEY(EndDate + CarId) order by __key__ DESC
>
> Now, you can use the Async query to start processing. You're going to get
> entries that you're not interested in but you're only getting the key field
> back and not the whole CarJourney entry and this key/id has the Date and Car
> ID, so the first time you hit a Car ID for each Car then you have the ID for
> the latest CarJourney for that car. Now, once you've found all car ID's your
> looking for you can abort the query or you'll reach the end of the query
> results. Now, as you're looping, store the KEYs of the entries your looking
> for and then do a batch GET on memcache to retrieve as many Car (you've got
> the car id) and CarJourney (you've got the carjourney id) entries that might
> be stored there and then for any that you didn't get from memcache, you can
> do a batch GET on the datastore using the keys/ids that you have.
>
> I think that if you memcache things appropriately and use the batch gets for
> memcache and datastore then this might just work for you.
>
> Let me know what you think. It's an interestng problem,
> Stephen
>
> On Mon, Feb 14, 2011 at 2:12 PM, Stephen Johnson wrote:
>
>
>
>
>
>
>
> > Or maybe it blocks on different result sets just not on getting the next
> > fetch block?? Hmmm. Sounds like a tough problem.
>
> > On Mon, Feb 14, 2011 at 2:09 PM, Stephen Johnson wrote:
>
> >> Are you using .asList (which I think blocks like you describe), but I
> >> thought asIterable or asIterator wasn't suppose to. (if you're using Java).
>
> >> On Mon, Feb 14, 2011 at 12:38 PM, Edward Hartwell Goose <> >> ed.go...@gmail.com> wrote:
>
> >>> Hi Calvin & Stephen,
>
> >>> Thanks for the ideas.
>
> >>> Calvin:
> >>> We can't do the filtering in memory. We potentially have a car making
> >>> a journey (the car analogy isn't so good...) making a journey every 3
> >>> seconds, and we could have up to 2,000 cars.
>
> >>> We need to be able to look back up to 2 months, so it could be up to
> >>> 1.8 billion rows in this table.
>
> >>> Stephen:
> >>> That's an interesting idea. However the Asynchronous api actually
> >>> fires the requests synchronously, it just doesn't block. (Or at least,
> >>> that's my experience).
>
> >>> So, at the moment we fire off 1 query (which Google turns into 2) for
> >>> each site. And although the method call returns instantly, it still
> >>> takes ~5 seconds in total with basic test data. If each call takes
> >>> 12ms, we still have to wait 24 seconds for 2,000 sites.
>
> >>> So, the first call starts at time 0, the second call starts at 0+12,
> >>> the third at 0+12+12... etc. With 2,000 sites, this works out about 24
> >>> seconds. Once you've added in the overheads and getting the list of
> >>> Cars in the first place, it's too long.
>
> >>> If we could start even 100 queries at the same time of time 0, that'd
> >>> be superb. We thought we could do it with multithreading, but that's
> >>> not allowed on App Engine.
>
> >>> Finally - I've also posted this on StackOverflow -
>
> >>>http://stackoverflow.com/questions/4993744/selecting-distinct-entitie...
>
> >>> I'll try and keep both updated.
>
> >>> Any more thoughts welcome!
> >>> Ed
>
> >>> On Feb 14, 6:47 pm, Calvin wrote:
> >>> > Can you do filtering in memory?
>
> >>> > This query would give you all of the journeys for a list of cars within
> >>> the
> >>> > date range:
> >>> > carlist = ['123','333','543','753','963','1236']
> >>> > start_date = datetime.datetime(2011, 1, 30)
> >>> > end_date = datetime(2011, 2, 10)
>
> >>> > journeys = Journey.all().filter('start >', start_date).filter('start
> >>> <',> >>> > end_date).filter('car IN', carlist).order('-start').fetch(100)
> >>> > len(journeys)
> >>> > 43 # <- since it's less than 100 I know I've gotten them all>
> >>> > then since the list is sorted I know the first entry per car is the
> >>> most
> >>> > recent journey:
>
> >>> > results = {}
> >>> > for journey in journeys:
> >>> > ... if journey.car in results:
> >>> > ... continue
> >>> > ... results[journey.car] = journey
>
> >>> > len(results)
> >>> > 6
>
> >>> > for result in results.values():
> >>> > ... print("%s : %s" % (result.car, result.start))
> >>> > 753 : 2011-02-09 12:38:48.887976
> >>> > 1236 : 2011-02-06 13:59:35.221003
> >>> > 963 : 2011-02-08 14:03:54.587609
> >>> > 333 : 2011-02-09 10:40:09.466700
> >>> > 543 : 2011-02-09 15:28:53.197123
> >>> > 123 : 2011-02-09 14:09:02.680870
>
> >>> --
> >>> You received this message because you are subscribed to the Google Groups
> >>> "Google App Engine" group.
> >>> To post to this group, send email to google-appengine@googlegroups.com.
> >>> To unsubscribe from this group, send email to
> >>> google-appengine+unsubscribe@googlegroups.com.
> >>> For more options, visit this group at
> >>>http://groups.google.com/group/google-appengine?hl=en.

Re: Policy for instance startup

2010-11-24T01:07:00.000+02:00

Not been a Googler can't help much with this.
Having said that, I suspect there is a kind of build in algorithm that
does some kind of application profiling taking into acount QPS,
response times, and other parameters which adjusts instance life time,
number of instances to start etc..
This could possibly explain the difference in behaviour between your
staging and production appls.

happy coding;-)

On Nov 23, 11:58 am, Tomas Alaeus <tala...@gmail.com> wrote:
> I'm curious when exactly instances are started. I have two
> applications running on GAE, one of them have billing enabled. The one
> with billing enabled have been stress tested and have at most started
> 100 simultaneous instances. The other is just for testing and staging
> purposes and have never handeled much traffic.
>
> What I experience is that the staging server never starts more
> instances than needed. If a single person views pages it will never
> load more than a single instance. The other one however seems to start
> about 5 instances before anyone can get hot responses, and it will
> continue to start up to about 10 before realizing that ~1 QPS isn't
> that much traffic (the requests finish in about 100ms each).
>
> So, why does GAE boot up lots of instances even though 1 instance can
> serve the incoming traffic without a problem (the requests doesn't
> even overlap, so no waiting is needed)?
>
> I realize that this isn't a very big issue, since when it gets lots of
> traffic it will indeed need all the instances. I'm just curious why it
> happens.

World Countries and IP geocoding API for App Engine

2010-11-14T23:32:00.000+02:00

I have been writing here again and again about the inherent problem that App Engine based Applications have in using third party API’s with quota limits based on ip addresses since all of those are served from the same block of ips alocated by Google © and therefore have to share those quotas with other applications hosted on App Engine using the same service.

Today I am happy to offer a free IP geocoding and world countries information service API and hope this solves to some extend the problem of Server side IP geocoding for fellow App Engine developers.

Detailed service description and more information is provided here : "World countries API"

Happy coding:-)

maximum number of requests that may be handled in a single process lifetime

2010-10-27T17:59:00.000+03:00

I was doing some load tests on app engine today when I noticed a new Info message in the logs: "After handling this request, the process that handled this request reached the maximum number of requests that may be handled in a single process' lifetime, and exited normally."

So what that supposed to mean ?
Up to know we new that application instances are automatically terminated after some inactivity time out. If I understand this message well now we know that a process can be terminated after handling so many requests. How many exactly ? is this a new magic number ? Lets hope we will have some definite answers from the always helpful App Engines team.

I am doing some latency optimization (lazy imports) in this app based on the assumption that an instance will stay live as long as there are more requests coming in and I was scared this new number spoils my optimization logic. After some more testing I have found out that this is not true since whatever this number is it must be in the order of thousands requests - so no big trouble in my use case. Still I think App Engine's developers deserve to know more about such parameters, it helps both developers and the platform.

Downloan app code feature or bug ? - download_app

2010-10-23T22:43:00.000+03:00

I think App Engines developers community has by a vast majority rejected the idea of code downloading at least as default (opt out) option.
I also do not like the idea of a payable service, since it :

will complicate the pricing model
will attract criticism against the platform and
help guys who are in the business of doing unreasonable GAE vs S3 vs whatever_looks_like_cloud comparisons happy.

But ... then again who am I to tell mother G what to do ? -:)
( see my post at google-appengine group )

On Oct 23, 10:00 pm, "A. Stevko" <andy.ste...@gmail.com> wrote:
> IMO, I think source code download is a great disaster recovery option that
> should have a $$$ price tag associated with it.
> On Sat, Oct 23, 2010 at 5:08 AM, Tim Hoffman <zutes...@gmail.com> wrote:
> > Hi
>
> > This was nearly introduced, and the community overwhelmingly rejected
> > the proposal.
> > There are a number of issues that such a facility introduces.
>
> > Using a shared fileservice or source code control (actually a much
> > better strategy) is what
> > you should be using.
>
> > I don't think Ikai was being humorous. It might be worth reviewing
> > this thread to see just how negative the facility was received.
>
> > Rgds
>
> > Tim Hoffman
>
> > On Oct 23, 9:43 am, mykhal <michal.bo...@gmail.com> wrote:
> > > On Oct 18, 10:26 pm, "Ikai Lan (Google)" <ikai.l+gro...@google.com<ikai.l%2Bgro...@google.com>
>
> > > wrote:
>
> > > > Have you looked into Dropbox?
>
> > > >https://www.dropbox.com/
>
> > > > There is a free offering.

Instances console in App engine's Admin Tools

2010-10-15T15:07:00.000+03:00

It seems a new feature has just been rolled out in App Engine's production Admin tools - instances - it shows a view of total running instances along with QPS, Latency and memory used Memory as well as averages for the above values.
There is also a summary of the above in application's dashboard.
It is a very useful feature that will help a lot with application performance monitoring and resolving issues on when and how many new instances are started and killed.
Well done dev team. :-)

Yahoo claims about spam filtering

2010-09-17T04:22:00.000+03:00

From Techcrunch :
Yahoo claims -55% less spam than Gmail. 40% less than hotmail.
I am sure that if Yahoo have not been an MS partner the numbers would have been reversed.
Something like 55% less spam than hotmail. 40% less than Gmail.
Any way ... lets wait to hear what MS have to say about those figures.

App Engine scalability issues

2010-09-16T01:06:00.000+03:00

There is an issue going on for sometime now about App Engine's scaling capability.
We know that all App Engine's user facing HTTP Requests should be processed within a time-frame of 30 seconds, other wise will throw an exception. That is a rule that started with App Engine's introduction some two and half years ago, and until some time it was the only law applicable as far as I know. Then some months ago a new magic number has been introduced in group posts. It started as 1000ms within which an application has to respond other wise it could not scale properly (No new threads will be started to cope with increasing demand) later this number was lowered to 800ms.
I have raised this issue a month ago here :
" This "800ms" rule started as 1000ms some time ago, now it moved to 800ms and 400ms enters the scene. I am afraid it has became a moving target approaching 0ms too fast. Somebody must stop the bar somewhere."
But I got no reply.
In that thread even "sub-400ms" as optimal number is mentioned by a Ikai L.
Today it turns out nobody stopped the bar, instead this magic number has decreased even further to 700ms which turns my above mentioned prophecy of approaching 0ms a step closer to true. I do agree with all people in this group who complain that this is troublesome and concerning. Main advantage of App Engine is scalability, but this number kills scalability in most practical application scenarios.
It is understandable to raise the bar a little to reflect for advances in datastore latency but talking about 700 and sub 400 ms renders App Engine's scalability irrelevant for most if not all practical use cases.
There are other issues here that I am sure concern a lot of us poor App Engine's developers:

1) The always changing numbers.-I think it is reasonable to expect that an application put into production a year ago should have the same or better scalability behaviour today or a year from now. I know we operate on a Beta platform so we have to take some risk and follow up and improve our applications when Google changes the parameters under which we operate but we will never be able to catch up with this kind of drastic changes.
2) Transparency - We should be informed about those limits, so we know in advance if what we have in mind is doable with App Engine. I understand there can be some trade secrets for engineering and/or marketing reasons, but still we must be in the loop and at least know basic things that affect our Appls.
Having complained about this I want also to be fair to App Engine and its team. Up to know the usual practice is that most of those limits that affect our applications (quotas etc.) are improving over time not vice versa. This is an exception and I think it must be rectified.

Bing search returns a stack trace

2010-09-09T18:31:00.004+03:00

Haven't seen that before ! bing search returned (just once) following stack trace after a manual search for: where are highest mountains in the world. So you see it is not only us small individual developers that can force a stupid piece of code to execute, bi(n)g guys are quite capable to make it happen too.

Server Error in '/search' Application.

An invalid reloadable resource was passed.

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.ArgumentException: An invalid reloadable resource was passed.

Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Stack Trace:

[ArgumentException: An invalid reloadable resource was passed.]
Microsoft.Search.Frontend.Configuration.CompositeConfigStoreRequestInitializer..ctor(IConfigStore configStore, ReloadableResourceStore reloadableResourceStore) in e:\bt\986564\private\frontend\snr\Core\Configuration\CompositeConfigStoreRequestInitializer.cs:24
BuildUp_Microsoft.Search.Frontend.Configuration.CompositeConfigStoreRequestInitializer(IBuilderContext ) +609
Microsoft.Practices.ObjectBuilder2.BuildPlanStrategy.PreBuildUp(IBuilderContext context) in e:\Builds\Unity\UnityTemp\Compile\Unity\Src\ObjectBuilder\Strategies\BuildPlan\BuildPlanStrategy.cs:40
Microsoft.Practices.ObjectBuilder2.StrategyChain.ExecuteBuildUp(IBuilderContext context) in e:\Builds\Unity\UnityTemp\Compile\Unity\Src\ObjectBuilder\Strategies\StrategyChain.cs:86

[BuildFailedException: The current build operation (build key Build Key[Microsoft.Search.Frontend.Configuration.CompositeConfigStoreRequestInitializer, null]) failed: An invalid reloadable resource was passed. (Strategy type BuildPlanStrategy, index 3)]
Microsoft.Practices.ObjectBuilder2.StrategyChain.ExecuteBuildUp(IBuilderContext context) in e:\Builds\Unity\UnityTemp\Compile\Unity\Src\ObjectBuilder\Strategies\StrategyChain.cs:112
Microsoft.Practices.ObjectBuilder2.Builder.BuildUp(IReadWriteLocator locator, ILifetimeContainer lifetime, IPolicyList policies, IStrategyChain strategies, Object buildKey, Object existing) in e:\Builds\Unity\UnityTemp\Compile\Unity\Src\ObjectBuilder\Builder.cs:61
Microsoft.Practices.Unity.UnityContainer.DoBuildUp(Type t, Object existing, String name) in e:\Builds\Unity\UnityTemp\Compile\Unity\Src\Unity\UnityContainer.cs:463

[ResolutionFailedException: Resolution of the dependency failed, type = "Microsoft.Search.Frontend.Configuration.IConfigStoreRequestInitializer", name = "". Exception message is: The current build operation (build key Build Key[Microsoft.Search.Frontend.Configuration.CompositeConfigStoreRequestInitializer, null]) failed: An invalid reloadable resource was passed. (Strategy type BuildPlanStrategy, index 3)]
Microsoft.Practices.Unity.UnityContainer.DoBuildUp(Type t, Object existing, String name) in e:\Builds\Unity\UnityTemp\Compile\Unity\Src\Unity\UnityContainer.cs:475
Microsoft.Practices.Unity.UnityContainer.Resolve(Type t, String name) in e:\Builds\Unity\UnityTemp\Compile\Unity\Src\Unity\UnityContainer.cs:155
Microsoft.Practices.Unity.UnityContainerBase.Resolve() in e:\Builds\Unity\UnityTemp\Compile\Unity\Src\Unity\UnityContainerBase.cs:466
Microsoft.Search.Frontend.CoreUX.MvcApplication.b__a(IUnityContainer c) in e:\bt\986564\private\frontend\Serp\app\Global.asax.cs:775
Microsoft.Practices.Unity.StaticFactory.<>c__DisplayClass1`1.b__0() in e:\Builds\Unity\UnityTemp\Compile\Unity\Src\Unity.StaticFactory\StaticFactoryExtension.cs:42
Microsoft.Practices.Unity.StaticFactory.FactoryDelegateBuildPlanPolicy.BuildUp(IBuilderContext context) in e:\Builds\Unity\UnityTemp\Compile\Unity\Src\Unity.StaticFactory\FactoryDelegateBuildPlanPolicy.cs:36
Microsoft.Practices.ObjectBuilder2.BuildPlanStrategy.PreBuildUp(IBuilderContext context) in e:\Builds\Unity\UnityTemp\Compile\Unity\Src\ObjectBuilder\Strategies\BuildPlan\BuildPlanStrategy.cs:40
Microsoft.Practices.ObjectBuilder2.StrategyChain.ExecuteBuildUp(IBuilderContext context) in e:\Builds\Unity\UnityTemp\Compile\Unity\Src\ObjectBuilder\Strategies\StrategyChain.cs:86

[BuildFailedException: The current build operation (build key Build Key[ConfigStoreService.IConfigStoreRequest, null]) failed: Resolution of the dependency failed, type = "Microsoft.Search.Frontend.Configuration.IConfigStoreRequestInitializer", name = "". Exception message is: The current build operation (build key Build Key[Microsoft.Search.Frontend.Configuration.CompositeConfigStoreRequestInitializer, null]) failed: An invalid reloadable resource was passed. (Strategy type BuildPlanStrategy, index 3) (Strategy type BuildPlanStrategy, index 3)]
Microsoft.Practices.ObjectBuilder2.StrategyChain.ExecuteBuildUp(IBuilderContext context) in e:\Builds\Unity\UnityTemp\Compile\Unity\Src\ObjectBuilder\Strategies\StrategyChain.cs:112
Microsoft.Practices.ObjectBuilder2.Builder.BuildUp(IReadWriteLocator locator, ILifetimeContainer lifetime, IPolicyList policies, IStrategyChain strategies, Object buildKey, Object existing) in e:\Builds\Unity\UnityTemp\Compile\Unity\Src\ObjectBuilder\Builder.cs:61
Microsoft.Practices.Unity.UnityContainer.DoBuildUp(Type t, Object existing, String name) in e:\Builds\Unity\UnityTemp\Compile\Unity\Src\Unity\UnityContainer.cs:463

[ResolutionFailedException: Resolution of the dependency failed, type = "ConfigStoreService.IConfigStoreRequest", name = "". Exception message is: The current build operation (build key Build Key[ConfigStoreService.IConfigStoreRequest, null]) failed: Resolution of the dependency failed, type = "Microsoft.Search.Frontend.Configuration.IConfigStoreRequestInitializer", name = "". Exception message is: The current build operation (build key Build Key[Microsoft.Search.Frontend.Configuration.CompositeConfigStoreRequestInitializer, null]) failed: An invalid reloadable resource was passed. (Strategy type BuildPlanStrategy, index 3) (Strategy type BuildPlanStrategy, index 3)]
Microsoft.Practices.Unity.UnityContainer.DoBuildUp(Type t, Object existing, String name) in e:\Builds\Unity\UnityTemp\Compile\Unity\Src\Unity\UnityContainer.cs:475
Microsoft.Practices.Unity.UnityContainer.Resolve(Type t, String name) in e:\Builds\Unity\UnityTemp\Compile\Unity\Src\Unity\UnityContainer.cs:155
Microsoft.Practices.Unity.UnityContainerBase.Resolve() in e:\Builds\Unity\UnityTemp\Compile\Unity\Src\Unity\UnityContainerBase.cs:466
Microsoft.Search.Frontend.CoreUX.MvcApplication.InitializeRms() in e:\bt\986564\private\frontend\Serp\app\Global.asax.cs:650
System.Web.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +80
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +171

Version Information: Microsoft .NET Framework Version:2.0.50727.4927; ASP.NET Version:2.0.50727.4927

Want to build voice, SMS, IM and Twitter apps in python? Tropo WebAPI library now available

2010-09-08T03:47:00.000+03:00

Want to build voice, SMS, IM and Twitter apps in python? Tropo WebAPI library now available

Best practice to import libraries or frameworks in App engine

2010-09-05T01:11:00.001+03:00

Usual procedure is to paste the library in your application's path then upload it with your application using appcfg - update.
This works but sometimes we have to deal with huge libraries consisting of hundreds of python files that sometimes exceed the upper limit of 1000 files / application limit or reduce the number of available files for your application. Also maintaining those files becomes a nightmare.
So what is the proper way ? :
Well Python's import works straight from the box with a zip file as well, if it finds it your sys.path it treats it as a directory. (for more complicated situations you can look at zipimport module)
Compress the library in a zip file.
You can optionally include an empty "__init__.py" in each directory (if not there already) so it can be treated as a normal python package.
Paste just the zip file in your application - or better put it somewhere and only paste a link to this file in your applications directory (this way you can maintain the library in a single place and use it on many applications)
Then whenever you want to import from that library you can use something like :

import sys
import os
myLibrary= os.path.dirname(__file__)+'/mylibrary.zip'
#(os.path.dirname(__file__) gives you the directory of currently executing script)
if not mylibrary in sys.path:sys.path.insert(0, mylibrary)
#(or sys.path.append(mylibrary)
import mylibrary

Thats it !

Allegro Non Troppo - Using Tropo API in App Engine

2010-09-03T02:43:00.005+03:00

I was looking for an SMS - Voice gateway for an application of mine and after thoughtfully looking into Clicatell - twilio etc. I opted for Voxeo's tropo.
Do not ask much for the reasons I am not ready for a war over which one is better or cheaper to use.
I just felt like that tropo's "develop free - pay as you go once in production" business model fits well with App Engine's model that is on the same line, same applies to application scaling which is analogous to App Engine's.

So I decided to give it a try and here are my preliminary assessment of the service:
Registration was a 30 seconds job, after which you can proceed and create an application.
Tropo is providing two different APIS a scripting API where you host your scripts in their servers and a WEB api.
Although I knew scripting would be much more easy I have chosen the WEB API for the flexibility it provides.
Setting up an application is also very easy you just provide URLS that will handle Json requests from tropo, then you get 2 keys (outbound tokens) one for voice and one for Messaging applications as well as as some telephone, sip, Inum and other numbers that you can use.
You can test those tokens right from that page if you click on those and select "launch" on the pop up, your server if set up properly will receive an HTTP GET on the URL you specified.

One note about the URLS initially i specified a URL of the form "http://www.myapp.appspot.com/trop/SMS/" it seems this form was not working then I tried something in the form "http://www.myapp.appspot.com/trop/SMS/foo.json" and it worked - but can't be sure if it is really so or I have missed something.

Now the part that puzzled me for half a day "how you initiate an action request from tropo ?" . i was very unlucky since I could not find a proper example or framework in python since this was only published just a few days ago and was not indexed by google yet, and I did not bother to ask on the tropo's forum because I was thinking I will sort it out my self in a matter of minutes. Well I did but it was a matter of many hours.
What was the problem i did'nt quite get the handshaking process and I thought I would initiate an action (i.e. sending an SMS) by sending to tropo the requesting json object, I did not even bother to check the online debugging utility as I thought this would work only for the scripting API. By the way, after i discovered it works for the web API as well, I find this a very good tool - you can find in a link at the bottom of any (?) tropo's page. Well to make a long story short you initiate a request by issuing a GET to tropo's API url then API replies with a Json post which initiates a session that is posted to your application's URL and you issue your action command by responding to this Post with a Json object.
By the time I was close enough to the solution I received and EMail from Adam Kalsey from Tropo (nice fellow who tries to be helpful all the time) who was checking how was I doing with the API, I mention something like "not much documentation for Python" and he replied right away pointing me to the framework ( http://github.com/tropo/tropo-webapi-python ). Well believe me it is a nice small MIT licenced framework by Dan York with tests and examples that will set you up and running in app engine in zero time. Pity I discovered it a little too late, but I can only blame me for that.

Lessons learned: after we deal with a problem for some time (how much ?) and before we start scratching our heads better reach out to the community, more probably than not somebody has been there before and is willing to give a hand. I know, I know I mention this same lesson some posts ago when I was dealing with panoramio API, lets hope this time I will really get my lesson.

So far so good with tropo's Web Api but, i have not done much development and testing yet. So stay tuned for more about tropo and app engine - python when i will have more to say about the service.

Understanding the concurrent requests limit

2010-08-18T00:47:00.000+03:00

This "800ms" rule started as 1000ms some time ago, now it moved to
800ms and 400ms enters the scene. I am afraid it has became a moving
target approaching 0ms too fast. Somebody must stop the bar
somewhere.

Happy coding ;-)
(From my post in : http://groups.google.com/group/google-appengine-python/browse_thread/thread/6353b6232e8851aa )

A bug in tweepy streaming API

2010-07-17T20:25:00.000+03:00

I found a bug in tweepy streaming API .
It can't filter by tag with non Latin characters.
I was astonished coz this was working when I used it some months ago (version 1.3 if I remember well).
So here is my small contribution to the project :
def filter(self, follow=None, track=None, async=False):
   params = {}
   self.headers['Content-type'] = "application/x-www-form-urlencoded"
should be :
def filter(self, follow=None, track=None, async=False):
   params = {}
   self.headers['Content-type'] = "application/x-www-form-urlencoded; charset=utf-8"
I have only tested streaming and only tag filtering - probably this applies to other parts but I am not sure.

App Engine & Google Geocoding Service II

2010-07-09T01:46:00.006+03:00

This issue comes and goes in this thread all the time.
I also suggested in the past to delegate the job to the client since I
could't see a scenario where we use server side request and be within
the TOR's.
see here : http://gaengine.blogspot.com/2010/05/google-maps-api-quotas-and-app-engine.html
I was wrong. Recently I run into a case where I had to do this and
still I am sure I am within legal limits, this is so when you need the
geolocation service in order to publish a static map served from
Google.
So this is a real problem and the issue has to be resolved somehow,
although I think we are discussing this in the wrong group. IMO that
this is an issue for the maps group and we should adress it there.
Happy coding:-)

Update :For an alternative solution you can take a look here.

http://groups.google.com/group/google-appengine/browse_thread/thread/28a0a39a479838da

On Jul 8, 10:29 pm, Zarko <eladza...@gmail.com> wrote:
> Yes, it is in use with a Google map, actually I am trying to save
> requests to Google from clients...
>
> On Jul 8, 6:43 pm, Barry Hunter <barrybhun...@gmail.com> wrote:
>
>
>
> > On 8 July 2010 16:08, Zarko <eladza...@gmail.com> wrote:
>
> > > By the way I can't transfer the job to the client (it's not a browser
> > > app).
>
> > Are you sure your app is using the API legally then?
>
> > *Note: **the Geocoding API may only be used in conjunction with a Google
> > map; geocoding results without displaying them on a map is prohibited.*
> > *
> > *
> > fromhttp://code.google.com/apis/maps/documentation/geocoding/

Re: What is the best way to convert a dictionary of data into a datastore entity?

2010-06-25T23:56:00.000+03:00

If you are going to use your dictionaries as dictionaries back in your
program then why not save those into a Blob or Text datastore property
(by pickling or repr - eval or other means).

There was a long thread going here about the more efficient way to do this here :
http://groups.google.com/group/google-appengine-python/browse_thread/thread/8b07e7c24cb434f2/035f1b4e247deef9
http://groups.google.com/group/google-appengine-python/browse_thread/thread/b929234f093f355c/806aa0c3de0d2528
an example where i use repr eval:

from google.appengine.ext import db

class DicPropertyEval(db.Property):
    data_type = dict
    def get_value_for_datastore(self, model_instance):
        return db.Text(repr(super(DicPropertyEval,self).get_value_for_datastore(model_instance) ) )
    def make_value_from_datastore(self, value):
        if value is None:
            return dict()
        return eval(value)
    def default_value(self):
        if self.default is None:return dict()
        else:return super(DicPropertyEval,self).default_value().copy()

class DataStoreDic(db.Model):
    dicVal = DicPropertyEval(indexed=False)

The above method is not the fastest one but I prefer it sometimes coz
the dictionary is readable - editable by a human while in datastore.

happy coding ;-)

On 25 Ιούν, 20:57, Barry Hunter <barrybhun...@gmail.com> wrote:
> I'm no python expert, but I think Expando Class is designed for this
> sort of situation
>
> http://code.google.com/appengine/docs/python/datastore/expandoclass.html
>
> On 25 June 2010 17:52, richardhenry <richardhe...@me.com> wrote:
>
> > I have millions of Python dictionaries that have some common fields
> > (clearer for me to call them "fields" than "keys" here), but the
> > fields will vary a great deal from one dict to another.
>
> > Since I don't know in advance the fields that a dictionary will
> > contain, I was thinking of iterating through the values in the dict,
> > picking a data store Property (IntProperty(), StringProperty(), etc.)
> > class to use, and then creating a class using type('Content',
> > (db.Model,), {"name": StringProperty() ...}). Then I can store the
> > data from the dictionary in this class and save() it.
>
> > Am I headed down the right route? Does this make sense, or am I
> > missing some built-in feature that makes this much easier?
>
> > To summarize: I have millions of dictionaries that I want to convert
> > to datastore entities. All of the dict are related, and they do have
> > *some* common fields, but the majority of the data varies from dict to
> > dict. What's the best way to convert these dicts into entities in the
> > datastore?
>
> > Richard

SDK version 1.3.5 Prerelease

2010-06-17T01:36:00.002+03:00

App Engine team announced V 1.3.5 prerelease, I do like the release early release often attitude.
From a quick source code reading, what I see as most important new feature in this release is the support for Content-range headers for Blobs and stream like interface to blobstore that will extend usage of blobstore to some new application scenarios.

Transparent App Engine logo

2010-06-16T03:09:00.006+03:00

The other day I was looking for a transparent App Engine Logo to use in this blog.
Unfortunately I could not find one so I had to make it. Now it is available here .
As for people asking how I made this flying animation ( I know I know ... it looks stupid and obscuring etc. etc. but I wanted this lovely plane to fly a bit !, it is all pure CSS plus a couple of lines of js all made very easy with the new blogger template designer. Here are the steps :

1) Save the image in an image gadget:
From template designer > Advanced > Add CSS : to add some css to make thing work.
3) Position the image :
#Image1{
position:fixed;
width:auto;height:auto;
left:20px;top:30px;}

#Image1 div {position: fixed;top: 7%;left:1%}

4)Define a class so it can be applied by js to auto start the animation on page load or on click using a little js :

.nmanimate {
-webkit-animation-name: flyplane;
-webkit-animation-duration: 60s;
-webkit-animation-iteration-count: 1;
}

5) Or make the animation start with hover ... :

#Image1 div:hover {
-webkit-animation-name: flyplane;
-webkit-animation-duration: 60s;
-webkit-animation-iteration-count: 2;
}

(*) It only works on Chrome but it degrades gracefully in other browsers, besides most of the visitors here are Chrome users.
(* *) Lets hope some body with a more creative mind than mine picks up from here an recreates it in a much better way ie. use multiple css backgrounds to create a parallax effect.

(*) For people wanting the transparent App Engine logo they can get it from here: Logo

DataStore Latency II

2010-06-13T23:06:00.005+03:00

As promised Google went on an data store overhaul today.

One good thing is that there was no significant application disruption during the maintenance period, except of a short ~~write-only~~ read-only period that was pre announced by App Engine team.

Second good thing is that judging by my appls average response time the results are excellent.

This is also depicted in Datastore Status Page, which for historical reasons I am attaching here.

From the picture we can see that after around 03:30 PM when maintenance finished datastore latency dropped dramatically, also we do not see any spikes (well not until the time I write this - fingers crossed).

So lets keep the picture here and see how today's maintenance will affect App Engine on the long run.

Meanwhile I can see a change in Python's average response time which seems a little elevated but more steady than before.

Kudos to the App Engine team for the work they have done during the weekend to minimize potential problems.

DataStore Latency

2010-06-11T00:09:00.002+03:00

There is a lot of talk going on in technical blogosphere regarding App Engine's elevating datastore latency lately. Google has acknowledged the issue and respondend here and promised a fix within a couple of weeks.

The problem does affect the applications running in app engine as you can see from one of my Appls Site performance charts as plotted by Google Webmasters tools:
The explanation given by App Engine's team above is that this is the result of rapid growth rate of App Engines usage during past few months and I do believe this explanation.
The problem is that it also happened while Google announce App engine for Google Appls. My point of view is that App Engine should hold from expanding to new areas till all technical problems, outstanding issues and features promised in the road map get materialized. But then again who am I to tell Google what to do ?.
By the way an interesting but rather naive view has been written in App Engines mail-group about DataStore Latency affecting SEO.
One of the contributors there seems to believe that "But it could also happen nothing too because google detects that your site runs on their service" i.e. Google search could possibly will not penalize for been slow applications running on App Engine since these are running on Google's infrastructure.
Well I can tell you this can't happen since it defies search's main purpose to "provide best results for the user" besides I do not believe search results and serps can change that much within couple of weeks time (caffeine is just designed to target this delay).

MultipleChoiceField Django field with select multiple

2010-06-05T03:50:00.002+03:00

Yesterday I was trying to find a way to use MultipleChoiceField Django field along with a CheckboxSelectMultiple Django widget for an input form that contains some tags used in an App Engine model field.

There is an excellent solution described in ( better-way-to-use-djangos-selectmultiple-in-google-app-engine-for-a-listproperty/) except that it relies on

google-app-engine-django package which is a nice package bridging App Engine db models and Django models.

But ... for this particular case I needed a light solution with a very small footprint and a minimum of imports.

It was more easy than I though.

First define our input choices and respective labels : tagChoises = ( ('Tag 1', 'Economy'), ('Tag 2','Geograpy'))

Then define your form item: tags = forms.MultipleChoiceField(required=False, choices=tagChoises , widget=forms.CheckboxSelectMultiple)

The form now displays the check boxes which a user can fill.

Problem is, if a user selects multi tags only the one is returned for saving into the db and since this value is a unicode string while a list of string values is expected it results in an form error.

Solution: intercept the retrurned value in Request Handler post this is a UnicodeMultiDict .

It turns out that getall method of this class returns a list of the selected strings so last item of code : self.request.POST['tags']=self.request.POST.getall('tags' ) makes the trick and posted values are ready to be put.

I know this is an ungly hack but still it is ok for situations where the complete Django package is an overkill.

MapReduce Support for GAE

2010-05-24T22:06:00.000+03:00

It seems Map Reduce support is coming to App Engine although no official anouncement yet, except what is available on code's Wiki page (http://code.google.com/p/appengine-mapreduce/wiki/GettingStartedInPython)
Well done App Engine team !

10 things we do not love in Ubuntu 10.4

2010-05-20T13:41:00.004+03:00

There was an article on techrepublic today titled "10 things you'll love about Ubuntu 10.4"

And what about those 10 things we do not love in Ubuntu 10.4 ?

Well ... make it just 1 - moving the window controls to the left I think it is a damn wrong move since It creates a lot of confusion especially to new users converting from Windows and quite a lot of ambiguity since some windows still have window controls on the right i.e. Chrome browser.

Of course you can move the controls back to the right using the configuration editor but this is not a solution for newbies.

I believe Canonical's first priority should be targeting Windows users coz there are much more of them and more easy to convert compared to Mac users.

Google Maps API quotas and App Engine

2010-05-13T17:55:00.004+03:00

There is a lot of talk in various forums that I follow about Maps API quota limits.
To summarize: there is a limit of 2500 requests per day per requesting IP (see: http://code.google.com/apis/maps/faq.html) , if your application issues those requests from any server you can reach this limit quite fast. The situation is much worst if your application is based in GAE then it has to share those quotas with any other application in GAE that happens to use the same API since all those applications are served from a limited number of google IP's making the API useless.
So far so bad except ... there is a solution, delegate the job to the client using JSON & JS. This way :
a) quotas are calculated against your user's IP and more probably than not he will never exceed those.
b) you offload some workload from your server.
c) save some bandwidth.
Practically I can't see a scenario where we can't use client side, since whatever we do we have to use the results only in connection with a Google map according to Maps TOS para 10.12 (unless of course you opt for the premier edition).
Update :For an alternative solution you can take a look here.

Happy Birthday

2010-04-21T23:22:00.000+03:00

It was two years ago today that I have started developing in App Engine, been one of the first 10000, when it was released to the first 10000 developers.
Since then I have spent a lot of days and nights (well ... mostly nights) working with App Engine having seen it advancing to its present form, something I do not regret.
On this occasion I must express my many thanks to the wonderful App engine team for their efforts and the support they give to developers.
http://googleappengine.blogspot.com/2010/04/happy-birthday.html

New Warning message in logs

2010-04-21T02:24:00.000+03:00

Any body noticed following new warning on their logs ?
"
04-20 04:02PM 48.495
This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application.
"
Very helpful and useful in many ways (code optimization etc.)
Oh ! just a short commend : could it be a little less verbose ? thus not waisting so much valuable space in logs database etc ?

Google Plans To Expand Cloud Computing Services

2010-04-14T02:13:00.000+03:00

This sounds like good news for App Engine developers
http://techcrunch.com/2010/04/12/google-plans-to-expand-cloud-computing-services-think-aws-on-steroids/

Post mortem of a different kind.

2010-04-06T02:26:00.000+03:00

Today's post has nothing to do with Application Engine neither with programming at large which are the subjects of this blog.

Or may be it has ?

My dearest brother died last Wednesday night at the age of 47.

I do regret that been occupied with work and programming, did not spent much time with him lately neither I have listened carefully on what he was trying to tell me.

Programming is work and hobby but there are lots of other things in life that we should not neglect and never let computers, social networks and all virtual paraphernalia replace real human relationships.

CSS Chrome now supports border-radius

2010-04-01T22:38:00.000+03:00

Just discovered that Chrome now supports standard CSS 3 border-radius.
So no need for "-webkit-border-radius" if you target chrome.
It works on 5.0.342.7 beta Linux still have to test for other OS & versions.

Geognos Help Page

2010-01-05T00:58:00.000+02:00

Many people contacted me asking how geognos operates. Except the technicalities for which I will post some other time, I have compiled a help page which you can find here : geognos-help or you can also try it your self by visiting a country's page like: Russia or United States and play by clicking the map and the photos.

Happy coding.

geognos

2010-01-03T16:42:00.000+02:00

Today I published beta I of geognos my little application hosted in the cloud by google app engine.
You can select and visit any country from here: http://www.geognos.com , or may be take a look at my own country here: http://www.geognos.com/geo/en/cc/gr.html
If you do not like it you can blame only me, since I have done all the staff my self (css, Html, Python - javascript , 4 different API interface programming etc. etc.)
This site is a mashup of 4 APIS and shows textual and visual (thanks to: Google Visualization API ) information for all countries of the world. Users can navigate to different countries by selecting a country from the compo box provided in top right corner of each page or from the links provided in home page.
For each specific country a live map provided by Google Maps API v3 is included. The map provides the functionality (move, pan, zoom, etc.) of Google Maps. Right clicking on any point of the map pops up an information window displaying geographical information about this point (Latitude Longitude, PO Code, city, country etc.) User can also navigate to points of interest (capital, cities, neighboring countries etc.) of the particular country by using the selector box provided in page top.
Interesting photos from the country are displayed (courtesy of panoramio.com api) on the top right part of the page. users can view a bigger photo by clicking on one of the thumbnail photos shown in photo region. Also photo's geographical coordinates and other information is shown on the map. This information is presented translated to user's language thanks to Google AJAX Language API Clicking on the bigger photo displays a real size of the photo on top of the map region. Users can navigate to next-previous set or photos by clicking the respective arrows located at the bottom of thumbnails. Clicking on the "play" button located between the left right arrow puts the photos on a "slide mode" which is activated when user clicks on "left" - "right" button. While in "slide show" mode photos are auto-displayed until user clicks on "pause" button. Slide show is self terminated after a number of photos, in order to eliminate over-demand on the servers. Clicking on the left "=" button synchronizes photos with the contents of the map at that time and displays only photos located on the region included in the map, try this after moving - zooming the map.

Panoramio Photo API (III)

2009-10-18T01:16:00.000+03:00

Well it is exactly a month since Panoramio API has started to misbehave after a major site maintenace, although they fixed that particular bug quite fast, still the API it is not working properly so I checked if their own application work properly. To my surprise it seems their own applications are also affected which quite strange for a Google company, see my post at their forum http://www.panoramio.com/forum/viewtopic.php?p=249432#249432
" Photos shown in world map ( http://www.panoramio.com/map/) do not correspond to the coordinates of the map but include photos from very distant places not shown in the map - this is particular true on low zoom levels of the map."
they requested some examples which I also posted there with some explanations.
Still waiting for an answer.

Panoramio Photo API (II)

2009-09-23T23:28:00.001+03:00

It seems somebody in Panoramio listened to my prayers and they were fast to respond. So according to their official Forum ( http://www.panoramio.com/forum/viewtopic.php?t=20934 )
" Panoramio API: "count" is not working and changes in call back function. We are working in bringing them back."

Google Maps API V3

2009-09-22T23:51:00.001+03:00

Some good news about API's now.
Today Google has released a new version of v3 maps API today, with a nice surprise this time it has included a badly needed feature, Poly-lines and Polygon overlays !.

Panoramio Photo API

2009-09-22T23:38:00.000+03:00

I am using Panoramio JSON Photo API since quite some time.
Well, during last weekend they have a major maintenance of their platform.
Unfortunately after this maintenance the API format has been changed without any notice, It took me half a night to realize what was happening and fix my mash-up application.
I have filled a report here ( http://www.panoramio.com/forum/viewtopic.php?p=239662#239662 )
Lessons learned : If you are using any third party APIs - have a plan and be ready for surprises !.

Server Instance Life time (part II)

2009-09-12T03:21:00.000+03:00

I got some remarks on the above mentioned post today at :
http://groups.google.com/group/google-appengine-python/browse_thread/thread/2337166b90dcce14?hl=en
Following is my response :
"server-instance-life-time" "that's not my experience. In the early days an instance of my app would live for about half day between hits," Well it depends on how we define 'early days' - I am talking about circa May 2008 since when I started keeping an eye on instance life time. It used to be a some seconds value, now it is something like a minute class value for me. I do not dispute other values given by some members of the group here since (I) some other - application specific parameters could be possibly (?) - used by the environment to determine how long it keeps an instance running. And (II) ... may be a particular application has a request served by a new instance although previous instance is still up and running. ? (just food for thinking). Anyway I do not see the point made by Mr. BJörn Lindqvist of pinging at 5'' intervals. It is a waste of recourses (let's think Green !). There are other solutions even if his numbers are right. i.e: {I really hate this initials reminding me of Internet explorer} think ... of a static landing page which will ping the server preparing it for successive visits etc. etc.. If we are talking about sub 30'' lifetime something must be really wrong, why you don't try a with a simple 'hello-world- application' to check the figures. By the way, i have a handler which was always giving me red figures in the vicinity of 1000 + cpu_ms. I was happy to see it going purple at around 750 cpu_ms since a week or so without me touching anything for this particular handler. Anybody else experienced something similar ? Happy coding Nick

Google Maps API V3 (part II)

2009-09-03T13:29:00.000+03:00

Glad to see that my bug issue report mentioned in a previous post was verified, accepted and fixed by Google's developers team in ZERO time (whole process took less than 1 working day). This tells me a lot about the quality of their work.
It is a pity that I have to revisit my code now and remove the work around I devised.
Lesson learned: If you run into a bug and you are damn sure it is a real bug and not your lousy code causing the problem do not hesitate, fill a report and wait a little before you start waisting your time devising a work around.

Gmail failure today.

2009-09-02T03:37:00.000+03:00

Well it happened again, after last February there was one more outage of Gmail for some hours today.

But ... a small remark - What was down and not functioning was only Gmail's web interface - Email Servers were up and running and we could send-receive through POP / IMAP agents.

And that was the case with (almost) all major Gmail failures - no real service interruption. I am sure it will make headlines tomorrow but nobody (well .. almost) will mention this fact.

Hm .... It has started already : "This is one of the reasons that corporate e-mail has not moved to the cloud," said Tim whatever, president of Creative Strategies in Campbell. "Under no circumstances do they want to have the system taken down."

Hope somebody tells this guy of Creative Stratospheric Disasters in Com(p)eBackToFortran that most of Corporate Email is in the cloud already one way or the other.
For system admins that were prepared for this it was really NO EVENT at all.

And ... for people asking what this has to do with Application Engine the answer is obvious :
This (non) event will be used by people of BackToEarthFromCloud camp to spread around FUD (Fear Uncertainty Doubt) for cloud computing in general - just because they were very late in this game or they do not really understand it. :)

Google Maps API V3

2009-08-30T00:16:00.000+03:00

I was doing some work with Google Maps API V3 during last couple of weeks,
and here are some of my experience with this version.

Very small footprint and quite fast.
Although in Beta I think it is quite stable to be used in production - if of course you are ready to follow with latest developments and fixings.
A small but annoying bug I discovered (for which I filled a ticket : http://code.google.com/p/gmaps-api-issues/issues/detail?id=1643 along with a work around)
A tip : Since there is no way to hide-show the zoom - pan and Map type selection controls through api you have do it by writing some javascript

   var housediv=yourmapObject.getDiv();
   if (housediv.childNodes[3]) {
   housediv.childNodes[3].className = myClass
   housediv.childNodes[4].className = myClass
   }
   // if exist
   //controls reside in childnode[3] and [4] of container div
   //rest is done with CSS class where you can hide-show or change the opacity

Be careful with Yaml editing

2009-08-16T20:53:00.000+03:00

The other day I have almost erased an application of mine in App Engine the only thing that saved my day is that I was working on a version.

So how this happened to me ?

Well you edit your yaml file which probably have been copy/pasted from some other application , you read it, you are satisfied with what you see and you decide to upload on the spot so here we are: appcfg ... and presto your old application is gone coz you forgot to save this yaml which looked good to your eyes.

You see appcfg does not read through your own eyes it prefers its own devices which can read a complete different story than yours.

Server Instance Life time

2009-08-16T20:43:00.000+03:00

This was always an issue since the beginning of GAE, with people suggesting fake requests or cron jobs to keep an instance alive.

I am following the issue since the very beginning and although I do not have indisputable numbers to share here there is one thing I can tell for sure the average timeout has increased quite a lot since the early days and currently is satisfactory to the extend that any application with reasonable hits, can expect to almost always be served from a hot instance.

CPU and latency of a bare bones application.

2009-08-16T20:15:00.000+03:00

I have done some research today regarding App Engine's response time cpu usage etc.

I got some conclusions but need more verification before posting those here but between some tests I decided to check what is the minimum cpu time usage when starting cold, so I creating a brand new application I uploaded a plain "Hello World application, with a bare minimum configuration.

Just a single yaml handler and a print statement for output no import whatsoever.

I was amassed to see in in the logs that a cold start would cost around 100 cpu_ms or more with successive requests consuming around 10 cpu_ms and request time latency of more than 100 ms in the hot start.

So if we have a proper application with cpu consuption and latencey close enough to above values we are doing wanders.

One more remark on the aftermath of these tests today, do not forget to include a favicon to the handler since most of the browsers will request for it any way and the cost of this request and the denial is probably greater than provide one in the first place.

And ... here are the real logs where you can see a Cold % Start and successive Hot Starts :

1. 08-16 10:37AM 50.074 /Hot%20Start 200 8ms 8cpu_ms 0kb Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US) AppleWebKit/532.0 (KHTML, like Gecko) Chrome/3.0.198.1 Safari/532.0,gzip(gfe)
1. 08-16 10:37AM 48.149 /favicon.ico 200 11ms 14cpu_ms 0kb Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US) AppleWebKit/532.0 (KHTML, like Gecko) Chrome/3.0.198.1 Safari/532.0,gzip(gfe)
1. 08-16 10:37AM 47.616 /Hot%20Start 200 37ms 55cpu_ms 0kb Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US) AppleWebKit/532.0 (KHTML, like Gecko) Chrome/3.0.198.1 Safari/532.0,gzip(gfe)
1. 08-16 10:37AM 37.885 /favicon.ico 200 24ms 16cpu_ms 0kb Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US) AppleWebKit/532.0 (KHTML, like Gecko) Chrome/3.0.198.1 Safari/532.0,gzip(gfe)
1. 08-16 10:37AM 37.340 /Cold%20Start 200 229ms 168cpu_ms 0kb Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US) AppleWebKit/532.0 (KHTML, like Gecko) Chrome/3.0.198.1 Safari/532.0,gzip(gfe)
1. 08-16 09:27AM 29.460 /favicon.ico 200 7ms 6cpu_ms 0kb Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US) AppleWebKit/532.0 (KHTML, like Gecko) Chrome/3.0.198.1 Safari/532.0,gzip(gfe)

Google Application Engine

2009-08-16T20:14:00.001+03:00

Well... after more than a year that I spent some of my free time doing development on the cloud mostly with Google Application Engine, I decided to start this blog as a means of keeping some notes of the whole process.

Hoping this will be valuable to me and any others working in this field.

Probably I am starting this a little too late, but then again "never is too late never is too far" since I do believe the "cloud" is here to stay.

Cloud Computing with Google Application Engine

New App Engine Pricing policy, the good the bad and the ugly.

Google Maps API rate limiting and Google App Engine again

App Engine's Email service future

Selecting distinct entities across a large table

Re: Policy for instance startup

World Countries and IP geocoding API for App Engine

maximum number of requests that may be handled in a single process lifetime

Downloan app code feature or bug ? - download_app

Instances console in App engine's Admin Tools

Yahoo claims about spam filtering

App Engine scalability issues

Bing search returns a stack trace

Want to build voice, SMS, IM and Twitter apps in python? Tropo WebAPI library now available

Best practice to import libraries or frameworks in App engine

Allegro Non Troppo - Using Tropo API in App Engine

Understanding the concurrent requests limit

A bug in tweepy streaming API

App Engine & Google Geocoding Service II

Re: What is the best way to convert a dictionary of data into a datastore entity?

SDK version 1.3.5 Prerelease

Transparent App Engine logo

DataStore Latency II

DataStore Latency

MultipleChoiceField Django field with select multiple

MapReduce Support for GAE

10 things we do not love in Ubuntu 10.4

Google Maps API quotas and App Engine

Happy Birthday

New Warning message in logs

Google Plans To Expand Cloud Computing Services

Post mortem of a different kind.

CSS Chrome now supports border-radius

Geognos Help Page

geognos

Panoramio Photo API (III)

Panoramio Photo API (II)

Google Maps API V3

Panoramio Photo API

Server Instance Life time (part II)

Google Maps API V3 (part II)

Gmail failure today.

Google Maps API V3

Be careful with Yaml editing

Server Instance Life time

CPU and latency of a bare bones application.

08-16 10:37AM 50.074 /Hot%20Start 200 8ms 8cpu_ms 0kb Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US) AppleWebKit/532.0 (KHTML, like Gecko) Chrome/3.0.198.1 Safari/532.0,gzip(gfe)

08-16 10:37AM 48.149 /favicon.ico 200 11ms 14cpu_ms 0kb Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US) AppleWebKit/532.0 (KHTML, like Gecko) Chrome/3.0.198.1 Safari/532.0,gzip(gfe)

08-16 10:37AM 47.616 /Hot%20Start 200 37ms 55cpu_ms 0kb Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US) AppleWebKit/532.0 (KHTML, like Gecko) Chrome/3.0.198.1 Safari/532.0,gzip(gfe)

08-16 10:37AM 37.885 /favicon.ico 200 24ms 16cpu_ms 0kb Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US) AppleWebKit/532.0 (KHTML, like Gecko) Chrome/3.0.198.1 Safari/532.0,gzip(gfe)

08-16 10:37AM 37.340 /Cold%20Start 200 229ms 168cpu_ms 0kb Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US) AppleWebKit/532.0 (KHTML, like Gecko) Chrome/3.0.198.1 Safari/532.0,gzip(gfe)

08-16 09:27AM 29.460 /favicon.ico 200 7ms 6cpu_ms 0kb Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US) AppleWebKit/532.0 (KHTML, like Gecko) Chrome/3.0.198.1 Safari/532.0,gzip(gfe)

Google Application Engine