Re: Policy for instance startup

Not been a Googler can't help much with this.
Having said that, I suspect there is a kind of build in algorithm that
does some kind of application profiling taking into acount QPS,
response times, and other parameters which adjusts instance life time,
number of instances to start etc..
This could possibly explain the difference in behaviour between your
staging and production appls.

happy coding;-)

World Countries and IP geocoding API for App Engine

I have been writing here again and again about the inherent problem that App Engine based Applications have in using third party API’s with quota limits based on ip addresses since all of those are served from the same block of ips alocated by Google  © and therefore have to share those quotas with other applications hosted on App Engine using the same service.

Today I am happy to offer a free IP geocoding and world countries information service API and hope this solves to some extend the problem of Server side IP geocoding for fellow App Engine developers.

Detailed service description and more information is provided here : "World countries API"

Happy coding:-)

maximum number of requests that may be handled in a single process lifetime

I was doing some load tests on app engine today when I noticed a new Info message in the logs: "After handling this request, the process that handled this request reached the maximum number of requests that may be handled in a single process' lifetime, and exited normally."

So what that supposed to mean ?
Up to know we new that application instances are automatically terminated after some inactivity time out. If I understand this message well now we know that a process can be terminated after handling so many requests. How many exactly ? is this a new magic number ? Lets hope we will have some definite answers from the always helpful App Engines team.

I am doing some latency optimization (lazy imports) in this app based on the assumption that an instance will stay live as long as there are more requests coming in and I was scared this new number spoils my optimization logic. After some more testing I have found out that this is not true since whatever this number is it must be in the order of thousands requests - so no big trouble in my use case. Still I think App Engine's developers deserve to know more about such parameters, it helps both developers and the platform.

Downloan app code feature or bug ? - download_app

I think App Engines developers community has by a vast majority rejected the idea of code downloading at least as default (opt out) option.
I also do not like the idea of a payable service, since it :

• will complicate the pricing model
• will attract criticism against the platform and 
• help guys who are in the business of doing unreasonable GAE vs S3 vs whatever_looks_like_cloud comparisons happy.

But ... then again who am I to tell mother G what to do ? -:)
( see my post at google-appengine group )

Instances console in App engine's Admin Tools

It seems a new feature has just been rolled out in App Engine's production Admin tools - instances - it shows a view of total running instances along with QPS, Latency and memory used Memory as well as averages for the above values.
There is also a summary of the above in application's dashboard.
It is a very useful feature that will help a lot with application performance monitoring and resolving issues on when and how many new instances are started and killed.
Well done dev team. :-)

Yahoo claims about spam filtering

From Techcrunch :
Yahoo claims -55% less spam than Gmail. 40% less than hotmail.
I am sure that if Yahoo have not been an MS partner the numbers would have been reversed.
Something like 55% less spam than hotmail. 40% less than Gmail.
Any way ... lets wait to hear what MS have to say about those figures.

App Engine scalability issues

There is an issue going on for sometime now about App Engine's scaling capability.
We know that all App Engine's user facing HTTP Requests should be processed within a time-frame of 30 seconds, other wise will throw an exception. That is a rule that started with App Engine's introduction some two and half years ago, and until some time it was the only law applicable as far as I know. Then some months ago a new magic number has been introduced in group posts. It started as 1000ms within which an application has to respond other wise it could not scale properly (No new threads will be started to cope with increasing demand) later this number was lowered to 800ms.
I have raised this issue a month ago here :
" This "800ms" rule started as 1000ms some time ago, now it moved to 800ms and 400ms enters the scene. I am afraid it has became a moving target approaching 0ms too fast. Somebody must stop the bar somewhere."
But I got no reply.
In that thread even "sub-400ms" as optimal number is mentioned by a Ikai L.
Today it turns out nobody stopped the bar, instead this magic number has decreased even further to 700ms which turns my above mentioned prophecy of approaching 0ms a step closer to true. I do agree with all people in this group who complain that this is troublesome and concerning. Main advantage of App Engine is scalability, but this number kills scalability in most practical application scenarios.
It is understandable to raise the bar a little to reflect for advances in datastore latency but talking about 700 and sub 400 ms renders App Engine's scalability irrelevant for most if not all practical use cases.
There are other issues here that I am sure concern a lot of us poor App Engine's developers:

1) The always changing numbers.-I think it is reasonable to expect that an application put into production a year ago should have the same or better scalability  behaviour today or a year from now. I know we operate on a Beta platform so we have to take some risk and follow up and improve our applications when Google changes the parameters under which we operate but we will never be able to catch up with this kind of drastic changes.
2) Transparency - We should be informed about those limits, so we know in advance if what we have in mind is doable with App Engine. I understand there can be some trade secrets for engineering and/or marketing reasons, but still we must be in the loop and at least know basic things that affect our Appls.
Having complained about this I want also to be fair to App Engine and its team. Up to know the usual practice is that most of those limits that affect our applications (quotas etc.) are improving over time not vice versa. This is an exception and I think it must be rectified.

Bing search returns a stack trace

Haven't seen that before ! bing search returned (just once) following stack trace after a manual search for: where are highest mountains in the world. So you see it is not only us small individual developers that can force a stupid piece of code to execute, bi(n)g  guys are quite capable to make it happen too.

Want to build voice, SMS, IM and Twitter apps in python? Tropo WebAPI library now available

Want to build voice, SMS, IM and Twitter apps in python? Tropo WebAPI library now available

Best practice to import libraries or frameworks in App engine

Usual procedure is to paste the library in your application's path then upload it with your application using appcfg - update.
This works but sometimes we have to deal with huge libraries consisting of hundreds of python files that sometimes exceed the upper limit of 1000 files / application limit or reduce the number of available files for your application. Also maintaining  those files becomes a nightmare.
So what is the proper way ? :

Allegro Non Troppo - Using Tropo API in App Engine

I was looking for an SMS - Voice gateway for an application of mine and after thoughtfully looking into Clicatell - twilio etc. I opted for Voxeo's tropo.
Do not ask much for the reasons I am not ready for a war over which one is better or cheaper to use.
I just felt like that tropo's "develop free - pay as you go once in production" business model fits well with App Engine's model that is on the same line, same applies to application scaling which is analogous to App Engine's.

So I decided to give it a try and here are my preliminary assessment of the service:

Understanding the concurrent requests limit

This "800ms" rule started as 1000ms some time ago, now it moved to
800ms and 400ms enters the scene. I am afraid it has became a moving
target approaching 0ms too fast. Somebody must stop the bar
somewhere.

Happy coding ;-)
(From my post in : http://groups.google.com/group/google-appengine-python/browse_thread/thread/6353b6232e8851aa )

A bug in tweepy streaming API

I found a bug in tweepy streaming API .
It can't filter by tag with non Latin characters.
I was astonished coz this was working when I used it some months ago (version 1.3 if I remember well).
So here is my small contribution to the project :
def filter(self, follow=None, track=None, async=False):
   params = {}
   self.headers['Content-type'] = "application/x-www-form-urlencoded"
should be :
def filter(self, follow=None, track=None, async=False):
   params = {}
   self.headers['Content-type'] = "application/x-www-form-urlencoded; charset=utf-8"
I have only tested streaming and only tag filtering - probably this applies to other parts but I am not sure.

App Engine & Google Geocoding Service II

This issue comes and goes in this thread all the time.
I also suggested in the past to delegate the job to the client since I
could't see a scenario where we use server side request and be within
the TOR's.
see here : http://gaengine.blogspot.com/2010/05/google-maps-api-quotas-and-app-engine.html
I was wrong. Recently I run into a case where I had to do this and
still I am sure I am within legal limits, this is so when you need the
geolocation service in order to publish a static map served from
Google.
So this is a real problem and the issue has to be resolved somehow,
although I think we are discussing this in the wrong group. IMO that
this is an issue for the maps group and we should adress it there.
Happy coding:-)

Update :For an alternative solution you can take a look here.

Re: What is the best way to convert a dictionary of data into a datastore entity?

If you are going to use your dictionaries as dictionaries back in your
program then why not save those into a Blob or Text datastore property
(by pickling or repr - eval or other means).

SDK version 1.3.5 Prerelease

App Engine team announced V 1.3.5  prerelease, I do like the release early release often attitude.
From a quick source code reading, what I see as most important new feature in this release is the support for Content-range headers for Blobs and stream like interface to blobstore that will extend usage of blobstore to some new application scenarios.

Transparent App Engine logo

The other day I was looking for a transparent App Engine Logo to use in this blog.
Unfortunately I could not find one so I had to make it. Now it is available here .
As for people asking how I made this flying animation ( I know I know ...  it looks  stupid and obscuring etc. etc.  but I wanted this lovely plane to fly a bit !, it is all pure CSS  plus a couple of lines of js all made very easy with the new blogger template designer. Here are the steps :

DataStore Latency II

As promised Google went on an data store overhaul today.

One good thing is that there was no significant application disruption during the maintenance period, except of a short write-only read-only period that was pre announced by App Engine team.

Second good thing is that judging by my appls average response time the results are excellent.

This is also depicted in Datastore Status Page, which for historical reasons I am attaching here.

From the picture we can see that after around 03:30 PM when maintenance finished datastore latency dropped dramatically, also we do not see any spikes (well not until the time I write this - fingers crossed).

So lets keep the picture here and see how today's maintenance will affect App Engine on the long run.

Meanwhile I can see a change in Python's average response time which seems a little elevated but more steady than before.

Kudos to the App Engine team for the work they have done during the weekend  to minimize potential problems.

DataStore Latency

There is a lot of talk going on in technical blogosphere regarding App Engine's elevating datastore latency lately. Google has acknowledged the issue and respondend here and promised a fix within a couple of weeks.

The problem does affect the applications running in app engine as you can see from one of my Appls Site performance charts as plotted by Google Webmasters tools:
The explanation given by App Engine's team above is that this is the result of rapid growth rate of App Engines usage during past few months and I do believe this explanation.
The problem is that it also happened while Google announce App engine for Google Appls. My point of view is that App Engine should hold from expanding to new areas till all technical problems, outstanding issues and features promised in the road map get materialized. But then again who am I to tell Google what to do ?.
By the way an interesting but rather naive view has been written in App Engines mail-group about DataStore Latency affecting SEO.
One of the contributors there seems to believe  that  "But it could also happen nothing too because google detects that your site runs on their service" i.e. Google search could possibly will not penalize for been slow applications running on App Engine since these are running on Google's infrastructure.
Well I can tell you this can't happen since it defies search's main purpose to "provide best results for the user" besides I do not believe search results and serps can change that much within couple of weeks time (caffeine is just designed to target this delay).

MultipleChoiceField Django field with select multiple

Yesterday I was trying to find a way to use MultipleChoiceField Django field along with a CheckboxSelectMultiple Django widget for an input form that contains some tags used in an App Engine model field.

There is an excellent solution described in ( better-way-to-use-djangos-selectmultiple-in-google-app-engine-for-a-listproperty/) except that it relies on

google-app-engine-django package which is a nice package bridging App Engine db models and Django models.

But ... for this particular case I needed a light solution with a very small footprint and a minimum of imports.

It was more easy than I though.

First define our input choices and respective labels : tagChoises = ( ('Tag 1', 'Economy'), ('Tag 2','Geograpy'))

Then define your form item: tags = forms.MultipleChoiceField(required=False, choices=tagChoises , widget=forms.CheckboxSelectMultiple)

The form now displays the check boxes which a user can fill.

Problem is, if a user selects multi tags only the one is returned for saving into the db and since this value is a unicode string while a list of string values is expected it results in an form error.

Solution: intercept the retrurned value in Request Handler post this is a UnicodeMultiDict .

It turns out that getall method of this class returns a list of the selected strings so last item of code : self.request.POST['tags']=self.request.POST.getall('tags' ) makes the trick and posted values are ready to be put.

I know this is an ungly hack but still it is ok for situations where the complete Django package is an overkill.

MapReduce Support for GAE

It seems Map Reduce support is coming to App Engine although no official anouncement yet, except what is available on code's Wiki page (http://code.google.com/p/appengine-mapreduce/wiki/GettingStartedInPython)
Well done App Engine team !

10 things we do not love in Ubuntu 10.4

There was an article on techrepublic today titled "10 things you'll love about Ubuntu 10.4"

And what about those 10 things we do not love in Ubuntu 10.4 ?

Well ... make it just 1 - moving the window controls to the left  I think it is a damn wrong move since It creates a lot of confusion especially to new users converting from Windows and quite a lot of ambiguity since some windows still have window controls on the right i.e. Chrome browser.

Of course you can move the controls back to the right using the configuration editor but this is not a solution for newbies.

I believe Canonical's first priority should be targeting Windows users coz there are much more  of them  and more easy to convert compared to Mac users.

Google Maps API quotas and App Engine

There is a lot of talk in various forums that I follow about Maps API quota limits.
To summarize: there is a limit of  2500 requests per day per requesting IP (see: http://code.google.com/apis/maps/faq.html) , if your application issues those requests from any server you can reach this limit quite fast. The situation is much worst if your application is based in GAE then it has to share those quotas with any other application in GAE that happens to use the same  API since all those applications are served from a limited number of google IP's making the API useless.
So far so bad except ...  there is a solution, delegate the job to the client using JSON & JS. This way :
a) quotas are calculated against your user's IP and more probably than not he will never exceed those.
b) you offload some workload from your server.
c) save some bandwidth.
Practically  I can't see a scenario where we can't use client side, since whatever we do we have to use the results only in connection with a Google map according to Maps TOS para 10.12 (unless of course you opt for the premier edition).

Happy Birthday

It was two years ago today that I have started developing in App Engine, been one of the first 10000, when it was released to the first 10000 developers.
Since then I have spent a lot of days and nights (well ...  mostly nights) working with App Engine having seen it advancing to its present form, something I do not regret.
On this occasion I must express my many thanks to the wonderful App engine team for their efforts and the support they give to developers.
http://googleappengine.blogspot.com/2010/04/happy-birthday.html

New Warning message in logs

Any body noticed following new warning on their logs ?
"
04-20 04:02PM 48.495
This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application.
"
Very helpful and useful in many ways (code optimization etc.)
Oh ! just a short commend : could it be a little less verbose ? thus not waisting so much valuable space in logs database etc ?

Google Plans To Expand Cloud Computing Services

This sounds like good news for App Engine developers
http://techcrunch.com/2010/04/12/google-plans-to-expand-cloud-computing-services-think-aws-on-steroids/

Post mortem of a different kind.

Today's post has nothing to do with Application Engine neither with programming at large which are the subjects of this blog. 

Or may be it has ?

My dearest brother died last Wednesday night at the age of 47.

I do regret that been occupied with work and programming, did not spent much time with him lately neither I have listened carefully on what he was trying to tell me.

Programming is work and hobby but there are lots of other things in life that we should not neglect and never let computers, social networks and all virtual paraphernalia replace real human relationships.

CSS Chrome now supports border-radius

Just discovered that Chrome now supports standard CSS 3 border-radius.
So no need for "-webkit-border-radius" if you target chrome.
It works on 5.0.342.7 beta Linux still have to test for other OS & versions.

Geognos Help Page

Many people contacted me asking how geognos  operates. Except the technicalities for which I will post some other time, I have compiled a help page which you can find  here :  geognos-help or you can also try it your self by visiting a country's page like: Russia or United States and play by clicking the map and the photos.

Happy coding.

geognos

 Today I published beta I of geognos my little application hosted in the cloud by google app engine.
You can select and visit any country from here: http://www.geognos.com , or may be take a look at my own country here: http://www.geognos.com/geo/en/cc/gr.html
If you do not like it you can blame only me, since I have done all the staff my self (css, Html, Python - javascript , 4 different API interface programming etc. etc.)
This site is a mashup of 4 APIS and shows textual and visual (thanks to: Google Visualization API ) information for all countries of the world. Users can navigate to different countries by selecting a country from the compo box provided in top right corner of each page or from the links provided in home page.  
For each specific country a live map provided by Google Maps API v3 is included. The map provides the functionality (move, pan, zoom, etc.) of Google Maps. Right clicking on any point of the map pops up an information window displaying geographical information about this point (Latitude Longitude, PO Code, city, country etc.) User can also navigate to points of interest (capital, cities, neighboring countries etc.) of the particular country by using the selector box provided in page top.
Interesting photos from the country are displayed (courtesy of panoramio.com api) on the top right part of the page. users can view a bigger photo by clicking on one of the thumbnail photos shown in photo region. Also photo's geographical coordinates and other information is shown on the map. This information is presented translated to user's language thanks to Google AJAX Language API Clicking on the bigger photo displays a real size of the photo on top of the map region. Users can navigate to next-previous set or photos by clicking the respective arrows located at the bottom of thumbnails. Clicking on the "play" button located between the left right arrow puts the photos on a "slide mode" which is activated when user clicks on "left" - "right" button. While in "slide show" mode photos are auto-displayed until user clicks on "pause" button. Slide show is self terminated after a number of photos, in order to eliminate over-demand on the servers. Clicking on the left "=" button synchronizes photos with the contents of the map at that time and displays only photos located on the region included in the map, try this after moving - zooming the map.