Google Maps API rate limiting and Google App Engine again

@bFlood : Maps Premium is an expensive service usually used by not public-facing, password protected Web sites, while what we are talking about here is the free Google Maps service, where G is penalizing all Apps running on top of GAE since they have to share the ip addresses pool of GAE ip adresses.
@Ikai : what you write applies to old maps (V2) API where you can obtain an application authorization key and applications are rate limited based on this key, although I feel that some kind of ip based limitations exists also.
Maps V3 which is the way to go especially for mobile appls, do not require an application key, instead there are rate limitations based just on originating IP addresses, this puts GAE based appls on a disadvantage since we have to share this with all other GAE based appls using the service.
(see : post publised in App Engine group)

App Engine's Email service future

From time to time I have seen posts in app engine's groups of people complaining for Email service glitches. Some of those real some not so. Sometimes coming from people who did not bother to read the documentation or just ignorant of what an Email service is all about.
App Engines team response alarmed me a little.
IMHO it is not a good policy for GAE to abandon(?) services middle way
instead of improving - enhancing those.
so I wrote the following post and now I am waiting to see how the conversation evolves.

@Icai
I agree on most of what you write above, and I understand that you prefer to focus on more important things, also having run Email services for enterprises in the past I do know it is not trivial.
But ....
still I believe Email service is a major asset for GAE and dropping it (or anything to that effect) will constitute a major blow to App Engine.
Gae offers a limited subset of services compared to what a LAMP box or a IaaS box can offer but been a PaaS provides trouble free operation and automatic scalability.
Email service usually is part of any web operation so by dropping it out you farther limit the number of potential applications that fit well into what GAE offers.
Of course, developers can look into alternative options but this makes our life difficult since we have to integrate several other third party services in order to make a working web solution i.e. setting up multiple accounts, feed traffic back and forth to other services, having to monitor and deal with one more possible point of failure. All this defies to some extend the benefits of GAE as a PaaS.
Also, dropping a service in a time when competition is adding services, will sent the wrong signal to App Engine's developers/users echo system and having in mind that G is associated with the best email service can possibly turn into a PR disaster.

Further more, been a regular reader of the groups and following App Engine since the very early days I do not see that Email service has raised a lot of issues. I believe for most people who know what they are doing and do not abuse the service it works quite smoothly. Some of the issues raised (mainly spam flagging)
a) happen to the best of Email services b) are addressed by well known techniques and practices described by others here and elsewhere.
In conclusion:
I would welcome any measure taken to fight service abuse like using GAE primarily as a mail server - we all understand that this is not what GAE is all about.
Instead of dropping the service I would prefer to consider:
a) put false positive spam flagging issues under the responsibility of developers.
b) exclude the service or part of it (like delivery assurances) from the future SLA offer.
c) think about the technical possibility to integrate it to gmail which is the *most* reliable email service in town.

Selecting distinct entities across a large table

I have faced this kind of problem some time ago.
I tried some of the solutions suggested below (in memory sort and filtering, encoding things into keys etc. and I have benchmarked those for both latency and cpu cycles using some test data around 100K entities)
An other approach I have taken is encoding the date as an integer (day since start of epoch or day since start of year, same for hour of day or month depending on how much detail you need in your output) and saving this into a property. This way you turn your date query filter into an equality only filter which does not even needs to specify an index) then you can sort or filter on other properties.
Benchmarking the latest solution I have found that when the filtered result set is a small fraction of the unfiltered original set, is 1+ order of magnitude faster and cpu-eficient. Worst case when no reduction of the result set due to filtering the latency and cpu usage was comparable to the previous solutions)

Hope this helps, or did I missed something ?
Happy coding-:)

Re: Policy for instance startup

Not been a Googler can't help much with this.
Having said that, I suspect there is a kind of build in algorithm that
does some kind of application profiling taking into acount QPS,
response times, and other parameters which adjusts instance life time,
number of instances to start etc..
This could possibly explain the difference in behaviour between your
staging and production appls.

happy coding;-)

World Countries and IP geocoding API for App Engine

I have been writing here again and again about the inherent problem that App Engine based Applications have in using third party API’s with quota limits based on ip addresses since all of those are served from the same block of ips alocated by Google  © and therefore have to share those quotas with other applications hosted on App Engine using the same service.

Today I am happy to offer a free IP geocoding and world countries information service API and hope this solves to some extend the problem of Server side IP geocoding for fellow App Engine developers.

Detailed service description and more information is provided here : "World countries API"

Happy coding:-)

maximum number of requests that may be handled in a single process lifetime

I was doing some load tests on app engine today when I noticed a new Info message in the logs: "After handling this request, the process that handled this request reached the maximum number of requests that may be handled in a single process' lifetime, and exited normally."

So what that supposed to mean ?
Up to know we new that application instances are automatically terminated after some inactivity time out. If I understand this message well now we know that a process can be terminated after handling so many requests. How many exactly ? is this a new magic number ? Lets hope we will have some definite answers from the always helpful App Engines team.

I am doing some latency optimization (lazy imports) in this app based on the assumption that an instance will stay live as long as there are more requests coming in and I was scared this new number spoils my optimization logic. After some more testing I have found out that this is not true since whatever this number is it must be in the order of thousands requests - so no big trouble in my use case. Still I think App Engine's developers deserve to know more about such parameters, it helps both developers and the platform.

Downloan app code feature or bug ? - download_app

I think App Engines developers community has by a vast majority rejected the idea of code downloading at least as default (opt out) option.
I also do not like the idea of a payable service, since it :

• will complicate the pricing model
• will attract criticism against the platform and 
• help guys who are in the business of doing unreasonable GAE vs S3 vs whatever_looks_like_cloud comparisons happy.

But ... then again who am I to tell mother G what to do ? -:)
( see my post at google-appengine group )