Selecting distinct entities across a large table

I have faced this kind of problem some time ago.
I tried some of the solutions suggested below (in memory sort and filtering, encoding things into keys etc. and I have benchmarked those for both latency and cpu cycles using some test data around 100K entities)
An other approach I have taken is encoding the date as an integer (day since start of epoch or day since start of year, same for hour of day or month depending on how much detail you need in your output) and saving this into a property. This way you turn your date query filter into an equality only filter which does not even needs to specify an index) then you can sort or filter on other properties.
Benchmarking the latest solution I have found that when the filtered result set is a small fraction of the unfiltered original set, is 1+ order of magnitude faster and cpu-eficient. Worst case when no reduction of the result set due to filtering the latency and cpu usage was comparable to the previous solutions)

Hope this helps, or did I missed something ?
Happy coding-:)

Re: Policy for instance startup

Not been a Googler can't help much with this.
Having said that, I suspect there is a kind of build in algorithm that
does some kind of application profiling taking into acount QPS,
response times, and other parameters which adjusts instance life time,
number of instances to start etc..
This could possibly explain the difference in behaviour between your
staging and production appls.

happy coding;-)

World Countries and IP geocoding API for App Engine

I have been writing here again and again about the inherent problem that App Engine based Applications have in using third party API’s with quota limits based on ip addresses since all of those are served from the same block of ips alocated by Google  © and therefore have to share those quotas with other applications hosted on App Engine using the same service.

Today I am happy to offer a free IP geocoding and world countries information service API and hope this solves to some extend the problem of Server side IP geocoding for fellow App Engine developers.

Detailed service description and more information is provided here : "World countries API"

Happy coding:-)

maximum number of requests that may be handled in a single process lifetime

I was doing some load tests on app engine today when I noticed a new Info message in the logs: "After handling this request, the process that handled this request reached the maximum number of requests that may be handled in a single process' lifetime, and exited normally."

So what that supposed to mean ?
Up to know we new that application instances are automatically terminated after some inactivity time out. If I understand this message well now we know that a process can be terminated after handling so many requests. How many exactly ? is this a new magic number ? Lets hope we will have some definite answers from the always helpful App Engines team.

I am doing some latency optimization (lazy imports) in this app based on the assumption that an instance will stay live as long as there are more requests coming in and I was scared this new number spoils my optimization logic. After some more testing I have found out that this is not true since whatever this number is it must be in the order of thousands requests - so no big trouble in my use case. Still I think App Engine's developers deserve to know more about such parameters, it helps both developers and the platform.

Downloan app code feature or bug ? - download_app

I think App Engines developers community has by a vast majority rejected the idea of code downloading at least as default (opt out) option.
I also do not like the idea of a payable service, since it :

• will complicate the pricing model
• will attract criticism against the platform and 
• help guys who are in the business of doing unreasonable GAE vs S3 vs whatever_looks_like_cloud comparisons happy.

But ... then again who am I to tell mother G what to do ? -:)
( see my post at google-appengine group )

Instances console in App engine's Admin Tools

It seems a new feature has just been rolled out in App Engine's production Admin tools - instances - it shows a view of total running instances along with QPS, Latency and memory used Memory as well as averages for the above values.
There is also a summary of the above in application's dashboard.
It is a very useful feature that will help a lot with application performance monitoring and resolving issues on when and how many new instances are started and killed.
Well done dev team. :-)

Yahoo claims about spam filtering

From Techcrunch :
Yahoo claims -55% less spam than Gmail. 40% less than hotmail.
I am sure that if Yahoo have not been an MS partner the numbers would have been reversed.
Something like 55% less spam than hotmail. 40% less than Gmail.
Any way ... lets wait to hear what MS have to say about those figures.