Cost profiling on Google App Engine

I've recently been measuring costs for various operations that are currently being performed on Google App Engine.  Google provides some cost estimates on the app engine dashboard, and you can get historical daily totals, but it's generally not straightforward to answer the question "How much does this operation cost (or is going to cost if I ramp up)?".

The google-provided appstats is fine for profiling individual requests, but sometimes you need a much more comprehensive view.

With a Chrome extension to monitor the app engine dashboard numbers, and a small app engine app to collect data, I've managed to collect some interesting intra-day profile data, as well as provide a means for fairly accurate estimates of discrete operations.

Group view (for multiple app IDs).  The artifact on the left is due to two days' worth of missing data.  The lower graph has an obvious daily cron job, while the upper has much more distributed activity:

 Zoomed view (detail for a single app ID).  On this graph, you can see some annotations have been added; the data collector provides an API for applications to post events that can be overlaid on the cost data, making it easy to pick start and end points and calculating the cost for the selected span:

This project is now available on github.  The Chrome extension is based on the OSE (Offline Statistics Estimator) which scrapes usage data and applies customizable usage rates from the GAE dashboard pages.