Potamus update

The GAE cost profiling graphs have gotten a facelift, now using flot instead of Google visualizations. I rapidly hit the limit of the GViz capabilities (one notable shortcoming is the lack of support for sparsely-populated data). Most of the controls are now completely client-side, which makes it a lot easier to tweak the graph to get just the information you'd like.

Flot generally provides more CSS-level control over styling, and a nice plugin system to allow for mixing features.

App Engine real-time cost profiling is available on github.  Some assembly required.

Cost profiling on Google App Engine

I've recently been measuring costs for various operations that are currently being performed on Google App Engine.  Google provides some cost estimates on the app engine dashboard, and you can get historical daily totals, but it's generally not straightforward to answer the question "How much does this operation cost (or is going to cost if I ramp up)?".

The google-provided appstats is fine for profiling individual requests, but sometimes you need a much more comprehensive view.

With a Chrome extension to monitor the app engine dashboard numbers, and a small app engine app to collect data, I've managed to collect some interesting intra-day profile data, as well as provide a means for fairly accurate estimates of discrete operations.

Group view (for multiple app IDs).  The artifact on the left is due to two days' worth of missing data.  The lower graph has an obvious daily cron job, while the upper has much more distributed activity:

 Zoomed view (detail for a single app ID).  On this graph, you can see some annotations have been added; the data collector provides an API for applications to post events that can be overlaid on the cost data, making it easy to pick start and end points and calculating the cost for the selected span:

This project is now available on github.  The Chrome extension is based on the OSE (Offline Statistics Estimator) which scrapes usage data and applies customizable usage rates from the GAE dashboard pages.

Enable cProfile in Google App Engine

If it's not readily apparent to you how to enable CPU profiling on Google App Engine (it certainly wasn't to me, aside from a few hand waves at cProfile), this code snippet should get you up and running so you can focus on finding the data you need rather than the implied interfaces you have to mimic.  It uses the standard WSGI middleware hook to wrap an incoming request in a cProfile call, formatting and dumping the resulting stats to the log when the request returns:

def cprofile_wsgi_middleware(app):
    """
    Call this middleware hook to enable cProfile on each request.  Statistics are dumped to
    the log at the end of the request.
    :param app: WSGI app object
    :return: WSGI middleware wrapper
    """
    def _cprofile_wsgi_wrapper(environ, start_response):
        import cProfile, cStringIO, pstats
        profile = cProfile.Profile()
        try:
            return profile.runcall(app, environ, start_response)
        finally:
            stream = cStringIO.StringIO()
            stats = pstats.Stats(profile, stream=stream)
            stats.strip_dirs().sort_stats('cumulative', 'time', 'calls').print_stats(25)
            logging.info('cProfile data:\n%s', stream.getvalue())
    return _cprofile_wsgi_wrapper

def webapp_add_wsgi_middleware(app):
    return cprofile_wsgi_middleware(app)