Finally got per-pixel alpha mask working on w32. The included image isn't the greatest for showing an antialiased edge (it does have one), but if you have something with a drop shadow lying about, just drop it on the demo window and that'll be used instead.
Update This demo works on Windows, OSX and Linux (Linux requires JRE1.5+).
Update Here's another image to drop on the demo window, which better demonstrates the alpha blending, in case you don't have any handy
I've got some working X11 code, too, but I need to figure out a decent API to make it happen. OSX lets you just set the window background pixel transparent and then everything you paint in the window is automatically composited with whatever alpha mask you paint with. That could probably work with Swing as well, but you have to magically drop anything with a default background. OSX does this by checking for a magic UIResource color; anything painted with that color is fully transparent.
Update Source is available on BRANCH_V2 from JNA.
I realize I may be an utter dolt, but I find it really hard to follow Microsoft APIs. I've worked with Qt, X11/Xt, and Mac, each of which has its peculiarities of architecture, but those have a consistency (might I say design) that spans more than two or three functions.