I added a drop shadow to the speech bubble. You don't need it on OSX (OSX provides one automatically), and I don't have compiz or similar compositing window manager installed on linux at the moment (although if anyone is willing to submit/test the corresponding code, I'm happy to include it).
Nothing fancy, just black with 50% alpha. The shadow mask is just the original mask, sheared, scaled, and offset. The drop shadow would probably look nicer with a blurred edge, but that likely requires a variable alpha setting for the window (although you might be able to fake it by varying the edge color). Applying per-pixel alpha masks prior to Windows Vista seems to be non-trivial (this example uses a single alpha value for the entire window).
The JNA function definitions are trivial:
int GetWindowLongA(Pointer hWnd, int nIndex);
int SetWindowLongA(Pointer hWnd, int nIndex, int dwNewLong);
boolean SetLayeredWindowAttributes(Pointer hwnd, int crKey,
byte bAlpha, int dwFlags);
As is the actual usage:
Pointer hWnd = getHWnd(w);
User32 user = User32.INSTANCE;
int flags = user.GetWindowLongA(hWnd, User32.GWL_EXSTYLE) | User32.WS_EX_LAYERED;
user.SetWindowLongA(hWnd, User32.GWL_EXSTYLE, flags);
user.SetLayeredWindowAttributes(hWnd, 0, (byte)(255*alpha), User32.LWA_ALPHA);
The only real trick is that DirectDraw must be disabled in order to get the transparency effect and avoid leaving behind painting artifacts.
java -Dsun.java2d.noddraw=true ...
NOTE: thanks to l2fprod for the hint and the original round, transparent clock!
Update: Demo is now available from the JNA homepage.