I've been meaning to play around with shaped windows for a while, but didn't relish the thought of walking through the tedium of JNI configurations and builds. Doing it on one platform is bad enough, but on several? No thanks.
So I thought I'd write a little scriptable shared library access stub once and be done with JNI entirely. Well, turns out it's already been done. Several times.
JNativeNLinkJNAJNative has some interesting features not found in the others, but actually using it is only slightly better than JNI. NLink is currently w32-only, and has a bit of a COM bent. JNA fit most closely with my objectives, already had implementations for w32 and solaris, and was already platform-agnostic. So I took a couple days to hack in some more features (mostly to get an understanding of the codebase), and here is what I got. The following code is what it takes to make a frame take an arbitrary shape.
User32 user32 = User32.INSTANCE;
GDI32 gdi32 = GDI32.INSTANCE;
JFrame frame = new JFrame(getName());
Pointer p = gdi32.CreateRoundRectRgn(0, -150, 300, 300, 300, 300);
int hWnd = user32.FindWindowA(null, getName());
user32.SetWindowRgn(hWnd, p, true);
Looking up the appropriate w32 calls probably took the most time. How are the w32 libraries defined? How is this for trivial:
public interface User32 extends StdCallLibarary {
User32 INSTANCE = (User32)Native.loadLibary("user32", User32.class);
int FindWindowA(String winClass, String title);
void setWindowRgn(int hWnd, Pointer p, boolean redraw);
}
public interface GDI32 extends StdCallLibrary {
GDI32 INSTANCE = (GDI32)Native.loadLibrary("gdi32", GDI32.class);
Pointer CreateRoundRectRgn(int left, int top, int right, int bottom, int w, int h);
}
Now, somebody's probably going to point me to how SWT has had this functionality for years (does it?), but this is nicely abstracted and based on a very small number of classes. I'll be updating the code at jna.dev.java.net (or maybe opening a new location if I can't get the existing project moved to subversion), but for now, check out the demo by clicking on everyone's favorite orange launch button.

Permissions required, because this runs some custom native code.
Oh, BTW, this is windows-only for the moment. I'll do X11 next and anyone's free to send me some code snippets for setting window masks on other platforms. I could also use some help porting to PPC and other platforms (a very small amount of native ASM to push arguments to the stack, not too hard).