Upcoming comic: "Can you pay me with Venmo?"

A little genZ gig economy, a little Lovecraft, and a little trans-genomics

Some initial character drafts

I think this is from a 20yo TODO list. I got tired of seeing a complex dependency tree kicking off seemingly unrelated builds, so I put together a quick python parser for make dependency and build output to render the DAG in an html page on demand. 

 Top-level targets:

Individual target dependency expansion:

Simple python script and html template 

Autoscrolling with respect for the user

I couldn't find an implementation of this that didn't have serious flaws, so here's a reliable React hook for ensuring a container scrolls to the bottom as its content grows, except when the user scrolls back to look at something. Returns a `ref` for the container, a `tailRef` for a component at the end of the container's children, and an `autoScroll` function that can be called to manually trigger the autoscroll behavior. Accepts a user scroll callback in case you want to respond to user scrolling events.

import { useCallback, useState, useEffect, useRef } from "react";

// Automatically scroll to the bottom of the container when the size or
// scrolling content changes, unless the user has started scrolling.
// Set onUserScroll to false to always scroll to bottom on component size
// changes, regardless of user activity.
export const useAutoScroll = (onUserScroll = null) => {
  const [container, setContainer] = useState(null);
  const ref = useCallback(node => setContainer(node), []);
  const tailRef = useRef();
  const isUserScrolling = useRef();
  const isAutoScrolling = useRef();
  const lastScrollHeight = useRef();
  const atBottomScrollTop = useRef();
  const isAtBottom = el => Math.abs(el.scrollTop + el.clientHeight - el.scrollHeight) < 1;

  const scrollToBottom = (onComplete = null) => {
    if (!container || isAtBottom(container)) {
      return;
    }
    isAutoScrolling.current = true;
    container.classList.add('auto-scrolling');

    // Check if any ancestor container is user-scrolling or auto-scrolling
    // If so, we need to avoid scrollIntoView which would scroll the ancestor
    const hasScrollingAncestor = container.parentElement?.closest('.user-scrolling, .auto-scrolling') !== null;

    // Use scrollIntoView only if no ancestor is scrolling (nice smooth scroll)
    // Otherwise use scrollTop to avoid triggering ancestor scroll and causing loops
    if (hasScrollingAncestor) {
      // Only scroll this container, don't trigger ancestor scrolling
      container.scrollTop = container.scrollHeight;
      // Immediately mark as complete since scrollTop is synchronous
      atBottomScrollTop.current = container.scrollTop;
      isAutoScrolling.current = false;
      container.classList.remove('auto-scrolling');
      onComplete && onComplete(container);
    } else {
      tailRef.current?.scrollIntoView({ behavior: "smooth" });

      // Check to see if we're at the bottom yet
      setTimeout(() => {
        if (container) {
          if (!isAtBottom(container)) {
            scrollToBottom(onComplete);
          } else {
            atBottomScrollTop.current = container.scrollTop;
            isAutoScrolling.current = false;
            container.classList.remove('auto-scrolling');
            onComplete && onComplete(container);
          }
        }
      }, 50);
    }
  };

  const autoScroll = (force = false, onComplete = null) => {
    if (!isUserScrolling.current || force) {
      scrollToBottom(onComplete);
    }
    else {
      onComplete && onComplete(container);
    }
  };

  const scrollHandler = e => {
    if (e.target !== container) {
      return;
    }
    const hasScrollbar = container.scrollHeight > container.clientHeight;
    lastScrollHeight.current = container.scrollHeight;
    if (atBottomScrollTop.current === null) {
      atBottomScrollTop.current = container.scrollHeight - container.clientHeight;
    }
    const currentScrollTop = container.scrollTop;

    // Clear user scrolling if at bottom or no scrollbar
    if (isAtBottom(container) || !hasScrollbar) {
      atBottomScrollTop.current = currentScrollTop;
      if (isUserScrolling.current) {
        isUserScrolling.current = false;
        container.classList.remove('user-scrolling');
      }
    }
    // Detect user scrolling up
    else if (!isAutoScrolling.current
             && atBottomScrollTop.current != null
             && atBottomScrollTop.current > currentScrollTop) {
      if (!isUserScrolling.current) {
        isUserScrolling.current = true;
        container.classList.add('user-scrolling');
        onUserScroll && onUserScroll(e);
      }
    }
  };

  // Ensure we trigger autoscroll on component content/size change
  useEffect(() => {
    if (container) {
      lastScrollHeight.current = container.scrollHeight;
      container.addEventListener("scroll", scrollHandler);

      const mutationObserver = new MutationObserver((mutations) => {
        // If content was cleared or significantly reduced, reset user scrolling flag
        const contentCleared = container.scrollHeight <= container.clientHeight;
        if (contentCleared) {
          isUserScrolling.current = false;
          container.classList.remove('user-scrolling');
        }

        autoScroll();
      });
      mutationObserver.observe(container, {childList: true, subtree: true, characterData: true});

      let resizeTimeout;
      const resizeObserver = new ResizeObserver((entries) => {
        // Debounce to avoid "ResizeObserver loop completed with undelivered notifications"
        clearTimeout(resizeTimeout);
        resizeTimeout = setTimeout(() => {
          // Clear user scrolling if scrollbar disappears
          const hasScrollbar = container.scrollHeight > container.clientHeight;
          if (!hasScrollbar && isUserScrolling.current) {
            isUserScrolling.current = false;
            container.classList.remove('user-scrolling');
          }

          autoScroll();
        }, 0);
      });
      Array.from(container.children).forEach(child => resizeObserver.observe(child));
      resizeObserver.observe(container);
      // Initialize state
      autoScroll();

      return () => {
        container.removeEventListener("scroll", scrollHandler);
        mutationObserver.disconnect();
        resizeObserver.disconnect();
      };
    }
  }, [container, onUserScroll]);

  return [ref, tailRef, autoScroll];
};

"Cambridge Chronicled" complete

I realized I hadn't posted the completion of this project (about two years ago now), mostly b/c the social media has been managed elsewhere.

The book is available Million Year Picnic in Cambridge, MA or online at Radiator Comics.

Other creative things

Been spending a lot of time on a comic (aka graphic novel memoir for those with upturned noses).

We offered a few chapters at the last MICE, but it's not yet available elsewhere.

It's a bit of humor about raising a kid in Cambridge, but it's also touches a lot of serious topics like racism, diversity, culture, and soul.  It's solely intended to share our unique perspective(s).

It's heartening to see so many around the world joining in protest after four years of xenophobia and hate in the US.  While it's sad that so little has changed since Rodney King (or Emmett Till, for that matter), I hope that more white folk (and by this I mean anyone who has questioned whether people of color are really treated any differently in this country) will take the opportunity to learn more.

Because ultimately, justice in society is everyone's responsibility, not just those with a boot on their neck.

Next time you go to read your favorite blog, pick up some James Baldwin or Ralph Ellison to read instead (you can probably get digital or audio versions of these from your local library for free).

Loading that pickle, what's going on?

tqdm is a handy little utility for showing how well some process is progressing (it works within python code, but you can also use it on any piped shell process).

Pickle is python's built-in serialization, which can take an awful long time on really large objects.  Unfortunately, pickle's only input is an open file.

So here's a little useful snippet for estimating how long it's going to take to deserialize that big python pickle file.

class TQDMBytesReader(object):
    """Show progress while reading from a file"""
    def __init__(self, fd, **kwargs):
        self.fd = fd
        from tqdm import tqdm
        self.tqdm = tqdm(**kwargs)
    def read(self, size=-1):
        bytes = self.fd.read(size)
        self.tqdm.update(len(bytes))
        return bytes
    def readline(self):
        bytes = self.fd.readline()
        self.tqdm.update(len(bytes))
        return bytes
    def __enter__(self):
        self.tqdm.__enter__()
        return self
    def __exit__(self, *args, **kwargs):
        return self.tqdm.__exit__(*args, **kwargs)

with open(filename, "rb") as fd, \
     TQDMBytesReader(fd, desc=f"Loading 'pickle", unit="b",
                     total=os.path.getsize(filename) as reader:
    obj = pickle.load(reader)

Here's the complement, although it's not terribly helpful since pickle serializes everything to memory and only then writes to disk:

class TQDMBytesWriter(object):
    """Show progress while writing to a file"""
    def __init__(self, fd, **kwargs):
        self.fd = fd
        from tqdm import tqdm
        self.tqdm = tqdm(**kwargs)
    def write(self, b):
        bytes_written = self.fd.write(b)
        self.tqdm.update(bytes_written or 0)
        return bytes_written
    def __enter__(self):
        self.tqdm.__enter__()
        return self
    def __exit__(self, *args, **kwargs):
        return self.tqdm.__exit__(*args, **kwargs)

Potamus update

The GAE cost profiling graphs have gotten a facelift, now using flot instead of Google visualizations. I rapidly hit the limit of the GViz capabilities (one notable shortcoming is the lack of support for sparsely-populated data). Most of the controls are now completely client-side, which makes it a lot easier to tweak the graph to get just the information you'd like.

Flot generally provides more CSS-level control over styling, and a nice plugin system to allow for mixing features.

App Engine real-time cost profiling is available on github.  Some assembly required.

Cost profiling on Google App Engine

I've recently been measuring costs for various operations that are currently being performed on Google App Engine.  Google provides some cost estimates on the app engine dashboard, and you can get historical daily totals, but it's generally not straightforward to answer the question "How much does this operation cost (or is going to cost if I ramp up)?".

The google-provided appstats is fine for profiling individual requests, but sometimes you need a much more comprehensive view.

With a Chrome extension to monitor the app engine dashboard numbers, and a small app engine app to collect data, I've managed to collect some interesting intra-day profile data, as well as provide a means for fairly accurate estimates of discrete operations.

Group view (for multiple app IDs).  The artifact on the left is due to two days' worth of missing data.  The lower graph has an obvious daily cron job, while the upper has much more distributed activity:

 Zoomed view (detail for a single app ID).  On this graph, you can see some annotations have been added; the data collector provides an API for applications to post events that can be overlaid on the cost data, making it easy to pick start and end points and calculating the cost for the selected span:

This project is now available on github.  The Chrome extension is based on the OSE (Offline Statistics Estimator) which scrapes usage data and applies customizable usage rates from the GAE dashboard pages.

Enable cProfile in Google App Engine

If it's not readily apparent to you how to enable CPU profiling on Google App Engine (it certainly wasn't to me, aside from a few hand waves at cProfile), this code snippet should get you up and running so you can focus on finding the data you need rather than the implied interfaces you have to mimic.  It uses the standard WSGI middleware hook to wrap an incoming request in a cProfile call, formatting and dumping the resulting stats to the log when the request returns:

def cprofile_wsgi_middleware(app):
    """
    Call this middleware hook to enable cProfile on each request.  Statistics are dumped to
    the log at the end of the request.
    :param app: WSGI app object
    :return: WSGI middleware wrapper
    """
    def _cprofile_wsgi_wrapper(environ, start_response):
        import cProfile, cStringIO, pstats
        profile = cProfile.Profile()
        try:
            return profile.runcall(app, environ, start_response)
        finally:
            stream = cStringIO.StringIO()
            stats = pstats.Stats(profile, stream=stream)
            stats.strip_dirs().sort_stats('cumulative', 'time', 'calls').print_stats(25)
            logging.info('cProfile data:\n%s', stream.getvalue())
    return _cprofile_wsgi_wrapper

def webapp_add_wsgi_middleware(app):
    return cprofile_wsgi_middleware(app)

Thread termination/exit handlers

I'd been hunting for a while for a good solution to automatically call cleanup operations on exit from threads I didn't own.  Windows (XP onward) has a pretty straightforward solution if you're working with a DLL, but for pthreads on most other platforms, the solution is not as obvious.

Some folks have been using JNA to allow Java code to be invoked as callbacks from various streaming (video, sound) libraries.  If these callbacks come from threads instantiated from native code, the JVM has to (at least temporarily) map the thread into Java space for the duration of the callback.  If the callbacks are frequent, and always come in on the same native thread, we don't want to incur the mapping overhead on every invocation.  The solution, then is to avoid detaching the thread when the callback finishes.

Now we have a new problem.  When the native thread actually terminates, the JVM has no idea that the thread went away because it's only really got a placeholder Thread object, and hasn't hooked up all the plumbing it normally does to detect that the thread has gone away and clean up/GC the mapped Thread object.  Thus the need for a thread exit handler.  In some cases, you may not care about the minimal object overhead, for instance if you've just got a few native threads.  However, with a lot of threads coming and going (we can't force folks to thread-pool).

Windows

On Windows, if you have a DllMain function defined, you'll get notices when threads and processes attach/detach your DLL's code.  This works out nicely, we can make the VM detach the current thread when we get that message:

// Store thread-local information required to detach the thread
// (in my case, only a JVM reference was required)
// TlsSetValue is only set when we recognize that we need the 
// extra cleanup
static DWORD dwTlsIndex;
BOOL WINAPI DllMain(HINSTANCE hDLL, 
                    DWORD fdwReason, 
                    LPVOID lpvReserved) {
  switch (fdwReason) {
  case DLL_PROCESS_ATTACH:
    dwTlsIndex = TlsAlloc();
    if (dwTlsIndex == TLS_OUT_OF_INDEXES) {
      return FALSE;
    }
    break;
  case DLL_PROCESS_DETACH:
    TlsFree(dwTlsIndex);
    break;
  case DLL_THREAD_ATTACH:
    break;
  case DLL_THREAD_DETACH: {
    extern void detach_thread();
    detach_thread(TlsGetValue(dwTlsIndex));
    break;
  }
  default:
    break;
  }
  return TRUE;
}

 Easy enough.  Note that TlsSetValue is called elsewhere only when the callback decides it doesn't want to detach immediately (rather than every time a native thread attaches).  Callbacks normally detach on exit if they weren't attached to begin with.

POSIX Threads (pthreads)

The pthreads library is a different beast.  Searches for "thread termination handler" don't turn up much, except for the stack-based pthread_cancel_push/pop, which seems to do the right thing, but have to arrive in pairs.  It's actually in the pthreads implementation of thread-local storage that we find a way to attach a termination handler to a given thread.

When you define a given thread-local variable in pthreads (a "key" in pthreads lingo), you can provide a destructor function to be used to clean up the storage...on thread exit.  It was a little hard to find since it's not a thread termination handler per se, but rather a mechanism to clean up thread local storage.  I'd overlooked it several times because I wasn't looking for thread local storage solutions.

// Basic plumbing to create a unique key identifying
// specific thread-local storage
static pthread_key_t key;
static void make_key() {
  extern void detach_thread();
  pthread_key_create(&key, detach_thread);
}

    ...
    // This code gets called whenever we identify that
    // a thread needs detach-on-exit behavior
    static pthread_once_t key_once = PTHREAD_ONCE_INIT;
    pthread_once(&key_once, make_key);
    if (!jvm || pthread_getspecific(key) == NULL) {
      pthread_setspecific(key, jvm);
    }
    ...

Now detach_thread will be called (with the value of the thread-local storage as its single argument) when the thread exits.

And voilà, you have a POSIX thread termination handler.

WebStart unit tests

I had a few bits of JNA functionality that were only active when in a web start environment, which made it a bit tricky to add tests for them that could be run at the same time as all my other tests. The functionality also showed up as a big splot of missing code coverage under clover, so I decided to tinker just a bit to see if I could somehow simulate a web start environment for the unit tests.

I managed to come up with a JUnit-based test fixture which ensures its test methods are all run in a web start environment. This works well under Windows and OSX; unfortunately *nix variations use NetX which needs some extra hand-holding when first run.

Finding the right hooks involved standard hacking to find whether the most configurability was offered by invoking the javaws executable or adding the javaws classes to my classpath and hooking directly into the Java. I ended up doing a Runtime.exec to ensure the running environment wouldn't interfere with that of Web Start.

The main obstacle to be overcome was signed code. You can get web start to run unsigned code by tweaking the local policy file to allow the code we're testing, but when you include any native code via the <nativelib> tag, JNLP requires the <all-permission> tag, which seems to re-trigger the requirement for signed code. Self-signing the code is no big deal, but that triggers web-starts authorization dialogs to allow the unknown CA to be used. Fortunately, the somewhat obscure deployment properties file can be used to temporarily allow the self-signed certs and bypass the dialogs. The tests can be run entirely without user input (except in the case of *nix, where NetX is not sufficiently configurable -- you have to dismiss the dialog on the first test run).

Unfortunately the exit codes for javaws don't correspond to the codes passed to System.exit by Java code, so I had to have the test fixture communicate to the running javaws instance via socket. Failures are transmitted by the fixture in such a way that you can't tell that the test is running in a separate process.

I encapsulated the deployment config, JNLP file creation, javaws launch, and test case execution into a single class, WebStartTest, which runs a few self tests and can be extended to add whatever other tests you need to run under web start. The JNLP is partly hard-coded to include files set up sepecifically for JNA testing, but should be trivial to change to accommodate a different project.

There are probably other ways to test WebStart code, but this fit nicely with the project's existing tests, allowing its WebStart features to be tested by test methods identical to those for the rest of the project.

Full source is here. This has been tested under Sun and IBM JDKs, on Windows, OSX, Solaris/sparc, and GNU/Linux.

JNA: increasing performance with large Structures

If you're using very large structures and using them often, here's a tip that can boost performance by several orders of magnitude. Note that you should follow this tip *only* if you really need the performance boost; otherwise you may wind up obfuscating your code.

By default, when JNA makes a native call it will copy the full contents of a Java Structure to native memory prior to the call and read it all back after the call. If your Structure is very large, this can result in significant overhead reflecting all the fields of the Structure. The reflection dwarfs the actual native communication time.

If you're only reading or writing a single field, it's much faster (although somewhat less elegant) to use the readField(String) and writeField(String) methods to access the data, while disabling the normal read and write. Depending on the size of your structure, you may see two orders of magnitude or more improvement in the native function call time.

Here's an example of performing the same operation two different ways:

class Big extends Structure {
    public int toNative;
    public int fromNative;
    // plus lots more fields
    // the more, the bigger the difference in performance
}

class FastBig extends Big {
    public void read() { }
    public void write() { }
}

Big big = new Big();
big.toNative = 42;
lib.callMyNativeFunction(big);
System.out.println("Got " + big.fromNative);

Big fast = new FastBig();
fast.toNative = 42;
fast.writeField("toNative");
lib.callMyNativeFunction(fast);
System.out.println("Got " + fast.readField("fromNative"));

If you wrap a loop and time these, you'll see what kind of difference it makes. On a test structure with 25 "int" fields, the fast version reduces time by a factor of 10.

Trivia: some other "struct" implementations (e.g. Javolution) use objects for all fields and require an explicit "write" or "set" on each. This reduces data transfer and/or reflection overhead, at the expense of simplicity of assignment and initialization.

JNA:
s.field = 1;
call(s.field);

Javolution:
s.field.set(1);
call(s.field.get());

JNA win64 port

JNA now supports 64-bit windows. The the build is a bit nonstandard, but it's automatic and reproducible. I was hoping to use mingw-64, but that project is not yet sufficient for JNA's purposes. Fortunately, the MS has a free 64-bit cross compiler which does work.

Thanks to the fine folks at Aquafold (actually mostly Niels G. the CTO) for making it possible.

Thanks also to T. Heller of python's ctypes for the initial glue bits.

Decorator Update

Thanks to some input from a few users, the AbstractComponentDecorator has been updated to fix a few visual and behavior artifacts.

Get the latest from Sourceforge SVN.

The decorator component allows you to apply arbitrary decorations to a component without affecting its layout or (necessarily) response to events.

This blog has many different examples of usage beyond simple decorations, including drag images and per-component input blocking.

Embedding DLL File Version Information with mingw32/cygwin

If you look at the properties window for a DLL file in Windows, you'll see a few bits about the file version and copyright info. I couldn't find any information on the web about how to embed that information into a DLL. Not terribly surprising, since it's probably something MSVC inserts automatically, so why would any one need to know how it's done?

Well, I'm building a DLL with cygwin/mingw32, so I do need to know how it's done. Fortunately, Mumit Khan, author of mingw32, wrote a utility called "windres", which compiles w32 resources for use in an executable or DLL. It'll compile directly to object format, which you can simply add to your final link.

For example:

% windres -i version.rc -o version.o
So the only thing I needed to figure out was the version resource format (I really didn't want to go grok the resource spec and write it from scratch). Luckily, Java's main DLL (java.dll, what else) has version information in it, and windres can also conveniently dump an exe or dll back into text format.

The java.dll resource info looks like this:

% windres -i java.dll -O rc

// Type: version
// Name: 1
LANGUAGE 0, 0
1 VERSIONINFO
 FILEVERSION 6, 0, 30, 5
 PRODUCTVERSION 6, 0, 30, 5
 FILEFLAGSMASK 0x3f
 FILEOS 0x4
 FILETYPE 0x2
BEGIN
  BLOCK "StringFileInfo"
  BEGIN
    BLOCK "000004b0"
    BEGIN
      VALUE "CompanyName", "Sun Microsystems, Inc."
      VALUE "FileDescription", "Java(TM) Platform SE binary"
      VALUE "FileVersion", "6.0.30.5"
      VALUE "Full Version", "1.6.0_03-b05"
      VALUE "InternalName", "java"
      VALUE "LegalCopyright", "Copyright \251 2004"
      VALUE "OriginalFilename", "java.dll"
      VALUE "ProductName", "Java(TM) Platform SE 6 U3"
      VALUE "ProductVersion", "6.0.30.5" 
  END 
END

With just a few tweaks to do a regexp replace on the new template and include the new obj in my build, my released dll now has visible version information (and in this case I'm perfectly happy having absolutely no clue what any of the resource file text actually means :).

Run your Java program as an NT service

Thomas Boerkel has contributed some mappings and code to the JNA project to facilitate access to Windows' user accounts, registry, and NT services. Included is an abstract class which is capable of installing, uninstalling, and running itself as an NT service. What sets this little gem apart from most available wrappers is that you don't need a Windows executable stub; the complete Windows service API is available to your Java program, including all system events and callbacks.

You can browse the source here. A NetBeans project is included.

I was working on an abstract service class as well but Thomas beat me to completion. Great work, Thomas!

Update: Make sure your classpath and classes are accessible by the local system account, or the service won't be able to start. To see if this is the issue, you can temporarily change the service to start using your user identity; if it works, then the problem is that the local system account doesn't have access.

Is final not final, or just not the final I wanted?

While adding some VM crash protection to JNA, I ran into this situation:

public class Pointer {
    public static final int SIZE = Native.POINTER_SIZE;
}

public class Native {
    public static final int POINTER_SIZE;
    static {
        initIDs();
        POINTER_SIZE = pointerSize();
    }
    private static native int pointerSize();
}

I had just moved the native initialization code from Pointer to Native. All the native methods were private, so the only changes were to update those two classes and rename the native methods. But when I reran the test suite, I started getting a host of errors.

There errors were all caused by Pointer.SIZE having a zero value. I figured that since Pointer.SIZE was final, the value it was assigned from must also have been zero. A quick inspection on Native.POINTER_SIZE, however, showed that it was propertly initialized and non-zero. How, then, could Pointer.SIZE be zero if Native.POINTER_SIZE was not?

Anyone who has done any JNI is probably aware that Java access modifiers don't mean anything to native code. My first thought was that perhaps "final" modifiers don't mean anything either. Only problem is that I was never touching that Pointer.SIZE field in native code. Never even accessed it.

Scanning though the native initialization code, I noticed that, among a host of standard JRE classes, a reference to the Pointer class was being created. This turns out to be the cause of my problem: during Native class initialization, the Pointer class was loaded from JNI. When the pointer final field was initialized, it got the value from a not-yet-initialized Native class.

Turns out, this can be duplicated with pure Java code:

public class LoadTest {
    public static final int VALUE;
    static {
        System.out.println("value=" + Secondary.VALUE);
        VALUE = 1;
        System.out.println("value=" + Secondary.VALUE);
    }
}

class Secondary {
    public static final int VALUE = LoadTest.VALUE;
}

When loaded, the class prints "value=0" twice, so final is final, just not the value I intended. Here, the problem is more apparent because the class dependency is clearly visible, which wasn't the case with the JNI initialization.

Maybe I'll call this antipattern something curiously obscure, like "Law of the Polygamist's Second Wife".

Demo sources

All source for demos in previous blog entries is now available via SVN on SourceForge. The project page is here.

All sources are provided under the LGPL.

Improved Window Shaping

Thanks to contributions from Olivier Chafik and Chris Deckers, the overhead for setting a window shape mask has been reduced by about two orders of magnitude. In this case, the balloon tip windows no longer have such a noticeable delay before showing.

Linux users should also have fewer issues with this version; some of the library setup previously required is now taken care of automatically by the JNA library.