Other creative things

Been spending a lot of time on a comic (aka graphic novel memoir for those with upturned noses).

We offered a few chapters at the last MICE, but it's not yet available elsewhere.

It's a bit of humor about raising a kid in Cambridge, but it's also touches a lot of serious topics like racism, diversity, culture, and soul.  It's solely intended to share our unique perspective(s).

It's heartening to see so many around the world joining in protest after four years of xenophobia and hate in the US.  While it's sad that so little has changed since Rodney King (or Emmett Till, for that matter), I hope that more white folk (and by this I mean anyone who has questioned whether people of color are really treated any differently in this country) will take the opportunity to learn more.

Because ultimately, justice in society is everyone's responsibility, not just those with a boot on their neck.

Next time you go to read your favorite blog, pick up some James Baldwin or Ralph Ellison to read instead (you can probably get digital or audio versions of these from your local library for free).

Loading that pickle, what's going on?

tqdm is a handy little utility for showing how well some process is progressing (it works within python code, but you can also use it on any piped shell process).

Pickle is python's built-in serialization, which can take an awful long time on really large objects.  Unfortunately, pickle's only input is an open file.

So here's a little useful snippet for estimating how long it's going to take to deserialize that big python pickle file.

class TQDMBytesReader(object):
    """Show progress while reading from a file"""
    def __init__(self, fd, **kwargs):
        self.fd = fd
        from tqdm import tqdm
        self.tqdm = tqdm(**kwargs)
    def read(self, size=-1):
        bytes = self.fd.read(size)
        self.tqdm.update(len(bytes))
        return bytes
    def readline(self):
        bytes = self.fd.readline()
        self.tqdm.update(len(bytes))
        return bytes
    def __enter__(self):
        self.tqdm.__enter__()
        return self
    def __exit__(self, *args, **kwargs):
        return self.tqdm.__exit__(*args, **kwargs)

with open(filename, "rb") as fd, \
     TQDMBytesReader(fd, desc=f"Loading 'pickle", unit="b",
                     total=os.path.getsize(filename) as reader:
    obj = pickle.load(reader)

Here's the complement, although it's not terribly helpful since pickle serializes everything to memory and only then writes to disk:

class TQDMBytesWriter(object):
    """Show progress while writing to a file"""
    def __init__(self, fd, **kwargs):
        self.fd = fd
        from tqdm import tqdm
        self.tqdm = tqdm(**kwargs)
    def write(self, b):
        bytes_written = self.fd.write(b)
        self.tqdm.update(bytes_written or 0)
        return bytes_written
    def __enter__(self):
        self.tqdm.__enter__()
        return self
    def __exit__(self, *args, **kwargs):
        return self.tqdm.__exit__(*args, **kwargs)