blog - git - desktop - contact

2026-08-05

This is my hex editor bine:

It's all black and white, basically. A while ago, Vim's xxd gained the ability to color the output (this is not included in Vim Classic, which I use):

I wanted to have this feature in bine as well. So I started implementing it. An early draft can be seen here:

I think this is already an improvement, because it's easier to spot ranges with mostly ASCII text. But it's not that great. More colors would be needed and it would probably be better if the hex columns were right next to each other, instead of being separated by a space.

My TUI framework movwin uses ncurses under the hood (this is entirely intentional, because I value ncurses' feature stability). The framework itself is written in Python, ncurses is a C library. I had already noticed that making calls from Python to ncurses can be very expensive (whether this is a problem in general or just with Python + ncurses or simply with ncurses itself, I don't know, and it doesn't matter).

Without colors, bine writes an entire hex line using one curses call. With colors, there would have to be many calls per line. That would be too expensive.

And the framework itself is also not particularly well suited to using lots of different colors in a speedy manner. It has a hierarchical palette system and all that is costly. And I would have to maintain at least two versions of the color palette, because movwin supports a dark and a light theme by default.

Eventually, I came up with this:

Instead of colorizing the hex column, bine uses special characters in the ASCII column -- ones, that normally cannot appear there, so there are no conflicts. NUL bytes use U+2593 (▓), bytes outside of the range 0x20 <= byte <= 0x7E use U+2593 (░), and the rest are printable ASCII chars that are shown as is.

This doesn't cover all the same cases as xxd does, but it's good enough. You get a quick overview over what's "plain text" and what isn't, NUL bytes stand out.

It was super easy to implement and it's cheap at runtime.

I also wanted to have a similar visualization feature for entire big files. The tiny ASCII pane in bine can only show so much. I wanted this for something that's a couple of megabytes in size, maybe gigabytes.

My visualization essentially is an image. So ... why not coerce the computer into interpreting an existing binary file as an image? Don't do any processing, just take all the bytes in the range of 0x00 thru 0xFF and map them to pixels.

A Portable Graymap is pretty much exactly that. The Wikipedia example shows the ASCII version of this format, but there's also a binary version.

So what I had to do, was basically just this:

printf 'P5\n%s %s 255\n' "$width" "$height"
cat "$infile"

Done.

The only "logic" I implemented was determining a good width and height.

The result can be shown in any image viewer (on Linux and BSD).

Here's a visualization of an EXE file of GORILLAS.BAS, unscaled:

(Yes, that's the entire game. You could even take this image and convert it back to an EXE file.)

What can we see here? Let me annotate a few interesting sections:

1. At the top, we have an almost "random" area. Very dark pixels, very bright pixels, gray pixels, it's all there. This is code.
2. Towards the middle, we have a very regular pattern. I have not yet figured out what that is, it's a story for another day. (It looks a bit like a Relocation Table, but it's at the wrong position. This file, however, is an EXEPACK file, which complicates things. We'll see.)
3. At the end, there's a longer section of "darker gray". It's relatively uniform. This is ASCII text: The high bit is never set, so all these values are distinctively smaller than code bytes. (Program files often have such a section towards the end, it's where the string literals are stored.)

I think it's nice that there is no pre-processing at all. All this simply exploits the properties of the data.

As a more "explicit" example, here's a tarball with two files in it, a text file and an image:

Also, notice the "pattern" in the text area? Looks like a tiled wallpaper? That's because the text is repeating.

Just some more examples. Here's a scaled-down visualization of ruff:

An unscaled excerpt of the black stripe of mostly NUL bytes towards the top:

Lots of neatly aligned data and tables.

The ruff binary is 25 MB in size and so is the resulting PGM file (4096x6225 pixels), so I scaled it down, because I don't want to have such a large file lingering in my blog forever. But GIMP and nsxiv can easily handle the PGM file, there are no performance or usability issues at all.

Here's a visualization of a 690 MB .iso file (heavily scaled down, of course):

Creating the PGM file takes 0.2 seconds, resizing the image using ImageMagick takes 6 seconds. This shows some limits of this approach: I tried doing this with a 4 GB file, but ImageMagick required too much RAM. I think that you could make this work, in theory, by iteratively resizing the image first (put only a couple of rows at a time into memory), then compressing the smaller version. I have not written such a tool, yet, and ImageMagick doesn't appear to implement this (or I don't know how -- haven't searched, though).

Still, depending on which kind of data I'm looking for, this can already be a useful tool. Inspecting large files in the range of gigabytes is something that I very rarely need to do -- if need be, I can easily split it into smaller parts. Just as a demo, here's an old 8 GB disk image:

Mildly interesting to look at, but as I said, depending on what I look for, it can be helpful.

I think I'll keep it just as it is now.