Moiré no more

I learned a thing long time ago, and it was: Once you print something, you can’t get it back.

On the surface, this statement doesn’t make any sense. Scanners are cheap, and in 2021 some digital cameras are as good as scanners, too. Optical character recognition is so fast that it can happen in real time, leaving enough processor power for simultaneous translation to a different language. And I myself bragged about scanning almost 400 printed items and putting them up on the Internet Archive the other day.

But something happens to photos in particular when they get printed. Pixels only make sense on displays. To survive in real life, pixels have to get translated into halftones – complex patterns of translucent cyan, magenta, yellow, and black dots that overlap, vary in size, and are rotated at strange, yet deliberate angles.

Here’s a simulation of printing a photo with smaller and smaller halftone dots. When the dots are big, you can only get a sense of the photo by squinting. But as you make the dots smaller, squinting becomes unnecessary – the photo on paper looks exactly like it did onscreen even as, under the hood, it’s put together in a very different way.

Grab a magnifying lens and examine any photo printed in a newspaper or a book, and you will see a similar complex pattern of dots. But if you find such a printed photo and scan it, you don’t automatically get its pixels back. No, you get new, smaller pixels that replicate the exact nature of halftone dots. And with those confused pixels, you also get one more thing: a warm hug from something called moiré.

Moiré is named after a type of material where a special process adds an organic, shimmering, and generally admired effect:

Moiré in silk is intentional, but the modern effect named after it – occurring when two similar frequencies overlap and create strange, pulsating patterns – is almost always something to avoid.

A version of moiré happens when you try to hum with your vacuum cleaner, and your hum gets really close in pitch (seriously, please do try this). It happens when you try to see through two chainlink fences – the links get out of and in phase with each other, creating a weird effect:

And it happens a lot in digital scanning and photography, because the sensor elements in scanners and cameras are on a grid, and therefore have a frequency of their own – a frequency another frequency can interfere with.

This is why taking a photo of just one fence with a digital camera can give you this:

This is why nobody will allow you to wear stripes on television:

This is why taking a photo of a completely white screen at an angle can result in weird splotches of colour:

And this is why dealing with a printed photo can be such a pain in the ass.

⌘

First of all, moiré can already happen when scanning a printed photo. It can happen later when the photo is resized. And when printing an already halftoned image and having it halftoned again, possibilities for moiré to appear are basically endless.

For all of these reasons, I tried to avoid putting once-printed things in my book. I reached out to newspaper archives for original photos, rather than scanning newspapers. I did the same for books. When I needed to, I took my own photos or asked others to take them for me.

But there were times when I didn’t have a choice. You know how much I care for a good photograph, and right here was another perfect example: an evocative hundred-years-old shot dripping in what used to be the future, coming from a company that no longer exists, scanned from perhaps the only remaining copy of said company’s old internal newsletter:

It was a great photo from a distance. But when I zoomed in, the halftone dots were clearly visible:

And even simply resizing the scan was like buying a ticket to Weird Pattern Central:

What to do then? 

There is a tried-and-true way to remove halftones, and it’s the digital equivalent of something I mentioned above: squinting. If you blur the photo enough, the dots disappear:

That might be good in some cases, for example when the reproduction is small in size. But it’s a blunt, naïve technique. Blurring doesn’t just remove halftones – it also removes a lot more information around them. On the way to lose the dots, you lose a lot more.

All the demoireing guides tell you to apply a sharpening filter after this to compensate, but that’s like reheating cold pizza. You will never get back to where you were before, no matter how hard you try:

I procrastinated and dreaded removing moiré on this and a few photos like it, because I was worrying about them becoming too blurry. 

But some of my wait was also deliberate. The world of digital image processing is not standing still. In recent years, there have been many examples of machine learning algorithms that seemed rather… magical. Machine learning cannot recover information from a blurred photo, but it can add new pixels based on what it learned from other photos in similar situations. If done well, this new information is exactly what the original information would’ve been, or close to it. It’s, basically, cheating – but it’s better than nothing.

And so I asked the friendly nerds of Twitter. I showed the original typewriter car scan, added my blurred-then-sharpened photo as a pathetic comparison, and asked: what is the latest in demoireing? Is there some new tech that could help me?

I didn’t get any obvious machine learning hints. Many people responded with downright resignation: what you did is still the best out there. But a friend of mine – a neuroscientist from Utah – had an answer I didn’t expect.

“You don’t need ML,” Bryan said. “What you need is inverse FFT.”

⌘

FFT – Fast Fourier Transform – was invented in 1805 and then, once again, in 1965. I learned about it long time ago in college. I never got it and I still don’t think I fully get it today. But the basic idea seems to be: applying FFT to an audio sample allows you to see its underlying frequencies, separated and exposed. (FFT-ing a recording of pressing three piano keys, for example, would tell you exactly which three piano keys were pressed.)

But since it’s not just audio that exists as waves, FFT can be applied to photography also. Here is an example of the same above photo of multiple generations of people admiring the vaunted Space Cadet keyboard, right next to its FFT:

There is not too much going on in the FFT as the photo is messy, without too much of an underlying structure. But take a look at another photo now, a close-up of an old computer monitor:

Now, the FFT tells a bit of a story: the thick horizontal line matches the horizontal lines of each character, and the dots represent various other reoccurring frequencies.

How about this random photo of the IBM 3277 terminal keyboard I have lying around?

Now, the FFT is really flying, showing strong lines matching the ones on the photo.

But this pales in comparison to the typewriter car photo I wanted to reuse, the one with all the dots, where we can see the FFT immediately betraying their repeated presence:

I got those FFTs by installing a strange piece of software called Fiji: an image processing powerhouse for scientists – “the original killer app for the Macintosh in bioscience labs,” my friend noted – with a user interface only a parent could love:

It’s always fun to play with some new nerdy tool. “This is cute and interesting,” I told my friend, “but how does it help me with my original problem?”

He responded by saying that FFT is not just for admiring. You can manipulate it and then – via inverse FFT – transform it back into an image. 

And how do you manipulate the FFT? You take the piece of software written by PhDs for PhDs, and treat it in a way a little kid would: by painting over the white peaks with black.

⌘

This sounded like a prank. You’re telling me that a problem I’ve witnessed for decades could be solved with a 1960s algorithm, and I don’t even have to be particularly careful?

But I tried it out. I started crudely drawing over the peaks, one by one. Things were weird at the beginning, but then I saw something astonishing – the halftone dots started shrinking:

When I was done, there was still a little bit of a semblance of the original structure, but only when I zoomed really closely. Otherwise, it’s like the halftone dots were never there:

This was miraculous. Messing with the FFT removed the dots and nothing else – the image was still very sharp, especially if you compare it to the naïve blurring method:

The comparison to the original, halftoned scan, was also great – now you could clearly see “Bloomington, Illinois” printed in a small font at the bottom of the truck’s gate.

After the return trip to the FFT land, the photo became almost as good as I imagine the original photo did. I could resize it any way I wanted, without ever again worrying about moiré.

Or, in other words, I could bring it to the printer without worry.

⌘

My mind was blown seeing all this happen during one mid-week evening. I didn’t understand it then, and I barely understand it now, which added to the sense that this is some sort of witchcraft that should not be possible. But it isn’t that – this is pure, old-fashioned math and science, which feels particularly meaningful in America of the summer of 2021.

Somehow, through hard work of people much smarter than me, I could go deep into the photo and not only inspect its frequencies, but actually change them. Imagine being able to edit the nutrition label and see the food transform in front of you. Imagine drawing on top of a weather forecast and summon rain this way. I was so excited I couldn’t sleep. 

Is FFT awesome? Yes. I want you to know that. It’s applied math that a theoretical mathematician would approve of. (If you’re so inclined, you should check out this video explaining its ingenuity in more detail.)

But I also wanted to share this with you because of how much this moment meant to me.

I’ve always had this theory that any long-term project requires two ingredients: things you’re good at, and things you want to learn. The first group gives you a feeling of accomplishment and mastery. The other one? It keeps things interesting.

When coming to a large project without some things you’ve already mastered, the entire endeavour can feel overwhelming. But without anything new to look forward to, it can become stale and repetitive.

Or here’s another lens: When you’re facing one of the many hard moments, something you’re good at can help you feel awesome. But when you’re bored, something new to learn can remind you how there’s more to life than feeling awesome all the time.

I’ve done enough creative projects to know that the cliché is actually true: the journey really is the reward. The moment of my book’s publication will be hard, really hard – perhaps less of a wedding and more of a break-up. There will be a typo I spot that can no longer be corrected. A mixed review that will touch on something I’m insecure about. And, more than anything else, a hollowness of a multi-year project that cannot immediately be filled by anything else.

Knowing all that, I promised myself to keep the book process itself interesting, and I’ve been determined and lucky enough that this has been the case so far. I’ve learned about interviewing, editing, proofreading, finding things, cover design, restoring keyboards, mortality, and many other things. I’m currently in the process of learning about paper and printing (more on that very soon), not to mention marketing.

But that moment when I was taught Fast Fourier Transform and within an hour went from “I guess blurry it is” to “holy shit, I am the god of sharpness”? It immediately became one of my favourite moments. And that this happened during the last year of the project, with the publication on the slowly approaching horizon? For this I will always be grateful.

This was newsletter №26 for Shift happens, an upcoming book about keyboards. Read previous issues · Check out all the secret documents