Hi there!

Yesterday I heard sad news — which quickly became good news (the good news is one of the tweets in this letter). Other than that, we’ll discuss the algorithm behind Spotify’s shuffle, see the pandemic’s impact on shopping with beautiful visualisations, reverse-engineered Resume ATS and my talk on personal websites.

Let’s dive in!

When you hit shuffle on Spotify (or Apple Music, or YouTube), do you get a random list of songs to play one after another? If you have ten artists, all with around 100 pieces each, you will likely have 13 back-to-back songs by the same artists by the time you finish the first 100 songs (little simulation).

Spotify was bombarded with complaints from users, and they worked intensely on this. Initially, Spotify used the Fisher-Yates shuffle algorithm for creating a perfectly random playing queue. It isn’t complicated.

Simple, right? Well, listeners didn’t like the results. The engineering team had to define a “more-random-sounding” algorithm. Ultimately, they used Floyd-Steinberg dithering (an algorithm primarily used in image compression) to space out songs by the same artists. The algorithm can then be used recursively to space out pieces from the same album.

thenewnormal.is is a visualisation project that explores the general interest in different products before and during the Covid-19 pandemic. The products are grouped into three categories: normal, new normal and unusual. Search interest for normal products remained consistent through the pandemic. Products, where search interest increased and remains high, are called new normal. Unusual products suddenly got popular and are now back to old levels; even though COVID cases are not :(

The website is a rabbit hole and you can explore search trends for hundreds of products. The website gives you an option to filter products by category as well.

If you have created a ggplot2 plot and want to add interactivity, here’s a simple way to do it. Just pass the plot p to ggplotly().

And by interactive, I mean you can hover on a point and get all the details; select points using a box or lasso tool; zoom in and zoom out; and even download the resulting graphics!

Well, not really. In the modern world, being on the job market means applying through hundreds of portals, most of which wouldn’t clear your resume through their ATS considering it filters 70% of applicants.

A (spoof) platform developed by Jess Peter provides you tools to provide you resume content, profile picture and profile video and even thoughtful leadership tweets. The project uses JSON resume to create Resume content, NVIDIA’s StyleGAN2 model which has been trained on 70,000 photos of people to create a profile picture, GPT-2 algorithm to generate thoughtful tweets and First Order Motion Model framework to animate still photos to act out recorded facial expressions.

I won’t recommend using it for actual applications, but it is definitely click-worthy.

Digital identities have come a long way since the internet. Physical identities gradually disappear, and projects like Aadhaar in India have transformed governmental identification. Today, our digital social identities are owned and managed by Twitter, LinkedIn, and Facebook.

Last Saturday, I gave a talk at the Trenton R Users group on creating and controlling our digital social identity. How? By having a personal website. For the first half of the talk, I argued why personal websites make more sense today than ever before. Later, I jumped on to a hands-on website development on Owlstown. Believe it or not, it takes a little more than 15 minutes.

We recorded the session, and you can view it for free on YouTube. Other materials and sources are present on my website.

plotly is an R package for interactive web-based plots in R. You can click on a point to learn more, pinch to zoom, and all other familiar gestures work as well. Check the guidebook for different plots you can make.

quanteda creator argues you should drop all other text mining packages in R and start using this for all things text. Check the guidebook for more details.

pavo provides several tools to manage the spectral and spatial analysis of colour patterns. It provides flexible solutions to input spectral data, processes it, extracts variables and produces graphics among others. See the vignette for more details.

tmap is a package to produce thematic maps in R. The syntax for creating plots is similar to that of ggplot2 but tailored to maps. Check the vignette to learn more.

Here’s the link to the spreadsheet with all packages mentioned in past editions.

pull() is used to extract a column from a data frame while working with pipes. df %>% pull(var) is way neater than using df %>% .$var.

Fair Bet: Fair bet is a bet for which the expected outcome is zero — after accounting for the cost of participation.

Sometimes you may need to edit the image in R — converting to black and white or ilk. The image_convert function from magick allows many options for doing so!

With R Function A Day discontinued and me running out of “interesting-but-not-complicated” jargon, I have decided to include R functions in the Three Jargons section of this letter.

As always, feedback is welcome. See you next week!

Harsh