View profile

Lessons In Preventing Mistakes from Our Stripe Webhook Outage

Lessons In Preventing Mistakes from Our Stripe Webhook Outage
By Mastering JS Weekly • Issue #75 • View online
Recently, Mastering JS’ Stripe webhook, which is responsible for sending eBooks to customers once they’ve paid, was down. If you were affected, I’m so sorry, please email me for your missing eBook.
What happened? We recently migrated the webhook from an Express server on an EC2 VM to Azure Functions as part of our efforts to consolidate our different handlers into one easy-to-deploy codebase. But, as part of that migration, we missed setting our Mailgun API keys. And the mailgun npm module doesn’t fail fast when you pass it an empty API key.

Smoke Testing, and Why It Sometimes Fails
In addition to TDD, we usually recommend our clients have some sort of automated smoke testing in place that lets you spot check that your API isn’t completely broken. A good smoke test is fast and simple. Nothing stateful: no logging in or updating data.
In our case, Mastering JS’ smoke test process pings the `/status` endpoint to make sure that Azure is serving the correct version, and then makes an OPTIONS request to every API endpoint. Every endpoint is configured to respond immediately if an OPTIONS request comes in.
Our endpoints always respond with `{ ok: 1 }` to OPTIONS requests
Our endpoints always respond with `{ ok: 1 }` to OPTIONS requests
This smoke test is usually good for catching devops snafus that normal local tests won’t catch: missing configs, missing dependencies, delayed deploys, syntax issues in configuration files, etc. But our smoke test didn’t catch this case, because mailgun doesn’t fail fast when it has an empty config.
Failing Fast
We are firm believers in using code to make sure you never make the same mistake twice. In this case, we decided to wrap our config object in an ES6 proxy that throws an error when you access a config value that isn’t set.
Wrap configs in a proxy that throws an error when accessing unknown properties
Wrap configs in a proxy that throws an error when accessing unknown properties
Combined with our development practice of only accessing config in global scope, this code will ensure that our smoke test catches missing config properties. OPTIONS requests will still fail, because the function using the missing variable will throw an error in global scope.
Failing fast makes smoke tests more effective. This is especially important in serverless setups: with a conventional server, it is easy to tell if a server is down. But with serverless, you need to check whether individual functions are down. And if a function fails fast when there’s some obvious devops error, smoke tests can catch the issue.
Most Recent Tutorials
The deleteMany() Function in Mongoose - Mastering JS
How to Use Mongoose's updateMany() Function - Mastering JS
What We're Reading
Source Code Readability, Elegance, and Complexity
Designing Error Messages and a Logging Strategy in Node.js | AppSignal Blog
Did you enjoy this issue?
Mastering JS Weekly

Pragmatic web development. No bloatware allowed!

In order to unsubscribe, click here.
If you were forwarded this newsletter and you like it, you can subscribe here.
Powered by Revue