The public enemy of junior app developers: push notifications

As developers we run into all kinds of challenges, each one more illogical and bizarre than the last. But the true nemesis of any app developer is push notifications, don't let anyone tell you otherwise. You can spend years fiddling with some pointless screen and nobody will bat an eye. Fire off one bad push to production, though, and everyone will remember your name.

Let me explain.

Back when I was a junior, my mission at the time was to build a system for sending push notifications to the apps we were developing. So I rolled up my sleeves: I dug into the SDKs, learned how pushes work under the hood, and figured out the best way to build a server for the job.

Once I had everything I needed, I got started. I spun up a test app, generated the right certificates, and got to work. It was flawless: pushes went out, and only to the right places, complete with icons and metadata the app could use to decide what to load. Great, right?

So, let's roll it out to every app. Why not.

The first apps shipped with my system baked in, and we started running controlled tests. The usual stuff: send a push by location and device ID. Worked like a charm. We layered on more filters, time, app opens, screens visited, and it all held up perfectly.

"So then, Kyngo, why are you even writing this article? Sounds like it all went great for you," you might ask.

It didn't. Trust me. The fun's only just getting started.

We rolled the system out globally. Hundreds of thousands of people were using our apps every day, and everything was humming along. Then, one day, a notification popped up in one of our most popular apps...

"test"

Thirty seconds later...

"test 2"

"test last opened at ..."

"test this one will work"

Everyone turned to me, eyes like saucers. I looked back, gave a nervous laugh, and asked what was going on. They shoved a phone in my face. My eyes went wide and I blurted out a "WHAT?!"

One of the developers pulled off his headphones and clocked the scene. I asked what he'd done, and someone showed him the pushes. His response?

"Oh, that was me. Just testing that the API works."

In production.

He was testing the system in production. A senior developer with nearly twenty years under his belt. You can't make this stuff up.

But Kyngo, this is supposed to be about pushes and junior developers, why are you telling us this? you might ask again.

Two reasons. First, to prove that "senior" doesn't always count for much. And second, to show you just how randomly pushes can turn into your worst enemy. You'll see.

Two years after that incident, and one developer lighter on the roster, we had to build a check-in system powered by push notifications. The idea: you'd walk into a venue, the staff would scan the QR code on your screen, and once processed you'd get a push welcoming you in.

I started building it and testing in a controlled environment. Everything seemed to work great.

Seemed.

We shipped it to production after testing and clocked off for the night. The next morning we walked into the office to find the CEO fuming. The push system hadn't worked properly. But not in the way you'd think.

Here's what happened: when the first person walked in, they got a notification, "Welcome to the venue!" So far so good. When the second person arrived, they got theirs too.

But so did the first person. Again. And it kept snowballing. Welcome, guest one. Welcome, guest two. Welcome, guest three. All night long.

The CEO was livid, because we'd blasted a single device with over seven hundred welcome notifications. The second person got six hundred and ninety-nine. And so on, all the way down the line.

What went wrong? The Lambda function that sent the pushes was choking on an internal AWS hiccup that slowed responses down, causing the function to time out... before it could clean up after itself.

Thirty seconds is more than enough to push to a thousand-plus devices under normal conditions. But if AWS stalls on a single push for that long... well, you get what you get.

The lesson? Don't use Lambda if you don't really know what you're doing. :)

We all came out of it understanding how Lambda behaves under the hood, but it was a stark reminder of how much we still had to learn about development and proper QA.

For all I know, the system I built might still be running today. I just hope they've patched the rookie mistakes I made along the way. And there were definitely more of them.

Cheers, and may your code be resilient!