‍

How do you go from zero to a million…in one second…with no warning?

With a global sporting event that garners over a hundred million viewers, that’s the question we needed to figure out as an engineering team.

‍
But before we dive into that question (and…it’s definitely a big question), let’s first set the scene with some context.

At Flowcode, we produce a suite of products that connect people in the Earthverse to the Metaverse — AKA, from the offline world to the digital world. Our primary product is a next-generation QR code that is backed with top of the line design, analytics, and privacy compliance. When someone scans one of our codes, they are brought to a digital destination. In addition to driving scanners to a product page for fast commerce or CRM collection form (to name just a few destinations), the creator of the code also has access to engagement metrics that help them make business decisions.

The main takeaway you need to know is that we’re in the business of connection. But not just any connection — fast connection.

Okay, back to the story. This past weekend, during the biggest football game of the year, our engineering team at Flowcode had the opportunity to work with multiple national brands. QR Codes were added to their 30 seconds ad spots, with the goal of driving more viewers to their respective channels. And the codes were lightening fast.

‍
This didn’t just happen by chance. It took weeks of hard work. Testing, failing, then testing again. Over and over and over. Here’s how we did it:

Considered the key challenges:

When commercials often receive more attention than the game, you need to bring your A-game. And when the brands you’re partnering with are spending $7 million on 30 second placement, you need to be flawless. Those are the social challenges. From a technical standpoint, we had to consider:

• Traffic for global sporting events is extremely challenging due to the massive amount of traffic that appears instantly.
  

• You need to be able to go from 0 to millions of requests per second with no warning.
  

• Engagement from events like this would normally be considered a denial of service attack, which is a technique used by hackers to take down websites. But, this traffic is legitimate!

How our team overcame this:

To address the technical challenges at hand, extensive testing and configuration was necessary. We needed to be able to serve content from multiple data centers in order to provide — this provided redundancy and helped distribute web traffic across the country, and the world. Additionally, our team:

• Scaled all of our servers before the traffic happened. If your team finds themself in a similar situation, you don’t have to scale in response to load, you need to scale ahead of load.
  

• We utilized a content delivery network or modern ‘edge caches’. This distributed the content close to users.
  

• We load tested. A lot. Some QR codes during past sporting events have crashed. We ensured we wouldn’t be a part of that storyline.

Our team profiled all services in the request path and spent hours making them as efficient as possible on serving the requests — starting from the entry point of the request to the exit of analytical data into our analytics store.

‍

The journey started months ago.

When we confirmed we’d see our codes alongside a global audience, we got to work. Early conversations began in December. The dedicated effort started in January, though. The order of operations were:

• Audited all of our systems. What was in the ‘hot path’ (required for redirect) and what was in the warm path (real time analytics).
  

• Divided up key systems. We allocated the work to different teams. From analytics and redirect services to AWS and Kubernetes infrastructure, dividing and conquering as a team was essential.
  

• Found the bottlenecks. We profiled and optimized our applications. Throughout the hunt, we were able to get our redirect service to respond in microseconds. Performance is key for scale.
  

• Put WD-40 on our network infrastructure. We needed an analogy in this piece somewhere. This feels like the right spot. Our SRE team tuned our infrastructure by switching to more performant Network Load Balancers in AWS and IPVS networking with Kubernetes. This reduced latency.
  

• Refined our DNS. We tuned our global Cloudflare DNS system by balancing global traffic across our AWS regions.
  

• Got real. We improved our real time analytics pipeline by refactoring several processing stages.
  

• Retained our integrity. Data integrity is important — we had backup plans to offload and preserve data if our real time pipelines became overloaded, which they didn’t.
  

• Load test. Load test load test. Sending millions of events concurrently to ensure we could handle whatever load came on from over a hundred million viewers.

Hard work paid off

Our team took what seemed like an impossible task a few weeks ago, and got everything to the point where the only limiting factor was the load testing tools we used — not the service. Meaning, we couldn’t generate enough load to break anything without throwing an excessive amount of compute at our systems. We over-prepared, and it paid off.

Now, we’re ready for any global event you toss our way. We’re not just any QR code provider, we’re a partner. That QR code you found online on some random website that you were thinking about tossing on your campaign? We recommend thinking again about that one. As you may have seen during the big game, there’s nothing fun about spending $7 million on your commercial, then having the $5 QR code you put on it crashing. Stick with our team.

‍