Huge effort and just getting started

I've been seeing posts about front-end devs fearing that AI is going to take their jobs. Is it just me, or is front-end not as trivial as people make it out to be. Building a performant web app ends up being a systems problem : how to lay out your data flow, SSR vs CSR (will be writing a longer thread on this soon), infrastructure hosting, incremental re-computation and more. The guys who can do this well are studs

I love working with @vercel from a DevX standpoint, and it truly does have some of the best support for NextJS as a programming framework... BUT I think that @Cloudflare 's infrastructure takes the cake. They're just faster for a globally distribted audience (Note: we use both for different things)

We're gonna Make Points Fun Again @monkeytilt

Ryan's been one of our earliest supporters. Super stoked for this

For the largest of my fellow degens, @monkeytilt just pushed our High Limit section to the Casino lobby. Enjoy : monkeytilt.com/en-ca/casino/…

Scaling problems 101 : @monkeytilt is experiencing a large growth in DAUs, and we spent a lot of today dealing with classic issues in database scaling for our backend services Small 🧵on how we think about scaling

We dropped the Leaderboard this week @monkeytilt, super proud of the team and could not be more excited. I like to refer to this internally as the "Make Points Fun Again" campaign. A small 🧵

@SunWukong_MT claims he's doing QA on our original games, but I think he's just addicted. Peep him "crash"-ing and burning below: monkeytilt.com/casino?bets-s…

No diddy!

Big fan of OpenAI building all our childhood fantasies. Now need to build an AI bookie openai.com/index/hello-gpt-4…

Hard to get back to coding after that night. What a performance by Ryan, against one of the P4P greats. @monkeytilt sees talent boys!

@SunWukong_MT is also bald

Happy to be along for this ride

Hats off to crypto for breathing life into the hugely slow moving online gaming industry. Operators of crypto casinos have set new standards on product innovation, pace and UI/UX Have a hunch we're going to see a large influx of talent soon as problems become more compelling 👇

Business Intelligence (BI) is one of those areas that seem like it's just an add on layer to your data warehouse. I think this trivializes one important assumption : the design of your data warehouse. Over the last few months, as our data has grown, we've been researching / implementing novel data warehousing techniques. Our metric of success is being able to deploy our in-house ML segmentation models in under 5 minutes and have them feed to our BI team right after. Getting there!

Pussymonster extending his lead on the @monkeytilt leaderboard, with a surprise entry and meteoric rise from Mattdeuce32

Predictable1 with a MASSIVE run @monkeytilt

What salary guys?

🔥🔥

A monorepo creates more problems than it solves if each sub-project tries to be independent in the way its built and versioned. We've worked with monorepos enough to appreciate this, however we thought we could get by with small disjunctions until reaching true scale However, a small incident on Friday , so we've taken the weekend to revamp our build system, CI pipelines and sem-versioning system. Its boring work, but one that pays massive dividends

To anyone who has dealt with race conditions / undefined behavior at scale : had a pesky little case with our cookies today that caused non-determinism hell! Kudos to the team for figuring it out in under 30 mins

(7/9) API calls can be optimized in a bunch of interesting ways. One thing we found success with was matching the API re-computation to the data access pattern. How often do the trending games really change? Every 24h? We just spun up server tasks that push this data onto a CDN. Client side work = close to 0

The boys are eating tonight!!!!! nitter.app/STONKS_IU/status/17819… @monkeytilt

(1/9) Online casinos are "content heavy / content catalogue" apps (think Netflix, Spotify) with thousands of games, sports and lines. Users expect snappy interactivity, in spite of a high volume of data A 🧵 on what @monkeytilt is doing to keep FE performance up

Super excited for Runes. Thinking of some cool interplays between them and @monkeytilt

@monkeytilt we chose to structure all our code as a monorepo, despite having many diverse projects : multiple front-ends for our product verticals, various different services in languages ranging from Java, Golang to Terraform and more. I'll be writing a long form on this soon, but this article from Google summarizes a lot of the reasons : cacm.acm.org/research/why-go…

At @monkeytilt we're building everything from microsecond scale systems, RAG infrastructure for domain specific problems to smart contracts and on-chain stuff. Don't know too many opportunities for that kind of breadth

I've seen multiple interesting solutions to this over my career. We're going to try some interesting stuff and I'll write back soon (stay tuned) Only after we're at the limit of what we can do will we scale up resources + replicas

UFC 300 was not only an amazing card (let's go Holloway and Pereira), it was also @monkeytilt 's highest volume day! Slowly but steadily building this thing

So true. Resolving to read = dopamine spike, reading = dopamine trough, finishing = dopamine spike. Been trying to double up this way

(2/9) Generally, FE performance is measured on three axes: - Resource delivery : how quickly are bytes shipped from your server to my browser (metrics such as TTFB, FCP and LCP) - Browser work : how much work does my browser have to do once bytes are shipped (metrics such as INP) - Visual stability : is it easy on the eyes (metrics such as CLS)

In the last some days, we've been noticing increasing load on the DB (this was expected, it wouldn't last forever). Rather than scale up immediately, however, we decided to extract as much as we could from the same hardware to make for interesting lessons in systems perf

You can actually see exactly where this Cache Stampede occurs below. All new requests, finding invalidated cache entries, blast the DB

(5/9) What we did to make X as small as possible: - Preload / prefetching comes for free with Next, but you can provide hints with tags as well. This amortizes X - Asset optimization : webp is a modern format we use, but we also played with image sizes + resolutions - Moved all fonts to the CDN (we use custom fonts, but this is static data) - Lots and lots of bundle profiling. NextJS 14 really doesn't require many external deps - Heavy use of React Suspense : our infra automatically streams these to the browser

Happy customers!

Phenomenal read

Major major major haram. This is what I get for staying home and fixing observability issues for our stack

(3/9) Each of these are fairly intricate tasks in and of themselves, but (as expected) follow a Pareto distribution in the effort required to fix Will talk about production measurement in a separate thread, but this is a must to make sure changes are actually working

We added a cache middleware for our API requests (caching on the Body, Content-Type and Status) and API caching on our CDN. This reduced our database load by ~50% Issue with caching under high load is a large num of cache keys tend to expire at the same time, generating a Thundering Herd

Quite surprised that there hasn't been more work done in large scale recommender systems for online gaming. Not one product I've used does anything more sophisticated then some broad local profiling + category collaborative filtering / popularity ranking

Crazy

Can't put into words how much I appreciate @ongardieand Prof. Osterhout for Raft (raft.github.io/raft.pdf) My first large-ish project in distributed systems was implementing it ground up in C++ for on-disk state management, exactly how was described in the paper (over RPC). My initial thoughts were: (1) wow this is so much easier on the programmer than Paxos and (2) I'll be surprised if a large number of future systems don't eventually migrate their consensus. However, I didn't foresee its use at the microsecond scale. Replacing the RPCs with a compact VLE messaging schema, shared memory and kernel bypass networking actually allowed us to use RAFT even in maintaining in-memory state on exchange grade systems, operating at millions of TPS!

(8/9) For visual stability (CLS), properly sizing images and adding Suspense boundaries while they loaded made sure things stayed put. We also tried invisible text during font loading so there were no flashes of old font

(4/9) Resource Delivery is concerned with how quickly you can move X bytes from point A to point B. Intuitively, you can improve this by: - Making X as small as possible : more on this later - Making the distance from A to B as small as possible : choice of infrastructure provider matters a LOT here. You want one that has many points of presence for both compute and data. We use Cloudflare for both edge compute and CDN data

They won't? Shite

Bigger downside having to deal with you everyday

(6/9) Browser work depends heavily on what kind of interactivity you want to offer. Browsers do a LOT of work from rendering DOMs, executing your JS code and more. Since JS uses a run to completion model, there's multiple thread starvation opps that come from poorly designed React Components A hacky thing we did early on was place an upper limit on the amount of stuff each React Component could do. Small components are your friends, because the browser can most likely inject higher prio work React Suspense is a godsend here as well, as Next streams HTML to the browser to amortize work

(9/9) Some other misc points: - We have a LOT of images in game cards, sports logos and more. Lazy loading all images below the fold once the user started scrolling was helpful - Server side compute : for data intensive calls such as fetching 100s of games, we keep those components on the server (NextJS has everything on server by default) - Recurrent timers run in the background on web workers - NextJS image optimization comes for free

It's been all over the internet bro. You've been too busy supporting customers

Traditionally, leaderboard points campaigns have heavily incentivized mercenary farming. It not only becomes a chore, but adversely selects the kinds of consumers while creating a subpar experience

We started off with a single DB instance on an 8 core machine (we knew we'd outgrow this quick, but cheapest option to start!). The main writers are backend stream processing services (Kafka msg persistence), and the main readers are API gateways

At Monkey Tilt, that just didn't jibe well. We had to rethink it. So Leaderboard V1 is just the start. Over the next few weeks, me and the devs will be tirelessly working at the game that is points accumulation. Be prepared to see lotteries, bonuses, points utility like no other platform, and more! Super excited for it