San Francisco, CA
- Tweets 11,691
- Following 162
- Followers 45,918
- Likes 35,261
SportsCenter
Many “anti-DEI” legal immigrants who celebrated Trump are now realizing that “America first” is fundamentally incompatible with their dreams of “global meritocracy”.
And that Trump’s core base is comprised of people who find their presence in America to be rather undesirable.
134
601
4,387
269,138
From personal experience, I know how hard and frustrating it can be to be an Indian immigrant stuck in a never ending green card queue, but I doubt acting entitled, or worse, *disparaging* other nationalities as less deserving serves the cause very well.
Let’s not be jerks.
63
88
2,957
413,170
The detailed AWS incident report is out, and it’s worth a read
- DNS records managed by 2 systems; a race condition led to regional record getting unset
- EC2 lease establishment was borked as it depends on DynamoDB
- fluctuating NLB health checks leading to EC2 DNS entry purges
55
411
2,463
473,753
Ok, Elon going thermonuclear on nativists and racists, defending the H-1B visa to *this*,degree wasn’t on my bingo card.
A very pleasant surprise, but given the amount of vitriol and hatred thats been aimed at immigrant tech workers, this is extremely welcome.
229
138
2,179
116,117
Microservices or not, one of the best things you can do to improve your own productivity is to have a fast, local development environment setup with minimum dependencies and ways to introspect isolated parts of your codebase.
25
346
2,150
Thinking of the time Elon publicly fired an Android engineer for saying that the app was slow because of technical debt and a full rewrite is needed to address perf problems.
3 months on, Elon himself proposes a full rewrite himself, because a config change caused an outage. 😅
19
171
1,881
166,733
- Microservices are hard.
- Building reliable and testable microservices is a lot harder than most folks think
- Effectively *testing* microservices requires a ton of tooling and foresight.
- A Netflix/Uber style microservices isn't required by many (most?) orgs.
- Macroservices?
33
354
1,653
Full report for yesterday’s Google outage
- not feature flagging a new code path
- null pointer crashing a binary when reading empty fields from Spanner
- remediation causing a thundering herd as there was no exponential backoff
- which required manual throttling to recover from
17
147
1,576
199,186
The most ill-conceived idea that seems to have ossified among certain group of people is that Twitter was failing because the engineering culture wasn’t high-performing enough
The problem with Twitter was product and leadership, and not engineering not executing well enough.
34
85
1,263
Dropbox famously built their own data centres and moved off of S3 to reduce costs, and now moved *back* to AWS (DynamoDB, S3) again to save costs for their metadata storage, because it turned out scaling MySQL was hard for this use case.
dropbox.tech/infrastructure/…
17
223
1,262
Tech Debt is one of those things that make sense to engineering, but to leadership it sounds like “we’ve created a mess over the years that slowed the product, we did nothing to fix it, and now we need to spend *even more* time and people on fixing it”.
Not a winning argument.
94
132
1,146
The hill I will die on: technical debt is an engineering problem and is owned by engineering, not “product” and not your C-suite.
By engineering I mean the engineering org and those at the higher echelons, not the junior-ish engineers who are the leaf nodes in the org chart.
69
91
1,140
This was the woman who posted a picture of herself sleeping in the office to meet Elon’s deadlines.
At the time, a lot of prominent people lauded this, saying “this is how the most impressive software is built”.
Fast forward a couple of months and she’s fired.
15
123
1,036
314,868
MY BOOK IS OUT! 🎉
It’s a small (free) ebook (25 pages) so there’s only so much I could cover. I hope people find it useful. 🙂
Link - bit.ly/2rOf4og
48
373
1,057
I was on an H-1B visa for 8.5 years before I got my green card, and never once did I think I was exploited. I also changed jobs twice, so I wasn't an indentured servant. Stay in your lane; I don't need your paternalistic concern of trying to "protect" me from evil corporations.
260
66
955
143,368
Twitter tech influencers of the 2015-2021 era were, by and large, deeply technical people who had deep insights. Think Kelsey Hightower.
“tpot” is mostly edgelord chuds engagement farming and being contrarian without any substance or even any pretense of intellectual integrity.
43
47
1,050
282,963
Blogged: Best of 2019 in tech talks
Topics include:
- Linux
- concurrency
- Lots of malloc and mesh
- performance
- large scale computing
- serverless
- Spectre
This is also the first year I’ve included presentations from academic conferences.
link.medium.com/nw50nEyvz3
12
239
994
Best of 2018 in tech talks.
It’s a list of 30 of my favorite talks covering pretty diverse topics - concurrency in Kotlin, writing an OS in Rust, consistency without distributed consensus, autonomous testing, fast safe mutable state and a lot more. bit.ly/2Sp4ggp
14
315
921
You can expect some convos to crop up on twitter every 2-3 months like clockwork: 😅
- friday deploys
- should you work weekends and nights to be successful
- ssh into prod
- do job titles matter
- do degrees matter
- do algorithms/data structures matter
- should managers code
29
101
868
X engineer realizing what was obvious to many of us who paid attention *years* ago.
that Twitter 1.0’s infrastructure was so battle-hardened given the caliber of talent that worked on core infrastructure, that it would continue to pay dividends for years to come.
14
71
931
90,357
Seriously, I’m just baffled at their credulity.
“America First” is fundamentally a protectionist, populist, myopic movement, whether it comes to global trade, or foreign policy, or immigration!
It’s incompatible with free markets, free trade, and free movement of labor.
19
58
852
25,427
"GPUs will increase 1000× in performance by 2025, whereas Moore’s law for CPUs essentially is dead. By replacing branch-heavy algorithms with neural networks, the DBMS can profit from these hardware trends."
Whoa, this whole paper is 🤯 bit.ly/2CGgiZE
9
288
857
The paper I've been looking forward to the most is now out: zero downtime deployments at Facebook.
Disruption free release of services that speak different protocols and serve different types of requests (long lived TCP/UDP sessions, requests involving huge chunks of data etc.)
6
209
852
If Twitter manages to stay up despite this chaos, it’s a testament to the decade long infrastructure engineering efforts and investment in resilience and testing.
Twitter’s blog posts detailed many of these efforts.
It’s the sort of “unseen” work that pays off in the long haul.
14
105
772
Perhaps I should blog about all the recent (5-7y) trends in distributed databases?
There’s just some really, really, ridiculously interesting stuff that’s being built in this domain, and it’s worth capturing some of the super cool, cutting edge research in one place.
28
8
779
Very first stab at a comparison between AWS lambda, Cloudflare Workers and Fastly’s new wasm based runtime.
I started with the AWS lambda sketch and then adopted the same convention for the others too.
Counting on the internet to call me out if any of this is inaccurate 😏
15
225
752
If there’s one thing that’s become more obvious than ever it’s that you don’t need mega teams of engineers, PMs, evangelists etc to build something great and useful.
Small, focused teams led by someone with a strong vision and sense of product often punch well above their weight
10
223
719
The post mortem of today’s @Cloudflare outage is up: it’s a sobering read.
- testing in production (dark launch) isn’t without its perils. It can take down your production
- don’t do *global* deploys, even for staged rollout of feature flags/dark traffic. bit.ly/2FQN77k
15
285
680
On the one hand, tech CEOs claim to have “overhired” in the past 2 years, yet a closer look at who they are laying off shows that many long tenured, high performing people are being impacted, while more recent, less experienced hires continue to remain employed.
29
50
672
219,805
Wrote down all of my thoughts on testing microservices -
the tradeoffs, antipatterns, what I call the "step up rule", how to test with service meshes, testing in production vs pre production testing, as well as a new test pyramid for distributed systems
bit.ly/2lpyVI5
16
279
696
What are the biggest pain points you believe tooling can address in the next decade (2020-2029)?
I’ll go first:
- CI/CD. Jenkins is currently the CI gold standard and it’s a very low bar.
- Easier abstractions and paradigms for building infra. Kube is too low level + complex.
74
114
690
*Incredible* slide deck from Google from SRECon on architecting a large scale distributed system, including how to approach scaling, how to evaluate tradeoffs along various axes as well as tons of back of the envelope calculations to justify each decision
usenix.org/sites/default/fil…
5
189
669
Dropbox just announced it is becoming a “virtual first” company.
honestly, this is the first such post that actually makes sense to me. It addresses all the pros and cons of a “remote first” culture and proposes something that addresses all the concerns.
blog.dropbox.com/topics/comp…
10
168
657
The best PR reviews I’ve ever got included comments on:
- concurrency bugs
- software layering
- how to write better tests
The worst PR review I’ve ever got told me to make functions smaller.
That’s why I get so triggered whenever “clean code” is wheeled out in PR reviews.
21
67
634
Blogged: A decade in review in tech
medium.com/@copyconstruct/a-…
Wrote about some of the most important trends in tech this decade and some ideas on the pain points the next decade must address.
It’s still 2019 in SF, so this is my last post of this decade.
Happy New Year! 🎆 🎊
7
251
646
There is so much spoken about engineer shortage and how hard it is to hire good engineers. But a bigger problem seems to be engineer burnout. So so many fantastic engineers I know are completely burned out.
18
154
630
Every year I post a “best tech/papers of the year” list.
I try to write a detailed blog post, with a few lines on why I liked or what I learned from each of the items on the list.
Here’s my list for 2023, if anyone is interested. I’ll have the blog post ready by early Jan
7
89
651
50,735
Look, if your metric for who is “excellent” corresponds to who has the most time to be writing, talking, tweeting, publishing about what they do or how they think, then you’re going to miss out on a ton of incredible people who just don’t spend their time publicizing themselves.
12
80
612
Sorry, if culture was so broken at companies it stopped from shipping products, the blame is entirely on founders (and those who funded these founders) and top leadership.
Instead these people blame literally everyone else imaginable - PMs, MBAs (lol), woke employees etc.
40
30
599
Submitted the *final* draft of my book to my editor. There won’t be any more changes.
Well, that’s that, I guess. Dotting the i’s and crossing the t’s took much longer than expected, but got there in the end.
39
78
611
Really hard to feel any excitement for SpaceX launching a rocket right now.
Maybe it’s just me, but dismantling systemic discrimination and stop people being murdered seems more important to me than colonizing mars or wherever.
21
108
549
Taiji: Dynamic Traffic Engineering at Facebook
This is just utterly freaking NEXT LEVEL! 🤯
- Load balancing modeled as a constraint satisfaction problem to generate the optimal dynamic routing table
- Dynamic, connection aware routing based on Social Hash
- in prod >4 years
2
130
635
One of the most valuable skills for engineers of any level of seniority is learning how to be very productive when dealing with extremely “messy” systems or codebases.
People who have this skill are often very successful in pretty much any conceivable scenario.
17
96
606
Infra engineering is this strange world where oftentimes the most impressive/difficult/time consuming works manifests in the form of “this change was made, there weren’t any fireworks and it was totally uneventful and boring”
15
75
594
“Build something users love” is a piece of advice that’s often given.
I’ve never encountered a piece of software that “users love” that isn’t built by a team of engineers who:
- love it themselves
- constantly use it
- understand the problem domain well
- listen to users
6
125
582
The real reason why some startups flounder is because CEOs/founders have no managerial experience or strong leadership skills, and simply don’t know how to build a high-functioning team.
Instead, bizarrely, we see *employees* being perceived as the major problem at such places.
11
57
556
Remember when I said anon tpot isn’t technical or even very intellectual?
Case in point: they think one of the most famous and highest ranked engineers at Google who was so successful he retired in his 40s isn’t “grinding hard enough”
Chuds doesn’t even begin to cover it. 🙄
17
13
600
44,999
We’re going to see a lot of founders who simply have no managerial experience taking all the wrong lessons from the Twitter saga.
Elon might be the exception that proves the rule, but highly doubtful copying his managerial style will produce outcomes desired by these copycats.
36
63
559
Docker succeeded not because of “the tech” (containers) but because it was a great developer tool.
Kubernetes built atop Docker, and decided to be all things tech and not much by way of UX. It was and remains a hostile piece of software to learn, run, operate, maintain.
Docker was a transformational user experience, one that remains unmatched today. Ask anyone running k8s if they wished it worked like that first ‘docker run’. The answer is yes.
21
165
561
Yep. These are called grey failures and Microsoft even had an entire paper on this
microsoft.com/en-us/research…
A salad of resiliency features doesn’t always make your services more reliable. It makes failure harder to detect
5
102
559
You could’ve chosen your words more carefully because your tweet was in very poor taste.
High visibility accounts like yours also play a role in coloring a lot people’s views about an entire ethnic group, as can be seen in tweets like this.
It’s kind of bizarre to me how enormous the empathy gap is between Indian men and women. I’ve had a lot of suboptimal professional experiences with Indian immigrant men and the opposite with the women, who are often disparaged by the men.
10
4
566
43,068
JFC so much wrong with this demagoguery.
If you abolish the H-1B program, employers aren’t going to start hiring Americans in droves.
The jobs themselves will shift to Europe/Canada/India or some other place with a good tech ecosystem and plenty of easily hireable talent.
326
43
520
69,481
As more and more exec types begin to wonder why companies are overstaffed, we’re going to enter an era where engineers are going to be questioned why they run such complex systems - think every tech trend of the 2010’s - when much simpler ones would suffice.
Brace yourselves.
13
45
553
Started writing something about zero downtime deployments since most existing literature overly generalize (blue-green vs rolling etc) and the devil is often in the detail.
But it’s down a total rabbit hole once you start breaking things down and doesn’t fit on a single slide.
3
62
571
Well, that settles it, I guess, as to why we should always deploy software at 5:00pm on a Friday. 😎
23
56
539
‘jq’ is such a freaking amazing tool.
The best part of ‘jq’ is it’s so Unix-y. Using a pipe-esque interface to munge data is the best. To wit:
curl foo | jq ‘.[]data | select(.name == “foobar”) | select (.age > 20) | ...’
❤️❤️
Even makes bash scripts bearable to write/read.
14
59
552
Autopilot: Workload autoscaling at Google
- dynamic vertical + horizontal autoscaling
- unlike k8s, the proc doesn't get killed when the static limits for CPU, memory is exceeded
- a *functional* architecture with a control loop (reminds me of PID loops used in some AWS systems)
4
100
545
Seeing a bunch of folks in my mentions saying “TIL Google doesn’t use Kubernetes” and ... 😐
I’m surprised there are people who think Google *does* run on k8s.
Even GKE (Google’s hosted Kubernetes offering) runs on Borg, which afaik, is far more sophisticated and battle-tested.
14
80
520
Wow - Lyft did a four part blog series on scaling engineering productivity in a microservices world. lft.to/32Nir5v
Key takeaways:
- “fullstack in a box” doesn’t scale
- fast, local, native dev environments are indispensable (h/t Tilt)
- staging overrides with Envoy
9
92
530
Blogged: Testing in Production, the hard parts
bit.ly/2oa2UYG
Topics include:
- patterns to curtail blast radius
- design of control and data planes
- deployments and lessons from high profile Google and Cloudflare outages born due to a test in prod gone wrong.
7
148
527
Another little secret: a lot of what most “developer influencers” say is fairly aspirational.
Their own companies don’t necessarily do things as smoothly as they preach to others.
This is especially true at larger companies where the culture might vary vastly between orgs/teams
In the last few years, I've talked with hundreds, if not thousands, of software developers.
What I've learned: there's a HUGE gap between what developer-influencers are writing about, versus the daily reality of most developers. BUT PEOPLE DON'T TALK ABOUT THE OTHER STUFF. 1/
10
87
500
The Google outage postmortem is up:
"The root cause was an issue in our automated quota management system which reduced capacity for Google's central identity management system, causing it to return errors globally."
status.cloud.google.com/inci…
6
125
519
Dropbox open sourced its career ladder. dropbox.github.io/dbx-career…
I’ve seen a fair few career ladders but this is the most comprehensive one I’ve seen so far.
Covers both the IC and the management track, and also specialized roles like Security Engineer/SRE/Quality Engineer.
3
98
510
Ahem! 😉
I said last week that as the going gets tough, execs who think companies are overstaffed are going to start asking engineers why the software is so complex when simpler alternatives exist.
“App doing over 1000 poorly batched RPC calls to load a homepage” is exhibit A.
Btw, I’d like to apologize for Twitter being super slow in many countries. App is doing >1000 poorly batched RPCs just to render a home timeline!
7
30
482
Started reading the Amazon Builder's Library today. The very first article I read on deployments was absolutely amazing!
It goes into great depth about the challenges of deploying *stateful* services, in a manner I've never seen it written about before.
amzn.to/2suymTz
2
111
490
I’d bet over 70% of stuff here is unusable.
The next decade is going to involve unfucking all this.
21
70
473
Were I to start a virtual book club (on Google Groups or similar), would folks be interested in joining?
- it will be a technical book (TBD)
- ever week we will discuss one chapter.
- we might get authors to “guest post” and participate in Q&A (maybe over Zoom)
- open to all
102
39
486
wrote an embarrassingly long post on cluster schedulers, the need for them, why we chose Nomad over Kubernetes bit.ly/2tndXuM
20
190
477
Blogged: know how your org works (or how to become a 1̴0̴x̴ more effective engineer)
bit.ly/3EXi4T0
Sorry for being a killjoy this early into the new year, but here’s a list of unpopular, unfashionable yet (IMO) very important things people rarely talk about.
17
89
471
This is a fantastic companion to the first Aurora paper from Amazon.
The main takeaway is the distributed consensus kills performance, that local state can actually be a good thing, use an immutable log as the source of truth (basically what the Kafka folks have been preaching)
4
123
476
Imagine frothing at the mouth over 85K H-1B visas and 140K employment based green cards for the highly qualified, when a whopping 870K people enter the country without requiring any qualifications or even basic language skills, and immediately have full access to the job market.
While @X is all heated up with debate around H1B & high skilled immigration, including per-country caps, here is some perspective:
Employment based category (140,000 or 13% of total) is the ONLY category in legal immigration where the applicants actually have to go through labor market test, often times over and over again, only to be scapegoated into being called "cheap labor" and "replacing american workers"
123
54
425
37,326
Tech interview antipattern: interviewer springs a “so do you have any questions for us” to the candidate after a long coding session or when there are like 5 mins left.
IMO this should be a whole separate, no-holds barred conversation, if the technical rounds went well.
30
31
468
Incredible talk (as always) by @kavya719 on modeling of systems with formal laws of performance from queueing theory. Video - infoq.com/presentations/litt…
6
114
476
Looks like engineers at Twitter now spend nights and weekends fixing things that worked perfectly for years, instead of shipping any groundbreaking new features that wow-ed the world.
This is what many investors and pundits were calling for more CEOs to emulate a few months ago.
12
39
454
80,174
Wrote down some thoughts on distributed tracing and how we've been doing it wrong. bit.ly/2XFPZhq
19
134
467
I feel there are lots of engineers who can write code. There are far fewer engineers who have a deep understanding of product and have the knack of nailing the product.
And yet, most engineering interviews overindex for “coding” skills and barely cover core product skills.
if i could go back and do one thing different in my career as db/infra engineer, it would be this: build some fucking product. not only in the darkest hour of our need, but like during the day. with a PM and that culty scrum shit.
it is harrrrrrrd leveling up on this shit now.
17
90
458
Now might be a good time to push a cap on CEO pay.
If you’re management skills are so dreadful you overhire and layoff thousands within a 1-2 years, you shouldn’t be allowed to walk away with your usual comp package (we’re talking 50-100million here for big tech CEOs)
9
74
448
40,440
Rust is now one of the 4 official server side languages recommended at Facebook (C++, Rust, Python, Hack).
Interesting how it’s considered “at par” with C++ for performance sensitive stuff, and the recommended tool for CLI usage.
engineering.fb.com/2022/07/2…
6
88
475
Every company has tech debt, but Twitter 1.0 had extensively invested (pioneered even) many mechanisms to safely test and deploy changes (including many types of config changes) in prod.
Guess all those thousands of engineers were actually doing something worthwhile, after all.
4
25
438
17,957
Wrote a post on health checks, the perils of treating it as a binary state, the need for adaptive feedback loops and effective backpressure.
Also why HAProxy style agent-checks are a fantastic pattern for getting started with this form of control loops.
bit.ly/2MdY5Jd
4
175
455
Finally got around to reading this absolutely EPIC post by @clare_liguori on CI/CD at AWS which covers everything from:
- unit testing
- integration testing (in prod!)
- backward compatibility testing
- staged rollouts in "waves"
- automated rollbacks
go.aws/2AX3lfP
3
104
526
If people (founders) really to learn lessons on how to turn a company around, look at the proven success story of how Satya turned around Microsoft, and not the jury-still-out experiment of Twitter (which has a very long way to go before it can be called a successful turnaround).
16
35
448
everything rotten about Twitter can be root caused to the fact that Jack was an absentee CEO who delegated to folks, some of whom made questionable choices.
Yet, he never seems to get the blame for anything at all, while engineers who played no part in any of this get vilified
23. The decision was made at the highest levels of the company, but without the knowledge of CEO Jack Dorsey, with former head of legal, policy and trust Vijaya Gadde playing a key role.
4
22
433
The full postmortem of the Google outage this week is now up:
- incomplete migrations can be dangerous
- it continues to baffle me that Google systems aren’t designed to minimize the blast radius.
- Automated tools shouldn’t have the ability to make *global* config changes.
9
106
445
Testing in Production, the safe way
Topics include:
- why test in prod when you can test in staging
- how to test in prod while minimizing risk
- how to test configuration changes in prod
- why proxies are your best friend
- what to monitor
- and more
bit.ly/2GbB9Dc
5
171
430
The Spectrum of Compute
This is not an exhaustive comparison, but what with bare metal servers, VMs (including lightweight options like Firecrackeror or alternatives like gVisor), containers, serverless, ‘edge compute’, dedicated hardware like TPUs, options galore in the cloud.
9
106
440
The first time I’m seeing my book in print.
Wow - these are my thoughts and my ideas in print. I didn’t think it was a big deal when I was asked to write a book last year, but now seeing it for real is pretty surreal!
20
29
427
At a previous job, we decided against doing K8s. A disgruntled engineer spent the entirety of their tenure being grumpy about this decision.
It takes well over a million dollars just in engineer salary to get K8s up and running from scratch. And you still might not get there.
Even with kubeadm and a team of 6 you can’t get upstream K8s production ready in 6 months. We tried.
Hard Problems:
- High Availability
- Single Sign On
- Multitennancy
- Resource Isolation
- Permission Management
- Upgrades
- Backups
- Package Management
- CI/CD Integration
24
163
414
The reason bash will always be around despite being pretty awful at best is because sometimes it takes 500 lines of code in a strongly typed language to achieve what 5 lines of bash can get done.
22
19
408
Google has a styleguide for bash. google.github.io/styleguide/…
- If you find you need to use arrays for anything more than assignment of ${PIPESTATUS}, you should use Python.
- If you are writing a script that is more than 100 lines long, you should probably be writing it in Python.
15
140
415
The existence of 27 year old “directors/VPs of engineering” in SF is probably one of the many reasons why titles, while important in some contexts, shouldn’t be taken very seriously in many others.
I said what I said.
9
23
410
Dropbox blogged about rewriting the core sync engine from Python to Rust.
- Rust has been a force multiplier for our team
- betting on Rust was one of the best decisions we made
- ergonomics + correctness + encoding complex invariants in type system helped tame sync’s complexity
6
95
414
Twitter has set a very high bar for the rest of us designing and building distributed systems:
Build systems so robust that even if 70% of you get fired, it can chug along for as long as possible, prompting ignorant people to wonder why you even had a job in the first place. 😞
8
51
402
Macroservice:
- not a monolith
- Has no more than 20 devs/3 teams working on the service (5 pizza rule?)
- may or may not have/need monorepo. Dependency management becomes a lot easier (though still non-trivial) the fewer the services/repos
- better observability, debugging
16
50
404
I ❤️ data structures.
I especially love learning about real world use cases of data structures.
What’s your favorite data structure (bonus if it’s something you’ve coded in real life to solve a fun problem). 😀
58
59
397
The $10m engineering problem bit.ly/2P8W7u1
This is plain awesome work from @segment using a wide variety of techniques used to curb infra spend:
- rearchitecting systems
- "tree shaking" to distribute optimal JS bundles
- optimizing Go code
- Consul in lieu of ALB
10
114
406
The 2010s began with the idea that scaling infinitely wasn’t possible with an RDBMS.
In 2020, it’s possible to scale “infinitely” using strong relational data models (MySQL with Vitess, Postgres with CockroachDB, Aurora, and custom solutions like Cloud Spanner), which is cool.
12
41
404
