VP/Distinguished Engineer at AWS. Misplaced New Yorker in PNW. Die-hard sports fan on hiatus.
- Tweets 3,136
- Following 452
- Followers 4,880
- Likes 3,080
The 8 year old is learning Python, and after a dealing with a syntax bug she asks: “If the computer knows I’m missing a semicolon here, why won’t it add it itself?”
I don’t know. I really don’t know.
797
12,682
131,485
It might be older vs younger millennial thing too. Wife who is 3 years younger has no trouble executing all sorts of complex transactions on the phone. Meanwhile, I need a grown up computer for any serious business.
30
9
2,658
203,843
Re-watching Matrix with my wife, and the least believable part of the story is a software engineer getting reprimanded for not being at his desk by 9am.
54
184
2,171
Wow, I've muted this thread a while ago, and didn't realize this went bonkers. So wanted to add two things:
1/ The OP was a typo that meant to say colon (a bunch of people figured that out already)
2/ I am now intentionally going to teach my daughter to use semicolons in Python
9
17
1,055
Unpopular opinion: the biggest impediment to feature velocity is not bad code or outdated tools. It’s organizational coupling at the architecture level, where launching any feature requires multiple teams to contribute, especially if those teams are far apart organizationally.
22
74
591
You should go ahead and build micro services. Or a monolith. Or anything else in between that makes sense. I just wouldn’t take architecture advice from Twitter, where folks don’t know details of your situation.
13
61
565
One of the most impractical things in tech is to take a system designed for single-tenant use, and make it safe for multi-tenant use.
12
43
447
I want to try something new today. Many people find networking intimidating, and that's certainly how I felt when I joined EC2 7 years ago. But behind the acronym soup, there're usually fairly understandable concepts, and I want to talk about one such concept today: MTU. 1/
8
64
367
One of the new services I'm building is being written in Rust, and I have to admit it has been a pleasure. Excited that we are investing into the community.
aws.amazon.com/blogs/opensou…
2
47
224
I think one of the mistakes we make as an industry is thinking of CI/CD as a pipeline. IMO, a constraint solver is a much better fit - you specify a list of constraints and the system executes deployments without violating those constraints.
14
27
219
This is BIG! You can now assign IPv4 and IPv6 prefixes to your ENIs. The net result is that EC2 instances will now support vastly larger number of IP addresses, and managing those addresses will become easier. 1/n
aws.amazon.com/about-aws/wha…
4
53
203
I'm growing convinced that one of the most challenging software control systems to get right is the deployment mechanism for a distributed system. It comes with lots of nuanced failure modes, many of which don't lend themselves to a centralized solution. 1/
5
25
202
VPC Lattice is another launch that I’m super excited about. When we first pitched the idea for VPC Lattice, we had an intuition about the kinds of customer benefits we could deliver if we built layer-7, application awareness into the VPC data plane. 1/
aws.amazon.com/blogs/aws/int…
6
51
202
Both of these are true statements:
• Caches are responsible for more outage minutes than most other design patterns in modern computing.
• Caches are an integral part of modern computing, without which computing like we know it wouldn't exist.
7
20
183
Somebody asked me today if it was good seeing another cloud provider outage. And the answer is no, absolutely not! A bad day in the cloud is a bad day for all of us, and I’ve been in enough operational events to know how hard and stressful they are for all involved.
4
8
181
🚨 My new Amazon Builders Library article is out! In it, I talk about the risks of correlated failures in distributed systems, and the patterns we use to minimize them. I also talk about a fun adventure we had with newly built cabinets in our house.
aws.amazon.com/builders-libr…
5
37
168
A few thoughts on AI aided coding, testing, and software development:
blog.joemag.dev/2025/10/the-…
8
21
172
52,265
There is a software truism, that I think applies broadly. When observing a complex system from the outside, it’s tempting to point out and laugh at some seemingly absurd decisions. However, most often it’s the uninitiated that are missing something obvious about the system.
5
25
139
TIL that a library I wrote in anger 12 years ago is still in use by a dozen Amazon teams, and for that I’m really sorry.
4
6
143
Recently, I've been discussing throttling with one of our teams. It's a super deep topic, probably deeper than it gets credit in modern systems, so I decided to write some thoughts on the matter: blog.joemag.dev/2025/06/the-…
3
20
152
16,581
I’m watching a bunch of first graders use video conferencing without any technical problems, each one muting themselves when not speaking. So I’m pretty sure it’s just our generation.
2
11
132
In recent design reviews, I'm observing more and more teams plan to write their new data planes in Rust. It looks like the tide has turned, and Rust is becoming the default choice for new data plane software.
2
26
134
My last work related activity of 2020 was writing a supportive promotion recommendation. I think that's a pretty good way to end the year.
1
122
A trip down the memory lane. When I joined Amazon 14 years ago, my very first project was to add a simple feature to one of our backend services. The service was backed by an Oracle database, and the feature required addition of a new table, and a few extra columns. 1/n
Migration Complete – Amazon’s Consumer Business Just Turned off its Final #Oracle Database - aws.amazon.com/blogs/aws/mig… #AWS
1
39
123
The irony: I’m running late for a meeting in which I talk to a bunch of Amazon Principal Engineers about time management.
5
4
112
I want to buy beer for the Apple employee that came up with "autofill security code from incoming message" idea.
Let's say nice things about technology today. I'll start. If it wasn't for @lkanies and @puppetize, there is no way we would have been able to adapt as an industry to the rise of the cloud. Quote tweet me with your own.
1
6
106
Towards Modern Development of Cloud Applications is an interesting paper, and I recommend reading it. The challenges the paper tackles are real, and our tools and frameworks haven't made nearly enough progress in solving them.
sigops.org/s/conferences/hot…
1/
2
17
114
27,726
As the first batch of millennials to turn 40 later this year, what are we doing for mid-life crisis? I’m not particularly interested in a fancy car, or a boat, and travel is unlikely. I feel underwhelmed with my choices.
35
5
103
It's been a lot of fun writing about our experiences building services at AWS for the latest Amazon Builder's Library article!
aws.amazon.com/builders-libr…
4
36
110
A few thoughts on using leader election algorithms:
blog.joemag.dev/2025/03/the-…
11
91
6,142
This is fascinating, but also an example where the goals of distributed systems are at odds with single host systems. The latter needs to make every effort to survive. The former prefers the host just crash, anything else is a liability and an invitation for weird gray failures.
2
18
83
A made a newbie Amazon mistake this week - I thought I can change an important decision without writing a 6 pager. I was wrong, and now must repent by catching up on writing for the time I wasted on talking!
2
1
94
If you use PrivateLink, then this is big: you can now control the DNS name that will be configured for your PrivateLink endpoints.
aws.amazon.com/about-aws/wha…
3
26
87
ENA Express is easily my favorite launch of this re:Invent, and we’ve been eagerly waiting having customers try it out. It is also worth a short thread. 1/
aws.amazon.com/blogs/aws/new…
6
23
88
Reading all the “Google squandered early cloud advantage” takes - it’s a pretty tough sell for me. There is such a big chasm between internal tools like MapReduce, GFS, and Borg and what is needed for a public service.
2
87
I just found out that a few weeks ago I've hit 15 years at Amazon, which earns me a fancy purple badge. There are times I still can't believe the series of (at the time) questionable choices that brought me to Amazon and Seattle, and eventually to AWS. /1
9
90
I was just catching up with AWS Infinidash team, and it’s amazing what a two-pizza team can accomplish without any principal engineers getting in the way!
6
1
87
I can’t help it, but every time I see the word “containerd”, I pronounce it as contain-nerd to myself.
5
12
82
Happy that we made the right decision here.
Amazon pledged to continue paying all hourly employees — some 10,000 people — who serve its Seattle and Bellevue buildings, which are largely empty since the company directed all employees to work from home if they can amid the coronavirus outbreak. bit.ly/2In5cdU
10
79
The boring answer is by being more complete, faster, and more usable than the alternatives. An interesting follow up question is what cultural aspects allowed them to be that.
3
1
80
We launched Network Load Balancer to support load balancing low latency, high throughput TCP and TLS connections. Now, you can use it to load balance your UDP traffic as well:
aws.amazon.com/blogs/aws/new…
3
30
81
A nice write up about role of Principal Engineers at AWS:
deloitte.wsj.com/cio/2020/09…
1
21
81
I wonder if every parent is familiar with the feeling of hearing running footsteps, followed by a loud thud, and then a nervous pause as you wait to hear if the next sound is a cry or a laugh.
5
80
One of the most desirable properties in queue or stream based systems is increased efficiency as the queue depth grows. Without that property, slight mode shifts can push the system into the state of perpetually growing backlog. Who here hasn't a seen a graph like this: 1/
1
5
80
Slack has so thoroughly replaced email for me that I’m now in 100 different Slack channels that I have no chance of keeping up with.
9
4
77
Recently I hit my 15th anniversary of moving to Seattle to join Amazon. I still remember the series of events that brought me here. I was a wet behind the ears junior SDE, working for a financial startup in NY area. 1/n
1
2
74
This might be harsh, but whenever I see folks schedule a weekly team or working group meeting to solve some problem, I just assume the problem will remain unsolved.
8
3
70
Today's twitter reminds me why I dislike the term microservice. It focuses attention on the least important aspect: size, rather than more meaningful factors (on both sides) like simplicity, agility, efficiency, scaling characteristics, and blast radius.
3
12
67
Amazon Chime feature I would pay for: when I search for a name, sort results based on how recently and how frequently I communicated with that person.
3
62
A question came up in a conversation with one of my teams: are feature flags harmful? My thoughts on this topic are nuanced, so I decided to write them down:
blog.joemag.dev/2022/04/feat…
7
14
66
I'm stoked that VPC Lattice went GA today! I'll do a longer thread on what lead to the creation of Lattice and how we are thinking about it long term later on, but for now I'm really excited about getting Lattice into customers hands!
aws.amazon.com/blogs/aws/sim…
1
22
68
14,396
Honest question: How can a tech startup have a multi-billion dollar valuation, if after reading their home page for 10 minutes I still can’t understand what their product is or what problem it is solving?
6
2
64
Reading about network overlays used by some container plugins irked me enough to talk about the concept of ‘flow’ used by networks. As developers, we are abstracted away from the underlying network and that’s great. However, the concept of flow is worth understanding. 1/n
3
24
61
I bet their day will be ruined when they discover: github.com/openjdk/jdk/blob/…
2
55
1,988
The irony of Trump presidency is that whoever becomes the president after him will have to spend the first few years making America great again.
1
2
54
The topic of SSD block erasure came up in a meeting today, and I think it's such an interesting example of complexity hiding behind simple abstractions, it's worth a short thread. 1/
1
11
56
When the VPC launched, it only supported IPv4 addresses. We added IPv6 support in 2016, but our IPv4 roots were still showing and you needed to assign a primary IPv4 address to each ENI.
With this release, that finally changes!
aws.amazon.com/about-aws/wha…
4
5
50
This is big! It has been a very common customer request, and something that S3 and EC2 Networking reams have been working on for a while. And now it’s out!
aws.amazon.com/blogs/aws/aws…
2
12
52
Imagine thinking that saying no to a meeting after 10pm or before 5am is a sign of vulnerability.
Sheryl refused to participate in late night meetings. She had the confidence to admit she went to bed at 10pm and told Mark she'd be happy to meet when she woke up at 5am if he still hadn't gone to bed yet. Her vulnerability was inspiring and signaled strength not weakness.
2
3
53
I will really miss EC2-Classic - it was the first system I worked on at AWS. But worth calling out that EC2-Classic APIs will still continue working, like they always had. We create a default VPC behind the scenes and launch your "faux" classic instances into it!
7
50
My one career goal remains to use a Bloom Filter in a real production setting. For the second time in the last five years I thought I had a solid reason to use one, only to finally settle on a better alternative.
8
2
52
Somedays I feel like the most valuable experience I gain by working at AWS is not how to solve hard technical problems, but rather how to run a successful business.
2
53
Nothing gets pitchforks and torches out faster than somebody posting @ here in a large slack room.
6
1
44
I’m punishing the 6 year old because of something she has done, and the 4 year old goes: “Yes, use your hate. Let it flow through you!”
3
47
VPC Traffic Mirroring is a feature that is near and dear to my heart, as I spent a fair amount of time working with teams that were building it. The hardest part was making sure that mirrored packets arrive without their left and right sides reversed!
aws.amazon.com/blogs/aws/new…
8
49
I think it’s a mistake to think one can just package their internal tools for external consumption. It takes a different product mindset, and often times a different design philosophy. Sure, Google could’ve started GCP earlier, but calling it a squandered advantage is unfair.
4
1
49
I occasionally have 5-to-6 meetings, which is a problem for me because I’m typically the one responsible for dinner.
Today, I dialed into the meeting while I was making dinner, and I have to admit I found the meeting a lot more enjoyable that way!
3
1
53
You may find it interesting to see how EC2 collects metrics and logs from all the EC2 servers. My recent builder’s library article was inspired by lessons learned from such systems.
aws.amazon.com/builders-libr…
1
1
48
7yo: Dad, since mom builds BigQuery, I think she is better at answering questions.
7yo: I think from now on I should direct all my questions to her.
Joe: <fighting to hold back a smile> Yes, I think that makes sense.
46
We talk a lot about economic benefits of multi-tenant systems, but I think the bigger benefit is ability to scale. There are few practical ways to build a single-tenant (or low cardinality multi-tenant) system that can handle sudden load changes as well as a multi-tenant one.
7
9
47
I remember that day! I was an SDE II in the team that owned some of the fulfillment systems for Amazon.com, and one of our immediate projects was to replace NFS that was a constant source of operational pain with S3. Our lives became so much better.
2
3
45
I’ve finally given up on digital whiteboards, and just bought a physical tabletop whiteboard and an overhead camera. Can’t believe I’m saying this, but I’m actually looking forward to my next meeting!
6
48
We started watching Westworld last night, and I’m super frustrated they didn’t immediately rollback after seeing issues pop up at the same time they deployed a new release to the first 10% of the robots.
8
3
43
My 4yo decided that her favorite band is Queen, and I think that means I’m succeeding as a parent.
7
45
Today convinced me that one of the most under-appreciated skills in life is to have an opinion on something, but keep it to oneself.
7
2
46
Another proof that containers are not a security boundary! I left a piece of cake in a Tupperware container, and somebody ate it!
4
5
45
3,994
Thanks to folks @Meta for inviting me to speak at Reliability@Scale conference. I had a lot of fun preparing and giving the talk, and can’t wait for the video to go live!
atscaleconference.com/reliab…
1
3
43
9,309
I’m really excited about this ALB feature! Most distributed systems are built to be redundant in the face of failures, and indeed they tend to do great when faced with fail-stop failures. 1/
Application Load Balancer increases application availability with Automatic Target Weights
Application Load Balancer (ALB) now supports Automatic Target Weights (ATW), wh... aws.amazon.com/about-aws/wha…
1
5
44
12,762
When a team at AWS delivers a feature that improves some operational aspect of their service, they send a broad email with details of the change, along with #pmet graphs demonstrating the impact. This is what those graphs typically look like.
Oh, and drop in CPU utilization ftw!
We are pleased to release the Amazon Corretto Crypto Provider (ACCP), available on Maven and GitHub, so that everyone can benefit from our work to significantly improve the performance of cryptography. #corretto amzn.to/2Y6KubQ
ALT graph: AWS service that terminates TLS at the host level, showing dramatically reduced CPU utilization after deploying ACCP.
8
42
About a decade (or maybe more?) ago Amazon started to consistently include principal engineers in organizational review meetings. These days many orgs include seniors as well. Both benefits Charity mentioned did happen.
If you do calibrations, consider making it your policy to include some senior+ individual contributors.
1. You'll gain valuable perspective that managers necessarily lack
2. It helps demystify a very scary, opaque process
3. It levels the power imbalance between managers & ICs
2
4
42
Some semi-serious math/software-engineering spelunking on a Thursday afternoon. When propagating state across systems (such as between replicated data stores or between control and data planes) one decision is how to keep that state in sync. 1/
1
4
44
At Amazon, we don’t change tenets lightly! A lot of thought and deliberation goes into these tenets, and this week principal community got a new one.
This just in: we updated Amazon's Principal Engineering Community tenets. 1/5
amazon.jobs/en/landing_pages…
1
4
40
Went clothes shopping with my 3 year old. She insisted I buy a pair of pink shorts. I pushed back. So did she.
I now own a pair of bright pink shorts.
5
1
41
My dad’s retired from active play now, but before he did he was (still is) a Grandmaster and a pretty successful chess player. So it was really cool to hear @Kasparov63 talk about his experience training with dad in his podcast: open.spotify.com/episode/1ME…
3
42
This is indeed big! The ideal development environment for me is one where I do all my coding on a laptop, but the actual build and indexing happen on a powerful EC2 instance (I’m looking at you, Rust!).
This is big! We’re introducing remote development in the 2021.3 JetBrains release wave:
- Run your IDE on a remote server
- Work from any laptop, anywhere
- Manage dev environments with @JetBrains_Space
Bonus: check out @JetBrains_Fleet
Find out more jetbrains.com/remote-develop…
3
1
39
The only way I know to fasten an ambiguous project is to remove as many principal engineers as possible. You may end up building the wrong thing, but you will build it really fast.
3
42
This launch really has everything! An unexpected new product. A big customer need. A tough technical challenge. Dave wearing delivery vest and drawing design diagrams on a whiteboard!
Run macOS on AWS for the first time with new Amazon EC2 Mac instances. Start developing, building, testing, and signing Apple apps on AWS. Learn more: go.aws/37CLRlp
1
43
Reliable DNS is table stakes for building reliable applications. Route53 and Nitro teams implement a lot of neat ideas to provide customers’ EC2 instances with reliable DNS, and I love seeing the teams talk about some of those details:
docs.aws.amazon.com/Route53/…
2
8
41
9,919
When the final FB postmortem is out, I kind of wish there will be a PKI somewhere in the causal chain, along with BGP and DNS. That way this would be the most cliche failure ever!
4
38
I spent a year and a half at Google, and have many positive memories and friends from that time. But one thing that always rubbed me the wrong way was how often the “you are the best engineers in the world solving the world’s most complex problems” rubric was repeated.
Where does engineering rep come from? Here's a Google engineer slagging on Amazon engineering, which they say mediocre.
I don't think this is an unusual opinion, I've heard this from people both inside and outside of Google. Google has the best engineering, Amazon is mediocre.
ALT "Amazon clearly does not compete by hiring the best engineers. Rather, they compete by throwing money at the problem and undercutting everyone else, getting by with mediocre engineering." -- googlemike
4
1
39