Founder of Dagster Labs. Creator of Dagster. GraphQL co-creator.
- Tweets 5,080
- Following 2,192
- Followers 12,990
- Likes 8,414
Jeff Dean and co at GOOG just released a paper showing how machine-learned indexes can replace B-Trees, Hash Indexes, and Bloom Filters. Execute 3x faster than B-Trees, 10-100x less space. Executes on GPU, which are getting faster unlike CPU. Amazing. arxiv-vanity.com/papers/1712…
38
2,594
5,299
The terms "data cleaning" and "data wrangling" should be retired. They undervalue and trivialize the work. It is the design and construction of an organization's data products that outlive any particular model or dashboard. It's the bulk of the work and it's hard.
40
215
1,600
In Cuomo's America, this nurse in Israel that grabbed a pizza boy to vaccinate when a vaccine was about to expire would be fined a million dollars.
nitter.app/erlichya/status/134473…
14
167
1,310
Here's a prediction: This is the moment Buffet has been waiting for his entire career. Berkshire has 125B in cash and a universe of distressed assets to buy (when the time is right) with real, underlying value. Buffet is going to end his career in a blaze of glory.
31
105
1,031
Microservices are such a fundamentally and catastrophically bad idea that there are going to an entire cohort of multi-billion companies built that do nothing but contain the damage that they have wrought.
54
120
806
All the dunking on the Zillow team––ICs and leadership alike––is not a good look. Business is hard, they took a shot, and they missed. Can happen to anyone.
54
35
788
How is this not the biggest story in the world right now?
Wow. Sea of protestors in #HongKong ‘s Victoria Park today, estimate of 1.7 million say organizers despite rain.
11th week, 5 demands & rejecting police brutality:
19
126
732
Disney acquiring LucasFilm and Marvel for 4 billion a pop turned out to be a steal in the streaming era. By comparison, this year Netflix got the rights to just Seinfeld for 500 million for 5 years. Can't really fathom what would Marvel and LF would go for today.
11
65
638
I can’t quote tweet @ryxcommar but this fake account fraud is one of my favorites yet. Outed by the excel row limit by bankers lmao
11
80
576
200,678
From the talk about the rewrite of fb using Relay and GraphQL. This feature is so amazing and intuitive. Deliver js only if the graphql query returns data that requires that js.
7
79
441
Degradation of Uber customer experience under CA’s new regulatory regime is pretty shocking. 5x wait time. 2.5x price increase. Thanks AB5!
29
45
418
One of the most high leverage things a non-technical person can learn at a tech company is SQL. Extraordinarily empowering.
17
27
385
My heartfelt congratulations to the PR geniuses at Lyft and Stripe for getting this story framed this way.
5
8
377
Every time I’m satisfied with some engineering thing I’ve done I remember what engineers who program video games pull off and most engineering is child’s play by comparison.
2
15
365
56,432
1/ Today we at Elementl are excited to launch an early release of Dagster, an open-source Python library for building data applications. Here's a post about what Dagster is, why I moved to data infra, why data is hard, and why we need a new system. medium.com/p/dbd28442b2b7
10
97
358
Unrelated to to WeWork S-1, but I am still amazed that Neumann (WeWork CEO) banned meat in the workplace for environmental reason while gallivanting around the world on a company-owned G650. The hypocrisy is so brazen I almost admire it in a perverse way.
7
19
341
Should unrealized cap gains and a QSBS repeal pass while preserving the step-up-in-cost-basis-on-death loophole, it should be understood as total, unrestricted warfare by those who are already wealthy against those trying to become wealthy.
15
27
326
"People think our new software is bad/not worth it so let's make them pay to not install it" is a galaxy brain business model.
13
26
310
1/ Wrote a document explaining to the @dagster team why I always write technical documents and discussion in GitHub Discussions rather than Notion, Google docs, or similar tools
9
29
326
107,399
California (and San Francisco in particular) is a grand, fascinating sociological experiment. Are the network effects of talent so strong that they can withstand a government that actively hates that network and wants it eliminated.
🤣 you mean Elon Musk is such a snowflake he moved from California because I said a censored bad word about him? Wow, isn’t that weak.
17
23
302
My father is an epidemiologist and believes that East-Asian-style universal mask wearing is critical to fighting coronavirus, but that healthcare workers must get priority. So he and my mother designed a DIY mask and produced this video. Proud son today.
piped.video/watch?v=q8GwjVih…
7
92
284
React team is now at Jedi-level adoption strategy: build some really good (hooks), call it alpha quality, and tell people not to use it. Reverse psychology at ecosystem scale.
5
50
297
Funny enough while watching the Jordan documentary I turned to my wife and said: “Jordan plays with such simultaneous elegance and aggressiveness. It is both beautiful and terrifying. The thing that compares that I know of is watching Sacca sign a term sheet.”
8
8
269
1/ As part of the FB interview series (see nitter.app/schrockn/status/113050…) I'm going to posting some summaries and commentaries/hot takes on each interview. First up: @floydophone.
softwareengineeringdaily.com…
1/ Last week @software_daily did a series of interviews on Facebook engineering that had interviews with myself, @dan_abramov , @floydophone , @jgoldfein, and @mvernal. This series will continue in spurts over the next few months.
3
64
252
Finally read High Output Management by Andy Grove this weekend. It's 235 pages, 35 years old, and from what I can tell is the primary source for nearly every other tech-oriented management book ever since. Pretty amazing.
5
17
257
Noise about AAPL reminds me of this line from @BillGates about being told FB was a platform: “That’s a crock of shit. This isn’t a platform. A platform is when the economic value of everybody that uses it, exceeds the value of the company that creates it. Then it’s a platform.”
4
30
229
1/ Last week @software_daily did a series of interviews on Facebook engineering that had interviews with myself, @dan_abramov , @floydophone , @jgoldfein, and @mvernal. This series will continue in spurts over the next few months.
3
37
238
A year ago I sent the attached message to the @dagster team, making @floydophone CEO. One year later: best decision I’ve made in my career. Company is thriving (Series B, >4x rev growth 🤯), team is on fire, product is better, org is healthier.
14
3
223
58,607
How did I just learn about this? This is genius. Drive incremental adoption of GraphQL by generating REST APIs from GraphQL, rather than the other way around. Counterintuitive but brilliant.
medium.com/the-guild/sofa-th…
@UriGoldshtein amazing work
5
36
205
1/ I posted about this paper rise.cs.berkeley.edu/blog/a-… this weekend and about how awesome it is. It's the Berkeley RISELab report on serverless computing. Wanted to elaborate the parts I found really interesting and insightful.
4
67
203
This is from wwdc 1997. I watch this entire Q&A at least once per year to remind myself what clarity of vision and communication should look like. It’s extraordinary. piped.video/GnO7D5UaDig
2
17
190
Thoughts and prayers to data/tech journalists:
Dbt–-the leading transform tool in the MDS–-acquires Transform––which is not a transform tool, but a metrics tool––to transform their transformation tool into a metrics and transformation layer to enable digital transformation.
2
12
188
27,055
Apple no longer sells any laptops without touchbars. Their insistence on active hostility towards developers (and vi people in particular) in this regard is both confusing and maddening. Might be time to corner market on MacBook Pro 2015 15”, which was peak MacBook.
24
15
187
1/ Dagster has been public over a year. Last week we pushed out a new version that marks a new level of maturity for the project. We now call Dagster a data orchestrator. Here is a post about what we’ve built, learned, and principles we've developed:
medium.com/dagster-io/dagste…
4
39
186
I thought the FB acquisition of Instagram was overpriced at the time. Turned out to be the best acquisition of the decade.
2
6
172
Whelp there is a GraphQL documentary now :-)
It's not just about GraphQL, but also about FB's transition to be mobile-first in 2012. It captures the urgency and stress of that *very* well.
Thanks to the team at @honeypotio for such amazing work.
piped.video/watch?v=783ccP__…
4
34
174
1/ We at @dagster have a Python monorepo. About 500K lines of code and 70 python packages. Not big compared to bigco, but big enough where some python tools fall over. One of these is pylint. We've switched to Ruff (github.com/charliermarsh/ruf…) by @charliermarsh. It's a gamechanger.
9
18
170
34,576
1/ I’m seeing a lot of hemming, hawing, and gnashing of teeth over AirBnB’s announcement that they are sunsetting React Native. They are unrolling a pretty massive investment. Obviously this is a setback for React Native ecosystem.
After two years, 220 screens, and 120,000 lines of javascript, we're moving away from React Native at Airbnb. I tried to summarize our experience in a single blog post but that wouldn't do it justice so it turned into five!
medium.com/airbnb-engineerin…
3
65
175
Tesla reaching eclipsing a trillion dollar market cap represents a stunning, epoch-defining defeat for the cynical, East coast financial class that set out to destroy the company. Victory has never tasted so sweet. Glory to the builders and the optimists.
3
9
172
The impact of Microsoft’s new leadership and strategy on the industry is simply put, stunning.
3
21
169
Just got caught up on the wallstreetbets GameStop drama and can confirm that it is the funniest shit to happen in a long time.
3
22
147
Truly shocked Twitter is still up. In 2012 while working on a risky, high stakes project at FB I asked @jayparikh what we do to bring down the site if we totally screwed it up. He said he would just unplug the global load balancer. Maybe Twitter should do that right now.
6
20
155
All the haterade for @LambdaSchool today is some the stupidest shit I’ve seen on Twitter in while
3
5
154
I never get sick of the magical DevEx that is TypeScript combined with modern GraphQL tools like Apollo/Relay. It really is amazing. Imagine traveling back in time to 2010 and showing a frontend dev. They wouldn't even comprehend it.
6
14
147
Dear LinkedIn: Never in my life have I wanted to say the words "Congrats on your work anniversary!" to anyone
7
4
144
1/ Well this got scooped by the press (didn't see that coming Business Insider!) but I am moving from CEO to CTO at Elementl (the company behind @dagster) and Pete Hunt (@floydophone) is now CEO.
7
5
148
This is how you dress for a conference. Ladies and gentlemen, I present to you @leeb @graphqleu
2
9
146
Frontend eng is more difficult and complicated than backend most of the time. Most backend eng's heads would explode if they had to deal with a modern, complicated frontend.
18
4
142
20,522
1/ I'm excited to announce Elementl's 14MM Series A, led by @mavolpi at @IndexVentures, alongside @sequoia, @slow, @AmplifyPartners, Coatue, @OSSCapital, @neo, and other leading investors. dagster.io/blog/decade-of-da…
12
16
142
1/ elementl.com/2020-2021-engin…
This year students are facing the prospect of returning to a degraded college experience while still paying full tuition. We at Elementl suggest an alternative: a well-paid, year-long fellowship where you work on an open source project, Dagster.
1
37
136
1/ @graphqleu 2018 Keynote is live! Title: "Revenge of the Monolith: The Past, Present, and Future of GraphQL-Native" It had three sections.
piped.video/watch?v=zMa8rfXI…
1
37
130
If you ever found a company, never, ever use your real phone number for anything ever. I get 5-15 spam/sdr calls a day now.
31
1
128
22,887
Approximately 100% of my social circle have contracted mid-grade alcoholism during the pandemic.
11
7
123
I read: "A deep dive into how Khan Academy took a 1 million-line Python monolith and split it into ~40 Go services in a more than 3 year-long project."
My triggered lizard brain translates: "Welcome to your nightmare made flesh."
newsletter.pragmaticengineer…
7
8
126
42,118
I actually think engineers underestimate the ability of people to write "code". For example python. Lots of people could do some work in python, but they would have *zero* ability to deal with virtual environments, etc. I think it less about code and more about the infra.
9
6
118
Just made a post about code reviews. This came up in an ex-Facebook discussion group so I got the original text and adapted it for the outside world. Code review is incredibly important and often a source of unnecessary frustration and churn, My thoughts: medium.com/@schrockn/on-code…
3
47
119
Because it natively executes in a web browser. Nearly all other reasons are either incidental or downstream from that underlying fact.
2
116
Remember how people acted this weekend. Remember who fomented panic; who spread misinformation and rumor in private threads; who abandoned business relationships in seconds that should be measured in years; who used their public platform to help and those who used it to destroy.
4
10
120
16,039
Might be time for an update on this article about @a16z after what happened today
theinformation.com/articles/…
4
3
123
"When you're operating on the maker's schedule, meetings are a disaster" -- @paulg 2009
paulgraham.com/makersschedul…
They actually think they invented the idea that meetings are a tax on productivity.
Eventually it happened. Paul Graham is quoting from REWORK, REMOTE, and It Doesn’t Have to Be Crazy at Work. It took a virus for the valley to validate our ideas.
4
3
116
Really excited to speak at the DataEngConf SF. It will be the first time discussing Dagster, the project I’ve been working on for nearly a year. Data processing — ETL, ML, data pipelining — is a massive, critical, growing discipline and better developer tooling is a huge need.
Data Council SF '19 Speaker Announcement: @schrockn from Elementl will talk about "#Dagster : A new programming model for data processing". Book your tickets now hubs.ly/H0gMNZF0
6
20
120
After seeing this picture I was extremely concerned about your health and working environment. Thanks for the clarification and reassurance.
4
112
18,749
From the chatter in FB eng groups, anytime an ex-FB engineer enters a company, they are appalled at the number of repos and immediately start to consolidate and merge them if they can.
3
3
120
TIL I learned that graphql.org has 2.2M yearly unique visitors 🤯🤯🤯 Pretty surreal.
3
13
119
The biggest gap in the ecosystem is a “rails for graphql” esque system. Vertically integrated, opinionated toolkit that manages db schema, data loader interactions, software abstractions etc. Huge opportunity. Way way too hard to write high quality graphql backends today.
7
22
112
2.1/ Technical Strategy: “Evolutionary Means For Revolutionary Ends: The Art of Changing Large Systems” Bunch of my career was spent building/tech-leading the abstraction stack that led to GraphQL. Operated mostly on instinct but formed a philosophy around what worked.
2
42
112
GraphQL exists not just because it was designed and implemented by engineers who could write backends, but also who had the ability to listen to empathize with the product developers they served, rather than insult and belittle them.
2
13
107
A new, successful Brazilian bank called Nubank has built a banking tech stack from scratch using Clojure, Dataomic, Kafka, and functional programming nerdery in general. Fascinating presentation.
infoq.com/presentations/nuba…
41
105
🤘🏻🤘🏻🤘🏻
Elementl raises $33M Series B for its data orchestration platform based on Dagster tcrn.ch/3Wz0ZJB by @fredericl
12
4
109
15,990
1/ It’s time to rebundle the data platform. Today @dagsterio is unveiling a new concept: the software-defined asset. It makes Python a true first-class citizen in the MDS and unifies disparate tools into a single fabric that models all your data assets: dagster.io/blog/rebundling-t…
1
24
107
1/ It’s an exciting day: Dagster Cloud, the new standard for productivity, scalability, and trust in enterprise data orchestration, is here. We’re opening up for Early Access today.
dagster.io/blog/introducing-…
1
14
102
I prefer “tech leadering” by being the bad cop about being the bad cop on code reviews too! link.medium.com/siaAdCa0xV
1
28
99
It's important to occasionally go back and read this thread to remind yourself of what @voxdotcom, @recode and their ilk really are.
Incredible.
Last week I called out a journalist who I thought was writing a piece that would be a disservice to public health.
Unfortunately, my concerns proved well-founded. Recode’s virus piece ignores WHO & CDC, gets the science wrong…and focuses on handshakes!
Thread: 🧵
1
6
94
The Rise of Medium Code:
Why the reports of software's demise are greatly exaggerated
dagster.io/blog/the-rise-of-…
5
19
96
22,381
“When men stop believing in God they don't believe in nothing; they believe in anything.” — GK Chesterton
4
7
95
Google trends isn’t at all scientific, but still, is there any open source piece of software that has had as much impact and adoption as @reactjs from zero to five years? It’s a stunning achievement. Props to the original, @jordwalke, and the entire team.
4
9
93
Turned on Apple two factor auth on a new laptop. I think I’ve had to enter my password 15-20 times across 4 devices. You can really feel their organizational compartmentalization in their software sometimes. No escaping Conway’s law.
5
6
97
Even xkcd knows about the reality of data pipelines.
Alt image text:
"Is the pipeline literally running from your laptop?"
"Don't be silly, my laptop disconnects far too often to host a service we rely on. It's running on my phone."
1
17
81
2/ links:
- @floydophone: softwareengineeringdaily.com…
- @schrockn: softwareengineeringdaily.com…
- @jgoldfein: softwareengineeringdaily.com…
- @dan_abramov: softwareengineeringdaily.com…
- @mvernal: softwareengineeringdaily.com…
2
10
86
Your view of @elonmusk says a lot about how much you value norms versus how much you value outcomes.
7
1
87
I also see a misuse of the rule to justify engineering mistakes that make it *impossible* to do optimizations/improvements in the future. Programs must be structured properly and have understand perf complexity. Premature *micro*-optimization is often a waste of time.
89
18,021
Five year anniversary of the GraphQL Open Source announcement today! reactjs.org/blog/2015/05/01/…
12
86