CEO of @confluentinc (confluent.io). One of the original creators of @apachekafka.
Palo Alto, California
- Tweets 7,403
- Following 974
- Followers 33,155
- Likes 3,716
Look, I’m biased, but about 97% of blockchain usecases I hear about don’t need anything like blockchain, they just need a commit log, and they’d be about 100x simpler and faster if they just used @apachekafka. Or, for that matter, @PostgreSQL.
45
213
1,111
Nikki Haley telling lazy Silicon Valley companies that if we just avoided immigrants like they do in South Carolina, we could have a tech industry like theirs. I suspect she’s right! :-)
When I was governor in SC our unemployment went from 11% to 4%. How? Not by hiring foreign workers. We recruited foreign companies to invest in SC but not their workers. We retrained South Carolinians in our tech schools for these new jobs.
The companies started apprentice programs for high school and college students. You know who paid for them? The companies, not govt. Today South Carolinians are building planes, automobiles, tires, etc. And building them well.
What is lazy is for the tech industry to automatically go to foreign workers for their needs. If the tech industry needs workers, invest in our education system. Invest in our American workforce. We must invest in Americans first before looking elsewhere. Don’t ever underestimate the talent of Americans or the American spirit.
28
63
745
54,113
The irony of distributed systems is that data loss is really easy but deleting data is surprisingly hard.
15
803
588
We think this is a big step in making stream processing easier. I'm really excited about it. confluent.io/blog/intro-to-k…
9
175
475
Apache Kafka 1.0 is out! confluent.io/blog/apache-kaf…
14
310
445
Having worked on both social networks and distributed systems, I’m pretty confident that decentralizing social media will not solve any of the problems people think it will and will make impossible the actual hard stuff you need to get right (relevance, abuse, search, etc).
18
47
410
Folks are working on what is probably the most requested feature in @apachekafka: removing ZooKeeper as a dependency. This makes the minimum footprint smaller, and makes it a lot easier to manage, secure, and scale. Lots of work to do to make it real.
12
111
397
I mean *I* think our streaming strategy is pretty good but maybe I should talk to this guy to be sure? 😀
16
7
387
89,268
Okay, since I can't resist a good argument I wrote up a few thoughts for any Kafka Exactly Once skeptics out there: medium.com/@jaykreps/exactly…
16
237
393
I'm excited to announce that Confluent has raised a $250m Series E funding round. We think this is just the beginning, and that event streams are going to be one of the most important data platform in a modern company. confluent.io/blog/series-e-r…
19
56
353
Software is mostly human capital (in people's heads): losing the team is usually worse than losing the code.
24
541
338
I could not be more excited to be working with the Anthropic team—an incredible group of smart, humble, and ethical people solving one of the world’s biggest, most fascinating problems.
Welcoming @jaykreps to Anthropic's Board of Directors: anthropic.com/news/jay-kreps…
19
18
317
69,672
This is ridiculously good: Apache Kafka as explained with Otters: gentlydownthe.stream/
4
92
279
I wrote a popular blog post on "The Log" back at LinkedIn. Here's a follow-up on what's happening with event streams, stream processing, databases, Kafka, @confluentinc and what it means for the software architecture of companies. confluent.io/blog/every-comp…
6
96
272
I wrote a new blog post: "It's Okay To Store Data In Kafka" confluent.io/blog/okay-store…
5
160
270
1. Thoughtworks notes that "Kafka continues toward its status as a de facto standard." noting that Kubernetes, Kafka, and the CSPs are becoming stable layers in the next gen stack and churn around alternative platforms seems to have waned. thoughtworks.com/content/dam…
5
80
237
Really good series of blog posts on the cap theorem by someone who actually bothered to read the paper thislongrun.blogspot.com
3
103
228
I'm excited to announce that a few of us from LinkedIn are starting a company around Apache Kafka and realtime data. linkedin.com/pulse/article/2…
54
255
222
1. The two-phase commit proposal for @apachekafka (KIP-939) is pretty interesting. Quick thread on why it matters. cwiki.apache.org/confluence/…
7
62
225
47,591
Something tells me he’s not a Kafka user.
Tech sales is wild sometimes.
Confluent is a glorified wiki that barely works and they grow 38% in the middle of a recession.
The only excuse I can think of is that it’s such a tedious tool to replace with poor alternatives that procurement always delays it for another year.
19
14
227
73,414
New blog post: Introducing Kafka Streams confluent.io/blog/introducin…
14
185
224
Really good overview of why Kafka's design is particularly well-suited to modeling events and event-driven services confluent.io/blog/apache-kaf…
5
105
218
“Bug me when I’ve left the heater on”, machine learning with Kafka and KSQL for home automation by @SimonAubury. Very cool! medium.com/@simon.aubury/mac…
2
81
217
Trick for productionizing research: read current 3-5 pubs and note the stupid simple thing they all claim to beat, implement that.
6
227
198
1/ There is a mini open source drama, this time because RedPanda, a company with a source-available Apache Kafka clone, bought an open source connector framework and made licensing and trademark changes to thwart other startup competition. (a thread) news.ycombinator.com/item?id…
1
34
209
96,510
Excited to announce @confluentinc has raised a Series D funding round. forbes.com/sites/alexkonrad/…
14
41
192
We're really excited to announce the open source release of KSQL, a streaming SQL engine for Apache Kafka confluent.io/blog/ksql-open-…
2
150
192
I've gotten a number of intelligent responses to my post on exactly-once in Kafka. I've posted a discussion here: medium.com/@jaykreps/exactly…
3
77
193
I'm proud of what @confluentinc has done in last 4 months:
1. Confluent Cloud: Kafka-as-a-service
2. Exactly-once
3. KSQL
4. 2 Kafka Summits
5
40
190
Confluent Earnings #2: 67% year-over-year growth in total revenue, 245% (!!!) year-over-year growth in Confluent Cloud. I’m incredibly grateful to our team, investors, partners, and especially our customers for being part of it.
Today we announced Q3 earnings, marking our 1st quarter with $100M+ in total revenue. Our results are driven by customers around the world who are setting their #DatainMotion to thrive in the modern era. See here for more: bit.ly/3wf1YRQ
6
25
197
Microsoft has released an open source .NET Kafka client on their github account. github.com/microsoft/kafkane…
1
191
192
"Microservices, events, and upside-down databases" @samnewman on event-driven microservices and event streams as a source of truth. oreilly.com/ideas/microservi…
5
69
178
One way of explaining product marketing to technical founders: humans are I/O bound and a limiting factor in caring about your product is getting them to understand it. Good product marketing is like a kind of “compression” that let’s you get big ideas through a maxed out bus.
5
24
176
We at Confluent promised a new set of product launches the first week of every month as part of Project Metamorphosis. There's obviously a larger conversation happening now that needs our attention as citizens and humans, so we're postponing this month's launch.
3
14
172
1/ Faust is a python library from for stream processing with @apachekafka from @RobinhoodApp. I think it's really cool. It highlights one of the things I think we got right with Kafka Streams: supporting stream processing in Kafka at the protocol level. github.com/robinhood/faust
4
53
174
Neha has done so much to help build both Apache Kafka and @confluentinc . From the early Kafka development at LinkedIn, to helping found and grow Confluent to where it is today, we wouldn't be where we are without her.
New decade, big leap: After five and half years at Confluent, I've decided to step down from an operational role in the company. Excited to continue supporting Confluent as a board member & evangelizing for @apachekafka and the event streaming category. Thrilled for what's next!
12
172
First earnings: 64% year-over-year growth in total revenue, 200% (!) year-over-year growth in Confluent Cloud. A huge thanks to all our customers, employees, partners, and investors! Onward!
Today we announced our first earnings results as a public company: Total revenue of $88.3M (up 64% YoY), 617 customers with ≥ $100K in ARR (up 51% YoY) and Confluent Cloud revenue of $20M (up 200% YoY). See here for more info: bit.ly/3CvgJTs #DataInMotion
3
25
165
I wrote a new blog post on stream processing and an alternative to the Lambda Architecture radar.oreilly.com/2014/07/qu…
16
130
163
I wrote a blog post on @apachekafka & the data pipeline challenges in building production machine learning systems oreilly.com/ideas/apache-kaf…
3
96
156
Really excited to announce that @ConfluentInc has raised a $50m Series C financing lead by Sequoia confluent.io/blog/the-arriva…
14
43
154
Kafka now has time-based indexes to allow seeking to a particular point in time cwiki.apache.org/confluence/…
3
135
153
Excited to announce we're kicking off a series of free online talks on Kafka, streaming data, & stream processing confluent.io/apache-kafka-ta…
4
70
155
The recording of my #kafkasummit talk, “Databases Are Only Half Done” is now available. piped.video/4QoCbhsQeyE
1
35
152
New blog post: "Putting Apache Kafka to Use: A Guide to Building a Stream Data Platform" blog.confluent.io/2015/02/25…
6
120
150
Started using a system where our 8 & 10 year old girls earn “screen bucks” for chores, homework, etc. 1 screen buck = 30 mins of screen time. Took about a week for them to recreate most every positive aspect & dysfunction of capitalism using this currency. Chores are done though.
14
5
149
"Eliminating Large JVM GC Pauses Caused by Background IO Traffic": This is a big problem in any java app that logs engineering.linkedin.com/blo…
2
112
152
All the recordings of the talks from Kafka Summit SF are now available for free online: confluent.io/kafka-summit-sf…
64
148
Another awesome blog by @benstopford on the relation between Kafka, KSQL, stream processing, and microservices confluent.io/blog/building-a…
68
141
For the kind of people who like to read distributed systems design docs, this is a good one.
Kafka will use the Raft protocol on top of Kafkas own native commit log to manage metadata in the coming ZooKeeper-free architecture. cwiki.apache.org/confluence/…
3
19
143
Really excited to welcome Steffan Tomlinson, former CFO of Google Cloud and Palo Alto Networks, to Confluent. cnbc.com/2020/06/22/google-c…
4
22
142
This article assumes that Kafka is great but too complex unless you are at massive scale and have no choice. That was once true. Now we offer Kafka as a service for a few cents per GB of writes. You can have nice things, even for normal apps! vicki.substack.com/p/you-don…
12
35
142
Interesting analysis of what happened to Hadoop with a good summary of why @apachekafka matters.
2
52
137
New blog post on Kafka, logs, distributed systems, and stream processing: engineering.linkedin.com/dis…
10
93
137
1/ In April we at @confluentinc kicked off what we call Project Metamorphosis, which is all about building a real cloud-native service around Kafka and it's ecosystem. I talked about why I think this is a big deal in my Kafka Summit Keynote today. Here's a twitter summary:
2
37
138
For distributed systems having some machines get very slow is often far harder to deal with than having them fail. danluu.com/limplock/
5
106
137
Really excited to be joining forces with the Warpstream team! Awesome people and a fantastic product.
BYOC 🤝 Confluent
🎉We’re excited to share that we acquired @warpstream_labs!
Now, our customers can get streaming data any way they need it – self-managed, fully managed, or bring your own cloud (BYOC).
Get the full scoop in @jaykreps’ blog → cnfl.io/3ZjKt3o
8
16
140
15,280
Good discussion of the CAP theorem and it's limits as a predictor of actual empirical availability yokota.blog/2017/02/17/dont-…
59
137
In the US system some things must transcend politics for our society to function. Among those are a belief in truth, democracy, and the rule of law. Trump should be impeached immediately and removed from social media platforms permanently. The bullshit must stop.
2
12
124
Good overview of the differences between "stream processing" and "CEP" softwareengineeringdaily.com…
1
74
134
And yes LinkedIn does own ~2% of Confluent. So needless to say I'm frantically deleting any past negative tweets about MS Windows.
3
52
130
We posted the videos for all the Kafka Summit talks kafka-summit.org/schedule/
3
96
135
We just added pure usage-based pricing to Confluent Cloud, our @apachekafka as a service offering. This means no up-front cost or provisioning up to 100 MB/sec, you just pay for the data you read, write, and store. Prices start at $0.13/GB of reads or writes.
Start streaming in seconds with @confluentcloud, the industry’s only truly cloud-native @apachekafka service. Read the blog about today's announcement from #KafkaSummit: cnfl.io/cloud-native-experie…
5
49
129
O'Reilly has published an expanded version of my long-ass log blog post as a mini book shop.oreilly.com/product/063…
8
103
127
Really good analysis of Spark performance. eecs.berkeley.edu/~keo/publi…
5
80
129
"Understanding Paxos and Consensus in Distributed Systems" ifeanyi.co/posts/understandi…
2
53
134
How @PinterestEng uses Kafka Streams for real-time predictive ad budgeting medium.com/@Pinterest_Engine…
1
52
131
Fantastic non-technical overview of deep learning and it’s limitations technologyreview.com/s/60891…
1
41
124
The use of Kafka for event-driven microservices has gone mainstream in a big way. "Only" half as prevalent as REST is no small thing.
3
27
124
The Calculus of Service Availability: You're only as available as the sum of your dependencies. queue.acm.org/detail.cfm?id=…
3
58
124
My younger daughter refers to reddit as “the happy app” (because I subscribe to r/funny and r/aww) and requests that we switch to it whenever she comes over and sees me on “the sad app” (aka Twitter), which mostly has pictures of angry people. Perceptive.
3
5
123
Good article on not cargo cutting technologies for scalability you don't need. blog.bradfieldcs.com/you-are…
4
71
123
Microservices are about scaling the number of engineers not the number of requests m.signalvnoise.com/the-majes…
5
94
121
It’s cool how @apachekafka KIPs (feature design proposals) have a following much larger than the set of people writing code in Kafka. A great thing about open source is seeing the thought process behind the system, and having smart people show up to tell you how to do it better.
2
21
125
There’s a meme that distributed systems like Kafka are operationally complex so only use them if you really must. That is pre-cloud thinking. Confluent offers Kafka as a fully managed service for < $0.11 per GB read/written/stored with and we do the ops. vicki.substack.com/p/you-don…
8
24
122
Got to read through a (now complete) version of @martinkl's distributed systems book dataintensive.net this weekend. It's superb.
2
43
122
Recording of the talk I gave at @reactivesummit on Kafka, stream processing, and event-driven microservices piped.video/playlist?list=PL…
2
44
118
LinkedIn just open sourced their Blob storage system, Ambry. engineering.linkedin.com/blo…
1
108
124
5. At the end of 2013 I tried to write out a blog post laying out how event streams could act as a central nervous system for companies and the use of data could transition into real-time with stream processing. engineering.linkedin.com/dis…
1
9
122
Great overview of Kafka's streams API for stateful stream processing applications by @DanLebrero of @IGcom danlebrero.com/2017/01/05/pr…
2
57
120
Video of a talk I gave: "Putting Apache Kafka to Use: Building a Real-time Data Platform for Event Streams" vimeo.com/128195441
39
119
Really thoughtful write-up on how @TwitterEng decided to move off of their BookKeeper-based streaming platform and adopt @apachekafka.
Want to know why Twitter decided to adopt @apachekafka as its publish-subscribe system? Check out what we learned through the process 👇blog.twitter.com/engineering…
54
123
I swear product announcements in the big data space sometimes read like the output of a Markov random generator trained on big data fluff.
7
37
118
"Making Sense of Stream Processing" -- free O'Reilly book by @martinkl on @apachekafka and stream processing confluent.io/making-sense-of…
73
117
Here's a super cool demo of Kafka scaling elastically to 10GB/sec with a few clicks in Confluent Cloud. It took us years and a big team to be able to run at that scale at LinkedIn. The cool thing about cloud services is that you can get these capabilities almost instantaneously.
See @apachekafka scale to 10+ GBps in Confluent Cloud with just seven clicks, and learn about what goes on behind the scenes in our latest blog post for #ProjectMetamorphosis by @DanRosanova: cnfl.io/pm-elastic-part-4
1
32
116
How Apache Kafka is Tested confluent.io/blog/apache-kaf…
1
50
117
"The contents of the DB are a cache of the
latest records in the log. The truth is the log. The database
is a cache of a subset of the log."
4
106
107
Ah, life wisdom passed on from 24 year olds to 19 year olds, a genre I didn't fully appreciate before Hacker News.
2
62
113
We did a series of short 10 minute videos introducing stream processing using the streams API in @apachekafka piped.video/Z3JKCLG3VP4
51
116
Really excited to be partnering with @Google to bring @apachekafka to @GCPcloud
Today, we're excited to announce a partnership with @GCPcloud and introduce Confluent Cloud Professional! Read more here: ow.ly/zYs030k7J4Q
3
34
116
Great blog post by @benstopford on building event-driven microservices with @apachekafka confluent.io/blog/build-serv…
2
54
116
This years Kafka Summit will be the biggest ever by far, there are over 15,000 attendee signed up already. I can't wait. Also, it's free! confluent.io/blog/top-5-reas…
3
44
112
I did a Q&A with the @sequoia folks on some lessons learned building @confluentinc sequoiacap.com/newsletter/20…
2
22
112
I don’t usually comment on politics but this one seems simple. Confederate generals (1) fought in an armed insurrection against the US, (2) aimed at perpetuating slavery and a racist ideology, (3) which they lost. Seems like any one of those would be a good reason to change?
It has been suggested that we should rename as many as 10 of our Legendary Military Bases, such as Fort Bragg in North Carolina, Fort Hood in Texas, Fort Benning in Georgia, etc. These Monumental and very Powerful Bases have become part of a Great American Heritage, and a...
5
9
103