Ship data pipelines with extraordinary velocity. Dagster+: dagster.io/plus GitHub: github.com/dagster-io Slack: dagster.io/slack
☁️
- Tweets 1,790
- Following 149
- Followers 5,592
- Likes 1,552
Pinned Tweet
Dagster+ delivers impressive ROI according to the latest Forrester TEI report: 432% ROI with data engineers shifting from maintenance to high-value work.
As one executive noted: “Because we now have a full test suite that ensures everything is actually running as expected, we trust our code more. And now it’s trivial to go through the process of managing deployments and pull requests and managing releases and deployments of pipelines and code.”
Get the full report today!
Link in thread
4
3
19
15,602
Can @duckdb replace our cloud data warehouses or data lakes?
@floydophone and @s_ryz build a data lake from scratch with @duckdb @ApacheParquet and S3.
dagster.io/blog/duckdb-data-…
2
23
117
With @coalesceconf just around the corner, we are pleased to share a new Dagster feature: using @duckdb, @getdbt, and @plotlygraphs with Dagster software-defined assets.
docs.dagster.io/integrations…
2
6
78
Why would somebody fake GitHub stars?
Because they influence serious, high-stakes decisions, including which projects get adopted and which startups get funded.
But fake ☆s are not all that hard to spot. Here we share a Dagster project to do just that.
dagster.io/blog/fake-stars
3
14
71
23,561
Introducing Dagster 1.0! 🎉
After 463 releases and with over 200 contributors, we are proud to release Dagster 1.0.
Data teams who want to access what’s unique about Dagster now have a stable foundation to build on.
Check it out!
dagster.io/blog/dagster-1-0-…
1
11
65
One Data engineering team saved $30K a year on their Fivetran bill by swapping out their data platform's database-to-database data movement tasks with Dagster's embedded ELT functionality.
On Tuesday, @pdrmnvd will walk us through this approach.
dagster.io/events/embedded-e…
2
3
55
5,610
Experience the future of data orchestration with us as we unveil the next generation of Dagster Cloud.
New features, more collaboration, trusted data delivery, and cost management await.
Mark your calendars!
April 17th at 12 PM EST.
Register here:
bit.ly/3TKo7Vt
3
7
51
196,354
We are very pleased to announce our Series B. Elementl - the company behind Dagster - has raised an additional $33M in capital to continue building out the open-source solution, the community, and the commercial adoption for Dagster Cloud.
1
6
50
3,914
We're excited to share that we are partnering with Dennis Hume (Data Engineer at @getdutchie), and @corise_ to build a course on #DataEngineering with Dagster!
The course launches on August 15th –sign up here: corise.com/course/dagster
#DataOrchestration
1
3
46
dbt is one of the most commonly used technologies in data transformation. But how you you fully leverage dbt in a modern data pipeline?
In our upcoming release - Dagster 1.4 - we will be introducing new capabilities to supercharge your dbt work...
Learn more on Aug 2nd
🧵 1/7
ALT Join the Dagster team on Aug 2nd for a special livestream on orchestrating dbt models.
1
5
45
16,870
While most data engineers working on Dagster are fully conversant with Python, others welcome an intro or maybe a refresher on Python basics.
For these folks, we are building out Python primers specific to data engineering.
We just published chapters 5 and 6. Here is a recap:
1
8
44
6,762
We're big fans of @getdbt. We're even bigger fans when you can seamlessly interleave dbt models with Python and other tools.
With software-defined assets, you can:
🌳 Declare data lineage across tools
🕐 Schedule jobs to ensure data assets are fresh
dagster.io/blog/dagster-0-15…
1
8
43
We’re proud to announce 0.13.0 of Dagster. We’ve made dramatic improvements to our core APIs, completely revamped our UI, and brought renewed clarity to our mission.
dagster.io/blog/dagster-0-13…
8
41
Elementl, the company behind the Dagster project, has been renamed Dagster Labs.
In the following blogpost, @schrockn and @floydophone briefly share the thinking behind the change.
tl;dr: it's simpler.
dagster.io/blog/introducing-…
2
9
39
11,876
1/ Today is Dagster Day! Find out how Dagster lets you ship data pipelines with extraordinary velocity. We start in 30 minutes, and you can join us here: piped.video/watch?v=70c84LDZ… . We will thread updates here as we go.
ALT Dagster Day is a live event to celebrate the launch of Dagster Cloud and Dagster version 1.0
2
19
42
We should organize a conference... what would we call it?
@Octolis_app @zentaskai @snowplow @SQreamTech @krakenfx @pictoryai @chainguard_dev @AirbyteHQ @argoproj
17
5
38
20,442
Dagster 1.6 is now available. Entitled "Back to Black" it offers many enhancements to the UI including - you guessed it - dark mode.
But there is a lot more to this release... 🧵
1
6
34
6,526
While we're at it...
dlt is now part of our Embedded ELT!
Enjoy expanded data ingestion from APIs & systems in a seamless, Pythonic approach that complements Sling's database and file system replication for efficient pipeline development.
Learn more:
bit.ly/3vJICsh
5
35
3,500
Dagster 1.8 is out!
• Pipes no longer experimental
• @SDFLabs Integration
• DbtProject integration enhancements
• New data catalog metadata
• Deduping and asset definitions merging
• Declarative automation API
Explore all of the changes here:
bit.ly/4dEFXAB
1
6
34
6,896
It's integration season 🍁
We've been shipping non-stop this month, and it's only the beginning.
Try out our revamped integrations with @AirbyteHQ, @ApacheAirflow, and @duckdb (with @noteable_io and @getdbt coming soon 👀)
ALT Pull requests to the Dagster GitHub repository have increased 51.2% this month.
1
4
31
The new Dagster docs experience is here, complete with dark mode!
This makes it easier than ever to get started with Dagster. Our core focus was on creating a more user-friendly docs structure to ensure that all of the information you need is available within just a few clicks.
2
5
29
5,053
Introducing @floydophone, Elementl's new Head of Engineering.
dagster.io/blog/pete-hunt-pa…
1
4
28
☁️😶🌫️☁️
🙏 Zero-downtime deployments
🙏 Offloaded operations burden
🙏 Enterprise auth and granular permissions
... and more!
Introducing Dagster Cloud, our enterprise grade data orchestration platform. Like Vercel, but for your data pipelines 👀
dagster.io/blog/introducing-…
2
5
26
Together with the community, Dagster is powered by a core team of contributors from Elementl.
Today, we're excited to announce our $14MM series A. Read more about our future timeline 👀☁️☁️
dagster.io/blog/decade-of-da…
4
24
Learn how to prompt ChatGPT to answer technical questions about your documentation!
@floydophone shows us how to use Dagster to power a chatbot trained on your latest support docs using @langchain and @OpenAI.
dagster.io/blog/chatgpt-lang…
3
2
29
3,521
With the launch of the Cloud solution, we revisit our popular "Poor Man's Data Lake" project, switching from local @duckdb to @motherduck.
"a huge usability improvement on top of S3 and Parquet, and it’s much easier to collaborate using Motherduck rather than vanilla DuckDB."
1
1
27
7,938
Viewing data work as asset creation, or "thinking in assets" results in clearer data lineage, easier maintenance, and better transparency in data pipeline development.
@tims_tangents walks through this approach in our latest blog post.
Read now:
hubs.ly/Q02k4hCs0
2
26
5,211
Fan-out in data engineering is when one operation splits into many parallel downstream tasks, boosting efficiency. It's vital for high-velocity data ingestion, distributed computing, and rapid data transformation.
Explore fan-out's impact in-depth here:
bit.ly/46t3NwR
3
27
1,554
If you’re looking to learn how to design, build, and maintain a data platform, look no further than Dennis Hume’s 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝘄𝗶𝘁𝗵 𝗗𝗮𝗴𝘀𝘁𝗲𝗿 course launching on CoRise on November 21st!
1
2
28
We are creating a guide to support data engineers who may be new to Python.
Today we publish part 3: Best Practices in Structuring Python Projects.
We dive into 9 key best practices, provide examples of folder structure and review the role of key files.
dagster.io/blog/python-proje…
1
4
26
2,350
Discover Dagster in this 10 minute overview with @lopp_sean and learn how progressive data engineering teams deliver high quality data assets faster and with greater control.
piped.video/watch?v=L5kTxCM-…
1
22
21,847
If you're a dbt user, you probably remember the new pricing plans that will come into effect this year.
Get ahead of the upcoming pricing changes by learning more on how you can use Dagster to orchestrate your dbt runs
Watch our detailed guide here:
piped.video/watch?v=yv97Xgbw…
1
2
23
2,663
Building better data analytics pipelines starts with managing the complexity inherent in todays data environments.
The complexity resides in the data, but also in the plethora of systems, stakeholders, schedules, versions, and compute environments you have to juggle.
2
2
25
2,468
The Dagster Labs team is growing! Here are our team photos from our Jan 2023 vs. Jan 2024 offsites. Welcome to all the new team members.
We have big plans for 2024 - come join us! We will be adding more roles as the year rolls on: dagster.io/careers
1
23
1,269
Last month @pdrmnvd shared a bird-centric MDS pipeline at MDS Fest '23.
Pedram showcased a wide range of free technology for data ingestion, data storage, data transformation, data orchestration, and data visualization.
[1/2]
1
1
19
3,246
Would you like to see Dagster in action?
Join us on Thursday, Nov 3rd at 9 AM PST for a live demo of Dagster. @OwenKephart will show how to create a pipeline using software-defined assets, then will hold a live Q&A.
Sign up below!
dagster.io/dagster-demo-sign…
24
Is your team frustrated by the limitations of @ApacheAirflow ?
Join us on Feb 8th as we share best practices for smoothly transitioning to Dagster.
It's a robust tool that provides a fantastic developer experience and fosters collaboration across teams.
dagster.io/events/dagster-ai…
1
5
22
18,353
How does the $2.5T #crypto market get #data?
@artemis__xyz uses Dagster+ to aggregate and deliver real-time crypto data to financial institutions and investors-- enhancing reliability, data lineage management, and collaboration.
Read the full story:
bit.ly/3WGaUyv
3
23
5,525
The Dagster GitHub repo has officially reached 10,000 stars!
We appreciate our growing community + the collective effort of the contributors, users, and supporters who believe in making data orchestration more approachable, reliable, and productive.
Here's to the next 10K.
2
20
1,570
Dagster 1.4 — “Material Girl” — is now live. The release includes dagster-dbt enhancements which we will demo on Aug 2nd. 1.4 also evolves asset materializations, giving you more fine-grained control and observability over when, why, and how your computations run.
1/6
1
4
21
4,579
Struggling with #data pipelines from local to cloud?
Check out this hands-on walkthrough by @__AlexMonahan__ and @coltonpadden. They tackle @motherduck's DuckDB integration and @dagster orchestration for smooth data management.
Learn by example:
bit.ly/3QpwuUi
5
22
1,401
Dagster Pipes and External Assets are quickly gaining adoption and will prove to be a game changer in how people think about data orchestration.
dagster.io/blog/dagster-pipe…
20
2,879
dbt is central to many Dagster projects, so it’s no surprise that we have focused on making @getdbt models easy to integrate into Dagster pipelines to centralize observability and run metadata.
And now we are adding Declarative Scheduling for dbt!
1
1
21
2,558
Build generative AI steps into your pipelines with our new dagster-openai integration!
You can now use OpenAI's powerful LLMs in your data pipelines for smarter automation, streamlined tasks, and cost-effective insights.
Explore the details:
bit.ly/48M45yg
2
20
1,837
We have exciting news: Dagster 1.0 and Dagster Cloud will be released on August 9 at 9AM PST / 6PM CEST.
Join us for Dagster Day and learn about our promises for API stability in our open-source framework, as well as the launch of our hosted offering.
dagster.io/blog/announcing-d…
5
21
As @DSJayatillake publishes his 3rd and final installment of 'Dabbling with Dagster,' here is a recap, ICYMI.
His conclusion? "I wholeheartedly recommend Dagster over Airflow."
David is the Head of Data @metaplane and wrote this series independently. And we love it!
[1/4]
1
3
23
Looking to migrate away from Apache Airflow? With Dagster, we provide built-in utilities to help you seamlessly transition away from the legacy platform.
Here are four new resources and an upcoming event to help teams migrate to Dagster!
🧵 1/6
1
1
21
2,232
Imagine if you had one pane of glass for your data team.
Imagine if you could understand the lineage of your data assets, all in one place.
ALT A lineage graph of data assets. Initially, there are assets from Airbyte, Snowflake, and Fivetran. A subset of assets are transformed using dbt. Then the processed assets flow into a Noteable asset.
1
22
We will be hosting our Fall Launch Week from October 6th to the 13th. Each day, we will announce and showcase new features and capabilities on the Dagster platform.
The theme of Launch Week is "Escaping the Modern Data Trap".
Why?
[1/3]
1
1
18
8,868
For data engineers new to Python, Python Packages can be a bit of a head-scratcher. @elliot_j_g wrote a handy Python primer as an intro to learning Dagster.
In Part 1, he covers modules, packages, __init__.py, pip, and relative/absolute imports.
dagster.io/blog/python-packa…
1
5
20
2,067
📢 Join us Dec 7th at 9 AM PST / 5 PM GMT for a special virtual 𝗗𝗮𝗴𝘀𝘁𝗲𝗿 𝗖𝗼𝗺𝗺𝘂𝗻𝗶𝘁𝘆 𝗠𝗲𝗲𝘁𝗶𝗻𝗴.
We'll share updates on the Dagster project, news on integrations and partners, new faces on the Elementl team, and hold a live Q&A.
dagster.io/community-day
1
1
17
In data pipelines, we often have processes that 'fan-out' - an operation that results in many identical downstream tasks.
A pipeline with a 'fan-out' step may require a scale-up of computing power, with each sub-task run in isolation from the others.
dagster.io/glossary/fan-out
1
5
17
2,814
#Dagster brings you fine-grained orchestration and deep observability.
#SDF brings you advanced SQL transformations and static analysis.
Together, you get the most efficient and transparent #data pipelines.
Explore one of the best combos since PB&J here:
bit.ly/4cv9ewm
1
3
17
3,908
The moment is finally here... 🥁
0.10.0, "Edge of Glory" has been released 🙏
Hope Lady Daga is proud of us ☺️
github.com/dagster-io/dagste…
4
16
Three main reasons to choose Dagster over Airflow:
1. Dagster is designed for end-to-end productivity
2. Dagster supports a declarative, asset-based approach to orchestration
3. Dagster is cloud- and container-native
@s_ryz and @schrockn make the case.
dagster.io/blog/dagster-airf…
1
4
18
For #dataengineering practitioners, embracing Domain-Specific Languages (DSL) is not just about technical efficiency but also about being able to scale, simplify, standardize, and democratize #data processes.
Learn more about DSLs in our latest blog:
hubs.ly/Q02kMYWK0
2
18
1,996
Dagster Cloud launches today! We just posted a submission to @ProductHunt here: producthunt.com/posts/dagste… - We look forward to the feedback as we release Dagster Cloud to the world!
1
18
"Upsert" is a basic data engineering operation: update a record in a database or file, or create it if it does not yet exist.
And yet, upserting has some nuance when it comes to performance and interpretation.
Learn more about upsert: dagster.io/glossary/upsertin…
2
16
2,388
Following the release of Dagster 1.0 and the launch of Dagster Cloud, we are delighted to present core concepts, demos, and best practices in #dataengineering at four upcoming conferences!
ALT On the road again: Dagster will be presenting at four upcoming conferences.
1
4
19
Would you like more structure in your ML experiment tracking?
In under 5 mins, @GusCavanaugh runs us through some best practices in building a ML pipeline, tracking experiments in @MLflow and using @github actions as a CI tool.
piped.video/watch?v=kSd_UvRH…
3
17
1,957
Dagster 0.15.0 "Cool for the Summer" has been released! Featuring...
🌟 Software-defined assets are marked fully stable!
👀 A new partitions and backfills experience in Dagit
⤵️ Top-level inputs can be passed to jobs
And more! Check out the recap 👇🏼
dagster.io/blog/dagster-0-15…
1
5
17
ngrok's Data Platform runs on #Dagster. Here's how they manage an influx of about 20TB of net new data each month in over 400 Dagster assets and ~170 dbt models.
Stack: @apachekafka @getsentry @AirbyteHQ @kubernetesio @ApacheIceberg @dag
ngrok.com/blog-post/how-ngro…
2
19
5,843
We’re thrilled to announce a new integration between Dagster and Great Expectations (@expectgreatdata). GE enables Dagster users to build data quality checks directly into their pipelines, making it easier to catch data issues early.
More on our blog: dagster.io/blog/great-expect…
3
18
In this updated tutorial we migrate the Poor Man’s Data Lake away from S3 and Parquet files into a single system. it’s straightforward and we realize all of the benefits of Motherduck without touching our business logic.
dagster.io/blog/poor-mans-da…
1
2
17
3,539
We're hosting our fourth Dagster Community Meeting, tomorrow, Jan 12, at 9 AM PST (UTC-7)!
We have three presentations lined up from the core team, running through some features in our 0.10.0 release "Edge of Glory"
👇🏼👇🏼👇🏼
1
7
18
We often get the question "Should I use Dagster or dbt? They both have dependency graphs". We view them as complementary tools. So the answer is "both." dagster.io/blog/dagster-dbt
2
17
The upcoming Dagster 1.3 release will bring major ergonomic improvements to the config and resource systems by using Pydantic for specifying schema and validation.
dagster.io/blog/pythonic-con…
2
1
17
1,744
Happy to announce of latest release, 0.7.0. We've made a ton of progress on a bunch of fronts, working with users with real scalability needs.
medium.com/dagster-io/dagste…
1
3
17
To bundle, or not to bundle, that is the question (though you already know our answer)
Live coverage of Bundlegate continues Tuesday, March 15th. See the showdown at atlan.com/great-data-debate/ @AtlanHQ
2
6
17
Ever wonder how other teams build their data platforms? Join us at 9:00 - 10:00 AM Pacific Time Tuesday, February 9, 2021 to meet Dennis Hume from @Drizly and @kantrn from @geomagical_labs and learn about their production Dagster setups. bit.ly/monthly-community-mee…
5
17
Learning Dagster opens many opportunities. For example, would you like to be a Mutineer? Here's your chance.
Mutinex.co - an Australia-based SaaS company specializing in marketing mix modeling - is looking for a Staff Software Eng. with Dagster & dbt experience to work with clients Samsung, Mars, CUB, and ING.
Join the team and help power better marketing decisions!
(PS: Dagster Labs, like many of our customers, is also hiring! -> dagster.io/careers )
mutinex.co/join-the-mutinex-…
2
16
1,508
We're LIVE.
Meet Dagster+.
We'll be covering the event in this thread, but you can join the event here:
piped.video/watch?v=_Z4xxZYE…
1
3
14
6,290
Madagnna's new remastered single, 0.11.0 "Lucky Star" has been released! 🌟
github.com/dagster-io/dagste…
3
5
17
Weekly Release Highlights: 1.1.11✨
☝️ One command `dagster dev` to run both UI and daemon in the same process during local dev.
🏎️ Utility to cache compilations from @getdbt Cloud jobs, which allows dbt assets to be loaded faster.
📜 New example for the branching I/O manager.
3
17
1,556
👀 See a preview of what we have in store for Dagster Day - join us on August 9 at 9AM PT to learn about our release of Dagster 1.0 and Dagster Cloud!
1
3
16
Dagster 1.9 will be out soon! We are targeting Halloween; in keeping with tradition, we are crowdsourcing the name from the community. Join the discussion on GitHub and submit your festive song choice.
Several exciting improvements and integrations are coming with this release, so stay tuned for more updates!
1
3
15
2,139
Stored procedures are a critical concept data engineers need to master.
In this Dagster Glossary entry, we provide an overview and a Python/Postgres example of these precompiled and stored SQL statements and procedural logic.
dagster.io/glossary/stored-p…
1
15
1,122
A software engineer’s commodity is a code change.
To be a productive data engineer, you need to master changes: how these affect the program and others on the team.
@alex_langenfeld walks us through a practice called “Stacked Diffs” or “Stacked PRs.”
dagster.io/blog/productive-s…
1
2
14
Today's the day.
Today, we present to you Dagster+.
New capabilities that embed data reliability, accelerate dev cycles, optimize costs + enrich metadata insights await.
The virtual event starts in 1 hour.
Can't wait to see you there.
Join here:
bit.ly/3UltlYd
5
16
2,948
From low-code to high-impact.
Discover how Sean Pool, a one-person data team at @ErewhonMarket, transformed their data strategy using @Dagster to build a scalable, cost-effective data platform that powers Erewhon’s growth.
Read the full story here:
bit.ly/46MSMX6
2
1
14
1,098
We're having a blast at the #DataAISummit!
If you didn't know already - your data engineering doesn't need to be a drag.
So come talk to us (at booth 36) and learn how to ship data confidently with the orchestration platform loved by engineers everywhere.
2
13
893
Dagster 0.10.0 will be 🔥🔥🔥
In our most recent community meeting, we showcased features in our upcoming release on January 14, 2021
Here’s our sneak preview (with time stamp links!)
👇👇👇
2
1
14
The Data team at @zephyr_ai is revolutionizing cancer treatment through bioinformatics and predictive analytics. They shared with us their journey of building ML pipelines on Dagster.
dagster.io/blog/zephyr-ai-ca…
ALT Zephyr AI's data pipeline tech stack (Oct 2022)
1
1
15
Are you attending Data Council in Austin in March? Come pre-game with us courtesy of the modern data stack dream team: @BrooklynData , @_hex_tech @HightouchData @AirbyteHQ and yours truly.
eventbrite.com/e/data-counci…
3
14
2,242
Exciting news!
We've relaunched a fan favorite - Dagster Deep Dives - a series of in-depth tutorials and interviews with industry experts on all things data engineering.
RSVP for the first session:
bit.ly/4dsmoeH
#Dagster #DataEngineering
2
12
755
Are you done with Airflow? If you’ve ever had to install Kubernetes locally just to test a simple pipeline, or have resorted to the push-to-prod-and-pray method, it’s time to take a look at how Dagster’s configuration and resource systems allow you to develop locally and ship confidently.
We will be hosting our second Dagster Deep Dive event later today: Configuration & Resources with Colton Padden
8:30 AM PST / 11:30 AM EST / 4:30 GMT / 5:30 CET
Add it to your calendar here: addevent.com/event/Gy2008622…
2
2
14
1,192
If you are at this year's #DataAISummit be sure to catch @s_ryz 's live presentation on "The Future of Data Orchestration: Asset-Based Orchestration" on the 28th @ 1PM.
Sandy's demo includes @AirbyteHQ , @getdbt and of course @databricks.
databricks.com/dataaisummit/…
2
15
1,520
Many ETL workflows start off as scripts.
After building these scripts, you need to put them into production. You want things like...
⏱️ Event or schedule-based triggers
🔍 Observability into your computations and data
and more!
Dagster provides all of this out-of-the-box.
1
1
14
1,286
It's been really fun seeing you all at the @SnowflakeDB Data Cloud Summit so far!
We're showing off the power of @dagster with live demos and talking all things data engineering, so stop by booth #1251 and see firsthand how Dagster can upgrade your data operations.
1
13
854
We've already supported generating software-defined assets from @getdbt Core.
Starting with next week's release, you can generate software-defined assets from dbt Cloud.
In one place, understand the lineage of your dbt Cloud models along with your other data assets.
2
15
Why Dagster over Airflow?
We show the advantages of using Dagster for:
🧪 Developing and testing computations
📦 Deploying and executing pipelines
🔍 Monitoring computations and observing data assets
Read our case 👇
dagster.io/blog/dagster-airf…
5
14
Define your data assets and their dependencies in code. We'll keep their materializations in storage up-to-date.
With software-defined assets, Dagster now brings a declarative model to data orchestration.
dagster.io/blog/software-def…
1
2
13