Simple ideas, pursued maximally. Co-Founder & Co-CEO @runwayml.
New York, NY
- Tweets 552
- Following 447
- Followers 5,315
- Likes 3,307
Text to video is coming!
Next week, will be sharing more on how this came to be, how it works, and our rollout strategy.
In the meantime, go to runwayml.com and sign up for early access :)
Make any idea real. Just write it.
Text to video, coming soon to Runway.
Sign up for early access: runwayml.com
21
232
1,671
Text-to-video was act zero, here's act one.
The future of video generation is in building tools that are controllable, expressive, and fun -- and in increasing the space of possible choices that you can make.
Introducing, Act-One. A new way to generate expressive character performances inside Gen-3 Alpha using a single driving video and character image. No motion capture or rigging required.
Learn more about Act-One below.
(1/7)
9
28
237
21,790
“You are all the movies you’ve watched.”
In collaboration with @c_valenzuelab.
4
47
230
For @stupidhackathon, made an app that shows the typing indicator to anyone messaging you, forever. Collab w @isiain agermanidis.github.io/facebo…
2
73
199
Models just want to generalize.
For the past years, we’ve been pushing the frontier of controllability in video, releasing new models and techniques for inpainting, outpainting, segmentation, stylization, keyframing, motion and camera control.
Aleph is a single in-context model that can solve all of those tasks at once, as well as other ones (e.g. novel view synthesis), in many cases with zero task-specific training.
The recipe of References continues to work, and scaled to new heights.
Introducing Runway Aleph, a new way to edit, transform and generate video.
Aleph is a state-of-the-art in-context video model, setting a new frontier for multi-task visual generation, with the ability to perform a wide range of edits on an input video such as adding, removing and transforming objects, getting new angles of a scene and modifying style and lighting, among many other tasks.
11
32
192
30,060
We're announcing a new research direction towards building general world models.
If next token prediction has gotten us this far, imagine where next frame prediction will take us.
Introducing General World Models.
We believe the next major advancement in AI will come from systems that understand the visual world and its dynamics, which is why we’re starting a new long-term research effort around general world models.
Learn more: bit.ly/3RexmuJ
8
14
167
25,332
👨🏻🔬 New experiment: Uncanny Road is a tool for collectively synthesizing a never-ending road with the help of Generative Adversarial Networks. Collaboration with @c_valenzuelab. uncannyroad.com/
2
54
156
Thrilled to finally share what we’ve been working on: Gen-3 Alpha, our new base model for video generation. A significant improvement in fidelity, geometric consistency, and prompt understanding over Gen-2, and a step towards building General World Models.
Introducing Gen-3 Alpha: Runway’s new base model for video generation.
Gen-3 Alpha can create highly detailed videos with complex scene changes, a wide range of cinematic choices, and detailed art directions.
runwayml.com/gen-3-alpha
(1/10)
9
22
150
14,848
Working on something special for the next @runwayml release: a new way to navigate GAN latent spaces. Coming soon!
3
18
106
Why I think that advancing simulation is the most important thing to be working on in AI
6
13
100
22,970
Just the very beginning of what's coming.
Generate everything.
Introducing AI Magic Tools
Dozens of creative tools to edit and generate content like never before. New tools added every week.
Available now: runwayml.com
1
7
94
I've learned that it takes about a year from initially setting a research direction internally to getting it to finally work.
It took a year of investing in scaling laws for video models to releasing Gen-3 Alpha. It took a year of investing in a unified multi-task approach to releasing Gen-4 References/Aleph.
Always a difficult balance to strike between trying to go somewhere ambitious too early and introducing uncertainty vs. moving too late and doing something incremental. But leading the way is more fun than catching up.
Excited about the new direction we're setting now and where we'll be next year!
2
10
89
5,777
Welcome to the storytelling era of generative models.
Incredibly proud of our team for this release. We set a very high standard internally for our next model. Building the world's best model for video generation was the baseline, what I'm most excited about is the new paradigm of free-form, reference-based control that you'll get to know soon.
Today we're introducing Gen-4, our new series of state-of-the-art AI models for media generation and world consistency. Gen-4 is a significant step forward for fidelity, dynamic motion and controllability in generative media.
Gen-4 Image-to-Video is rolling out today to all paid plans and Enterprise customers.
1/8
6
6
81
8,341
As a field, we're just scratching the surface in building multimodal simulators of the universe. This will be a long journey, and the true purpose of deep learning.
3
12
80
20,941
Total Pixel Space, which won the Grand Prix at this year's AIFF, is a wonderful video essay and, by the way, one of the clearest descriptions of universal simulation (as search in the space of all possible universes)
piped.video/watch?v=zpAeygE4…
4
11
74
20,064
“Since my musical ideas are always changing, so does my notation.” -- John Cage
If your ideas change, why should the interface remain the same?
1
7
70
Lucid Dream Test
Imagine the following scenario. You enter a room and you are asked to wear a VR headset that has a camera and supports a passthrough mode (which can display a real-time feed of your surroundings). Once you put on the headset, you find yourself in what appears to be the exact same room.
For the next five minutes, you are asked to walk around, interact with objects, and have conversations with people who enter the room.
Finally, while still wearing the headset, you are asked the question: do you believe that you are viewing the real world via your headset’s passthrough mode, or is everything you're experiencing generated by an AI model?
As the quality of world simulators further improves, we’ll need to increasingly focus on evaluations that include the ability to interact with the simulated environment. This is a harder task than simply generating physically plausible videos, where the models can “cheat” by avoiding generating difficult futures.
7
19
73
23,605
Excited to announce the release of #stablediffusion to make state-of-the-art image generation models accessible, responsibly, to a wider AI community.
Incredible work by @pess_r (@runwayml) @robrombach (LMU) and the team at @StabilityAI!
Right one more time.
Happy to announce the release of #StableDiffusion for researchers. Public release soon.
GitHub here: github.com/CompVis/stable-di…
Blogpost (correct link) here: stability.ai/blog/stable-dif…
More on @StabilityAI later when I figure out how to use the internet 🫣
16
63
Just published a technical overview of Green Screen, which you can read here:
medium.com/runwayml/building…
Summary in thread 🧵
Introducing Green Screen!
The first real-time web tool for cutting objects out of videos. Using machine learning, it makes rotoscoping (a.k.a. masking) a lot faster and a lot less painful.
Start creating now: app.runwayml.com/video-tools…
1
11
61
The latest @runwayml release introduces Vector Input, a real-time interface for interacting with generative models that take a vector of random numbers as input (think BigGAN, StyleGAN, etc). Thread ⬇️
Runway beta v0.4 is here! 🏄♀️
Major updates include:
- 5 new models: StyleGAN, GPT-2, MaskRCNN, PhotoSketch, Visual Importance.
- Vector Input: a grid-like interface for exploring the outputs of generative models such as BigGAN and StyleGAN.
More features on the way! 🎁
1
18
61
We’re still vastly underestimating the amount of reasoning that visual generative models do.
A lot more to come on this front.
Gen-3 Alpha exhibits several simulation capabilities, including the ability to generate dynamic camera motions, complex fluid motion, and interactions between objects. We expect further simulation capabilities to emerge as we continue to scale our models. To learn more about our long-term research efforts to build General World Models, visit:
research.runwayml.com/introd…
Prompt: A cinematic top down shot of cold water being poured into a hot frying pan.
(1/5)
9
58
5,276
Gen-4 Turbo is an amazing feat of research and engineering. To give a sense of the improvement, in our internal evals its outputs were preferred ~90% of the time compared to those of non-Turbo Gen-3 Alpha.
Today we’re introducing Gen-4 Turbo. The fastest way to generate with our most powerful video model yet. With Gen-4 Turbo it now takes just 30 seconds to generate a 10 second video, making it ideal for rapid iteration and creative exploration. Now rolling out across all plans.
18
56
5,648
The entire Runway team was involved in one way or another in building Gen-3 Alpha: from the model architecture, to the infrastructure, data, and evaluation. Everyone showed so much care and passion building this; art and science come together in this company in a way that they do nowhere else. Consider joining us: runwayml.com/careers/
2
6
56
6,087
References has been the biggest demonstration so far for me that if you focus on the problems that you really need to solve, rather than the problems that feel most solvable, deep learning will reward you for it.
Today we are releasing Gen-4 References to all paid plans. Now anyone can generate consistent characters, locations and more. With References, you can use photos, generated images, 3D models or selfies to place yourself or others into any scene you can imagine.
More examples below.
(1/4)
2
5
57
3,707
Gen-2, our text-to-video model, is now available for everyone to use. A new kind of camera that captures ideas, stories, visions into moving pictures.
Thank you to all the early research testers for all your creativity, enthusiasm, and feedback.
If you can imagine it, you can generate it.
Gen-2 is now available on web and mobile: bit.ly/3qtxeh9
8
9
53
13,746
Time and again, we've seen that engineers can excel in AI projects even without formal training in the field, when given the opportunity.
Excited to formalize this into the Runway Acceleration Program, a full-time 3-month initiative to fast-track engineers into ML practitioners.
Today, we’re excited to announce the launch of Runway’s Acceleration Program: an initiative designed to support exceptional software engineers in becoming ML practitioners.
Learn more about the Runway Acceleration Program and apply: runwayml.com/blog/introducin…
2
9
49
30,788
My prediction is that our ghostwriting credits for big company roadmaps will increase exponentially this year.
Introducing General World Models.
We believe the next major advancement in AI will come from systems that understand the visual world and its dynamics, which is why we’re starting a new long-term research effort around general world models.
Learn more: bit.ly/3RexmuJ
6
50
6,193
👨🏻🔬 New experiment: A chatroom for algorithmic copies of real people. Are you ready to be simulated? copyof.me
4
18
47
The Runway research team is coming to #CVPR2023!
We’re hosting a small dinner on Tuesday 6/20. DM if you’re interested in joining to chat about generative video, creative tools, and more.
3
7
42
25,737
NeurIPS is coming up soon, and we’re hosting another dinner. This is one of the best ways to meet the Runway research team.
RSVP here: lu.ma/cpr98pmj
1
14
44
16,278
Great work by @graceluo_ @jongranskog training diffusion models to be aligned with VLM feedback in minutes, which can be used to improve commonsense reasoning and enable many kinds of visual prompting.
✨New preprint: Dual-Process Image Generation! We distill *feedback from a VLM* into *feed-forward image generation*, at inference time.
The result is flexible control: parameterize tasks as multimodal inputs, visually inspect the images with the VLM, and update the generator.🧵
1
4
46
5,416
This summer has a similar feeling as that of 2022. Something has clicked with the latest generation of models and there’s suddenly so much low hanging fruit. Back to making new prototypes every weekend.
2
2
44
2,293
Tired: Build camouflage tech to avoid being identified by ML systems
Wired: Make the concept of ‘being identified’ meaningless
2
8
39
I sometimes compare Runway's end-of-week demo meetings with the demo meetings of any other company I've experienced in the past and it makes me feel immense joy and gratefulness
1
3
39
4,463
Thrilled to partner with Canva to bring video generation capabilities to 150+ million more users
Today, we are thrilled to announce that we are partnering with @canva to bring the power of Runway and AI video generation to Canva users all around the world.
runwayml.com/blog/runway-par…
2
2
39
4,317
Want to find out what happens when you put together the best density of talent in the industry, introduce a strong conviction that deep learning is early and will simulate everything, and remove all barriers that prevent researchers, engineers, and artists from working together?
If any of the following sequence of events resonates with you, email me now at c at runwayml.com
- For the last couple of years, you have been exploring interesting AI research ideas, either as a masters/PhD student or inside a big AI research lab. Nice people, good ideas.
- As the field is moving at breaking speed, you have been trying to make sure your lab/team stays up to date.
- But it's impossible to do anything. Everything has bureaucracy and requires multiple approvals.
- You realize there are multiple teams trying to accomplish the same goals, but due to internal politics, resources get allocated somewhere else. You have to wait.
- You wait. A lot.
- You realize those who get the resources internally aren't actually the best technically, but the best politically.
- It's too late. The idea you had gets published somewhere else. It's the same thing you were thinking, although you thought you could do better.
- After waiting six months for manager approval, you manage to get some resources. You work incredibly hard to catch up, to prove your ideas.
- You are told not to release or speak about your work until competitor X does it first. It will then take some time to decide what to do.
- After five more months of waiting, you get approval to publish a blog with benchmarks that get you close to SOTA, but you're late by ~12 months. There's already something more interesting to solve.
- Although your research is very promising, there is no product priority or plan to do anything with it.
- You lose patience. You know your work can have meaningful impact. But your manager tells you to wait three more months, there's a "big new project coming."
- You keep waiting.
- And the loop starts again.
2
6
41
6,925
Game Worlds is now available to everyone in Beta.
It represents an entire new research direction for us towards generative storytelling and world-building:
Today, we're launching the Runway Game Worlds Beta.
Over the last few months, we have been working on research and products that are moving us closer toward a future where you will be able to explore any character, story or world in real time.
While generating the pixels of these experiences is one aspect of this new frontier, another is the need for novel mechanics and interfaces. From how stories unfold to how your choices affect the worlds you’re simulating.
Today’s beta release marks a first step in this direction, learn more below.
(1/5)
2
1
42
3,311
Frames is our newest image generation model. It's an incredibly versatile model, built in close partnership between our research and creative teams, with a focus on precise control over style and aesthetics.
Introducing Frames: An image generation model offering unprecedented stylistic control.
Frames is our newest foundation model for image generation, marking a big step forward in stylistic control and visual fidelity. With Frames, you can begin to architect worlds that represent very specific points of view and aesthetic characteristics.
See below for examples.
World 1089: Mise-en-scène
(1/11)
2
3
41
2,676
New funding to help us tell more stories, simulate everything:
Announcing our Series D: Towards a new media ecosystem with world simulators
runwayml.com/news/runway-ser…
3
1
38
1,589
Most of the received wisdom about how to build technology companies was developed in the 2010-2020 era of relative technological stagnation and was overly focused on becoming great at distribution over the ability to continuously invent new things.
2
3
36
2,218
I spent way too much time over the past week exploring the 1,000 categories of BigGAN; enough time to convince myself that more and more artists, designers, writers, etc. will start working with these kinds of “generative discovery” tools in the coming years.
1
4
35
Out today: more control over subject motion, more control over style, and significant quality improvements to our image generation tools
Today we're releasing new features and updates to provide more control, greater fidelity and even more expressiveness when using Runway.
We are excited to introduce Motion Brush, Gen-2 Style Presets, updated Camera Controls and more.
Thread below
2
3
33
6,534
Happening this evening in London — it’s an amazing lineup of speakers.
Join us for RNA (Research and Art) 003 in London on Thursday, October 24th.
With presentations from Tim Fu, @GKopanas, @arkitus and @RebeccaFiebrink, each followed by a rapid fire Q&A.
RSVP: lu.ma/j8rkkhmo
2
4
25
2,500
Runway Gen-3 Alpha will soon be available in the Runway product, and will power all the existing modes that you’re used to (text-to-video, image-to-video, video-to-video), and some new ones that only are only now possible with a more capable base model.
2
2
28
1,560
A different framing: we're in one continuous era of simulation. The only thing that changes is what's being simulated: from toy worlds, to the world as perceived by humans, to the world beyond human perception.
The short paper "Welcome to the Era of Experience" is literally just released, like this week. Ultimately it will become a chapter in the book 'Designing an Intelligence' edited by George Konidaris and published by MIT Press.
goo.gle/3EiRKIH
1
1
26
9,943
Intuition in a fast-moving field is overrated because what didn't work yesterday might work today and vice versa. Building the appetite, skillset, & infrastructure to try as many ideas as possible, as quickly as possible, is still surprisingly underrated.
1
7
25
3,735
Imagine this kind of latent space exploration, some years from now, where each individual output is not a single image but a narratively coherent feature-length film.
3
3
21
This is a plot of the number of PRs merged across all @runwayml repos every week over the past year. Anyone want to venture a guess on what changed in December?
2
24
After months and months of tinkering and a few lifetimes of learning, we're finally ready to open the floodgates. Runway public beta is here :)
Hello world, we have some exciting news to share with you today!
🔥🔥 Runway is now in public beta, available for anyone to download! 🔥🔥
Along with that, we are launching a brand new identity and website, check them out! ✒️ 💅
Learn more at runwayml.com
1
5
21
Pushing the Pareto front of quality and efficiency in video generation, one release at a time.
It's been surprising to see how well this new model performs, and it makes exploring new ideas so much more interactive.
We trained a new version of Gen-3 Alpha, Turbo, that can generate videos 7x faster than the original Gen-3 Alpha, while matching its performance on many use cases. We’ll be rolling out Turbo for Image to Video with significantly lower pricing over the coming days while also making it available to free users.
Gen-3 Alpha Turbo redefines the efficiency frontier for high-fidelity video generation, unlocking many new possibilities of near real-time interactivity.
2
20
1,403
Higher concurrency & new features for our API:
We've released a number of updates to the Runway API that give users more features, flexibility and control when integrating Gen-3 Alpha Turbo into their apps and products.
Updates include added support for keyframes, a new self-serve tier with a higher max concurrency limit, and multi-user organization support for optimized admin management.
Learn more: docs.dev.runwayml.com
1
20
1,496
The release of Sequel marks a new chapter for @runwayml, where we seek out to build the fastest way to turn thought into moving image, and increase the number of really good films in the world.
Introducing Runway Sequel.
The first professional video editor made for the web, powered by machine learning.
Start creating now: runwayml.com
4
19
Since we released Gen-2 last year, we learned a lot. We learned that multi-modal artistic control is key, that video diffusion models are nowhere close to saturating performance gains from scaling, and that those models, in learning the task of predicting video, build really powerful representations of the visual world.
1
1
21
489
A consistent thrill of the past month has been pushing a Gen-2 model change and seeing Paul direct it to brilliant unexpected places within minutes. This beautiful short film sets the tone for all that's to come.
1
1
17
3,681
New work from Runway Research, led by @deeptigp, where we investigate bias in text-to-image models and propose a method (DFT) to reduce it that goes beyond hidden prompt engineering
Runway Research is proud to share its latest paper, Mitigating Stereotypical Biases in Text to Image Generative Systems. This work is an important step towards better representation for all people.
Read the paper and access our open sourced prompts: research.runwayml.com/public…
20
1,613
Brought back to life some code I wrote a while ago for visually tracing the execution of a Python program (featuring @PyTorch code)
1
2
16
Expanding the Gen-4 API with generalist image capabilities:
Earlier this month, we released Gen-4 References, our most general and flexible image generation model yet. It became one of our most popular releases ever, with new use cases and workflows being discovered every minute.
Today, we’re making it available via the Runway API, enabling anyone to integrate its powerful multimodal generation capabilities directly into their apps, products, platforms and websites.
1
16
2,048
Hi! I made Antipersona, an app that lets you use Twitter as any user you choose for 24h: antipersona.co
1
3
16
We seem to be collectively in SOTA-chasing mode where decimal point improvements make news. Time to bet fully on new benchmarks & new architectures
1
1
15
1,817
.@maxhawkins and I are organizing a meet up about speech synthesis. The first meeting is this Friday 6:30pm at @ITP_NYU. Come if you're interested in making disembodied singing versions of yourself or for any other reason. facebook.com/events/19833169…
1
15
We just raised our Series C!
Thrilled to continue building the tools of the future with the most brilliant team of technologists & artists.
So much more to invent. We're just getting started.
runwayml.com/blog/runway-rai…
1
1
15
New project: A tool that helps you change your personality to match the average personality in a location. IWantToFit.in
1
13
Together with the number of parameters, model releases should include the total number of hours spent by the researchers staring at the model's outputs. It might be more predictive of real-world performance.
12
900
Time from tweet-to-paper:
Chain-of-thought prompting: ~a year and a half
Prompt injection: ~2 months
How long until a tweet turns into an ML paper overnight?
12
For being such a critical infrastructure of modern life, video formats and codecs are frustratingly under-documented and difficult to work with. I hope we can change this soon @runwayml.
12
Finally, in case you want to help us navigate the labyrinth of possible interfaces for human-AI collaboration, we’re hiring! Send us a note: hello@runwayapp.ai
1
11
Thought-to-moving-picture latency 📉
#stablediffusion text-to-image checkpoints are now available for research purposes upon request at github.com/CompVis/stable-di…
Working on a more permissive release & inpainting checkpoints.
Soon™ coming to @runwayml for text-to-video-editing
1
11
constraints breed creativity (constraints being having to wait 5 hours to get a single P100 on the NYU HPC)
11
465
Speaking at #ttw18 in about an hour. Catch the livestream here: theorizingtheweb.org/ny/ttw1…
1
10
r/aivideo is a 2023 subculture that's destined to become 2024 culture teddit.net/r/aivideo
1
10
3,260
Within each of us, there is a vast and unique world model filled with a wide range of original characters, systems of knowledge, expressive behaviors. We just need the right prompts to surface them.
1
10
A reminder that the more legible and familiar your vision of the future is, the more ridiculous it will seem in retrospect
8
1,038
Vector Input consists of an infinite tile grid, where each tile corresponds to an input vector. We start with a random seed vector in the middle, we synthesize its neighbors by adding noise to it, each subsequent tile by averaging its neighbors and adding more noise, and so on.
1
8
NYC folks, Randomly Generated Social Interactions will be at the @creativetechwk Arts Hub all week.
7
We had to shut down the GPU server to be financially prudent grad students :) If you're interested in keeping the project running, please reach out.
👨🏻🔬 New experiment: Uncanny Road is a tool for collectively synthesizing a never-ending road with the help of Generative Adversarial Networks. Collaboration with @c_valenzuelab. uncannyroad.com/
4
7
indexing my film library with convolutional nets, based on setting: bedroom, kitchen, restaurant, highway, sky
2
5
We're experimenting with a new fully remote format for our residency program. Apply here ⬇️
📣 OPEN CALL! 📣
We are trying out a new format for our Something-in-Residence program for these times: A 3-week paid web residency in April for creatives working with machine intelligence
Apply now! 👇🏃♀️
runwayml.com/flash-residency…
4