Generative AI MLE, FOSS toolmaker, innovation catalyst @CoreWeave + @AiEleuther. bsky.app/profile/digthatdata…
Seattle, WA
- Tweets 10,191
- Following 1,945
- Followers 4,047
- Likes 11,333
Thanks for your patience everyone, first draft of colab for music video automation with #stablediffusion is published! colab.research.google.com/gi…
34
161
957
Absolutely dope #AIart transformation of @marcrebillet with @devdef's #stablediffusion #warpfusion + a few other tools (see OP for workflow and links to tutorials), by redditor AthleteEducational63: teddit.net/r/StableDiffusion…
33
165
848
88,032
yoooooo.... 3D gaussian splatting + simulated kinematics xpandora.github.io/PhysGauss…
27
91
670
73,074
OP ChatGPT corrective prompt:
"this isn't what I wanted. read my previous instructions carefully and try again. start by explaining how this most recent response did not follow my previous instructions, and then try again."
Have it explain what it did wrong as part of its reply
7
18
429
60,704
AI assisted live painting with @Krita_Painting frontend and #ComfyUI backend via LCM and this boss krita extension: github.com/Acly/krita-ai-dif…
16
113
394
234,557
With all the fear mongering about how generative models are gonna steal all the artist jobs, no one's talking about how prompt engineering has created a tangible incentive for people to study art history and learn about different artists and art styles.
17
53
370
#stablediffusion experiment with some fancy masking
12
32
357
The GLAZE technique is specifically adversarial to finetuning. Decided to take an alternate approach "style mimicry" and used PEZ to reverse engineer viable prompts. Seems to work as expected: generated prompts capture content but bork on style. great work
@ravenben and team!
1/ This might be the most important oil painting I’ve made:
Musa Victoriosa
The first painting released to the world that utilizes Glaze, a protective tech against unethical AI/ML models, developed by the @UChicago team led by @ravenben. App out now 👇
glaze.cs.uchicago.edu/downlo…
ALT Woman representing a creative muse, holds a laurel wreath in victory. She is surrounded by eagles who celebrate her accomplishments and stands on a pedestal overlooking the horizon. The colors are warm and comforting. This painting is a love letter to the art community and the accomplishments of research team creating technology for the benefit of mankind.
8
59
288
81,954
ByteDance's "Universal Source Separation (USS) with Weakly labelled Data" project deserves way more than 130 stars. The source separation quality and granularity it achieves is really spectacular. github.com/bytedance/uss
5
42
297
53,427
After a month of work: Open Source PyTTI released! Docs: pytti-tools.github.io/pytti-… Colab: colab.research.google.com/gi…
17
44
267
Aw shit, code released for Magic123 - Single Image to HQ 3D mesh!
github.com/guochengqian/Magi…
7
36
206
26,109
Happy hannukah everyone! Presenting ComfyUI-Keyframed nodes to facilitate prompt scheduling and all your other keyframing needs! github.com/dmarx/ComfyUI-Key… 1/
6
29
201
28,242
Happy Saturday! New release of my fork of @RiversHaveWings KLMC2 notebook adds:
* custom checkpoints
* init image
* keyframing for prompt weights and all supported parameters
* multi-prompt conditioning
* fancy spinning logo!
colab.research.google.com/gi…
14
30
201
21,195
Gorgeous results here from training a separate motion prior, which has the added benefit that it can be composed with any other pre-trained SD checkpoint. BYOM plug-n-play text2video!
* animatediff.github.io/
* arxiv.org/abs/2307.04725
12
32
190
26,839
Improved my #stablediffusion "animating w variations" colab using a TSP solver to re-order frames. Definitely makes the final result a lot smoother! colab.research.google.com/dr…
11
20
187
SDXL-Turbo is cool, but SD-Turbo is almost as good and even inferences faster (base model is sd-2.1 instead of sdxl) but for some reason no one is talking about it. huggingface.co/stabilityai/s…
16
30
188
34,476
Using #stablediffusion image variations to add just the tiniest bit of life to a painting
7
14
156
The "animating with variations" thing was so well received I was motivated to evolve it into an automated music video maker. This animation was made with NO editing. Scene timings inferred completely from video content (subtitles). Notebook coming soon, need to tidy up some :)
12
21
154
A lot of people seemed to be having trouble getting FiLM (an AI for video frame interpolation) working, so I put together a colab that hopefully makes it a bit easier or at least more reliable: colab.research.google.com/gi…
10
22
146
New features in #stablediffusion music video automation notebook, #VKTRS! #DreamStudio API is optional; connect google drive; robust resume; in-notebook spreadsheet for prompt editing, overriding, flagging images for regen... Short tutorial vid (6.5min): piped.video/0pfzQ-cZU0E
9
23
142
Been dropping teasers for some new comfy nodes i've been working on for the past two weeks. Planning to do a proper release later today after updating the docs. check back in a few hours!
9
8
145
14,866
PyTTI-Tools v0.10 release is live! Lots of new features, including: AudioReactivity, ViT-L/14@336px, numpy functions in weight formulae, and notebook QOL improvements! Lemme know what I broke :) colab.research.google.com/gi…
6
19
124
Ok so check it out, I think i've already figured out a stupid simple trick for improving quality even further with the new #texttovideo model: just run it through the XL model multiple times, varying the strength. Here's after running it through twice at .75 then twice at .7. 🧵
10
16
123
34,802
2y ago, I released pytti-tools. hours l8r, @EMostaque reached out to recruit me as 1 of @StabilityAI's 1st eng hires. Easily 1 of happiest, most validating days my life. The co was SNAFU, but Emad is a gr8 guy who does a lot of good for open source AI. 😢
3
5
124
19,551
Although the #stablediffusion #AIart bots don't formally support prompt weights at the moment, there are still several ways you can manipulate prompt influence in multi-component prompts. Here are a few prompt-engineering tricks I've found useful with SD: 1/n
6
23
111
text-to-video is already crazy, and it' still early days. the prompt was literally just "the godfather". this is part of a... "classifier guided random walk" where I am the classifier. still exploring, will share a travelogue soon. safe to say zeroscope has untapped potential.
7
16
108
9,933
Important benchmark here. This is serious AI research.
silverware etiquette at fancy restuarants confuses me too, #texttovideo model.
7
5
105
23,142
StabilityAI just lost the CompVis team, the people responsible for the Stable Diffusion and Stable Video architectures.
* Robin Rombach
* Andreas Blattmann
* Dominik Lorenz
Scoop: @robrombach , one of two of the original developers of Stable Diffusion has quit Stability AI. Rombach leaving the company represents the departure of the person responsible for the tech that made the company famous: sifted.eu/articles/stability…
5
8
103
25,533
hacked together a slightly more user friendly AnimateDiff notebook, lemme know if you have issues colab.research.google.com/gi…
9
20
104
9,795
in addition to being generally interesting content, this blog post is worth thumbing through just for the fantastic graphics ai.googleblog.com/2023/01/go…
2
13
100
6,154
Exciting news: I've joined @CoreWeave's ML team!! I worked closely with their lead Wes Brown while driving early DreamStudio backend development @StabilityAI and am super looking forward to working with Wes full time. Open Source AI goes brrrrrr!
20
1
99
4,708
Made a video tutorial to help folks get set up with the new zeroscope_v2_XL #texttovideo madness that's been making the rounds. Setup is the first 5 minutes, all the links you need are in the description.
piped.video/cYmGALAQKFs
7
10
95
9,843
32 frames from the 14 frame image-to-video SVD model??? Yes, you can indeed use the last frame of the output as the input condition for another round of video generation, BUT CHANGE THE SEED FIRST. Discussion and comfyui workflow here: github.com/dmarx/digthatdata…
4
10
90
17,095
#aiart made with zeroscope #texttovideo
6
7
80
4,841
omg, extra shoutout required for this amazing picture of the team via Ben's twitter banner
1
81
6,817
I'm totally exhausted from the launch, but I had a weird idea and I had to try it. Cobbled together a little experiment demonstrating how you can interpolate in "prompt-space" with #stablediffusion : colab.research.google.com/dr…
4
9
80
thanks @KaliYuga_ai for reminding me: my fork of @RiversHaveWings' #KLMC2 nb had new features I forgot to merge and share!
* Archive old work instead of deleting!
* Resume! Choose starting frame!
* Naive video upscaling (for better encoding on socials)!
colab.research.google.com/gi…
2
17
80
7,367
little animation experiment, #stablediffusion + FiLM interpolation
6
9
75
The ability to drop in a generic SD LoRA for text-to-video is quite a super power. pre-LoRA, I was getting all shutterstock-watermarked outputs a la modelscope. Add a LoRA previously trained on text-to-image: BOOM, cinematic animation.
8
11
76
8,280
Very interesting looking tool, makes it easier to interact with ComfyUI workflows via a more standard form UI, while also giving you the ability to modify the workflow graph as well. Also has the ability to wrap workflows into reusable "actions"
github.com/rvion/CushyStudio
6
9
74
7,962
madlad on the @AiEleuther discord has SDXL-Turbo at 50FPS on their hyper-optimized custom inference backend. This video is not sped up. It really generates images that fast.
Finally figured out how to speed up my #sdxlturbo
frontend! It's so fast that the only way to show the actual speed is to delete the prompt, since I can't type fast enough 😆 .. built with next.js frontend & tensorrt backend.
1
10
75
10,826
This could be the beginning of a new publication paradigm. Citation, model, data, and code provenance all living in the same space. Looking forward to the future of diff-able research!
1
14
69
30,244
zero shot intent classifier for arbitrary intent slot filling.
That's it. That's the whole thing.
4
5
74
6,753
upcoming pytti-tools release with video source animation fixed! gimme a few days to tie a bow on it :)
3
7
67
New favorite ChatGPT prompt: "please implement ... following functional design principles and satisfying the user stories listed above. respond only with functioning python code and a full coverage test suite. when the implementation is complete, end with the phrase "ship it!"
3
5
69
6,134
Dad has dementia. mom just had surgery to remove cancer, poor prognosis. just picked dog up from vet, probably has cancer. check email after getting home from vet: best friend's dad passed unexpectedly. Not sure when I stepped on to this ride, but I'd like to get off now thanks.
27
64
6,049
Yesterday was my last day at StabilityAI. Assured it had nothing to do with performance. Just dropped like a dime Friday afternoon. Neither mgr nor skip were consulted. Plan to continue making free AI tools, you can support my work here: patreon.com/DigThatData
8
8
64
10,089
succesful test using huggingface's diffusers library in the music video automation notebook! Calling it a night: api-optional notebook coming your way tomorrow, bright and early :)
8
7
63
Want to integrate #stablediffusion directly into your notebook work? Take our new SDK pacakge for a spin! Check out this colab for a simple usage demo: colab.research.google.com/gi…
Delighted to announce the public open source release of #StableDiffusion!
Please see our release post and retweet! stability.ai/blog/stable-dif…
Proud of everyone involved in releasing this tech that is the first of a series of models to activate the creative potential of humanity
2
12
61
With AI poised to impact every corner of human productivity, maybe now would be a good time to revisit that whole UBI thing. Crazy idea.
4
5
59
4,338
Here's my fork of @RiversHaveWings KLMC2 notebook which adds keyframing for prompts and a few other things colab.research.google.com/gi…
KLMC2 aging a portrait of the late queen by incrementing her age in the prompt. The sampler seems to identify the "aging trajectory" very quickly and so she already looks 100 by the time the prompt is asking for a portrait of her at 50, but still: lots of potential here!
7
16
59
12,082
95% of nascent AI startups are just a simple prompt that wraps an API call. This business model is unsustainable, hinging on temporary lay ignorance of AI accessibility. Users will leave as laypeople get accustomed to using LLMs directly.
9
6
59
6,978
... software library coded in the style of gerald sussman and bjarne startstroup, optimized for readability, LGTFM, ship it, passes all tests, no errors, stateless, trending on artstation
3
8
58
3,276
"Low background steel" is steel made prior to the detonation of the first nuclear bomb. It's important for making sensitive equipment. The same way "1945" is a cutoff year for steel, I bet "2022" will be a cutoff year for reliably human-generated training data.
3
12
74
7,196
New shoes with my #stablediffusion #aiart printed on em!!! Letting em air out a bit (gdamn this printing process stinky), but excited af. Maybe I'll take em dancing tomorrow?
6
4
52
AnimateDiff motion model for the LCM LoRA
huggingface.co/wangfuyun/Ani…
animatelcm.github.io/
1
8
51
3,857
hon, wake up: new @srush_nlp MLE exercises dropped github.com/srush/Triton-Puzz…
3
53
4,853
debugging plots and animations available now in my (more fully featured) fork of @RiversHaveWings' amazing #KLMC2 #stablediffusion notebook! visualize precisely how settings changes and timings impact the generated output colab.research.google.com/gi…
6
3
51
2,942
thanks for your patience, pytti-tools fans! the ol' girl is up and running again (and thanks @sportsracer48 for spotting the fix!) colab.research.google.com/gi…
3
4
50
experiment overplotting prompt weights and step size together. probably would be better as separate subplots (ugh). def need to at least make sure they share x-axes (there should only be one moving vertical line). making progress anyway.
2
7
49
4,411
Great news! Vet just called with post-op histopathology results: the tumor was small and they're confident they got it all! Here's Marley as a sea monster to celebrate
2
1
49
2,859
I like building complex animations parameterized with keyframes and parameter curves, but I find the notation to be burdensome. The motivation of these nodes is to facilitate parameterizing keyframed animations, but leaning on the node UX as much as possible 2/
2
3
38
11,805
i'm increasingly coming across anecdotal descriptions of people for whom making AI art is therapeutic. any art therapy researchers investigating? would love to see some numbers to go with the anecdotes. bet there're also some novel interventions waiting to be discovered here too.
11
4
47
13,351
Simple, clever (and controllable!) style transfer by just computing the KV features from a reference image.
project: curryjung.github.io/VisualSt…
paper: arxiv.org/abs/2402.12974
code: github.com/naver-ai/Visual-S…
2
49
3,563
Every little thing is just so much easier. and we're just scratching the surface of this iceberg.
2
4
49
4,647
Twitter isn't the only place where you can "follow" researchers to keep up with cutting edge AI projects: I get a ton of my news from Github directly. Here's a starter pack of users I follow whose "starring" activity is one of my most valuable news feeds. 1/n
5
6
46
6,123
looping a step size curve through a few different prompt transitions, fancy dreaming with klmc2
3
5
48
3,396
Something I dislike about the whole NFT thing is the tacit implication that an art piece has no cultural value unless it's also a commodity. A lot of influential generative artists are completely ignored by the art show/museum circuit because they don't mint.
4
3
44
2,967
Oh hey whaddayaknow, my KLMC2 fork supports custom output resolution. Also updated the demo to illustrate how to do a traversal with an accelerating step size colab.research.google.com/gi…
3
2
48
5,684
decided I wanted that last one to cook a bit longer, here's 2k frames at 60fps, klmc2 dreaming a #deepfloyd init image with #stablediffusion
4
4
45
4,276
Couldn't decide what kind of pizza to get, so I ordered a superposition of every pizza on the menu.
4
5
47
9,217
Trying out a print-on-demand service for the first time. Fuck NFTs: I'm minting shoes over here. Non-fungible textiles.
6
6
42
coming soon to my #klmc2 fork: debugging animations! here's an example plotting relative prompt weights over the generated images
2
7
44
4,468
Fun fact: remember that "multiple passes" trick i figured out for improving modelscope outputs? Yeah, it works pretty great for AnimateDiff too.
third pass:
4
3
45
4,269
yooooo how tf did I miss this?? make any sd1.5 checkpoint inferenceable in 2-8 steps by slapping on a LoRA! arxiv.org/abs/2311.05556 huggingface.co/latent-consis…
7
1
41
3,405
Let's be honest here: the vast majority of data professionals work on "AI for maximizing ad revenue"
2
1
38
cranking up the guidance scale on modelscope's text2video produces better outputs (imho) and lets you get away with fewer timesteps. This 64 frame video was generated from just 27 steps at cfg 50 (model defaults are steps=50, cfg=9, frames=16)
2
4
41
12,139
And did they share code for this awesome project? why yes, yes they did! github.com/buaacyw/GaussianE…
Excited to present our new paper "GaussianEditor", a new #3D #gaussiansplatting #editing framework! GaussianEditor provides controllable, diverse, and interactive high-resolution 3D editing, needing only 2-7 minutes.
Project page 👉: buaacyw.github.io/gaussian-e…
2
41
4,228
inference in just 4 steps github.com/luosiallen/latent…
1
6
43
3,394