prolific open-source contributor. former 100x engineer; now replaced by an LLM. early eng team @cursor_ai and @github
San Francisco, CA
- Tweets 5,669
- Following 5,970
- Followers 9,453
- Likes 21,434
Excited to share that I am working with the amazing team at @anysphere to help build @cursor_ai
19
2
240
27,256
This month I got PRs merged into cutlass, flash-attention, xformers, transformers, peft and vllm.
All relatively trivial changes, but still feels good.
6
2
88
9,559
the github.com test suite runs in under a minute with github.com/tmm1/test-queue
6
48
81
Testing Llama-2-7b-chat inference speeds on my local hardware:
Mac M1 Ultra + GGML (q4_K_M)
25.18 tokens/s
3090 + hf load_in_4bit
8.61 tokens/s
3090 + hf
11.62 tokens/s
3090 + vllm/PagedAttention
43.1 tokens/s
6
4
75
12,438
FYI:
flash-attn 2.1.0 was released, with inference fixes
xformers 0.0.21 was released, with FA2 now bundled
vLLM 0.1.4 was released, with bumped xformers dependency
2
4
51
4,981
Ruby 2.1.0 in Production: known bugs and patches gist.github.com/tmm1/8393897
1
68
47
ffmpeg's hls (http live streaming) demuxer used to speak only http/1.0
taught it to use http keepalive and pipelining, and overall streaming throughput increased by 3x
ffmpeg.org/pipermail/ffmpeg-…
5
39
Hacked up line_profiler to capture VRAM usage instead of clock time, and its already pretty useful!
In this example, training_step uses 4GB VRAM and optimizer state uses 160MB
github.com/pyutils/line_prof…
3
40
7,172
Faster ruby code with 2.1 and stackprof tmm1.net/ruby21-profiling/
3
29
34
Four years ago today, my first day and first ship at GitHub github.com/blog/793-introduc…
6
4
36
TIL CUDA_VISIBLE_DEVICES=0 won't always give you the same GPU, unless you also set CUDA_DEVICE_ORDER=PCI_BUS_ID
github.com/gpuopenanalytics/…
3
8
32
4,746
Started a blog thing to show off some of the cool stuff shipping with Ruby 2.1. First up, the method cache: tmm1.net/ruby21-method-cache…
6
26
32
Tried ctranslate2 on 3090 w/ Llama2-13B-chat
Impressive inference performance, but exllama still ahead.
34 tok/s vs 40 tok/s
And exllama loads way faster.
1
1
31
9,613
The next release of @fridadotre will support tvOS and watchOS!
github.com/frida/frida/pull/…
8
31
Initial tests show ruby 2.1 spends 70% less time in GC on the github.com rails app.
7
91
28
started exploring some LLM training tooling today
cobbled together monkey patches to get QLoRA + FlashAttention v2 working together, then optimized to eek out a bit more perf
peft:
37/3693 [05:57<9:51:49, 9.71s/it]
peft + flash-attn:
37/3693 [05:45<9:32:35, 9.40s/it]
2
28
3,430
I built a little web app that can estimate income tax owed for US taxpayers: taxsim.app #TaxTwitter
2
1
23
Here's the partial profiler hack I mentioned in my talk gist.github.com/tmm1/63dc9ec… /cc @1337807
View Profiler
View Profiler. GitHub Gist: instantly share code, notes, and snippets.
gist.github.com 15
24
i've been working on a small benchmark suite to compare various training methods and optimizations, and am finally starting to get back some results!
4
2
25
3,612
Want to store key/value metadata about your git commits?
Use git-notes + jq as a simple append-only json database
3
1
23
10,628
All this took was 20 minutes of research and a one line PR
github.com/Dao-AILab/flash-a…
Huge, yet very very niche news: Flash Attention (flash-attn) now uses Metadata 2.2, which means uv can resolve it without building the package from source.
1
21
7,433
In case you missed any of my 2.1 posts, here's a list: tmm1.net/ruby21/
16
17
got something working locally: github.com/ggerganov/llama.c…
2
20
19,167
Whoa, a docker fork that adds native macOS container support
github.com/moby/moby/compare…
macOS Containers (native, not linux) macoscontainers.org/
Very cool !
3
19
3,517
TIL PyTorch has built-in tooling to snapshot and visualize memory usage
zdevito.github.io/2022/08/16…
2
1
19
2,616
VRAM is precious and there are all sorts of tools and techniques to see how it's being used
github.com/li-js/gpu_memory_…
github.com/stonesjtu/pytorch…
stackoverflow.com/questions/…
1
1
19
9,373
success at last, thanks to help from @chpwn. hubotsiri is alive: f.cl.ly/items/0v3Y2W1z242B2e…
17
27
13
Created a jenkins job today that runs a bash script that uses a ruby script to talk to a windows server over http to run powershell cmdlets.
3
5
16
Do your future self a favor and install @atuinsh
My top commands from 2023
ALT A terminal, showing atuin stats year 2631 lvim 2462 k get 2244 git status 2191 cd 2191 fg 2047 ls 1483 ssh 1439 git checkout 1360 git add 1178 git commit Total commands: 37007 Unique commands: 9289
1
4
16
3,288
when i saw people monkey patching pytorch modules, everything clicked for me
it's a dynamic language runtime. i know this! 🧵
1
5
16
4,527
Incredibly proud of @p_reynolds and the GH systems team for turning a simple PoC into a full blown production system
1
16
attention is quadratic, but what does that mean?
processing 2k tokens requires 2.7GB of VRAM
processing 4k tokens requires 7.8GB of VRAM!
ALT benchmarking results showing number of prompt input tokens vs VRAM usage during the text generation process
3
1
17
4,130
Thanks to @charliesome's fstrings, ruby trunk now boots our rails app with 40% fewer long-lived strings on the heap. bugs.ruby-lang.org/issues/91…
3
19
16
exllama + GPTQ was fastest for me
vLLM also very competitive if you want to run without quantization
TGI for me was slow even tho it uses exllama kernels. Maybe it's better optimized for data centers (A100) vs what I have locally (3090)
1
15
1,436
them:
>convert your azure pipelines to github actions using our automated tool
me:
>run azure pipeline inside github action
4
10
1,147
Landed my first golang patch: github.com/golang/go/commit/…
I guess I'm a windows programmer now.
15
Working on a heap dumper for ruby 2.1 to simplify memory debugging: github.com/ruby/ruby/pull/42…
1
15
13
ruby 2.1.0-preview2 is out. Try the new ObjectSpace.dump_all or profiler api: github.com/tmm1/stackprof
1
8
12
If you're going to try the Llama2 chat models, make sure you're using the right prompt format!
github.com/ggerganov/llama.c…
gpus.llm-utils.org/llama-2-p…
2
14
1,752
Ruby 2.1 will ship with a simple heap dumper for debugging memory growth and reference leaks github.com/ruby/ruby/pull/42…
2
34
11
wrote a sampling callstack ruby profiler in less than 300 lines using new c-apis available in ruby 2.1: github.com/tmm1/stackprof#re…
4
13
12
however for vLLM you need github.com/vllm-project/vllm… in order to use xformers' flash-attn backend
1
12
1,615
Good resource on configuring nvidia driver on Linux
github.com/ventureoo/nvidia-…
2
1
11
1,564
IIRC, my initial implementation downloaded `git ls-files` to the browser and ran a js fuzzy text lib over it. Probably never changed much.
12
Wanted to try 13B-chat on 3090 next, but I was getting OOMs with hf/vllm (expected), and then 4bit also failed strangely (mismatched shape on mul)
Ended up using GPTQ with exllama, which loads quick (<5s) and runs fast!
7B: 43 tokens/s
13B: 39 tokens/s
github.com/turboderp/exllama
1
1
12
2,431
Helped a friend put together a build for local ML dev:
Amazon:
NUC12WSHi5 = $489
64GB DDR4 = $105
2TB NVMe = $89
PCIe power Y-splitters = $16
OWC TB4 Hub = $129
Facebook:
Sonnet 750 = $175
Razer Core X = $300
Mantiz Saturn Pro II = $250
3x 3090 = $2100
Total: $1553 + $2100
2
11
1,757
lol TIL python lets you alter your caller's local variable namespace?
inspect.currentframe().f_back.f_locals.pop
stackoverflow.com/a/70606157
4
1
10
6,810
Slides from my @demuxed talk this afternoon: speakerdeck.com/tmm1/video-d…
2
10
TIL the prlimit64 syscall on Linux lets you change resource limits (including the open file limit) on any running process.
1
8
10
good overview of security and sandbox implementation in iOS: media.blackhat.com/bh-us-11/… semantiscope.com/research/BH…
4
4
10
great presentation about writing good ruby code // RT @jeremyevans0: Finally got slides for my #mwrc presentation up http://is.gd/ayPcY
9
really enjoyed slides from @idangazit's Design for Developers talk http://scr.bi/e3eDdN
10