Mojo 🔥 0.5.0 is released! 🚀 Even more epic updates unleashed! 😱 Checkout the highlights ⬇️ or read the full changelog here & happy weekend hacking! 👩🏼‍💻 ➡️ bit.ly/463X9eq

The first crack at llama2.🔥 is here 🚀 A Mojo 🔥 community member - Mojician - did a simple port from Python to Mojo, and shows its already 20% faster than Karpathys llama.c implementation 😱 How much faster can it go? 📈

Wow, impressive!

New blog post by Mojician 🔥 and guest contributor @tairov 💯⬇️ Aydyn discusses his journey from discovering Mojo 🔥 to implementing llama2.🔥 which has over 1.2k stars 🤩 on GitHub! 🚀 modular.com/blog/community-s…

Just in time for the holidays, we are releasing some new software today from Apple machine learning research. MLX is an efficient machine learning framework specifically designed for Apple silicon (i.e. your laptop!) Code: github.com/ml-explore/mlx Docs: ml-explore.github.io/mlx/bui…

whisper.cpp v1.5.0 github.com/ggerganov/whisper…

@tairov 🔥🔥🔥#modcon23

I think I'll need to output more debug symbols/ do you think better to read the report straight from perf

Mistral's team literally just used learned weights over data instead of programmed rules and raised 120+M. Investors what are you doing 😢 this is horrendous

New open weights LLM from @MistralAI params.json: - hidden_dim / dim = 14336/4096 => 3.5X MLP expand - n_heads / n_kv_heads = 32/8 => 4X multiquery - "moe" => mixture of experts 8X top 2 👀 Likely related code: github.com/mistralai/megablo… Oddly absent: an over-rehearsed professional release video talking about a revolution in AI. If people are wondering why there is so much AI activity right around now, it's because the biggest deep learning conference (NeurIPS) is next week.

Llama2.mojo performance on Mac is right up there with llama.cpp (!!!), and even outperforms plain C in many cases. This is insane!

Project #2: LLM Visualization So I created a web-page to visualize a small LLM, of the sort that's behind ChatGPT. Rendered in 3D, it shows all the steps to run a single token inference. (link in bio)

Are you kidding? There has never been a green pasture of this size with this low barrier to entry.

🦙 .🔥

T-minus 2 hours for @thursdai_pod live recording, and as always, if you can't make it to the live one, make sure you're subscribed on thursdai.news to receive the episode in newsletter and podcast form

Mojo 🔥 is coming to Mac 💻 very soon 😱 Here’s a little sneak peak of us testing LLama2.🔥 out of the box by @tairov. Look for this to drop in the next couple of weeks 💯🚀

Epic work, the thing I love about this is how small and clean the code is - literally reimplementing everything down to the metal instead of depending on thick layers of magic.

Q-Learning works better in Mojo🔥🚀Amazing work @tairov 💯

It's absolutely crazy how cheap the computing power is if you stay outside of the Cloud 🤯 The usage of iximiuz Labs keeps growing, so I'm upgrading my bare metal servers. And I just doubled the fleet's CPU capacity with the price going from $44 to $53 per server per month.

So excited to share what the team and I have been working on these last months! #AlphaCode 2 is powered by Gemini and performs better than 85% of competition participants in 12 contests on Codeforces! More details at goo.gle/AlphaCode2 @GoogleDeepMind

Thank you Aydyn! I love the idea of a pure Python implementation with no dependence to an external package. I started from your version and introduced numpy: github.com/chris-ch/llama2-p… ... using the standard interpreter we get to 2x slower from 10x slower compared to C version.

Claude 2.1 (200K Tokens) - Pressure Testing Long Context Recall We all love increasing context lengths - but what's performance like? Anthropic reached out with early access to Claude 2.1 so I repeated the “needle in a haystack” analysis I did on GPT-4 Here's what I found: Findings: * At 200K tokens (nearly 470 pages), Claude 2.1 was able to recall facts at some document depths * Facts at the very top and very bottom of the document were recalled with nearly 100% accuracy * Facts positioned at the top of the document were recalled with less performance than the bottom (similar to GPT-4) * Starting at ~90K tokens, performance of recall at the bottom of the document started to get increasingly worse * Performance at low context lengths was not guaranteed So what: * Prompting Engineering Matters - It’s worth tinkering with your prompt and running A/B tests to measure retrieval accuracy * No Guarantees - Your facts are not guaranteed to be retrieved. Don’t bake the assumption they will into your applications * Less context = more accuracy - This is well know, but when possible reduce the amount of context you send to the models to increase its ability to recall * Position Matters - Also well know, but facts placed at the very beginning and 2nd half of the document seem to be recalled better Why run this test?: * I’m a big fan of Anthropic! They are helping to push the bounds on LLM performance and creating powerful tools for the world * As a practitioner of LLMs, it’s important to build an intuition for how they work, where they excel and their limits * Tests like these, while not bulletproof, help showcase real world examples and get a feeling for how they work. The goal is to transfer this knowledge to productive use cases Overview of the process: * Use Paul Graham essays as ‘background’ tokens. With 218 essays it’s easy to get up to 200K tokens (repeated essays when necessary) * Place a random statement within the document at various depths. Fact used: “The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day.” * Ask Claude 2.1 to answer this question only using the context provided * Evaluate Claude 2.1s answer with GPT-4 using @langchain evals * Rinse and repeat for 35x document depths between 0% (top of document) and 100% (bottom of document) (sigmoid distribution) and 35x context lengths (1K Tokens > 200K Tokens) Next Steps To Take This Further: * For rigor, one should do a key:value retrieval step. However for relatability I did a San Francisco line within PGs essays for clarity and practical relevance * Repeat test multiple times for increased statistical significance Notes: * Amount Of Recall Matters - The model's performance is hypothesized to diminish when tasked with multiple fact retrievals or when engaging in synthetic reasoning steps * Changing your prompt, question, fact to be retrieved and background context will impact performance * The Anthropic team reached out and offered credits to repeat this test. They also offered prompt advice to maximize performance. It's important to clarify that their involvement was strictly logistical. The integrity and independence of the results were maintained, ensuring that the findings reflect my unbiased evaluation and are not influenced by their support. * This test cost ~$1,016 for API calls ($8 per million tokens)

“Animate Anyone” was released last night for making pose guide videos. Lets dive in. Paper: arxiv.org/abs/2311.17117 Project: humanaigc.github.io/animate-… 🧵1/

New YouTube video: 1hr general-audience introduction to Large Language Models piped.video/watch?v=zjkBMFhN… Based on a 30min talk I gave recently; It tries to be non-technical intro, covers mental models for LLM inference, training, finetuning, the emerging LLM OS and LLM Security.

You don’t have to train from scratch whenever developing a smaller model of an existing model family. Sharing our latest work - “Initializing Models with Larger Ones” arxiv preprint: arxiv.org/abs/2311.18823 code: github.com/OscarXZQ/weight-s…

Let's gooo! We actually had Aydyn on ThursdAI and talked about LlaMa.🔥 sub.thursdai.news/p/thursdai…