Ever wondered what neural networks are and how they work? Systems like ChatGPT use neural networks to work as well as they do. Neural networks are composed of "layers" of neurons, layers with different functions; connections between layers called "weights"; and mathematical functions called "activation functions". If you’re interested in learning about these systems, check the comments. Ultimately, the neural network structure of the model serves to visually demonstrate that it is, in fact, a complex mathematical equation. When companies release the model's weights, they are releasing a key component needed to run the model's complete equation. Without the weights, the equation is incomplete. For the math-minded: the weights of a model are the learned numbers (they are variables during training) that are then used as constants in the mathematical functions that make up the model. Neural networks are ultimately just one big, hyper-complex mathematical function, and when a model is trained, it learns the constants associated with the high-dimensional input.

Elon Musk is launching a school for children & it launches next month. The school is called "Ad Astra" which means "to the stars" in Latin - not surprising given this is Elon Musk. it's going to be a Montessori-style school, which means it's based on hands-on learning & collaboration & project-based learning, focused on STEM. The school boasts a 40-acre plot right outside Austin & it will take 50 children aged 3-9. This is apparently just the beginning of Elon's foray into education.

Here’s a video showing how to fine-tune Mixtral, Mistral's 8x7B Mixture of Experts (MoE) which outperforms Llama2 70B! The video walkthrough is easy-to-follow and uses QLoRA so you don’t need A100s YT link below 🤙

🫶 Important wisdom I think of #3 often Thank you @karpathy

I fine-tuned Mistral 7b on my personal journal entries and it’s hilarious 😂 Here’s how you can fine-tune it on your own data— guide is below 👇🤙🏄‍♂️

Exciting news! @karpathy is known to be a legend in AI. Former Sr. Director of AI at Tesla, Stanford alum and professor, and a founding member of OpenAI. He is also an exceptional teacher, capable of explaining complex concepts in clear terms. The world needs his videos, but as you might imagine, he's quite busy. He just released another video, an intro to LLMs. We're all very excited. Link below. 🥳🎉🎊🎓

BioMistral is the new open-source model for medical domains. It has a completely free license, without concern for royalties. Use it yourself - with no-code needed - using the guide below. We are entering an era of domain-specialized, free models.

ChatGPT Plus paused signups, so we made it easy to chat with open source models. Zephyr 7B Beta outperforms GPT 3.5 + Llama 2 70B and all 7B models Run it with the template - link below 🤙

The transformer architecture, explained. You may have heard about the transformer architecture: it resulted in momentous improvements in large language models (LLMs) and thus helped enable systems like ChatGPT. It also produced “emergent behavior” - capabilities of an AI model

New video walkthrough tutorial! Chat with your PDFs using OCR and Amazon's 32K context-length Mistral Lite. We use pd3f, an open-source optical character recognition (OCR) tool. Enjoy! 👋🤙🤗

Fine-tune Llama 2 - Video Walkthrough Step through pre-filled Jupyter Notebooks to fine-tune Llama 2 on either a Hugging Face dataset or your own dataset YT link below 🤙 Happy Friday

Fine-tune Mixtral 8x7B, Mistral's Mixture of Experts (MoE), on your own dataset. Video walk-through and Jupyter Notebook out now, links below 🤙

Hi, I’m Harper, and I share about AI/ML! 🤙

Mistral 7B outperforms Llama 13B on all tested benchmarks Made a guide showing how to fine-tune it cost-effectively using QLoRA Jupyter notebook linked below 🤙

🤭🤭🤭 Can confirm, new guide to fine-tune Mistral 7B on your own data shared sooooon!!!!! Honestly this was SO fun, I love my Mistral-as-angsty-teen-and-adult-Harper model 💓 #besties

Apple's AI Reasoning Model Paper Explained Apple’s disruptive new AI reasoning model research paper, “The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models” explained, and what it means for AGI.

One of my most-asked questions is, Should I still learn to code? Get a computer science degree, etc.? Here’s my answer. Let me know what you think in the comments.

Fine-tune Phi-2 - Video Walkthrough Microsoft's 2.7B model demonstrated nearly state-of-the-art performance among models with < 13B parameters. YT link below; includes notebooks to follow along

piped.video/watch?v=zjkBMFhN… Thank you @karpathy !!

BioMistral Fine-tune Guide is Out! BioMistral, the new open-source model for medical domains, has a free license without concern for royalties. Fine-tune it for your use case with the guide below. P.S. Are you interested in using Google's new Gemma models? Let me know...

On this day, 10 years ago, I accepted my admission to Stanford. Stanford (and the West Coast) would change my life forever, and - I really believe - for the better. Coincidentally, also on this day but *6* years ago, I was admitted to the computer science AI Master’s program at Stanford, where I would continue my undergraduate work and receive an additional Master of Science. This photo is not from then; it’s actually from recently, but I think it reflects about how happy I also was back then. I felt some of the most profound joy on that day 10 years ago that I’ve ever experienced, and I’m grateful for the journey that life has taken me on since then. And I’m grateful to be feeling that happy again recently, and that it was captured by the legend @blabacphoto. When I was admitted on this day in 2018, my senior year, to the Master’s program, I wrote up a reflection on my time at Stanford doing computer science and AI. I talked about my struggles and failures and rejections. If you’d like to read it, you can check out the link below. Thanks for all your love and support. I love connecting and learning with you.

10 DAYS OF AI BASICS: Day 1

Still think AI “wrapper” companies are a joke? @a16z general partner @martin_casado said, “‘GPT wrapper’ was this derogatory term… we’ve come to the conclusion thats that’s not even a thing… when someone writes software on the cloud, you don’t call it a cloud wrapper.”

Instant, podcast-style research papers with AI Have you tried NotebookLM’s other features? Is this something you’d use? More broadly, how do you see AI enhancing your ability to learn, grow, and create or produce, rather than holding you or others back?

The AI notebooks repo has reached 1K stars! From fine-tuning guides for Mistral, Llama 2, Mixtral, Phi-2, & BioMistral, to guides for AUTOMATIC1111, ControlNet, Oobabooga, to chatbot guides like using OCR on your PDFs... So glad the repo has provided value!! @brevdev

Amazon has entered the open source arena! 🏇🔥 Amazon's Mistral allows for 32K tokens, which means that depending on your use case, you won't need to fine-tune or use RAG. and it's open source. free.

Someone in our Discord asked when it's better to fine-tune an LLM vs. use RAG The answer? It depends. Are you trying to emulate data or reference it? For example, fine-tuning a model on your user data is useful if you want to generate fake user data. In contrast, if you want to understand what your users are doing, then you'll want RAG. One caveat: you can overfit the model to have it know the data it's trained on really well - for example, I had my journal model overfit, so it used my friends' real names and talked about actual events from my life. An analogy: fine-tuning a model is like having Person A learn mannerisms/details/etc. from Person B by spending time around them, and RAG is like having Person A literally search Person B's journal to answer questions about life events and point to specific places in the journal

You may have heard rumors that the OpenAI drama was caused by Q* bringing about AGI. What you may not know is that Q-Learning (and Q*) has been around for a while! At Stanford I TA'ed a class covering Q-Learning and Q* - Decision Making Under Uncertainty (CS238/AA228). Here are some slides from 2017. I can post an explanation if people are interested!

Just made a video of me fine-tuning Mistral 7B using the notebook released last week Link below to follow along!! 👇🤗🤙✨

In SF for TechCrunch Disrupt!

How Much Does It Cost to Train a Large Language Model? A Guide Machine learning is affecting every sector, and no one seems to have a clear idea about how much it costs to train a specialized LLM. This week at OpenAI Dev Day 2023, the company announced their model-building service for $2-3M minimum. This is a steep price to pay for a specialized model, and many are wondering, is it necessary? The question of how much it costs to train an LLM is a really hard one, and while there’s not a straightforward, plug-and-chug cost calculation, the answer mainly depends on two factors: compute requirements and how long it takes to train. To help provide clarity on how to estimate the cost of training an LLM, I’ve compiled a structured overview of the different levers that affect model training time and compute requirements. Note that this article does not include costs of: - Development and operation (eng salaries, debugging, IDEs, version control systems, tooling to monitor model performance, infrastructure set-up (try Brev lol) using more optimized ML libraries / APIs (decreasing cost) - Code licensing / legal considerations - Data privacy/security & regulatory compliance - Model bias & fairness assessments / ethical reviews - Adversarial training (to protect against adversarial attacks) & other security measures - Deployment in a production environment The four main variables to consider when determining compute requirements and training time are model architecture, training dynamics, and methods for optimizing training performance. First, however, we should learn a bit about the hardware these models fit on, so we understand the context of where these variables fit. 1. Hardware Costs This refers to access to GPUs and their associated cost, and GPU memory tends to the bottleneck. This is how much “stuff” (model, parameters, etc.) the GPU is able to hold in memory at one time. Something we’ve noticed is that most people think they need an expensive, highly elusive A100 or H100 with 40GB or 80GB of GPU memory. However, something smaller, cheaper, and more available may suffice. I’ve released a few guides on fine-tuning (Mistral on HF dataset, Mistral on own dataset, Llama on own dataset). In these guides, I used QLoRA with 4-bit quantization and LoRA on all linear layers, reducing the trainable parameters by LoRA 98%. As a result, I was able to train these models on a single A10G (24GB of GPU Memory, and only $1/hr on Brev, which provides cloud GPUs without vendor lock-in across cloud providers, like AWS, GCP, and Lambda Labs). Training on my own dataset took about 10 minutes for 500 iterations over 200 samples, and training on the HF dataset took about an hour for 6,000 samples and 1000 iterations. These models would likely not be production-grade; I am just providing these values as base references. Cloud provider costs and the choice between spot and reserved instances are direct cost factors. If using cloud GPUs, different providers and regions can have vastly different pricing. Spot instances are cheaper but less reliable as you may lose them while training, while reserved instances cost more but ensure availability. 2. Model Architecture a. Size and Structure The depth (number of layers), width (neurons per layer), and the total number of parameters affect both GPU memory requirements and training time. A model with more and/or wider layers has the capacity to learn more complex features, but at the expense of increased computational demand. Increasing the total number of parameters to train increases the estimated time to train and the GPU memory requirements. Techniques like low-rank matrix factorization (e.g., LoRA) and sparsity, where tensors are pruned to have a high number of 0 values, can reduce the number of trainable parameters and mitigate these costs, but they require careful tuning. Sparsity is often done in transformer attention mechanisms (see below) or in weights (as in block-sparse models). b. Attention Mechanisms Transformers leverage self-attention mechanisms, with multiple heads attending to different sequence parts, enhancing learning at the cost of increased computation. The traditional Transformer attention style compares every token in the context window with every other token, leading to memory requirements that are quadratic in the size of the context window, O(n^2). Sparse attention models offer a compromise by focusing on a subset of positions, for example with local (nearby) attention, thereby reducing computational load, often down to O(n • sqrt(n)). c. Efficiency Optimizations Choices of activation functions and gating mechanisms can impact computational intensity and training time. Different activation functions have varying levels of mathematical complexity; ReLU, for example, is less complex than sigmoid or tanh. Additionally, parameter sharing, for example weight sharing across layers, can reduce the number of unique parameters and hence memory requirements. 3. Training Dynamics a. Learning Rate and Batch Size Learning rate and batch size significantly influence the model's training speed and stability. The learning rate of a model affects the step size it takes in the opposite direction of the gradient (i.e. the direction towards minimizing the cost or loss function). This is called gradient descent. The batch size is the number of samples processed before the model’s parameters are updated. It is true that the larger your batch, the more memory you need; it scales linearly with the size of the batch. However, a larger batch size can lead to faster convergence because at each step, you get better estimates of the true gradient. One subtlety to consider: Even if you had a terabyte of GPU memory, you still may not want to use the largest batch size possible. Downsampling (i.e. using a smaller batch size than the total number of training samples) introduces noise into the gradient, which can help you avoid local minima. That’s why it’s called stochastic gradient descent: the stochasticity refers to how much you’re downsampling from your training set in each batch. The learning rate's size (magnitude) and schedule (rate of change over training) can affect the speed and stability of convergence. A higher learning rate means the model takes bigger steps during gradient descent. While this can speed up convergence, it can also lead to overshooting minima and potentially unstable training. Conversely, a learning rate that is too small can slow down convergence (as getting to a minimum takes longer), and the model may get stuck in local minima. See the drawing below for an example of local vs. global minima. In simple terms, a local minimum that is not equal to the global minimum is a location on the graph where it seems like the optimal loss has been found, but we had just gone a little further - up a hill and dealing with some worse performance to get there - we could have found a better place in the graph. b. Precision and Quantization The precision of calculations, like FP16 versus FP32 - using 16 bits to represent each floating point versus 32 - and techniques such as quantization balance memory usage with performance trade-offs. Using half-precision (FP16) instead of single-precision (FP32) floating points cuts the tensor sizes in half, which can save memory and speed up training by enabling faster operations and more parallelization. However, this comes with a trade-off in precision, which can lead to potential numerical errors, like overflow/underflow errors, as fewer bits can’t represent as large or as small numbers. It can also reduce accuracy, but if not too extreme, it can serve as a form of regularization, reducing overfitting and allowing the model to actually perform better on the held-out dataset. Another technique is to use mixed precision training, where some floating points are FP16 and some are FP32. Determining which matrices should be represented as FP16 vs. FP32 may take some experimentation, however, which is also a cost consideration. Quantization is another technique that maps high-precision floating points to lower-precision values, usually 8- or even 4-bit fixed-point representation integers. This reduce tensor sizes by 75% or even 87.5%, but usually results in a significant reduction in model accuracy; as mentioned before, though, it may actually help the model generalize better, so experimentation may be worthwhile. c. Hyperparameter Sweeps Hyperparameters are external configuration variables for machine learning models, i.e. they aren’t learned by the model itself, like weights are. Hyperparameters are basically all the variables we discussed here: learning rate, model architecture like number of neurons or layers, attention mechanisms, etc. Hyperparameter sweeps are when experiments are run training different models with combinations of various hyperparameter settings, and they enable a model to find the best possible combinations of hyperparameter values for its specific dataset and task. However, it is computationally expensive, as you must train many models to find the best configuration. d. Checkpointing/Early Stopping Frequent model state saving (checkpointing) can increase disk usage but provides more rollback points; if a model overfits or performs better at an earlier state in training, you can have those weights saved at a checkpoint and load that model. Early stopping is a method where one stops model training after it ceases to improve on the held out validations set. This can save training time. 4. Optimizing Training Performance a. Base Model State Starting with a pre-trained model, especially one that is trained in a task similar to the new task being trained, can significantly reduce training time. If the initial weights are closer to the optimal solution’s weights, training can be faster. Building a model from scratch - i.e. with randomized initial weight matrices or similar - takes significantly more compute and is usually not advised. b. Parallelism and Distributed Training Parallel computing is usually done with one computer that has multiple processors, which execute multiple tasks simultaneously for increased efficiency. Distributed computing involves several machines (that can be physically distant) working on divided tasks and then combining their results. Usually these two techniques are used together. Parallelism can speed up training but adds complexity and compute requirements. There are various parallelization methods, like pipeline model parallelization, where models are split into different stages and distributed across GPUs, and data parallelization, where the dataset is divided across GPUs. Distributed training can be more efficient but requires more compute resources and adds complexity. c. Data Considerations How quickly the training data can be fed from storage into the model can affect training time. Some variables to consider: - Where is the GPU located? Transferring your own data to cloud machines in more remote regions may take longer - Machine I/O bandwidth affects time to transfer between storage and GPU - Data caching, pre-fetching, and parallel loading on the GPU can decrease this time Additionally, more complex data might take the model longer to learn the patterns, i.e. loss convergence time may increase. The relevance and quality of training data also have a profound effect on training efficiency. Preprocessing and augmentation can improve outcomes but may increase the computational overhead. 5. Conclusion I hope this helps to understand the complexities behind calculating how much it costs to fine-tune or train an LLM. There’s no one-size-fits-all answer or plug-and-chug equation; the main takeaway I’d like you to have is that there’s a lot of experimentation to find what works best for you and your use case, but that’s part of the fun of ML. So try things, expect a lot of it to not work, but by doing so, you’ll see what gets you the best results. Ultimately, the cost of training LLMs like those offered by OpenAI does seem steep. For many, fine-tuning smaller models and maintaining control over proprietary data might be a more viable solution.

POV: You taught an AI to tweet exactly like you & you can do it too. Here’s how 👇 I fine-tuned a model on my tweets, so now it creates tweets like me! Check out the side-by-side - the standard model compared to the one fine-tuned on my posts. It’s awesome, and ‼️you can do it too‼️ Note that it also works for other types of writing samples — poems, journal entries, scripts, essays… you name it. Watch til the end for the $10 off code 🤗 Fine-tuning is when you actually tweak the parameters of the model - you’re rewiring the model’s brain, rather than simply feeding data into the prompt. This is a great first fine-tuning project that involves minimal code. Nebius’s fine-tuning web interface makes it really user-friendly and straightforward! You can then use the final model for whatever you’d like! You can play with it on the “playground,” or even query it with code to build it into your apps! Also, if you have another dataset and are curious about whether you should fine-tune, or just how to go about it, let me know about your specific use case in the comments! I’d be happy to weigh in and help you get started. 🤗⬇️ There is no better time than now to learn to vibe code. The barrier to entry is lower than it has ever been. Try this out! 🔐 DATA PRIVACY: Also, something I LOVE about Nebius AI Studio is their consideration of your data. Check my post in the comments below for their answers to my data questions - do they store your prompts, who can access fine-tuning data, etc… 🔐 Nebius paid me to make this but not to post on X… I just think it’s fun so sharing here!

What's Q*?! Reinforcement Learning, Q-Learning, Model-Based vs. Model-Free Methods, & Temporal Difference Learning This Brev Concepts video covers the basics, is accessible to all backgrounds, and includes a little math for those interested. YouTube link & timestamps below 🥳👇

Meta’s $14.3B investment in Scale AI ExplAIned Scale AI: what it is and why it’s crucial for AI (and AGI)

Here’s why the ex-CTO of OpenAI’s company, and first product, is a big deal. Thinking Machines aims to democratize access to AI. AI currently has the capacity to transform our world. IMPORTANT: I’m NOT just talking large language models (LLMs, ie models like ChatGPT) or image generation models; I’m talking about AI in general, ie all types of optimization over numbers and applying intelligence to all domains. I’ve felt that many of these major tech companies have been missing the mark by investing enormous amounts of money and talent in building something that could ultimately replace humans — AGI — when the technology in its current form can actually improve human life and benefit every area of society. From innovation in health, energy, science, resource allocation, food development, etc… The list is endless if we can just get the tech out to people of all industries. That is what Thinking Machines aims to do, and their first product Tinker allows even those without a machine learning background to create customized open-source models for their use cases. I have a number of videos explaining what open-source models are and why they’re important, but I will review the topic again in my upcoming video explaining Tinker. Follow and/or turn on post notifications to see this. Fine-tuning open source models is a topic that is close to my heart, since it is how I got started teaching AI online :)

wake up babe new RLAIF (reinforcement learning from ai feedback) model just dropped starling.cs.berkeley.edu/ jk it was like 5 days ago but i missed it

!!! OK!! Thank you @pmarca @a16z @networkdotpress Techno-Optimist Manifesto is out today!

AI Watermarking by Google: How SynthID Works & Why It's Not Enough (in under 5 mins) Google DeepMind's new invisible AI watermarking technology, SynthID, adds watermarks to AI-generated content so that you can tell if a piece of content was generated by AI.

Open Source vs. Open Weights & my view on it Made this a while ago when Llama 3.3 came out & never released it - but figured it is relevant as we talk about DeepSeek & etc.

Why AI is the Next Electricity

AI x Nuclear Electricity, Day 5: Why Nuclear? Welcome to Day 5 of AI x Nuclear Electricity where we discuss why nuclear might be the solution to AI energy needs! A couple of things you should know: ⚡️ The capacity factor for nuclear electricity in the US is 93%! Compare that to solar’s capacity factor of just 24%. We explain what that means. ⚡️ Nuclear power doesn’t require a lot of land to make massive amounts of electricity. Stay tuned for Day 6 where we go deeper into nuclear electricity and why it’s a great solution for AI energy demand.

Generate AI Images with the most popular open source Stable Diffusion Web UI • Video Tutorial Generate your own AI images using AUTOMATIC1111. No Midjourney or Dall•E needed, and no coding needed. Watch here on 𝕏, or YT link below.

Tutorial: How to run LLMs locally on your laptop with @ollama !

I was taught by an expert that this is the only targeted ab exercise you need, and that it's the best one - crunches etc are bad for your pelvic floor I can attest that it is incredible Start with bringing your knees up to 90 degrees Then as you get stronger, pike to 90 degrees Then bring all the way up like I'm doing 5 sets of 5

Mistral’s Mixture of Experts (MoE) model dropped on Friday, so we’re dropping everything to get you a guide + GPUs to fine-tune it ASAP Stay tuned 🤙

10 Days of AI x Nuclear Electricity, Day 1: AI’s Energy Needs

AI (LLMs) vs. Google Search They are not the same, and knowing the difference is crucial

Meta’s release of Llama 3.3 is a big deal, and here’s why. Plus, you’ll soon be able to chat with it directly from Meta AI inside Instagram. Did you know that Meta was contributing so much to open source, even though it’s not nearly as lucrative?

These AI avatars are arguably the best in the world. Which do you think is actually me: A or B? Please answer the poll below. I’m curious what percentage are fooled! Was in NYC to interview @synthesiaIO CEO/cofounder @vriparbelli and had fun making this avatar. — 👋 Hi, I’m Harper, AI educator, engineer and advisor. I have two degrees in Computer Science specializing in Artificial Intelligence from Stanford, experience teaching PhD-level AI courses there, and 4+ years building AI at Facebook / Meta

Wake up babe new SOTA quantization method just dropped!!!! TEQ (Trainable Equivalent Transformation) preserves FP32 precision of the model output while using low-precision quantization. Training requires only 1K steps and < 1% of the original model’s trainable parameters. Results are on par with state-of-the-art (SOTA) methods on typical LLMs. Paper below 👇

I had such a blast talking AI for kids on @tbpn . You guys are rad - thanks for having me!

yay here: github.com/brevdev/notebooks…

Wanna deploy your Mistral 7B model once you finish fine-tuning it? I made a super-simple guide to deploy any fine-tuned model on @replicate 🔥 Video tomorrow 🤙 Notebook below 👇

We’re collaborating to talk about a super important but often overlooked issue… AI’s extraordinary energy needs and how we can meet them without burning more fossil fuels. Stay tuned for our 10 Days of AI x Nuclear Electricity ⚡️ @isabelleboemeke @isodope

How QLoRA Lets You Fine-Tune Models that have Billions of Parameters on a Relatively Small GPU QLoRA is a fine-tuning method that combines Quantization and Low-Rank Adapters (LoRA). QLoRA is revolutionary in that it democratizes fine-tuning: it enables one to fine-tune massive models with billions of parameters on relatively small, highly available GPUs. • Quantization in the context of deep learning is the process of reducing the numerical precision of a model's tensors, making the model more compact and the operations faster in execution. This is by nature lossy and usually has some negative effect on accuracy. • LoRA, which stands for "Low Rank Adaptation," is a method designed to fine-tune large pre-trained language models more efficiently by reducing the number of trainable parameters. First, some definitions. A low-rank approximation of a matrix aims to approximate the original matrix as closely as possible, but with a lower rank. The rank of a matrix is a value that gives you an idea of the matrix’s dimensionality. A lower-rank matrix reduces computational complexity, and thus increases efficiency of matrix multiplications. Low-rank decomposition refers to the process of effectively approximating a matrix A by deriving low-rank approximations of A. Singular Value Decomposition (SVD) is a common method for low-rank decomposition. More details for the math people; feel free to skip this. The rank of a matrix is the linear space spanned by its rows or columns; to be precise, it is the maximal number of linearly independent columns (or rows) in a matrix. Low-rank decomposition approximates a given matrix A with lower-rank matrices by a product of two matrices U and V, which have reduced dimensions related to a lower rank than the original rank of A. Now, onto LoRA. Fine-tuning usually involves updating the entire set of parameters in the model, which can be computationally expensive and time-consuming for large language models. LoRA makes this process more efficient by creating and updating low-rank approximations of the original weight matrices (called update matrices) which are formed using low-rank decomposition on the original weight matrix. Only these matrices are updated during fine-tuning - the original model weights remain the same - and thus LoRA’s total number of trainable parameters is equal to the size of the low-rank update matrices. Since the low-rank matrices are being updated instead of all of the larger matrices with far more parameters, we can do this on a smaller, cheaper GPU. Here's a simplified explanation of how LoRA works: 1⃣ Initial Model: Start with a large pre-trained model (e.g., Llama 2, Mistral, etc.). 2⃣ Low-Rank Matrix: Introduce low-rank approximations for the matrices that will be used to adapt the model for the specific task at hand. Low-rank matrices (adapters) are typically formed for all linear layers of the model, but this can vary based on the model architecture and the task. 3⃣ Transform Layers: Instead of directly modifying the original weights of the model, LoRA applies a transformation using the low-rank matrices to the outputs of affected layers. 4⃣ Fine-tuning: During the fine-tuning process, only the parameters in the low-rank matrices are updated. The rest of the model's parameters are kept fixed. Again, updating only low-rank matrices allows for fine-tuning on smaller, cheaper GPUs. 5⃣ Prediction: For making predictions, the adapted layers are used in conjunction with the original pre-trained model. The low-rank adapted layers act as a kind of "add-on" to the existing architecture, adjusting its behavior for the specific task. The benefits of LoRA include: 🌟 Efficiency: Because it only updates a small subset of parameters, fine-tuning is faster and requires less computational power. According to the LoRA paper, it can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times. 🌟 Specialization: The low-rank adaptation allows the model to specialize in a particular task without a complete overhaul of the original weights. 🌟 Preservation: By keeping the bulk of the model fixed, LoRA helps preserve the generalization capabilities of the original pre-trained model while still enabling task-specific adaptations. ⭐️⭐️⭐️ Quantization + LoRA = QLoRA ➡️ In this method, the original model's parameters are first quantized to lower-bit values based on a user-defined quantization configuration. This makes the model more compact. Subsequently, LoRA is applied to the model's layers to further optimize for the specific task. This combination in QLoRA allows for fine-tuning on significantly less computational power, which essentially democratizes the ability to fine-tune models. TL;DR QLoRA, a method which combines Quantization and Low-Rank Adaptation (LoRA), presents a groundbreaking approach to fine-tuning large pre-trained models. By applying quantization, it efficiently compresses the original model, and through LoRA, it drastically reduces the number of trainable parameters. This synergistic combination democratizes the fine-tuning process, making it feasible to perform on smaller, more accessible GPUs. By democratizing access to sophisticated fine-tuning methods, QLoRA stands as a significant advancement in the field of machine learning, promising a more inclusive future for both researchers and practitioners alike. Try it Yourself! To see QLoRA in action and fine-tune a model yourself, you can follow the guide linked in the comments below. It takes you through everything step-by-step.

Here’s how you can use the fastest AI in the world (for LLM inference) for free to write a book about whatever you want. 📖💭🪄 @GroqInc is amazing

🎄🎅INTRODUCING... Brev 12 Days of Christmas! 🎅🎄 It's an advent calendar that's kinda like a launch week. New release every day til Christmas! Day 1⃣: Fine-tune Mixtral 8x7B using QLoRA on a 4xT4! Guide below 🤙🎅❤️‍🔥 Another drop tomorrow

Recorded a pod today with @Liv_Boeree about AI and we covered many topics including…. intuition Super fun. Excited for it to be out.

Day 4⃣: Fine-tune Phi-2! 2 guides to fine-tune Microsoft's 2.7B Phi-2, which has a nearly SOTA performance among under 13B-parameter models Got this one out ASAP after hearing the requests! Hope it's helpful! 🎅 link to the guides below // Day 5 tomorrow 🎁

Just read “Blending is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM”. TL;DR: Blending, or randomly selecting from just 3 different 6B/13B parameter models at each generation of a chat response, conditioning on chat history, outperformed 175B+ parameter ChatGPT. Pretty amazing. Not only do Blended chat AIs have higher user retention, indicating that they are more engaging/useful, but their inference costs are equivalent to a single 6B/13B system. Also, this is just from randomly-selected models. Imagine if there were an intelligent system to select the best model based on the type of question. We know that optimizing a model for one metric, like mathematical reasoning, usually reduces its performance on other metrics, for example reading comprehension. Why don’t we just blend thousands of specialized models? The future looks bright.

Scammers and con artists are nothing new, but they are now using AI to prey on innocent people in ways that are increasingly difficult to detect. I’ve teamed up with poker champion and risk expert @Liv_Boeree to create this series to help educate and protect yourselves and your loved ones (share this with them!) in this rapidly-changing new world.

What are CPUs vs. GPUs vs. TPUs, and what are AI chips, like what NVIDIA makes? Why do we need them? Are they just a marketing ploy? A few of you have asked questions along these lines: “What does an Al chip have that separates it from a normal chip!? Isnt it a bit of marketing from companies like NVIDIA to capture investors? Also do Al systems need chips to function? Do we really need does advanced chips? And does it all work!?” Let me know in the comments if you have any questions about GPUs / AI chips, and if this video clears it all up! And as always, please use the comment section to ask more general questions about AI or request topics for us to cover.

Just recorded the first episode of our new joint podcast! What should we call it? @femalelongevity

Expect engineering to hurt.

New studio! So much AI to talk about.

Google DeepMind has released an AI watermark-injector (& detector) open source, and has placed it in text, image, video, and music-generation models. It's open source, so the hope is that other models adopt it to watermark their content as AI-generated too 🧵(1/4)

10 DAYS OF AI BASICS Day 2: DATA

AI Explained: Transfer Learning” & “Fine-tuning” explAIned - what are they, and how are they different than plain-old model training? This is a clip from my “10 Days of AI Basics, Day 4: Model Training“ long-form video posted on YouTube. It’s 45 minutes and goes into the various steps of model training. Days 1, 2 & 3 are also on YouTube and they cover AI/ML basics, data, and model architecture. Link to Day 4 below.

A short explanation of “model sizes” in AI and why you should care about them. #learnai

Ilya basically confirmed it was over AGI safety which in fairness is the mission of the nonprofit which they are legally bound to abide by but the way it happened seems.... less than ideal

The smartest, most successful women I know believe in God

Why is NVIDIA SO valuable, particularly compared to other GPU companies? NVIDIA’s Value Explained, Part I

I had the privilege to try out Claude 3.5 Sonnet before its release, and I was blown away by its capabilities. It *significantly* outperformed ChatGPT & Gemini in raw text cleanup, generated useful code, and even gave recs to improve a marketing visual. I’m excited about this.

10 Days of AI x Nuclear Energy, Day 3: the future energy projections of AI

DeepSeek Stock Market Plunge Explained Here's a quick overview of what is happening with the Chinese AI model DeepSeek, NVIDIA, and the stock market. I’ll have a long-form podcast on this soon - follow me to get notified.

10 Days of AI Basics, Day 2: DATA Types of data, collection and preprocessing strategies, validating and dealing with missing data, storage and management, etc. Plus, are vector databases necessary?

no filter, no edits, just my broken iphone 13 camera

Orrrr you fine-tune your own model with our guide & @brevdev for a few bucks an hour 😜 The future will be fine-tuning small models to teach them about your use-case, not paying $3m to hand over your proprietary data for a personalized GPT-4 And more palatable. No need for billions of tokens; your Apple notes can suffice

By popular request.... New OCR + chatbot + PDF analysis tutorial out this week!! Using all open source tools! See you soon 🤙

Instant AI Slide Decks from Notes: AI Tool Gamma This is NOT an ad. I’ve been a big fan for a while, and I want to share it with you! What AI tools do you use that help you with the tasks you typically don’t enjoy?

Claude 3.5 Sonnet is out!! It is FREE to use, extremely fast, and sets new industry benchmarks for • graduate-level reasoning (GPQA) • undergraduate-level knowledge (MMLU) • coding proficiency (HumanEval) It has state-of-the-art vision capabilities, like image and graph reasoning. I found it performs spectacularly well on everything I need it to. Plus, its creator team at @AnthropicAI is unmatched in its commitment to AI alignment. I’ll be using it in my coding tutorial coming out tomorrow! Will you be using this? Have you used Claude before? By the way, I am not affiliated with Anthropic, and I wasn’t paid to say this… but I was honored to be offered early access.

omg yessssss @hubermanlab

Chat with your PDF using all open-source tools This simple guide 👇 uses OCR + MistralLite + prompt engineering to create a chatbot for your PDFs MistralLite, Amazon's variation of Mistral, allows for 32K context tokens, which can fit about 48 pages / 24,000 words. P.S. enjoy this meme @NaderLikeLadder made of me. he has covid and is delirious

AI SCAM ALERT More of us are turning to online dating to find love, and unfortunately, this means more scammers are using AI-generated dating profiles to find victims, using techniques like catfiishing & blackmail. So how can we stay safe AND find love in the age of AI and synthetic media? I’ve teamed up with poker champion and game theory expert @Liv_Boeree to create this series on spotting AI enhanced scams to help you protect you and your loved ones. Follow to not miss an episode!

10 DAYS OF AI BASICS, DAY 4: MODEL TRAINING #demystifyai

Big announcements coming soon from your frands at @brevdev 🤙🌊🏄‍♂️

can confirm that cold emails change lives.

MIT’s ChatGPT Brain Scan Paper: AI/ML Engineer’s Opinion The implications are important, so let’s get it right. Let’s discuss in the comments. What do you think?

Want to chat with or fine-tune a model locally without sharing your data or your history? In most cases where you download an open source model locally to your device (or pseudo locally through a cloud GPU), you won’t send your chat history or your data back up to external servers or providers. Neither the master model nor its providers will have a memory of your chat history, or the outputs that it created for you. You can be the sole handler of your chats, data, and/or fine-tuned models.

What are GPUs (Graphical Processing Units) and why are they fundamental to AI?

With a guide, fine-tuning isn’t hard. The future is small individualized models, which is why I build @brevdev and build notebooks guide people through fine-tuning and other ways to use their data, open source, so they control and own the model and their data.

AI x Nuclear Electricity, Day 2: Where does AI energy come from? Hint: ✨ Data Centers ✨

How Groq is Changing AI with GPU-Alternative LPUs + Demos I had the privilege to chat with Graham Steele of Groq at the first-ever AI Engineer World's Fair in SF. The company is founded by Jonathan Ross, who began Google's TPU effort as a 20% project, where he designed and built the core architecture of the original chip. In the 20-minute chat, we cover LPUs, Groq's software-first approach to hardware, Groq's present + future, advice to beginners in AI, & more. We also go over three Groq demos - including a side-by-side demo with Meta's platform, and writing a book in seconds - to showcase just how fast Groq LPUs are. Hope you like it. Groq is definitely a company to watch.

Yesterday as my 90-year-old grandma was walking back to her car with her new Apple Watch (her first), she fell It knew right away and was ready to call emergency services. She lives alone, and we got the Watch for this reason— for its health/emergency detection— and it proved itself immediately. The kind couple who helped me lift my grandma up said the Watch had saved their own elderly mother when she fell, too I love these moments of seeing how technology is saving lives. Thank you Apple 🙏

Top researchers from OpenAI, Anthropic, Google DeepMind, Meta & more released a position paper last week calling to prioritize research into models' chain-of-thought (CoT) processes and preserving CoT monitorability. Though not perfectly indicative of models' thinking, the human-language CoT outputs are a stark contrast to the historic "black box" issue of neural networks. The researchers urge that they enable a "unique opportunity for AI safety" but warn that there is "no guarantee that the current degree of visibility will persist" going forward. More info on CoT and the position paper ↓

STANFORD AI PROFESSOR @aiprof_mykel: Al is *technically* a probabilistic machine, but the fact that it works as well as it does is mystifying - even to the Stanford Al profs. - - I had the best time teaching a lecture on Al ML at the HAI (Stanford Human-Centered Al @StanfordHAI) Advanced Al Leadership Executive Program this week! Met some exceptional people - really enjoyed the conversations.

Here's a tutorial to deploy your AI model for inference and get APIs for your app Watch til the end for fun times 🥳 YT link in comments 👇 Happy Halloween!!🎃👻

Grok 3 is really good.

A couple clarifications I’d make to this: - RAG and fine-tuning are complements, not substitutes. Combining both can often significantly improve performance. (@atroyn) - Fine-tuning allows for true learning - understanding how different concepts and variables fit together - not just emulation. I liked @connorshepherd 's suggestion: “If you're trying to create a capability that doesn't exist in the base model, finetune. if you're trying to make use of a capability that exists in the base model, but with your data, RAG.” I'll also pull out my analogy again (which of course is non-exhaustive of all fine-tuning applications): fine-tuning a model is like having Person A learn mannerisms/details/etc. from Person B by spending time around them, and RAG is like having Person A literally search Person B's journal to answer questions about life events and point to specific places in the journal.

Chain-of-Thought Prompting ExplAIned The power of this special prompting - how you can use it, how it works, and how it has contributed to a whole new wave of the world’s most advanced and reliable AI models (specifically large language models, LLMs, like ChatGPT). Some bonus info: One of the side effects of chain-of-thought prompting is the model outputs more tokens than it would have otherwise. (A “token” is a fragment of a word and when you talk to a model and it gives you a response, those are a series of tokens appended together.) This happens because the model shows its step-by-step reasoning process before giving the final answer, rather than jumping straight to the conclusion. More tokens means higher costs, since the model needs to do expensive (in terms of compute, which uses energy) calculations for every outputted token. What researchers have found is that those extra tokens are worth the cost because they significantly improve the model’s accuracy. If you need to improve your accuracy and aren’t using a reasoning model (which we will talk about in an upcoming video) try adding “think step-by-step” to the end of your prompt. This simple addition can substantially improve your performance. There are ways that you can refine this command which may further improve the performance, but start there, and we can talk more about this in upcoming videos - let me know in the comments if you’d be interested in that.

What is a LLM (Large Language Model)? Here I explain in basic terms.