Combining_Whisper3HuggingFace_&_Claude3_4Summarization.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "97bef6c8-ea57-440a-aceb-941e6307d5fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import torch\n",
    "from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline, AutoTokenizer, AutoModelForCausalLM\n",
    "from dotenv import load_dotenv\n",
    "from anthropic import Anthropic\n",
    "\n",
    "\n",
    "# hide sensible Claude API key information by importing environment variables  from an env.file\n",
    "load_dotenv(r\"anthropic_access.env\")\n",
    "# store the token in a variable\n",
    "api_key= os.environ.get(\"ANTHROPIC_KEY\")\n",
    "\n",
    "# hide sensible HuggingFace token information by importing environment variables from an env. file\n",
    "load_dotenv(r\"huggingface.env\")\n",
    "# store the token in a variable\n",
    "token=os.environ.get(\"HF_TOKEN\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a13d5db8-7697-4b81-8863-a3aa3d34f2f6",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Hello good people! Have you ever wondered what Langchain was? Or maybe you've heard about it and you've played around with a few sections but you're not quite sure where to look next? Well in this video we're going to be covering all of the Langchain basics with the goal of getting you building and having fun as quick as possible. My name is Greg and I've been having a ton of fun building out apps in Langchain. Now, I share most of my work on Twitter, so if you wanna go check it out, links in the description, you can go follow along with me. Now, this video is gonna be based off of the new conceptual docs from Langchain. And the reason why I'm doing a video here is because it takes all of the technical pieces and abstracts them up into more theoretical, qualitative aspects of Langchain, which I think is extremely helpful for it. And in order to understand this a little bit better, I've created a companion for this video. And that is the Lane Chain Cookbook. Link's in the description. If you want to go check that out, please go and check out the GitHub and you can follow along here. I'm going to put a lot of timestamps in the description as well. There's going to be a fair amount of content in this one, so you can watch it all the way through, or if you wanna skip to a certain section, feel free to jump to that timestamp, all right? Without further ado, let's jump into it. All right, here are the new conceptual docs from Langchain. Now, the reason why these are different is because there are the Python docs, which well which is also more technical documentation however these concepts are more qualitative so you can understand what is going on in the background of these different sections here now we're going to focus on these components of langchain there's an entire section on use cases which is when you actually put these into practice and that is going to be a part two of this video so we won't jump into this today that would be a too long for us we're going to run through schema models prompts indexes index indexes, memory, chains, and agents with a working code sample for each one of those. Well, without further ado, let's jump into some code. Here we are with the Langchain Cookbook. Now, my goal is to make this a dense document with a ton of links, so you can go and self-service right into there. Links in the description description and if you want to follow along i encourage you to get this on your computer and go from it go for it from there so the goal of this doc is to provide an introductory understanding of the components and use cases of langchain in a explain like m5 way with examples and code snippets for use cases check out part two which is not made yet that is coming soon hopefully by the time you see this it will be be. A bunch of links here. We go into what is Langchain. So Langchain is going to be a framework for developing applications powered by language models. Well, Greg, OpenAI just came out with plugins. Yes, but there is a whole lot of other things that you can do with language models outside of those. And Langchain helps abstract a ton of that so that you're able to work with it more easily and intermix different pieces and customize really how you need to. So Langchain makes the complicated parts of working and building with AI models easier. It does this in two main ways. The first big way is going to be through integration. So you can bring external data such as your files, other applications, API data to your language models which is cool the other big way that it helps do this is through agency so it allows your language models to interact with its environment via decision making basically you're using the language model to help decide which action to take next and you do this when the path isn't so clear or it may be unknown and we'll get into more of that later. So why Langchain specifically? There are four big reasons why I like Langchain. The first one is going to be for the components. Langchain makes it easy to swap out abstractions and components necessary to work with language models. Basically, they've created a ton of tools that make it super simple to work with language models like ChatGPT or anything on Hungryface, how you may want. Also because it allows you to customize chains really easily. So there's a ton of out of the box support for using and customizing chains. Basically combining a series of actions together. On the qualitative side of why Langchain is awesome is because the speed is great. Almost every day I need to go and make sure that i'm on the latest branch of langchain and i go and i update it every time so the speed is awesome the other really cool part is the community so there's a ton of meetups there's a discord channel and there's a ton of events like webinars that go on throughout the week that are really awesome learning resources for us cool now again to all this, why do we need Langchain? Well, because language models can be pretty straightforward. It is text in, text out, and you may have experienced this yourself. However, once you just start developing applications, there's a ton of friction points that Langchain is going to help you develop. There's a ton of friction points that they're going to help you with, basically. Now, the last thing that I'll say about this before we jump into it is that this cookbook isn't going to cover all of the aspects of Langchain. This isn't meant to be a replacement for the documentation online. This is meant to show you a very broad overview about the capabilities that there are with my interpretation of them and my voiceover with it. And with that, I'm hoping that you can get to building an impact as quick as possible. I'm super curious to see what you build, so please let me know and I will hopefully, I would love to see it. First thing we're going to do is we're going to import our OpenAI API key. Now, I have a hidden cell here, but you're going to replace your API key right here. Just throw that in there. The first aspect of lane chain components that we're going to look at is the schema. Now, I almost didn't even include this one, but the first one is going to be text. Now, what's really cool about these language models is that text is the new programming language. Not verbatim, not per se, but we're using a lot more English language to tell language models what to do. In this case, what day comes after Friday is an example of something I may go tell a language model and it is going to respond back to me with a natural language response. Very cool. Next up is going to be chat messages. So like text, chat messages are similar, but they have different types. The first type is going to be system. And this is helpful background context that tell the AI what to do, right? Like your helpful teacher assistant bot or something. Then we have human messages. And these are messages that are intended to represent the user. And so literally user input or something that I may text from it. Then we have AI messages, and these are messages that show what the AI responded with. And the cool part about this is the AI may or may not have actually responded with it, but you can tell it that it did so that it has additional context on how to answer you. So what I'm going to do here is i'm going to import chat open ai and my three message types and then i'm going to create my chat model i'm going to do that and then i'm going to type in two messages the first system message is you are a nice ai bot that helps a user figure out what to eat in a short sentence and then a human message i like tomatoes what should i eat let me go ahead and run this. And you get an AI message back because this is what it responds with. You could try making a tomato salad with fresh basil and mozzarella cheese. Thanks, AI. That's cool. What you can also do is you can also pass more chat history and get responses from the AI. So in this case, you're a nice AI bot that helps a user figure out where to travel to in one short sentence. I'm saying, I like the beaches. Where should I go? I'm telling it that it responded to me. It didn't actually do this, but I'm telling it that it did. You should go to Nice, France. Cool. What else should I do when I'm there? And so the reason why I did this one is because you'll notice that I didn't say where I went. It's going to have to infer from the history on where I went. And it says, while in Nice. And so it picked up where I was because it gets the history of the chat messages. Now if you're making a chatbot, you could see how you could append different messages that have been back and forth. I'm not sure if that's a verb, but back and forth through the user. The next model that we're going to look at is going to be documents. So documents are important because this represents a piece of text along with associated metadata. Now, metadata is just a fancy word for things about that document. And in this case, this document or the text is held within a field called page content. So this is my document. It's full text that I've gathered from other places. Awesome. And then I'm going to pass in some metadata. So this is my document. It's full text that I've gathered from other places. Awesome. And then I'm going to pass in some metadata. And this metadata is a dictionary of key value pairs, my document ID, which is my key here, and then some random document ID here that happens to be an int. It could be whatever you want it to be. My document source, this is the langchain papers. And then my document create time is going to be some timestamp whatever you want it to be and this is going to be whatever format you want this is extremely helpful for when you're making large repositories of information and you want to be able to filter by it so instead of just going and asking langchain to look at all your documents in your database you can go ahead and filter these by certain metadata go ahead and run run this. And you can see here, I get a document object with a bunch of metadata on it from there. Cool. If those are the schemas that we work with, the next thing we're going to look at is the different models. Now, these are the ways of interacting with, well, different models. But the reason why this is important is because they're different model types. Let me just show an example here. The normal one that we're looking at is going to be the language model. And this is when text goes in and text comes out. Now, the first thing I'll do is I'll import OpenAI and I'll make my model. And you'll notice here that I changed my model in case you ever want to change your model as well. And so I'm going to pass in a regular string into this one, into my language model. What day comes after Friday? Go ahead and run this. And I get Saturday comes out the other end. But not all models are like this. You actually have chat models as well. And we looked at this in the previous example, but I didn't call it out specifically. So for this one, I'm going to import chat OpenAI. I'm going to import my messages again. I'm going to put temperature equals one, which means the model is going to get a little spicy on me. No, but really just means it's going to have more creativity. And it's a little bit more exaggerated. And so in this case, I'm going to say, you are an unhelpful AI bot that makes jokes at whatever the user says. And in this case, the user says, I would like to go to New York. How should I do this? I'm going to go ahead and run this model. You could try walking, but I don't recommend it unless you have a lot of time on your hands. Maybe try flapping your arms really hard to see if it can fly there. So as you can see, it took that system message and it understood those directions and it wasn't very helpful for me. Well, because I told it not to be very helpful. The last type of model that we're going to look at is going to be your text embedding model. The reason why this one is important is because we do a lot of similarity searches and a lot of comparing texts when working with language models. Now, in this case, OpenAI also has an AI embeddings model that we're going to use. There's a lot of embedding models out there. You can use whatever you want. I just use OpenAI because it feels like it's a standard and it's very simple right now. So I'm going to pass in my API key. I'm going to get my embeddings engine ready, and then I'm going to define a piece of text. Hi, it's time for the beach. Let me go ahead and do that text. And what I'm going to pass that text and I'm going to embed that text. So what that means is, is it's going to take this string, which is just a series of letters, and it's going to convert it into a vector. And in this case, a vector is just simply a one-dimensional array, meaning a list of numbers. And that'll be a semantic representation of that text. That's a fancy way of saying is that meaning of that text is going to be embedded in those numbers right there, which makes it really easy to compare across others as well. So I'm going to put that in a variable called text embeddings. I'm going to see how long my text embeddings is, and I'm going to get a preview of it. So you'll notice here that my text embedding length is 1536. This means that there are 1536 different numbers within that list that represent the meaning of my text. That's a lot of numbers and I'm glad I don't have to deal with them. I'm glad the computer can. So here's a sample of what those look like in case you're curious. I only show the first five here, but I put a dot, dot, dot. So you know that there are 1531 other numbers out there. Next, let's look at prompts. So prompts are gonna be the text that you send over to your language model. We've already sent some prompts over to the language model, but they've been pretty simple. In this case, we're gonna start doing more instructional prompts and passing those to our model. So again, a prompt is what we pass to our language model. I'm gonna import OpenAI. In this case, I'm using DaVinci as my model. And I'm going to say prompt equals this string. Now, I use three double quotes because, well, I think it looks fancier. No, but really, it's just easier to use, which is why I like it. In this case, I'm not doing anything fancy. And I could have passed this string right within my language model. But in this case, I made a variable for it because it's a little bit easier to understand. So today's Monday, tomorrow's Wednesday. What is wrong with the statement? The statement is incorrect. Tomorrow's Tuesday, not Wednesday. So you can see how it picked it up from there. Now, why prompts are cool is because we start to get into the prompt template world. The reason why prompt templates are important is because most of the time you're going to be dynamically generating your prompts meaning they won't just be static strings that you type out but you're actually going to be inputting tokens or inputting placeholders based off of the scenario that you're working with so in this case what i'm doing here is i'm importing my packages again in this case prompt template is going to be the new one i'm going to to do DaVinci again. Okay, great. And in this case, I'm going to create a template to start. So I really want to travel to location. You'll notice my opened and closed brackets around location, which means that this is going to be a token that I'm going to be replacing later. What should I do there? Respond in one short sentence because or else it just responds with too much. I'm going to create a prompt template. In this case, I'm going to put it in this variable prompt. My input variable is going to be location, which matches the same name that we had up here. And then the template is this, this whole thing that I had here. The final prompt is going to be prompt dot format, which means go and insert the values. I tell you, go and insert the value Rome into where it says location right here. Let's go ahead and run this so final prompt i really want to travel to rome which replaced location up above and here we have our prompt template that's finally filled out and then in terms of the output um it tells me what i should do so it took that information in with rome and respond in one short sentence it gives me this which is cool all right the next cool part that we're going to look at is the example selectors. So often when you're constructing your prompts you're going to do something called in context learning. This means that you're going to show the language model what you want it to do and one of the main ways that people do this is through examples. This could be about how to answer a customer service request or it could be how to respond to some nuanced question And in this case, I'm gonna pick examples however, we have example selectors because Say you had 10,000 different examples. You don't want to throw all those into your in your prompt They may not fit and they may not be as relevant So you want to select which ones you want and in this case? What I'm gonna do is I going to import a lot of things here, but the main star of the show is going to be the semantic similarity example selector. That's a long name for a functionality that's going to select similar examples. So I'm going to get my language model going again. I'm going to get my example prompt, and this is just a prompt template like we saw up above and then I'm going to define a list of different examples. So in this case I want to name a noun and then I want the language model to tell me where this noun is usually found. So in this case a pirate on a ship, a pilot on a plane, driver in a car, a tree, oh that's not true, a tree in the ground, or a bird in a nest so i'll go ahead and run that one and then what we're going to do is we're going to get our example selector ready so we have our similar example selector we're going to pass it the list of examples that i just defined above but then we're also going to pass it our embedding engine and the reason why we do this is because we're actually going to match examples on their semantic meaning. So not just matching them off of similar strings, but off of what they actually mean. So in this case, we're going to use the open AI embeddings, which is one of the models that has been shared by Facebook, which is really cool. And this is going to help store our embeddings. And then we're going to tell it how many we want, how many examples we want back. And then we're going to tell it how many we want, how many examples we want back. In this case, I want K equals two. Let me go ahead and run that. And then we're going to have a new prompt template here, and this is going to be the few shot prompt template, meaning the few shot part means that there's going to be a few examples in there for us. So we give it our example selector. We give it our example prompt, which we we made up above and then we're going to add on just some little strings before and after to make it easier for the model so give the location that an item is usually found in cool and then the suffix will be the input and output that we have from here based off of what the user inputs then the input variable go ahead and do that so here i'm going to say my noun is student. So based off of the noun of student, it's going to go and find me the examples up above that are most closely related to student. And we're going to use those examples. So if I were to go ahead and do this, I'm going to say print, and it's going to print me the prompt that we're actually going to use within or give to our language model. In this case, it found the driver and it found the pilot one being most similar to student, which is cool. Now, if I were to do a different one, say flower, it's going to give me the tree and the bird examples. But I'm going to stick this with student. And what I'm going to do is I'm going to take this prompt that we just made and I'm going to pass that into the language model. And all of a sudden you get classroom. The next thing we're going to look at is output parsers now that's kind of a complicated way to say we need some structured output like we want the language model to return a json object back to us why well because it makes it a heck of a lot easier to go deal with and work with on the other side there's two big concepts when we talk about output parsers. First going to be the formatting instructions piece. So this is the prompt template that is going to tell your language model how to respond back to you. And Langchain provides us some conventions to do this automatically, which is cool. And then the second thing we're going to have is going to be the parser. And so this is going to be the tool that is going to parse the output of your language model. So the language model can only return back a string, but if we want a JSON object, well, we need to go and parse that string and extract the JSON from that. Okay. So we're going to get a structured output parser, and we're going to get the response schema from there. Let's import our language model again. So we're going to have a response schema. So in this case, I just want it to be a two field json object it i'm going to have a bad string which is a poorly formatted user input string and then a good string this is your response a formatted response and so the really nice response from the from the language model there and in this case i'm going to go ahead and create my output parser which is going to read the response schema and it's going to be able to parse it for us. But we won't use that until just a second here. So first thing we're going to have is our format instructions. So on the output parser, we're going to say get format instructions and then let's print those out. In fact, I don't need to do that. I could just print this out directly right here. Cool. And so this is a piece of text that is going to be input or insert put into the prompt. The output should be a markdown code snippet formatted in the following schema, JSON, and then the two fields that I input up above, but it did the formatting for me, or at least lane chain did for him to put it in here. So let's go ahead and create a prompt template. We're going to do a placeholder variable for our format instructions. And then we're also going to do a placeholder for user input. This will be the poorly formatted string that the user is going to input. And then finally, I put your response here just to tell it. It's like, hey, I'm done telling you instructions. Give me your response. We go ahead and we get the prompt template. We have the user input. We have a partial variable of format instructions. And this will be the format instructions we had up above we have our template which is the string up above here and then we have our prompt value so this will be the actual value that is filled out with the variables i tell it and i'm going to say welcome to california with an exclamation point let's go ahead and do that one and here i print out the final prompt that is going to be sent to the llm we have user input welcome to california everything we had up above. Let's go ahead and run this. Let's see what it responds back to us. So we get a string here. It kind of looks like gobbledygook, but if we were to print this out, it'd make more sense. But before printing out, let's just go ahead and parse this. And now we can actually parse this and we get a nice JSON object back. Well, in this case, it's going to be a dict, but you can see here it's type dict. The next thing we're going to look at is different indexes. So in this case, we're going to be structuring documents in a way that language models have a better time working with them. And one of the main ways that Langchain does this is going to be through document loaders. Now, this is very similar to the OpenAI plugins that just were released. However, there's a lot of support for a lot of really cool data sources in Langchain that aren't yet supported within the plugin world. In this case, I'm going to be doing a Hacker News data loader. So all I'm doing is just passing a simple URL to this data loader. I'm going to say, hey, go get me that data. And so I'm asking, hey, how many pieces of data did you find? And in this case, it found seventy six different comments within this Hacker News post. And I asked it to print me out a sample. And here we see one of the responses by the moderator, Deng, within Hacker News. And we see the response there. We see different comments. You can go and work with these within your language model now, which is pretty cool. Another big piece of what we do a ton of is text splitting. So in your language model now, which is pretty cool. Another big piece of what we do a ton of is text splitting. So oftentimes your document, like your book or your essay or whatever, is gonna be too long for your language model. You need to split it up into chunks and text splitters will help with this. Now the reason why you do this is because if you want a single answer out of a book, it wouldn't behoove you too much to input that entire book into the prompt one because it's too long but two is because the signal to noise ratio is too much or is too little for your language model to effectively do its job it'd be a lot better if you just put in a few pieces of text into there and in order to get those few pieces of text we need to do splitting or chunking of those so in this case i'm going to do text splitting and the one that i use most often is going to be the recursive character text splitter. There's a lot of different types of text splitters depending on your use case. I encourage you to go check those out. And in this case, I'm going to pull in a Paul Graham essay, his worked essay. This one is quite long, maybe his longest actually. So if I were to read his document, I just have one big long document right now which means it's a really long piece of text but in this case what i want to do is i want to have the recursive character text splitter and i'm going to say chunk size equals 150. this means that i'm going to have a size of 150 when i end up splitting my star document there and if you want chunk overlap that means that the venn diagram of your docs is going to overlap just a little bit i encourage you to play with these variables to see which one works best for your use case normally i wouldn't do 150 i'd probably do a thousand or two thousand but for demonstration purposes i'm doing 150. go ahead and run that and so we had one document up above but after i split it i now have 606 documents all right and if i wanted to preview those, I can go ahead and preview these and see how they're nice and small. They're super small. And if I wanted to make this 50, for example, well, then my chunks will be a whole lot smaller, but let me go ahead and make that bigger. The next thing we're going to look at is going to be retrievers. Now retrievers are easy ways to combine your documents with your language models. There's going to be a lot of different types of retrievers and the most widely supported one is gonna be the vector store retriever. And it's most widely supported because we're doing so much similarity search within embeddings. Let's look at an example here. We're gonna load up a Paul Graham essay just like how we had beforehand. I'm gonna do some splitting of it and so we're gonna get a whole bunch of documents. We're gonna split the documents and then i'm going to create embeddings out of those documents and so all those little chunks we're going to create vectors out of them which is the semantic meaning of them and then i'm going to store those vectors within a document store here okay and i'm going to call that within my db there and then i'm going to say hey this retriever is going to be the db but we're going to set it as the retriever okay so it knows to go get stuff and if i were to look at this you can see here that we have our vector store retriever that's output right here okay we're going to take our retriever and i'm going to say hey go get me the relevant documents what types of things did the author want to build now in the background what it's doing here is it's taking the string and it's converting it to a vector it's taking that vector and it's going to go compare it to the vector store that you have and find the similar documents that come from there so what I'm going to do here is I'm just going to print out this is a one-liner kind of complicated one just to print out the preview of the documents that we have here I'm just going to have it print out the first two docs is not defined great let's go ahead and run those so all of a sudden these are the previews of the docs that it found there what i wanted was to not just build things but build things that would last so you can see here that out of all those documents that i found i found the two that were most similar to what i was looking for which is really cool i wanted to build things nice next let's look at vector stores so we briefly just talked about vector stores right before this but to go into it a little bit further think of a vector store really the way that i think about it is a table with rows with your embeddings and associated metadata that comes with it an example of it is right here two main players in the space right now are going to be pinecone and weviate however if you want to you can go check out open ai's retriever documentation and they list a whole bunch of other ones that you may find awesome for you. Okay, so let's go ahead and look at these. Again, we're gonna import our models. We got our embeddings. Okay, cool. Now with these embeddings, I'm gonna look at that. And based off of how I split my document up above with a thousand chunks or a thousand as a chunk size, we get 78 documents out of paul graham's worked essay okay what i'm going to do is i'm going to create those as embeddings and i'm going to get my embeddings list from there and i'm going to let's look at the length of the embedding list i have 78 embeddings reason why is because i have one vector for each one of my documents so all right makes sense and here's a sample of one so here's an example of what the embedding would look like. It's a numerical representation of the semantic meaning of your document there. So your vector store is going to be storing your embeddings, and it makes them easily searchable. So in this case, it is going to take my embedding, and it's going to store it like a database. The next topic I want to look at is going to be memory. So this is going to be memory so this is going to be how you help your language models remember things the most common use case for this is going to be your chat history so if you're making a chat bot then you can tell it the history messages that you've had beforehand which makes it a whole lot better at helping your user do whatever it needs to do so in this case i'm going to import chat message history and i'm going to import my chat open ai again and so i'm going to create my chat model and then i'm going to import my chat open AI again. And so I'm going to create my chat model and then I'm going to create my history model. And to my history model, I'm going to add an AI message. Hi. And then I'm going to add a user message. What is the capital of France? So let me go ahead and run that. And if I were to take a look at my history messages, I get my two that are input right there and they're in the right order as we would expect there to be so what's cool is that I can pass my history of messages to the language model and so in this case it is gonna read oh I said hi to start and then the human message was what's the capital of France and let's see what it responds back to us the capital of France is Paris and it gives us an AI message which is cool and so what I want to do here is I want to add an AI message to my history which I shouldn't repeat this but I am actually no I'm not repeating it I'm taking the AI response and I'm just putting out the content let me print out those messages again and you can see here that it adds the capital Francis Paris to the end of my chat history which makes it easy for me to work with and another cool functionality of this too is langchain makes it extremely simple to save this chat history so you can go ahead and load it later a lot of really cool functionality i encourage you to go check out the next concept we're going to look at is chains so in this case we're going to be combining different llm calls and actions automatically so say you have one input but then the output of that language model you want to use as the input to another call, and then another call, and then another call. Well, in that case, you're going to be using chains, which is where the chain and lang chain comes from. So in this case, we're going to cover two of them. There's a lot of really complicated examples here. I encourage you again to go check out the documentation to see if one of them would cover your use case better than what you're seeing here. The first one is going to be a simple sequential chain. And in this case, I'm going to go ahead and tell it, hey, I want you to do X and then Y and then Z. Now, the reason why this is important or why I like to do it is because it helps break up the tasks. Now, language models can get distracted sometimes. And if you ask it to do too many things in a row, it could get, it could get confused. It could start to hallucinate and that's not good for anybody. Plus I want to make sure that my thinking is sound. And that way I can kind of check out the different outputs of each one of my different actions here. So in this case, I'm going to import the simple sequential chain. Let me go ahead and run this. And I'm going to put two different things to here. I'm going to use two different prompt templates. So your job is to come up with a classic dish from the area that the user suggests. I'm going to input the user location and I'm going to give it the user location, which we'll do in a second here. And I'm going to create a LLM chain with this and I'm going to call it location chain, which basically is going to take my language model. It's going to take my language model. It's going to take a prompt template. And then the next one we're going to look at, given a meal, give a short and simple recipe on how to make that dish at home. So in this case, we have the user location, which that's not actually what we want. We want user meal output. This wouldn't have mattered because I had the variables the same, but just to make it more clear. Given a meal, cool your response i'm going to do the same thing i'm going to put that into a meal chain so what it's going to do is it's going to output a meal a classic dish and then it's going to output a simple recipe for that classic dish okay i'm going to create my simple sequential chain and in this case i'm going to specify my chains as my location chain, and then the meal chain. Order matters. Be careful on that. I'm going to set verbose equals true, which means that it's going to tell us what it's thinking. And it's actually going to print those statements out. So it's easier to debug what's going on. Let's go ahead and create that. And then I'm going to say my overall chain, I want you to run. And in this case, I only have one input variable which is going to be roam which is going to be the user location that i start in the first place let me go ahead and run this so you can see here that it's entering the new sequential chain and it ran roam against the first prompt template and got me a classic dish which is really cool and then it gave me a recipe to on how to make that classic dish which is really cool so all of a sudden it just did two different runs for me all in one go and i didn't have to run any complicated code i could just use langchain for that it's pretty sweet now the next one that i want to show is one that i use quite often which is going to be the summarization chain the reason why this one is so cool is because if you have a long piece of text and you want it summarized, or say you have an article you want summarized or a tweet thread or a hacker news post or whatever it may be, you're going to want to chunk up your longer piece of text and you're going to want to find summaries of those different chunks and then get a final summary. And in that case, what we're going to do is we're going to load in load summarized chain and we're going to do Paul Graham's essay disc. Not even sure what that one's about. Then we're going to split it up into different texts right here. The chunk size is going to be 700, and then I'm going to load summarize chain. And the chain type that I'm going to do is going to be that one that I mentioned beforehand, which is where you get the small summaries of the individual sections, and then you get a summary of the small summaries. I have a whole video on different chain types. And so if you're curious go check out the video up above and you can go see it. Let me go ahead and run this. And so as you can see here the language model is asking, I'm sorry, the chain or lang chain is asking the language model to summarize this piece of text right here and then this piece of text right here and then this piece of text right here because we only had two chunks that we wanted to summarize and then it's asking for a final concise summary so here's the summary of the chunk number one here's the summary of chunk number two and it's asking for a summary of the summaries and we finally get a summary of the summaries which is really cool because all built into this one liner right here was all the different calls back and forth to figure out how to do the summary of the summaries, which is one of the powers of Langchain, which is really sweet. The last thing we're going to look at is agents. And this is one of the most complicated concepts within Langchain, which is why we're talking about it last here. But I thought that the official Lang chain documentation did a great job describing what agents are. Some applications will not require just a predetermined chain of calls to LLMs and other tools. What we did up above was a predetermined chain here. But potentially an unknown chain that depends on the user input. An unknown chain, emphasis mine, means that we're not really sure what route we wanna take, but we want the language model to tell us which route it thinks that it should take. In these types of chains, there is an agent, which has an access to a suite of tools. Depending on the user input, the agent can then decide which, if any, of these tools to call. So for example, hey, you have two databases you could pick information from. They're on completely different topics. The user just asked you a question about trees. Which database should you go look in to find your tree information? Well, an agent can decide that, which is really sweet. So I'm going to go over the vocabulary first, and then we're going to look at an example. So an agent is the language model that is going to be driving the decision making cool tools or tool is going to be a capability of the agent so you can think of this as similar to the open ai plugins that just came out you can also think of this as the ability to go search google the ability to go look your email whatever it may be a tool kit is going to be a collection of tools so an agent will have a tool kit of tools uh an agent will have a toolkit of tools and that's what that's what it's going to do there i'm going to import load tools i'm going to initialize the agent i'm going to import open ai as well with that i'm going to create my language model now i've made i've insert my serp api key because that's the example that we're gonna be running through here which is an easy way to search google and then with the toolkit i'm gonna go ahead and load the tools now in this case i'm only loading one tool and it's the surf api however you could load in a lot of tools here and you may naturally think well let me just load it up with all the tools in the world you could it's just gonna get difficult for the model or the agent to know which tool to use at which time so you kind of only want to use the ones that you know you're gonna be needing at that at that point so I'm gonna pass in my language model and I'm gonna pass in my surf API here API key then I'm gonna create my agent so I'm gonna pass in the toolkit that I just made I'm gonna pass in the language model again. I'm going to say, what type of agent are you? Now there's different agent types for different types of tasks. And I encourage you to go check out the language or the documentation to see which one would be best for you. I'm going to say verbose equals true. So we can see it thinking. I'm also going to return the intermediate steps. It just means that we get more granularity into what it's actually doing. With this, I'm gonna say response, oh, agent is not defined. Then what I'm gonna do here is I'm gonna pass in my query to the agent itself. So what was the first album of the band that Natalie Bergman is a part of? The reason why I asked this question specifically is because keep in mind, I haven't uploaded any documents here, so there's no information preloaded, and it's kind of a complicated question that has multiple steps that need to be answered for it. And this is a perfect question for an agent here. So let's go ahead and run this and let's see how the agent is thinking about it. Entering the new agent executor class. And it said, I should try to find out what band Natalie Bergman is a part of. So it knows that it needs to go search, which it has a search tool up above, which I gave it. And it's saying Natalie Bergman band. So it's searching for that one. And it says observation, which is what it observed from its action. Natalie Bergman is an American singer songwriter. She is one half the duo of wild bell. Okay, cool. I should search for the debut album of wild bell. It understood the band that she's a part of. And now it needs, now it knows it needs to go search for that band. So it's gonna search again, and it's gonna say Wild Belle debut album, and it observes that the debut album is Ailes. I know the final answer, which is good. We want it to know the final answer. Ailes is the debut album of Wild Belle, the band that Natalie Bergman is a part of. That is really cool because that is a multi-step question and the agent knew what it needed to go find out without me telling it the chain so this chain could have been a whole lot longer if it needed more steps with it but it dynamically figured that out along the way which is really really cool and so if we were to print out the intermediate steps you get more information about what it actually did and how it searched and all that good information from there um and if we were to confirm this let's go ahead and run this wow yep wild bell there's natalie bergman brother and sister duo band um beautiful i would play their song if it wasn't going to give me copyright trouble but i encourage you to go look it up link to my favorite song of theirs is in the description well my friends that was a very broad overview of all of the nuts and bolts of Lion Chain, the tactical nuts and bolts. I congratulate you for making it to the end of this video. And if you have any questions, please let me know. I encourage you to subscribe to check out for part two when we go through actual use cases for these nuts and bolts. And again, I share a lot of tools on Twitter, so I encourage you to follow me there like always please leave comments let me know what you think of the video and let me know if you have any questions we'll see you later\n"
     ]
    }
   ],
   "source": [
    "# load the whisper large model from HuggingFace\n",
    "\n",
    "device = \"cuda:0\" if torch.cuda.is_available() else \"cpu\"\n",
    "torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32\n",
    "\n",
    "model_id = \"openai/whisper-large-v3\"\n",
    "\n",
    "model = AutoModelForSpeechSeq2Seq.from_pretrained(\n",
    "    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True,)\n",
    "model.to(device)\n",
    "\n",
    "processor = AutoProcessor.from_pretrained(model_id)\n",
    "\n",
    "pipe = pipeline(\n",
    "    \"automatic-speech-recognition\",\n",
    "    model=model,\n",
    "    tokenizer=processor.tokenizer,\n",
    "    feature_extractor=processor.feature_extractor,\n",
    "    max_new_tokens=128,\n",
    "    chunk_length_s=30,\n",
    "    batch_size=16,\n",
    "    return_timestamps=True,\n",
    "    torch_dtype=torch_dtype,\n",
    "    device=device,\n",
    ")\n",
    "\n",
    "# use a sample mp3 file, in this case a mp4 to mp3 converted youtube video explaining the key concepts of Langchain\n",
    "sample= r\"audio directory\\The LangChain Cookbook - Beginner Guide To 7 Essential Concepts.mp3\"\n",
    "result= pipe(sample)\n",
    "print(result[\"text\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "98c6772c-4f9c-4d86-890d-bd4d95a25900",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Here's a summary of the key concepts of Langchain based on the provided transcript, with headlines for each key concept:\n",
      "\n",
      "#1 What is Langchain?\n",
      "Langchain is a framework that helps developers build applications with large language models (LLMs) like GPT-3. It provides a modular and extensible approach to working with LLMs, making it easier to integrate them into various applications.\n",
      "\n",
      "#2 Agents\n",
      "Agents are a core concept in Langchain. They represent the ability to break down complex tasks into smaller steps, retrieve relevant information, and iteratively refine the solution. Agents can be thought of as autonomous entities that can reason and plan to achieve a specific goal.\n",
      "\n",
      "#3 Tools\n",
      "Tools are the building blocks that agents use to accomplish tasks. They can be anything from search engines, databases, APIs, or even other LLMs. Agents can leverage multiple tools to gather information, perform calculations, or execute actions.\n",
      "\n",
      "#4 Memory\n",
      "Memory is a crucial component that allows agents to keep track of their thought process and the information they've gathered. It enables agents to maintain context and build upon previous steps, rather than treating each step as isolated.\n",
      "\n",
      "#5 Prompting\n",
      "Prompting is the process of providing instructions or prompts to LLMs. Langchain provides utilities for creating and managing prompts, which are essential for guiding the LLM's behavior and output.\n",
      "\n",
      "#6 Chains\n",
      "Chains are sequences of operations or steps that can be chained together. They provide a way to combine different components, such as agents, tools, and prompts, into a larger workflow or pipeline.\n",
      "\n",
      "#7 Tracing\n",
      "Tracing is a feature in Langchain that allows developers to visualize and understand the thought process of an agent. It provides insights into how the agent broke down a task, what tools it used, and how it arrived at a particular solution.\n",
      "\n",
      "#8 Memory Streams\n",
      "Memory streams are a way to persistently store and retrieve the memory of an agent. This feature enables agents to maintain long-term memory and recall information from previous interactions or sessions.\n",
      "\n",
      "#9 Callbacks\n",
      "Callbacks are functions that can be executed at different stages of an agent's lifecycle, such as before or after a tool is used or a step is completed. They provide a way to extend the functionality of agents and integrate custom logic.\n",
      "\n",
      "#10 Customization and Extensibility\n",
      "Langchain is designed to be highly customizable and extensible. Developers can create their own custom agents, tools, prompts, and chains, allowing them to tailor the framework to their specific needs and use cases.\n"
     ]
    }
   ],
   "source": [
    "# summarize the transcripted result using the Anthropic API with Claude3 Opus, a state of the art LLM that is extremely fast and exerces excellence in all tasks\n",
    "transcription= result[\"text\"]\n",
    "\n",
    "# use the Claude API to summarize the text with Claude 3 Sonnet due to its large output and input capablities\n",
    "# Claude 3 Opus is also available but cost/million token is 5x higher than for Claude 3 Sonnet\n",
    "# Pre-define the prompt for cleaner code\n",
    "\n",
    "prompt=\"You are a fast learner getting started in the field of AI. Please summarize the key concepts of Langchain, following the transcript about a video explanation about the framework.\"\\\n",
    "\"Highlight the key concepts and use headlines for every key concept you explain in the summary. Here is the transcript: {transcription}\"\n",
    "\n",
    "# Initialize the client\n",
    "client= Anthropic(\n",
    "    api_key=api_key,\n",
    ")\n",
    "\n",
    "#configure the model for message creation\n",
    "message= client.messages.create(\n",
    "    model=\"claude-3-sonnet-20240229\",\n",
    "    max_tokens=4096,\n",
    "    messages=[\n",
    "        {\"role\":\"user\",\"content\":prompt}\n",
    "    ]\n",
    ")\n",
    "\n",
    "# print only the text part of the response body which includes the summarization\n",
    "print(message.content[0].text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd0f515b-bed6-4e7e-87fe-79b643d6879c",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}