Basic Prompt#
According to bedroc docs
Prompts are a specific set of inputs provided by you, the user, that guide LLMs on Amazon Bedrock to generate an appropriate response or output for a given task or instruction.
- Proper format prompt for a model such as anthropic.claude-v2
- Add historical conversation or templates
Let create a simple example prompting an anthropic.calude-2
const messages = [{ role: "user", content: "how to cook chicken soup?" },{role: "assistant",content:" Here is a basic recipe for homemade chicken soup:\n\nIngredients:\n- 1 whole chicken or 2-3 pounds chicken pieces (breasts, thighs, legs)\n- 2 tbsp olive oil\n- 1 onion, diced \n- 3 carrots, peeled and sliced\n- 3 stalks celery, sliced\n- 6 cups chicken broth\n- 2 bay leaves\n- 1 tsp thyme\n- Salt and pepper to taste\n- Egg noodles or rice (optional)\n\nInstructions:\n\n1. If using a whole chicken, remove skin and fat and cut the chicken into pieces. If using chicken pieces, trim off any excess fat or skin.\n\n2. Heat olive oil in a large pot over medium heat. Add the chicken pieces and cook for 3-4 minutes on each side until browned. Remove chicken to a plate. \n\n3. Add the onion, carrots and celery to the pot. Sauté for 5 minutes until vegetables are softened. \n\n4. Return the chicken to the pot and add the broth, bay leaves, thyme, salt and pepper. Bring to a boil, then reduce heat and simmer for 45 mins to 1 hour.\n\n5. Remove the chicken pieces and shred or cut into bite-sized pieces. Discard bay leaves.\n\n6. Add the shredded chicken back to the pot and add noodles or rice if desired. Cook for 8-10 minutes until noodles or rice are tender. \n\n7.",},{role: "user",content: "how to customize it for a little 3 years old girl?",},{role: "assistant",content:" Here are some tips for customizing homemade chicken soup for a 3 year old girl:\n\n- Use a mild tasting broth or reduce the amount of herbs/seasonings so it's not too strong flavored. You can use low sodium chicken broth.\n\n- Add small pasta shapes like stars, alphabets or mini noodles. Toddlers enjoy fun shaped pastas. \n\n- Dice the vegetables into small pieces so they are easy to eat. Good veggie additions include carrots, peas, corn, green beans.\n\n- Shred or cut the chicken into very small, bite-sized pieces. Remove any bones or skin to be safe. \n\n- Add a spoonful of uncooked rice to thicken the broth slightly. This makes it easy for a toddler to eat.\n\n- Stir in some spinach or kale at the end for extra nutrients. Blanch quickly in the hot soup to wilt.\n\n- Keep the soup on the milder side for spiciness. Avoid pepper or hot spices.\n\n- Mix in a dollop of plain yogurt or sour cream to provide thickness and tang.\n\n- Garnish with a sprinkle of shredded cheddar cheese for added flavor and nutrition. \n\n- Let the soup cool slightly before serving to prevent burns. Check temperature first.\n\n- Serve with soft bread sticks or rolls that are easy to dip and chew.\n\nThe key is tailoring the textures and flavors to a",},{role: "user",content: "should I add some spicy ingredients for the 3 years old one? ",},];let buffer = "";for (let i = 0; i < messages.length; i++) {if (messages[i]["role"] == "user") {buffer += "\n\nHuman:" + " " + messages[i]["content"];}if (messages[i]["role"] == "assistant") {buffer += "\n\nAssitant:" + messages[i]["content"];}}buffer += "\n\nAssistant:";console.log(buffer);
Then send a request to bedrock
const callBedrock = async ({ prompt }: { prompt: string }) => {const response = await bedrockClient.send(new InvokeModelCommand({modelId: "anthropic.claude-v2",contentType: "application/json",accept: "application/json",body: JSON.stringify({prompt: prompt,max_tokens_to_sample: 2048,}),}));if (response.body) {const text = decoder.decode(response.body);console.log(text);}};
LangChain Prompt#
Let use LangChain to create prompts from template or messages
const generatePrompt = async () => {const template = PromptTemplate.fromTemplate(`Human: The following is a friendly conversation between a human and an AI.The AI is talkative and provides lots of specific details from its context. If the AI does not knowthe answer to a question, it truthfully says it does not know.Current conversation:<conversation_history>{history}</conversation_history>Here is the human's next reply:<human_reply>{input}</human_reply>Assistant:`);const prompt = await template.format({history: "",input: "How to cook chicken soup?",});console.log(prompt);return prompt;};
Another way to call prompt is using messages
const prompt = ChatPromptTemplate.fromMessages([["system", "You are a world class technical documentation writer."],["user", "{input}"],]);console.log(await prompt.format({ input: "How to cook chicken soup?" }));
Then prompt the model
const prompt = await generatePrompt();await callBedrock({ prompt: prompt });
LangChain Chain#
Let create a simple chain here
The most basic and common use case is chaining a prompt template and a model together
const testChain = async () => {const parser = new StringOutputParser();const prompt = PromptTemplate.fromTemplate(`Human: The following is a friendly conversation between a human and an AI.The AI is talkative and provides lots of specific details from its context. If the AI does not knowthe answer to a question, it truthfully says it does not know.Current conversation:<conversation_history>{history}</conversation_history>Here is the human's next reply:<human_reply>{input}</human_reply>Assistant:`);const previousMessages = [{ role: "user", content: "how to cook chicken soup?" },{role: "assistant",content:" Here is a basic recipe for homemade chicken soup:\n\nIngredients:\n- 1 whole chicken or 2-3 pounds chicken pieces (breasts, thighs, legs)\n- 2 tbsp olive oil\n- 1 onion, diced \n- 3 carrots, peeled and sliced\n- 3 stalks celery, sliced\n- 6 cups chicken broth\n- 2 bay leaves\n- 1 tsp thyme\n- Salt and pepper to taste\n- Egg noodles or rice (optional)\n\nInstructions:\n\n1. If using a whole chicken, remove skin and fat and cut the chicken into pieces. If using chicken pieces, trim off any excess fat or skin.\n\n2. Heat olive oil in a large pot over medium heat. Add the chicken pieces and cook for 3-4 minutes on each side until browned. Remove chicken to a plate. \n\n3. Add the onion, carrots and celery to the pot. Sauté for 5 minutes until vegetables are softened. \n\n4. Return the chicken to the pot and add the broth, bay leaves, thyme, salt and pepper. Bring to a boil, then reduce heat and simmer for 45 mins to 1 hour.\n\n5. Remove the chicken pieces and shred or cut into bite-sized pieces. Discard bay leaves.\n\n6. Add the shredded chicken back to the pot and add noodles or rice if desired. Cook for 8-10 minutes until noodles or rice are tender. \n\n7.",},];// console.log(formatMessages(previousMessages));const promptValue = await prompt.invoke({history: formatMessages(previousMessages),input: "How to customize it for a little 3 years old girl?",});console.log(promptValue.toString());const chain = prompt.pipe(model);const stream = await chain.stream({history: formatMessages(previousMessages),input: "How to customize it for a little 3 years old girl?",});for await (const chunk of stream) {process.stdout.write(chunk);}const response = await chain.invoke({history: formatMessages(previousMessages),input: "how to customize it for a little 3 years old girl?",});console.log(response);};
LangChain Memory#
Most LLM applications have a conversational interface. An essential component of a conversation is being able to refer to information introduced earlier in the conversation. At bare minimum, a conversational system should be able to access some window of past messages directly. A more complex system will need to have a world model that it is constantly updating, which allows it to do things like maintain information about entities and their relationships.
-
Customize memory AI Prefix
const testLLMChain = async () => {const prompt = PromptTemplate.fromTemplate(`Human: The following is a friendly conversation between a human and an AI.The AI is talkative and provides lots of specific details from its context. If the AI does not knowthe answer to a question, it truthfully says it does not know.Current conversation:<conversation_history>{history}</conversation_history>Here is the human's next reply:<human_reply>{question}</human_reply>Assistant:`);const memory = new BufferMemory({memoryKey: "history",returnMessages: true,inputKey: "question",aiPrefix: "AI Assistant",});const llm = new Bedrock({model: "anthropic.claude-v2",region: "us-east-1",maxTokens: 2048,temperature: 0.8,});const conversation = new LLMChain({llm: llm,prompt: prompt,verbose: true,memory: memory,});const response = await conversation.invoke({question: "How to cook chicken soup?",});console.log(response);// const response1 = await conversation.invoke({// question: "How to customize it for a 3 years old little girl?",// });// console.log(response1);console.log(await conversation.memory?.loadMemoryVariables({}));};
LangChain Integration#
There are multiple ways to call bedrock
Let call Bedorck using aws sdk Bedrock runtime
const bedrockClient = new BedrockRuntimeClient({region: "us-east-1",});const response = await bedrockClient.send(new InvokeModelCommand({modelId: "anthropic.claude-v2",contentType: "application/json",accept: "application/json",body: JSON.stringify({prompt: prompt,max_tokens_to_sample: 2048,}),}));
Let call Bedrock using LangChain LLM Bedrock model
const model = new Bedrock({model: "anthropic.claude-v2",region: "us-east-1",maxTokens: 2048,temperature: 0.8,});const prompt = await generatePrompt();const response = await model.invoke(prompt, {});
Let call Bedrock using LangChain Bedrock Chat Model
import { BedrockChat } from "@langchain/community/chat_models/bedrock";// Or, from web environments:// import { BedrockChat } from "@langchain/community/chat_models/bedrock/web";import { HumanMessage } from "@langchain/core/messages";// If no credentials are provided, the default credentials from// @aws-sdk/credential-provider-node will be used.const model = new BedrockChat({model: "anthropic.claude-v2",region: "us-east-1",// endpointUrl: "custom.amazonaws.com",// credentials: {// accessKeyId: process.env.BEDROCK_AWS_ACCESS_KEY_ID!,// secretAccessKey: process.env.BEDROCK_AWS_SECRET_ACCESS_KEY!,// },// modelKwargs: {},});const res = await model.invoke([new HumanMessage({ content: "Tell me a joke" }),]);console.log(res);/*AIMessage {content: " Here's a silly joke: \n" +'\n' +'What do you call a dog magician? A labracadabrador!',name: undefined,additional_kwargs: {}}*/
Code#
Here is the sample code
sample.ts
import {BedrockRuntimeClient,InvokeModelCommand,} from "@aws-sdk/client-bedrock-runtime";import { ChatPromptTemplate, PromptTemplate } from "@langchain/core/prompts";import { Bedrock } from "@langchain/community/llms/bedrock";import { StringOutputParser } from "@langchain/core/output_parsers";import { LLMChain } from "langchain/chains";import { BufferMemory } from "langchain/memory";const messages = [{ role: "user", content: "how to cook chicken soup?" },{role: "assistant",content:" Here is a basic recipe for homemade chicken soup:\n\nIngredients:\n- 1 whole chicken or 2-3 pounds chicken pieces (breasts, thighs, legs)\n- 2 tbsp olive oil\n- 1 onion, diced \n- 3 carrots, peeled and sliced\n- 3 stalks celery, sliced\n- 6 cups chicken broth\n- 2 bay leaves\n- 1 tsp thyme\n- Salt and pepper to taste\n- Egg noodles or rice (optional)\n\nInstructions:\n\n1. If using a whole chicken, remove skin and fat and cut the chicken into pieces. If using chicken pieces, trim off any excess fat or skin.\n\n2. Heat olive oil in a large pot over medium heat. Add the chicken pieces and cook for 3-4 minutes on each side until browned. Remove chicken to a plate. \n\n3. Add the onion, carrots and celery to the pot. Sauté for 5 minutes until vegetables are softened. \n\n4. Return the chicken to the pot and add the broth, bay leaves, thyme, salt and pepper. Bring to a boil, then reduce heat and simmer for 45 mins to 1 hour.\n\n5. Remove the chicken pieces and shred or cut into bite-sized pieces. Discard bay leaves.\n\n6. Add the shredded chicken back to the pot and add noodles or rice if desired. Cook for 8-10 minutes until noodles or rice are tender. \n\n7.",},{role: "user",content: "how to customize it for a little 3 years old girl?",},{role: "assistant",content:" Here are some tips for customizing homemade chicken soup for a 3 year old girl:\n\n- Use a mild tasting broth or reduce the amount of herbs/seasonings so it's not too strong flavored. You can use low sodium chicken broth.\n\n- Add small pasta shapes like stars, alphabets or mini noodles. Toddlers enjoy fun shaped pastas. \n\n- Dice the vegetables into small pieces so they are easy to eat. Good veggie additions include carrots, peas, corn, green beans.\n\n- Shred or cut the chicken into very small, bite-sized pieces. Remove any bones or skin to be safe. \n\n- Add a spoonful of uncooked rice to thicken the broth slightly. This makes it easy for a toddler to eat.\n\n- Stir in some spinach or kale at the end for extra nutrients. Blanch quickly in the hot soup to wilt.\n\n- Keep the soup on the milder side for spiciness. Avoid pepper or hot spices.\n\n- Mix in a dollop of plain yogurt or sour cream to provide thickness and tang.\n\n- Garnish with a sprinkle of shredded cheddar cheese for added flavor and nutrition. \n\n- Let the soup cool slightly before serving to prevent burns. Check temperature first.\n\n- Serve with soft bread sticks or rolls that are easy to dip and chew.\n\nThe key is tailoring the textures and flavors to a",},{role: "user",content: "should I add some spicy ingredients for the 3 years old one? ",},];const decoder = new TextDecoder();const bedrockClient = new BedrockRuntimeClient({region: "us-east-1",});const model = new Bedrock({model: "anthropic.claude-v2",region: "us-east-1",maxTokens: 2048,temperature: 0.8,});const formatMessages = (messages: any) => {let buffer = "";for (let i = 0; i < messages.length; i++) {if (messages[i]["role"] == "user") {buffer += "\n\nHuman:" + " " + messages[i]["content"];}if (messages[i]["role"] == "assistant") {buffer += "\n\nAssitant:" + messages[i]["content"];}}return buffer;};const callBedrock = async ({ prompt }: { prompt: string }) => {const response = await bedrockClient.send(new InvokeModelCommand({modelId: "anthropic.claude-v2",contentType: "application/json",accept: "application/json",body: JSON.stringify({prompt: prompt,max_tokens_to_sample: 2048,}),}));if (response.body) {const text = decoder.decode(response.body);console.log(text);}// console.log(response);};const testPrompt = async () => {const template = PromptTemplate.fromTemplate(`Human: The following is a friendly conversation between a human and an AI.The AI is talkative and provides lots of specific details from its context. If the AI does not knowthe answer to a question, it truthfully says it does not know.Current conversation:<conversation_history>{history}</conversation_history>Here is the human's next reply:<human_reply>{input}</human_reply>Assistant:`);const prompt = await template.format({history: "",input: "How to cook chicken soup?",});console.log(prompt);return prompt;};const testBedrockModel = async ({ prompt }: { prompt: string }) => {const response = await model.invoke(prompt, {});console.log(response);};const testChain = async () => {const parser = new StringOutputParser();const prompt = PromptTemplate.fromTemplate(`Human: The following is a friendly conversation between a human and an AI.The AI is talkative and provides lots of specific details from its context. If the AI does not knowthe answer to a question, it truthfully says it does not know.Current conversation:<conversation_history>{history}</conversation_history>Here is the human's next reply:<human_reply>{input}</human_reply>Assistant:`);const previousMessages = [{ role: "user", content: "how to cook chicken soup?" },{role: "assistant",content:" Here is a basic recipe for homemade chicken soup:\n\nIngredients:\n- 1 whole chicken or 2-3 pounds chicken pieces (breasts, thighs, legs)\n- 2 tbsp olive oil\n- 1 onion, diced \n- 3 carrots, peeled and sliced\n- 3 stalks celery, sliced\n- 6 cups chicken broth\n- 2 bay leaves\n- 1 tsp thyme\n- Salt and pepper to taste\n- Egg noodles or rice (optional)\n\nInstructions:\n\n1. If using a whole chicken, remove skin and fat and cut the chicken into pieces. If using chicken pieces, trim off any excess fat or skin.\n\n2. Heat olive oil in a large pot over medium heat. Add the chicken pieces and cook for 3-4 minutes on each side until browned. Remove chicken to a plate. \n\n3. Add the onion, carrots and celery to the pot. Sauté for 5 minutes until vegetables are softened. \n\n4. Return the chicken to the pot and add the broth, bay leaves, thyme, salt and pepper. Bring to a boil, then reduce heat and simmer for 45 mins to 1 hour.\n\n5. Remove the chicken pieces and shred or cut into bite-sized pieces. Discard bay leaves.\n\n6. Add the shredded chicken back to the pot and add noodles or rice if desired. Cook for 8-10 minutes until noodles or rice are tender. \n\n7.",},];// console.log(formatMessages(previousMessages));const promptValue = await prompt.invoke({history: formatMessages(previousMessages),input: "How to customize it for a little 3 years old girl?",});console.log(promptValue.toString());// const chain = prompt.pipe(model);// const stream = await chain.stream({// history: formatMessages(previousMessages),// input: "How to customize it for a little 3 years old girl?",// });// for await (const chunk of stream) {// process.stdout.write(chunk);// }// const response = await chain.invoke({// history: formatMessages(previousMessages),// input: "how to customize it for a little 3 years old girl?",// });// console.log(response);};const testPromptMessages = async () => {const prompt = ChatPromptTemplate.fromMessages([["system", "You are a world class technical documentation writer."],["user", "{input}"],]);console.log(await prompt.format({ input: "How to cook chicken soup?" }));};const testLLMChain = async () => {const prompt = PromptTemplate.fromTemplate(`Human: The following is a friendly conversation between a human and an AI.The AI is talkative and provides lots of specific details from its context. If the AI does not knowthe answer to a question, it truthfully says it does not know.Current conversation:<conversation_history>{history}</conversation_history>Here is the human's next reply:<human_reply>{question}</human_reply>Assistant:`);const memory = new BufferMemory({memoryKey: "history",returnMessages: true,inputKey: "question",aiPrefix: "AI Assistant",});const llm = new Bedrock({model: "anthropic.claude-v2",region: "us-east-1",maxTokens: 2048,temperature: 0.8,});const conversation = new LLMChain({llm: llm,prompt: prompt,verbose: true,memory: memory,});const response = await conversation.invoke({question: "How to cook chicken soup?",});// console.log(response);// const response1 = await conversation.invoke({// question: "How to customize it for a 3 years old little girl?",// });// console.log(response1);console.log(await conversation.memory?.loadMemoryVariables({}));};const main = async () => {// const prompt = await testPrompt();// await callBedrock({ prompt: prompt });// await testBedrockModel({ prompt: prompt });// await testChain();// await testPromptMessages();await testLLMChain();};main();
And package.json
{"name": "next-prisma-hello","version": "0.1.0","private": true,"scripts": {"dev": "next dev","build": "next build","start": "next start","lint": "next lint"},"dependencies": {"@aws-sdk/client-bedrock-runtime": "^3.490.0","@aws-sdk/client-s3": "^3.504.0","@aws-sdk/s3-request-presigner": "^3.504.0","@langchain/community": "^0.0.28","@langchain/core": "^0.1.28","@prisma/client": "^5.8.0","ai": "^2.2.33","langchain": "^0.1.18","next": "14.0.4","package.json": "^2.0.1","react": "^18","react-dom": "^18"},"devDependencies": {"@types/node": "^20","@types/react": "^18","@types/react-dom": "^18","autoprefixer": "^10.0.1","eslint": "^8","eslint-config-next": "14.0.4","postcss": "^8","prisma": "^5.8.0","tailwindcss": "^3.3.0","typescript": "^5"}}