CDK Entest

Introduction#

GitHub this note shows

Bedrock Python SDK batch and stream
Create a simple Flask app
JavaScript handle stream response
Deploy on Lambda web adapter

Beckrock SDK#

Let call bedrock with batch response

import boto3
import json
from flask import jsonify

client = boto3.client('bedrock-runtime')

def bedrock_batch(topic='chicken soup'):
    """
    demo
    """

    instruction = f"""
    You are a world class writer. Please write a sweet bedtime story about {topic}.
    """

    body = json.dumps({
        'prompt': f'Human:{instruction}\n\nAssistant:',
        'max_tokens_to_sample': 1028,
        'temperature': 1,
        'top_k': 250,
        'top_p': 0.999,
        'stop_sequences': ['\n\nHuman:']
    })

    response = client.invoke_model(
        body=body,
        contentType="application/json",
        accept="*/*",
        modelId="anthropic.claude-v2",
    )

    response_body = json.loads(
        response.get('body').read()
    )

    print(response_body)

Let call becrock with stream response

def bedrock_stream(topic='chicken soup'):
    """
    demo
    """
    # prompt
    instruction = f"""
    You are a world class writer. Please write a sweet bedtime story about {topic}.
    """
    # request body
    body = json.dumps({
        'prompt': f'Human:{instruction}\n\nAssistant:',
        'max_tokens_to_sample': 1028,
        'temperature': 1,
        'top_k': 250,
        'top_p': 0.999,
        'stop_sequences': ['\n\nHuman:']
    })
    # response
    response = client.invoke_model_with_response_stream(
        body=body,
        contentType="application/json",
        accept="*/*",
        modelId="anthropic.claude-v2",
    )
    # parse stream
    stream = response.get('body')
    if stream:
        for event in stream:
            print(event)

Flask App#

Create a flask app with project structure as the following

|--static
   |--output.css
   |--script.js
|--templates
   |--bedrock.html
   |--index.html
|--app.py
|--Dockerfile
|--requirements.txt

JavaScript Stream Response#

Let create a simple javascript at client side to handle stream response from Flask

const storyOutput = document.getElementById('story-output')

const callBedrock = async () => {
  const topic = document.getElementById('topic').value
  storyOutput.innerText = 'thinking ...'
  console.log('call bedrock request', topic)

  if (topic) {
    try {
      const response = await fetch('/bedrock', {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json'
        },
        body: JSON.stringify({ topic: topic })
      })
      const body = await response.json()
      console.log(body)
      storyOutput.innerText = body.completion
    } catch (error) {
      console.log(error)
    }
  }
}

const callBedrockStream = async () => {
  console.log('call bedrock request')

  try {
    const response = await fetch('/bedrock-stream', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json'
      },
      body: JSON.stringify({ topic: 'chicken soup' })
    })

    console.log(response)
    const reader = response.body.getReader()
    const decoder = new TextDecoder()

    while (true) {
      const { done, value } = await reader.read()
      if (done) {
        break
      }
      const text = decoder.decode(value)
      console.log(text)
      storyOutput.innerText += text
    }
  } catch (error) {
    console.log(error)
  }
}

document
  .getElementById('submit-button')
  .addEventListener('click', callBedrockStream)

Lambda Web Adapter#

Let create a lambda web adapter for hosting the web. Please note that it supports only batch response

import { Stack, StackProps, aws_iam, aws_lambda } from 'aws-cdk-lib'
import { Effect } from 'aws-cdk-lib/aws-iam'
import { Construct } from 'constructs'
import path = require('path')

export class LambdaWebAdapter extends Stack {
  constructor(scope: Construct, id: string, props: StackProps) {
    super(scope, id, props)

    const func = new aws_lambda.Function(this, 'FlaskWebAdapter', {
      functionName: 'FlaskWebAdatper',
      code: aws_lambda.EcrImageCode.fromAssetImage(
        path.join(__dirname, './../../flask/app/')
      ),
      runtime: aws_lambda.Runtime.FROM_IMAGE,
      handler: aws_lambda.Handler.FROM_IMAGE,
      memorySize: 1024
    })

    func.addToRolePolicy(
      new aws_iam.PolicyStatement({
        effect: Effect.ALLOW,
        resources: [
          'arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-v2'
        ],
        actions: [
          'bedrock:InvokeModel',
          'bedrock:InvokeModelWithResponseStream'
        ]
      })
    )
  }
}

Summary CV#

post request to send pdf cv
parse the file pdf using PyPDF2
summarize content using Bedrock Claude 2

Let install somme packages

python -m pip install Flask
python -m pip install langchain-community
python -m pip install pypdf
python -m pip install langchain-text-splitters
python -m pip install PyPDF2

Let create a simple backend

"""
haimtran 17/04/2024
hello flask
python -m pip install Flask
python -m pip install langchain-community
python -m pip install pypdf
python -m pip install langchain-text-splitters
https://python.langchain.com/docs/get_started/installation/
"""

from io import BytesIO
from PyPDF2 import PdfReader
import boto3
from flask import Flask, jsonify, request, render_template
from langchain_community.document_loaders import PyPDFLoader
import json

client = boto3.client("bedrock-runtime")


def bedrock_batch(content):
    """
    demo
    """

    instruction = f"""
    Your are an expert in HR (Human Resource), please summarize the following CV in 3 bullet points and response in Vietnamese. Here is the CV {content}.
    """

    body = json.dumps(
        {
            "prompt": f"Human:{instruction}\n\nAssistant:",
            "max_tokens_to_sample": 1028,
            "temperature": 1,
            "top_k": 250,
            "top_p": 0.999,
            "stop_sequences": ["\n\nHuman:"],
        }
    )

    response = client.invoke_model(
        body=body,
        contentType="application/json",
        accept="*/*",
        modelId="anthropic.claude-v2",
    )

    response_body = json.loads(response.get("body").read())
    return response_body["completion"]


app = Flask(__name__)


@app.route("/")
def hello():
    """
    hello
    """
    return render_template("upload.html")


@app.route("/cv")
def summary_cv():
    """
    summary cv
    """
    return jsonify({"name": "Hai", "age": 20})


@app.route("/upload-json", methods=["POST"])
def upload_json():
    """
    process upload
    """
    # parse request
    data = request.get_json()
    # process data
    print(data)
    # return
    return jsonify({"status": "ok"})


@app.route("/upload", methods=["POST"])
def upload():
    """
    process upload
    """
    # parse request
    data = request.form
    # parse user question
    question = data["question"]
    # parse file
    file = request.files["file"]
    print(f"file name {file.filename} and question {question}")

    # load pdf
    content = ""
    # pdf loader
    # loader = PyPDFLoader("Profile.pdf")
    # load document
    # docs = loader.load_and_split()

    # pypdf2 pdfreader
    loader = PdfReader(file)
    for page in loader.pages:
        content += page.extract_text()

    # summary by bedrock claude 3
    # content = ""
    # for doc in docs:
    #     content += doc.page_content
    # # call bedrock
    summary = bedrock_batch(content)
    # return
    return jsonify({"summary": summary})


if __name__ == "__main__":
    app.run(host="0.0.0.0", port=3000, debug=True)

Here is the frontend page

upload.html

<html>
  <head>
    <title>Upload PDF</title>
    <meta name="viewport" content="width=device-width" />
    <style>
      :root {
        box-sizing: border-box;
      }

      *,
      ::before,
      ::after {
        box-sizing: inherit;
      }

      body {
        background-color: antiquewhite;
      }

      .container {
        max-width: 500px;
        margin: auto;
      }

      .container-form {
        position: relative;
      }

      .input-question {
        width: 100%;
        padding: 15px 10px;
      }
      .button-submit {
        background-color: orange;
        padding: 10px 25px;
        border-radius: 2px;
        border: none;
        outline: none;
        position: absolute;
        top: 50%;
        right: 10px;
        transform: translateY(-50%);
        cursor: pointer;
      }

      .input-file {
        width: 100%;
        padding: 15px 10px;
        background-color: aquamarine;
        cursor: pointer;
        margin-top: 10px;
      }

      .container-image {
        position: relative;
        background-color: gainsboro;
        padding: 10px 10px;
        align-items: center;
        justify-content: center;
        display: flex;
        max-height: 600px;
        height: 50%;
      }

      .description-image {
        position: absolute;
        bottom: 0;
        left: 0;
        background-color: azure;
        padding: 10px;
        opacity: 0.9;
      }

      .text-model {
        /* color: #4caf50; */
        font-family: system-ui, -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto,
          Oxygen, Ubuntu, Cantarell, 'Open Sans', 'Helvetica Neue', sans-serif;
        font-size: medium;
        font-weight: 400;
        letter-spacing: normal;
        line-height: 25px;
      }
    </style>
  </head>
  <body>
    <div class="container">
      <div>
        <form onkeydown="return event.key != 'Enter';">
          <div class="container-form">
            <input
              type="text"
              id="question"
              name="question"
              class="input-question"
              placeholder="what is in this image?"
            />
            <button class="button-submit" id="submit" type="submit">
              Upload
            </button>
          </div>
          <input type="file" id="file" name="file" class="input-file" />
        </form>
      </div>
      <div id="list" class="text-model"></div>
    </div>
  </body>

  <script>
    const fileInput = document.getElementById('file')
    const submit = document.getElementById('submit')
    let file

    // get file
    fileInput.addEventListener('change', event => {
      // get file
      file = event.target.files[0]
      if (file) {
        console.log(file)
      }
    })

    // cal backend cv endpoint
    const summaryCV = async () => {
      // Get the list container element
      var listContainer = document.getElementById('list')

      // clear content before query
      listContainer.innerHTML = ''

      // get user prompt
      let question = document.getElementById('question').value
      if ((question == '') | (question == null)) {
        question = 'what is in this image?'
      }
      // formdata
      const formData = new FormData()
      formData.append('question', 'question')
      formData.append('file', file)
      console.log(formData)

      try {
        const response = await fetch('/upload', {
          method: 'POST',
          body: formData
        })

        const result = await response.json()
        console.log(result)

        listContainer.innerText = result.summary
      } catch (error) {
        console.log(error)
      }
    }

    submit.addEventListener('click', async event => {
      event.preventDefault()

      await summaryCV()
    })

    // listen on enter
    document
      .getElementById('question')
      .addEventListener('keydown', async event => {
        if (event.code === 'Enter') {
          await summaryCV()
        }
      })
  </script>
</html>

Beckrock SDK#

Flask App#

JavaScript Stream Response#

Lambda Web Adapter#

Summary CV#

Reference#