Skip to main content

4 posts tagged with "app"

View All Tags

· 4 min read
Sparsh Agarwal

/img/content-blog-raw-blog-name-&-address-parsing-untitled.png

Introduction

Create an API that can parse and classify names and addresses given a string. We tried probablepeople and usaddress. These work well separately but need the functionality of these packages combined, and better accuracy than what probablepeople provides. For the API, I'd like to mimic this with some minor modifications. A few examples:

  • "KING JOHN A 5643 ROUTH CREEK PKWY #1314 RICHARDSON TEXAS 750820146 UNITED STATES OF AMERICA" would return type: person; first_name: JOHN; last_name: KING; middle: A; street_address: 5643 ROUTH CREEK PKWY #1314; city: RICHARDSON; state: TEXAS; zip: 75082-0146; country: UNITED STATES OF AMERICA.
  • "THRM NGUYEN LIVING TRUST 2720 SUMMERTREE CARROLLTON HOUSTON TEXAS 750062646 UNITED STATES OF AMERICA" would return type: entity; name: THRM NGUYEN LIVING TRUST; street_address: 2720 SUMMERTREE CARROLLTON; state: TEXAS; city: HOUSTON; zip: 75006-2646; country: UNITED STATES OF AMERICA.

Modeling Approach

List of Entities​

List of Entities A - Person, Household, Corporation

List of Entities B - Person First name, Person Middle name, Person Last name, Street address, City, State, Pincode, Country, Company name

Endpoint Configuration​

OOR Endpoint

Input Instance: ANDERSON, EARLINE 1423 NEW YORK AVE FORT WORTH, TX 76104 7522

Output Tags:-
<Type> - Person/Household/Corporation
<GivenName>, <MiddleName>, <Surname> - if Type Person/Household
<Name> - Full Name - if Type Person 
<Name> - Household - if Type Household
<Name> - Corporation - If Type Corporation
<Address> - Full Address
<StreetAddress>, <City>, <State>, <Zipcode>, <Country>
~~NameConfidence, AddrConfidence~~

Name Endpoint

Input Instance: ANDERSON, EARLINE

Output Tags:-

- <Type> - Person/Household/Corporation
- <GivenName>, <MiddleName>, <Surname> - if Type Person/Household
- <Name> - Full Name - if Type Person
- <Name> - Household - if Type Household
- <Name> - Corporation - If Type Corporation
- ~~NameConfidence~~

Address Endpoint

Input Instance: 1423 NEW YORK AVE FORT WORTH, TX 76104 7522

Output Tags:-

- <Address> - Full Address
- <StreetAddress>, <City>, <State>, <Zipcode>, <Country>
- ~~AddrConfidence~~

Process Flow​

  • Pytorch Flair NER model
  • Pre trained word embeddings
  • Additional parsing models on top of name tags
  • Tagging of 1000+ records to create training data
  • Deployment as REST api with 3 endpoints - name parse, address parse and whole string parse

Framework

/img/content-blog-raw-blog-name-&amp;-address-parsing-untitled-1.png

/img/content-blog-raw-blog-name-&amp;-address-parsing-untitled-2.png

Tagging process

I used Doccano (https://github.com/doccano/doccano) for labeling the dataset. This tool is open-source and free to use. I deployed it with a one-click Heroku service (fig 1). After launching the app, log in with the provided credentials, and create a project (fig 2). Create the labels and upload the dataset (fig 3). Start the annotation process (fig 4). Now after enough annotations (you do not need complete all annotations in one go), go back to projects > edit section and export the data (fig 5). Bring the exported JSON file in python and run the model training code. The whole model will automatically get trained on the new annotations. To make the training faster, you can use Nvidia GPU support.

fig 1: screenshot taken from Doccano&#39;s github page

fig 1: screenshot taken from Doccano's github page

fig 2: Doccano&#39;s deployed app homepage

fig 2: Doccano's deployed app homepage

fig 3: create the labels. I defined these labels for my project

fig 3: create the labels. I defined these labels for my project

fig 5: export the annotations

fig 5: export the annotations

Model

I first tried the Spacy NER blank model but it was not giving high-quality results. So I moved to the PyTorch Flair NER model. This model was a way faster (5 min training because of GPU compatibility comparing to 1-hour Spacy training time) and also much more accurate. F1 results for all tags were near perfect (score of 1). This score will increase further with more labeled data. This model is production-ready.

Inference

For OOR, I directly used the model's output for core tagging and created the aggregated tags like recipient (aggregation of name tags) and address (aggregation of address tags like city and state) using simple conditional concatenation. For only Name and only Address inference, I added the dummy address in name text and dummy name in address text. This way, I passed the text in same model and later on filtered the required tags as output.

API​

I used Flask REST framework in Python to build the API with 3 endpoints. This API is production-ready.

Results and Discussion

  • 0.99 F1 score on 6 out of 8 tags & 0.95+ F1 score on other 2 tags
  • API inference time of less than 1 second on single CPU

· 2 min read
Sparsh Agarwal

Live app​

This app can detect COCO 80-classes using three different models - Caffe MobileNet SSD, Yolo3-tiny, and Yolo3. It can also detect faces using two different models - SSD Res10 and OpenCV face detector. Yolo3-tiny can also detect fires.

/img/content-blog-raw-blog-object-detection-with-yolo3-untitled.png

/img/content-blog-raw-blog-object-detection-with-yolo3-untitled-1.png

Code​

import streamlit as st
import cv2
from PIL import Image
import numpy as np
import os

from tempfile import NamedTemporaryFile
from tensorflow.keras.preprocessing.image import img_to_array, load_img

temp_file = NamedTemporaryFile(delete=False)

DEFAULT_CONFIDENCE_THRESHOLD = 0.5
DEMO_IMAGE = "test_images/demo.jpg"
MODEL = "model/MobileNetSSD_deploy.caffemodel"
PROTOTXT = "model/MobileNetSSD_deploy.prototxt.txt"

CLASSES = [
    "background",
    "aeroplane",
    "bicycle",
    "bird",
    "boat",
    "bottle",
    "bus",
    "car",
    "cat",
    "chair",
    "cow",
    "diningtable",
    "dog",
    "horse",
    "motorbike",
    "person",
    "pottedplant",
    "sheep",
    "sofa",
    "train",
    "tvmonitor",
]
COLORS = np.random.uniform(0, 255, size=(len(CLASSES), 3))

@st.cache
def process_image(image):
    blob = cv2.dnn.blobFromImage(
        cv2.resize(image, (300, 300)), 0.007843, (300, 300), 127.5
    )
    net = cv2.dnn.readNetFromCaffe(PROTOTXT, MODEL)
    net.setInput(blob)
    detections = net.forward()
    return detections

@st.cache
def annotate_image(
    image, detections, confidence_threshold=DEFAULT_CONFIDENCE_THRESHOLD
):
    # loop over the detections
    (h, w) = image.shape[:2]
    labels = []
    for i in np.arange(0, detections.shape[2]):
        confidence = detections[0, 0, i, 2]

        if confidence > confidence_threshold:
            # extract the index of the class label from the `detections`,
            # then compute the (x, y)-coordinates of the bounding box for
            # the object
            idx = int(detections[0, 0, i, 1])
            box = detections[0, 0, i, 3:7] * np.array([w, h, w, h])
            (startX, startY, endX, endY) = box.astype("int")

            # display the prediction
            label = f"{CLASSES[idx]}: {round(confidence * 100, 2)}%"
            labels.append(label)
            cv2.rectangle(image, (startX, startY), (endX, endY), COLORS[idx], 2)
            y = startY - 15 if startY - 15 > 15 else startY + 15
            cv2.putText(
                image, label, (startX, y), cv2.FONT_HERSHEY_SIMPLEX, 0.5, COLORS[idx], 2
            )
    return image, labels

def main():
  selected_box = st.sidebar.selectbox(
    'Choose one of the following',
    ('Welcome', 'Object Detection')
    )
    
  if selected_box == 'Welcome':
      welcome()
  if selected_box == 'Object Detection':
      object_detection() 

def welcome():
  st.title('Object Detection using Streamlit')
  st.subheader('A simple app for object detection')
  st.image('test_images/demo.jpg',use_column_width=True)

def object_detection():
  
  st.title("Object detection with MobileNet SSD")

  confidence_threshold = st.sidebar.slider(
    "Confidence threshold", 0.0, 1.0, DEFAULT_CONFIDENCE_THRESHOLD, 0.05)

  st.sidebar.multiselect("Select object classes to include",
  options=CLASSES,
  default=CLASSES
  )

  img_file_buffer = st.file_uploader("Upload an image", type=["png", "jpg", "jpeg"])

  if img_file_buffer is not None:
      temp_file.write(img_file_buffer.getvalue())
      image = load_img(temp_file.name)
      image = img_to_array(image)
      image = image/255.0

  else:
      demo_image = DEMO_IMAGE
      image = np.array(Image.open(demo_image))

  detections = process_image(image)
  image, labels = annotate_image(image, detections, confidence_threshold)

  st.image(
      image, caption=f"Processed image", use_column_width=True,
  )

  st.write(labels)

main()

You can play with the live app *here. Source code is available here on Github.*

· 2 min read
Sparsh Agarwal

Live app​

This app can detect COCO 80-classes using three different models - Caffe MobileNet SSD, Yolo3-tiny, and Yolo3. It can also detect faces using two different models - SSD Res10 and OpenCV face detector. Yolo3-tiny can also detect fires.

/img/content-blog-raw-blog-object-detection-with-yolo3-untitled.png

/img/content-blog-raw-blog-object-detection-with-yolo3-untitled-1.png

Code​

import streamlit as st
import cv2
from PIL import Image
import numpy as np
import os

from tempfile import NamedTemporaryFile
from tensorflow.keras.preprocessing.image import img_to_array, load_img

temp_file = NamedTemporaryFile(delete=False)

DEFAULT_CONFIDENCE_THRESHOLD = 0.5
DEMO_IMAGE = "test_images/demo.jpg"
MODEL = "model/MobileNetSSD_deploy.caffemodel"
PROTOTXT = "model/MobileNetSSD_deploy.prototxt.txt"

CLASSES = [
    "background",
    "aeroplane",
    "bicycle",
    "bird",
    "boat",
    "bottle",
    "bus",
    "car",
    "cat",
    "chair",
    "cow",
    "diningtable",
    "dog",
    "horse",
    "motorbike",
    "person",
    "pottedplant",
    "sheep",
    "sofa",
    "train",
    "tvmonitor",
]
COLORS = np.random.uniform(0, 255, size=(len(CLASSES), 3))

@st.cache
def process_image(image):
    blob = cv2.dnn.blobFromImage(
        cv2.resize(image, (300, 300)), 0.007843, (300, 300), 127.5
    )
    net = cv2.dnn.readNetFromCaffe(PROTOTXT, MODEL)
    net.setInput(blob)
    detections = net.forward()
    return detections

@st.cache
def annotate_image(
    image, detections, confidence_threshold=DEFAULT_CONFIDENCE_THRESHOLD
):
    # loop over the detections
    (h, w) = image.shape[:2]
    labels = []
    for i in np.arange(0, detections.shape[2]):
        confidence = detections[0, 0, i, 2]

        if confidence > confidence_threshold:
            # extract the index of the class label from the `detections`,
            # then compute the (x, y)-coordinates of the bounding box for
            # the object
            idx = int(detections[0, 0, i, 1])
            box = detections[0, 0, i, 3:7] * np.array([w, h, w, h])
            (startX, startY, endX, endY) = box.astype("int")

            # display the prediction
            label = f"{CLASSES[idx]}: {round(confidence * 100, 2)}%"
            labels.append(label)
            cv2.rectangle(image, (startX, startY), (endX, endY), COLORS[idx], 2)
            y = startY - 15 if startY - 15 > 15 else startY + 15
            cv2.putText(
                image, label, (startX, y), cv2.FONT_HERSHEY_SIMPLEX, 0.5, COLORS[idx], 2
            )
    return image, labels

def main():
  selected_box = st.sidebar.selectbox(
    'Choose one of the following',
    ('Welcome', 'Object Detection')
    )
    
  if selected_box == 'Welcome':
      welcome()
  if selected_box == 'Object Detection':
      object_detection() 

def welcome():
  st.title('Object Detection using Streamlit')
  st.subheader('A simple app for object detection')
  st.image('test_images/demo.jpg',use_column_width=True)

def object_detection():
  
  st.title("Object detection with MobileNet SSD")

  confidence_threshold = st.sidebar.slider(
    "Confidence threshold", 0.0, 1.0, DEFAULT_CONFIDENCE_THRESHOLD, 0.05)

  st.sidebar.multiselect("Select object classes to include",
  options=CLASSES,
  default=CLASSES
  )

  img_file_buffer = st.file_uploader("Upload an image", type=["png", "jpg", "jpeg"])

  if img_file_buffer is not None:
      temp_file.write(img_file_buffer.getvalue())
      image = load_img(temp_file.name)
      image = img_to_array(image)
      image = image/255.0

  else:
      demo_image = DEMO_IMAGE
      image = np.array(Image.open(demo_image))

  detections = process_image(image)
  image, labels = annotate_image(image, detections, confidence_threshold)

  st.image(
      image, caption=f"Processed image", use_column_width=True,
  )

  st.write(labels)

main()

You can play with the live app *here. Source code is available here on Github.*

· One min read
Sparsh Agarwal

You can play with the live app here. Souce code is available here on Github.

Live app​

/img/content-blog-raw-mobilenet-ssd-caffe-pre-trained-model-untitled.png

Code​

#------------------------------------------------------#
# Import libraries
#------------------------------------------------------#

import datetime
import urllib
import time
import cv2 as cv
import streamlit as st

from plugins import Motion_Detection
from utils import GUI, AppManager, DataManager

#------------------------------------------------------#
#------------------------------------------------------#

def imageWebApp(guiParam):
    """
    """
    # Load the image according to the selected option
    conf = DataManager(guiParam)
    image = conf.load_image_or_video()
    
    # GUI
    switchProcessing = st.button('* Start Processing *')

    # Apply the selected plugin on the image
    bboxed_frame, output = AppManager(guiParam).process(image, True)

    # Display results
    st.image(bboxed_frame, channels="BGR",  use_column_width=True)

def main():
    """
    """
    # Get the parameter entered by the user from the GUI
    guiParam = GUI().getGuiParameters()

    # Check if the application if it is Empty
    if guiParam['appType'] == 'Image Applications':
        if guiParam["selectedApp"] is not 'Empty':
            imageWebApp(guiParam)

    else:
        raise st.ScriptRunner.StopException

#------------------------------------------------------#
#------------------------------------------------------#

if __name__ == "__main__":
    main()