Please visit the below blog for referring to the Original article.
https://medium.com/@naveen.vijapurapu/diy-build-your-own-jarvis-7fae3801c49
Credit — Marvel Studios
- Motivation
- Problem Statement
- Introduction
- Different Types of Chatbots –
- Metrics Used/ Referenced:
- Improvements and Results:
- Dialogflow
- ChatterBot
- RASA
- Slack and IBM Watson Movie Recommendation System
- Conclusion:
* Future Improvements:
* These are my bot examples:
- References:
Conversational AI is re-defining the future of Customer Service. I wanted to explore on the options that are available for deploying chatbots and help answer the below questions to help make a difference for start-ups and smaller companies to compete with larger organizations –
- What is a Chatbot and why is it becoming a Big Deal?
- How can it be made accessible to all for deploying a chatbot to improve the Customer Satisfaction Index.
- Are bot messages the new email, when it comes to generating traffic and leads? (Spoiler – Research and Results say ‘Yes!’)
What are the options available for creating Personal bots? No single source of information is easily available for comparing the options available for creating bots Customer wait times through traditional methods and IVRS have long wait times and cause to bad Customer experience and frustration. How can we look at ways to accelerate and improve customer service by establishing an innovative messaging service across multiple digital platforms
It is no longer just a figment of Scify imagination on the big movie screens to have an Artificial Intelligent Robot similar to Jarvis for IronMan replying to your queries. And NO, you don’t have to be wealthy rich like Tony Stark either to own one (or one of the big MNC’s). Sounds interesting?? then please continue reading..
Below we go through some of the popular options for creating AI Chatbots.
But first, let us understand the architecture and the inner working details of a Chatbot.
Process flow of NLP (Natural Language Processing) Pipeline:
Most of the chatbots follow the below structure to identify the intent and entity classifications which is fed in to a pre-trained machine learning model to predict suitable responses to the input query. The steps are as follows:
- Text Processing (Input & Output)
- Cleaning
- Normalization
- Tokenization
- Stop Word Removal
- Part of Speech Tagging
- Named Entity Recognition
- Stemming and Lemmatization
- Feature Extraction
- Bag of Words
- TF-IDF (Term Frequency — Inverse Document Frequency)
- Word Embeddings
- Modeling
- The final stage of the NLP pipeline is modeling, which includes designing a statistical or machine learning model, fitting its parameters to training data, using an optimization procedure, and then using it to make predictions about unseen data.
- The final stage of the NLP pipeline is modeling, which includes designing a statistical or machine learning model, fitting its parameters to training data, using an optimization procedure, and then using it to make predictions about unseen data.
Credit — Wall-E & Eva characters are Copyrighted by Disney Pixar Studios
- Contextual AI- (Set of Rules or State Machines)
- Notifications (Automated messages on expiry or for renewal)
- FAQ Chatbots (Hardcoded commonly asked FAQ’s)
- Conversational AI & Types-
- Rule Based ChatBot
- Retrieval Based (TF-IDF)
- Few of the frequently used models for Chatbots –
- CNN (Convolutional Neural Networks)
- DNN (Neural Networks)
- Few of the frequently used models for Chatbots –
- Generative Based
- Deep Learning
- RNN (Recurrent Neural Networks)
Now that we understood the inner workings of a chatbot, let us now compare the three poplar options available and used in the industry based on varying degrees of Ease of Use versus Open Source.
We will go over Diaglogflow, Chatterbot & RASA as our three options to build chatbots.
- RNN (Recurrent Neural Networks)
### Side by side comparison
Now as we can clearly see, there is no one single solution that fits all, we will go over the details of each of the options and leave it to you to pick the right tool for your needs!
The below metrics can be considered while deploying a chatbot and analysing the gains
- Customer Satisfaction
- Reduction in Customer Wait Times
- Effective Utilization of key personnel
- Retention Rate - Effectiveness through Engagement and Re-engagement
- Reduction in repetitive FAQ's asked freeing up employee's time for other critical tasks
- Repetitive tasks and human intervention
- Market Research on Metrics–
- When tested whether its chatbot or email marketing drove more traffic to its website, the results were staggering. On average, the bot had an 84% “read” rate and a 53% CTR – 1,428% higher engagement rate than the email funnel.
- Though its email content and bot content were both super-personalized, bot appeared to have avoided the spam filters and “inbox fatigue” that led to the lower email metrics.
- Automated qualification and immediate response time increased new leads by 70% and new opportunities by 170% in three months.
- Improved Customer Service - Improved Customer Engagement - 6-8 minute average conversations - 11 turns per conversation - 50% user re-engagement - Always Available (24 X 7) - Reduction in wait times for Customers - FAQ Chatbot Reduces Customer Support Emails by 83% - Data for insights and lead generation - Van Leasing Sales Chatbot Generates 30,000+ Leads - Scalability - Cost Savings - Bot saves 950 minutes of Human Time per day - Enables smaller start-ups to compete against larger more established Organizations - Chatbots can be a great back up for employees for relatively basic and repetitive tasks being cost-effective, easy to implement, maintain and use. - Recruitment Chatbot Reduces Unqualified Job Applications by 73%
Courtesy:
Dialogflow (formerly Api.ai, Speaktoit) is a Google-owned developer of human–computer interaction technologies based on natural language conversations.
Gives users new ways to interact with your product by building engaging voice and text-based conversational interfaces, such as voice apps and chatbots, powered by AI. Connect with users on your website, mobile app, the Google Assistant, Amazon Alexa, Facebook Messenger, and other popular platforms and devices.
Dialogflow is an end-to-end, build-once deploy-everywhere development suite for creating conversational interfaces for websites, mobile applications, popular messaging platforms, and IoT devices.
Natural language understanding recognizes a user’s intent and extracts prebuilt entities such as time, date, and numbers. You can train your agent to identify custom entity types by providing a small dataset of examples. You can also use 40+ prebuilt agents as templates.
In September 2014, Speaktoit released api.ai (the voice-enabling engine that powers Assistant) to third-party developers, allowing the addition of voice interfaces to apps based on Android, iOS, HTML5, and Cordova. The SDK’s contain voice recognition, natural language understanding, and text-to-speech. api.ai offers a web interface to build and test conversation scenarios. The platform is based on the natural language processing engine built by Speaktoit for its Assistant application. Api.ai allows Internet of Things developers to include natural language voice interfaces in their products.
Dialogflow now does sentiment analysis for each user query, powered by Cloud Natural Language. Sentiment analysis scores can be used to hand off unsatisfied users to live agents, or to get a better understanding of which intents lead to the highest customer sentiment.
References: https://www.diagflow.com
Wikipedia: Dialogflow
is a machine-learning based conversational dialog engine build in Python which makes it possible to generate responses based on collections of known conversations. The language independent design of ChatterBot allows it to be trained to speak any language.
How it works
An untrained instance of ChatterBot starts off with no knowledge of how to communicate. Each time a user enters a statement, the library saves the text that they entered and the text that the statement was in response to. As ChatterBot receives more input the number of responses that it can reply and the accuracy of each response in relation to the input statement increase. The program selects the closest matching response by searching for the closest matching known statement that matches the input, it then returns the most likely response to that statement based on how frequently each response is issued by the people the bot communicates with.
References:
https://github.com/gunthercox/ChatterBot
https://chatterbot.readthedocs.io/en/stable/
__RASA__ is a set of open source Machine learning tools for developers for Conversational AI. An open source machine learning framework to automate text-and voice-based conversations.
Rasa’s primary purpose is to help you build contextual, layered conversations with lots of back-and-forth. To have a real conversation, you need to have some memory and build on things that were said earlier. Rasa lets you do that in a scalable way.
Rasa contains two main components:
* NLU — Natural Language Understanding for Intent Classification and Entity extraction
* Takes unstructured data and converts them to structured data on the form of intents and entities
* Core — Framework for machine learning based, contextual decision making. Dialog Management component. Brains of the Conversational AI
Tutorial:
https://rasa.com/docs/rasa/user-guide/rasa-tutorial
Try RASA
Reference: https://rasa.com/
GitHub: https://github.com/rasaHQ/
This bot has been created as a separate reposistory to keep the folder structure clean and un-cluttered.
#### Please refer the below repo for the code implementation and installation instructions. https://github.com/VijapurapuS/Chatbot/tree/master/bot
Data Exploration, Pre-processing, Results and Future improvements will be covered here for completeness of the article and Project report.
One important thing to note with this design is that, the data and processing is all handled in the local system. Even though we use IBM, it is used as an API service and none of the internal data is sent to IBM. This way the entire design can be implemented in your workplace without having to worry about data transfers.
Users can interact with the bot via Slack. Once the user post a question, it is passed to the backend system for analysis.
All the natural language processing happens in step 2. This includes IBM Watson processing, similarity search, recommendation based on collaborative filtering. After the NLP processing is completed, we have three outputs from it Intents — What the user is trying to ask or query? Entities — What is the exact field or column they are looking for? Dialog/Interaction — Provide the appropriate request/response for the user question.
The results obtained from the backend is posted to user via Slack
Installation and Bot Setup Implementation & Refinement from Other bots created (DiagFlow & Chatterbot)
This file will walk you through the steps to setup your bot. Download the entire folder and the follow the steps below.
Step 1: Create Slack Bot user
Step 2: Create a IBM Watson account and Upload the bot.json workspace
Step 3: Install the required packages listed in the requirements.txt file. To install the required packages, please use the code below.
pip3 install -r requirements.txt
It would be recommended to use Python 3.5.x or 3.6.x version for this project.
Step 4: Update the config files with the Slack and Watson API details Please make sure that you modified the API details both for Slack and Watson in the config.py file
Step 5: Download data from source and perform Data Preparation. The data for this example is downloaded from the location below:
https://www.kaggle.com/rounakbanik/movie-recommender-systems/data
Name of the dataset - movies_metadata.csv
"metadata_prep.csv" will be created after you run the data preparation code which will be later used in nlp models to train the movie recommendation system.
Step 6: Create "onetime.txt" file Navigate to the folder where the main.py file resides and execute the code below.
python3 nlp/nlp_solutions/onetime_run_file.py
This will create the "onetime.txt" file automatically.
Step 7: Initiate Bot Navigate to the folder where the main python script exists and run the code below.
python3 main.py
Please refer to the data folder under the repo - https://github.com/VijapurapuS/Chatbot/tree/master/bot for code to follow along.
The data for this exercise is taken from the Kaggle link below. The name of the dataset is “movies_metadata.csv”. https://www.kaggle.com/rounakbanik/movie-recommender-systems/data?source=post_page-----7c69af7a7439----------------------
The dataset contains a lot of information related to movies with less preprocessing required from users. Hence I chose it for training the bot.
Data_Preparation.ipynb https://github.com/VijapurapuS/Chatbot/blob/master/bot/data/Data_Preparation.ipynb
- Chatscript
- Chattypeople
- Pandorabots
- Botsify
- Rebot.me
Hope this article has simplified the process for creating your own bot! Message me in the comments with your example bots for interaction.
- Improvement based on the language of the film
- Extend recommendations based on actors
- Collaborative user based recommendations
- Chatterbot — https://shankachakra.pythonanywhere.com/
- Dialogflow bot https://bot.dialogflow.com/dialogflow-demo
Udacity Nano Degree for Data Scientist
Searches on stackoverflow.com for research
https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet#code
https://www.kaggle.com/rounakbanik/the-movies-dataset