IVRobot – An IVR-based ChatBot for Small Business Owners

IVR-based chatbot that can easily be used by small business owners to improve their digital presence, attract more customers, and enhance their customer support

Product Description

An integrated voice chatbot that can easily be used by small business owners to improve their digital presence, attract more customers and enhance their customer support.

Problems the product intends to solve     

  • Enhance digital presence for small businesses
  • Automate answering of popular/repeat queries 
  • Save time spent in answering calls by in-store staff thereby increasing productivity
  • Provide easy online customer service for small stores which can increase their customer volume and loyalty

Target customers/users

  • Customer: Small shop owners
  • User: People calling these shops

Why is this an important problem?

On one hand we have people who prefer calling up before visiting the store of their choice.  Based on user interviews, we discovered that these users are spread across different age groups and call because of any/all of the following:

  • They don’t trust what is on the store website (if there is any)
  • They like to get a real-time status but calling in
  • They don’t like text based chatbots 

On the other end, we spoke to the store owners who highlighted that during Covid-19 times, the incoming calls have increased. As a result, the amount of time store staff spend on these calls, hampers productivity. Most of the inquiries are repetitive. We have collected this information and mapped it into features of the IVRobot. There are 3 types of inquiries that are answered: Store Open hours, ongoing discounts, specific product availability with an option of talking to the store staff in case callers have additional questions. 


  • As a caller, they do not have to dial multiple extensions (like in traditional IVR systems) to get their query answered
  • As a SMB owner, they get more time to do the actual work in running the shop without losing out on digital interactions with their customer base
  • Analytics and customizable console mean store owners can derive actionable insights on their customer behavior



A IVR (interactive voice response) system using Twilio’s Programmable API that get user’s inquery and send it to endpoint then reply as the endpoint’s response.


A backend that handle user’s inquery based on store’s information (discounts/promotions, open/close hours, product availability, etc.)


A Python NLP engine to verify user’s inquery (e.g. product availability)


A store owner console that manage the customers’ call and analyze of it (e.g. # of calls, popular inquery, etc)



  • Node.js/Express
  • MongoDB
  • Twilio API
  • ngrok (for the testing localhost to public url)

API Integration

  • Google Speech-to-Text API (Speech-to-text)
  • Twilio Programmable Voice API (Text-to-speech)




Step-by-step Installation

Enable Google Cloud API

To use the speech-to-text, you should have a Google Cloud API key. We’re using Google Cloud’s Speech-to-Text to transcribe speech (especially phone audio) coming from Twilio, a leading cloud communications PaaS. We’re discovered that it’s now easier than ever to integrate live call data with Google Cloud’s Speech-to-Text using Twilio’s Media Streams.

The new TwiML <stream> command streams call audio to a websocket server. This makes it simple to move your call audio from your business phone system into an AI platform that can transcribe that data in real time and use it for use cases like helping contact center agents and admins, as well as store it for later analysis. 

When you combine this new functionality with Google Cloud’s Speech-to-Text abilities and other infrastructure and analytics tools like BigQuery, you can create an extremely scalable, reliable and accurate way of getting more value from your audio.


The overall architecture for creating this flow looks something like what you see below (We modified this architecture without using GCP.) Twilio creates and manages the inbound phone number. Twilio’s new Stream command takes the audio from an incoming phone call and sends it to a configured websocket which runs on a simple App Engine flexible environment. From there, sending the audio along as it comes to Cloud Speech-to-Text is not very challenging. Once a transcript is created, it’s stored in BigQuery (in our project, we directly receive it to our IVR system) where real-time analysis can be performed.

To use speech-to-text API, you must have a valid Google Cloud API that requires a credit card number but it will provide you $300 free credit. After you have a GCP account, create a project, then go to the API browser and Enable the Speech-to-Text API for that project.

Afterward, create a service account. Then, download a private key as JSON. You can view and manage these resources at any time in the Cloud Console. Set the environment variable GOOGLE_APPLICATION_CREDENTIALS to the path of the JSON file that contains your service account key. This variable only applies to your current shell session, so if you open a new session, set the variable again. We’ll use this JSON key in our IVR.

Twilio ML

You should have an account of Twilio. Once you sign up as a developer, Twilio provides you a test phone number. Check https://www.twilio.com/console and sign up. Then, you can have Twilio’s phone number. For that, go developer console and add phone number (that will charge you $1 per month) 


Ngrok is a Go program, distributed as a single executable file for all major desktop platforms.  This is super rad – no additional frameworks to install or other dependencies.  Grab the version for your development system of choice and simply unzip the file somewhere on your computer. Download: https://ngrok.com/download 

In a Terminal window, navigate to the directory where you unzipped ngrok. Going forward you may want to add ngrok to your system path. That would allow you to access the ngrok command from any directory, but for now we will run the ngrok utility directly from this location.

We now need to start ngrok, telling it which port we want to expose to the public Internet. In the Terminal, type ./ngrok http 3000 on Linux and Mac, and just ngrok http 3000 on Windows. After starting up, you should see something like the following:


See that URL on the ngrok domain? That’s the brand new home of your local web app on the public Internet. In a browser window, open up https://[your generated ID].ngrok.io/voice – this should display the same XML found at http://localhost:3000/voice – verify this is the case.

Now, we can configure our Twilio number to hit our laptop when it receives an inbound call or text. In your Twilio account dashboard, go to your list of incoming numbers and choose one to work with. Enter your ngrok URL, including the “/voice” route, in the Messaging Request URL field as shown. Also, make sure to change the HTTP request type to a “GET” rather than a “POST”. Save your changes.

Go to Twilio Account -> Programmable Voice API -> Click your number


Firebase (Testing only)

We mainly use MongoDB for our real version. But for testing purpose, you may use Firebase for convenience. If you have a Google Cloud account, you may have a Firebase account easily. Go to firebase.google.com and create project, then add apps in Project Overview -> Settings

Click Add app and create a key, then keep your config file like as below:

We’ll use it for IVR and Store Owner Console.

MongoDB (Real)

You should have mongodb instance either local one or Mongodb Atlas (free resource is available. Check https://www.mongodb.com/cloud/atlas ) Once you get it, please change the mongoDbUri (in the index.ts for IVR, Endpoint, and Store Console) 

Run IVR system

We will provide a docker compose file but if you want to run without dockerized ones, you can run our system as separately.

First, clone our repository

You have to run those things: IVR, endpoint, store-owner-console


IVR is our main router handling inquiry from Twilio and replies whatever we ask. Please follow as below:

  1. Go to ivr folder, and run npm install
  2. Go ivr > handler.js file, then change json file to your firebase key (you can download GCP console)

Now you can have your IVR router as default port of 3000.


Endpoint handles the REST request regarding merchant, customer, inquiries, promotions, and items. Please follow as below:

  1. Go to endpoint folder, and run “npm install”
  2. Go to src > index.ts and change mongoDbUri to your own mongoDB connection string.
  1. Run npm start

Now you can have your IVR endpoint at localhost with a default port of 3001.

Regex matching rules for incoming inquiry:

  • If the user inquiry contains keyword “menu”, return menu response
  • If the user inquiry contains keywords “discount” or “promotion”, return promotion response
  • If the user inquiry contains keywords “open”, “close”, “hour” or “hours”, return store hour response
  • If the user inquiry contains keyword “have”, return store inventory response. 

NLP server

  1. Go to /nlp folder
  2. Install pip by running “python get-pip.py”, and use pip to install flask and nltk
  3. Run “python proper_noun_api.py” and the nlp server will start at 

Store Owner Console

Store owner console is an integrated dashboard that can handle your customer and store information for the store owner only. To run, please follow the instructions:

  1. Go to endpoint folder, and run “npm install”
  2. Go to src > firebase.js and change firebase config as your one.
  1. Run npm start

Now you can have a store console. Go http://localhost:3000/ and you can see login screen:

To use it, you must first register your store account.

The most important part is you should add your store phone number and Twilio Phone Number. 


  • MIT / Carnegie Mellon University – Silicon Valley


  • Ikwhan Chang
  • Manuja Gokulan
  • Wei Li