In today’s world of conversational AI, the ability to accurately identify and extract information from messages is paramount. This is where Named Entity Recognition (NER) comes into play. Today, we’ll explore an open-source framework called Chatbot NER designed specifically for conversational AI, particularly in handling Indian languages.

What is Chatbot NER?

Chatbot NER is an open-source framework specifically tailored for entity recognition in text messages. Developed by the Haptik team, it emerged from extensive research into existing NER systems, highlighting the need for a framework that could effectively support Indian languages. Currently, it encompasses languages like English, Hindi, Gujarati, Marathi, Bengali, and Tamil, along with their code-mixed forms.

Installation

Setting up Chatbot NER on your system using Docker is a breeze. For detailed instructions, please check out the installation documentation.

Supported Entities

The framework can identify a variety of entity types:

• Time: Example – “tomorrow morning at 5” (Supports en, hi, gu, bn, mr, ta).
• Date: Example – “next Monday” (Supports en, hi, gu, bn, mr, ta).
• Number: Example – “50 rs per person” (Supports en, hi, gu, bn, mr, ta).
• Phone Number: Example – “9833530536” (Supports en, hi, gu, bn, mr, ta).
• Email: Example – “hello@haptik.co” (Supports en).
• Text: Example – “Order me a pizza” (Supports en, hi, gu, bn, mr, ta).

For more specialized detection such as PNR codes and custom regex, reference the respective types provided in the documentation.

API Structure

The API for Chatbot NER is designed for easy integration with conversational AI applications. It’s also versatile enough to be used in other applications. For further details, explore the API documentation.

Framework Overview

In conversational AI, we have various entities to recognize, each requiring unique detection logic. The repository organizes entities into four types:

• Numeral: Includes number detections and size.
• Pattern: Identification based on regular expressions (e.g., emails, phone numbers).
• Temporal: Focuses on detecting time and dates.
• Textual: Relies on dictionary lookups for various entities.

To make it user-friendly, the numeral, temporal, and pattern categories have been upgraded to support multiple languages effectively.

Contributing to Chatbot NER

If you’re interested in contributing to Chatbot NER, there are two primary ways:

• Adding Training Data: Enhance the framework’s capabilities by adding data via CSV files.
• Adding Detection Patterns: You can introduce custom patterns for different languages through simple functions.

Refer to the official guidelines for more detailed steps on contributing.

Troubleshooting Common Issues

While using Chatbot NER, you might encounter some common issues. Here are a few troubleshooting ideas:

• Check if your Docker setup matches the installation guidelines provided in the documentation.
• Ensure that the required language packs are correctly added if you’re working with specific Indian languages.
• If you’re facing API integration issues, validate your implementation against the provided API documentation.
• Review any error logs to identify specific problems with entity detection.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

Final Thoughts

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.