Powering up first line support with AI

Conversation Analysis

Central to creating an effective and reliable AI chatbot is ensuring the answers provided are accurate but also that they are delivered in a natural dialogue flow. To quality check the system we prepared collections of tasks for users to preform against the system. These chat sessions were tracked and reviewed by subject matter experts for accuracy of response and by our build team to ensure the dialogue flow was on point. Where we found deviations we were able to track back through the engine to determine where additional source data was required or where inconsistencies or contrary data existed. We found key to this was to run the same tests multiple times against the ChatBot, this allowed us to identify areas where the same question may be answered in different ways, these pointed to contrary data in our dataset to be eliminated. Key takeaways from this were to ensure web pages, when used as a source, must be manually scrubbed for hidden or commented out content, these old development patches can cause issues. To ensure we could find the source of inconsistencies we strictly tracked the source for each data point when scraping and vectorizing our public site data.

Prompt Engineering

As we push further into AI development and AI systems, prompt engineering is becoming more critical to generating effective systems. Through prompt engineering the system can be tailored to improve responses and results. In ChatBots, prompts are used to define what the bot is to respond to, how it should respond and to provide it with boundaries to which it can and, importantly, cannot respond to. Like all things in programming, short cycles through the DESIGN > BUILD > TEST > REFINE process are key to prompts. Regular testing is key to good prompt engineering especially as new functionality and content is added to the system.

Animation for Effect

In order to humanize the system we provided some simple “moments-of-joy” in the interactive process by creating an animated application visual. In application design, simple visual queues can be used to improve the user acceptance of the app as well as ensure visual attention is drawn at the correct moments in time. Our animated action provided these simple key transformations:

• Wave animation draws attention when we detect user has paused for an extended period of time on page
• Transition to quesiton mark icon when clarifying with user.
• Track icon eyes to pop-over modal when expanding to draw attention.

Design Systems

Even in a simple interface with minimal design elements, following a design system is key. In this case we worked to a design pattern based on multiples of eights ensuring padding margines and rounding behaviours followed. This provides a visually appealing look to the interface and can work to improve user satisfaction raitings.

Deliverables

One key to this project was to ensure extensibility, we wanted a solution that would both meet the clients needs today as well as provide a solid base to grow. Thus this solution was build with a SaaS model in mind with us providing the following components to the client:

• Hosted scalable infrastructure for Service
• Javascript integration code for existing website
• End points for LLM training and tuning
• Support materials and Documentation

Learnings

Fundamental understanding of how AI “magic” works can be difficult to express to non technical audiences. Due to this it is very important to ensure users have a sense for what they are communicated with be it human or AI. Avoidance of the conversation loop is key to avoid user frustration.

Additionally the quality of production code and structure can be a factor if using website content as a source for training AI systems. Hidden elements can easily creep in and influence the data pool.