Having Beacon requires ongoing attention and is a commitment to testing and maintaining the chatbot. It’s not just an AI you turn on and it’s good to go. Site owners will need to go through a testing process, engage their subject matter experts to review the accuracy of responses, and make improvements to content.
Start testing by asking Beacon any questions you believe it should be able to answer based on the content that can be found on your YaleSite.
Use a question bank to document which questions are being tested and the quality of the responses provided.
Evaluate whether the responses are accurate, complete, and helpful. The questions Beacon struggles with often reveal gaps or weaknesses in your website content that can affect human visitors too.
If you’re not satisfied with the responses go to the content’s location on the website and make the changes necessary. A consistent finding from piloting Beacon is when the chatbot gives poor responses, the root cause is usually content.
After making changes to the website’s content retest Beacon by asking it the previous questions to verify any improvements. Changes made to site can take a few minutes to be reflected in Beacon.
Content quality is not a one time achievement. It requires ongoing attention.
Regular reviews: Schedule periodic reviews of your content, especially pages that address policies or procedures that needs attention.
Monitor Beacon performance: Pay attention to the questions Beacon struggles with. They often indicate content that needs attention.
Update when things change: When policies, procedures, or contact information change, update your content promptly. Beacon can only be as current as your website.
Engage subject matter experts: The people who know your content best should be involved in reviewing and maintaining it, the better Beacon and your site’s visitors will be served.
Learn more about the support YaleSites provides for Beacon and what site owners are responsible for.