Introducing Confidence Score
We're pleased to announce the launch of our highly-anticipated Confidence Score feature in Waii 1.20. This powerful new capability let's product engineering teams to build more sophisticated, context-aware AI-powered data products.
What is the Confidence Score?
The Confidence Score is a predictive metric that assesses the likelihood that the generated SQL query or response precisely matches the user's intent. This score is calculated by analyzing multiple factors, including:
- Query and instruction complexity
- Alignment with the knowledge graph and semantic context
- Correlation with historical queries and responses
Our testing process, involving hundreds of databases, thousands of tables, and a diverse array of queries, has demonstrated the accuracy and reliability of this score across various domains.
Why it matters
The Confidence Score opens up new possibilities for creating nuanced, user-centric experiences in your data products. It allows you to fine-tune your application's behavior based on:
1. User personas and their risk tolerance
2. The criticality of the data being queried
3. Specific use cases and compliance requirements
How you can leverage the Confidence Score
1. Risk management: Implement thresholds to block low-confidence responses in critical scenarios.
2. Enhanced UX: Trigger additional review processes or cross-checking for moderate-confidence results.
3. Contextual enrichment: Automatically supplement responses with supporting information when confidence is below a certain level.
Real-world application
Consider a compliance analyst working in a highly regulated industry. Their need for accuracy and reliability is paramount. With the Confidence Score, you can:
- Set a high confidence threshold (e.g., 95%) for automatic query execution
- Implement a mandatory review process for queries scoring between 80-95%
- Require manual query construction or expert intervention for scores below 80%
This approach ensures that the compliance analyst always works with the most reliable data, reducing risk and enhancing overall efficiency.
The introduction of the Confidence Score demonstrates our commitment to empowering your team to build more intelligent, adaptive, and trustworthy AI-powered data products. We're excited to see how you'll leverage this feature to create exceptional user experiences and drive business value.