Improving AI Answer Quality: A Practical Guide

Patterns
1

Improving AI Answer Quality: A Practical Guide

Making your datasets AI-ready doesn’t need to be complicated. In fact, most of the steps are things you’re probably already doing to make data understandable to humans - now we’re just extending that same clarity to help the AI interpret it. Think of this as a quick tuning pass, not a full rebuild. You can make a meaningful impact in under an hour. Perfect is the enemy of good.

Here’s a tactical guide to help you get started.

1. Structure Your Data Like You Would for Humans

Good AI performance starts with solid, user-friendly data modeling. The goal is to make each dataset clear, focused, and include the most important context and logic - just like you would if you were handing it to a user in your company. In Omni, that means building well-scoped Topics. A Topic should represent a specific slice of your business logic and include just the fields, filters, and joins needed to answer common questions about that area.

Here’s what that looks like in practice:

  • Create subject-specific datasets. Keep each dataset focused and scoped to a clear business purpose.
  • Add joins so users (and the AI) knows what tables it can access.
  • Apply default filters to remove noise (e.g. exclude deleted records).
  • Set appropriate permissions. Omni’s AI always respects the user’s data permissions, so make sure row-, column-, or topic-level controls are in place if needed.
  • Hide extraneous fields like unused columns or foreign keys.
  • Use good field labels.
    • Instead of scheduled_task_id_count_distinct, label it “Number of Schedules”
  • Add field descriptions or pull directly from your warehouse or dbt. These help the AI disambiguate fields.
  • Add all_values for fields that commonly appear in filters.
    This helps the AI map user-friendly input to the actual values in your data. For example, if someone asks for “users in CA” but the underlying value in the column is “California,” providing all_values ensures the AI knows they’re the same.
  • Define a short overview of what the dataset is for - this is the ai_context parameter and it helps the AI choose the right dataset for each question.

2. Test and Tune the AI

Once your datasets are modeled well, the next step is to see how the AI performs in practice. We recommend collecting 10–20 real questions your users are already asking, and running them through the AI to see how it does. The best place to do this is on the Workbook. It’s easy to see exactly how the AI is formulating the answer (fields selected, filters applied, SQL), and adjust the answer/re-steer it as needed.

If the AI gets things wrong, you can immediately add some context to improve it, then try the question again. Here’s some examples:

Is it getting filter values wrong?

  • Example: User asked for total orders in CA. AI applied a filter on State = ‘CA’, but the actual values in the database are full state names (e.g. ‘California’)
  • Solve: Add all_values to the State field so it can match the user input to values in the database.

ai_values.png866×464 25.6 KB

Is it confused between similar fields?

  • Example: A user asks for “revenue,” and the AI picks the wrong field – maybe it chooses total_revenue instead of net_revenue.
  • Solve: Add field synonyms or a more explicit description to help guide selection. If you have duplicative or outdated fields, consider hiding or removing them to simplify the dataset.

synonyms.png876×230 11 KB

Is it picking the wrong dataset?

  • Example: A user asks about product inventory, but the AI chooses a marketing dataset because of overlapping field names.
  • Solve: Add more detail to the ai_context parameter on each topic, and include examples of real user questions to help the AI learn when each dataset should be used.

topic_context.png1288×280 32.1 KB

Is there hidden nuance in your business language?

  • Example: A user asks about “closed deals.” In your org, folks really mean deals that are both closed and won. But without that context, the AI is going to just filter on closed=true.
  • Solve: Clarify how common business terms are used in your ai_context, so the AI can apply your team’s language correctly to the data.
    Example: if asked about closed deals, most often the user means both closed and won.

topic_context2.png1324×314 29.8 KB

You can also use Omni’s built-in tools to help automate this tuning process. The Learn from Conversation feature can extrapolate business context, field definitions, synonyms, and more from an AI chat conversation. It’s a quick way to capture real user language and feed it back into your AI setup with minimal effort.

3. Monitor What People Are Asking

Once you’re up and running, check your prompt logs in the Analytics section regularly. You’ll learn a lot by seeing how people interact with AI and where it struggles.

Look for:

  • Questions the AI couldn’t answer - are there data gaps?
  • Repeated follow-ups or corrections - is the topic or fields missing critical context?
  • Business terms people are using - are there synonyms or preferences you should capture in your context?

session_logs.png1412×648 68.6 KB

Optimizing for AI doesn’t require a massive overhaul. In most cases, you’re just making the same improvements you’d make for any well-modeled BI experience - with a little extra metadata to help the AI connect the dots.

Start small, iterate quickly, and use real user behavior to guide what you improve next. Happy AI-ing!

4 Likes