Hey Founders building AI features, what was the first question from a customer or partner that made you realize you needed better answers around AI usage, data, or security?
From simple to difficult questions
How many times our APIs were called due to the exposed MCP server.
How many agent invocations happened?
Token consumed per account (many users from the account)
Margin per account (MRR-$ tokens consumed - $cogs)
Great question. I see this topic being more and more important in the coming months. Gururaj P. How do you go about measuring these things?
Feels like teams either go really deep on metrics like Gururaj mentioned, overthink AI vendor security, or they don’t ask much at all yet. Part of it might just be that the expectations aren’t that clear yet, so people don’t know what “good” looks like.
I think even the frameworks haven’t quite caught up yet either, so people are kind of guessing what matters.
Yes! So looks like there are 3 modes right now: Some teams go deep on metrics (calls, tokens, margins), some over-index on vendor/security concerns, and a lot don’t ask much at all yet. The tricky part is there’s no shared baseline for what “acceptable” AI usage actually looks like yet, so teams are either guessing or reacting once someone asks. Curious what people think the first thing a team should be able to clearly explain about their AI usage is?
Most teams I talk to can explain it at a high level, but it falls apart when you go one level deeper. Curious what people are actually showing when they get pushed on it?
