AI Vendor Vetting Questions

Vendors love listing logos on their website. Ask to see the actual documents. A vendor with real compliance will hand them over under NDA without hesitation. One that stalls or deflects probably does not have what they claim.

AI models change constantly, and updates can shift accuracy, bias profiles, and output quality overnight. You need to know their update cadence, testing process, and notification policy. A good vendor gives advance notice and documents what changed.

Model cards, algorithmic impact assessments, or plain-language decision explanations are all acceptable. Vendors who say their model is proprietary and cannot be explained are asking you to trust a black box with your customers' outcomes.

Traditional incident response covers data breaches and outages. AI failures are different: biased outputs, hallucinated data, model drift. A mature vendor has a separate plan for AI-specific incidents, including detection, triage, notification, and remediation steps.

Insert the regulation that applies to your industry and state. If the vendor does not know the law you are referencing, that tells you a lot about how closely they are tracking the regulatory landscape they operate in.

Multiple upcoming regulations require AI impact assessments. Forward-thinking vendors already produce them. If a vendor has never conducted one, they are likely not evaluating the downstream risks of their own product.

Look for escalation paths, confidence thresholds that trigger human review, override capabilities, and audit logs. Any AI system making consequential decisions should have a human-in-the-loop option, not just a theoretical one, but a designed and documented one.