Artificial Intelligence: From Task to Evaluation
Artificial intelligence builds systems that infer, generate or choose actions for a defined task.
- Author
- absequ
- Published
- Updated
- Category
- AI and data
- artificial intelligence
- fundamentals
- practical guide
Artificial intelligence builds systems that infer, generate or choose actions for a defined task. Start with the decision you need to support, the evidence available and the cost of being wrong. A useful AI feature needs an evaluation plan as much as a model.
On this page
Key concepts
AI includes approaches such as explicit rules, search and machine learning. Generative systems produce new outputs; predictive systems estimate outcomes. Neither fluent text nor a high confidence score establishes that an answer is correct. Keep the model distinct from the surrounding product, permissions and review process.
Practical example
For a support assistant, begin with a narrow task: draft an answer from approved help pages. Retrieve relevant passages, ask for a draft with references, then let a person approve it. Keep account changes outside the assistant until authorization and recovery paths are designed.
Create a synthetic evaluation set with answerable questions, ambiguous requests and questions absent from the help pages. Record whether each draft is supported by its references, whether it admits missing information and whether it exposes restricted information. Compare against a simple keyword search before adding complexity.
Common mistakes
Do not treat retrieved documents or user messages as trusted instructions. Limit tool permissions, avoid sending private information without authorization and provide a way to report failures. NIST frames AI risk work through Govern, Map, Measure and Manage: assign ownership, understand context, assess outcomes and act on findings.
Practice
Try changing one help page after evaluating the assistant. Which examples should change, and which should stay stable? Keep the evaluation cases versioned so improvements can be checked against earlier failures.