AI Security Engineer Interview Questions

Technical Questions: Adversarial ML

1. Explain the difference between evasion attacks and poisoning attacks on ML models.

Evasion attacks manipulate inputs at inference time to cause misclassification. Poisoning attacks corrupt training data to alter model behavior. The key distinction is timing: evasion happens after training, poisoning happens during training. A strong answer includes examples of each and discusses how defense strategies differ.

2. How would you detect a data poisoning attack on a model training pipeline?

Discuss statistical analysis of training data distributions, anomaly detection on feature values, monitoring model behavior for unexpected changes during training, data provenance tracking, and differential testing (training with and without suspicious data subsets). Mention that detection is harder for clean-label poisoning attacks where the poisoned samples appear correctly labeled.

3. What is model extraction and how would you prevent it?

Model extraction uses API queries to reconstruct a model's behavior or weights. Prevention includes rate limiting, query pattern analysis, output perturbation (adding calibrated noise to predictions), monitoring for systematic querying patterns, and watermarking model outputs to detect stolen models in production elsewhere.

4. Describe three different types of prompt injection attacks.

Direct injection (user provides malicious instructions in input), indirect injection (malicious instructions embedded in content the model processes from external sources), and prompt leaking (extracting the system prompt). A strong answer includes attack examples and discusses why each is difficult to defend against.

5. How does adversarial training work and what are its limitations?

Adversarial training augments the training dataset with adversarial examples, improving model robustness against known attack types. Limitations include computational cost (2x to 10x training time), reduced accuracy on clean inputs, limited generalization to unseen attack types, and the ongoing arms race between new attacks and defensive training.

Technical Questions: Security Fundamentals

6. How would you threat model an LLM-powered chatbot?

Walk through STRIDE or similar framework applied to the LLM application. Identify assets (model weights, system prompt, user data), threat actors (external users, compromised data sources), attack vectors (prompt injection, data extraction, denial of service), and trust boundaries (user input, external data retrieval, tool use). A strong answer discusses both AI-specific and traditional application security threats.

7. Explain the principle of least privilege applied to an AI agent system.

AI agents with tool use should have minimal permissions for each tool. A coding assistant should not have email access. A customer service bot should not have database write permissions. Discuss how to implement capability restrictions, approval gates for high-risk actions, and sandboxing for code execution.

8. What is the MITRE ATLAS framework and how does it differ from MITRE ATT&CK?

ATLAS (Adversarial Threat Landscape for AI Systems) catalogs attack techniques specific to ML systems, while ATT&CK covers traditional cybersecurity attack techniques. ATLAS includes techniques like model evasion, model poisoning, and ML supply chain attacks that are not covered by ATT&CK. Discuss how both frameworks complement each other for comprehensive threat coverage.

9. How would you implement a defense-in-depth strategy for an LLM application?

Layer defenses: input validation and sanitization, system prompt hardening, output filtering and classification, rate limiting and abuse detection, monitoring and logging for post-hoc analysis, and architectural isolation (separating LLM processing from sensitive actions). Discuss why no single layer is sufficient.

10. What security considerations apply to fine-tuning a model on customer data?

Data isolation between customer tenants, preventing training data memorization and extraction, model access controls post-fine-tuning, data retention and deletion compliance, and ensuring fine-tuning does not degrade safety properties of the base model. Discuss differential privacy as a technical control.

System Design Questions

11. Design a secure ML model deployment pipeline.

Cover model provenance verification (signed model artifacts), automated security scanning (backdoor detection, adversarial robustness testing), staged rollout with canary deployments, runtime monitoring (input/output logging, anomaly detection), access control for model registry, and rollback capabilities. Draw the architecture and discuss trust boundaries.

12. How would you architect a prompt injection detection system?

Discuss multi-layer approach: keyword-based initial filtering, ML classifier trained on known injection patterns, semantic similarity checking against the system prompt, output analysis for instruction-following behavior, and continuous learning from false positives and negatives. Address latency requirements and false positive tradeoffs.

13. Design a model monitoring system that detects adversarial attacks in production.

Cover input distribution monitoring (statistical tests for drift), model confidence analysis (unusual prediction patterns), output consistency checks (comparing model behavior across similar inputs), rate-based anomaly detection (query patterns consistent with extraction), and alerting and escalation workflows. Discuss the tradeoff between detection sensitivity and alert fatigue.

14. How would you secure a multi-tenant AI platform where customers deploy their own models?

Cover tenant isolation (compute, storage, networking), model scanning on upload (malicious model detection), inference isolation (preventing cross-tenant data leakage), resource limits (preventing abuse), API authentication and authorization, and audit logging. Discuss shared responsibility boundaries between platform and tenant.

Behavioral and Scenario Questions

15. A product team wants to ship an LLM feature next week but you have not completed the security review. What do you do?

Prioritize the highest-risk aspects of the review. Communicate the risks of shipping without a full review. Propose a phased approach: ship with additional monitoring and restrictions, complete the full review post-launch, and expand capabilities as security confidence increases. Demonstrate collaboration, not obstruction.

16. You discover that a deployed model is memorizing and potentially leaking training data. How do you respond?

Immediate assessment of the scope and sensitivity of leaked data. Determine if the data includes PII or confidential information. Implement output filtering to block known leakage patterns while a permanent fix is developed. Notify relevant stakeholders (privacy team, legal, affected data owners). Retrain or fine-tune the model with differential privacy controls. Document the incident and update security review procedures.

17. How would you prioritize AI security investments for a company just starting its AI security program?

Start with the highest-impact, lowest-effort controls: input validation for LLM applications, rate limiting on model APIs, basic monitoring and logging. Then build toward adversarial testing capabilities, model supply chain security, and compliance frameworks. Discuss how to make the case to leadership for investment using risk quantification.

18. Describe a situation where you had to explain a complex technical security issue to a non-technical audience.

Use a real example. Structure the answer as: situation, what made it complex, how you translated it into business terms, and the outcome. Emphasize using analogies, quantifying risk in business terms (cost, probability, impact), and tailoring the message to the audience's priorities.

Advanced Technical Questions

19. How would you detect a backdoor in a pre-trained model from Hugging Face?

Discuss Neural Cleanse (optimizing for minimal input perturbations that cause targeted misclassification), activation analysis (looking for neurons that activate strongly on specific trigger patterns), meta-neural analysis (training a classifier to distinguish clean from backdoored models), and behavioral testing with held-out test sets. Acknowledge that no method is foolproof and defense requires multiple approaches.

20. Explain membership inference attacks and how to defend against them.

Membership inference determines whether a specific data point was in the training set by exploiting the model's higher confidence on training data. Defenses include differential privacy during training, regularization techniques that reduce overfitting, output perturbation, and restricting access to detailed model confidence scores. Discuss the privacy implications in healthcare and financial contexts.

21. What is the Carlini and Wagner attack and why is it significant?

The C&W attack is an optimization-based adversarial example method that minimizes the perturbation needed to cause misclassification. It is significant because it showed that defensive distillation (an early defense) was ineffective, and it remains a strong baseline for evaluating adversarial robustness. Discuss L0, L2, and Linf variants.

22. How would you evaluate the safety of a model before deploying it to production?

Cover automated red teaming (systematic adversarial testing across categories), safety benchmarks (TruthfulQA, BBQ for bias, HarmBench), manual evaluation by domain experts, differential testing against previous model versions, and monitoring during staged rollout. Discuss how evaluation criteria should be aligned with the specific deployment context and risk profile.