The gate is honest. So is the feedback.

Five public sample items

Representative of the bar. Not in the locked bank used for scoring. Click Reveal answer on each to see the answer and reasoning. Treat these as a calibration of whether the actual assessment is for you.

1. Sample 1Multiple choiceRAG diagnostics

   Users complain a RAG-powered product gives stale answers despite documents being updated weekly. The retrieval layer uses dense embeddings refreshed hourly. The LLM is Claude Sonnet 4.6.
   
   Which of the following is the MOST LIKELY cause?
   
   (a) Embedding model is too small
   (b) Embedding cache is keyed by document_id and is not invalidated when document content changes
   (c) Vector index dimensionality is too low
   (d) The LLM's knowledge cutoff predates the document update
   (e) Need to add a re-ranking layer

2. Sample 2Code review (multi-select)Production hygiene

   A junior engineer ships this code to production. Which of the following are problems? (Select all that apply.)
   
   ```python
   def chat(user_id, message):
       response = anthropic.messages.create(
           model="claude-sonnet-4-6",
           messages=[{"role": "user", "content": f"User {user_id} asks: {message}"}],
           max_tokens=4000,
       )
       return response.content[0].text
   ```
   
   (a) No system prompt — no role, boundaries, safety framing
   (b) Prompt-injection vulnerability via message interpolation
   (c) user_id is leaked into the prompt unnecessarily, with PII risk
   (d) No error handling for API failures, timeouts, or rate limits
   (e) response.content[0].text is fragile and assumes structure
   (f) max_tokens=4000 is excessive for typical chat and inflates cost
   (g) No streaming — poor UX
   (h) No observability or cost tracking per call

3. Sample 3Tradeoff scenario (short answer)Cost discipline

   Your LLM-based feature costs $1,200/day at 50k DAU. Roadmap requires it to scale to 500k DAU within 90 days at the same per-DAU cost (i.e., roughly $12,000/day). Without significantly degrading quality:
   
   Name your first three interventions, in priority order, with one-sentence justification each.

4. Sample 4Conceptual (short answer)Evals

   Your LLM-as-judge agrees with human labels at Cohen's kappa = 0.42. What does this mean, and what should you do? (≤100 words.)

5. Sample 5Failure mode (short answer)Agents

   A multi-agent product is occasionally fabricating tool outputs that don't match the actual tool's response. The user-facing answer is wrong as a result. What's most likely happening, and how would you debug? (≤150 words.)