Choosing a partner

Choosing an AI delivery partner is a procurement decision as much as a technical one.

Five routes for staffing AI work and 24 questions to ask any of them. Five named ways an engagement fails. The cases where we are the wrong choice. Copy the questions into your own RFP.

Updated October 20, 2026 First published August 8, 2026

27 min read

Markdown

Every guide on this site so far is written for an engineer deciding how to build something. This one is for whoever signs the contract and decides who builds it. The options are to hire in-house, or use a big consultancy, an offshore dev shop, a freelancer, or a specialist studio. People make that decision all the time. They rarely get a real checklist for it.

This page is that checklist. It covers five routes, written straight, including the three we compete against. It lists twenty four questions any of them should answer, with what a passing answer contains and what a failing one sounds like. It also shows where we fit and the plain cases where we do not.

The five routes

Five ways to staff this, compared plainly.#

Each one has a real strength. None of them wins every time, including us.

Hire in-house

You hire one or more engineers onto your own payroll. They own the AI work full-time, inside your existing team structure.

Best at: Institutional knowledge compounds. Someone who sits in your standups and knows your data model carries context across projects for years. No outside party can match that, and it grows the longer they stay.
Where it breaks: Hiring takes months, and strong AI engineers have their pick of offers. A single hire has no second opinion on their architecture calls until the team grows past one person.
Time to first ship: 3-6 months. Sourcing, interviewing, and onboarding all come before the first meaningful production code.
Six-week tell: Six weeks in, the requisition is still open. Or the new hire is still ramping on your codebase with nothing shipped. Name this in a leadership review. It is not a failure on its own.

Big consultancy or systems integrator

A large, multi-practice firm that staffs your engagement from a bench of consultants. A partner-level relationship usually sits above the team doing the work.

Best at: Scale and coverage. Suppose the work spans a dozen legacy systems, needs a change-management program beside the code, or needs twenty people next quarter. A large integrator is built to absorb that shape.
Where it breaks: The people who ran the sales conversation rarely write the code. Staffing can follow bench availability across the firm's other clients as much as fit for your problem.
Time to first ship: 2-4 months to first production ship. The statement of work, staffing, and onboarding for a multi-week engagement all count.
Six-week tell: Six weeks in, the person who ran the pitch is absent from the daily calls. The team roster does not match the résumés shared at kickoff.

Offshore or generalist dev shop

A software agency, often offshore or nearshore, that builds whatever a client asks for across many technologies. AI is one more line on the menu, not the specialty.

Best at: Cost-effective throughput on well-specified work. You hand over a clear, detailed spec and get a competent build back at a lower day rate than a specialist charges. A generalist shop is built to deliver that trade.
Where it breaks: AI work rewards judgment on ambiguous problems more than execution on a fixed spec. A shop staffed for throughput on clear tickets has little incentive, or practice, to push back on a weak brief.
Time to first ship: 4-8 weeks to a first demo. Reaching something that survives real production traffic often takes three to four times longer. Fixed-spec engagements usually skip eval discipline and guardrails first.
Six-week tell: The demo looks finished. Nobody can say what happens when the model is wrong, and no eval set backs the accuracy number they quoted.

Freelancer or fractional lead

An independent contractor, often a senior engineer between roles. You engage them directly for a defined chunk of work or an ongoing fractional commitment.

Best at: Speed and cost per hour. A good freelancer with production AI experience can move faster than a team with a kickoff deck and a project manager. The rate is a fraction of a studio's or integrator's.
Where it breaks: One person is a single point of failure for delivery and continuity. Nobody reviews an architecture call before it ships. A freelancer with several clients has a hard ceiling on how many hours are yours.
Time to first ship: 2-6 weeks. The range is wide because it depends on the individual and how much of their week they commit to you.
Six-week tell: You cannot say what happens to the project if this person is unavailable for two weeks. There is no documentation and no second person who could pick it up.

Specialist AI studio

A small team that works only on applied AI. You engage it for a scoped build or an ongoing retainer. It is sized between a freelancer and a full consultancy practice.

Best at: Judgment on the ambiguous parts: the build-buy-or-skip call, the architecture fork, the eval discipline. That judgment is the whole business. It is not one service line competing for staff against ten others.
Where it breaks: Small teams have a capacity ceiling. A studio sized for a handful of engagements cannot absorb a client that needs twenty engineers next quarter. A bad pick with no track record carries as much risk as a bad hire.
Time to first ship: 4-8 weeks to a working, evaluated pilot. A studio optimizes for AI judgment over raw headcount.
Six-week tell: There is still no eval set and no named production owner for the pilot. The same tell sinks a bad engagement on any of the other four routes.

Question bank

24 questions, six gates.#

Filter to the gates that matter for your situation, then copy the visible list as a Markdown checklist for your RFP.

Six gates run in order before a contract gets signed. Each one tests something the last gate did not.

01 Evidence & track record

A sales deck proves a firm can present. It does not prove a firm can deliver. This gate replaces the pitch with checks you can run. Ask for a named system still running today. Ask who will do the work. Ask for a reference who will talk about what went wrong.

Can you show me a named, live production system you built that's still running today?

Why you ask it: A portfolio slide with client logos proves a sales relationship. It does not prove a shipped system. Check whether the work survives real users and real data after the invoice is paid.
What passes: A specific system, named, live today, with roughly when it shipped and what it does. It is a real reference you can call. A case-study PDF does not count.
What fails: A list of client logos with no named system. Or "we can't say due to NDA" offered as the whole answer, with no redacted but specific description.
The artifact: A reference call with the team that operates the system now. The team that sold it does not count.

What happens when your own delivery goes wrong, and can you show me an example?

Why you ask it: Every delivery team ships a defect eventually. A good answer names one and says what changed afterward. A claim of a flawless record is a bad sign.
What passes: A specific, named incident with the root cause and what changed in their own process afterward. They offer it without being pressed.
What fails: A claim of a flawless track record. Or a generic answer about continuous improvement with no specific incident behind it.
The artifact: A published incident record or postmortem, or a specific verbal example you can check with a reference.

The Delivery Record

Who specifically will be doing the work, and can I see their own production AI experience?

Why you ask it: The person on the sales call and the person writing the code are not always the same. Check the named individuals. The firm's combined résumé tells you little.
What passes: Named individuals, with their prior production AI work described. You get this before you sign the contract, not after.
What fails: A generic "our team has years of combined experience" with nobody named. Or a roster where the senior names roll off right after the sale.
The artifact: Named CVs or profiles for the staffed team, checked against who shows up to the first working session.

Have you shipped something at roughly our scale and data sensitivity before, or would this be a first for you too?

Why you ask it: A team's best work is usually close to what they have already done. A partner whose biggest prior engagement was a tenth of your scale takes on unstated risk on your project.
What passes: A specific precedent at comparable scale or sensitivity. Or an honest "this would be a first for us at this scale", said up front.
What fails: Confidence with no specific precedent. Or scale questions answered with generic capability claims and no named comparable engagement.
The artifact: A description of the most similar prior engagement, detailed enough for you to judge the comparison yourself.

02 Technical fit

A partner who ships a demo differs from one who runs a system for years. Look for model-agnostic architecture, eval discipline, and production experience past launch day. This gate checks for all three.

Is your architecture tied to one model provider, or can you swap models without a rewrite?

Why you ask it: A provider outage, price change, or deprecation will happen at some point. Check whether that day is a config change or a rewrite.
What passes: A named abstraction layer between the application and any single model provider, with a specific past example of a model swap they did.
What fails: Everything hardcoded to one provider's SDK. Or "we'd just rebuild that part" offered as an acceptable answer to a provider outage.
The artifact: The architecture diagram showing the provider abstraction layer.

Glossary: Model-Agnostic Architecture

How do you know your system works before you ship it, and how do you know it still works after the next change?

Why you ask it: Anyone can demo a system that works once, on a case chosen to show it off. Look for a repeatable, checkable measurement.
What passes: A labelled eval set with a pass/fail condition per case. It runs in CI on every change, and they can show you the current pass rate.
What fails: "It works well in our testing" with no eval set named. Or evals described as a one-time check before launch, not a running gate.
The artifact: An eval report, or a description specific enough that you could ask to see one before signing.

The Eval Stack

What's the most complex production AI system you've operated after launch, past the day it shipped?

Why you ask it: Building a demo and operating a system through months of real traffic are different skills. This question checks for the second one.
What passes: A specific system, with a description of what broke after launch and how they caught it. That story only exists if someone operated the system.
What fails: Every example stops at launch day. Nothing says what happened afterward or how anyone would notice a regression.
The artifact: A description of the observability or monitoring setup on a real post-launch system.

Pattern: Trace-first observability

Walk me through a build-versus-buy or agent-versus-pipeline call you made recently, and why it went that way.

Why you ask it: The reasoning behind a past architecture decision reveals more about judgment than any credential. Check whether they decided the fork on purpose or defaulted into it.
What passes: A specific past decision with the tradeoff named. It includes a case where the simpler option won over the flashiest one.
What fails: Every past decision defaults to the most complex or most current option. No example shows them choosing the simpler path when it was right.
The artifact: A specific past project where the simpler option won, described in enough detail to check.

Handbook: Design stage

03 Ownership & exit

You pay someone to build this so that you end up owning it. This gate checks what transfers at handover. It also checks what it costs to leave if the relationship stops working.

Do we own the source code, prompts, and eval sets outright at handover, or are we licensing something back from you?

Why you ask it: Owning the work and licensing access to it are different things. One is independence and the other is a vendor relationship. Check which one you are signing.
What passes: A written clause that transfers source code, prompts, and eval sets to you in full at handover. Running or changing them needs none of the vendor's tooling.
What fails: Ownership described verbally but absent from the contract. Or a clause that transfers code but omits the eval sets or infrastructure-as-code needed to run it.
The artifact: The ownership clause in the signed engagement contract.

Ownership terms

If we wanted to walk away and hire someone else tomorrow, what would that take?

Why you ask it: A partner confident in their own value has an easy answer to this. A partner who relies on lock-in does not.
What passes: A short, specific answer: handover documentation, a knowledge-transfer session, and the code and credentials you already own. Nothing else is required.
What fails: Hesitation or a vague answer. Or you find that key infrastructure or credentials sit in the vendor's accounts, not yours.
The artifact: A documented handover checklist, ideally reviewed before the engagement starts. It should exist before you ask to leave.

What documentation and runbooks do we get, and can someone outside your team operate this from them?

Why you ask it: Undocumented systems create a dependency on the people who built them, whatever the intent. Check whether the knowledge left their heads.
What passes: Runbooks and documentation specific enough that a competent engineer who never worked on the project could run and modify the system from them alone.
What fails: "We'll walk your team through it" offered in place of written documentation. Or documentation that turns out to be a slide deck, not an operational runbook.
The artifact: The documentation itself, reviewed for specificity before the engagement closes. A promise of documentation later does not count.

What happens the week after handover if something breaks, and is that covered or a new invoice?

Why you ask it: An undocumented dependency shows up in the gap between the engagement ending and you running alone. Check the terms of that gap before you are in it.
What passes: A stated support window after handover. It says what is covered and what triggers a new engagement, agreed before work starts.
What fails: No stated post-handover terms. You find out when something breaks and a new statement of work appears.
The artifact: The support or warranty clause in the contract.

04 Delivery model & velocity

Most AI engagements fail on process, not on the model. Common causes are no named owner, no visibility into a bad week, and scope that drifts with no way to renegotiate it. This gate checks the operating model.

Once this ships, who is the named owner on our side and on yours, and does that person have the authority to act?

Why you ask it: A pilot with no owner stalls after the demo. Check that a specific, accountable person exists on both sides before it ships. Do not wait until it stalls.
What passes: A named individual on each side with real decision authority. Both are agreed before the pilot starts and not assigned informally after launch.
What fails: Ownership described as "the team" and not as a person. Or a named owner with no authority to approve changes.
The artifact: The ownership line in the project charter or kickoff document.

How will I see progress week to week, and what does a bad week look like when it happens?

Why you ask it: A partner who only shows you good news has decided what you may see. Look for a cadence that surfaces a problem the same way it reports success.
What passes: A regular cadence, such as a demo, a written update, or a dashboard. It would show a stalled or blocked week as clearly as a good one.
What fails: Updates that are verbal reassurance with no artifact. Or a cadence that only exists when things go well.
The artifact: A sample status update or dashboard from a past engagement.

When the scope needs to change mid-engagement, because it usually does, what happens?

Why you ask it: Scope drift is normal on real AI work. What matters is whether a defined, fast path exists to renegotiate it. Without one, scope expands the original quote in silence.
What passes: A named change-control process with a stated turnaround time, agreed before the engagement starts.
What fails: No defined process. Scope changes are handled ad hoc, and disputes about what the parties agreed surface after the fact.
The artifact: The change-control clause in the statement of work.

What does the path from pilot to production look like, and who decides the pilot is done?

Why you ask it: A pilot with no graduation criteria can run forever or die with nobody able to say which. Check that both outcomes have a name and an owner.
What passes: Stated success criteria for the pilot, agreed up front. A named decision point and decision-maker choose between graduating to production and stopping.
What fails: "We'll know it when we see it" as the whole graduation criteria, with no stated metric or decision point.
The artifact: The pilot success criteria, written down before the pilot starts.

05 Security, privacy & compliance

Whatever a partner builds will meet your own InfoSec, privacy, and procurement review. Our security review page covers the same four gates. This gate checks whether that work happens before the review or during a rejected one.

Could this system pass an internal security or privacy review today, and if not, what specifically is missing?

Why you ask it: A system built without review in mind usually needs weeks of retrofitting before it can ship. Check whether that work is done or still ahead of you.
What passes: A specific, honest list of what is ready and what is not, mapped to the gates your own reviewers will run.
What fails: An assumption that security review is a formality to handle later. Or no knowledge of what your own reviewers will ask.
The artifact: A completed pre-flight artifact list against your own review gates.

AI Security Review

Where does our data go once it leaves our systems, and is that written down?

Why you ask it: Model providers, subprocessors, and logging destinations form a chain. A vague answer here usually means nobody has mapped it. It does not mean the chain is simple.
What passes: A named data-flow description covering every model provider and subprocessor your data reaches, checked against your own compliance requirements.
What fails: "It's all handled securely" with no specifics. Or a chain that turns out to include a subprocessor nobody mentioned.
The artifact: The data-flow diagram and subprocessor list.

If this system can take actions as well as answer, what stops it from doing something irreversible by mistake?

Why you ask it: An agent that can write, send, or delete inherits real consequences from a hallucinated call or a successful prompt injection. Check whether the team designed around that risk or assumed it away.
What passes: Named tool scoping to least privilege, with a human checkpoint on anything irreversible. They describe it specifically and do not just assert it.
What fails: A claim that "the model is pretty reliable" offered as the control. No scoping or checkpoint is named.
The artifact: The tool/permission matrix.

Pattern: Human checkpoint before irreversible actions

Which compliance framework does our industry map to, and how does that change what you build?

Why you ask it: GDPR, SOC 2, and industry-specific rules change what you must log, retain, and disclose. A partner who cannot name yours is guessing at your requirements.
What passes: The specific framework named for your situation, with a concrete example of how it changes a design decision. A generic list of acronyms fails.
What fails: Every industry gets the same generic compliance answer. No framework is named for your situation.
The artifact: A written mapping of your compliance requirements to specific design decisions.

06 Commercials & risk

Three things price an engagement's risk: the pricing model, the production cost, and a reference call with a client who left. The rate card does not.

Is this priced fixed-fee, time-and-materials, or value-based, and does that match who's holding the risk?

Why you ask it: A pricing model assigns risk. Time-and-materials rewards slow delivery. A fixed fee on an unscoped problem punishes honest scope discovery. A mismatch shows up later as friction.
What passes: A pricing model matched to the scope certainty, explained plainly, with the tradeoff named.
What fails: A pricing model presented as the only option, with no tradeoff explained. Or open-ended time-and-materials billing on a poorly scoped problem.
The artifact: The pricing clause in the proposal, with the model and the reasoning behind it.

Pricing & engagement

What's the cost per completed task if this runs in production, and what's the worst case if it misbehaves?

Why you ask it: A sticker-price estimate routinely undercounts real spend once retries, fallbacks, and agent loops are counted. Check whether the number you are quoted is the real one.
What passes: A modeled cost per successful outcome, not per API call. Plus a stated ceiling on worst-case spend, given an enforced budget.
What fails: A per-token estimate presented as the total cost. No mention of retries, fallbacks, or a spend ceiling.
The artifact: The cost model, including the worst-case scenario.

Handbook: Cost stage

What specifically triggers a change order versus what's included in the original quote?

Why you ask it: A quote with no stated boundary invites scope disputes later. Check that the line between included and extra is named up front. You do not want to discover it mid-engagement.
What passes: A specific, written boundary between what is included and what triggers additional cost, agreed before work starts.
What fails: "That wasn't included" surfaces for the first time after work is underway, against a quote with no stated boundary.
The artifact: The scope-boundary section of the statement of work.

Can I talk to a past client who didn't renew or extend, as well as one who did?

Why you ask it: A reference list made only of happy renewals shows you the best case. It does not show the average. A partner who offers a mixed reference tells you something about themselves.
What passes: A willingness to offer, or at least discuss, a reference from an engagement that ended and did not renew, with an honest account of why.
What fails: Every reference offered is a renewal or expansion. Nothing acknowledges that an engagement ever ended for a reason worth naming.
The artifact: A reference call with a past client, including one that didn't continue.

Five ways it goes wrong

These sink an engagement, not a model.#

Each one is a contract problem with a contract fix.

The pilot with no production owner

Symptom: A pilot ships and gets a good reaction in the room. Then it sits untouched for months. Nobody on either side owns the push into production.

Root cause: The kickoff named a project and no person. "The team is on it" implied accountability. Nobody with authority to act held it.

The clause that prevents it: Name a production owner on both sides in the kickoff document. Do it before the pilot starts. Add a decision date for graduate-or-stop.

Most common in Big consultancy or systems integratorOffshore or generalist dev shop

Demo-driven scope

Symptom: The system looks finished in a conference room on curated examples. It falls apart on the first messy, real-world input a week after launch.

Root cause: The engagement optimized for sign-off, which means a good demo. Nobody wrote down what "done" meant, so nothing pushed it to survive production.

The clause that prevents it: Define the eval set and its pass/fail conditions in the statement of work. Do it before anyone builds a demo.

Most common in Offshore or generalist dev shopFreelancer or fractional lead

The Eval Stack

The black-box handover

Symptom: The engagement ends and the invoice is paid. Then the client finds that nobody in-house understands the system or can safely change it.

Root cause: The team treated documentation and knowledge transfer as a nice-to-have at the end. Nobody specified them as a deliverable at the start.

The clause that prevents it: Make documentation and a knowledge-transfer session a named deliverable in the contract. Give it its own sign-off, separate from the code shipping.

Most common in Big consultancy or systems integratorFreelancer or fractional lead

Model lock-in by omission

Symptom: A provider raises prices or deprecates the model the system depends on. Switching turns out to mean a rewrite.

Root cause: Nobody required model-agnosticism. The fastest path to a working demo was to call one provider's SDK everywhere, and that became the permanent architecture.

The clause that prevents it: Require a model-agnostic abstraction layer as a named technical requirement in the contract. Do not assume it as a best practice.

Most common in Offshore or generalist dev shopFreelancer or fractional lead

Glossary: Model-Agnostic Architecture

Seniority bait-and-switch

Symptom: The senior names who ran the pitch are gone from the team three weeks later. They impressed everyone in the room and do none of the work.

Root cause: Staffing on the proposal was aspirational, or a sales asset. Nothing in the contract tied the named people to the delivered work.

The clause that prevents it: Name the staffed individuals in the contract. Define a process, and client sign-off, for any substitution.

Most common in Big consultancy or systems integrator

A technical sibling to several of these already exists on our failure modes reference.

Our answers

Our answers, with receipts.#

Six of our own 24 questions, each pointing at something already published.

What happens when your own delivery goes wrong, and can you show me an example?

Six real defects, the root cause, and the build-time guard added after each one. We publish all of it in full.
The Delivery Record
Who specifically will be doing the work, and can I see their own production AI experience?

Named bios and no bench. See who is on the team before you sign anything.
About
How do you know your system works before you ship it, and how do you know it still works after the next change?

Six eval layers and 24 named checks, published in full. A proposal alone would only assert them.
The Eval Stack
Is your architecture tied to one model provider, or can you swap models without a rewrite?

Why the abstraction layer belongs in the design from day one. Adding it after the first price change costs more.
Handbook: Design stage
Do we own the source code, prompts, and eval sets outright at handover, or are we licensing something back from you?

Source code, infrastructure-as-code, the eval suite, and documentation all transfer at handover. Keeping it running needs no retainer.
Ownership terms
What's the cost per completed task if this runs in production, and what's the worst case if it misbehaves?

Cost modeled per completed outcome and not per API call. It includes the retries and fallbacks that a sticker price leaves out.
Handbook: Cost stage

When we're wrong for you

Honest disqualifiers.#

If any of these describe you, another route above fits better than an engagement with us. So might the resource we point at.

You need twenty engineers staffed and working by Monday. You do not need a small team making judgment calls.

A large integrator or a dev shop is built to solve that scale problem. A studio sized for a handful of engagements is the wrong shape for it, whatever the pitch says.
Compare the five routes above
You want a fixed price against a spec that nobody has written yet.

A fixed price needs a fixed scope. The risk in AI work almost always sits in the ambiguous part nobody has scoped. A diagnostic turns that ambiguity into a written scope before anyone prices it.
Book a Ship Audit
You want someone to sit inside your org and own this forever. You do not want a team that builds it, proves it, and hands it over.

An in-house hire is for that. We hand over full ownership at the end of an engagement. Nobody here can become a permanent member of your team.
The Decide stage of the Handbook
There is no budget for a paid engagement right now, and you need something today. Next quarter is too late.

A paid engagement is the wrong instrument for a zero-budget problem. Check what already exists for free before you pay anyone.
FreeTier
You are not sure this is an AI problem yet. You are talking to partners before you test the deterministic alternative.

Every route on this page, including us, is slower and more expensive than a rule-based system that already solves the problem. Test that first.
AI Readiness Scorecard

This page lists the questions a buyer usually needs answered before signing an AI delivery contract. It also lists the evidence that tends to answer them well. It is not legal or procurement advice. Your own legal and procurement teams make the sign-off call for your organization.

Ready to run this against a specific partner

A Ship Audit turns an unscoped idea into a written plan. Do it before you take the idea to any of the five routes above, including us.

Book a Ship Audit → See pricing →

Questions

Before you sign anything.#

What buyers ask us before they run this checklist against a partner.

01 Is this page trying to talk me out of hiring you?

No. It gives you a decision framework. Two of the five routes and several of the disqualifiers point away from us on purpose. A guide that concludes "hire us" for every question is an ad.

Link to this answer

02 Do all 24 questions apply to every engagement?

No. Filter to the gates that matter for your situation. A ten-day proof of concept does not need the full commercials gate. A system that touches regulated data needs every security question answered before anyone signs.

Link to this answer

03 What if a partner can't answer one of these yet?

A specific "we don't have this yet, here's the plan and the date" beats a vague or invented answer. Our security review page holds reviewers to the same standard.

Link to this answer

04 Can I use this if I've already picked a partner?

Yes. Run the 24 questions against the partner you chose. Catching a gap before the contract is signed costs less than catching it three months in.

Link to this answer

05 Why publish the routes where you're not the best fit?

A page that always concludes "hire us" helps nobody make a real decision. The honesty is the point of the wrong-choice section below.

Link to this answer

Check the record, not the pitch

Seniority bait-and-switch and a black-box handover are two of the five ways an engagement fails. Our delivery record shows the actual commits and merge requests behind every project, not a highlight reel.

Read the delivery record → Talk to us →