How to Vet an AI Development Company Out of 8,149 Listings
The questions that separate production AI experience from a service line added last year
To vet an AI development company, check three things before you watch the demo: a production system the firm still operates, a written record of how it measures the model being wrong, and a contract that names where your data goes. Learning how to vet an AI development company is no longer about finding a scarce specialist. It is about filtering a market where nearly every software firm now claims the title.
Updated November 2026. Edverise listing figures were read on 28 October 2026.
Every page that currently answers this question was written by a company that sells AI development, so none of them mention how many competitors they have. That omission changes the advice. If specialists were scarce, your job would be to find one and keep it happy. They are not, so your job is to eliminate most of the market fast.
How to vet an AI development company when the category is this crowded
As of October 2026, Edverise lists 8,149 firms under Custom AI Chatbot Development and 8,190 under Generative AI. Those are two separate category counts, and they are not added together: a firm offering both is counted in both, which is also why no honest total exists for "AI companies."
That scale only lands next to the rest of the catalogue. Edverise carries 1,249 active service categories, and the median one lists 5 firms; 703 list fewer than 10. The AI build categories are not a thin specialist niche in that catalogue. They are among its deepest pools.
One number says more about posture than depth. Across the catalogue, in a typical category 57.4% of the firms that declare a minimum project size declare $0. Under Custom AI Chatbot Development only 14.3% do, and 32.5% name a $1,000 floor. In Generative AI the $1,000 band holds 33.6%. These firms are not shy about taking work, and they have priced their entry point low enough that almost nothing filters itself out before it reaches your inbox.
Treat "AI development company" the way you treat "web development company": a search term, not a credential. The filtering has to happen in your questions, because it is not happening in the supply.
Where the supply thins out, and what that tells you
The counts get far more uneven once you look at the categories that only matter after a model is live.
| Service category | Firms listed |
|---|---|
| Custom AI Chatbot Development | 8,149 |
| AI Model Fine-Tuning & Training | 1,578 |
| Data Annotation Services | 548 |
| MLOps & Model Deployment Infrastructure | 374 |
The chatbot category is more than twenty times the size of the MLOps & Model Deployment Infrastructure category. Listing yourself is a choice, so the gap is not proof of capability either way, but it tells you what the market thinks it sells. Thousands of firms describe themselves by the thing a buyer asks for. A few hundred describe themselves by the deployment, monitoring and retraining work that decides whether the thing keeps working in March.
Use that asymmetry as a question rather than a verdict. Ask each shortlisted firm which of those four categories it would list itself under and why. A firm that reaches for AI Model Fine-Tuning & Training had better be able to describe a training run it did and what it cost.
One limit on all of this: a listing on Edverise starts as a profile compiled from public information and becomes Verified only when the company claims it and proves control of its domain. A listing is a place in a filterable index, not a judgement about engineering quality, and Edverise publishes how its rankings work so you can see what it does and does not assess.
How to tell if an AI development company has production experience
Production experience shows up in the failure stories, not the case studies. A firm that has run an AI feature for a year has been paged at night by it, has a list of inputs that break it, and has a number for how often it is wrong. A firm that shipped a pilot and moved on has none of those and cannot invent them under questioning.
That stage is where most projects die. In a survey of 600 CIOs run by The Harris Poll for Dataiku, 85% said gaps in traceability or explainability have already delayed or stopped AI projects from reaching production, and 74% said they regret at least one major AI vendor or platform decision made in the last 18 months. Read the sample before borrowing the number: those respondents were CIOs at companies with annual revenue of $500 million or more. The vendor blogs quoting the 74% never mention that limit, and it is the limit that decides whether the figure applies to you.
Six questions separate operators from first-timers. Ask them in one call and listen for whether the answers need preparing.
- Which AI system that you built are you still operating today, and who gets paged when it fails?
- What did your evaluation set look like — how many test cases, written by whom, and what score shipped?
- What was the error rate at launch, what is it now, and what changed it?
- Where is the human-in-the-loop review step, and what fraction of outputs goes through it?
- What did the model do wrong that you did not anticipate, and how did you find out?
- Show me the runbook for the day the model provider changes the endpoint.
Question two is the one that cannot be bluffed. An evaluation harness is a fixed set of inputs with known good answers, scored automatically on every change. A firm with one will describe its size and who wrote the answers in a sentence. A firm without one will talk about accuracy in general terms, which is the tell.
What to ask an AI vendor about training data and where your data goes
Ask which model provider the system calls, on which tier, and under which written terms. The vendor's reassurance is not the relevant document, because the vendor is not the party holding your data.
The provider terms are public and specific. Anthropic's commercial terms state that Anthropic "may not train models on Customer Content from Services" and that the customer retains all rights to its inputs and owns its outputs. OpenAI's API documentation states that data sent to the API is not used to train or improve its models unless the customer opts in, that abuse-monitoring logs are kept for up to 30 days by default, and that zero data retention is available subject to prior approval. Those are the protections your vendor is relying on when it says your data is safe.
So the useful follow-ups are about the chain, not the promise:
- Name every subprocessor in the request path, including any gateway, vector store or logging service, and put the list in the data processing agreement.
- Is the account yours or the vendor's? If it is the vendor's, the provider's terms protect the vendor, and you are relying on a flow-down clause that has to exist in writing.
- Is prompt and response logging on? Where does it land, who can read it, and for how long?
- If you supply documents or transcripts for retrieval-augmented generation or fine-tuning, who owns the resulting index or weights when the engagement ends, and what is deleted?
- What happens to the system if you revoke the data tomorrow?
A vendor who answers with a provider name, a tier and a retention setting has done this before. One who says "your data never trains anyone's model" without naming the provider is repeating a line from a pricing page.
What an AI pilot should cost, and what the price has to include
A stated minimum is a floor on what a firm will accept, not an estimate of your project. The spread under one AI category shows how little the floor narrows things.
| Stated minimum | Share of firms declaring one |
|---|---|
| $0 | 14.3% |
| $1,000 | 32.5% |
| $5,000 | 24.1% |
| $10,000 | 16.3% |
| $25,000 | 7.9% |
| $50,000 or more | 4.9% |
Nearly a third of the category will start at $1,000 and only 4.9% require $50,000 or more, so a quoted price tells you about the firm's business model rather than the difficulty of your problem. Four proposals at four numbers are not four estimates of the same work. They are four guesses at what you will pay.
Bound the first phase instead of negotiating the whole thing. Across a sample of 5,392 IT projects, Flyvbjerg and colleagues found that cost overruns follow a power-law distribution with a fat tail, so the risk is not a small miss but the rare project that runs away entirely. A capped first phase is how you stay out of that tail, and whether you then move to fixed price, time and material or an outcome-based contract is a decision you make with evidence rather than before it.
A first phase worth paying for includes four things: your own data rather than a sample, an acceptance threshold agreed in writing before work starts, the evaluation set handed over as a deliverable, and a documented exit with the prompts, configuration and index in your repository.
Ask for the running cost separately, because it is the line most proposals omit. Model list prices differ by a factor of a hundred: OpenAI publishes $0.10 per million input tokens for its cheapest current model and $10.00 per million for its most expensive. A vendor that cannot name the model it plans to call and estimate your monthly token volume has not priced inference at all, and that is the number that decides whether the feature survives its second quarter.
Where this advice stops applying: if the whole budget sits at the $1,000 or $5,000 end, a separate paid pilot is overhead you cannot afford. Buy the build, keep the written acceptance threshold, and skip the discovery phase.
Red flags when hiring an AI development company, and the case against them
Take the opposing argument at full strength first. Most of an AI project is not model work. It is data plumbing, integration with systems that predate the idea, permissions, interface design and change management. A good web and backend shop already does that well, and a firm that knows your industry will usually beat an AI boutique that has to learn it. "They added AI last year" is not by itself a reason to cut anyone.
What is a reason to cut them:
- No named system in production that they still support, only pilots and prototypes.
- No evaluation set, and no answer to how they would build one for you.
- No answer to what happens when the model is wrong — no fallback path, no escalation, no review step.
- A fixed package quoted before any discovery, identical whatever your data looks like.
- Unwillingness to name the model provider, the account owner or the retention setting.
- An accuracy figure with no test set, no date and no baseline behind it.
That last one deserves its own line. The question about an accuracy claim is not how high it is but what it was measured on, how many cases, who wrote the expected answers, when, and what the previous process scored on the same set. A firm that answers all five has an evaluation practice. A firm that answers none has a slide.
Write your evaluation criteria before you talk to any vendor
Federal procurement solves this by forcing the buyer to commit first. Under FAR 15.304(d), a solicitation must clearly state all factors and significant subfactors that will affect award and their relative importance, and say whether the non-price factors combined are more important than price; the rating method need not be disclosed. The discipline is the useful part. Weights written after the pitches are weights written to justify the pitch you liked.
Write yours down before the first call, with a benchmark for what earns full marks. The weights below are a starting point for a first AI build at a 50-250 person company, not a formula.
| Criterion | Weight | Full marks requires |
|---|---|---|
| Production evidence | 25 | A named system live for a year or more, still supported by the same firm, plus a reference call |
| Evaluation practice | 20 | An existing harness described by size and authorship, and a proposed acceptance threshold for your project |
| Data handling | 20 | Provider, tier, account owner, subprocessor list and retention setting named without being chased |
| Domain fit | 15 | Two clients in your industry, and an unprompted question about your constraints |
| Exit terms | 10 | Prompts, configuration, index and weights in your repository, and deletion on request |
| Running cost | 10 | A monthly inference estimate with the model named and the token volume shown |
Score each firm on the day you speak to it, not at the end. If two land within a few points of each other, the tie-breaker is which one asked you harder questions, and a software development RFP template is a reasonable way to get the same answers from everyone in writing.
Frequently asked questions
How do I vet an AI development company without an in-house data scientist?
Vet the firm's practices rather than its technical choices. You do not need to assess a model to ask which system it still operates, how large its evaluation set is, who wrote the expected answers, what the error rate was at launch, and who gets paged when it breaks. Those answers are verifiable by reference call, and the vague ones are obvious without any machine learning background.
Should an AI project start with a paid pilot?
Yes, for anything beyond the small end of the market, and only if the pilot is bounded. A pilot worth paying for runs on your data, has an acceptance threshold agreed in writing before work starts, hands over the evaluation set as a deliverable, and ends with the prompts and configuration in your repository. If your whole budget sits at the $1,000 or $5,000 floors, skip it and buy the build.
What should I ask an AI vendor about where my data goes?
Ask which model provider the system calls, whose account it runs under, which subprocessors sit in the request path, whether prompt logging is on, and how long anything is retained. Get the subprocessor list into the data processing agreement. If the account belongs to the vendor, the provider's terms protect the vendor, so you need a written flow-down clause.
How can I tell a real AI specialist from a rebranded web agency?
Ask about the work that only exists after launch: monitoring, retraining, drift, fallback paths and the review step for wrong outputs. Specialists answer from memory because they live with it. Rebranded agencies answer in principles. The honest rebrand will say it does the integration and bring in a partner for the model work, which is often a fine arrangement.
What accuracy claims should I be suspicious of?
Be suspicious of any accuracy figure given without a test set, a date and a baseline. A claim worth something names how many cases were tested, who wrote the expected answers, when it was measured, and what the previous manual process scored on the identical set. A number with none of that attached was produced for the proposal, not for engineering.
What to do with those four proposals this week
Send all four firms the same six production questions and the same data-handling list, in writing, with a deadline. Score the replies before anyone presents. Usually two answer in specifics and two answer in adjectives, and the decision stops being hard — which is the practical effect of knowing the supply is deep rather than scarce.
If you are still assembling the shortlist, browse the firms listed under Custom AI Chatbot Development on Edverise and filter by minimum project size, city and industry before reading a single homepage, so the proposals you hold are at least comparable. The guide for buyers covers comparing and vetting from there.