7 Questions to Ask AI Website Agencies in Australia
Just about every web agency in the country has added AI to its capability list. Some have built something. Others resell a third-party widget. From outside, the two can look identical right up until the invoice or the incident.
Seven questions separate them, and none require you to be technical. What matters isn't really the answer. It's whether the person across the table has thought about it before, or is composing it live.
Key Takeaways
- A resold widget and a genuine build look identical in a demo. They stop looking alike the moment you ask about failure, cost and maintenance.
- Ask where data is processed and whether it trains anyone's model. Get it in writing, not in a meeting.
- Usage-based billing is the norm, so ask what happens to the bill when traffic triples.
- An agency that can't tell you what its AI feature does when it doesn't know the answer hasn't finished designing it.
- The best signal is an agency willing to tell you which AI feature not to build.
1. Where Does Our Data Go, and Does It Train Anyone's Model?
Start here, because it's the one with legal consequences attached. You want to know which provider processes your content and user input, which region, whether it's retained, and whether it feeds model training.
A good answer names the service and the contract terms. A bad one says "it's secure" and moves on. (Anyone who can't say whether your content sits under an enterprise agreement or a consumer chat product is guessing with your data.)
2. Which Model, and What Happens When It's Retired?
Models are rented, not owned, and they get deprecated on the vendor's schedule. That's normal now in a way it wasn't a few years ago. What isn't normal is having no plan.
Ask which model powers the feature, whether it can be swapped, and who pays for the migration when the current one retires. Agencies building properly abstract the model behind their own layer, so moving between OpenAI, Azure AI or Anthropic's Claude is a config change, not a rebuild. A reseller often can't answer this, because the choice was never theirs.
3. What Does This Cost at Full Traffic?
Pilots are cheap because almost nobody uses them. Model APIs generally bill by usage, so cost tracks volume the way licence fees never did.
Get an estimate at your real traffic, then at triple that, and ask what stops a runaway. Rate limits, caching, a spend ceiling, alerting: ordinary engineering, and its absence tells you the feature was assembled, not designed. Then ask who pays if it runs away. (The pause before that answer is informative.)
4. How Will We Know When It's Wrong?
This one tends to get vague answers, and it's the one I'd weight most heavily.
Language models are fluent whether or not they're correct, so errors don't announce themselves. There needs to be a test set: real questions with known good answers, re-run whenever the model, the prompt or the content changes. Ask to see it. If quality assurance means "we tried it and it seemed good", you're buying a feature nobody can prove works.
Be wary, too, of a headline accuracy figure offered without the test behind it.
5. What Does It Do When It Doesn't Know?
Related, but distinct. This one's a design question, not a technical one.
A well-built assistant grounded in your own content should decline, cite the page it drew from, and offer a route to a human. A poorly built one invents something plausible. Listen for grounding, refusal behaviour, escalation, and what the site does when the AI service is simply down, because sometimes it will be.
Ask them to demo the failure, not the success. Any decent build has a story about what it deliberately refuses to answer.
6. How Does This Sit With Australian Privacy Obligations?
If the feature touches personal information, the Privacy Act 1988 and the Australian Privacy Principles apply as they do everywhere else on your site. APP 8 covers cross-border disclosure, which matters the moment processing happens offshore.
You're not asking your agency to be your lawyer. You're checking they know the questions exist, can tell your privacy officer where data flows, and won't be blindsided by a residency requirement mid-build. Government and health buyers should raise it early.
7. Who Maintains It, and What Does Year Two Look Like?
AI features rot in a way ordinary pages don't. Content changes and answers drift. A model retires. A provider adjusts pricing. None of it throws a 500 error, so nobody notices until a customer does.
Ask what maintenance covers, who reviews outputs, how often the test set is re-run, and what happens if you part ways. If the answer is "it's all in our platform", you've found a widget.
The Question Behind the Questions
Ask what they'd tell you not to build. An agency with real AI development experience has views on which features aren't worth the money, and says so before the contract. One that finds every idea exciting is selling, not advising.
Where Webcoda Fits
Webcoda has built websites and web applications in Sydney since 2005. Our AI work covers AI-assisted content creation, AI-powered personalisation, search and recommendation engines, intelligent chatbots and virtual assistants, automated content workflows, and custom dashboards and internal tools, built on OpenAI, Azure AI and Anthropic's Claude, plus the native AI features in HubSpot and Umbraco's OpenAI connector. Most of the hard work sits in integration and content structure, not the model. Put these seven questions to us too. That's rather the point of writing them down. Nobody's finished working out the answers.
Frequently Asked Questions
Ask which model it uses, whether it can be swapped, and what the feature does when the model is unavailable. A team that built the integration answers immediately and in detail. A reseller describes the product rather than the architecture, because the architecture belongs to someone else.
Worried is too strong, but insist on estimates at real traffic rather than pilot traffic, since model APIs are usage-billed. Also ask what controls exist: rate limits, caching, spend ceilings and alerts. The controls matter more than the estimate, because the estimate will be wrong.
They don't prohibit it, but they do impose obligations. The Australian Privacy Principles under the Privacy Act 1988 include APP 8, which sets out steps to take before personal information is disclosed overseas. Your privacy officer or legal advisers make that call. Your agency's job is to tell them precisely where the data goes.
Be wary of anyone offering a single number without describing the test behind it. Accuracy depends on the questions asked, the quality of the content the assistant is grounded in, and whether it's allowed to decline. A useful commitment is a documented test set, a measured baseline and an agreed process for when the score falls, rather than a percentage on a slide.
Quality data in, quality responses out.
Settle this in the contract before the build starts. Custom integration code, prompts, test sets and configuration can all be yours. A third-party AI platform your agency resells usually cannot be, and that's the difference that bites when you want to move.
Talk to Webcoda
Webcoda is a Sydney-based, B Corp certified digital agency that has been building websites and enterprise web applications since 2005, for organisations including Sony, BridgeClimb, the Australian Federal Government and NSW Health. We are an Umbraco Gold Partner, a Gold partner for Xperience by Kentico and a HubSpot Solutions Partner, so platform advice comes without a single-vendor agenda.
Get in touch to talk through your project, or browse our full range of services and platforms.
