Public, open-weight or dedicated AI? A CIO’s guide
Public AI services are the natural starting point for most universities. But as experimentation turns into production, CIOs face a broader set of decisions around model capability, cost, control, sovereignty and infrastructure. The question is not when to move away from public AI. It is when a particular workload gives you a reason to take more control.
For the first phase of generative AI, the technology decision was relatively straightforward.
Give people access to capable tools. Experiment with Microsoft Copilot, ChatGPT, Claude, Gemini and the major cloud AI platforms. Find the use cases that create value. Learn where the technology performs well and where it does not.
For many university workloads, that will continue to be the right model.
The conversation changes when an experiment becomes an institutional service.
An internal assistant starts answering thousands of questions. AI becomes part of an admissions or finance process. A service desk starts using models continuously. An agent calls enterprise systems rather than simply generating text. What began as occasional consumption starts developing a measurable workload profile.
Gartner is seeing this transition in Australia. At its Sydney Infrastructure, Operations & Cloud Strategies conference in May 2026, it described organisations moving from early AI experimentation towards real-time inference and agentic AI. Gartner also identified a shift towards inference-optimised approaches, smaller domain-specific models and hybrid architectures rather than relying exclusively on large general-purpose models. (Gartner)
For university CIOs, that creates a new architectural question:
When should we keep consuming public AI, when should we introduce open-weight models, and when does dedicated AI capacity actually make sense?
There is no single answer for the university. There should be an answer for each workload.
Public AI is usually the right place to start
There is a strong argument for not optimising AI infrastructure too early.
Public AI gives universities immediate access to rapidly improving models without requiring them to forecast GPU demand, operate inference infrastructure or commit to a particular model architecture.
It is particularly effective for workloads where demand is uncertain or where access to frontier capability is valuable. Staff productivity is an obvious example, but the same applies to prototypes and emerging applications where the problem itself is still being understood.
At this point, flexibility matters more than infrastructure efficiency.
The first principle is therefore simple:
Do not optimise the AI infrastructure before you understand the workload.
As a use case becomes established, however, different questions become possible.
How frequently does it run? How much inference does it consume? What does a successful transaction cost? What data is moving through the model? Does the workload need the strongest available reasoning capability? Is latency important? Is the service becoming business-critical? How predictable is demand?
Once those things can be measured, architecture becomes a real decision rather than a forecast.
Does the production workload still need the frontier model?
One of the more important questions for CIOs may be whether the model used to prove a concept is necessarily the model that should run it for the next five years.
During experimentation, using the most capable model available can make sense. It removes model quality as a variable and allows the team to focus on whether the idea works.
Production changes the calculation.
Consider an application that repeatedly:
- classifies incoming requests
- extracts information from standard documents
- answers questions against an approved knowledge base
- generates constrained correspondence
- summarises known institutional information
- routes a request to the appropriate workflow
- calls a defined set of enterprise APIs.
The useful question is not whether another model is generally as capable as the current frontier model.
It is:
Does it reliably meet the quality threshold for this specific task?
Gartner’s Australian research points in this direction, with smaller domain-specific models increasingly being used as organisations optimise inference rather than defaulting every workload to a large general-purpose model. (Gartner)
Globally, Gartner also expects increasingly specialised model use as AI budgets come under greater scrutiny around cost, latency, performance and reliability. (Gartner)
This is where open-weight models become interesting.
When open-weight starts to make sense
Open-weight should not be treated as inherently better than a hosted frontier model.
It is also not synonymous with private AI, small models or open-source software.
Its practical advantage is choice and control.
Depending on the model and licence, an organisation can have considerably more choice over where the model runs, the inference environment around it, model lifecycle, optimisation, integrations and the operational controls applied to the workload.
That starts to matter when several conditions appear together.
The workload has become repeatable. A suitable model can be benchmarked against the task. The data being processed is becoming more sensitive. The institution wants greater control over the inference path. Model changes need to be managed rather than simply accepted. Or utilisation has become regular enough that capacity-based economics deserve consideration.
But moving to open-weight does not mean the university needs to buy GPUs.
There is a significant middle ground between calling a public model API and becoming an AI infrastructure operator.
Shared managed open-weight
Shared managed inference can be a sensible first step once an open-weight model has been proven for a workload.
The university gains access to a managed inference environment and greater deployment choice without having to reserve an entire physical capacity pool.
This may suit repeatable institutional applications where demand is material but still relatively modest, or where sovereignty and control are becoming more important without yet justifying dedicated infrastructure.
Dedicated managed open-weight
Dedicated inference becomes more relevant as the workload gets larger, more predictable, more sensitive or more important to the institution.
Dedicated capacity can provide greater isolation, clearer performance characteristics and stronger control over the environment and model lifecycle.
Importantly, the institution is taking more control of the AI workload without necessarily taking responsibility for operating the GPU platform underneath it.
Dedicated GPU capacity
Dedicated GPU infrastructure should have a much higher threshold.
It starts becoming worth serious consideration when a university has not one promising AI application, but a portfolio of workloads capable of making productive use of the capacity.
That may include sustained inference, multiple production AI services, specialised models, fine-tuning, higher concurrency, demanding performance requirements or a deliberate internal AI platform strategy.
This is consistent with the broader infrastructure market. Gartner forecasts inference spending to overtake training within AI-optimised infrastructure in 2026 as production, fine-tuned and domain-specific models create continuous compute demand. (Gartner)
But that is not an argument for every university to buy GPUs.
Dedicated GPUs should be the result of demonstrated demand, not the starting point of an AI strategy.
Sovereignty should follow the workload
Model capability and cost are only part of the decision.
As AI moves deeper into university processes, sovereignty can become materially more important.
A staff member using an approved service to summarise public material creates a very different risk profile from an institutional AI service processing sensitive student information, employee information, legal documents, security information or privileged university data.
The second workload deserves a different set of questions.
Where is inference actually taking place? Where do prompts and retrieved context travel? Are prompts or responses retained? Who can administer the environment? From which jurisdictions? Which legal entities and subprocessors are involved? What telemetry leaves the platform? Can the university replace the model without rebuilding the application?
That is a much more useful way to assess sovereignty than simply asking whether the GPU is physically located in Australia.
For some workloads, mainstream SaaS and public AI platforms will remain entirely appropriate.
For others, Australian-operated shared inference may provide the right balance.
A smaller number may justify a dedicated sovereign environment because the data, controls or operating model require it.
This is why sovereignty should be a workload placement factor, alongside performance, model capability, cost, integration and operational criticality.
It should not be a blanket rule that sends every AI workload to the same environment.
Don’t reduce the economics to tokens versus GPUs
The same principle applies to cost.
Consumption-based services transfer utilisation risk to the provider. If a workload is bursty or uncertain, that is enormously valuable.
Capacity-based models do something different. They give the organisation a known capacity envelope and can make spending more predictable, but the customer carries more of the utilisation risk.
Neither is automatically cheaper.
As applications become more sophisticated, the economics also become harder to understand from token pricing alone. Context size, model selection, retrieval, caching, concurrency, retry behaviour and agent loops can all change the cost of producing an outcome.
Gartner predicts inference cost per agentic workflow will increase more than fivefold through 2028 despite falling unit inference costs, because increasingly sophisticated agents will perform more steps and consume more inference in completing a task. (Gartner)
That makes a different metric more useful:
- What does it cost to complete the task?
- Cost per application processed.
- Cost per enquiry resolved.
- Cost per document assessed.
- Cost per automated workflow completed.
Until the university can understand AI at that level, deciding whether tokens or dedicated capacity are cheaper is largely theoretical.
Universities have another GPU question
Universities also have an infrastructure consideration that most organisations do not.
They already operate, consume or fund significant research computing environments.
Research AI and institutional AI are different. Their users, funding models, governance, workloads and service expectations should not simply be collapsed into a single environment.
But if institutional AI eventually creates sustained GPU demand, there is a legitimate question about whether the underlying capacity should always remain completely separate.
- Could different security domains and allocation models draw from a common physical GPU pool?
- Could steady institutional inference improve baseline utilisation around variable research demand?
- Could excess capacity move between environments while preserving grant accounting, data separation and governance?
- Or are research workloads, funding constraints and service requirements sufficiently different that separation remains the better model?
There will not be one answer across the sector.
That is a question we will be exploring with university IT leaders, researchers and shared infrastructure operators at the 2026 eResearch Australasia Conference. The MCS workshop, Sharing the GPU: Can Research and Institutional IT Build AI Capacity Together?, will examine the practical realities of cost recovery, grant acquittal, ethics, governance, sovereignty, security and the trade-offs between institutional, shared and national capacity.
The important point for a CIO is that the calculation for dedicated GPU capacity may eventually need to consider the university’s broader compute estate, not only the corporate AI team.
Don’t choose one AI architecture for the university
The likely destination is not public AI or private AI.
It is both, plus several points between them.
A university could reasonably have Copilot supporting staff productivity, frontier APIs solving complex reasoning tasks, shared open-weight inference supporting a high-volume knowledge service, dedicated open-weight capacity for a sensitive operational application and dedicated GPUs supporting a portfolio of specialised workloads.
That is not architectural inconsistency.
It is workload placement.
MCS has built Launch AI around the same principle. Universities can begin with a shared managed environment and move specific workloads to dedicated capacity as their requirements become clearer, while retaining a vendor-neutral architecture rather than treating public and private models as mutually exclusive choices. The service is designed around Australian sovereign infrastructure, capacity-based pricing and a managed operating model rather than requiring customers to build their own GPU platform.
The important part is not the sequence. It is maintaining choice.
University applications may remain in service for years. The leading model, pricing structure and hardware platform will change much faster than that.
The model should therefore be a component of the architecture, not the architecture itself.
Five questions to ask as an AI workload matures
When an AI experiment starts turning into an institutional service, five questions can help determine what should happen next:
- Does this workload genuinely require a frontier model, or can another model reliably meet the required quality threshold?
- Is demand still uncertain, or can we now measure utilisation and cost per completed outcome?
- Do the data, integration, control or sovereignty requirements justify greater control over the inference environment?
- Would shared managed inference meet those requirements, or is there a specific reason to reserve dedicated capacity?
- Do we have enough sustained demand across the portfolio to justify dedicated GPU infrastructure?
The goal is not to graduate from public AI to private AI.
It is to know enough about each workload to make the architecture decision deliberately.
Start with consumption while demand is uncertain. Consider open-weight models when the workload is sufficiently understood to benchmark them. Introduce dedicated capacity when utilisation, control or sovereignty justify it. Invest in dedicated GPU infrastructure when the portfolio of workloads can actually support it.
And retain access to frontier models wherever they remain the best tool for the job.
The next phase of institutional AI is not about choosing one model or one infrastructure platform. It is about giving universities enough choice to put the right model, on the right infrastructure, for the right workload.