AI Operating Models: CoE, Federated, or Both?

Why Operating Models Matter

AI success is a "team sport." BCG research shows only 5% of companies have truly scaled AI across the enterprise—and a key factor is whether they treat AI as an organizational capability rather than isolated pockets of experimentation.

Intel's AI Center of Excellence demonstrates what's possible with the right model: best practices led to doubling business impact over three years, delivering over $1 billion in value. But the right model depends on your context.

Workplace Scenario: The AI Steering Committee

You join a newly formed AI Steering Committee meeting. Different leaders are debating how to organize AI efforts:

Alice (AI CoE Lead) advocates for centralization:

"We need a single AI platform for everyone. Without central standards, we'll have governance chaos and duplicate efforts. My team can ensure quality, share best practices, and prioritize investments strategically."

Bob (Business Unit Manager) argues for decentralization:

"My team needs to move fast and tailor AI to our specific domain. Waiting for a central team slows us down. We know our customers and data better than anyone. Let each division own their AI initiatives."

Your role: As an AI Champion, you're tasked with connecting your department's AI efforts with either the central team or across departments. Understanding the tradeoffs helps you bridge these perspectives.

The Three Operating Models

Model 1: Centralized Center of Excellence (CoE)

Structure: A dedicated AI team serves the entire organization.

Advantages:

• Consistent standards and quality
• Efficient use of scarce AI talent
• Easier knowledge sharing
• Clear governance and oversight

Disadvantages:

• Can become a bottleneck
• May lack deep domain expertise
• "Throw it over the wall" dynamics
• Business units feel less ownership

Best For: Early AI adoption, regulated industries, limited AI talent

Model 2: Federated (Embedded in Business Units)

Structure: Each business unit has its own AI capabilities.

Advantages:

• Deep domain expertise
• Direct business alignment
• Faster iteration
• Strong local ownership

Disadvantages:

• Duplicated efforts
• Inconsistent practices
• Siloed knowledge
• Governance challenges

Best For: Mature AI organizations, diverse business units, abundant talent

Model 3: Hybrid (Hub and Spoke)

Structure: Central team provides infrastructure and standards; business units have embedded AI staff for domain work.

Central Team Provides:

• Shared platforms and tools
• Standards and best practices
• Governance and oversight
• Specialized expertise (MLOps, ethics)
• Training and capability building

Embedded Teams Handle:

• Domain-specific use cases
• Business requirements
• Change management
• Day-to-day operations

Best For: Most medium-to-large enterprises scaling from experimental to systematic AI

McKinsey observes that many leading firms combine central and distributed efforts: centralize platforms and governance, decentralize execution in business units.

Debate Simulation: Alice vs. Bob

Consider each argument. Who makes the stronger point?

Case Study: Intel's AI Center of Excellence

Intel established a centralized AI CoE that delivered measurable results:

What made it work:

• Strong executive sponsorship
• Standardized processes and platforms
• Regular best-practice sharing across units
• Clear prioritization framework
• Investment in reusable components

Contrast: Organizations with fully federated approaches often see siloed, duplicate efforts—multiple teams solving the same problem independently without sharing learnings.

The Champion's Role in Each Model

Regardless of operating model, AI Champions serve as connectors:

Key Question: Who do you need to connect with? Identify at least one person or team outside your immediate group who you should coordinate with on AI initiatives.

Poll: Your Current State

Reflect on how AI is currently organized in your context:

Completion: Your Collaboration Plan

To complete this module, answer two questions:

Question 1: Operating Under Your Model

Given your organization's structure (or the one you expect), how will you operate as an AI Champion to ensure knowledge is shared and efforts aren't duplicated?

Write 2-3 sentences describing your approach.

Example: "In our hybrid model, I will submit our use case ideas to the monthly AI CoE review meeting and advocate for leveraging existing platform components. I'll also share our learnings in the AI community Slack channel."

Question 2: Your Coordination Contact

Name one resource, team, or person you will coordinate with for AI initiatives.

Identify who they are and why they're relevant.

Example: "I'll connect with Sarah from the central data platform team—she can help us leverage the existing feature store rather than building our own data pipelines."

Key Takeaways

• No single operating model is universally best—context determines the right choice
• Centralized enables consistency and governance; federated enables speed and domain expertise
• Most mature organizations adopt hybrid models: central platforms, distributed execution
• AI Champions are connectors in any model—bridging gaps, sharing knowledge, preventing silos
• Intel's CoE success shows that with the right structure, AI can deliver billions in value

Next Steps

In the final practitioner module, we'll explore The AI Factory—understanding how to leverage reusable platforms and components to accelerate AI delivery across the enterprise.