The Architecture at a Glance

M365 Copilot → Copilot Studio Agent → Custom Connector → Hybrid Bridge → Your On-Prem REST API

The main challenge is that last hop — getting from Microsoft's cloud into your network securely. There are a few options for that bridge, and the most common one is outlined below.

Step-by-Step

1. Expose Your On-Prem API Securely (Without Opening It to the Internet)

You have two primary options here:

Option A — Azure Relay / Hybrid Connection (Recommended for simplicity)

• Set up an Azure Relay resource in your Azure subscription.
• Install the Hybrid Connection Manager on a server inside your network that can reach the KB API.
• This creates an outbound-only connection from your network to Azure — no inbound firewall rules needed.
• Your on-prem API becomes reachable from Azure services through the relay.

Option B — On-Premises Data Gateway (Power Platform native)

• Install the Microsoft On-Premises Data Gateway on a machine inside your network.
• This is the same gateway used by Power BI, Power Automate, etc.
• It creates an outbound encrypted channel to the Power Platform.
• This is natively supported by Power Platform custom connectors.

Option B is often the path of least resistance since it's built into the Power Platform ecosystem that Copilot Studio sits on.

2. Create a Custom Connector in Power Platform

• Go to make.powerapps.com → Custom Connectors (under Data or Dataverse).
• Create a new connector and point it at your API. You can import a Swagger/OpenAPI spec if your KB API has one, which makes this much easier.
• Define the actions (e.g., SearchKnowledgeBase, GetArticleById) with the request/response schemas.
• Under the General tab, set the host to your on-prem API endpoint. If using the on-prem data gateway, check the "Connect via on-premises data gateway" option.
• Configure authentication to match whatever your KB API requires (API key, OAuth, basic auth, etc.).

3. Create a Power Automate Flow as Middleware (Optional but Recommended)

This step gives you more control over how data flows:

• Create a Power Automate flow with an HTTP trigger (so Copilot Studio can call it).
• In the flow, call your custom connector, pass the user's query to your KB API, and format the response.
• This lets you add logic like filtering results, formatting citations, handling errors gracefully, etc.

4. Build the Copilot Studio Agent

• Go to copilotstudio.microsoft.com.
• Create a new agent (you can publish it to M365 Copilot later).
• Define Topics or use the generative answers capability:
  • For a structured approach: create topics that capture the user's question, call your Power Automate flow or custom connector as an action, and return the response.
  • For a more natural approach: use Generative AI orchestration and register your connector/flow as a "plugin action" — Copilot will decide when to call it based on the user's question.
• Under Actions, add your Power Automate flow or custom connector directly.

5. Publish to M365 Copilot

• In Copilot Studio, go to Channels → Microsoft 365 Copilot.
• Enable the channel and publish.
• An admin then approves the agent in the Microsoft 365 Admin Center → Integrated Apps section.
• Once approved, users across your org can invoke the agent from within M365 Copilot (in Teams, the web experience, etc.) by @mentioning it or through the Copilot interface.

Key Decisions You'll Need to Make

• Gateway choice: The On-Premises Data Gateway is simpler for Power Platform. Azure Relay or Azure API Management gives you more flexibility if you want other services to use the same bridge.
• Authentication: How does your KB API authenticate requests? You'll need to replicate that in the custom connector.
• Response formatting: Copilot works best when it gets clean, well-structured text back. If your API returns complex JSON, the Power Automate flow layer is really helpful for shaping that into something the AI can reason over.

Licensing Note

Copilot Studio agents that are published to M365 Copilot require either M365 Copilot licenses for the users accessing them, or Copilot Studio capacity depending on how you deploy. Worth checking with your M365 admin on what licenses you already have in play.

APIM is a really strong choice, especially in an enterprise setting. It actually simplifies some steps while giving you a lot more control. The overall flow changes surprisingly little; it mainly replaces and consolidates Steps 1 and 2.

Updated Architecture

M365 Copilot → Copilot Studio Agent → Custom Connector → Azure APIM → Self-Hosted Gateway → Your On-Prem REST API

The key piece is the APIM Self-Hosted Gateway, which is a containerized version of the Azure API Management gateway that you run inside your network.

What Changes

Step 1 — Replace the Hybrid Bridge with APIM + Self-Hosted Gateway

• Create an Azure API Management instance in your Azure subscription. The Developer or Standard v2 tier works for this; Premium has built-in VNet integration but is expensive.
• Deploy a Self-Hosted Gateway inside your on-prem network (runs as a Docker container or on Kubernetes). This gateway maintains an outbound connection to your APIM instance in Azure — again, no inbound firewall rules needed.
• In APIM, import your KB's REST API definition (Swagger/OpenAPI spec, or define it manually).
• Assign that API to the self-hosted gateway so requests route through your on-prem gateway rather than the Azure-managed one.

This single step replaces both the hybrid bridge and a lot of what the custom connector definition was doing.

Step 2 — The Custom Connector Gets Simpler

• Your custom connector in Power Platform now just points to your APIM endpoint (e.g., https://your-apim.azure-api.net/kb/search), which is a public, managed, secured URL.
• No need to check "connect via on-premises data gateway" — APIM handles the on-prem routing transparently.
• Authentication becomes cleaner: you can use APIM's built-in subscription keys, OAuth validation, or managed identity on the front door, regardless of what your backend KB API uses. APIM handles the credential translation.

Steps 3–5 — Essentially Unchanged

The Power Automate flow (if you use one), Copilot Studio agent setup, and M365 publishing all stay the same. They just talk to APIM instead of going through the on-prem data gateway.

Why APIM Is Worth the Extra Setup

• Single front door — Other tools, apps, or agents in your org can also hit the same APIM endpoint. You're not locked into Power Platform.
• Policies — Rate limiting, caching, request/response transformation, all configurable without touching your backend API. The caching alone could meaningfully reduce load on your KB.
• Monitoring — Built-in Application Insights integration gives you full visibility into who's calling what, latency, error rates, etc.
• Security layering — You can enforce OAuth, IP filtering, JWT validation, or subscription keys at the APIM level, adding a security layer in front of your KB API without modifying it.
• Response shaping — APIM policies can transform your KB API's response format before it even reaches Copilot Studio, potentially eliminating the need for that Power Automate middleware layer entirely.

Cost Consideration

The self-hosted gateway is included with most APIM tiers, but the APIM instance itself has a monthly cost. The Developer tier is around $50/month and fine for dev/test. Standard v2 starts around $170/month and is better for production. Compare that to the on-prem data gateway, which is free but gives you far fewer capabilities.

That's a perfectly valid approach — and very common. But it does raise the key question: if APIM is running in Azure (not on-prem) and your KB API isn't open to the internet, how does APIM reach your backend?

The self-hosted gateway was solving that problem. Without it, you need a network path from Azure to your on-prem network. Here are the options:

Option A — Site-to-Site VPN (Most Common)

• Your org sets up a Site-to-Site VPN between your on-prem network and an Azure Virtual Network (VNet).
• APIM is deployed with VNet integration so it can route traffic through that VPN tunnel.
• APIM calls your KB API's internal IP/hostname, and the traffic flows through the VPN — never touching the public internet.

Requirements:

• A VPN gateway appliance on-prem (many enterprise firewalls already support this — Cisco, Palo Alto, Fortinet, etc.)
• An Azure VPN Gateway resource
• APIM on a tier that supports VNet integration (Developer or Premium for full VNet injection; Standard v2 supports VNet integration in a lighter model)

Worth asking your network team: You may already have a Site-to-Site VPN or ExpressRoute connection to Azure. A lot of orgs with M365 Copilot and Azure AD already have this in place.

Option B — Azure ExpressRoute

• Same concept as the VPN, but uses a private dedicated circuit instead of an encrypted tunnel over the internet.
• Higher bandwidth, lower latency, more predictable — but significantly more expensive and complex to set up.
• If your org already has ExpressRoute for other Azure workloads, APIM can ride on it.

Option C — Azure Relay as the Bridge Behind APIM

• If a VPN isn't in place and standing one up is a heavier lift than you want right now, you can use Azure Relay as the tunnel.
• APIM's backend would point to the Azure Relay endpoint, and the Hybrid Connection Manager on-prem forwards requests to your KB API.
• This keeps the "no inbound firewall changes" benefit and doesn't require VPN infrastructure.
• The trade-off is a small amount of added latency and a dependency on the relay service.

How This Affects the Steps

The Copilot Studio agent, custom connector, and publishing steps are all completely unchanged regardless of which option you pick. The only thing that changes is how APIM's backend is configured:

That's the ideal scenario — the hard part is already done. You essentially have a proven pattern in place, and this becomes a matter of replicating it for your KB API and wiring up the Copilot Studio agent.

Your Simplified Steps

1. Add Your KB API to APIM

Since you've done this before with other APIs, this is familiar territory:

• Import your KB API into APIM (OpenAPI spec, or define operations manually).
• Configure the backend to point to your on-prem KB server's internal hostname/IP — same as your other on-prem APIs.
• Apply your existing APIM policy pattern for the on-prem hop (you likely already have a reusable policy fragment for this).
• Set up the inbound authentication policy to validate Microsoft Entra ID (Azure AD) tokens, just like your other APIs. validate-jwt or validate-azure-ad-token policy — whatever you're already using.

At this point, your KB API is callable from Azure, secured by a Microsoft token, and routed to on-prem through your existing VNet. Nothing new here.

2. Create a Custom Connector in Power Platform

• Go to make.powerapps.com → Custom Connectors.
• Point it at your APIM endpoint (e.g., https://your-apim.azure-api.net/kb).
• For authentication, configure OAuth 2.0 with Azure AD — register an app in Entra ID (or reuse one) that has permission to call your APIM. The connector will acquire a token and pass it with each request, which APIM validates just like it does for your other APIs.
• Define the actions that map to your KB API operations — something like SearchArticles(query: string) and maybe GetArticle(id: string).

3. Build the Copilot Studio Agent

• Go to copilotstudio.microsoft.com and create a new agent.
• Under Actions, add your custom connector.
• Configure how the agent uses it — either through explicit Topics ("When the user asks a knowledge question, call SearchArticles") or through generative orchestration (let the AI decide when to call it based on the user's intent).
• The key thing to get right: the system prompt / instructions for the agent. Tell it what the KB contains, when to search it, and how to present results. This is where the quality of the experience lives.

4. Publish to M365 Copilot

• In Copilot Studio → Channels → enable Microsoft 365 Copilot.
• Publish, then have your M365 admin approve it in the Admin Center → Integrated Apps.
• Users can then invoke it from within Copilot.

Summary

Because your APIM infrastructure, VNet connectivity, and authentication pattern are already established, you've eliminated all the hard infrastructure questions. The work is:

1. One more API in APIM (routine for your team)
2. One custom connector (straightforward since APIM gives you a clean, well-documented endpoint)
3. One Copilot Studio agent (the new piece, but not complex)
4. Publish and approve (admin workflow)

The thing worth spending the most time on is that agent system prompt in Step 3 — how you instruct the agent to interpret user questions, when to call the KB vs. answer from general knowledge, and how to format/cite the KB responses. That's what makes the difference between a useful agent and a frustrating one.