Building AI Solutions with Azure AI Foundry and Copilot Studio

You've deployed a model. You've played with the chat playground. Now you need to build something real.

This guide takes you from "AI is working in my tests" to "AI is integrated into my application and serving users." We'll use Azure AI Foundry for orchestration and Copilot Studio for user-facing interfaces.

What We're Building

A practical internal IT support assistant that:

• Answers questions using company documentation
• Escalates to humans when confidence is low
• Integrates with Microsoft Teams for user access
• Logs interactions for improvement

This isn't a toy demo. It's a pattern you can adapt for real use cases.

Prerequisites

Before starting, you need:

• Azure subscription with AI services enabled
• Access to Azure AI Foundry (ai.azure.com)
• Copilot Studio license (included with some M365 plans)
• Basic understanding of REST APIs
• Some documents to use as a knowledge base

Part 1: Building the AI Backend in AI Foundry

Step 1: Set Up Your Project

1. Go to ai.azure.com
2. Create a new project: "IT-Support-Assistant"
3. Select your resource group and region
4. Enable Azure AI Search connection (we'll need this for RAG)

Step 2: Prepare Your Data

Create an index from your IT documentation:

1. Navigate to Data + indexes in your project
2. Click New index
3. Choose data source:
   • Azure Blob Storage for large document sets
   • Upload files for smaller collections
4. Upload your IT documentation (PDFs, Word docs, Markdown)
5. Configure chunking:
   • Chunk size: 1024 tokens (good balance for Q&A)
   • Overlap: 128 tokens (maintains context across chunks)
6. Create the index and wait for processing

Tip: Start with 10-20 representative documents. You can add more later, but it's easier to iterate on a smaller set.

Step 3: Deploy Your Model

1. Open Model catalog
2. Deploy GPT-4o (or GPT-4o-mini for cost savings)
3. Configuration:
   • Tokens per minute: Start with 10K (scale later)
   • Content filter: Keep default settings
4. Note your deployment name — you'll need it

Step 4: Create the Prompt Flow

This is where the magic happens. A prompt flow chains together data retrieval and AI generation.

1. Go to Prompt flow → Create
2. Choose template: Chat with your data
3. Configure the flow:

Node 1: Query Rewriting

You are a query rewriter for an IT support system.
Rewrite the user's question to be more specific and searchable.
If the question is already clear, return it unchanged.

User question: {{user_question}}
Rewritten query:

Node 2: Document Retrieval

• Connect to your index
• Top K results: 5
• Search type: Hybrid (vector + keyword)

Node 3: Response Generation

You are an IT support assistant for [Company Name].

CONTEXT (from internal documentation):
{{retrieved_documents}}

GUIDELINES:
- Answer based ONLY on the provided context
- If the context doesn't contain the answer, say "I don't have information about that. Let me connect you with the IT team."
- Be concise but complete
- For step-by-step procedures, use numbered lists
- Never invent procedures not in the documentation

USER QUESTION: {{user_question}}

RESPONSE:

Node 4: Confidence Check

Based on your response, rate your confidence (HIGH, MEDIUM, LOW):
- HIGH: Answer directly from documentation
- MEDIUM: Answer derived from documentation but not explicit
- LOW: Limited relevant documentation found

Confidence:

4. Test the flow with sample questions
5. Iterate on prompts until quality is acceptable

Step 5: Add Evaluation

Before deploying, measure quality:

1. Create a test dataset:

[
  {
    "question": "How do I reset my password?",
    "expected": "Contains steps for password reset portal"
  },
  {
    "question": "What's the VPN server address?",
    "expected": "Contains specific server URL"
  }
]

2. Run evaluation:

   • Groundedness: Is the response based on retrieved docs?
   • Relevance: Does it answer the question?
   • Coherence: Is it well-structured?

3. Review results and refine prompts

Step 6: Deploy as Endpoint

1. Click Deploy from your prompt flow
2. Choose Managed online endpoint
3. Configure:
   • Instance type: Standard_DS3_v2 (start small)
   • Instance count: 1 (scale based on usage)
4. Deploy and note the endpoint URL + API key

Test the endpoint:

curl -X POST "https://your-endpoint.inference.ai.azure.com/score" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"question": "How do I connect to VPN?"}'

Part 2: Building the User Interface with Copilot Studio

Now we'll create a Teams bot that calls your AI Foundry endpoint.

Step 1: Create a New Copilot

1. Go to copilotstudio.microsoft.com
2. Create new copilot: "IT Support Bot"
3. Skip the AI-generated topics for now

Step 2: Create a Power Automate Flow for AI Foundry

Copilot Studio needs to call your AI Foundry endpoint. We'll use Power Automate as the bridge.

1. In Copilot Studio, go to Actions → Add an action
2. Choose Create a new flow
3. Build the flow:

Trigger: When Power Virtual Agents calls a flow

• Input: UserQuestion (text)

HTTP Action:

• Method: POST
• URI: Your AI Foundry endpoint URL
• Headers:
  • Authorization: Bearer YOUR_API_KEY
  • Content-Type: application/json
• Body:

{
  "question": "@{triggerBody()['text']}"
}

Parse JSON:

• Content: Body from HTTP action
• Schema: Generate from sample response

Response:

• Return response and confidence to Copilot Studio

4. Save and publish the flow

Step 3: Create the Conversation Topic

1. In Copilot Studio, create a new topic: "IT Support Question"

2. Add trigger phrases:

   • "I have a question"
   • "Help me with"
   • "How do I"
   • "What's the process for"

3. Build the conversation:

Question node:

• "What do you need help with?"
• Save to variable: UserQuestion

Action node:

• Call your Power Automate flow
• Pass UserQuestion
• Store response in AIResponse
• Store confidence in Confidence

Condition node:

• If Confidence equals "LOW":
  • Message: "I'm not certain about this. Let me connect you with the IT team."
  • Transfer to agent or create ticket
• Else:
  • Message: AIResponse
  • Ask: "Did this answer your question?"

Follow-up:

• If yes: "Great! Anything else I can help with?"
• If no: Offer to create a support ticket

Step 4: Add Fallback Handling

For questions outside IT scope:

1. Go to System topics → Fallback
2. Customize to redirect to IT topic or explain scope

Step 5: Publish to Teams

1. Go to Channels → Microsoft Teams
2. Configure bot details:
   • Name: "IT Support Assistant"
   • Description: "Get help with IT questions"
3. Submit for admin approval (or approve yourself if admin)
4. Add to Teams

Part 3: Production Considerations

Monitoring

Set up observability:

AI Foundry:

• Enable Application Insights for your endpoint
• Track: latency, token usage, error rates

Copilot Studio:

• Review Analytics dashboard
• Monitor: session counts, resolution rates, escalations

Create alerts for:

• Latency > 5 seconds
• Error rate > 5%
• Daily cost exceeding threshold

Cost Management

AI costs can surprise you. Implement controls:

1. Set Azure budgets with alerts at 50%, 80%, 100%
2. Use GPT-4o-mini where possible (10x cheaper than GPT-4o)
3. Cache common responses for frequently asked questions
4. Rate limit the Teams bot to prevent abuse

Continuous Improvement

Your AI gets better with feedback:

1. Log all interactions (question, response, confidence, user feedback)
2. Review low-confidence responses weekly
3. Add missing documentation when gaps appear
4. Refine prompts based on failure patterns
5. Expand test dataset with real user questions

Security

• Store API keys in Azure Key Vault, not in flows
• Implement authentication for the Teams bot
• Review content filters in AI Foundry
• Ensure documentation doesn't contain sensitive data unintentionally

Troubleshooting Common Issues

"I don't have information about that" for everything

• Check index is properly created
• Verify search is returning results (test in AI Foundry)
• Adjust chunk size or search parameters

Slow responses

• Check endpoint instance size
• Consider caching for common questions
• Reduce Top K results if retrieval is slow

Inconsistent answers

• Add examples to your system prompt
• Increase temperature if too repetitive, decrease if too random
• Improve document quality (AI is only as good as your data)

Teams bot not responding

• Check Power Automate flow is published
• Verify API key hasn't expired
• Review Copilot Studio error logs

What's Next

Once this pattern is working:

1. Add more data sources — Connect SharePoint, ServiceNow, etc.
2. Implement ticket creation — Auto-create tickets from the bot
3. Add analytics — Dashboard of common questions and trends
4. Expand scope — HR questions, facilities, other departments

You've built the foundation. The same architecture scales to more complex scenarios.

The code is the easy part. The hard part is curating good documentation and iterating based on real user feedback.

Start small. Measure everything. Improve continuously.