On Saturday, May 2, we held our second full-day Kubernetes training event with the Boston Kubernetes Meetup community at the Microsoft NERD Center in Cambridge. The focus this year was AI networking on Kubernetes — moving beyond foundational Kubernetes networking into the operational, security, and architectural challenges introduced by modern AI systems and AI agents.

The event was intentionally hands-on and instructor-led, with attendees spending the day working through labs, discussions, demos, and real-world design considerations. About 30 attendees gave up a Saturday to attend. They ranged from relative beginners with K8s to experts who knew K8s but came for the LLM API overview or the agentic governance talks.

A huge thank-you goes to everyone who attended, asked questions, and helped create a highly interactive atmosphere throughout the day.

Why We Ran This Event

AI systems are changing traffic patterns, application architectures, and operational models inside Kubernetes environments. Traditional north/south web traffic assumptions no longer fully apply. Clearly, agentic AI is bringing even greater compliance and governance challenges to the already new threat landscape of Generative AI.

I believe everyone in IT needs to know at least the basics of:

• Large language models (LLMs)
• AI gateways
• AI agents
• Retrieval systems
• Vector databases
• Tool-calling frameworks
• External APIs
• Multi-model inference services

These systems introduce new networking and security concerns that many Kubernetes engineers have not had to solve before. The goal of the training day was to bridge that gap with practical content rather than high-level theory.

Agenda Overview

The event agenda focused on both foundational and advanced topics related to AI traffic inside Kubernetes environments. Sessions included:

• K8s fundamentals deck (First presentation from Michael)
• Killercoda.com/f5-se
  • Hello App Basics (online lab showing pod-to-pod communication)
• LLM API & prompting best practices (Second presentation from Michael)
• Hands-on demo (presentation from Ben)
  • use a Github codespace
  • deploy kind, a cluster, a demo app
  • deploy k8gpt, grafana, and kagent
• agentgateway for agent governance (Nina’s presentation)
  • Sandbox environment (lab we did with Nina)
  • agentgateway and MCP auth (additional lab we did not cover)

The structure combined whiteboarding, architecture walkthroughs, live demos, and hands-on labs so attendees could immediately apply concepts during the sessions themselves.

Instructor-Led and Community-Focused

The event was led by myself, Isabella Langan, Benjamin Hautefeuille, and Nina Polshakova. There was a very strong emphasis on practical engineering discussions rather than vendor marketing - this is something I hold very strictly to, especially for Training Days.

Key Takeaways

AI Changes Traffic Patterns

AI applications often create dramatically different internal traffic flows compared to traditional microservices applications. Tool-calling agents and chained inference systems can generate complex service-to-service communication paths that are difficult to predict and govern. The LLM API is itself important to learn. Attacks are semantic in nature.

Security Models Need to Evolve

Traditional API security approaches are not always sufficient for AI systems. Identity propagation, agent permissions, prompt handling, data governance, and outbound access controls all become increasingly important.

Observability Becomes Critical

AI systems introduce more non-deterministic behavior into distributed systems. Strong observability practices become essential for debugging latency, failed agent workflows, and inference bottlenecks.

Kubernetes Remains the Operational Foundation

Despite all the excitement around AI frameworks and models, Kubernetes continues to be the operational backbone enabling scalable deployment, networking, and governance for enterprise AI systems.

Community Feedback

From my discussions and the survey results, people REALLY want hands-on training and are willing to give up a weekend to find it. I intend to do more here. What’s my motive? I’m forced to learn this stuff in order to teach it.

Looking Ahead

This training day reinforced how quickly AI infrastructure engineering is evolving. It’s incredibly overwhelming, even for those of us who have been in the industry for decades.

I think I will focus a future event at a more advanced persona - the K8s-fluent AI engineer who already has the 200-level knowledge, perhaps the 300-level knowledge. I recognize that beginners to the industry need to learn the foundations also, but I am not sure I can keep teaching K8s to a new crop of beginners every session. I’ll try to find a balance.

Thanks again to everyone who attended and helped make the event successful.

See you at the next one.

I’ve replaced two recessed lights on the ground floor in the past month, and since pretty much all of them were installed around the same time — about five years ago — I’m expecting more to go. These are notes to my future self because I’d forgotten which breakers and the brand of light I used last time.

The breaker box

The ground floor lights are split across a few circuits. Before touching anything, flip the right breaker and use a non-contact voltage tester to confirm the power is actually off.

The fixture

This is the replacement I used. Remember:

• canless: these are really flat
• Size: (e.g., 4-inch in my mudroom and pantry but larger in the living room)
• Color temperature: (e.g., 2700K warm white to match the others)

Replacement steps (quick reference)

1. Shut off the correct breaker. Verify with voltage tester.
2. Pull down the existing trim — it’s usually held by spring clips.
3. Disconnect the wires from the existing box.
4. Connect the new driver box
5. Push the new driver box up into the ceiling.
6. Clip the new trim into place.
7. Restore power and test.

The whole job takes about 15 minutes once you’ve done it once.

Done

Recently I’ve been feeling an instinctive alarm bell sound as I feel fundamental technology skills slipping away in favor of moving faster with AI. At this point I’m not judging bad vs good, but in technology, AI helps us move faster while it frees us from learning how we did this.

There is a pattern in systems engineering that repeats itself across every generation of tooling: we build an abstraction that is brilliant at hiding complexity, celebrate the productivity gains, and then spend the next decade paying back the debt that abstraction quietly accumulated.

I recently felt this again when reading an example agent tooling with catastrophic consequences. Anthropic’s MCP specification deserves the “USB-C for AI” metaphor. It defines a clean, universal interface between an AI host, a language model, and the external world — local filesystems, databases, APIs, Kubernetes clusters. If you can expose a tool over JSON-RPC, the LLM can use it.

My instinctive alarm bells are set off by a new factor: non-deterministic systems making decisions. I used to feel that abstractions just allowed for faster development at the cost of foundational learning. Now I feel we have a combination of:

• abstractions making development faster (fewer foundational skills)
• AI making coding accessible to beginners (vibecoding)
• frameworks like MCP mass-enabling interaction with external systems (real world consequences)
• AI models being non-deterministic (intelligent guessing)
• Natural language prompts (requiring semantic reasoning from LLMs)

Each factor builds on the others, but it’s the combination of intelligent guessing and real-world consequences that summarizes my concern. The vibecoding and “speed over accuracy” culture adds to the concern.

The Traffic Flow Nobody Is Auditing

Before I get philosophical, it’s worth being precise about what actually happens when an MCP-enabled agent runs a request. The flow is:

1
2
3
4
5
6
7
8
9
10
11
12
Host Application
    │
    ├─► LLM (probabilistic reasoning layer)
    │       │
    │       └─► Tool Call Selection  ← "intelligent guess"
    │
    └─► MCP Server (executes side effects)
            │
            ├─► Filesystem operations
            ├─► API calls
            ├─► Database writes
            └─► Shell commands

Notice the component in the middle: probabilistic reasoning. The LLM does not parse a deterministic rule to decide which tool to invoke. It reasons about the semantics of the available tools — their names, their descriptions, their parameter schemas — and makes an inference about which one best serves the user’s intent. It is, in the most literal sense, an educated guess.

In isolation, that’s fine. In combination with tools that carry real-world side effects, it’s concerning. Especially when the tools themselves may be vibecoded (potentially low quality) and the natural language input will be imperfect.

Semantic Access Is Not Deterministic Control

Here’s an example I fear will happen with home-grown MCP servers for BIG-IP, for example.

When you configure an F5 BIG-IP, you work with objects that have precise, non-overlapping definitions. A VIP (Virtual IP) is the front-end address that external clients connect to. A virtual server is the F5 configuration object that listens on that VIP and applies policies. These terms are close enough in everyday language that a reasonable person — or a language model — might use them interchangeably. In the F5 management plane, they are not interchangeable. Conflating them means you are operating on the wrong object, applying the wrong policy, and potentially taking down traffic for a production service while believing you are doing something benign.

Now expose your network automation layer as MCP tools. Name one get_virtual_server_config and another get_vip_status. Ask an LLM agent to “check the load balancer for the payments service.” Which tool does it call? It depends entirely on which description the model finds semantically closer to “load balancer” and “check.” There is no compilation step to catch the mismatch. There is no type system enforcing the distinction. There is a probability, and a reliance on the description of the tools learned by the model.

This is what I mean by Abstraction Debt — when human operators lose understanding of the systems they govern. The debt is not visible in normal operations. It becomes visible when you need it: during an incident, when you are trying to understand what the agent actually did and why, when you are trying to teach others, etc.

Two Classes of Engineer Are Emerging

Again, this is Claude helping me build my thoughts into a case:

Orchestrators assemble systems using semantic reasoning. They write prompts, configure MCP servers, chain tool calls, and build agents that can traverse a multi-hop workflow across half a dozen services. They are productive at a pace that would have seemed impossible five years ago. They are, right now, extremely employable.

Foundationalists understand fundamentals. They know what happens at the TCP level when a JSON-RPC request leaves the MCP client. They can read a packet capture during an incident. They understand process management well enough to know why a forked subprocess inherited the wrong file descriptors. They can reason about the OSI model when an abstraction leaks and the stack trace is unhelpful.

The risk is not that Orchestrators exist. The risk is that we’re incentivizing engineers to become only Orchestrators — and that the feedback loops which historically taught foundational knowledge (debugging, reading logs, understanding failure modes) are being bypassed by systems that hide their internals behind a semantic interface.

The demand curve for Orchestrators will remain high until the first major incident at scale: a misconfigured MCP server with filesystem access, a semantically confused tool call that mutates production data, an agent that “helpfully” applies a Kubernetes manifest to the wrong namespace because the cluster names were similar enough in the model’s embedding space. At that moment, the Orchestrator reaches for their incident playbook and finds it empty. The Foundationalist is already reading the audit logs.

Knowledge Atrophy

(Again, somewhat ironically, Claude has helped me refine the following thoughts.)

A claim made by others but I believe in: when automation handles a task reliably, the human practitioner stops practicing the underlying skill. The skill atrophies. When the automation fails, the human is less capable of recovering than they would have been without the automation.

MCP is a powerful automation layer systems integration. When it works, it removes the need to understand transport protocols, to handle authentication edge cases, to reason about retry semantics. Engineers who use it exclusively will, over time, become less capable of debugging systems at the level where those concerns live. The protocol doesn’t teach you what it hides.

This is not an argument against abstraction. Abstraction is the mechanism by which the industry makes progress. It is an argument for deliberate practice at the layer beneath the abstraction you rely on. The engineers who built the most reliable systems I have worked with could always go one layer deeper than the tooling required. They understood TCP because they had debugged socket timeouts. They understood HTTP because they had read raw request logs. That reservoir of foundational knowledge is what gets drawn on when the abstraction leaks.

MCP makes it easy to never build that reservoir. That is the danger.

Where the Abstraction Leaks

MCP’s abstraction is not watertight. Here are the specific seams where it fails, ranked roughly by incident frequency:

1. Tool Description Drift
Tool descriptions are the primary signal the LLM uses for selection. As tools are updated, renamed, or repurposed, their descriptions often lag. The model continues reasoning against stale semantic signals. There is no schema enforcement to catch this.

2. Implicit State Assumptions
MCP tools are stateless by design at the protocol level. But the systems they connect to are stateful. A tool that “lists files in a directory” and a tool that “moves files to archive” share no transactional context within the MCP call graph. An agent that calls both in sequence across a failure boundary may leave the system in an inconsistent state that neither the agent nor the operator anticipated.

3. Authentication Context Collapse
In a traditional system, the principal making a request is explicit: a service account, a user token, an IAM role. In an MCP-enabled agent flow, the effective principal is often the host process. The LLM’s reasoning — which is untraceable in any cryptographic sense — is the de facto access control layer. This is not a security model. It is the absence of one.

4. Error Signal Ambiguity
When an MCP tool call fails, the LLM receives a text error message and reasons about what to do next. It may retry with different parameters. It may call a different tool. It may hallucinate a recovery strategy. None of this is logged in a way that a standard SIEM or observability platform can parse without custom instrumentation.

The Call to Action

The engineers who will be valuable in five years are not the ones who orchestrated the most agent workflows. They are the ones who understand why those workflows fail, can trace a failure through the full stack, and can reason about the system’s behavior at the level of the protocol rather than the semantic description.

The path is not to avoid MCP. It is to use it with deliberate awareness of what it hides, and to maintain active practice in the disciplines it makes it easy to skip.

Concretely:

• Read the MCP specification transport layer documentation. Understand that stdio and SSE/HTTP are meaningfully different transport modes with different security boundaries.
• When something goes wrong in an agent workflow, resist the urge to re-prompt. Capture the raw tool call log and trace it manually before asking the model to self-correct.
• Build at least one MCP server from a bare HTTP server before using a framework. The framework abstracts the right things, but you need to know what is being abstracted.
• Keep your Wireshark skills current. JSON-RPC over HTTP is readable in a packet capture. Read it.
• Study your infrastructure tools at the API level, not just through the semantic lens an agent provides. The F5 API, the Kubernetes API, the AWS SDK — the objects in these systems have precise definitions that semantic reasoning does not respect.

The Foundationalist who orchestrates is not just more resilient than the pure Orchestrator. They are the person in the room when the incident happens and the AI-built system has failed in a way no one anticipated. That person will not be replaced by an AI agent. They will be the one telling the AI agent what actually went wrong.

Stay foundational.

In my last post I briefly covered how to install Coder with very basic config options. Now let’s configure authentication away from local accounts in Coder to an OIDC provider.

Architecture Overview

The BIG-IP sits at the edge and handles all the enterprise concerns: who can get in, via what method, and with what level of access. Coder sits internally and manages the lifecycle of dev environments — templated, reproducible, ephemeral.

Pre-requisites

• Coder server deployed
  • I installed Coder on a VM: curl -fsSL https://coder.com/install.sh | sh
  • Docker installed on the Coder server
    • I have Docker configured to use a pre-selected IP range for the default bridge network ¹
• At least one Coder template defined (I use the built-in Docker template to start)
• Docker or Kubernetes available as the workspace runtime

Create a template and then a workspace

From the web UI it’s simple — create a template, create your workspace from this template, click Create. Within 30–60 seconds you have a fresh isolated environment.

The key thing I want to highlight: every workspace is ephemeral by design. Coder supports workspace autostop, so you can configure workspaces to shut down after a period of inactivity. For agentic AI workloads, I set a conservative TTL — the agent runs its task, the environment stops, nobody forgets to clean up.

Verify the environment

After creating the workspace, a quick sanity check:

1
2
3
4
5
6
7
8
9
10
11
# Confirm you're inside an isolated container
hostname
# → my-agent-workspace

# Check that no residual state exists from previous runs
ls -la ~
# → clean home directory, no lingering dotfiles

# Verify network isolation
ip addr
# → only loopback + container veth — no access to prod network

This is the point: each workspace is a clean slate. An agentic AI runs here, does its thing, and the environment is discarded.

Part 2: Configuring F5 BIG-IP APM for OIDC

This is where things get more interesting — and more enterprise. BIG-IP APM (Access Policy Manager) is F5’s solution for identity-aware access control. It handles VPN, ZTNA, SSO, MFA, OAuth, SAML, and more. It’s heavy machinery, but for organizations that already run BIG-IP, it’s incredibly powerful.

My goals here is to have APM handle authentication (who are you?) and authorization (can you access Coder?) before traffic ever hits the Coder server. Users should log in via APM and get SSO into Coder without re-authenticating.

Coder supports OIDC out of the box. APM can act as the OAuth 2.0 / OIDC Authorization Server.

Configure Coder using environment variables

In my last post I mentioned the file /etc/coder.d/coder.env. Here is the file now, with my OIDC settings configured:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# Coder must be reachable from an external URL for users and workspaces to connect.
# e.g. https://coder.example.com

#basic coder network config
CODER_ACCESS_URL=https://coder.my-f5.com
CODER_HTTP_ADDRESS='0.0.0.0:3000'

#OIDC config
CODER_OIDC_ISSUER_URL="https://auth.my-f5.com/f5-oauth2/v1"
CODER_OIDC_EMAIL_DOMAIN="f5.com"
CODER_OIDC_CLIENT_ID="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
CODER_OIDC_CLIENT_SECRET="yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"

# Scopes to request — offline_access gets you refresh tokens
CODER_OIDC_SCOPES="openid,profile,email,offline_access"

#additional OIDC config lines below
CODER_OIDC_IGNORE_EMAIL_VERIFIED=true
CODER_OIDC_SIGN_IN_TEXT="Sign in with F5 APM"
CODER_OIDC_ICON_URL=https://media.ffycdn.net/us/f5-networks-inc/qi443UWRoMTME9ELdEsJ.svg

There are many other configuration options that are set with either environment variables (as in the above file) or using flags at the command line. For a list of these, run coder server --help

After your coder.env file looks as mine does above, you can stop and restart the coder service. Coder will likely fail to start until your OAuth authorization server, confired above with the CODER_OIDC_ISSUER_URL option, is correctly configured.

Configure F5 APM as an Oauth authorization server

Now let’s configure APM.

Create a local user DB

We will not use an external IdP such as Active Directory for this lab. I’ll use local user db on BIG-IP. Almost every enterprise setup would use another IdP, since AzureAD and other IdP’s are so common.

Access > Authentication > Local User DB > Instances.

• Create an Instance
• Create at least 2 test users
• Give them usernames in the format of email address (eg. john.smith@example.com)
  • the email address domain should match CODER_OIDC_EMAIL_DOMAIN configured
• Do not require password change

Create scopes

Access > Federation > OAuth Authorization Server > Scope

• Create 3 scopes, email, offline_access, and profile

Create Client Application

Access > Federation > OAuth Authorization Server > Client Application

• Create a client application with a unique Client ID and Secret. These are what must match the values in the EnvironmentFile above.
• Notice the scopes are added to this application
• The Redirect URI must match what your Coder app will send. Coder’s instructions tell us this will be your CODER_ACCESS_URL + /api/v2/users/oidc/callback
• Ensure Support OpenID Connect is checked, and Authorization Code / Hybrid is checked.
• The Website URL and Website Logo URL are not critical, but the URL points to an image that makes the APM sign-on page look more appealing.

Create Resource Server

Access > Federation > OAuth Authorization Server > Resource Server

• Create a Resource Server. I have None selected for Authentication of this.

Configure JWT claims

Access > Federation > OAuth Authorization Server > Claim

• create claims to be included in JWT tokens issued by APM:
• Coder’s instructions tell us that we can include claims such as email, username, preferred_username, and email_verified. Running the –help flag will uncover more.
• For this lab, I have only provided 2x claims: email and email_verified, the second of which I set to true for every JWT. I found that only email is mandatory.

My F5 APM GUI stopped working at this point and I could not create claims via the GUI. So I created these 2 claims by editing the /config/bigip.conf file.

1
2
3
4
5
6
7
8
9
apm oauth oauth-claim /Common/email {
    claim-name email
    claim-value "%{session.logon.last.username}"
}
apm oauth oauth-claim /Common/email_verified {
    claim-name email_verified
    claim-type boolean
    claim-value true
}

Configure JSON Web Key (JWK)

Access > Federation > JSON Web Token > Key Configuration

Again, my web UI failed me and I couldn’t create JWK config via the GUI. I did so via TMSH, which I will share below

• Create a self-signed RSA cert and key pair to use for signing JWT’s.
  • System > Certificate Management > Traffic Certification Management > Create
  • I created a self-signed cert, named the object JWT_Signing_RSA.
  • The Common Name is self-signed-rsa-keypair-for-JWK
  • Key is 2048 bits

Now that I have a cert-key pair, I can run the tmsh command to configure JWK config:

1
tmsh create apm oauth jwk-config My_Manual_JWK { alg-type RS256 cert JWT_Signing_RSA cert-key JWT_Signing_RSA key-id "unique-id-123" }

Notice my JWK config is called My_Manual_JWK. This will be referenced later.

Create an OAuth Profile

Access > Federation > OAuth Authorization Server > OAuth Profile

• Client Application: reference your created object
• Resource Server: reference your created object
• Check the boxes for Support JWT Token, and Open ID Connect
• The Issuer URL is critical. This URL will be looked up by Coder and must match what Coder expects.
• Choose JWS, and JWT Primary Key, and ID Token Primary Key, as shown.
• create a JWT Refresh Token Encryption Secret. This can be anything but write it down.

Create an Access Profile (Per-Session Policy)

Access > Profiles/Policies > Access Profiles (Per-Session Policies ) > Create

• Create a Profile
  • Profile Type: All
  • OAuth Profile: Reference OAuth Profile created, as seen in screenshot
  • Choose English

• Now, edit using Visual Policy Editor as seen in screenshot

Create a Virtual Server

• Create VirtualServer
  • Configure a HTTP Profile
  • Configure ClientSSL profile
  • Configure Access Profile (Per-Session Policy)

Key takeaways

Ephemeral environments are a prerequisite for safe Agentic AI development. When the entity running code is autonomous, shared environments become unacceptable. You need clean slates, bounded blast radius, and short lifetimes. Coder delivers all three.

Enterprise access control is a first-class concern. It’s easy to prototype with a Coder instance that’s wide open on the network. In production — especially when AI agents are involved — you need identity-aware access. F5 BIG-IP APM is heavy, but it’s the right tool for organizations that already operate BIG-IP infrastructure. The combination of VPN tunnel control, MFA, group-based authorization, and header injection into Coder gives you a layered security posture without requiring changes to Coder itself.

Automation is the next step. Right now my workspace creation is manual. The obvious extension is an agentic workflow that calls the Coder API to provision a workspace, runs a task, and tears it down — all triggered by a CI event or an agent orchestration layer. The Coder REST API and CLI make this straightforward. The BIG-IP doesn’t need to know anything about individual workspaces; it just routes to the subnet.

Don’t skip the firewall rules. Header-based SSO only works if Coder is unreachable except through the proxy. Enforce this at the network layer.

References

• Coder - OIDC Auth
• F5 BIG-IP APM — OAuth Authorization Server configuration

Coder is an open-source platform for self-hosted dev environments. It uses Terraform templates to define workspaces, which means your dev environment is code. You can version it, review it, and destroy it with confidence.

Why ephemeral environments matter for Agentic AI

Agentic AI is different from regular AI-assisted coding. When an agent runs autonomously — executing shell commands, calling APIs, modifying files, spinning up processes — it needs an environment it can own. Shared dev environments are a liability. A runaway agent command, a dependency conflict, a leaked credential in a log file: these become everyone’s problem in a shared space.

Ephemeral environments solve this. Spin one up, let the agent do its work, tear it down. No residue. No blast radius. This is the same logic that drove us to containerize workloads and adopt immutable infrastructure — and it applies even more strongly when the “developer” is an AI.

There’s another angle here too: enterprise compliance. When you’re operating in a large organization, you don’t just need isolation for safety reasons. You need auditability, access control, and the ability to restrict who can reach which environment. That’s where F5 BIG-IP APM comes in.

In this post I’m documenting my setup: Coder for ephemeral workspace management, running on a single EC2 instance to begin with.

Deploy AWS EC2 instance, install Coder, configure basics

Coder can run on different platforms, including K8s. I’m going to use Ubuntu 24.04 on EC2 because it will be a quick lab. I’ll use a m7i.xlarge instance and give myself 100 GB of disk space.

Install Docker

First, I have created a file at /etc/docker/daemon.json with this content:

1
2
3
{
  "bip": "10.10.0.1/24"
}

This is because I want to choose the default bridge network CIDR block for this demo. Why? I am going to use a pre-configured IP range for my containers. This might come in handy later if I decide to implement a VPN for users to reach their docker containers.

After this file exists, I install Docker on the Coder server. I use my own instructions.

Install Coder

Now install Coder:

1
2
3
4
5
6
# 1. Install Coder
curl -L https://coder.com/install.sh | sh

#Run this command to allow the binary to open raw sockets without requiring root:
sudo setcap cap_net_raw+ep $(which coder)

Notice that a user was created called coder:

1
sudo more /etc/passwd | grep coder

Notice these files that were created:

• /usr/lib/systemd/system/coder.service. - this defines the service and references an EnvironmentFile
• /etc/coder.d/coder.env - this EnvironmentFile holds configuration for the service

Notice that the files above exist, but the service is disabled:

1
2
sudo systemctl list-unit-files --state=disabled #Show disabled services
sudo systemctl status coder #this will show the daemon is not enabled

Edit the file /etc/coder.d/coder.env and add/edit these values:

1
2
CODER_ACCESS_URL=https://coder.my-f5.com
CODER_HTTP_ADDRESS='0.0.0.0:3000'

Also, add coder user to docker group, which will allow coder to run docker commands without sudo:

1
2
sudo usermod -aG docker coder
newgrp docker

Enable and start the coder service:

1
2
sudo systemctl enable coder
sudo systemctl start coder

I have a URL coder.my-f5.com that I have pointed at a typical VirtualServer on BIG-IP. It listens on HTTPS (tcp/443) and decrypts traffic with a clientSSL profile, but passes traffic to the Coder server on HTTP (tcp/3000) with no serverSSL profile.

In the next post, I’ll offload authentication from Coder to BIG-IP APM.

Introduction to LiteLLM

This post is based on a talk I gave at the Boston Kubernetes Meetup on Tue Apr 7, 2026. The goal was to introduce LiteLLM to the group and show a progressive demo — from a local Python install all the way to Kubernetes. This post is my write-up of that content.

Overview

If you’ve been building anything with LLMs recently, you’ve probably noticed that every provider has a different API. OpenAI looks one way, Anthropic looks another, Gemini is different again. Switching models means rewriting your API calls. Testing two providers side-by-side means maintaining two different integrations. It gets messy fast.

LiteLLM is the answer to this problem. It gives you a single OpenAI-compatible interface for 100+ models and providers. You talk to LiteLLM, and LiteLLM figures out how to talk to whoever is behind it.

But LiteLLM is more than just a translation layer. Put it in your architecture as a proxy and you now have a central point where you can enforce policies, track costs, manage API keys, set rate limits, and observe everything that’s happening across your AI workloads.

What is LiteLLM?

From the project itself: LiteLLM lets you call all LLM APIs (Azure, Gemini, Anthropic, and more) using the OpenAI format. It translates inputs, standardizes exceptions, and guarantees consistent output format for completion() and embedding() calls. It does three things really well:

1. Consistent I/O: Removes the need for provider-specific if/else logic in your application code.
2. Reliable: Extensively tested with 50+ test cases and used in production environments.
3. Observable: Native integrations with Sentry, PostHog, Helicone, Prometheus, Langfuse, and others.

Two ways to use LiteLLM

Before jumping into the demo, it’s worth understanding the two distinct modes.

As a Python SDK

Import LiteLLM directly into your Python code. It’s a drop-in replacement for the OpenAI Python client and works similarly to how other frameworks like LangChain are used.

1
2
3
4
5
6
7
from litellm import completion

response = completion(
    model="anthropic/claude-sonnet-4-20250514",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

Change the model string and nothing else changes. That’s the whole pitch for the SDK. This is the right choice if you’re embedding LiteLLM directly in a Python application.

As a Proxy Server

LiteLLM can run as a standalone HTTP server — an OpenAI-compatible REST API that any application can call, regardless of language. This is what we’ll focus on for the rest of this post.

1
Your App  →  LiteLLM Proxy  →  OpenAI / Anthropic / Gemini / etc.

The proxy exposes standard endpoints like POST /v1/chat/completions and GET /v1/models. Any tool that already knows how to talk to OpenAI — Open WebUI, LangChain, your own app — can point at LiteLLM instead, with zero code changes.

The proxy mode is more interesting from an infrastructure standpoint because it gives you a centralized control plane for all your LLM traffic.

Demo: LiteLLM as a Proxy

I showed a progressive demo at the meetup — starting simple and adding complexity at each step. Here’s the full walkthrough.

Option 1: Local Python in a virtual environment

This is the simplest possible setup. Good for development and testing.

On Ubuntu 24.04 you’ll hit an “externally managed environment” error if you try to pip install system-wide. This is by design — Ubuntu 24.04 introduced PEP 668 to protect the system Python. Use a virtual environment:

1
2
3
4
5
6
7
8
# Create a virtual environment
python3 -m venv ~/.venv

# Activate it
source ~/.venv/bin/activate

# Install litellm with proxy dependencies
pip install 'litellm[proxy]'

Note: the [proxy] extra is required. The base litellm package doesn’t include the dependencies for the proxy server (FastAPI, uvicorn, websockets, etc.).

Start the proxy with a single model:

1
2
export GEMINI_API_KEY=your-key
litellm --model gemini/gemini-2.5-flash-preview-04-17 --port 4000

That’s it. The proxy is running. Simple enough — but not how you’d run this for anything beyond a quick test.

Option 2: Docker with a config file

For anything more serious you want Docker and a config file. The config file is how you manage multiple models, set a master key for authentication, and control routing.

Create litellm_config.yaml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

  - model_name: claude
    litellm_params:
      model: anthropic/claude-sonnet-4-20250514
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: gemini
    litellm_params:
      model: gemini/gemini-2.5-flash-preview-04-17
      api_key: os.environ/GEMINI_API_KEY

general_settings:
  master_key: "sk-xxxx" #change this. It can be anything starting with "sk-".

A few things to note here:

• model_name is the alias your applications use — you choose this, it doesn’t have to match the real underlying model name.
• os.environ/OPENAI_API_KEY tells LiteLLM to read the key from the environment at runtime, so you never hardcode a secret in the config file.
• master_key is what your applications use to authenticate to the proxy. This is separate from — and replaces — your real provider API keys in your application code.

Run it with Docker:

1
2
3
4
5
6
7
8
9
10
11
12
13
export OPENAI_API_KEY=xxxx
export ANTHROPIC_API_KEY=xxxxx
export GEMINI_API_KEY=xxxx

docker run --name litellm-proxy \
  --restart unless-stopped \
  -v $(pwd)/litellm_config.yaml:/app/config.yaml \
  -e OPENAI_API_KEY=$OPENAI_API_KEY \
  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  -e GEMINI_API_KEY=$GEMINI_API_KEY \
  -p 4000:4000 \
  docker.litellm.ai/berriai/litellm:main-stable \
  --config /app/config.yaml

Test it:

1
2
3
4
curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-xxxx" \
  -d '{"model": "gemini", "messages": [{"role": "user", "content": "Hello!"}]}'

Option 3: Adding a database

The config file handles static startup settings. If you want the admin UI, user management, spend tracking, and virtual key management, you need a Postgres database.

Start a Postgres container:

1
2
3
4
5
docker run --name litellm-postgres \
  -e POSTGRES_DB=litellm \
  -e POSTGRES_USER=litellm \
  -e POSTGRES_PASSWORD=mypassword123 \
  -p 5432:5432 -d postgres

Restart LiteLLM with the database connection and UI credentials:

1
2
3
4
5
6
7
8
9
10
11
12
docker run --name litellm-proxy \
  -v $(pwd)/litellm_config.yaml:/app/config.yaml \
  -e OPENAI_API_KEY=$OPENAI_API_KEY \
  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  -e GEMINI_API_KEY=$GEMINI_API_KEY \
  -e DATABASE_URL="postgresql://litellm:mypassword123@litellm-postgres:5432/litellm" \
  -e UI_USERNAME="admin" \
  -e UI_PASSWORD="admin" \
  --link litellm-postgres:litellm-postgres \
  -p 4000:4000 \
  docker.litellm.ai/berriai/litellm:main-stable \
  --config /app/config.yaml

Once running, the admin UI is at http://[YOUR_IP]:4000/ui. From there you can:

• Create users and teams
• Manage virtual keys with rate limits and spend caps
• View usage and cost per model, per key, per team
• Use the playground to test models directly
• Set up alerting

The general rule of thumb for where things live: structure in config.yaml, secrets in environment variables, runtime data in the database.

Option 4: Adding Open WebUI

At this point we have a solid LLM proxy. Let’s add a proper chat interface in front of it.

First, add an image generation model to your litellm_config.yaml:

1
2
3
4
5
6
  - model_name: dall-e-3
    litellm_params:
      model: openai/dall-e-3
      api_key: os.environ/OPENAI_API_KEY
    model_info:
      mode: image_generation

Create a Docker network so the containers can reach each other by name:

1
2
docker network create llm-network
docker network connect llm-network litellm-proxy

Run Open WebUI:

1
2
3
4
5
6
7
8
9
10
11
12
docker run --name open-webui \
  --network llm-network \
  -e OPENAI_API_BASE_URL=http://litellm-proxy:4000/v1 \
  -e OPENAI_API_KEY=sk-xxxx \
  -e ENABLE_IMAGE_GENERATION=true \
  -e IMAGE_GENERATION_ENGINE=openai \
  -e IMAGES_OPENAI_API_BASE_URL=http://litellm-proxy:4000/v1 \
  -e IMAGES_OPENAI_API_KEY=sk-xxxx \
  -e IMAGES_OPENAI_API_MODEL=dall-e-3 \
  -e IMAGE_SIZE=1024x1024 \
  -p 3000:8080 \
  ghcr.io/open-webui/open-webui:main

Note http://litellm-proxy:4000/v1 — Docker resolves container names as hostnames within the same network. Open WebUI has no idea it’s talking to LiteLLM rather than OpenAI directly.

Open WebUI is now at http://[YOUR_IP]:3000. You get chat history stored locally, model switching, image generation, and a much nicer interface than curl.

Option 5: Moving to Kubernetes

This is where things get more interesting. Once you’re comfortable with the Docker setup, moving to K8s is a natural next step. You get high availability, rolling updates, better secret management, and proper scaling. Here’s what the migration involves:

• Config file → Kubernetes ConfigMap, mounted into the LiteLLM pod
• API keys → Kubernetes Secrets, mapped as environment variables
• --link / --network → Kubernetes Services (K8s has built-in DNS, pods find each other by service name)
• -p port:port → Service of type LoadBalancer or NodePort
• -v volumes → PersistentVolumeClaims for Postgres and Open WebUI (you want chat history and DB data to survive pod restarts)
• --restart unless-stopped → automatic with K8s; it restarts crashed pods by default

I’ll cover the full K8s deployment with manifests in a follow-up post.

LiteLLM vs AgentGateway

This came up at the meetup so I want to address it here. Are LiteLLM and AgentGateway competing solutions?

The short answer is: not really. They solve different problems, and can be layered.

LiteLLM is a common and good choice today for most teams. It’s battle-tested, has a large community, and covers the core use cases well. AgentGateway is the one to watch if you’re building agentic systems — it’s specifically designed for the MCP and A2A protocols that are becoming the standard way agents communicate with tools and each other.

They are not mutually exclusive. You could use LiteLLM for LLM routing and cost tracking today, and layer AgentGateway in front of it as your agent infrastructure matures.

The bigger picture

Here’s the message I wanted to leave the meetup with, and it’s the same message I’ll leave here.

Your AI and agent workloads should traverse a central point of policy governance.

Right now a lot of teams are building AI features where every application has its own API keys, its own connection to whatever LLM provider it uses, and zero visibility into what’s happening across the organization. That might be fine for a prototype, but it doesn’t scale, and it creates real security and compliance problems.

A proxy like LiteLLM (or AgentGateway, or both) is that central point. Every LLM call goes through it. You get:

• Visibility: Who is calling what model, when, and at what cost.
• Control: Rate limits and budget caps per team, per key, or per model. Rogue applications can’t run up your bill.
• Security: Applications never hold real provider API keys. If a key is compromised, you rotate it in one place.
• Flexibility: Swap providers or models behind the scenes without touching application code. When OpenAI releases a new model, update the proxy config — your apps don’t change.
• Reliability: Fallback to another model or provider if one goes down.

This is exactly the pattern we already apply to other critical services. We don’t let every microservice talk directly to the database — we have connection pooling, access control, and monitoring. We don’t let every service handle its own TLS — we terminate it at an ingress controller or load balancer. LLM traffic should be no different.

A word on security

One thing that came up at the meetup and is worth calling out explicitly: in March 2026 there was a supply chain attack on the LiteLLM PyPI package. Versions 1.82.7 and 1.82.8 were compromised via a maintainer’s hijacked account. The malicious code stole credentials — environment variables, SSH keys, cloud provider tokens, Kubernetes tokens — and exfiltrated them to an attacker-controlled server.

The good news: the official Docker images (GHCR and Docker Hub) were not affected, as they were running 1.82.6. PyPI quarantined both versions within 46 minutes, but not before they were downloaded nearly 47,000 times.

The lessons are not new, but they’re worth repeating:

• Pin your dependencies and use lock files with checksums.
• Audit packages before upgrading, especially for production workloads.
• Be able to rotate credentials fast. If you had a central proxy, rotating your provider API keys after an incident like this is a single config change. If every application holds its own keys, it’s a much bigger problem.

This incident is actually a good argument for the centralized proxy pattern. The fewer places your real API keys live, the smaller your blast radius when something goes wrong.

Next post I’ll cover the full Kubernetes deployment with manifests. Thanks for reading!

Updated: April 8, 2026

This is a follow-up to my original post from November 2023 and another from Sept 2024. A lot has changed since then — not in how OpenShift installs, but in my AWS environment and in some installer requirements that crept in quietly.

This post covers the problems I hit deploying OCP 4.21 after not doing this since 4.16, and the changes I made to end up with a cheaper, leaner dev/test cluster.

What broke since last time

It’s been 6-12 months since I remember installing Openshift on AWS. Two things had changed that caused immediate failures.

1. Subnet tagging is now required

The installer now validates subnet tags before proceeding. If your subnets are untagged, you’ll get a fatal error before it even starts. Previously the installer would just tag your subnets itself — now it checks first.

Subnets you’re handing to the installer, ie those in your install-config.yaml, need to be tagged:

1
2
3
aws ec2 create-tags \
  --resources subnet-xxxxxxxxxxxxxxxxx subnet-xxxxxxxxxxxxxxxxx \
  --tags Key=kubernetes.io/cluster/ocpcluster,Value=shared

Use your cluster name from metadata.name in place of ocpcluster.

Any other subnets in the same VPC that the installer should ignore need this tag:

1
2
3
aws ec2 create-tags \
  --resources subnet-xxxxxxxxxxxxxxxxx \
  --tags Key=kubernetes.io/cluster/unmanaged,Value=true

If you skip this step, the installer will fail with an error about untagged subnets in the VPC.

2. Corporate IT had tightened my IAM permissions

My AWS credentials hadn’t changed, but the policy attached to my IAM user had. The new policy uses a NotAction block that explicitly excludes iam:*User* and iam:*AccessKey* operations (except on my own user). The installer’s default mint mode for the Cloud Credential Operator (CCO) needs to create and delete IAM users for each cluster component — so it now fails hard with:

1
2
3
WARNING Action not allowed with tested creds   action=iam:DeleteAccessKey
WARNING Action not allowed with tested creds   action=iam:DeleteUser
FATAL failed to fetch Cluster: failed to fetch dependency of "Cluster": failed to generate asset "Platform Permissions Check": validate AWS credentials: current credentials insufficient for performing cluster installation

The fix is to add credentialsMode: Passthrough to your install-config.yaml. In passthrough mode, the CCO copies your existing credential to each cluster component instead of minting new scoped IAM users. It never needs iam:CreateUser or iam:DeleteUser.

1
credentialsMode: Passthrough

This is a supported and documented mode, and for a dev/test cluster it’s a perfectly reasonable trade-off. The main thing to know is that your AWS credential becomes a long-lived dependency of the cluster — don’t rotate or delete it without updating the cluster secret first.

Cost optimizations

The original setup deployed 3 control plane nodes and 3 workers. That’s 6 EC2 instances running at all times. For a dev/test cluster this is overkill. Here’s what I changed.

Drop to 1 master and 1 worker

For dev/test, a single master and single worker is fine. A 3-node control plane gives you etcd quorum tolerance — if you don’t care about that (and for a throwaway lab cluster, I don’t), 1 master works.

Use Spot instances for workers

Workers are safely replaceable. The MachineSet will automatically provision a new one if a Spot interruption occurs. In us-east-1, an m5.xlarge Spot instance typically runs 60-70% cheaper than on-demand.

Important: do not use Spot for the master node. If it gets interrupted, the cluster is dead. Keep the master on on-demand.

Single availability zone

Running across multiple AZs incurs cross-AZ data transfer costs. For a dev/test cluster, just pick one AZ. This also means you only need 2 subnets (one public, one private) instead of 4.

Here’s my updated install-config.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
additionalTrustBundlePolicy: Proxyonly
apiVersion: v1
baseDomain: my-f5.com
credentialsMode: Passthrough
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  replicas: 1
  platform:
    aws:
      type: m5.xlarge
      spotMarketOptions: {}   # Spot pricing - ~60-70% cheaper than on-demand
      rootVolume:
        size: 100
        type: gp3
      zones:
      - us-east-1a
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  replicas: 1
  platform:
    aws:
      type: m5.xlarge         # On-demand - spot interruption = dead cluster
      rootVolume:
        size: 100
        type: gp3
      zones:
      - us-east-1a
metadata:
  creationTimestamp: null
  name: ocpcluster
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:
  - cidr: 10.0.0.0/16
  networkType: OVNKubernetes
  serviceNetwork:
  - 172.30.0.0/16
platform:
  aws:
    region: us-east-1
    subnets:
    - subnet-xxxxxxxxxxxxxxxxx   # Private subnet - us-east-1a
    - subnet-xxxxxxxxxxxxxxxxx   # Public subnet  - us-east-1a
publish: External
pullSecret: 'my-pull-secret'
sshKey: |
  ssh-rsa xxx...yyy imported-openssh-key

On-demand GPU nodes — deploy when you need them, delete when you don’t

I occasionally need to test an app that requires a GPU. Running a GPU instance full-time is expensive — a g4dn.xlarge (1x NVIDIA T4) runs about $0.53/hr on-demand, or around $0.17/hr on Spot. The right pattern here is to create a MachineSet for the GPU node after the cluster is up, scale it to 1 when you need it, and scale it back to 0 when you don’t.

When scaled to 0, the EC2 instance is terminated — you pay nothing for compute. The only ongoing cost is the EBS root volume that remains (~$10/month for 100GB gp3). If that bothers you, delete the MachineSet entirely and recreate it next time.

Creating the GPU MachineSet

Use an existing MachineSet as your template:

1
2
MACHINESET_NAME=$(oc get machineset -n openshift-machine-api -o name | head -1)
oc get machineset -n openshift-machine-api -o yaml $MACHINESET_NAME > gpu-machineset.yaml

Edit gpu-machineset.yaml and make these changes:

• Change the name (in both metadata.name and the selector labels) to something like ocpcluster-gpu-us-east-1a
• Set replicas: 0
• Set instanceType: g4dn.xlarge
• Add a GPU label to the node so workloads can target it:

1
2
3
4
5
6
spec:
  template:
    spec:
      metadata:
        labels:
          node-role.kubernetes.io/gpu-worker: ""

Apply it:

1
oc apply -f gpu-machineset.yaml

Install the NVIDIA GPU Operator

Before scaling up, install the NVIDIA GPU Operator from OperatorHub. It handles driver installation automatically via a DaemonSet that targets GPU nodes. Without this, your GPU node will come up but your app won’t be able to use the GPU.

Scale up when you need the GPU, scale down when done

1
2
3
4
5
6
7
8
# Spin up the GPU node
oc scale machineset ocpcluster-gpu-us-east-1a -n openshift-machine-api --replicas=1

# Wait for node to be Ready
oc get nodes -w

# When finished, terminate the instance (MachineSet definition is preserved)
oc scale machineset ocpcluster-gpu-us-east-1a -n openshift-machine-api --replicas=0

Scaling to 0 is the better habit over deleting the MachineSet — you keep the definition and can scale back up any time without rebuilding the YAML.

Summary of changes since previous posts

The result is a cluster that costs a fraction of the original setup to run, handles GPU testing without ongoing GPU spend, and works within the tighter IAM restrictions my org has put in place.

Overview

F5 Container Ingress Services (CIS) is a powerful way to manage BIG-IP configuration directly from Kubernetes. Using CIS Custom Resource Definitions (CRDs) like VirtualServer and TransportServer, you can define rich traffic management policies in native Kubernetes manifests and have CIS automatically create and update Virtual IPs (VIPs) on BIG-IP.

One common question that comes up: “What if I want DNS records created automatically when a VirtualServer comes up, but I’m not using F5 DNS?”

This article answers exactly that question. We’ll walk through how to combine CIS VirtualServer resources with the community project ExternalDNS to automatically register DNS records on external DNS providers like AWS Route 53, Infoblox, CoreDNS, Azure DNS, and others — all without touching a zone file by hand.

Background: How DNS Automation Typically Works in Kubernetes

Before diving into the solution, it’s worth grounding ourselves in how DNS automation normally works in Kubernetes.

The Standard Pattern: Services of Type LoadBalancer

The most common pattern is:

1. You create a Service of type LoadBalancer.
2. A cloud controller (or a bare-metal equivalent like MetalLB) assigns an external IP and updates the status.loadBalancer.ingress field of the Service object.
3. ExternalDNS watches for Services of type LoadBalancer with specific annotations, reads the IP from the status field, and creates a DNS A record on your external DNS server.

This is clean, well-understood, and widely supported. ExternalDNS can also watch Ingress objects or Services of type ClusterIP and NodePort, but the LoadBalancer pattern is by far the most common integration point.

Where F5 CIS Fits In

CIS supports creating VIPs on BIG-IP in multiple ways:

1. VirtualServer / TransportServer CRDs — Most customers prefer to use VS or TS CRDs because they expose richer BIG-IP capabilities: iRules, custom persistence profiles, health monitors, TLS termination policies, and more. This is where the DNS automation story gets more nuanced and is the focus of this article.

2. Service of type LoadBalancer — CIS watches for Services of type LoadBalancer. Typically an IPAM controller or a custom annotation will be used to configure an IP address. CIS allocates a VIP on BIG-IP, and updates the Service’s status.loadBalancer.ingress field with that IP. This is not the focus of this article.

3. Other — CIS can also use Ingress or ConfigMap resources, but these are out of scope for this article.

The Gap: F5 CRDs and Non-F5 DNS

CIS does include its own ExternalDNS CRD (not to be confused with the community project of the same name). However, F5’s built-in ExternalDNS CRD only supports F5 DNS (BIG-IP DNS / GTM). If you’re using Route 53, Infoblox, PowerDNS, or any other DNS provider, you need a different approach.

That’s where the community ExternalDNS project comes in.

The Solution: VirtualServer + Service of Type LoadBalancer + ExternalDNS

The trick is straightforward once you see it:

CIS can manage a VIP on BIG-IP via a VirtualServer CRD while simultaneously updating the status field of a Service of type LoadBalancer. ExternalDNS then reads that status field and creates DNS records.

Here’s the flow:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
VirtualServer CRD
      │
      │  references pool members via
      ▼
Service (type: LoadBalancer)
      │
      │  CIS updates status.loadBalancer.ingress
      │  with the VIP IP address
      ▼
ExternalDNS watches Service status
      │
      │  creates/updates DNS record
      ▼
External DNS Server (Route 53, Infoblox, etc.)

Let’s walk through the manifests.

Step-by-Step Walkthrough

Step 1: Deploy Your Application

A standard Deployment — nothing special here.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: my-namespace
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-app
          image: my-app:latest
          ports:
            - containerPort: 8080

Step 2: Create the Service of Type LoadBalancer

This Service is the linchpin of the whole solution. It serves three purposes:

1. It acts as a target for the CIS VirtualServer pool (either via NodePort or directly to pod IPs in cluster mode).
2. CIS updates its status.loadBalancer.ingress field with the BIG-IP VIP address.
3. ExternalDNS reads its status and annotations to create a DNS record.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
apiVersion: v1
kind: Service
metadata:
  name: my-app-svc
  namespace: my-namespace
  annotations:
    # ExternalDNS annotation — tells ExternalDNS what hostname to register
    external-dns.alpha.kubernetes.io/hostname: myapp.example.com
    # Optional: set a custom TTL
    external-dns.alpha.kubernetes.io/ttl: "60"
spec:
  selector:
    app: my-app
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
  type: LoadBalancer
  # Prevent other LB controllers from acting on this Service
  loadBalancerClass: f5.com/bigip
  # Do not allocate NodePort endpoints — more on this below
  allocateLoadBalancerNodePorts: false

Two fields here deserve extra explanation.

loadBalancerClass: f5.com/bigip

In a typical cluster, multiple controllers may be watching for Services of type LoadBalancer — MetalLB, the cloud provider controller, etc. If you’re using CIS VirtualServer CRDs to manage the VIP (rather than having CIS act directly as a LoadBalancer controller for this Service), you likely don’t want any of those other controllers touching this Service.

Setting loadBalancerClass to a value that no other running controller claims means this Service will be ignored by all LB controllers except the one that explicitly handles that class. In this pattern, no controller is assigning the VIP from the Service side — CIS does it via the VirtualServer CRD and writes back the IP into the status field programmatically.

Note: The exact value of loadBalancerClass depends on your environment. The key goal is to prevent unintended controllers from assigning IPs or creating cloud load balancers for this Service.

allocateLoadBalancerNodePorts: false

By default, LoadBalancer Services in Kubernetes allocate NodePort endpoints. This means traffic could reach your pods directly via <NodeIP>:<NodePort> — bypassing BIG-IP entirely, bypassing your security policies, and bypassing your iRules.

Setting allocateLoadBalancerNodePorts: false prevents this. The Service effectively behaves like a ClusterIP service in terms of access — the only way to reach it from outside the cluster is via the BIG-IP VIP. This is the right posture when:

• Your CIS deployment uses --pool-member-type=cluster, sending traffic directly to pod IPs via the BIG-IP’s overlay network (VXLAN or GENEVE).
• You want BIG-IP to be the sole external entry point for policy enforcement.

Step 3: Create the VirtualServer CRD

Now we define the VirtualServer. Note how it references the Service by name in the pool configuration:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
apiVersion: cis.f5.com/v1
kind: VirtualServer
metadata:
  name: my-app-vs
  namespace: my-namespace
  labels:
    f5cr: "true"
spec:
  host: myapp.example.com
  ipamLabel: prod          # Optional: use F5 IPAM Controller for IP allocation
  # virtualServerAddress: "10.1.10.50"  # Or specify IP directly
  pools:
    - path: /
      service: my-app-svc
      servicePort: 80
      monitor:
        type: http
        send: "GET / HTTP/1.1\r\nHost: myapp.example.com\r\n\r\n"
        recv: ""
        interval: 10
        timeout: 10

When CIS processes this VirtualServer, it:

1. Creates a VIP on BIG-IP (using either the IP you specified in virtualServerAddress or one allocated by the F5 IPAM Controller if ipamLabel is used).
2. Configures the BIG-IP pool with the backends from my-app-svc.
3. Writes the VIP IP address back into my-app-svc’s status.loadBalancer.ingress field.

That last step is what makes the whole chain work.

IP Address: Specify Directly or Use F5 IPAM Controller

You have two options for IP allocation:

Option A — Specify the IP directly in the VirtualServer manifest:

1
2
spec:
  virtualServerAddress: "10.1.10.50"

This is simple and predictable. Good for static, well-planned deployments.

Option B — Use the F5 IPAM Controller:

1
2
spec:
  ipamLabel: prod

The F5 IPAM Controller watches for CIS resources with ipamLabel annotations and allocates IPs from a configured range. CIS then picks up the allocated IP automatically. This is ideal when you want full automation without managing IP addresses in YAML files.

Step 4: Verify CIS Updates the Service Status

After CIS processes the VirtualServer, check the Service:

1
kubectl get svc my-app-svc -n my-namespace -o jsonpath='{.status.loadBalancer.ingress}'

You should see output like:

1
[{"ip":"10.1.10.50"}]

This is the IP that ExternalDNS will use to create the DNS record.

Step 5: ExternalDNS Does Its Job

With ExternalDNS deployed and configured for your DNS provider (Route 53, Infoblox, etc.), it will:

1. Discover my-app-svc because it’s of type LoadBalancer with an external-dns.alpha.kubernetes.io/hostname annotation.
2. Read 10.1.10.50 from status.loadBalancer.ingress.
3. Create an A record: myapp.example.com → 10.1.10.50.

ExternalDNS handles the rest automatically, including updates if the IP changes.

A minimal ExternalDNS deployment for Route 53 would look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
  namespace: external-dns
spec:
  replicas: 1
  selector:
    matchLabels:
      app: external-dns
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      serviceAccountName: external-dns
      containers:
        - name: external-dns
          image: registry.k8s.io/external-dns/external-dns:v0.14.0
          args:
            - --source=service
            - --domain-filter=example.com
            - --provider=aws
            - --aws-zone-type=public
            - --registry=txt
            - --txt-owner-id=my-cluster

Refer to the ExternalDNS documentation for provider-specific configuration (IAM roles for Route 53, credentials for Infoblox, etc.).

Putting It All Together: Summary of the Architecture

Key Considerations and Design Choices

When to Use This Pattern vs. CIS as a LoadBalancer Controller

CIS can act directly as a LoadBalancer controller — watching Services of type LoadBalancer and creating VIPs on BIG-IP without any VirtualServer CRD involvement. If that’s sufficient for your needs, it’s simpler. ExternalDNS works with that mode natively, since CIS updates status.loadBalancer.ingress in both cases.

Use the VirtualServer CRD approach when you need:

• Custom iRules or iApps on the VIP
• Advanced persistence profiles
• Fine-grained TLS termination control
• Traffic splitting or A/B routing policies
• Any BIG-IP capability that doesn’t map directly to Kubernetes Service semantics

allocateLoadBalancerNodePorts: false — When It Applies

This setting is appropriate when your CIS deployment uses --pool-member-type=cluster. In cluster mode, BIG-IP sends traffic directly to pod IPs, not through NodePort endpoints. Disabling NodePort allocation:

• Prevents back-door access to your application via <NodeIP>:<NodePort>.
• Reduces iptables rule sprawl on your nodes.
• Aligns with a clean security boundary where BIG-IP is the sole ingress.

If your CIS deployment uses --pool-member-type=nodeport, you should not set allocateLoadBalancerNodePorts: false, as CIS will need those NodePorts to forward traffic.

F5 IPAM Controller Integration

The F5 IPAM Controller pairs particularly well with this pattern. Rather than managing VIP IP addresses in your VirtualServer manifests, IPAM handles allocation from a configured pool. This means:

• Platform teams manage IP ranges in the IPAM controller config.
• Application teams simply specify an ipamLabel in their VirtualServer manifest.
• CIS picks up the IPAM-assigned IP and writes it to the Service status automatically.

The ExternalDNS chain remains identical regardless of whether the IP comes from IPAM or is statically assigned.

Frequently Asked Questions

Q: Can I use this pattern with TransportServer CRDs instead of VirtualServer?

Yes. CIS similarly updates the status of a referenced Service when using TransportServer. The same approach applies.

Q: What if I want ExternalDNS to also create a CNAME instead of an A record?

Use the external-dns.alpha.kubernetes.io/target annotation on the Service to override the IP with a hostname, causing ExternalDNS to create a CNAME. Refer to ExternalDNS documentation for specifics.

Q: Can I use multiple hostnames for the same VirtualServer?

Add multiple external-dns.alpha.kubernetes.io/hostname annotations (comma-separated values are supported by ExternalDNS) or create additional Services pointing to the same pods.

Conclusion

Combining F5 CIS VirtualServer CRDs with the community ExternalDNS project gives you the best of both worlds: rich BIG-IP traffic management via CIS, and flexible, provider-agnostic DNS automation via ExternalDNS.

The core insight is simple — CIS writes the BIG-IP VIP IP address back into the Kubernetes Service status field, and ExternalDNS reads from that same field. By using loadBalancerClass and allocateLoadBalancerNodePorts: false, you ensure the Service is a clean “status carrier” that doesn’t accidentally expose your application through unintended paths.

Whether you assign VIP IPs statically in your manifests or use the F5 IPAM Controller for full automation, this pattern integrates naturally into any Kubernetes-native GitOps workflow.

Additional Resources

• F5 CIS Documentation
• F5 CIS VirtualServer CRD Reference
• F5 IPAM Controller on GitHub
• ExternalDNS on GitHub
• ExternalDNS: Service Source Documentation
• Kubernetes: LoadBalancer Service specification

Remote CR discovery for CIS multi-cluster

Background

I’ve watched one of my favorite projects at F5, CIS (Container Ingress Services), mature over the years. When I joined F5 in late 2018, it watched the K8s API for the creation of Ingress resources and configured a VirtualServer on the BIG-IP with settings determined by the annotations on the Ingress resource. Then I watched over the years:

• it could send AS3 declarations to BIG-IP based on ConfigMap resources
• it supported Custom Resources (CRD’s) like VirtualServer and TransportServer
• the CRD schema grew and matured to support almost all available BIG-IP settings
• it integrated with IPAM and DNS
• it started supporting pool members using pods from multiple clusters

Remote CR discovery

What’s the latest improvement? Until now, even when operating in multi-cluster mode, CIS only discovered Custom Resources that were installed on the same cluster on which CIS was running. While pods from remote clusters could become pool members in BIG-IP, the VirtualServer and TransportServer resources had to be on the same cluster as CIS.

Why is this a problem?

Imagine you have 4 clusters and they all run different parts of the same overall application. You may have multiple clusters for the sake of:

• redundancy (in case of cluster failure)
• migration (you’re moving workloads between namespaces and clusters)
• cost reasons (some workloads on more costly clusters)
• etc

You may choose to install CIS on Cluster 1 and have CIS “watch” all 4 clusters for services to expose via BIG-IP. But before now, you had to install CR’s - VirtualServer and TransportServer - on Cluster 1 only. That means you have to control where your app teams deploy their resources.

Now, CIS will watch all 4 clusters for services AND Custom Resources. That’s the feature introduced here.

Before remote CR discovery

With remote CR discovery

How to configure remote CR discovery

As you can see from the diagrams, the difference is quite simple, but the operational impacts are large. Enabling this is equally simple: a single line in the extended spec config file (see line 15 below)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
kind: ConfigMap
apiVersion: v1
metadata:
  name: extended-spec-config
  namespace: kube-system
  labels:
    f5type: virtual-server
    as3: "true"
data:
  extendedSpec: |
    mode: default
    externalClustersConfig:
    - clusterName: cluster2
      secret: kube-system/kubeconfig2
      customResourceDiscovery: true #enables this new feature

For the sake of full repeatability, I am going to share my manifests here.

Also, for the sake of thinking about how this would apply to CIS in HA setup, here’s a diagram that depicts our problem with an Active/Standby CIS cluster.

Thanks for reading!

How I’ve updated my KillerCoda labs to use more features

Most of this can be found in the short doc for creators but I’ll share what I’ve done here:

Environments

Right now I’m still using the kubernetes-kubeadm-1node image but I do plan to have a good reason to use ubuntu or kubernetes-kubeadm-2nodes

Custom Code Markdown Actions

• The {{copy}} and {{exec}} shortcuts that turn code blocks into easily copied or run text is handy. I’m using that now.
• The {{TRAFFIC_HOST1_80}} is a nice way to display a URL easily, I’m using that too.

Scripts

• There is now a validation step in some of my scenarios. I have made scripts to check basic things like “are there 2x running pods in the nginx-ingress namespace?”
• I’m using a background script to do things like create YAML files when the scenario loads
• I’m using a foreground script to show a small “loading” alert in the command window while the background script runs

Formatting

I’m using html <details> and <summary> tags to have hints. Here’s an example:

This is done with formatting like this:

1
2
3
4
5
<details>
  <summary>Hint 1</summary>
  
  **Hint:** You will need to edit the `replicas` and `image` values in `nginx-ingress.yaml`, and add an argument for the container image.
</details>

The result

Check them out for yourself: