The short version: I swapped Antigravity for Claude Code via VS Code, switched LLM from Gemini to Claude, cleaned up the instruction architecture, and added a few plugins to Obsidian.

Why I Swapped Tools

Part 1 used Antigravity from Google. This is a decent agent with a good UI, but two annoying things got me to migrate to Claude Code:

• it was constantly prompting me for permission to do anything and I couldn’t just blanket allow operations on local files
• recently I’ve been getting “model unavailable” errors with retries in a fall-back fashion. Super annoying and basically unuseable.

Claude Code — Anthropic’s agentic CLI — is built for exactly this. It runs an autonomous loop, issues its own file reads and writes, and doesn’t drift mid-task. The VS Code extension puts a Claude Code terminal in the same window as your vault. For a system that manages files as its primary output, this is the right tool. Also, many colleagues use it so I have people to go to.

Model switch: Gemini → Claude Sonnet

Antigravity used Gemini by default. Claude Code lets you pin a specific model in settings.json:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
	"claudeCode.environmentVariables": [
    {
      "name": "ANTHROPIC_AUTH_TOKEN",
      "value": "xxxxx"
    },
    {
      "name": "ANTHROPIC_BASE_URL",
      "value": "https://<intranet_url>/anthropic"
    },
    {
            "name": "ANTHROPIC_MODEL",
            "value": "claude-sonnet-4-6"
    }
]

I’m using Claude Sonnet 4.6. For a second brain, the job is following complex multi-step instructions reliably — not reasoning over hard math problems or generating creative writing. Sonnet hits the right balance of speed, cost, and instruction-following for this use case. Opus is more capable but noticeably slower for what is essentially document management.

Updating the scaffolding

Part 1’s setup wasn’t terrible but there was some overlap:

• CLAUDE.md — Claude Code’s instruction file
• AGENTS.md — the universal agent rulebook for any AI tool
• .agent/rules/leadership-rules.md — a condensed copy for tools that load rules from that path

The fix: CLAUDE.md is now 25 lines. It delegates everything to AGENTS.md and only adds Claude Code-specific configuration — which skills directory to use, where memory lives, file naming conventions. AGENTS.md is the single source of truth. .agent/rules/leadership-rules.md now just says “read AGENTS.md.”

AGENTS.md also gained five new sections: a daily notes standard, inbox routing rules, staleness thresholds (14-day follow-up flag, 21-day 1:1 flag, 60-day radar archive recommendation), a LOG.md spec, and definitions for the new slash commands.

Five New Slash Commands

/morning is the one I expect to use daily. It’s read-only, takes about ten seconds, and gives me a clear starting point without having to open the master index manually.

/lint is the one I’m most curious about long-term. The staleness problem — notes that quietly become outdated — killed every previous note system I tried. Having an automated health check changes the maintenance model from “trust yourself to remember” to “trust the agent to flag it.”

People Profile Frontmatter

Every person ledger in people/ now has YAML frontmatter:

1
2
3
4
5
6
7
---
name: [Name]
role: Senior Solutions Architect
status: active
last_updated: "2026-09-30"
last_1on1: "—"
---

This is the piece that enables everything else. Dataview queries read from frontmatter. /lint compares last_1on1 against today’s date. The /morning brief surfaces anyone with an overdue cadence. Without frontmatter, all of this requires the agent to parse prose — fragile and slow. With it, the data is structured and queryable.

The agent is also required to update these fields on every write. The dates stay accurate automatically.

Obsidian extensions: Dataview + Omnisearch

Two community plugins:

Dataview turns Obsidian into a live database over your vault. The master index used to be a manually-maintained Markdown table. It now renders from a live query:

TABLE WITHOUT ID
  link(file.path, name) AS "Name",
  role AS "Role",
  last_1on1 AS "Last 1:1",
  last_updated AS "Last Updated"
FROM "people"
WHERE status = "active" AND role != "Manager (skip-level)"
SORT last_1on1 ASC

The table sorts ascending by last 1:1 date, so the people I most need to check in with are always at the top. When the agent updates frontmatter after a debrief, the table updates automatically. No manual editing.

I also created dashboard.md — a new file with six live views: 1:1s never formally logged, the full team sorted by last 1:1, active initiatives, radar items, recent daily notes, and inbox status. It’s the first file I open each day.

Omnisearch replaces Obsidian’s built-in search with a fuzzy full-text engine that actually works. Searching for a customer or topic across dozens of daily notes returns results instantly. Install it, forget Obsidian has a native search.

LOG.md: The Audit Trail

Every file mutation the agent makes now gets one appended line in LOG.md:

1
- 2026-09-30 `/1on1-debrief Dave`: updated `people/dave-remington.md` (1:1 history, follow-ups, frontmatter), `INDEX.md`

This matters for two reasons. First, debugging: when the agent does something unexpected, there’s a record of what it changed and when. Second, trust: I was more comfortable giving the agent write access to my vault once I knew I could audit its actions without reading every file.

LOG.md is append-only by instruction. Neither I nor the agent edits prior entries.

What’s Next - maybe

• Templater + Periodic Notes: daily notes are still created manually. Templater auto-creates daily/YYYY-MM-DD.md on schedule with the right structure. This week’s task.
• Claude Desktop + MCP: the Obsidian MCP plugin exposes your vault to Claude Desktop, so you can query it from the Claude.ai interface without opening VS Code. Useful for mobile and quick lookups.
• Terminal plugin: runs a shell inside Obsidian’s sidebar. Means I can type /morning without switching apps.
• Canvas team map: a visual node graph of the team, color-coded by specialty and development path. Good for skip-level prep and org conversations with leadership.

The core principle from Part 1 holds: the agent is the programmer, the vault is the codebase. What changed is the quality of the tooling. Better agentic loop, cleaner instructions, structured data. The system is more reliable because it makes fewer demands on me to keep it consistent.

Summary: Switched from Antigravity to Claude Code for its agentic file loop. Pinned Claude Sonnet 4.6 in settings.json. Reduced CLAUDE.md to a thin delegation layer, made AGENTS.md the single source of truth, and added five new slash commands. Added YAML frontmatter to every people profile. Installed Dataview and Omnisearch in Obsidian, replaced the static master index table with a live query, and created dashboard.md as the default home view. Added LOG.md as an append-only audit trail for all agent mutations.

Here is how I (mostly) replaced OneNote with a local, agent-operated markdown Second Brain that stays private, backs up to OneDrive, and avoids lock-in to any single AI vendor.

What’s with OneNote?

Manual use of OneNote works, but feels outdated in the age of AI. As a people leader, you share info with your team, but you also track dozens of micro-initiatives, proposals, and sometimes things that are private and should not be shared.

OneNote is breaking under that weight:

• Filing friction: Forcing you to decide where a fleeting thought belongs upfront means half your notes get lost on generic pages.
• Sluggish search: Grepping across years of tabs and sections to recall a specific technical escalation is painful.
• Stenographer syndrome: Typing during 1:1s in 2026 kills active listening and eye contact.

However, OneNote is at least a manual, local tool. I don’t have to worry (much) about corporate spyware, and I know that only I have access to Notebooks that are not shared. So there is certainly a place for it.

The Architecture: Decoupling Storage, Viewer, and Intelligence

I see proprietary AI note tools (Notion, Mem, Roam) as a trap. For one, I don’t know if I can control where my data resides or which model my data may transit. If corporate IT blocks them or your company mandates a new platform, your data is trapped.

A resilient second brain decouples three layers:

1. Storage (OneDrive): Plain text Markdown (.md) files stored in a synced OneDrive folder. It stays local to your machine, backs up automatically, and complies with enterprise DLP rules.
2. Viewer (Obsidian): Opens the OneDrive folder directly as an interactive wiki with clickable links ([[people/alice]]) and instant fuzzy search. There is no local web server to start up (localhost:8080), no build step, and zero database lock-in.
3. Intelligence (The AI Agent): Antigravity, Claude Code, or Cursor. The agent acts as a filing clerk—ingesting brain dumps, cross-linking files, updating status tables, and pulling pre-meeting context.

How Agents Actually Work With Your Corpus (Token Economics)

A common misconception is that an AI agent sends your entire folder to the model with every prompt. If you have 50 files totaling 80,000 words, sending that entire corpus on every prompt would be slow, expensive, and drown the LLM in irrelevant context. Modern agentic assistants interact with your files using selective tool execution:

1. Workspace Discovery: The agent inspects AGENTS.md and discovers specialized skills (like /1on1-debrief).
2. Targeted Reading: When you say “Debrief with Alice”, the agent checks INDEX.md to resolve the filename, reads only people/alice.md, and ignores the rest of your team’s files.
3. Intent Disambiguation: A well-instructed agent separates general knowledge from private data. If you ask “Where is Bank of America headquartered?”, it answers from its base training without touching your files. If you ask “What did Alice say about the BoA migration?”, it reads the relevant customer and personal files.
4. State Mutation: It generates the user-facing summary in chat, appends private coaching notes to people/alice.md, and updates the master tracking table in INDEX.md.

The Anti-Lock-In Rule: Switching Models and Agents in 60 Seconds

I use Antigravity today, but corporate tooling shifts. I refuse to build a system I can’t migrate in a minute. We achieve portability through open standards:

• Universal Instructions (AGENTS.md): All operational rules—confidentiality, intent separation, and formatting templates—live in a root AGENTS.md file. If I switch to Claude Code, CLAUDE.md points to it. If I use Cursor, .cursorrules points to it.
• Model Agnostic: Standard Markdown and YAML frontmatter mean the prompt setup works with any model—Gemini, Claude, or open-source.

Key File Best Practices

A clean directory structure keeps the agent fast and token usage low:

1
2
3
4
5
6
7
second-brain/
├── HOWTO.md          # Human reference manual read in Obsidian
├── AGENTS.md         # Universal operating rules read by AI agents
├── INDEX.md          # Master register of people, initiatives, and follow-ups
├── people/           # Confidential direct report ledgers
├── initiatives/      # Project cloud (active, radar, archive)
└── daily/            # Chronological daily timeline logs

• HOWTO.md: The human cheat sheet visible in Obsidian. Lists command shortcuts, voice dictation tips, and natural language query recipes.
• INDEX.md: The agent’s compass. Maintaining a single table of active initiatives, team members, and open commitments lets the agent locate files without traversing the whole folder tree.
• people/[name].md: Separates public from confidential. Shared 1:1 bullet points live in one section; private coaching notes or career goals live under a Manager Eyes Only section that the agent is strictly forbidden from sharing. (To be clear, I don’t put anything sensitive in this second brain. I will manually take notes or sometimes keep conversations verbal. I do not trust AI models with personal details, even corporate-approved models. I’m certainly willing to do this work manually.)
• initiatives/: Broken into active/ (in-flight projects) and radar/ (low-touch signals or rumors). If a radar item heats up, the agent promotes it; if it goes stale for 60 days, it moves to archive/.

The 1:1 Workflow in Practice: Voice + Custom Skills

My team is used to our standard 5-header format in their shared OneNote pages. I didn’t want to disrupt their habits.

1. Zero Typing During the Call During the 1:1, I stay fully present. No typing, no distraction.

2. The 60-Second Voice Debrief (Win + H) Immediately after the meeting, I open Antigravity, click into chat, hit Win + H for Windows voice dictation, and speak naturally:

/1on1-debrief Alice. Customer activity: Spent two days assisting Acme Corp on an architecture review. SA initiatives: Started building an automated lab script for team enablement. Other F5 things: Attended all-hands. Personal: Taking Friday off. Follow-ups: I will send her the lab credentials by Tuesday, and Alice will share her draft doc by Thursday. Private reflection: She seemed confident and energized about the new project.

3. Deterministic Output & Vault Sync The /1on1-debrief skill executes in seconds and outputs the exact copy-pasteable block:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#Customer Activity
- Acme Corp: Assisted with architecture review.

#SA initiatives / enablement
- Automated Lab Script: Developing deployment script for team enablement.

#Other F5 things
- Attended quarterly all-hands.

#Personal
- PTO: Taking Friday off.

#Follow-ups
- [ ] Mike: Send lab credentials by Tuesday.
- [ ] Alice: Share draft documentation by Thursday.

I copy that block into Alice’s shared OneNote page in two seconds.

Meanwhile, Antigravity silently logs the private reflection (“Seemed confident and energized…”) into people/alice.md, registers my follow-up commitment in INDEX.md, and updates her last meeting date.

Next week, asking /prep-1on1 Alice automatically surfaces my open commitments and previous discussion points before we jump on the call.

To Do

Now that I’ve written this post there’s a few things I want to do:

1. Switch from Antigravity to Claude Code. I am not loving Antigravity.
2. Find a way to make my Wiki a bit prettier. Now that I’ve read similar blog posts from people that know more than me, I have room for improvement here.
3. I’ll switch from Gemini (current LLM for corporate reasons) to a different model, also for corporate reasons.

Summary

Tools and AI vendors will rotate. By keeping your second brain in local Markdown, syncing through OneDrive, viewing it in Obsidian, and instructing your AI assistant via AGENTS.md, your system remains private, responsive, and completely under your control.

Problem #1: Hot water would not turn off

I’m not sure what my kid did, but he yelled for help as he constantly turned the hot water faucet clockwise but the water would not stop flowing. I tried to quickly trace a water line from our water heater to somewhere near the bathroom but couldn’t find a shut off, so I shut off the main water to the whole house. Then I unscrewed the faucet and stem by turning it counter-clockwise. The whole stem came out and I saw there was no washer left on the end of the stem.

In retrospect, the fact that I didn’t have to use a wrench at all, and simply turning the faucet counter-clockwise removed the entire stem assembly, probably means the stem body threads were loose or altogether unscrewed after being loose for a while.

Also, I found the half-disintegrated washer stuck in the pipe in the wall. I don’t know what to call that “pipe”, but Gemini suggests I called it the the faucet body, or fixture body.

At this point I thought it would be a simple replacement of that rubber washer.

Problem #2: A Sheared Brass Stub

Look closely at the photo of the stem with no washer: the head of the brass bibb screw holding the old rubber washer had completely sheared off. How am I supposed to replace that washer when there’s no screw to hold it down? There was only a tiny stub of the old screw left in there, but no way to grip it to screw it out.

Trial, Error, and Crumbling Brass

My first instinct was to grip the stub with pliers — no luck. So I sprayed WD-40 around that stub and went to bed, hoping it would loosen the headless screw overnight and the pliers would work in the morning - still, no luck.

I clamped the stem into a vise and bought mini hacksaw with a blade with 24 TPI (teeth per inch), intending to cut a clean slot across the stub face to back it out with a flathead screwdriver. I have no picture of this, but unfortunately, 100-year-old brass is deceptively soft. As soon as I applied counterclockwise pressure with the screwdriver, the notched brass simply crumbled apart, leaving me with a tiny, rounded nub and no surface left to grip.

The Precision Drilling Operation

With no material left to grab, drilling was the only option left. Because both the screw stub and the surrounding stem body were made of soft brass, any drill bit drift would instantly ruin the internal threads. The drill would have to perfectly be aligned. I kind of eye-balled it and got lucky.

I started with a tiny pilot bit, keeping the drill perfectly perpendicular to the face.

In the photo below, the pilot hole is drilled and you can see the first drill bit was thinner than the screw itself. I took this picture with my bibb screws in hand because I uploaded it to Gemini, asking AI which of the two sizes of bibb screw it thought I would be replacing this with.

From there, the key was stepping up incrementally without overshooting. I expanded the hole to 1/8”, not quite to 9/64”, which is the exact root diameter for a standard #8 bibb screw. Going slow paid off: I don’t remember if it was the 1/8” or the 9/64” bit, but one of them suddenly made a noise like something loosened. I stopped immediately and pulled out my drill.

It had hollowed out the screw core, leaving behind a paper-thin spiral shell of old brass threads that peeled right out on the bit tip. Check out the photos!

Washer: 1/2 flat

Mostly for the sake of a note so I don’t forget, I used 1/2 inch flat washers (not bevelled) because the screw wasn’t long enough to hold a bevelled washer, which is thicker. A replacement bibb screw holds this in nicely.

Another note: packing nut

The packing nut is the smaller, front-most hex nut—the one closest to the splined shaft where your handle attaches. Tightening this nut squeezes the internal packing seal against the shaft to stop leaks coming out along the stem.

Another note: slow drain

Lastly, this bathtub has never drained very fast. It’s not too slow, but I thought I could speed it up. Long story short, I was able to get some hair and scum from the drain with a thin plastic snake, but it hasnt’ improved much. I was able to confirm the slowness is in the horizontal drain pipe from underneath the bath to the vertical drain pipe right behind the bath in the wall. When I removed the barrel plunger from the cylinder (picture below), I couldnt find any hair or blockage to clear.

Finishing up

Reinstalled the stem with new washer, turned on mains, and faucet works nicely! I did the cold water faucet proactively. The old washer on the cold stem turned out to be a flat washer (looking a hair wider than my standard 1/2” pack, likely an old 1/2L size) that had turned completely stiff since it’s last replacement.

Now, shutting off both hot and cold side feels soft and smooth! Of course, showing my kids how the stem is screwed into the wall was the most satisfying part for me.

Same edge pattern, a very different gateway — and a trail of real-world friction worth documenting

In an earlier post I put open-source Kong behind an F5 BIG-IP on OpenShift and showed the two are complementary: the ADC owns the edge (TLS, WAF, pod-direct load balancing), the gateway owns API management (routing, rate-limiting, auth). The whole point is that the pattern is modular — the edge layer and API gateway layer can be switched for alternatives.

I’ll show this again. This time the in-cluster gateway is MuleSoft Omni Gateway — the product formerly (and still, in most logs and docs) called Flex Gateway. Same BIG-IP, same CIS, same OpenShift cluster, same “client → BIG-IP → gateway → backends” architecture. Only the middle layer changed.

But where the Kong write-up was a clean “here’s the architecture” piece, this one is a field report. Omni Gateway on Kubernetes had some troubleshooting - at least for me, with a trial license, learning by doing. If you’re a platform engineer about to try this, the goal here is to save you that day.

The end state:

1
2
client ──HTTPS──▶ F5 BIG-IP ──HTTP──▶ Omni Gateway ──▶ backend apps
                  (TLS + LB via CIS)   (routing + rate-limit + JWT)

I stood up https://mulesoft.my-f5.com/, edge TLS on the BIG-IP, CIS configured with OpenShift Route, Omni Gateway enforcing path routing, rate limiting, and JWT auth against two backends on /a and /b.

A note on branding. “Omni Gateway” is the current name; “Flex Gateway” is what you’ll find in almost all documentation, the Helm chart (flex-gateway), the container image (mulesoft/flex-gateway:1.13.3), and every log line (flex-gateway-agent, flex-gateway-envoy). If you’re searching for docs, search for Flex Gateway. I use “Omni Gateway” for the product and “Flex Gateway” wherever the software actually says so.

Part 1 — The setup journey

The trap: --connected=true

I created a trial login on https://anypoint.mulesoft.com/, logged into Anypoint platform, and clicked Runtimes > Runtime Manager > Omni Gateways > Self-Managed Gateways > Add Self-Managed Omni Gateway.

This gives me instructions to deploy a Gateway on Openshift. First step is to run a docker command that will generate a registration.yaml file to be used when deploying the gateway via helm. Token and Org GUID’s are populated - I’ve blurred my screenshots but you will have your own in your instructions.

Note this line from the instructions in the screenshot:

1
--connected=true

I pasted it, registered, deployed the Helm chart, and the pod never went Ready in K8s. Not crash-looping — just a readiness probe that never passed. And the maddening part: the logs showed no errors. The container was up. The process was running. It simply wasn’t serving.

I did the whole checklist:

• SCC? Checked (it was a problem too — see below — but not this problem).
• kubectl describe Fine.
• Pod logs Nothing obvious I could see.
• DNS / service resolution? Clean.

The breakthrough came from reading the full startup log rather than grepping for error. In connected mode, the gateway expects its configuration to come down from the Anypoint control plane — API Manager contracts, Runtime Manager config. It is not looking at Kubernetes-native resources. So my ApiInstance and PolicyBinding CRDs were simply ignored, no listener was ever configured for them, and readiness — which depends on a configured listener — never flipped to true. No error, because from the gateway’s point of view nothing was wrong. It was patiently waiting for a control plane that was never going to talk to it.

The fix is to register in local mode instead:

1
2
3
4
5
6
7
8
# Registration is immutable per (name, org). You cannot flip an existing
# registration from connected to local — you must register a NEW name.
docker run --entrypoint flexctl -u $UID   -v "$(pwd)":/registration mulesoft/flex-gateway \ 
  registration create --organization=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx \
  --token=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx   \
  --output-directory=/registration   \
  --connected=false \ ## this line is key when creating your registration.yaml file.
  my-gateway3  

Then deploy with the chart in local mode:

1
helm -n gateway upgrade -i --create-namespace ingress flex-gateway/flex-gateway   --set-file registration.content=registration.yaml

The learning: registration mode is the first architectural decision, and the most consequential one. connected vs local is not a fallback you can toggle later — it determines whether your gateway is configured by Anypoint contracts or by Kubernetes CRDs. Pick wrong and you get a silent readiness failure.

Bonus UX papercut — the vocabulary problem. This one setting has four names depending on where you look:

connected=false = mode=local = disconnected = offline. Same thing, four words. Worth knowing when you’re grepping logs.

The SCC storm

Once registration mode was fixed, the pod got further and then hit OpenShift’s Security Context Constraints — twice, in quick succession.

1. UID range. The gateway image wants to run as a specific built-in user. OpenShift’s default restricted-v2 SCC forces a randomized UID from the namespace’s allocated range and rejects the image’s requested UID.

2. Low-port binding. The gateway wants to bind privileged ports (< 1024). Doing that as a non-root user requires the NET_BIND_SERVICE Linux capability, and OpenShift’s default seccomp/capability posture drops it.

The fast way through — and what I did first to get unblocked — is to bind broad SCCs to the gateway’s ServiceAccount:

1
2
oc adm policy add-scc-to-user anyuid     -z ingress -n gateway
oc adm policy add-scc-to-user privileged -z ingress -n gateway

That works, but privileged is a sledgehammer and no security reviewer will sign off on it for production. The cleaner landing spot — which is what the running demo actually uses now — avoids privileged entirely by (a) running as nobody, (b) adding only the one capability that’s needed, and (c) using a sysctl to let a non-root process bind low ports:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# pod securityContext
securityContext:
  runAsNonRoot: true
  runAsUser: 65534                 # nobody
  seccompProfile:
    type: RuntimeDefault
  sysctls:
    - name: net.ipv4.ip_unprivileged_port_start
      value: "0"                   # allow unprivileged bind to low ports
---
# container securityContext
securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    add: ["NET_BIND_SERVICE"]
    drop: ["ALL"]

The even simpler move — and the one I’d recommend if you control the values file — is to sidestep the whole low-port fight by not using low ports at all (see the :8081 decision in Part 3). Then you don’t need NET_BIND_SERVICE, the sysctl, or privileged.

Why this matters: SCCs and Pod Security Standards are constraints, not misconfigurations. They’ll bite you in any hardened Kubernetes environment, not just OpenShift. Budget time for them as a setup prerequisite, and reach for a minimal custom SCC (or a high port) before you reach for privileged.

Part 2 — The architecture

Three layers, each independently testable, each ignorant of the others’ internals.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
┌───────────────────────────────────────────────────────────────────┐
│ Layer 1 — Network / edge  (F5 BIG-IP via CIS)                       │
│   • TLS termination (self-signed cert for this demo)                │
│   • Pod-direct load balancing to the gateway (pool = pod IPs:8081)  │
│   • WAF attach point (present via Policy CR; not exercised here)     │
└───────────────────────────────────────────────────────────────────┘
                                 │  HTTP :8081
                                 ▼
┌───────────────────────────────────────────────────────────────────┐
│ Layer 2 — API gateway  (MuleSoft Omni Gateway v1.13.3, local mode)  │
│   • Path routing:  /a → app-a,  /b → app-b                          │
│   • Rate limiting: 5 requests / 10 s, keyed on client IP            │
│   • JWT validation: HS256 bearer token, offline (no API Manager)    │
└───────────────────────────────────────────────────────────────────┘
                                 │  HTTP :8080
                                 ▼
┌───────────────────────────────────────────────────────────────────┐
│ Layer 3 — Backends  (namespace mulesoft-demo, 2 replicas each)      │
│   • app-a: traefik/whoami           (echoes request/pod metadata)   │
│   • app-b: mendhak/http-https-echo  (echoes headers + body)         │
└───────────────────────────────────────────────────────────────────┘

The gateway lives in namespace gateway; the backends live in mulesoft-demo. BIG-IP routes on hostname to the Mulesoft gateway.

The BIG-IP doesn’t know what’s behind the gateway. The gateway doesn’t know a BIG-IP exists.

Here’s the running environment:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ oc get pods -n gateway -o wide
NAME                      READY   STATUS    RESTARTS   AGE     IP            NODE
ingress-f88b6c598-txxlk   1/1     Running   1          4d21h   10.129.0.19   ip-10-0-2-184...

$ oc get pods -n mulesoft-demo -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP            NODE
app-a-6fb74c6c6c-9qvj9   1/1     Running   1          4d19h   10.129.0.26   ip-10-0-2-184...
app-a-6fb74c6c6c-nxhvh   1/1     Running   1          4d19h   10.129.0.24   ip-10-0-2-184...
app-b-5ff456446f-txr29   1/1     Running   1          4d19h   10.129.0.27   ip-10-0-2-184...
app-b-5ff456446f-w87kj   1/1     Running   1          4d19h   10.129.0.28   ip-10-0-2-184...

$ oc get svc -n gateway
NAME           TYPE        CLUSTER-IP       PORT(S)
ingress        ClusterIP   172.30.194.53    80/TCP,443/TCP
ingress-8081   ClusterIP   172.30.137.137   8081/TCP        # <- the demo listener
ingress-hl     ClusterIP   None             <none>

The pod is named ingress because that’s the Helm release name for the Flex/Omni gateway chart — don’t let it fool you into thinking it’s the OpenShift ingress router. The single container is named app, and the workload’s label selector is app=ingress (not app.kubernetes.io/name=flex-gateway, which some docs imply — worth checking with oc get svc <svc> -o jsonpath='{.spec.selector}' before you write a NetworkPolicy or a CIS selector against it).

Part 3 — MuleSoft Omni Gateway configuration deep dive

Everything here is done the Kubernetes-native way: ApiInstance for the listener + routing, PolicyBinding for policy. That’s exactly why local mode matters — in connected mode these CRDs are ignored.

Getting routing right: the ApiInstance

First friction point: kubectl explain wasn’t helpful. The published examples I found show one structure but I had to do slightly differently than what I found. Here’s the structure that works on v1.13.3:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
apiVersion: gateway.mulesoft.com/v1alpha1
kind: ApiInstance
metadata:
  name: demo-api
  namespace: gateway
spec:
  address: http://0.0.0.0:8081        # listen on :8081 — see the port note below
  services:
    backend-a:
      address: http://app-a.mulesoft-demo.svc.cluster.local:8080
      routes:
        - rules:
            - path: /a
    backend-b:
      address: http://app-b.mulesoft-demo.svc.cluster.local:8080
      routes:
        - rules:
            - path: /b

Why :8081? The Helm chart already stands up listeners on :80 and :443 (you’ll see ApiInstance/ingress-http and ingress-https in the namespace). Rather than fight those — and rather than deal with the low-port SCC drama from Part 1 — I gave the demo API its own listener on a high port. Cleaner, and it sidesteps NET_BIND_SERVICE entirely.

Prove routing works before you add a single policy. Port-forward to the listener and hit each path:

1
2
3
4
oc port-forward -n gateway svc/ingress-8081 18081:8081 &

curl -s http://localhost:18081/a   # -> app-a (traefik/whoami)
curl -s http://localhost:18081/b   # -> app-b (mendhak/http-https-echo)

/a returns a whoami dump (hostname, pod IPs, request headers); /b returns the echo server’s JSON. Once that’s solid, layer policy on top.

The schema mismatch: Extension definitions vs. PolicyBinding reality

Here’s the one that will cost you the most time if you’re not warned. I could not find a way to learn the expected schema or Mulesoft CRD’s. Check this out:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
ubuntu@ubuntu-Virtual-Machine:~/mulesoft/cis-resources$ kubectl get crd policybindings.gateway.mulesoft.com -o yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  creationTimestamp: "2026-07-22T20:38:50Z"
  generation: 1
  name: policybindings.gateway.mulesoft.com
  resourceVersion: "829989"
  uid: c3fffbe6-9644-477f-bf3f-3ee953f12bf2
spec:
  conversion:
    strategy: None
  group: gateway.mulesoft.com
  names:
    kind: PolicyBinding
    listKind: PolicyBindingList
    plural: policybindings
    singular: policybinding
  scope: Namespaced
  versions:
  - additionalPrinterColumns:
    - description: The target to apply the Policy
      jsonPath: .spec.targetRef.name
      name: Target
      type: string
    - description: The Policy to apply
      jsonPath: .spec.policyRef.name
      name: Policy
      type: string
    name: v1alpha1
    schema:
      openAPIV3Schema:
        properties:
          spec:
            x-kubernetes-preserve-unknown-fields: true
        type: object
    served: true
    storage: true
......[redacted status fields].....

The key section is below.

1
2
3
4
5
6
    schema:
      openAPIV3Schema:
        properties:
          spec:
            x-kubernetes-preserve-unknown-fields: true
        type: object

This means that you can put anything in the CRD and it’s treated as valid by kubectl. That’s bad - I had no real way of troubleshooting my CRD’s. Perhaps these are just the CRD’s installed by Helm, but this is not helpful.

The only reliable method I found was to switch to a different AI model that I had helping me :)

But AI’s suggestion as I’m writing this blog post is now: pull the definition off the pod, then build the binding empirically against a minimal working example.

1
2
3
POD=$(oc get po -n gateway -l app=ingress -o name | head -1)
oc exec -n gateway $POD -- \
  cat /etc/mulesoft/flex-gateway/conf.d/policies/definitions/rate_limit_definition.yaml

Read it for the property meanings, not as gospel for field names and nesting. Then test.

Rate limiting — the complete working binding

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
apiVersion: gateway.mulesoft.com/v1alpha1
kind: PolicyBinding
metadata:
  name: demo-rate-limit
  namespace: gateway
spec:
  policyRef:
    name: rate-limiting-flex        # NOT "rate-limit"
  targetRef:
    kind: ApiInstance               # required — not optional
    name: demo-api
  config:
    rateLimits:                     # keySelector/exposeHeaders/clusterizable
      - maximumRequests: 5          # live INSIDE this array item
        timePeriodInMilliseconds: 10000
        exposeHeaders: true
        clusterizable: false        # false for this single-replica demo; true in prod
        keySelector: "#[attributes.clientAddress.ip]"   # rate-limit per client IP

Test it — five in, then rejections:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ for i in $(seq 1 10); do
    curl -s -o /dev/null -w "req $i -> %{http_code}\n" localhost:18081/a
  done
req 1 -> 200
req 2 -> 200
req 3 -> 200
req 4 -> 200
req 5 -> 200
req 6 -> 429
req 7 -> 429
req 8 -> 429
req 9 -> 429
req 10 -> 429

$ curl -s localhost:18081/a          # body once the budget is spent
{"error":"Too Many Requests"}

Gotchas that actually cost me time:

• keySelector is effectively required. Leave it off and the policy loads happily but the enforcement behavior is not what you expect. Set it explicitly.
• It’s attributes.clientAddress.ip, not attributes.request.ip. Wrong expression → no useful key.
• targetRef.kind: ApiInstance is mandatory. Omit it and the binding won’t attach.
• Honesty check on exposeHeaders. I set exposeHeaders: true expecting RateLimit-Limit / RateLimit-Remaining / RateLimit-Reset headers on allowed responses. On v1.13.3 in this local-mode setup I did not see them emitted on the 200s — the enforcement (200→429) is rock solid, but the informational headers didn’t surface for me. If your clients depend on RateLimit-* headers to self-throttle, validate that behavior on your version before you promise it downstream.

JWT validation

This is where I’ll correct my own initial notes. If you follow blog posts (including drafts of this one) that show jwtSigningMethod, jwtSigningKeyLength, and jwtKey, the binding will not behave. The actual working field names on v1.13.3 are signingMethod, signingKeyLength, and textKey:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
apiVersion: gateway.mulesoft.com/v1alpha1
kind: PolicyBinding
metadata:
  name: demo-jwt-auth
  namespace: gateway
spec:
  policyRef:
    name: jwt-validation-flex       # NOT "jwt-validation"
  targetRef:
    kind: ApiInstance
    name: demo-api
  config:
    signingMethod: hmac             # not "jwtSigningMethod"
    signingKeyLength: 256           # not "jwtSigningKeyLength"
    jwtKeyOrigin: text
    textKey: "12345678901234567890123456789012"   # not "jwtKey"; plain text, NOT base64
    jwtOrigin: httpBearerAuthenticationHeader
    validateAudClaim: false
    mandatoryExpClaim: true
    skipClientIdValidation: true

Generate a token (standard PyJWT):

1
2
3
4
import time, jwt
now = int(time.time())
payload = {"sub": "test", "iat": now, "nbf": now, "exp": now + 3600}  # 1-hour expiry
print(jwt.encode(payload, "12345678901234567890123456789012", algorithm="HS256"))

Test the three cases:

1
2
3
4
5
6
7
8
9
10
11
12
$ curl -s -o /dev/null -w "%{http_code}\n" localhost:18081/a
400                                   # no token
$ curl -s localhost:18081/a
{"error":"JWT Token is required."}

$ curl -s -o /dev/null -w "%{http_code}\n" \
    -H "Authorization: Bearer $TOKEN" localhost:18081/a
200                                   # valid token → whoami

$ curl -s -o /dev/null -w "%{http_code}\n" \
    -H "Authorization: Bearer not.a.real.token" localhost:18081/a
401                                   # bad token

Note the status semantics: missing token is 400 (malformed request), present-but-invalid is 401 (unauthorized). Some clients treat those very differently.

Why plain-text textKey, not base64

The jwtKeyOrigin field selects how the key is interpreted:

• text — the key is a literal string (correct for HMAC / HS256).
• jwks — the key is a URL to a JWKS endpoint (for RS/ES asymmetric keys).
• customExpression — the key comes from a DataWeave expression.

For HS256 use text and supply the secret as-is, in plain text. If you base64-encode it out of habit, the policy tries to interpret the decoded bytes as a PEM key and fails with:

1
Error generating PEM InvalidData(InvalidByte(4, 45))

That error is a red herring the first time you see it — it looks like a certificate problem, but the real cause is “you gave me base64 and text origin.” Keep the secret plain.

The 32-bit WASM timestamp gotcha

Flex/Omni Gateway runs each policy as a WebAssembly module (wasm32-unknown-unknown) inside Envoy. The jwt-validation policy is Rust compiled to that 32-bit WASM target, and its claim-parsing code calls .unwrap() on a timestamp conversion. Feed it an exp that’s large enough to overflow the conversion and the .unwrap() panics on None, which traps the WASM module on an unreachable instruction — and the gateway returns 503 for that request.

I mapped the boundary empirically on v1.13.3 by walking exp upward:

1
2
3
4
5
6
7
exp = 2147483647   (2^31-1, "2038")  -> HTTP 200   # accepted
exp = 2147483648                      -> HTTP 200
exp = 4294967295   (2^32-1)           -> HTTP 200
exp = 4294967296   (2^32)             -> HTTP 200
exp = 5000000000                      -> HTTP 200
exp = 9999999999                      -> HTTP 503   # panic
exp = 99999999999                     -> HTTP 503

So — and this corrects the common “it breaks at 2038 / 2³¹” claim — on this version the wall sits above 2³², somewhere between 5×10⁹ and ~10¹⁰, not at the signed-32-bit 2038 boundary. Wherever exactly the threshold is, the point stands: unrealistically large exp values crash the request.

The smoking gun, straight from oc logs deploy/ingress:

1
2
3
4
5
6
7
8
[flex-gateway-envoy][critical] wasm log demo-jwt-auth ...
  panicked at .../jwt-lib/src/model/claims.rs:340:65:
  called `Option::unwrap()` on a `None` value
[flex-gateway-envoy][error] Function: proxy_on_request_headers failed:
  ... jwt_validation.wasm!..extract_datetime_value ...
      jwt_validation.wasm!..JWTClaims::new ...
Caused by:
  wasm trap: wasm `unreachable` instruction executed

extract_datetime_value → .unwrap() on None → abort → unreachable. That’s a shipping robustness bug: the policy should clamp or reject an out-of-range timestamp gracefully, not abort the WASM VM and 503 the caller.

The fix is boring and correct: use realistic Unix timestamps. now + 3600 for a one-hour token is fine. You only hit this if you (or a buggy token minter) emit absurd expiries — but it’s exactly the kind of thing a fuzzer or a copy-pasted “never expires” hack will trigger in the wild, so it’s worth knowing the failure mode is a 503, not a clean 401.

Part 4 — F5 CIS integration

The reader here is assumed to know F5, so this is deliberately brief — the CIS side is nearly identical to the Kong post. What changed is that this cluster drives BIG-IP config from an OpenShift Route plus an extended-spec ConfigMap, rather than a VirtualServer CRD.

CIS runs in kube-system, watching the relevant namespaces, in OpenShift controller mode with cluster (pod-direct) pool members:

1
2
3
4
5
6
7
8
9
$ oc get deploy f5cis-f5-bigip-ctlr -n kube-system \
    -o jsonpath='{.spec.template.spec.containers[0].args}'
--bigip-partition=openshift
--bigip-url=10.0.4.11
--controller-mode=openshift
--pool-member-type=cluster        # pool members = pod IPs, not NodePorts
--orchestration-cni=ovn-k8s
--static-routing-mode=true
--namespace=kube-system --namespace=gateway --namespace=mulesoft-demo

pool-member-type=cluster with orchestration-cni=ovn-k8s and static-routing-mode=true is what gives us pod-direct load balancing on OVN-Kubernetes: BIG-IP balances straight across the gateway’s pod IPs, no NodePort or router hop, and pool membership tracks pods automatically.

The Route (edge TLS decryption, HTTP→HTTPS redirect, targeting the :8081 Service):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: mulesoft-route
  namespace: gateway
  labels:
    f5type: systest                 # picked up by CIS
spec:
  host: mulesoft.my-f5.com
  path: /
  port:
    targetPort: 8081
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Redirect
  to:
    kind: Service
    name: ingress-8081
    weight: 100

The extended-spec ConfigMap pins the VIP, partition, and a Policy CR (TCP profiles + SNAT; the WAF attach point lives here too):

1
2
3
4
5
6
extendedRouteSpec:
  - namespace: gateway
    vserverAddr: 10.0.0.101           # the BIG-IP VIP
    vserverName: mulesoft-demo-routes
    bigIpPartition: mulesoft-demo
    policyCR: kube-system/security-policy

A quick confirmation that BIG-IP actually admitted the Route — note it’s admitted by two routers, the OpenShift default and F5 BIG-IP (CIS):

1
2
3
$ oc get route mulesoft-route -n gateway -o jsonpath='{range .status.ingress[*]}{.routerName}{"\n"}{end}'
default
F5 BIG-IP

Put together, the flow is:

1
2
3
4
5
Internet ─▶ BIG-IP VIP  mulesoft.my-f5.com : 443
         ─▶ TLS decrypt (edge policy on the Route)
         ─▶ CIS pod-direct routing to gateway pod IP : 8081
         ─▶ Omni Gateway  (JWT + rate-limit)
         ─▶ backend  /a (app-a)  or  /b (app-b)

Cert caveat. This demo terminates TLS with the BIG-IP’s default self-signed cert (CN=localhost.localdomain), which is why every curl below uses -k. Use a real, trusted certificate in production.

Part 5 — End-to-end testing (layered, so failures are localized)

The value of the three-layer design is that you can test each layer in isolation. If layer N misbehaves, the bug is in N — not upstream.

Layer 1 — gateway routing only (port-forward, no edge):

1
2
curl -s localhost:18081/a   # app-a
curl -s localhost:18081/b   # app-b

Layer 2 — rate limiting (port-forward):

1
2
$ for i in $(seq 1 10); do curl -s -o /dev/null -w "%{http_code} " localhost:18081/a; done
200 200 200 200 200 429 429 429 429 429

Layer 3 — JWT (port-forward):

1
2
3
$ curl -s -o /dev/null -w "%{http_code}\n" localhost:18081/a                              # 400 (no token)
$ curl -s -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer $TOKEN"   localhost:18081/a   # 200
$ curl -s -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer garbage"  localhost:18081/a   # 401

Layer 4 — the whole stack through BIG-IP (https://mulesoft.my-f5.com, self-signed → -k):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# no token
$ curl -sk -o /dev/null -w "%{http_code}\n" https://mulesoft.my-f5.com/a
400
$ curl -sk https://mulesoft.my-f5.com/a
{"error":"JWT Token is required."}

# invalid token
$ curl -sk -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer bad" https://mulesoft.my-f5.com/a
401
$ curl -sk -H "Authorization: Bearer bad" https://mulesoft.my-f5.com/a
{"error":"Invalid token."}

# valid token → routed to app-a (whoami)
$ curl -sk -H "Authorization: Bearer $TOKEN" https://mulesoft.my-f5.com/a
Hostname: app-a-6fb74c6c6c-nxhvh
IP: 10.129.0.24
GET /a HTTP/1.1
Host: app-a.mulesoft-demo.svc.cluster.local
...

# valid token → routed to app-b (echo)
$ curl -sk -H "Authorization: Bearer $TOKEN" https://mulesoft.my-f5.com/b
{ "path": "/b", "headers": { "host": "app-b.mulesoft-demo.svc.cluster.local", ... } }

# HTTP is redirected to HTTPS at the edge
$ curl -sk -o /dev/null -w "%{http_code} -> %{redirect_url}\n" http://mulesoft.my-f5.com/a
302 -> https://mulesoft.my-f5.com:443/a

# rate limiting still enforced end-to-end (window mid-cycle here)
$ for i in $(seq 1 10); do
    curl -sk -o /dev/null -w "%{http_code} " -H "Authorization: Bearer $TOKEN" https://mulesoft.my-f5.com/a
  done
200 200 200 429 429 429 429 429 429 429

Same behavior through the edge as at the gateway — which is exactly what “the layers are decoupled” is supposed to mean.

Part 6 — Key learnings

The three teaching moments

1. Registration mode is architectural, and it fails silently. --connected=true (the wizard default) makes the gateway ignore your Kubernetes CRDs and wait for Anypoint contracts — readiness never passes, and there’s no error to grep for. Register --connected=false / gateway.mode=local. Registration is immutable per name, so budget for throwaway registrations while you experiment.

2. Extension definitions describe the policy, not the binding schema. Policy names differ (rate-limit → rate-limiting-flex), field names differ (signingMethod/textKey, not jwtSigningMethod/jwtKey), and nesting differs (rate-limit knobs live inside the rateLimits array item). Pull the definition off the pod for meaning, then build the binding empirically against a minimal working example — and test.

3. The JWT policy runs in 32-bit WASM and will 503 on absurd exp values. The panic is a real .unwrap() on None in claims.rs, trapping the WASM VM. On v1.13.3 the boundary is above 2³² (not the “2038”/2³¹ myth), but the takeaway is the same: use realistic expiries, and know that the failure mode is a 503, not a clean 401. Related: use plain-text textKey for HMAC — base64 makes the policy try to parse PEM and fail.

Operational insights

• Namespace isolation pays off. Gateway in gateway, apps in mulesoft-demo, CIS scoped to just the namespaces it needs. Clean blast radius, no cross-talk.
• Use a high port for demo listeners. :8081 dodges the chart’s :80/:443 listeners and the entire low-port SCC saga. If you must bind low ports as non-root, prefer NET_BIND_SERVICE + the ip_unprivileged_port_start sysctl over privileged.
• Logs are the tool. oc logs -f deploy/ingress -n gateway shows policy load/apply events and the WASM panics in real time. When the readiness probe is silent, the full startup log is where the truth is.
• SCC/PSP are constraints, not suggestions. Plan for them in any hardened cluster.

Wrapping up

Two gateways, one edge pattern. Kong or MuleSoft Gateway, the BIG-IP side barely changed: TLS and pod-direct load balancing at the edge via CIS, API routing/rate-limiting/auth inside the cluster. That’s the payoff of clean layering — the gateway is a swappable component.

Summary

After getting the Alpine jump host working for SSH access to my internal Hyper-V VM while on VPN, a few real-world gaps showed up once I started actually using it day to day. This post covers three follow-up issues and how I fixed each one.

1. WinSCP doesn’t read my SSH config file

VSCode’s Remote-SSH extension shells out to the real ssh.exe, so it picked up my ~/.ssh/config ProxyJump entry automatically, with zero extra config. WinSCP does not — it has its own SSH implementation and never reads that file, so ProxyJump is invisible to it.

The fix is to configure the jump manually as a Tunnel inside WinSCP itself:

1. Session settings → Advanced → Connection → Tunnel
2. Check Connect through SSH tunnel
3. Tunnel host name: the jump host’s external IP
4. Tunnel user name + private key (converted to .ppk via WinSCP’s bundled PuTTYgen if needed)
5. Leave the main session’s host name as the target VM’s internal IP

Once that’s set, WinSCP opens the tunnel to the jump host first, then connects on to the target VM through it — same end result as ProxyJump, just configured in WinSCP’s own UI instead of inherited from a shared config file.

2. Outbound internet from the internal VM stopped working

Separately from the SSH/inbound story, I noticed my internal Ubuntu VM couldn’t reach the internet at all while on VPN (apt-get update and friends). This is a different traffic direction than the jump host was originally built for, so it needed its own fix.

I pointed the Ubuntu VM’s default gateway at the Alpine jump host’s internal IP, and added FORWARD rules plus MASQUERADE for outbound traffic — the same shape of rule as the SSH forwarding, just for return-path traffic heading out to the internet instead of into the internal switch.

The rules alone didn’t get it working immediately, though. Troubleshooting turned up two separate issues:

• Strict reverse-path filtering (rp_filter). With rp_filter=1 (strict) on both interfaces, the kernel was silently dropping forwarded packets before they reached eth0, even though the routes and FORWARD rules were correct. Disabling it (rp_filter=0 on both interfaces, plus all/default) cleared that up:

  1
  2
  3
  4
  echo 0 > /proc/sys/net/ipv4/conf/eth0/rp_filter
  echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter
  echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
  echo 0 > /proc/sys/net/ipv4/conf/default/rp_filter
  

• net.ipv4.ip_forward was actually still 0. This was the real root cause — my earlier sysctl -p hadn’t actually applied it (or something reset it), and I didn’t catch it until I checked sysctl net.ipv4.ip_forward directly rather than assuming the config file entry meant it was live. With forwarding disabled at the kernel level, packets arriving on the internal NIC were dropped before ever reaching the FORWARD chain — which is why the FORWARD counters stayed at zero no matter what rules I added.

Once ip_forward was actually confirmed as 1, outbound internet access from the internal VM started working immediately, routed through Alpine’s own MASQUERADE’d IP on the way out.

Lesson: when packets aren’t reaching a chain at all (0 packets/0 bytes on every rule), check the more fundamental kernel switches first (ip_forward, rp_filter) before troubleshooting the iptables rules themselves — a correct ruleset won’t matter if forwarding is off at the kernel level.

3. Forwarding other ports besides SSH

The original setup only handled SSH (port 22) via ProxyJump/tunneling, which works because it’s a client-initiated hop through an existing SSH session. But I also wanted to reach a plain HTTP service running on the internal VM (port 4000) directly by hitting the jump host’s external IP — no SSH involved at all.

That needs actual DNAT (destination NAT / port forwarding), not just the FORWARD-and-MASQUERADE setup used for outbound traffic:

1
2
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 4000 -j DNAT --to-destination 172.20.100.101:4000
iptables -A FORWARD -i eth0 -o eth1 -p tcp -d 172.20.100.101 --dport 4000 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT

The second rule mattered as much as the first: my existing eth0 → eth1 FORWARD rule only permitted RELATED,ESTABLISHED traffic (since it was written for SSH return traffic, not fresh inbound connections). A brand-new inbound connection to port 4000 doesn’t match either of those states, so without explicitly allowing NEW traffic for that specific port, the DNAT rule alone wasn’t enough — the first packet still got dropped at FORWARD.

With both rules in place, http://<jump-host-external-ip>:4000 now reaches the internal VM’s service directly.

Where things stand now

• SSH/SFTP via CLI and VSCode: works transparently through ~/.ssh/config, on or off VPN.
• WinSCP: works, via a manually configured tunnel (not automatic).
• Outbound internet from the internal VM: works, routed through Alpine.
• Other TCP services on the internal VM: reachable via explicit DNAT rules per port, as needed.

Bit by bit this jump host has turned into the actual gateway for my internal switch, in both directions — worth keeping this post as the reference for which rule handles which direction next time I need to add another port.

Summary

I run my daily-use Ubuntu VM on a custom Hyper-V Internal virtual switch (see my previous post on giving it a persistent static IP). That setup works great — until I connect to my corporate VPN (F5 Access / BIG-IP APM), at which point the VM becomes completely unreachable from my laptop. The fix wasn’t a routing tweak, it was a small Alpine Linux VM acting as a jump host with one leg on my physical LAN and one leg on the internal switch.

The problem

Connecting to VPN broke SSH/WinSCP/VSCode access to my internal VMs. My first assumption was routing: maybe the VPN client was pushing a full-tunnel default route that was winning over my internal subnet.

Turned out to be wrong on both counts, but it took some diagnosis to get there.

Checking the routing table

Before/after connecting to VPN, I compared route print output. The route to my internal switch’s subnet (172.20.100.0/24) was actually still present and correctly installed as a persistent route:

1
2
Network Destination        Netmask          Gateway       Interface  Metric
172.20.100.0     255.255.255.0   On-link      172.20.100.1     16

Since this is a more specific prefix than the VPN’s 0.0.0.0/0 default route, longest-prefix-match rules mean it should always win regardless of metric. So the routing table wasn’t lying to me — the route was there, on-link, correct.

And yet, ping to my VM and even to my own internal switch’s gateway (172.20.100.1) went completely silent while on VPN. No “destination unreachable,” just nothing. That silence is the tell: a route being present but traffic still not passing means something below the routing layer is filtering it.

It’s not routing — it’s the VPN client’s packet filter

F5 Access (like most enterprise VPN clients doing split-tunnel enforcement) installs a Windows Filtering Platform (WFP) callout driver that blocks traffic to non-tunnel destinations at the kernel level, independent of the routing table. This is deliberate anti-split-tunnel policy enforcement, controlled server-side by the BIG-IP APM Access Policy (a “Local Subnet Access” setting, or a split-tunnel exclude list) — not something fixable from the client side or with local admin rights.

I confirmed the scope of the block with a couple of pings while connected to VPN:

• ping 172.20.100.1 (my own internal switch gateway) — failed
• ping 192.168.1.1 (my physical LAN gateway) — succeeded

So the block is scoped specifically to the Hyper-V internal-switch adapter, not a blanket “nothing but the tunnel” policy. That distinction is what made a workaround possible at all.

What I considered that didn’t work

For completeness, here’s what I ruled out before landing on the jump host:

• Bumping the interface metric on the internal vEthernet adapter — didn’t matter, since the route was never the problem.
• Adding a persistent static route with admin rights (route -p add ...) — the route survived the VPN connection just fine, but traffic still didn’t pass. This is what confirmed it was a filter, not a route.
• Trying to remove/override the WFP filter directly — technically possible with admin rights, but a bad idea in practice: the F5 client service re-asserts its policy on a timer or reconnect, so any local override would likely get stomped anyway. More importantly, this is my own company’s security policy working as designed so I didn’t touch. Also, I’m out of my depth here, guided with AI tools but am not confident here anyway.

The actual fix for the root cause is a server-side APM policy change (Local Subnet Access, or excluding 172.20.100.0/24 from the tunnel) — that’s a conversation with whoever owns the BIG-IP policy, not something to solve locally. In the meantime, I wanted a workaround.

The workaround: a jump host VM

The idea: a small VM with one NIC on my physical LAN (External switch) and one NIC on my internal switch, sitting between my laptop and my other VMs. Since traffic to my physical LAN gateway survives the VPN’s filter, SSH-ing to the jump host’s LAN-facing IP works fine even on VPN — and from there, it’s a local hop to the internal switch, which the VPN never touches.

1
Host (VPN on) → SSH to jump host on 192.168.1.x → forwards to target VM on 172.20.100.x

VM vs. container

I considered a Windows container with the transparent network driver instead of a full VM, mainly to save on footprint. Decided against it:

• Docker Desktop’s default WSL2 backend shares one NAT’d network view across containers — no clean way to attach a container directly to two different Hyper-V virtual switches.
• The transparent driver can attach a container to a vSwitch directly, but that requires Windows Server Core/Nano containers, not Linux containers, and is a much less common, less documented setup.
• A bare Alpine VM ends up with a smaller footprint anyway than Docker Desktop’s backing WSL2 VM, so the “container to save memory” reasoning didn’t actually hold up once I compared the numbers.

A VM with two vNICs attached to two switches is just how Hyper-V networking normally works — nothing exotic, nothing to fight.

Building the jump host

• OS: Alpine Linux, alpine-virt edition (the ISO built for VM use) — smallest realistic option with normal package management, iptables, and OpenSSH support.
• VM generation: started on Generation 2, but hit a boot failure — the VM would fall through to PXE boot even with Hard Drive first in the firmware boot order and Secure Boot disabled. Switched to Generation 1 (legacy BIOS/MBR boot) and the install was reliable. For a small headless VM with no need for Gen2-specific features, Gen1 is the simpler choice anyway.
• NICs: one attached to my External virtual switch (physical LAN), one attached to my existing Internal virtual switch (172.20.100.0/24).
• IP forwarding + NAT:

  1
  2
  3
  4
  5
  6
  echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
  sysctl -p
  apk add iptables iptables-openrc
  iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
  rc-update add iptables
  /etc/init.d/iptables save
  

• SSH: created a non-root user, added my public key to ~/.ssh/authorized_keys, enabled and started sshd. Had to explicitly set AllowTcpForwarding yes in sshd_config — it was blocking the ProxyJump hop with an “administratively prohibited” error until I turned it on.

Making it transparent from Windows

A couple of entries in ~/.ssh/config on my laptop:

1
2
3
4
5
6
7
8
Host jumphost
    HostName 192.168.1.x
    User jumpuser

Host devvm
    HostName 172.20.100.101
    User <my-vm-user>
    ProxyJump jumphost

With that in place, ssh devvm (or ssh 172.20.100.101 directly, since the config matches on hostname) transparently hops through the jump host. No change needed in WinSCP or VSCode Remote-SSH either, since both understand ProxyJump.

I don’t actually work from a PowerShell or CMD prompt day to day — I live in VSCode. Since VSCode’s Remote-SSH extension reads the exact same ~/.ssh/config file the ssh CLI does, this fix applied with zero changes on my end there. I opened VSCode on VPN, connected to the same host entry I always use, and it just worked — routed through the jump host without me touching any settings.

The final result

• Off VPN: my Ubuntu dev VM is reachable directly on 172.20.100.101, same as always.
• On VPN: SSH to 172.20.100.101 is automatically routed via the jump host at 192.168.1.205, and it works — no change to my daily workflow, no manual jump-host step to remember.

Feedback and summary

The lesson that took the longest to land: a correct, present route in route print doesn’t mean traffic will actually pass — WFP-level filtering happens below routing and won’t show up there at all. Ping tests to a couple of different gateways (internal switch vs. physical LAN) were what actually isolated the scope of the block and pointed me at a workaround instead of a dead end.

The proper long-term fix is still a policy change on the APM side, and I’ll follow up on that separately. This jump host is a workaround, not a fix — but it’s a workaround that survives reboots, VPN reconnects, and switching between Ethernet and WiFi, so it’s good enough for now.

A reference architecture where the API gateway and the ADC each do what they do best

Often, an API gateway and F5 BIG-IP are complementary, not competing. They live at different layers of the stack and solve different problems. Put them together and you get a clean separation of concerns: the ADC owns the edge, the gateway owns API management.

For this article, I have chosen specific products to show configuration examples:

• For my Kubernetes (K8s) cluster, I’m using OpenShift
• For my API Gateway I’m using open source Kong
• For my ADC I’m using F5 BIG-IP

Of course, I could use a different K8s distribution (Rancher, AKS, GKE, etc) and a different API Gateway (NGINX, Gloo, Apigee, etc), and even a different ADC. The pattern is identical: F5 BIG-IP fronts the gateway, and the gateway fronts the services.

The split of responsibilities

The whole idea fits on one line:

1
2
client ──HTTPS──▶ F5 BIG-IP (TLS + WAF + LB) ──HTTP──▶ Kong (routing + rate-limit + auth) ──▶ backend apps
                      

F5 BIG-IP — the edge / entry point:

• TLS termination. BIG-IP presents the TLS certificate, decrypts, and forwards as HTTP to Kong inside the cluster.
• Advanced WAF. L7 protection (SQLi, XSS, etc.) is enforced at the edge, before traffic reaches Kong.
• Load balancing straight to pods. F5 Container Ingress Services (CIS) runs in cluster mode, so the BIG-IP pool members are the Kong proxy pod IPs.

Kong — behind F5, inside OpenShift:

• Request routing to backend services by path.
• Rate limiting per route.
• Authentication (here, API key / key-auth).

Notice there is no overlap. BIG-IP is not handling API keys; Kong is not performing WAF. Each can be operated by the team that owns it — NetOps/SecOps at the edge, the platform/API team inside the cluster.

The environment

Kong Gateway (open source) runs as a single Deployment kong-kong with replicas=1, installed via helm in namespace kong-f5-demo. The proxy Service is ClusterIP on purpose: the external entry point is BIG-IP, not the OpenShift router.

Two sample backends sit behind Kong:

• httpbin (mccutchen/go-httpbin) exposed at /httpbin
• app2 (nginx serving a static page) exposed at /app2

BIG-IP integration is via CIS running inside the cluster watching custom resources. A WAF policy /Common/kong_demo_waf_policy already exists on the BIG-IP.

Here is the running namespace kong-f5-demo shown with CLI output and UI screenshots:

1
2
3
4
5
6
7
8
9
10
11
12
$ oc get pods -n kong-f5-demo -o wide
NAME                         READY   STATUS    RESTARTS   AGE     IP            NODE
app2-64bcb6968b-plfp4        1/1     Running   0          6h24m   10.129.0.49   ip-10-0-2-184...
httpbin-67f5ffcb5f-b7b85     1/1     Running   0          6h22m   10.129.0.50   ip-10-0-2-184...
kong-kong-54b7bf4765-78f9j   2/2     Running   0          6h24m   10.129.0.47   ip-10-0-2-184...

$ oc get svc -n kong-f5-demo
NAME              TYPE        CLUSTER-IP       PORT(S)
app2              ClusterIP   172.30.4.204     80/TCP
httpbin           ClusterIP   172.30.29.73     80/TCP
kong-kong-admin   ClusterIP   172.30.27.62     8001/TCP,8444/TCP
kong-kong-proxy   ClusterIP   172.30.130.33    80/TCP,443/TCP

The configuration

BIG-IP: one VirtualServer, one TLSProfile

BIG-IP is configured by CIS, which watches K8s custom resources. Two resources describe the BIG-IP config:

TLSProfile — TLS edge termination, referencing a K8s TLS secret that holds the cert for kong-demo.my-f5.com:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
apiVersion: cis.f5.com/v1
kind: TLSProfile
metadata:
  name: kong-f5-demo-tls
  namespace: kong-f5-demo
  labels:
    f5cr: "true"
spec:
  hosts:
    - kong-demo.my-f5.com
  tls:
    termination: edge
    reference: secret
    clientSSL: kong-f5-demo-tls   # K8s TLS secret

VirtualServer — Note the three F5 responsibilities in one object: the TLS profile, the WAF policy reference, and a pool that points straight at the Kong proxy Service (which CIS translates to pod IPs):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
apiVersion: cis.f5.com/v1
kind: VirtualServer
metadata:
  name: kong-proxy-vs
  namespace: kong-f5-demo
  labels: 
    f5cr: "true"
spec:
  host: kong-demo.my-f5.com
  virtualServerAddress: 10.0.0.101 
  tlsProfileName: kong-f5-demo-tls # reference to TLSProfile object
  waf: /Common/kong_demo_waf_policy # reference to existing policy
  pools:
    - path: /
      service: kong-kong-proxy #pod IP's of this service become pool members
      servicePort: 80   

That servicePort: 80 shows that BIG-IP VIP is listening on tcp/443 (HTTPS), decrypting, and forwarding to the pool on tcp/80 (HTTP).

Kong: Ingresses and Plugins

Kong is also configured the K8s-native way. Each backend gets an Ingress with ingressClassName: kong and konghq.com/strip-path: "true" (so /httpbin/get reaches the backend as /get). Policy is defined as KongPlugin objects and attached to routes with the konghq.com/plugins annotation.

Routing — one Ingress per backend:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
  namespace: kong-f5-demo
  annotations:
    konghq.com/strip-path: "true"
    konghq.com/plugins: rate-limiting-demo   # policy attached here
spec:
  ingressClassName: kong
  rules:
    - http:
        paths:
          - path: /httpbin
            pathType: ImplementationSpecific
            backend: { service: { name: httpbin, port: { number: 80 } } }

The app2 Ingress is identical in shape but points at the app2 Service on /app2 and attaches key-auth-demo instead.

Rate limiting — 10 requests/minute, local counter:

1
2
3
4
5
6
7
8
9
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata: 
  name: rate-limiting-demo
  namespace: kong-f5-demo
plugin: rate-limiting
config:
  minute: 10
  policy: local

Authentication — API key carried in the apikey header, plus a consumer identity:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata: 
  name: key-auth-demo
  namespace: kong-f5-demo
plugin: key-auth
config:
  key_names: [ apikey ]
---
apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: demo-consumer
  namespace: kong-f5-demo
  annotations:
    kubernetes.io/ingress.class: kong
username: demo-consumer
credentials:
  - demo-apikey     # Secret holding the key value

Again — this is Kong syntax, but the concepts (routes, a rate-limit policy, a key-auth policy, a consumer) map onto every API gateway on the market.

Proving it works

All requests below go to https://kong-demo.my-f5.com, which resolves to the BIG-IP VIP 10.0.0.101.

1. TLS is terminated on the BIG-IP

The served certificate is the BIG-IP’s, with CN=kong-demo.my-f5.com:

1
2
3
4
$ curl -kv https://kong-demo.my-f5.com/httpbin/get 2>&1 | grep -E 'subject:|issuer:'
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
*   subject: CN=kong-demo.my-f5.com
*   issuer:  C=US; O=Let's Encrypt; CN=YE2

2. The WAF blocks attacks before Kong sees them

A classic SQL-injection probe is stopped at the edge — HTTP 403 with the F5 block page:

1
2
3
4
5
6
7
8
$ curl -k -i "https://kong-demo.my-f5.com/httpbin/get?id=1' OR '1'='1'--"
HTTP/1.1 403 Forbidden
Content-Type: text/html; charset=utf-8
...
<html><head><title>Blocked by F5 BIG-IP WAF</title></head>
<body><h1>403 Forbidden</h1>
<p>Request blocked by F5 Advanced WAF before reaching Kong.</p>
<p>Support ID: 1487496205675004032</p></body></html>

An XSS attempt gets the same treatment:

1
2
3
4
$ curl -k -i "https://kong-demo.my-f5.com/httpbin/anything?q=<script>alert(1)</script>"
HTTP/1.1 403 Forbidden
...
<p>Request blocked by F5 Advanced WAF before reaching Kong.</p>

Rendered in a browser, the block page looks like this:

The key architectural point: Kong and the backends never process the malicious payload. The attack surface is reduced before the API-management layer.

3. Kong routes clean traffic to the backends

Legitimate requests sail through F5 and Kong routes them by path. /httpbin/get (strip-path turns it into /get):

1
2
3
4
5
6
7
8
9
$ curl -k https://kong-demo.my-f5.com/httpbin/get
{
  "args": {},
  "headers": {
    "Host": [ "kong-demo.my-f5.com" ],
    "User-Agent": [ "curl/8.18.0" ]
  },
  ...
}

4. Kong enforces rate limiting

The rate-limiting plugin caps /httpbin at 10 requests/minute. On every allowed request Kong returns the standard rate-limit headers, and once the budget is spent it returns HTTP 429 with a JSON body:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
### RateLimit headers on a normal request:
RateLimit-Limit: 10
RateLimit-Remaining: 9
RateLimit-Reset: 26

### 12 requests inside one minute:
req  1 -> 200
req  2 -> 200
 ...
req 10 -> 200
req 11 -> 429
req 12 -> 429

### Body once the limit is exceeded:
{ "message": "API rate limit exceeded", "request_id": "d75e6d49..." }

Note: This example is hitting Kong proxy directly to show 429 responses. My WAF will re-classify Kong’s 429 response and serve its own 403 block page to the client. It’s a good illustration of the layering: the gateway decides “too many requests, send a 429” the edge decides “I see a 429 response but I’ll forward a 403 to the client”.

5. Kong enforces authentication

The /app2 route requires an API key.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# No key → Kong returns 401

curl -k -i https://kong-demo.my-f5.com/app2/
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Key
Content-Type: application/json; charset=utf-8

{"message":"No API key found in request"}

# Key in `apikey` header → 200 and the app2 page:

curl -k -i -H "apikey: demo-secret-key-12345" https://kong-demo.my-f5.com/app2/
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8

Why this layering is worth it

Walk the request path one more time and note who owns each decision:

A few architectural advantages:

• Clean ownership boundaries. NetOps/SecOps manage the edge (certs, WAF policy, VIPs) via BIG-IP; the API/platform team manages routes and API policy via K8s objects. Neither steps on the other.
• Blast-radius reduction. SQLi/XSS/other L7 attacks are dropped at the edge.
• Direct-to-pod load balancing. CIS cluster mode means the BIG-IP balances across Kong pod IPs — no extra NodePort/router hop, and pool membership tracks pods automatically.
• Portability. Everything is configured using K8s resource, for both Kong and BIG-IP. The gateway is replaceable — the edge pattern doesn’t change.

Kong here is just the example. The architecture here is: let the ADC do TLS, WAF, and pod-level load balancing at the edge, and let your API gateway do routing, rate limiting, and auth. They’re better together.

Appendix — asset files

Raw CLI captures used above are saved alongside this article under blog-assets/:

• cli-01-openshift-resources.txt — pods / services / ingress / plugins / CRs
• cli-02-tls-termination.txt — served-cert subject/issuer
• cli-03-waf-block.txt — SQLi + XSS 403 responses
• cli-04-routing.txt — httpbin + app2 200 responses
• cli-05-ratelimit.txt — rate-limit headers + 200→429 progression + 429 body
• cli-06-keyauth.txt — 401 without key, 200 with key
• cli-07-manifests.txt — full VirtualServer / TLSProfile / Ingress / KongPlugin / KongConsumer manifests

My kid’s closet was a disaster. Two fabric hanging organizers crammed with clothes, shoes piled on the floor, and absolutely no system. The standard single rod and one high shelf weren’t cutting it.

I decided to build a custom built-in: two flanking shelving towers, a center hanging zone, and four shallow drawers below the hang area. Total interior width was 50.5 inches. I wanted everything white, clean, and furniture-quality.

This is a record of what I did, what I learned, and what I’d do differently.

Design decisions

The closet is 50.5 inches wide with a 30-inch door opening and 10-inch recesses on each side behind the door frame. The depth is 26 inches total, but with clearance for reaching in, the shelving depth is 12 inches.

Key constraints:

• An existing top shelf support at 67 inches height (kept it — no reason to remove it)
• An existing mid-height wall support at 34.5 inches (used it as a natural shelf rest)
• Baseboard 0.75 inches thick and 5 inches tall, running along all three walls
• No reliable stud locations (old house, unreliable stud finder)

The final layout:

• Two inner divider panels — 12 inches deep, 67 inches tall, notched at the back to clear the baseboard, mid support, and top support
• Left and right shelving towers — 14.5 inches interior width each, 5 shelves per tower at 12.5”, 23.5”, 34.5”, 45.5”, and 56.5” from the floor
• Center zone — 20 inches wide: hanging rod above 34.5”, four shallow drawers below
• Full-width top shelf at 67 inches resting on the existing wall support

Tools and materials

Tools:

• main tools: drill, impact driver, jigsaw, miter saw, circular saw, sander
• other: Clamps, big and small spirit levels, square ruler, pocket hole jig

Materials:

• 3/4 inch birch plywood (shelving and dividers)
• 1/2 inch birch plywood (drawer box sides, fronts, backs) and 1/4 inch (bottoms)
• white primer and white paint
• other: drawer slides, screws, wood glue

Build process

The store cut my plywood into 12-inch strips on their panel saw, which made transport easy and reduced my circular saw work significantly.

Divider panels first. Each panel needed three notches cut from the back edge to clear the wall supports: baseboard (0.75 deep × 5 tall), mid support (0.75 deep × 3.75 tall, starting at 30.75 inches), and top support (0.75 deep × 3.5 tall, from the top). I did a terrible job with a cheap jigsaw but it will work.

Dry fit everything before committing. I assembled the full structure with one temporary screw per divider panel before any glue or painting. This confirmed the notches cleared correctly, the top shelf sat level, and the proportions looked right in person. A few minor adjustments.

Cleats for shelf support. I cut simple 11-inch wooden cleats from offcuts (3/4 × 1 inch strips) and glued and screwed them to the wall faces and divider panel faces at each shelf height. At shelf positions that coincide with the existing wall supports, the support itself carries that side and only the divider-side cleat is needed.

Paint before final assembly. I sanded everything to 220 grit outdoors, primed one coat, lightly re-sanded, then applied two coats of matte white. Should have done semi-gloss but bought the wrong paint and will leave with it.

Drawers built from scratch. Each drawer box: front and back at 19.25 × 7 inches, sides at 10 × 7 inches (10 inches to account for the front and back panel thickness), and a 1/4 inch plywood bottom at 19.25 × 11 inches. Assembled with wood glue screws.

Slide heights from the floor (with 1/8 inch offset per Everbilt instructions):

• Drawer 1: 5 and 1/8 inches
• Drawer 2: 12 and 5/8 inches
• Drawer 3: 20 and 1/8 inches
• Drawer 4: 27 and 5/8 inches

Drawer faces cut at 21 × 7.5 inches (full overlay, covering the 20.25-inch opening with slight overlap onto the divider panel faces). Positioned using a US quarter as a consistent 1/16-inch spacer between faces, starting from the bottom drawer and working up. This was not 100% perfect, but they work.

Lessons learned

Use the store’s panel saw. Getting the store to rip your sheet goods into 12-inch strips saves enormous time and effort.

Dry-fit before painting, label before disassembly. Mark every piece with a pencil before you take it apart.

Wooden cleats are underrated. I considered shelf pin strips, L-brackets, and other hardware, but simple plywood offcut cleats glued and screwed in place are strong, cheap, and I think they look nice.

Pocket screws need the right length. 1-1/4 inch Kreg screws are for 3/4 inch material. For 1/2 inch drawer box material I had to switch to standard wood screws — the Kreg jig isn’t set up for thin stock without a different collar setting and shorter screws. Plan ahead.

Measure slide heights from the actual installed base, not the theoretical one. My baseboard height was 4.5 inches, not 5 inches as I’d initially thought. Small difference, but it cascades through all four slide heights if you get it wrong.

Drawer face positioning: don’t use a coin as a spacer. This was annoying in the end. Next time I’ll plan ahead and have something better as a spacer.

What I’d do differently

• Face panels. Seeing 3/4 inch plywood edges makes this look like a dad’s DIY project (which it is), not a professional job. Next time I might plan for something more.
• Plan better. If I did this again I’d spend more time drawing this out.
• Invest in a better stud finder earlier. I’ve done this now.
• Double-check dimensions if Home Depot does the cuts. After getting 1x 48-inch piece cut into 4 pieces, I had 3x 12-inch widths, and 1x ~11 inch. I forgot to account for minor variances.
• Semi-gloss paint for the shelving. I used matte, which looks clean, but semi-gloss would be more durable and wipeable for a kids’ room.

In my last post I installed the F5 CIS operator from the CLI and brought up the F5BigIpCtlr operand. This post picks up right after that: the operator is running, CIS is healthy, and now a customer wants to expose an application with a plain Service of type: LoadBalancer instead of an Ingress, a Route, or a VirtualServer custom resource.

Why Service Type LB?

Some teams don’t want to manage a CIS-specific CRD for every app - they just want the same type: LoadBalancer experience they’d get from a cloud provider’s native load balancer, and they want BIG-IP to be the thing that satisfies it on-prem.

The short version: CIS doesn’t watch Service objects for LoadBalancer behavior out of the box. Two operator-level args need to be set before CIS will even look at the Service, and then the Service itself needs an annotation telling CIS what IP to use.

Why a Service alone isn’t enough

If you just create a Service with type: LoadBalancer against a default CIS install, nothing happens. EXTERNAL-IP sits at <pending> forever, because there’s no cloud controller manager assigning it and CIS, by default, isn’t watching for this resource type at all.

CIS has to be put into a mode where it watches Custom Resources, and on OpenShift, it needs to know which CNI plugin is in play so it can correctly read pod and node networking information. Both of these are operator-level settings, not something you can fix from the Service manifest alone.

Notable parameters

Notice these parameters were all included when we defined our operand in our last post.

Worth calling out: custom_resource_mode is the parameter that’s actually mandatory for Service type LoadBalancer support. static_routing_mode and orchestration_cni enable routes to be created on F5 BIG-IP so that the BIG-IP can know which Pod IP address blocks belong to which OpenShift nodes.

The F5BigIpCtlr operand

This is the same operand I had in the last blog post.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
apiVersion: cis.f5.com/v1
kind: F5BigIpCtlr
metadata:
  name: cis1
  namespace: kube-system
spec:
  namespace: kube-system
  args:
    bigip_url: 10.0.4.11
    bigip_partition: openshift
    log_level: DEBUG
    insecure: true
    pool_member_type: cluster
    manage_routes: false
    custom_resource_mode: true
    static_routing_mode: true
    orchestration_cni: ovn-k8s
  bigip_login_secret: bigip-login
  image:
    repo: f5networks/cntr-ingress-svcs
    user: registry.connect.redhat.com
    pullPolicy: Always
  version: 2.20.4-ubi10
  ingressClass:
    create: false
    defaultController: false
    ingressClassName: f5
  rbac:
    create: true
    namespaced: false
  serviceAccount:
    create: true

Exposing the application

With CIS configured, the customer’s application Service just needs type: LoadBalancer plus one CIS-specific annotation that tells CIS which IP to configure on BIG-IP for the virtual server.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
apiVersion: v1
kind: Service
metadata:
  annotations:
    cis.f5.com/ip: 10.8.3.1 #the VIP gets created with this IP
  labels:
    app: hello-world
  name: hello-world-svc-1
  namespace: hello-world
spec:
  ports:
    - name: port8080
      port: 8080
      protocol: TCP
      targetPort: 8080
  selector:
    app: hello-world
  type: LoadBalancer

Check that the IP landed on the Service:

1
oc get svc hello-world-svc-1 -n hello-world 

EXTERNAL-IP should now show 10.8.3.1 instead of <pending>, and CIS will have created a corresponding LTM virtual server on BIG-IP at that address.

A note on the IPAM alternative

cis.f5.com/ip is the hardcoded path - good for a customer who already owns a static IP and just wants CIS to honor it, which is the case I’m documenting here. F5 also ships an F5 IPAM Controller that can assign the address automatically from a pre-defined range, using a cis.f5.com/ipamLabel annotation instead of cis.f5.com/ip. That path needs the additional ipam: true arg on the CIS operand and an F5IPAM resource to define the range.

If both annotations happen to be present on the same Service, cis.f5.com/ip wins - CIS treats the hardcoded IP as the more specific instruction. For most customer environments I’ve seen, where IP assignment is already governed by an IPAM process outside the cluster, hardcoding via cis.f5.com/ip is the simpler and more predictable choice. IPAM earns its keep once you have many Services and don’t want to hand-manage individual addresses.

Final thoughts

Simple blog post intended for a copy/paste option for a customer that has installed CIS via OpenShift operator and wants to expose services of type LoadBalancer.

If you’re working through this for a real customer deployment, feel free to reach out!

I’ve installed F5 Container Ingress Services (CIS) into OpenShift a different ways over the years - raw manifests, Helm, and the OperatorHub tile. This week I wanted to install it via the OperatorHub operator, but skip the console entirely and do it from the CLI instead. The goal is to replicate, step by step, what a customer’s GitOps pipeline does when it subscribes to an operator, so I understand exactly what objects get created and in what order.

This post walks through:

1. A quick reminder of what’s actually happening when you click “Install” in OperatorHub
2. Doing the same install manually through the console, for reference
3. Doing it again from the CLI, object by object
4. Deploying the CIS operand (F5BigIpCtlr) with a plain YAML manifest once the operator is running

Cluster is OpenShift 4.21, freshly built, nothing else installed yet.

Why bother with the CLI when the GUI already works

A few reasons this was worth the time:

• GitOps. If a customer manages their cluster with ArgoCD or Flux, the operator subscription is just another manifest in a repo. Understanding the underlying objects means I can hand them exactly what they need instead of “go click around in OperatorHub.”
• Repeatability. I rebuild lab clusters constantly. oc apply -f three times beats re-navigating the console every time.
• It demystifies OLM. The console hides a sequence of object creations behind a single “Install” button. Once you’ve created the Namespace, OperatorGroup, and Subscription yourself, the rest of Operator Lifecycle Manager (OLM) stops being a black box. None of this is news if you’ve installed operators via CLI before, but it’s the first time I’ve done it specifically to mirror a customer’s workflow with the F5 CIS operator, so I wanted to write it down.

What OLM actually creates

OperatorHub installs, whether from the console or the CLI, boil down to the same three objects:

The InstallPlan and the resulting ClusterServiceVersion (CSV) are generated automatically by OLM in response to the Subscription - you never author those two yourself. Red Hat’s official walkthrough of this is in Administrator tasks for Operators in OpenShift Container Platform, specifically the “Installing from OperatorHub using the CLI” section. (If you’re on 4.21 docs and the link above 404s, just bump the version number in the URL - Red Hat’s doc structure is stable across releases.)

Part 1: Installing via the console (for reference)

I’m not going to dwell on this part since it’s not the point of the post, but for completeness:

1. Operators → OperatorHub, search for F5
2. Select F5 Container Ingress Services, click Install
3. Choose an installation mode (kube-system namespace, in my case), an update channel, and an approval strategy
4. Click Install again and wait for the CSV to reach Succeeded That’s the entire console flow. Behind the scenes it’s creating the same Namespace/OperatorGroup/Subscription trio described above - the console just bundles the choices into a wizard.

Part 2: Installing via the CLI

Prerequisites from the F5 documentation:

1. Create BIG-IP secret: oc create secret generic <SECRET-NAME> -n kube-system --from-literal=username=<USERNAME> --from-literal=password=<PASSWORD>
2. Install CRD’s: oc create -f https://raw.githubusercontent.com/F5Networks/k8s-bigip-ctlr/${CIS_VERSION}/docs/config_examples/customResourceDefinitions/customresourcedefinitions.yml

Step 1: Find the exact package name and channel

Don’t guess at the package name. Query the catalog directly:

1
oc get packagemanifests -n openshift-marketplace | grep -i f5

Then get the details for the one you want:

1
oc describe packagemanifest f5-bigip-ctlr-operator -n openshift-marketplace

This output tells you three things you need for the Subscription: the available channels, the catalog source name (e.g. certified-operators), and the default channel.

Step 2: Create the Namespace

(Since we are going to install the Operator into the kube-system namespace which already exists, we won’t worry about this)

Step 3: Create the OperatorGroup

The OperatorGroup scopes which namespace(s) the operator watches. Since I only need CIS watching one namespace, this is a single-namespace install:

1
2
3
4
5
6
7
8
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: f5-cis-og
  namespace: kube-system
spec:
  targetNamespaces:
  - kube-system

1
oc apply -f operatorgroup.yaml

Step 4: Create the Subscription

This is the object that actually kicks OLM into action.

1
2
3
4
5
6
7
8
9
10
11
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: f5-cis-subscription
  namespace: kube-system
spec:
  channel: stable
  name: f5-bigip-ctlr-operator
  source: certified-operators
  sourceNamespace: openshift-marketplace
  installPlanApproval: Automatic

A couple of fields worth calling out:

• channel, name, and source come straight from the packagemanifest describe output in Step 1 — don’t hardcode mine without checking yours

  1
  oc apply -f subscription.yaml
  

Step 5: Watch it converge

1
2
3
oc get subscription f5-cis-subscription -n kube-system -o yaml
oc get installplan -n kube-system
oc get csv -n kube-system

Once the CSV’s PHASE reads Succeeded, the operator pod is up and the F5BigIpCtlr CRD is registered on the cluster.

Part 3: Deploying the operand with a YAML manifest

The operator itself doesn’t do anything until you create an instance of its CRD. For the F5 CIS operator, that CRD is F5BigIpCtlr under the cis.f5.com/v1 API group. Since this is a Helm-based operator, the spec fields map directly onto the underlying Helm chart’s values.yaml keys - which is the whole reason this operator is “just” a packaging layer on top of Helm.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
apiVersion: cis.f5.com/v1
kind: F5BigIpCtlr
metadata:
  name: cis1
  namespace: kube-system
spec:
  namespace: kube-system
  args:
    bigip_url: 10.0.4.11
    bigip_partition: openshift
    log_level: DEBUG
    insecure: true
    pool_member_type: cluster
    manage_routes: false
    custom_resource_mode: true
    static_routing_mode: true
    orchestration_cni: ovn-k8s
  bigip_login_secret: bigip-login
  image:
    repo: f5networks/cntr-ingress-svcs
    user: registry.connect.redhat.com
    pullPolicy: Always
  version: 2.20.4-ubi10
  ingressClass:
    create: false
    defaultController: false
    ingressClassName: f5
  rbac:
    create: true
    namespaced: false
  serviceAccount:
    create: true

1
oc apply -f f5bigipctlr.yaml

Confirm the controller pod comes up:

1
2
oc get pods -n kube-system
oc logs deploy/cis1-f5-bigip-ctlr -n kube-system

Final thoughts

None of the individual steps here are complicated. What’s useful about going through them by hand is seeing exactly which three objects OLM needs (Namespace, OperatorGroup, Subscription) versus which two it generates for you (InstallPlan, CSV). For my purposes - matching a customer’s deployment workflow rather than just getting CIS running - that distinction matters more than the YAML itself.

If you’re doing this for real rather than as an exercise, feel free to reach out!