kea-ha: switch from hot-standby to active/active load-balancing #6

Merged

gosseljl merged 1 commit from feature/kea-ha-load-balancing into main 2026-05-10 20:11:59 +00:00

Conversation 0 Commits 1 Files changed 3 +10 -5

gosseljl commented 2026-05-10 19:33:04 +00:00

Owner

Copy link

Summary

Flips Kea's HA hook from hot-standby (one peer serves, other warm) to load-balancing (both peers serve a deterministic hash-split of the client space). Doubles steady-state DHCP throughput ceiling, halves per-peer load.

Diff

File	Change
ansible/roles/kea/defaults/main.yml	kea_ha_mode: hot-standby → load-balancing; kea_ha_max_unacked_clients: 0 → 5
ansible/roles/kea/templates/kea-dhcp4.conf.j2	Parametrise kea-02's peer role (standby → secondary) by kea_ha_mode
ansible/inventory/hosts.yml	kea-02 kea_node_role: standby → secondary (informational; template no longer reads it)

Why bump max-unacked-clients from 0 to 5

In hot-standby, 0 is fine — only the primary serves, so tripping PARTNER_DOWN on the first unacked client during a comm interruption is safe. In load-balancing, both peers are actively serving; a single packet hiccup shouldn't flip a peer to PARTNER_DOWN. ISC's recommended floor for active/active is 2–5; 5 rides out brief packet loss without state churn.

Validation done locally

• ansible-playbook playbooks/kea-render-prod.yml rendered the new conf to /tmp/kea-prod.conf on kea-01.
• kea-dhcp4 -t /tmp/kea-prod.conf parsed clean — HA hook accepts the new mode + role: secondary.
• ansible-lint passes on production profile.

Rendered HA stanza:

"mode": "load-balancing",
"max-unacked-clients": 5,
"peers": [
  { "name": "kea-01", "role": "primary",   ... },
  { "name": "kea-02", "role": "secondary", ... }
]

Trade-offs

• Split-brain visibility is slightly higher: in hot-standby a partition hides because only one peer was serving anyway; in load-balancing both stay actively serving until max-response-delay (60s) elapses on each, then transition to PARTNER_DOWN. The HA hook's COMMUNICATION_INTERRUPTED → PARTNER_DOWN machinery bounds the window. Acceptable at homelab cardinality.
• Per-peer restart blip: rolling restart bounces ~10s of DHCP per peer.

Rollout plan (post-merge)

1. CI lint + ansible-check green.
2. CI ansible-apply on push to main; the kea role's validate: kea-dhcp4 -t %s task gates the file install on successful validation. Both nodes converge sequentially.
3. The role's restart kea-dhcp4 handler bounces isc-kea-dhcp4-server on each node. Watch:
   • kea_ha_local_state flips through WAITING → SYNCING → READY → LOAD_BALANCING.
   • kea_ha_remote_in_touch returns to 1 within ~30 s.
   • Monitor task bnkm7caa0 flags any NAK / DECLINE / DROP storm in the same envelope as the per-VLAN cutover work.
4. Verify post-restart on each peer:

   curl -u $CRED -X POST http://10.0.2.232:8001/ \
     -d '{"command":"status-get","service":["dhcp4"]}' | \
     jq '.[].arguments."high-availability"[].ha-mode'
   

   Expected: "load-balancing" on both.
5. Spot-check a DORA cycle from a real client. Either peer can ACK; the HBA hash decides which.

Test plan

• kea-dhcp4 -t validates rendered conf
• ansible-lint passes on production profile
• CI lint workflow green
• CI ansible-check workflow green
• Post-merge: ansible-apply succeeds, both kea nodes restart cleanly
• kea_ha_local_state == load-balancing on both, kea_ha_remote_in_touch == 1
• Synthetic DORA probe still completes (dora_last_success_timestamp_seconds advances within 5 min)
• No KeaHAPartnerUnreachable / KeaHAClockSkewHigh firing for 30 min after cutover

🤖 Generated with Claude Code

## Summary Flips Kea's HA hook from **`hot-standby`** (one peer serves, other warm) to **`load-balancing`** (both peers serve a deterministic hash-split of the client space). Doubles steady-state DHCP throughput ceiling, halves per-peer load. ## Diff | File | Change | |---|---| | `ansible/roles/kea/defaults/main.yml` | `kea_ha_mode: hot-standby` → `load-balancing`; `kea_ha_max_unacked_clients: 0` → `5` | | `ansible/roles/kea/templates/kea-dhcp4.conf.j2` | Parametrise kea-02's peer role (`standby` → `secondary`) by `kea_ha_mode` | | `ansible/inventory/hosts.yml` | `kea-02 kea_node_role: standby` → `secondary` (informational; template no longer reads it) | ## Why bump `max-unacked-clients` from 0 to 5 In hot-standby, `0` is fine — only the primary serves, so tripping `PARTNER_DOWN` on the first unacked client during a comm interruption is safe. In load-balancing, both peers are actively serving; a single packet hiccup shouldn't flip a peer to `PARTNER_DOWN`. ISC's recommended floor for active/active is 2–5; `5` rides out brief packet loss without state churn. ## Validation done locally - `ansible-playbook playbooks/kea-render-prod.yml` rendered the new conf to `/tmp/kea-prod.conf` on kea-01. - `kea-dhcp4 -t /tmp/kea-prod.conf` parsed clean — HA hook accepts the new mode + `role: secondary`. - `ansible-lint` passes on production profile. Rendered HA stanza: ``` "mode": "load-balancing", "max-unacked-clients": 5, "peers": [ { "name": "kea-01", "role": "primary", ... }, { "name": "kea-02", "role": "secondary", ... } ] ``` ## Trade-offs - **Split-brain visibility** is slightly higher: in hot-standby a partition hides because only one peer was serving anyway; in load-balancing both stay actively serving until `max-response-delay` (60s) elapses on each, then transition to `PARTNER_DOWN`. The HA hook's `COMMUNICATION_INTERRUPTED` → `PARTNER_DOWN` machinery bounds the window. Acceptable at homelab cardinality. - **Per-peer restart blip**: rolling restart bounces ~10s of DHCP per peer. ## Rollout plan (post-merge) 1. CI lint + ansible-check green. 2. CI `ansible-apply` on push to `main`; the kea role's `validate: kea-dhcp4 -t %s` task gates the file install on successful validation. Both nodes converge sequentially. 3. The role's `restart kea-dhcp4` handler bounces `isc-kea-dhcp4-server` on each node. Watch: - `kea_ha_local_state` flips through `WAITING` → `SYNCING` → `READY` → `LOAD_BALANCING`. - `kea_ha_remote_in_touch` returns to `1` within ~30 s. - Monitor task `bnkm7caa0` flags any NAK / DECLINE / DROP storm in the same envelope as the per-VLAN cutover work. 4. Verify post-restart on each peer: ``` curl -u $CRED -X POST http://10.0.2.232:8001/ \ -d '{"command":"status-get","service":["dhcp4"]}' | \ jq '.[].arguments."high-availability"[].ha-mode' ``` Expected: `"load-balancing"` on both. 5. Spot-check a DORA cycle from a real client. Either peer can ACK; the HBA hash decides which. ## Test plan - [x] `kea-dhcp4 -t` validates rendered conf - [x] `ansible-lint` passes on production profile - [ ] CI lint workflow green - [ ] CI `ansible-check` workflow green - [ ] Post-merge: `ansible-apply` succeeds, both kea nodes restart cleanly - [ ] `kea_ha_local_state == load-balancing` on both, `kea_ha_remote_in_touch == 1` - [ ] Synthetic DORA probe still completes (`dora_last_success_timestamp_seconds` advances within 5 min) - [ ] No KeaHAPartnerUnreachable / KeaHAClockSkewHigh firing for 30 min after cutover 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gosseljl added 1 commit 2026-05-10 19:33:04 +00:00

feat(kea-ha): switch from hot-standby to active/active load-balancing ... 51a1665c7b

Changes the Kea HA hook from `hot-standby` (single peer serves, other
warm) to `load-balancing` (both peers serve, each owns a deterministic
hash bucket of the client identifier space). Doubles the steady-state
DHCP throughput ceiling and cuts per-peer load roughly in half.

Mechanical changes:
  - kea_ha_mode: hot-standby -> load-balancing
  - kea_ha_max_unacked_clients: 0 -> 5
        Reasoning: in hot-standby `0` means trip PARTNER_DOWN on the
        very first un-acked client during a comm interruption — fine
        when only one peer is serving. In load-balancing both peers
        are actively serving, so a single packet hiccup shouldn't
        flip the surviving peer to PARTNER_DOWN. Bump to 5 per ISC
        guidance for active/active deployments.
  - kea-dhcp4.conf.j2: parameterise the second peer's role
        ("standby" in hot-standby, "secondary" in load-balancing).
  - inventory/hosts.yml: kea-02 kea_node_role -> secondary
        (informational; the template no longer reads this var, but
        keep the inventory metadata consistent.)

Validation:
  - `ansible-playbook playbooks/kea-render-prod.yml` renders the new
    conf to /tmp/kea-prod.conf on kea-01.
  - `kea-dhcp4 -t /tmp/kea-prod.conf` validates clean — HA hook
    accepts the load-balancing mode + secondary peer role.
  - ansible-lint passes on production profile.

Rollout:
  1. Merge.
  2. Watch CI ansible-apply on push to main; the kea role's
     `validate: kea-dhcp4 -t %s` task gates the file install on
     successful validation. Both nodes converge in series via
     ansible's default linear strategy.
  3. The role's restart handler bounces isc-kea-dhcp4-server. On
     each restart watch:
       - `kea_ha_local_state` flips through WAITING -> SYNCING ->
         READY -> LOAD_BALANCING.
       - `kea_ha_remote_in_touch` should return to 1 within ~30s.
       - Monitor task (bnkm7caa0) will alert on any anomalies in
         the same envelope as the per-VLAN cutover work.
  4. Verify post-restart from each peer:
       curl -u $CRED -X POST http://10.0.2.232:8001/ \
         -d '{"command":"status-get","service":["dhcp4"]}' | \
         jq '.[].arguments."high-availability"[].ha-mode'
     Expected: "load-balancing" on both.

Trade-offs documented in conversation: split-brain visibility is
slightly higher (both peers actively serving means a network
partition has both staying live until max-response-delay elapses
on each), bounded by the HA hook's COMMUNICATION_INTERRUPTED ->
PARTNER_DOWN semantics. Acceptable at homelab cardinality.

gosseljl force-pushed feature/kea-ha-load-balancing from 51a1665c7b to 87c0c278aa 2026-05-10 19:38:32 +00:00 Compare

gosseljl merged commit daaab473b0 into main 2026-05-10 20:11:59 +00:00

gosseljl referenced this pull request from a commit 2026-05-10 20:12:00 +00:00

Merge pull request 'kea-ha: switch from hot-standby to active/active load-balancing' (#6) from feature/kea-ha-load-balancing into main

Commenting is not possible because the repository is archived.

Reviewers

No reviewers

Labels

Clear labels   

drift

Auto-filed by drift-detect.yml

No labels

drift

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

gosseljl/homelab-ddi!6

No description provided.