Kong Gateway in Production: Failure Clinic (Course #kong-clinic)
For platform engineers, SREs, API operations leads.
By the end, you can
By the end of this course, you will be able to operate Kong Gateway rate limiting on measured behaviour — a strategy decision matrix, a partition-tested blast-radius figure and a plugin-order audit — so that a counter-store outage produces a known multiplier rather than an unbounded one.
Running case study
Vantry Payments fronts card authorisation and merchant onboarding APIs with Kong Gateway in hybrid mode across 6 data-plane nodes in 2 regions, contracted at 600 req/min per merchant. During a 4-minute Redis incident, merchants sustained roughly 6× their contracted rate without a single 429, and the downstream authoriser shed load instead. Risk now requires the limiter's behaviour under counter-store loss to be a measured number in the runbook, not a configuration screenshot.
Programme every line is something you do
4 modules · 12 topics · in course-flow order
-
Module 1
The limiter fails open
Objective. Measure what Rate Limiting Advanced enforces when a node loses its shared counter store, and record the multiplier for your topology.
- Reproduce the documented fallback: on losing Redis, the plugin falls back to local rate limiting on each node
- Compute the effective ceiling for a 6-node data plane — a 600 req/min limit becomes up to 3,600 req/min during the outage
- Instrument the fallback so the transition is alertable within 60 seconds instead of discovered from merchant complaints
-
Module 2
Which strategy is even legal
Objective. Select a counting strategy against deployment-mode constraints rather than against accuracy preferences alone.
- Rule out the cluster strategy where it is unsupported — hybrid mode and Konnect deployments both exclude it
- Weigh accuracy against cost: a sync_rate of 0.1 is 10 counter syncs per second, and very high sync frequencies with cluster mode are not recommended
- Account for scale-down behaviour, where the probability of false negatives increases as the cluster shrinks
-
Module 3
Order changes under your feet
Objective. Audit the plugin execution chain and detect priority changes that silently reorder policy between releases.
- Reconstruct the actual execution order of the policy chain rather than the order the configuration is written in
- Trace the consequence of a priority change: Service Protection moved from 915 to 901 and now runs after other rate-limiting plugins
- Pin the expected order in a test that fails on upgrade, covering all plugins on the 2 highest-traffic routes
-
Module 4
Prove it under partition
Objective. Run the partition drill end to end and produce the runbook entry with measured numbers.
- Cut the counter store for 4 minutes under production-shaped load and record requests admitted against the contracted 600 req/min
- Capture the evidence a post-incident review needs: admitted rate, node count, fallback start and recovery time
- Write the runbook entry as measured behaviour with its date, since strategy support and plugin priorities both change between releases
References & Official Documentation every claim above has a source
Module 1
- RFC 6585 — Additional HTTP Status CodesSpecification
- Gateway rate limitingMuleSoft Docs
- Rate Limiting Advanced pluginMuleSoft Docs
Module 2
Module 3
- Kong plugin execution and priorityMuleSoft Docs
- Plugin entityMuleSoft Docs
Module 4
- Kong Gateway monitoringMuleSoft Docs