Testkit¶

zio-openfeature-testkit gives you TestFeatureProvider: an in-memory OpenFeature provider you control from your tests. Set flags, flip the provider's status, emit events, inject failures, and check which flags your code read. It runs the same evaluation path as production — hooks, context, transactions and error handling included.

libraryDependencies += "io.github.etacassiopeia" %% "zio-openfeature-testkit" % "<version>" % Test

Your first test¶

Say your service reads a flag:

// The code under test: a greeting gated by a flag
trait Greeter:
  def greet(user: String): IO[FeatureFlagError, String]

object Greeter:
  val live: URLayer[FeatureFlags, Greeter] =
    ZLayer.fromFunction((flags: FeatureFlags) =>
      new Greeter:
        def greet(user: String): IO[FeatureFlagError, String] =
          flags
            .boolean("new-greeting", default = false, EvaluationContext(user))
            .map(on => if on then s"Hey $user!" else s"Welcome, $user.")
    )

Provide TestFeatureProvider.layer in place of the real FeatureFlags layer. It provides both FeatureFlags and the TestFeatureProvider itself, so a test can set flags and then run the code under test:

val greeterSuite = suite("Greeter")(
  test("old greeting when the flag is off") {
    for
      provider <- ZIO.service[TestFeatureProvider]
      _        <- provider.setFlag("new-greeting", false)
      greeting <- ZIO.serviceWithZIO[Greeter](_.greet("alice"))
    yield assertTrue(greeting == "Welcome, alice.")
  },
  test("new greeting when the flag is on") {
    for
      provider <- ZIO.service[TestFeatureProvider]
      _        <- provider.setFlag("new-greeting", true)
      greeting <- ZIO.serviceWithZIO[Greeter](_.greet("bob"))
    yield assertTrue(greeting == "Hey bob!")
  }
).provide(TestFeatureProvider.layer, Greeter.live)

Each test gets its own provider and its own isolated OpenFeature API instance, so tests can run in parallel.

Choosing a layer¶

Every layer provides TestFeatureProvider & FeatureFlags.

Layer	Provider starts	Use it when
`layer`, `layer(flags)`	`Ready`	most tests
`asyncLayer`, `asyncLayer(flags)`	`NotReady`, until you call `setStatus(Ready)`	testing how your code behaves while the provider starts up
`asyncReadyLayer(flags, initDelay)`	`NotReady`, then `Ready` after `initDelay` (default 100 ms)	a provider that becomes ready on its own
`layerFrom(provider)`	as the provider is	you built the provider yourself with `make` / `makeNamed`

None of these needs a Scope: each layer owns its own and shuts the provider down when it is released. layer and layerFrom return only after the OpenFeature SDK has delivered its initial ready event, so a setStatus straight after the layer builds is not overwritten by that late event. scopedLayer and scopedAsyncLayer are deprecated aliases of layer and asyncLayer from when layers did need one.

flags is a Map[String, Any], or typed overrides (Flag := value, see below). providerLayer provides just a TestFeatureProvider, without FeatureFlags.

To build a provider without a layer, use TestFeatureProvider.make(flags), or makeNamed(name, flags) to give it its own metadata name. A name matters when you swap providers with FeatureFlags.setProvider, which tells the old and new provider apart by name. Every other factory names the provider TestFeatureProvider.DefaultName.

Typed fixtures¶

A fixture keyed by string accepts any value, so a typo such as "premum" passes the test and fails in production. Build the fixture from your FlagDefs instead, and the compiler checks the value's type:

enum Tier derives FlagType:
  case Free, Premium

object Flags:
  val NewGreeting = FlagDef("new-greeting", false)
  val Plan        = FlagDef[Tier]("user-plan", Tier.Free) // name the type: see the note below

val typedFixture = test("typed fixture") {
  for plan <- FeatureFlags.value(Flags.Plan)
  yield assertTrue(plan == Tier.Premium)
}.provide(TestFeatureProvider.layer(Flags.Plan := Tier.Premium, Flags.NewGreeting := true))
// Flags.Plan := "premium" does not compile

:= stores the value the way a provider would carry it (flagType.encode(value)), so the test reads it back through the same decoding as production. It also checks that the codec can decode its own encoding, and throws at fixture construction if it cannot.

Every factory that takes flags (make, layer, layer, asyncLayer, asyncLayer, providerLayer) also takes FlagOverride*. For asyncReadyLayer the delay comes first: asyncReadyLayer(100.millis, Flag := v).

Note

Name the type parameter when the default is a case object: FlagDef("k", Tier.Free) infers Tier.Free.type and finds no FlagType for it. Write FlagDef[Tier]("k", Tier.Free).

Changing flags during a test¶

val managingFlags = test("changing flags during a test") {
  for
    provider <- ZIO.service[TestFeatureProvider]
    _        <- provider.setFlags(Map("a" -> true, "b" -> "blue")) // merges into what is there
    _        <- provider.setFlag(Flags.Plan, Tier.Premium)         // typed, by definition
    _        <- provider.removeFlag("b")                           // "b" is now absent
    _        <- provider.replaceFlags(Map("c" -> 3))               // drops everything else first
    c        <- FeatureFlags.int("c", 0)
    a        <- FeatureFlags.booleanOrDefault("a", default = false)
  yield assertTrue(c == 3, !a)
}.provide(TestFeatureProvider.layer(Map("a" -> false)))

Method	Effect
`setFlag(key, value)`, `setFlag(flagDef, value)`	set one flag
`setFlags(map)`, `setFlags(overrides*)`	set several, keeping the others
`replaceFlags(map)`, `replaceFlags(overrides*)`	drop every flag, then set these
`removeFlag(key)`, `removeFlag(flagDef)`	remove one flag
`clearFlags`	remove every flag

Unset flags¶

A key the provider does not hold resolves as FLAG_NOT_FOUND, as it does with real providers:

val absentKeys = test("an unset key is FLAG_NOT_FOUND") {
  for
    typed <- FeatureFlags.boolean("never-set", default = false).either
    total <- FeatureFlags.resolveOrDefault[Boolean]("never-set", default = false)
  yield assertTrue(
    typed == Left(FeatureFlagError.FlagNotFound("never-set")), // the typed tier fails
    total.value == false,                                      // the total tier serves the default...
    total.reason == ResolutionReason.Error,                    // ...and says why
    total.errorCode.contains(ErrorCode.FlagNotFound)
  )
}.provide(TestFeatureProvider.layer)

Hooks see the error stage for it, not after. So when a test means "this flag is off", set it to false instead of leaving it unset.

Reporting FLAG_NOT_FOUND is also what lets a test provider sit in a provider chain.

A value of the wrong type is an error too, reported as TYPE_MISMATCH as a real provider reports it. The typed tier fails with TypeMismatch and the total tier serves your default; neither silently returns the default as if the flag were fine:

val wrongType = test("a value of the wrong type is a TYPE_MISMATCH, not the default") {
  for
    typed <- FeatureFlags.boolean("flag", default = false).either
    total <- FeatureFlags.resolveOrDefault[Boolean]("flag", default = false)
  yield assertTrue(
    typed.left.exists(_.isInstanceOf[FeatureFlagError.TypeMismatch]),
    total.errorCode.contains(ErrorCode.TypeMismatch),
    total.value == false
  )
}.provide(TestFeatureProvider.layer(Map("flag" -> "not a boolean")))

A key-based value is served only when its type is exactly the one read: Boolean, String, Int, Long or Double. Nothing is converted. A Long read as int is a mismatch even when it fits, and so is an Int read as long or double, a string read as a number, or any value read as string. Store 42L for a long flag and 2.0 for a double one. Typed fixtures (setFlag(flagDef, v), Flag := v) always store the right type.

Checking which flags were read¶

The provider records every evaluation that reaches it, with the context it received. That includes evaluations a simulated failure made fail: the provider was asked, even though it did not answer.

val recorded = test("which flags were read, how often, with what context") {
  for
    provider <- ZIO.service[TestFeatureProvider]
    _        <- ZIO.serviceWithZIO[Greeter](_.greet("alice"))
    _        <- ZIO.serviceWithZIO[Greeter](_.greet("bob"))
    read     <- provider.wasEvaluated(Flags.NewGreeting)
    count    <- provider.evaluationCount("new-greeting")
    evals    <- provider.getEvaluations // List[(String, EvaluationContext)]
    _        <- provider.clearEvaluations
    after    <- provider.evaluationCount("new-greeting")
  yield assertTrue(
    read,
    count == 2,
    evals.map(_._2.targetingKey) == List(Some("alice"), Some("bob")),
    after == 0
  )
}.provide(TestFeatureProvider.layer(Map("new-greeting" -> true)), Greeter.live)

Method	Returns
`wasEvaluated(key)`, `wasEvaluated(flagDef)`	whether the flag was read
`evaluationCount(key)`, `evaluationCount(flagDef)`	how many times
`getEvaluations`	every `(key, EvaluationContext)`, oldest first
`getRawEvaluations`	the same, with the Java SDK's `EvaluationContext`
`clearEvaluations`	forgets them
`trackedEvents`	every `track` call received, oldest first (the last 1000): `(name, EvaluationContext, Option[TrackingEventDetails])`, with the context the SDK merged
`clearTrackedEvents`	forgets them
`boundDomain`	the domain the provider was registered under, if any

Evaluations answered without the provider, by a transaction override or cache, are not recorded.

Provider status¶

setStatus changes what the provider reports. While it is NotReady or Fatal, evaluations fail fast:

val notReady = test("fails fast while the provider is not ready") {
  for
    provider <- ZIO.service[TestFeatureProvider]
    _        <- provider.setStatus(ProviderStatus.NotReady)
    down     <- ZIO.serviceWithZIO[Greeter](_.greet("carol")).either
    _        <- provider.setStatus(ProviderStatus.Ready)
    up       <- ZIO.serviceWithZIO[Greeter](_.greet("carol"))
  yield assertTrue(
    down == Left(FeatureFlagError.ProviderNotReady(ProviderStatus.NotReady)),
    up == "Hey carol!"
  )
}.provide(TestFeatureProvider.layer(Map("new-greeting" -> true)), Greeter.live)

getStatus (or status) reads the current status.

Startup¶

asyncLayer builds at once, with the provider still NotReady, as FeatureFlags.fromProviderAsync does with a slow provider. Evaluations fail with ProviderNotReady until you call setStatus(ProviderStatus.Ready):

val asyncInit = test("a provider that is still starting up") {
  for
    provider <- ZIO.service[TestFeatureProvider]
    before   <- FeatureFlags.boolean("feature", default = false).either
    _        <- provider.setStatus(ProviderStatus.Ready) // simulate the provider finishing init
    after    <- FeatureFlags.boolean("feature", default = false)
  yield assertTrue(before.isLeft, after)
}.provide(TestFeatureProvider.asyncLayer(Map("feature" -> true)))

asyncReadyLayer becomes ready on its own after initDelay. The delay runs on the test clock, so advance it:

val asyncReady = test("a provider that becomes ready on its own") {
  for
    _      <- TestClock.adjust(100.millis) // the layer's delay runs on the test clock
    status <- FeatureFlags.awaitReady()
    on     <- FeatureFlags.boolean("feature", default = false)
  yield assertTrue(status == ProviderStatus.Ready, on)
}.provide(TestFeatureProvider.asyncReadyLayer(Map("feature" -> true), initDelay = 100.millis))

Events¶

emitEvent sends a provider event through the OpenFeature SDK, so it reaches FeatureFlags.events and the on* handlers as a real provider's event would:

val events = test("emit a provider event") {
  for
    provider <- ZIO.service[TestFeatureProvider]
    changed  <- Promise.make[Nothing, Set[String]]
    _        <- FeatureFlags.onConfigurationChanged((keys, _) => changed.succeed(keys).unit)
    _        <- provider.emitEvent(ProviderEvent.ConfigurationChanged(Set("banner"), provider.metadata))
    keys     <- changed.await
  yield assertTrue(keys == Set("banner"))
}.provide(TestFeatureProvider.layer)

Use ProviderEvent.Ready, Error, Stale or ConfigurationChanged, with provider.metadata as the provider's metadata. Reconnecting has no OpenFeature SDK equivalent, so it does not reach FeatureFlags. The provider's own events stream sees every event you emit.

Simulating failures¶

val failures = suite("when the provider misbehaves")(
  test("every evaluation fails with a chosen error") {
    for
      provider <- ZIO.service[TestFeatureProvider]
      _        <- provider.setErrorMode(TestFeatureProvider.ErrorMode.ParseError)
      typed    <- FeatureFlags.boolean("flag", default = false).either
      total    <- FeatureFlags.resolveOrDefault[Boolean]("flag", default = false)
    yield assertTrue(typed.isLeft, total.errorCode.contains(ErrorCode.ParseError))
  },
  test("the same, as an aspect that resets after the test") {
    for resolution <- FeatureFlags.resolveOrDefault[Boolean]("flag", default = true)
    yield assertTrue(resolution.value, resolution.errorCode.contains(ErrorCode.General))
  } @@ TestFeatureProvider.withFailures
).provide(TestFeatureProvider.layer(Map("flag" -> false))) @@ TestAspect.sequential

Control	Effect
`setErrorMode(mode)`	every evaluation fails with `mode`: `FlagNotFound`, `ParseError`, `TypeMismatch`, `ProviderNotReady` or `General`
`setFailing(true)`	same as `setErrorMode(ErrorMode.General)`; `setFailing(false)` clears it
`setFailureProbability(p)`	each evaluation fails with probability `p` (clamped to 0–1)
`setDelay(d)`	each evaluation sleeps `d` first, on the real clock
`clearErrorMode`, `clearDelay`, `clearBehavior`	undo the above

Each control has a test aspect that sets it before the test and, after it, restores the settings it found: TestFeatureProvider.withErrorMode(mode), withFailures, withFailureProbability(p) (clamped like setFailureProbability) and withDelay(d). Because each restores what it found rather than the defaults, aspects nest: a test-level withDelay inside a suite-level withErrorMode keeps the error mode. They need TestFeatureProvider in the environment.

An error mode applies to every key. To make one key missing, leave it unset.

Fault injection¶

The controls above make evaluations fail. To test what your code does when the provider itself misbehaves, wrap it in a FaultyProvider: an initialize() that never returns, a shutdown() that hangs, an error event fired while starting up, or a failure on the first call only.

val faultInjection = suite("when the provider itself misbehaves")(
  test("an initialize() that never returns fails the layer at the init timeout") {
    for
      faulty <- FaultyProvider.scoped(Map("flag" -> true)) // the scope releases any hang left parked
      _      <- faulty.script(Op.Initialize, Fault.HangUntilReleased) // ignores shutdown(), like a stuck SDK
      error  <- FeatureFlags.fromProvider(faulty, FeatureFlagsConfig().withInitTimeout(200.millis)).build.flip
    yield assertTrue(error.isInstanceOf[java.util.concurrent.TimeoutException])
  },
  test("the first evaluation fails, the ones after it succeed") {
    for
      faulty <- FaultyProvider.scoped(Map("flag" -> true))
      _      <- faulty.script(Op.EvaluateBoolean, Fault.Throw(new RuntimeException("timeout")), Fault.Succeed)
      both <- (FeatureFlags.boolean("flag", default = false).either <*> FeatureFlags.boolean("flag", default = false))
        .provide(FeatureFlags.fromProvider(faulty))
      calls <- faulty.calls(Op.EvaluateBoolean)
    yield assertTrue(both._1.isLeft, both._2, calls.size == 2)
  }
) @@ TestAspect.withLiveClock

FaultyProvider.script(op, faults*) gives one Op a sequence of faults, used one per call. The last one repeats, so script(Op.Initialize, Fault.Throw(e), Fault.Succeed) fails the first initialize() and passes every later one. An op with no script calls the wrapped provider.

`Fault`	Effect
`Succeed`	call the wrapped provider
`Throw(error)`	throw `error` without calling it
`Delay(d, andThen)`	block the calling thread for `d`, then apply `andThen` (default `Succeed`)
`HangUntilReleased`	block until `release`; `shutdown()` does not unblock it
`HangUntilShutdown`	block until `shutdown()` or `release`
`Emit(event, details, andThen)`	emit an OpenFeature event (e.g. `PROVIDER_ERROR` from inside `initialize()`), then apply `andThen`

There is an Op for every FeatureProvider method: Initialize (both overloads), Shutdown, GetMetadata, GetProviderHooks, GetState, IsDomainScoped, Track, and one per getter, EvaluateBoolean to EvaluateObject.

Factory	Wraps
`FaultyProvider.scoped(flags, name)`	a fresh `TestFeatureProvider`; the scope releases every hang still parked when it closes
`FaultyProvider.make(flags, name)`	the same, without the finalizer
`FaultyProvider.wrapScoped(provider, name)`, `wrap(provider, name)`	any provider

Prefer the scoped factories: a test that fails while a call is parked then still frees the thread. delegate is the wrapped provider (for make, the TestFeatureProvider, so setFlag works). calls lists every completed call with its op, flag key, thread, start time, outcome and arguments; a hung call appears once it is released. awaitEntered(op) waits until a call has started, for example until initialize() is parked.

The wrapper emits its own events, so the wrapped provider's events do not reach FeatureFlags: use Fault.Emit or the wrapper's emit.

Per-test flag overrides¶

To pin flags for one test, or for every test in a suite, use a flag aspect. It needs TestFeatureProvider in the environment:

val Greeting = FlagDef[Boolean]("new-greeting", false)

val perTestFlags = suite("Greeter with per-test flags")(
  test("new greeting while the aspect pins the flag on") {
    for msg <- ZIO.serviceWithZIO[Greeter](_.greet("dave"))
    yield assertTrue(msg == "Hey dave!")
  } @@ TestFeatureProvider.withFlag(Greeting, true),
  test("the layer's value is back for the next test") {
    for msg <- ZIO.serviceWithZIO[Greeter](_.greet("dave"))
    yield assertTrue(msg == "Welcome, dave.")
  }
).provideShared(TestFeatureProvider.layer(Greeting := false), Greeter.live) @@ TestAspect.sequential

Aspect	Pins
`withFlags(overrides*)`	typed overrides, `Flag := value`, as `layer` takes them
`withFlags(map)`	key/value pairs, as `layer(map)` takes them
`withFlag(flag, value)`	one typed flag
`withFlag(key, value)`	one key

Each aspect restores per key. Before the test it saves the current value of every key it pins, or notes that the key was absent; after the test it puts that value back, or removes the key. So keys it does not pin are left alone: a setFlag("other", ...) inside the test survives. Aspects nest: a test-level aspect inside a suite-level one on the same key gives the test the inner value and leaves the suite's value afterwards.

Unlike a transaction override, the value comes from the provider, so wasEvaluated, the error modes and provider chains all apply.

Warning

With provideShared, tests share one provider, so flag aspects on tests that run at the same time race each other. Add @@ TestAspect.sequential to the suite, as above, or use provide to give each test its own provider.

Recording hook calls¶

RecordingHook is a hook that records every stage call it receives, in order. Install it like any other hook, with FeatureFlags.addHook, FeatureFlagsConfig.withHook or per call through EvaluationOptions. It works with any FeatureFlags, not only the testkit layers:

val recordedHooks = test("the hook pipeline ran before, after, finally") {
  for
    recorder <- RecordingHook.make()
    _        <- FeatureFlags.addHook(recorder)
    _        <- ZIO.serviceWithZIO[Greeter](_.greet("erin"))
    calls    <- recorder.calls
    resolved = calls.collect { case HookCall.After(_, resolution, _) => resolution.value }
  yield assertTrue(
    calls.size == 3, // Before, After, Finally
    resolved == Chunk[Any](true),
    calls.forall(_.ctx.flagKey == "new-greeting")
  )
}.provide(TestFeatureProvider.layer(Map("new-greeting" -> true)), Greeter.live)

calls returns a Chunk[HookCall], oldest first, and clear empties it. Each HookCall carries the full arguments of its stage, including the HookContext (flag key, wire type, default value, context, metadata) and the hints:

Case	Stage	Also carries
`HookCall.Before`	`before`
`HookCall.After`	`after`, on a clean resolution	`resolution`
`HookCall.Error`	`error`, on a failure or an error-coded resolution such as `FLAG_NOT_FOUND`	`error`
`HookCall.Finally`	`finallyAfter`, last on every exit	`resolution: Option[...]`

RecordingHook.make(types) records only evaluations of those flag types; by default, all of them. Its before always returns None, so it never changes the evaluation context.

Transactions¶

Transactions work on top of the test provider, so you can override a flag for one block:

val withOverrides = test("override flags for one block") {
  for
    base <- FeatureFlags.boolean("new-greeting", default = false)
    tx <- FeatureFlags.transaction(overrides = Map("new-greeting" -> true)) {
      FeatureFlags.boolean("new-greeting", default = false)
    }
  yield assertTrue(!base, tx.result)
}.provide(TestFeatureProvider.layer(Map("new-greeting" -> false)))

Provider chains¶

A test provider reports FLAG_NOT_FOUND for keys it does not hold, so a MultiProvider chain moves on to the next provider:

val chain = test("a test provider in a provider chain") {
  for
    primary  <- TestFeatureProvider.makeNamed("primary") // has no "flag"
    fallback <- TestFeatureProvider.makeNamed("fallback", Map("flag" -> true))
    chained = FeatureFlags.multiProvider(List(primary, fallback))
    value <- FeatureFlags
      .boolean("flag", default = false)
      .provide(FeatureFlags.fromProvider(chained))
    asked <- primary.wasEvaluated("flag")
  yield assertTrue(value, asked) // the primary was asked, said FLAG_NOT_FOUND, and the chain moved on
}

Warning

FeatureFlags.fromProvider and the other public factories use the global OpenFeature API, so tests that use them directly must not run in parallel. Add @@ TestAspect.sequential. The testkit's own layers are isolated and need nothing.

Checking a custom `FlagType`¶

If you write a FlagType by hand, check it with FlagTypeLaws. Give it a Gen for your type:

final case class Celsius(degrees: Int)
object Celsius:
  given FlagType[Celsius] = FlagType.mapped[Celsius, Int]("Celsius", Celsius(0))(Celsius(_), _.degrees)

val celsiusLaws = FlagTypeLaws.all(Gen.int(-100, 100).map(Celsius(_)))

celsiusLaws is a Spec; put it in a suite. all runs two laws:

Law	Checks
`roundTrip`	`decode(encode(a)) == Right(a)`
`throughValueBridge`	the same, after a trip through the OpenFeature SDK's `Value` type

The second one finds the real problems: the SDK turns every number into a Double, so a Long above 2^53 does not survive, and it drops structure members it cannot represent. It matters most for object-backed types, which always take that path. Both laws work on Scala 2.13 too.

Checking your own wrapper¶

A provider that wraps another one (for caching, metrics, retries) must pass every call on. The OpenFeature SDK adds methods to FeatureProvider with default bodies, so a wrapper written before one of them keeps compiling and answers it from the default instead of asking the provider it wraps: track events vanish, or a Long evaluation goes through getDoubleEvaluation. WrapperLaws.delegation catches this. Give it a name and a function that builds your wrapper around a provider:

val countingDelegates = WrapperLaws.delegation("CountingProvider")(
  inner => new CountingProvider(inner),
  handledByWrapper = Map(Op.GetMetadata -> "reports its own name")
)

The result is a suite with one test per Op. Each test builds your wrapper around a fresh FaultyProvider, calls one method with distinctive arguments, and checks the wrapped provider received the same call with equal arguments (contexts and tracking details are compared by content). For Initialize it checks both overloads, so a wrapper that drops the domain from initialize(ctx, domain) fails. The law does not look at the results.

handledByWrapper lists the ops your wrapper answers itself, each with a reason that becomes part of the test name. For those, the test checks the call does not reach the wrapped provider, and fails with "allow-list entry no longer needed" if it does, so the list cannot go stale. The law runs on Scala 2.13 and 3. The library's own wrappers (CachingProvider, CircuitBreakerProvider, DeferredProvider, IntegerWideningLongProvider, CachingReasonProvider) are checked with it.

Comparing a provider against the testkit¶

A test is only as good as the test provider's answers. DifferentialLaws checks them against a real provider. It generates flag sets over every wire type, puts each into TestFeatureProvider and into your provider, reads every key (and one absent key) through every typed accessor, and compares what FeatureFlags returns: the value, the reason class (Served, Default or Error) and the error code.

Some differences are intended. List each one as a Deviation with its reason; any other difference fails the test. Here an environment variable holds text, so a read of another type parses that text:

def environment(flags: FlagSet): Map[String, String] =
  flags.flags.map((key, value) => s"FF_${EnvVarProvider.defaultKeyTransform(key)}" -> value.toString)

def mismatchedInTestkit(testkit: Observed) = testkit.errorCode.contains(ErrorCode.TypeMismatch)

val envVarAgrees = DifferentialLaws.agreesWithTestkit("env-var")(
  seed = flags => ZIO.succeed(EnvVarProvider.withLookup(environment(flags).get)),
  deviations = List(
    Deviation(
      "env-var",
      (_, _, testkit, env) => mismatchedInTestkit(testkit) && env.reasonClass == ReasonClass.Served,
      "an env var is text: a read of another type is served when the text parses"
    ),
    Deviation(
      "env-var",
      (_, _, testkit, env) => mismatchedInTestkit(testkit) && env.errorCode.contains(ErrorCode.ParseError),
      "an env var is text: text that does not parse is PARSE_ERROR, not TYPE_MISMATCH"
    )
  ),
  flagSets = DifferentialLaws.scalarFlagSets, // an env var cannot hold a long or an object
  corpus = DifferentialLaws.scalarCorpus
)

seed builds your provider holding the generated flags.
A deviation that matched nothing in the run also fails the test, as stale. The list cannot outlive the behaviour it excuses.
Each run checks 50 random flag sets (samples) plus a fixed corpus of edge cases: NaN, -0.0, Longs beyond the Int range, text that looks like a number. zio-test seeds the randomness, so a failure is reproducible.
flagSets and corpus default to every wire type. scalarFlagSets and scalarCorpus keep to booleans, strings, ints and doubles, for a provider that holds nothing else.
compare takes the same arguments and returns the Report instead of a test.

This repository runs it for HoconProvider, EnvVarProvider and the OFREP provider (against an in-process OFREP stub). The HOCON run found that the provider truncated 1.5 to 1 for an int read.

`CachingReasonProvider`¶

CachingReasonProvider wraps any OpenFeature provider and reports the CACHED reason from the second evaluation of a key on. It always calls the wrapped provider; only the reason changes. This repository uses it to run the OpenFeature spec's CACHED scenario. It does not cache values: for that, see CachingProvider.

val cachingReason = test("CachingReasonProvider reports CACHED on a repeat") {
  for
    provider <- TestFeatureProvider.make(Map("flag" -> true))
    layer = FeatureFlags.fromProvider(new CachingReasonProvider(provider))
    reasons <- (FeatureFlags.booleanDetails("flag", false) *> FeatureFlags.booleanDetails("flag", false))
      .map(_.reason)
      .provide(layer)
  yield assertTrue(reasons == ResolutionReason.Cached)
}

Scenario matrices with zio-bdd¶

To run one scenario against several flag configurations, the zio-bdd framework can build a FeatureFlags layer per @flags(...) tag. Override flagLayer in your ZIOSteps suite. Tag values arrive as strings, so turn them into flag values of the right type:

override def flagLayer(meta: ScenarioMetadata, flags: Map[String, String]): ZLayer[Any, Throwable, FeatureFlags] =
  TestFeatureProvider.layer(Map("new-greeting" -> flags.get("greeting").contains("new")))

@flags(greeting=new, plan=pro) runs the scenario once, with both keys in flags. Two @flags(...) tags on separate lines run it twice, once per tag. The hook also has coarser siblings: scenarioLayer, featureLayer and applicationLayer. This repository's conformance-zio-bdd module has working matrix suites (FlagMatrixSpec) that drive the Optimizely provider this way.

To test a real provider's network code rather than its states, see Testing real providers.