Monitoring Changes In Firebase Remote Config Using Kotlin, Slack, and Google Cloud Functions

by Nate Ebel, Android Engineer

Our mobile team here at Premise uses Firebase for a number of things, the most heavily used feature being Firebase Remote Config. We use Remote Config to control feature flags. We also use it in our internationalization efforts as well as in conjunction with A/B Testing to gather insights into user behavior and application performance.

With this usage comes a fair bit of operational overhead.

As the number of Remote Config parameters and conditions increases, it becomes more difficult to keep track of individual values. This is exacerbated by increasing team sizes and more regular release cadences.

To help with this, we’ve been using Google Cloud Functions to monitor for changes in Remote Config values and to send notifications of those changes to Slack. This system has worked well, but over time, we’ve identified some potential improvements.

We recently built out a v2 of this Remote Config notification system, which included the following goals:

• Build the new Cloud Function in Kotlin so it’s easier for the Mobile team to maintain
• Notify on changes to Remote Config parameters
• Notify on changes to grouped Remote Config parameters
• Notify on changes to Remote Config conditions
• Clearly highlight diffs between parameter/condition versions

In this post, we’ll walk through the system design, project setup, code, and basic deployment that we’ve used to monitor Remote Config changes in our DEV, QA, and PROD environments. You should be able to easily adapt this sample code to your team’s needs or even deploy it to your Google Cloud project as is and start monitoring Remote Config changes immediately.

In the end, you should end up with something similar to this:

If you want to skip right to the code, you can check out the sample project on GitHub.

mobile-tutorials/002-remote-config-monitor-cloud-function at main · premisedata/mobile-tutorials

Designing the Remote Config Notification System

The end-to-end flow of our notification system has several steps:

• Changes to Firebase Remote Config trigger the remoteconfig.update event within GCP.
• That event triggers our monitoring Cloud Function.
• The Cloud Function parses the updated Remote Configversion from the event payload.
• The previous, and new, Remote Config templates (sets of values) are loaded using the Firebase Admin SDK.
• Those changes are compared to identify additions, removals, updates, etc.
• Changes are sent to Slack using the Slack SDK.

System diagram for our Firebase Remote Config monitoring service

This workflow could be customized to suit your team’s needs in several ways:

• If you don’t care to see diffs in your Slack messages, then you could skip the use of the Firebase Admin SDK, and simply notify Slack of the updated version number any time the Cloud Function is run.
• The comparison of new and old templates could be simplified, or made more complex, depending on specific output needs.
• The notification could be sent to any other service, email, etc. It would be straightforward to send to multiple destinations as well.

Creating the Kotlin Cloud Function Project

We’re going to use IntelliJ to create a new Kotlin project. Cloud Functions supports the Java 11 runtime, and we can include the required dependencies using either Maven or Gradle. We’re using Maven as the setup is a bit simpler and most of the Google docs use Maven as well.

Use IntelliJ to create a new Kotlin project

Once we’ve created our project template, we need to add a dependency on the Cloud Functions API so we can define our entry point class.

// pom.xml
<dependencies>
    <dependency>
        <groupId>com.google.cloud.functions</groupId>
        <artifactId>functions-framework-api</artifactId>
        <version>1.0.1</version>
    </dependency>
    ...
</dependencies>

Google Cloud Functions may be triggered in response to HTTP requests and/or some GCP event coming from a Pub/Sub topic, a Cloud Storage change, or a variety of other events such as changes to Firebase Remote Config.

In our case, we only need to respond to the remoteconfig.update event, so we will implement one of two available interfaces when defining our Cloud Function:

• RawBackgroundFunction
• BackgroundFunction<T>

Both interfaces provide an accept() method that we may override to defined the behavior of our Cloud Function. In the case of RawBackgroundFunction, the event payload will be returned as raw JSON, whereas the BackgroundFunction interface will return to us a typed object of our choosing.

We will use RawBackgroundFunction as it gives us more flexibility in how we log and parse the incoming event.

We’ll create a RemoteConfigUpdateEvent data class to hold the remoteconfig.update event payload.

import com.google.gson.annotations.SerializedName
data class RemoteConfigUpdateEvent(
    @SerializedName("updateType")
    val type: String,
    @SerializedName("updateOrigin")
    val origin: String,
    @SerializedName("versionNumber")
    val version: Long,
    @SerializedName("updateUser")
    val user: User
) {
    data class User(val email: String)
}

With this event, we can now define our Cloud Function.

class RemoteConfigMonitorService
    : RawBackgroundFunction> {
    override fun accept(
        json: String,
        context: Context
    ) {
        TODO("Not yet implemented")
    }
}

Now, we can add some basic logging to help identify changes in our Cloud Function logs. This also helps familiarize ourselves with the structure of the event payload before we start using it for more complex things.

class RemoteConfigMonitorService
    : RawBackgroundFunction {

    private val logger: Logger = Logger.getLogger(RemoteConfigMonitorService::class.java.name)
    private val gson: Gson = Gson()
    override fun accept(
        json: String,
        context: Context
    ) {
        val event: RemoteConfigUpdateEvent = 
            gson.fromJson(json, RemoteConfigUpdateEvent::class.java)        
        logger.info("Remote Config updated to version ${event.version} by ${event.user.email}")
    }
}

With our function in place and the update event data available to us, we can start working on notifying Slack of the changes.

Notifying Slack Of Remote Config Version Changes

For our first pass here, we’ll send a notification message to Slack that simply includes the new version number and the email address of the person who made the change.

For this, we’ll use the Slack SDK for Java.

// pom.xml
<dependencies>
    ...
    <dependency>
        <groupId>com.slack.api</groupId>
        <artifactId>slack-api-client</artifactId>
        <version>1.21.1</version>
    </dependency>
</dependencies>

To help work with the Slack SDK, we’ll create a small class SlackClient.

import com.slack.api.Slack
import com.slack.api.methods.MethodsClient
import com.slack.api.methods.response.chat.ChatPostMessageResponse

class SlackClient(token: String, private val channelId: String) {

    private val methods: MethodsClient = Slack.getInstance().methods(token)

    fun notify(message: String): ChatPostMessageResponse = methods.chatPostMessage { req ->
        req.channel(channelId).text(message).mrkdwn(true)
    }
}

This class will manage the instantiation of a Slack MethodsClient and the sending of our notification message to the desired Slack channel. The authentication and routing of these calls is controlled via the token and channelId values passed to the SlackClient.

The Slack token must be authorized to post messages to the provided channelId. How to do that is beyond the scope of this post. I recommend reviewing the “Building a Slack app” guide on how to get started building a Slack bot that can post messages on your service’s behalf.

We will access the token and channelId as runtime environment variables. We’ll examine how to pass those values as part of the deployment section of this post.

Once we’ve retrieved the necessary Slack credentials, we can create a new instance of our SlackClient and notify the channel when the Remote Config version has been changed.

class RemoteConfigMonitorService
    : RawBackgroundFunction {

    private val slackChannelId: String =
        System.getenv("SLACK_CHANNEL_ID")
    private val slackToken: String =
        System.getenv("SLACK_TOKEN")

    private val slack =
        SlackClient(token = slackToken, channelId = slackChannelId)
    private val logger: Logger =
       Logger.getLogger(RemoteConfigMonitorService::class.java.name)
    private val gson: Gson = Gson()
override fun accept(
        json: String,
        context: Context
    ) {
        val event: RemoteConfigUpdateEvent = 
            gson.fromJson(json, RemoteConfigUpdateEvent::class.java)
        val msg = "Remote Config updated to version ${event.version} by ${event.user.email}"
        logger.info(msg)
        slack.notify(msg)
    }
}

At this point, we should have a functional (if still limited) notification service ready to be deployed to GCP.

Deploying The Cloud Function From A Local Development Machine

To actually put our new Cloud Function into service, we need to deploy it to the Google Cloud project that our Firebase project is in. If you’re not sure which GCP project backs your Firebase instance, you can review your Firebase Project Settings.

To deploy our Cloud Function from our local development machine, we need to do several things:

• Install gcloud cli
• Authenticate gcloud with our desired GCP project
• Run the functions deploy command with any necessary arguments

The general deployment command format looks something like the following:

gcloud functions deploy <desired deployment name> \
--entry-point <class name> \
--runtime <target runtime> \
--trigger-event <target event>

We need to indicate the desired deployment name of our function, which class to load in response to an event, which event(s) should trigger the function, and which runtime environment to use for the deployment.

Beyond the basics, we can further customize the deployment by passing any necessary environment variables or any number of other properties.

In our case, the deployment command should look something like the following:

gcloud functions deploy remote-config-monitor-service \
--entry-point com.premise.RemoteConfigMonitorService \
--runtime java11 \
--trigger-event google.firebase.remoteconfig.update \
--set-env-vars GOOGLE_CLOUD_PROJECT=<target gcp proejct>,SLACK_TOKEN=<Slack Bot Token>,SLACK_CHANNEL_ID=<Id for target Slack channel>

The GOOGLE_CLOUD_PROJECT env variable will be used internally to ensure that the Cloud Function is authorized to interact with other GCP services such as Firebase. This variable is set automatically by some Cloud Function runtimes, but must be explicitly set when using the Java runtime.

Once this command has been run, it typically takes a couple of minutes to deploy to the target GCP project. Once deployed, you should be able to make a change in Remote Config and see a Slack notification within a few seconds.

Sending Remote Config Diffs With Slack Notification

At this point, our notification simply shows the new versionand the email of whoever made the change. It’s not particularly useful yet. So let’s dive deeper and take a look at how we can generate detailed diffs of each template change.

Loading Remote Config template versions

The remoteconfig.update event payload does not include the actual changes to the Remote Config template. Instead, the payload gives the version of the updated template. If we want to notify of the specific changes to our Remote Config parameters and conditions, we need to use the version to fetch that information.

We can use the Firebase Admin SDK to load this information from our Firebase project, and we’ll use coroutines to load both the old and new versions in parallel.

// pom.xml
<dependencies>
    <dependency>
        <groupId>com.google.firebase</groupId>
        <artifactId>firebase-admin</artifactId>
        <version>8.1.0</version>
    </dependency>
    <dependency>
        <groupId>org.jetbrains.kotlinx</groupId>
        <artifactId>kotlinx-coroutines-core</artifactId>
        <version>1.6.1</version>
        <type>pom</type>
    </dependency>
</dependencies>

With these dependencies added, we can now load the updated Remote Config version and version — 1 to get the complete set of parameters and conditions for each version.

private val projectId:String = System.getenv("GOOGLE_CLOUD_PROJECT")
init {        
    FirebaseApp.initializeApp()    
}
override fun accept(
    event: RemoteConfigUpdateEvent,
    context: Context
): Unit = runBlocking {
    val event: RemoteConfigUpdateEvent = 
            gson.fromJson(json, RemoteConfigUpdateEvent::class.java)
    val newConfig = async(Dispatchers.IO) {
        FirebaseRemoteConfig
            .getInstance()
            .getTemplateAtVersion(event.version)
    }
    val previousConfig = async(Dispatchers.IO) {
        FirebaseRemoteConfig
            .getInstance()
            .getTemplateAtVersion(event.version - 1)
    }

    val msg = "Remote Config updated to version ${event.version} by ${event.user.email}"
    logger.info(msg)
    slack.notify(msg)
}

Notice here that we are using runBlocking{} to provide a CoroutineScope and to block completion of accept() until all coroutines have completed. We then use the async{} coroutine builder to load both Remote Config template versions in parallel.

Diffing Remote Config template versions

Now that we have the full set of Remote Config parameters and conditions, we can diff the two versions, and use that diff to build a more helpful notification message.

We’ll start by creating several model classes to assist with the diff operations.

/**
 * Holds a diff between two Parameters or Conditions
 */
data class RemoteConfigDiff<T>(
    val key: String,
    val original: T,
    val updated: T
)

/**
 * A container for a set of changes to Parameters or Conditions
 */
data class RemoteConfigDiffContainer<T : Any>(
    val label: String,
    val removed: List<RemoteConfigDiff<T>>,
    val added: List<RemoteConfigDiff<T>>,
    val updated: List<RemoteConfigDiff<T>>,
) {
    val numberOfChanges: Int = removed.size + added.size + updated.size
}

/**
 * A container for the overall diff between two Remote Config template versions
 */
data class RemoteConfigDiffCollection(
    val projectId: String,
    val update: RemoteConfigUpdateEvent,
    val parameters: RemoteConfigDiffContainer<Parameter>,
    val conditions: RemoteConfigDiffContainer<Condition>
) {
    val numberOfChanges = parameters.numberOfChanges + conditions.numberOfChanges
}

Next, we’ll write a function to diff the updates into a resulting RemoteConfigDiffCollection, which will then be used to format our output notification for Slack.

/**
 * Constructs the diffs for Parameters and for Templates
 */
fun getRemoteConfigDiff(
    projectId: String,
    update: RemoteConfigUpdateEvent,
    original: Template,
    updated: Template
) = RemoteConfigDiffCollection(
    projectId = projectId,
    update = update,
    parameters = constructDiff(
        label = "Parameters",
        originalValues = original.getAllParameters(),
        updatedValues = updated.getAllParameters(),
        getDefault = { Parameter() }
    ),
    conditions = constructDiff(
        label = "Conditions",
        originalValues = original.conditions.associateBy { condition -> condition.name },
        updatedValues = updated.conditions.associateBy { condition -> condition.name },
        getDefault = { Condition(" ", " ") }
    )
)

private fun Template.getAllParameters() = parameters + buildMap {
    parameterGroups.entries.forEach { entry ->
        putAll(entry.value.parameters)
    }
}

/**
 * Uses Set math to identify which keys were updated, added, or removed
 * Builds List<RemoteConfigDiff> for each of these sets of keys
 */
private fun <T : Any> constructDiff(
    label: String,
    originalValues: Map<String, T>,
    updatedValues: Map<String, T>,
    getDefault: () -> T
): RemoteConfigDiffContainer<T> {

    val keysInBothConfigVersion = originalValues.keys intersect updatedValues.keys
    val removedKeys = originalValues.keys subtract keysInBothConfigVersion
    val addedKeys = updatedValues.keys subtract keysInBothConfigVersion

    val removedDiffs: List<RemoteConfigDiff<T>> = removedKeys.map { key ->
        RemoteConfigDiff(
            key = key,
            original = originalValues.getOrDefault(key, getDefault()),
            updated = getDefault()
        )
    }

    val addedDiffs: List<RemoteConfigDiff<T>> = addedKeys.map { key ->
        RemoteConfigDiff(
            key = key,
            original = getDefault(),
            updated = updatedValues.getOrDefault(key, getDefault())
        )
    }

    val updatedDiffs: List<RemoteConfigDiff<T>> = keysInBothConfigVersion.map { key ->
        val originalParameter = originalValues.getOrDefault(key, getDefault())
        val updatedParameter = updatedValues.getOrDefault(key, getDefault())
        Triple(key, originalParameter, updatedParameter)
    }.filter { it.second != it.third }
        .map {
            RemoteConfigDiff(
                key = it.first,
                original = it.second,
                updated = it.third
            )
        }

    return RemoteConfigDiffContainer(
        label = label,
        removed = removedDiffs,
        added = addedDiffs,
        updated = updatedDiffs
    )
}

Now that we understand how the Remote Config values have changed, we can format that data into something easier to glance at and understand in Slack.

Formatting Remote Config diffs

We will use the Java Diff Utils library to format the diffs between Remote Config template versions. This library is particularly helpful because it can annotate changes using simple markdown, which allows us to do things like strikethrough removed values and bold added ones. This helps a lot when viewing the output in Slack.

// pom.xml
<dependencies>
    <dependency>
        <groupId>io.github.java-diff-utils</groupId>
        <artifactId>java-diff-utils</artifactId>
        <version>4.11</version>
    </dependency>
</dependencies>

With the dependency added, we can create a RemoteConfigDiffFormatter class that will use a passed RemoteConfigDiffCollection to format an output string including:

• The total number of changes made
• A link to the Firebase Remote Config dashboard
• The email address of the person that made the change
• Removed, Added, and Updated sections for Conditions
• Removed, Added, and Updated sections for Parameters

private const val REMOVAL_SLACK_EMOJI = ":github-changes-requested:"
private const val ADDITION_SLACK_EMOJI = ":github-check-mark:"

/**
 * Generates a markdown-formatted string for a given [RemoteConfigDiffCollection]
 */
class RemoteConfigDiffFormatter(
    private val diff: RemoteConfigDiffCollection,
    private val firebaseProjectUrl: String,
) {
    private val generator = DiffRowGenerator.create()
        .showInlineDiffs(true)
        .inlineDiffByWord(true)
        .oldTag { f: Boolean? -> "~" }
        .newTag { f: Boolean? -> "*" }
        .build()

    fun toMarkdownString(): String {
        return diff.run {
            buildString {
                appendLine("$numberOfChanges changes made to <$firebaseProjectUrl|Remote Config> in $projectId")
                appendLine("Updated by: ${update.user.email}")

                markdownFromContainer(conditions)
                markdownFromContainer(parameters)
            }
        }
    }

    private fun Any.toJson(): String {
        return Gson().toJson(this)
    }

    private fun <T : Any> StringBuilder.markdownFromContainer(container: RemoteConfigDiffContainer<T>) {
        removedSection(container)
        addedSection(container)
        updatedSection(container)
    }

    private fun <T : Any> StringBuilder.updatedSection(container: RemoteConfigDiffContainer<T>) {
        if (container.updated.isNotEmpty()) {
            appendLine("")
            appendLine("")
            appendLine("$ADDITION_SLACK_EMOJI *Updated ${container.label}:*")
            appendLine("")

            container.updated.forEach { diff ->
                val rows = generator.generateDiffRows(
                    listOf(diff.original.toJson()),
                    listOf(diff.updated.toJson())
                )
                appendLine("  • _${diff.key}_")
                appendLine("      • ${rows.first().oldLine}")
                appendLine("      • ${rows.first().newLine}")
            }
        }
    }

    private fun <T : Any> StringBuilder.addedSection(container: RemoteConfigDiffContainer<T>) {
        if (container.added.isNotEmpty()) {
            appendLine("")
            appendLine("")
            appendLine("$ADDITION_SLACK_EMOJI *Added ${container.label}:*")
            appendLine("")

            container.added.forEach { diff ->
                val rows = generator.generateDiffRows(
                    listOf(""),
                    listOf(diff.updated.toJson())
                )
                appendLine("  • _${diff.key}_: ${rows.first().newLine}")
            }
        }
    }

    private fun <T : Any> StringBuilder.removedSection(container: RemoteConfigDiffContainer<T>) {
        if (container.removed.isNotEmpty()) {
            appendLine("")
            appendLine("")
            appendLine("$REMOVAL_SLACK_EMOJI *Removed ${container.label}:*")
            appendLine("")

            container.removed.forEach { diff ->
                val rows = generator.generateDiffRows(
                    listOf(diff.original.toJson()),
                    listOf("")
                )
                appendLine("  • _${diff.key}_: ${rows.first().oldLine}")
            }
        }
    }
}

With our formatter in place, we can put it all together to send our more detailed notifications to Slack.

class RemoteConfigMonitorService
    : RawBackgroundFunction {

    private val projectId: String = 
        System.getenv("GOOGLE_CLOUD_PROJECT")
    private val slackChannelId: String =
        System.getenv("SLACK_CHANNEL_ID")
    private val slackToken: String =
        System.getenv("SLACK_TOKEN")
    private val firebaseProjectUrl: String = 
   "https://console.firebase.google.com/project/${projectId}/config"


    private val slack =
        SlackClient(token = slackToken, channelId = slackChannelId)
    private val logger: Logger =
       Logger.getLogger(RemoteConfigMonitorService::class.java.name)
    private val gson: Gson = Gson()
    init {        
        FirebaseApp.initializeApp()    
    }
    override fun accept(
        json: String,
        context: Context
    ): Unit = runBlocking {
        val event: RemoteConfigUpdateEvent = 
            gson.fromJson(json, RemoteConfigUpdateEvent::class.java)
        
        val newConfig = async(Dispatchers.IO) {
            FirebaseRemoteConfig
                .getInstance()
                .getTemplateAtVersion(event.version)
        }
        val previousConfig = async(Dispatchers.IO) {
            FirebaseRemoteConfig
                .getInstance()
                .getTemplateAtVersion(event.version - 1)
        }

        val diffCollection = getRemoteConfigDiff(
            projectId, 
            event, 
            previousConfig.await(), 
            newConfig.await()
        )

        val response = slack.notify(
            message = RemoteConfigDiffFormatter(
                diff = diffCollection,
                firebaseProjectUrl = firebaseProjectUrl
            ).toMarkdownString())

        when (response.isOk) {
            true -> logger.info("Slack channel $slackChannelId notified")
            false -> logger.info("Failed to notify Slack channel $slackChannelId: ${response.error}")
        }
    }
}

The resulting output in Slack should look something like this:

Sample Remote Config diff sent to Slack

What’s Next?

This v2 of our Firebase Remote Config notification system has been a nice improvement so far, but we’re not done. Now that we have an improved framework on which to build, we’re looking for more ways to improve our monitoring of Remote Config changes.

The following are some of the planned improvements:

• Explicitly warn when a parameter’s type changes
• Automated deployment via GitHub Actions
• Improved notification formatting — possibly using Slack’s Block Kit APIs

You can find the full sample code for this post on GitHub:

mobile-tutorials/002-remote-config-monitor-cloud-function at main · premisedata/mobile-tutorials

Premise is constantly looking to hire top-tier engineering talent to help us improve our front-end, mobile offerings, data and cloud infrastructure. Come find out why we’re consistently named among the top places to work in Silicon Valley by visiting our Careers page.