DataWeave - Modify values depending on their key

Introduction

A little while ago I shared a function named applyToValues. This is a function that takes an element and a function, and applies the function to every element (you can read more about that here). While applyToValues works well in many situations (like modifying all the values in the same way, or modifying all the values based on the type of the value, to name a couple), what if you need to modify the value depending on the key? That's when you need applyWhenKey

Since the post for applyToValues went through a story of how I developed applyToValues, I wanted this post to be a little different. So instead of going over how this function might be developed iteratively, I'll go into detail about how it actually works.

If you don't care about how the code works, the first code block in the Code Dive section is all you need to get up and running. You can copy it, and use it as needed. I would recommend you check out the Examples and Gotcha sections just so you know how to use it, though. And you might as well check out the Additional Considerations section while you're at it and to pick up a tip that others reading your code will thank you for.

Examples

This section will go over some use cases I've found for this function.

Masking Passwords

This example will go over the use case of masking passwords for a flow's output.

Code:

%dw 1.0
%output application/json

%var input = [
  {
    "meta" : {
      "date":     "some date here",
      "password": "DontLookAtMe!"
    }
  {
    "field1":   "one",
    "password": "DontLookAtMe!",
  },
  {
    "field1":   "one",
    "password": "DontLookAtMe!",
  }
]

%var fieldsToMask = ["password"]

%function needsMask(k)
  fieldsToMask contains (k as :string)
  
%function mask(v) "*****"

%function applyWhenKey...
---
applyWhenKey(input, mask, needsMask)

Output:

[
  {
    "meta" : {
      "date":     "some date here",
      "password": "*****"
    }
  {
    "field1":   "one",
    "password": "*****",
  },
  {
    "field1":   "one",
    "password": "*****",
  }
]

You could describe this as: Replace the values in input with "*****" when the key is "password".

Modifying a single value in an object (potentially nested):

This example will review the use case of modifying a single value in the object based off of its key, while leaving the others untouched. This will work for a flat object, as well as object contained within arrays, nested objects, etc.

Code:

%dw 1.0
%output application/json

%var input = {
  field1: 1,
  field2: {
    field1: 1,
    field3: 3
  }
}

%function applyWhenKey...
---
applyWhenKey(input, ((n) -> n + 1), ((k) (k as :string) == "field1"))

Output:

{
  "field1": 2,
  "field2": {
    "field1": 2,
    "field3": 3
  }
}

You could describe this as: Increment each value in input by 1 when the key is "field1".

Gotchas

There isn't too much about applyWhenKey that can bite you, but there is one thing: the lambda you pass in for the predicate parameter, needs to cast the key to the necessary type. So if you're trying to compare the key to a :string, your lambda should look like this:

((k) -> (k as :string) == "string")

If you're working with this function and the output of it is the same as the input, chances are you're forgetting to cast the key.

Code Dive

Let's take a look at the code:

%function applyWhenKey(e, fn, predicate)
  e match {
    :array  -> $ map applyWhenKey($, fn, predicate),
    :object -> $ mapObject ((v, k) -> { 
                   (k): fn(v)
                 } when predicate(k) otherwise {
                   (k): applyWhenKey(v, fn, predicate)
                 }),
    default -> $
  }

Let's start with the function signature. This function is a lot like applyToValues in that it takes an element as its first parameter, and a function to apply to the element's values as its second parameters. However, unlike applyToValues, applyWhenKey takes an additional third parameter called predicate. You may be asking yourself what a predicate is. Predicate, in this context, is referring to the mathematical definition: a function that returns a boolean value (i.e., true or false). So the predicate parameter in the applyWhenKey function represents a function that takes in a single value (an object key) and returns a boolean value. Everytime you write a data transformation that uses filter, you supply it with a predicate function. Here's a concrete example of a predicate function:

%function isLarge(n) n > 100

Now that we understand what kind of parameters applyWhenKey expects, let's discuss how it uses those parameters. The first thing you'll notice is that applyWhenKey matches on the e parameter. Depending on the type of e, it will do one of three things. Let's start with the most simple and move to the most complicated.

First, let's take a look at this line:

default -> $

If e is neither of type :array, or of type :object, applyWhenKey just returns e as-is. This should make sense. After all, if e isn't an :array or an :object, there's not a chance it contains a key that we can run the predicate function on!

Second, we'll look at how applyWhenKey deals with e when it is of type :array. Here's the relevant line:

:array  -> $ map applyWhenKey($, fn, predicate)

This line has the potential to be confusing, so let's get that part out of the way right now. There are two $s in this line, and they both refer to something different. The first $ refers to the value of e, and the second $ refers to the current value that map is iterating over. So what does applyWhenKey do when it encounters e of type :array? Not much, actually. Arrays don't have keys, they only contain values. However, any value could potentially be a single value like a number or a string, or a collection like another array or an object. Which function do we know that knows how to deal with all three situations in the manner we need? applyWhenKey. So applyWhenKey iterates through each element in the array and calls itself to deal with each one. I'm going to refer this action as "passing the buck". If you're not familiar with this phrase, here's how Wikipedia describes it:

the act of attributing to another person or group one's own responsibility

So whenever I say "passing the buck" I just mean at that point the function doesn't know what to do with e, so it defers this decision to another function (in this case, itself).

This can be a confusing concept as well. Let's check out the following code and step through the execution:

%var arr = [1,2,[3,4,5]]
%function double(n) n + n
%function keyIsOne(k) (k as :string) == "one"
---
applyWhenKey(arr, double, keyIsOne)

The first two elements that applyWhenKey will encounter are 1 and 2. These are neither :array or :object, so they will remain as-is in the result. The third element encountered is of type :array, so this branch of the match will be executed:

:array  -> $ map applyWhenKey($, fn, predicate)

applyWhenKey will pass the array through ... map applyWhenKey($, fn predicate). In other words, it will pass each value in the array through the mapping function, and return the resulting array. But in this case, our mapping function is applyWhenKey, and therefore will just return any value as-is that's not an :object or :array, remember?

default -> $

So the result ends up being the same as the input: [1,2,[3,4,5]]. You might notice that this allows applyWhenKey to deal with arrays that are nested as deep as the language (or computer memory) will allow.

Let's check out the final branch of the match:

:object -> $ mapObject ((v, k) -> { 
               (k): fn(v)
             } when predicate(k) otherwise {
               (k): applyWhenKey(v, fn, predicate)
             })

This is where things get exciting. In the case that e is of type :object, for each key:value pair we will apply our fn function to the value if the key passes the predicate function (i.e., the predicate function returns true). If the key does not pass the predicate function, we pass the buck to applyWhenKey. Why do we do that? Because it allows us to work with nested data structures. The value of a key could be a singular value like a number or a string, or it could be a collection of values like an array or an object. What function do we know that knows how to deal with all three? applyWhenKey. So we pass the buck on to applyWhenKey. You might also notice that with this piece we can not only deal with objects that are nested as deep as the language (or computer memory) will allow, but we can also navigate through any arrays in those objects, and any objects/arrays in those arrays, etc.

Additional Considerations

applyWhenKey is a general-purpose function. It doesn't care what the shape of your data is, and it doesn't care what you want to do with it either, it just provides a mechanism to iterate through all your data and check keys along the way. It's your job to pass it every thing else. Because of this, it's a good idea to provide another layer of abstraction over applyWhenKey so that others reading your code can more easily identify the intention of your code. Take these two situations that build off of our first example in the Examples section:

%var input = ..

%var fieldsToMask = ["password", "secretKey"]

%function needsMask(k)
  fieldsToMask contains (k as :string)
  
%function mask(v) "*****"

%function applyWhenKey...
---
applyWhenKey(input, mask, needsMask)

and

%var input = ...

%function applyWhenKey...

%function maskFields(input, fields, mask="*****")
  applyWhenKey(input,
               ((value) -> mask),
               ((key) -> fields contains (key as :string)))
---
maskFields(input, ['password', 'secretKey'])

Which one is easier to read? Which one would you rather work with? To me, the second example is much more clear than the first. With the second example, I don't even need to know how applyWhenKey works. Unless I identify a bug in the code, I'm likely never going to need to dig into it. The name of the function and its inputs are enough information for me to make an informed decision about what it's doing.

Can you think of any other cool uses for applyWhenKey? Let me know.