Oleg's software

NodeJS functional programming in data transformation tasks.

There is a common task in many software development cases – get one list and transform it to another list. For example, implementing consesus protocol in a blockchain requires a function that receives an array of transactions and produce another list of transactions which it considers correct. In this article we will examine a case where we are given a data set with contacts of people based on which we need to produce another list of related one time tokens to send to a client. As data transformation tasks tend to become intricated, implementation will be done in accordance to functional programming paradigm to make code simplier to understand and reason about.

Recently I had to carry out a refactoring of a JavaScript code which have been typical for many years:

/**
* @author Oleg Kubrakov yellowred.github.com
* @since 2017
**/
const Promise = require('Bluebird')
const jwt = require('jsonwebtoken');
let generateOneTimeToken = (value) => {
// we use promise to stub a request to MongoDB
let token = jwt.sign({ value }, 'privkey');
return Promise.resolve({value, token});
}
let data = [
{
name: "Michelle Rung",
contacts: [
{ phone: "7281681726" },
{ emailAddress: "michelle@gmail.com" },
{ emailAddress: "michelle@gmail.com" },
{ emailAddress: "" }
]
},
{
name: "Johann Sigh",
contacts: [
{ phone: "7812787-2667" },
{ emailAddress: "johann@gmail.com" },
{ emailAddress: "johann-md@gmail.com" }
]
}
]
let generateResponse = (data) => {
return Promise.coroutine(function* () {
let electronicContacts = []
let tmpArray = []
for (let dataValue of data) {
for (let individualValue of dataValue.contacts) {
if (individualValue.emailAddress && tmpArray.indexOf(individualValue.emailAddress) < 0) {
let tokenObject = yield generateOneTimeToken(individualValue.emailAddress)
electronicContacts.push({
electronicAddress: individualValue.emailAddress,
token: tokenObject.token
})
tmpArray.push(individualValue.emailAddress)
}
} // end for (let individualValue of dataValue.individualAddress)
}
return electronicContacts
})()
}
generateResponse(data).then(console.log)
view raw imperative.js hosted with ❤ by GitHub

First of all it looks monolytic, hard to comprehend, therefore wtf meter is overshooting. There are no logical steps and, in order to find out how it works, you need to execute the whole function in your mind. Loops can be pretty big so we have to put comments beside curly braces to identify where a loop ends. Basically what it does is loops through some data, collects unique emails and generates an array of tokens for them. We use the yield keyword to get a value from asynchronous functions before continue to execute the rest of a function.

So lets fix these downsides converting our code to functional style. One of the traits of functional programming is to make a value immutable which is passed to a function which transforms it in some way and in turn pass to another function and so on until the required structure is shaped. It is the opposite to conventional approach where you assign a new value every loop.

Eliminate loops

Lets start from the part that I like the most – getting our code free from for statements.

let generateResponse = (data) => {
return Promise.coroutine(function* () {
let emails = data
.map(getEmailsFromIndividual)
return emails
})()
}
let getEmailsFromIndividual = (individual) => {
if (!Array.isArray(individual.contacts)) return []
return individual
.contacts
.map(value => value['emailAddress'])
}
view raw imperative_noloops.js hosted with ❤ by GitHub

We get rid of the whole cycle and instead engage JavaScript function map (node.js supports it since v4) which will interate through an array and produce a new array where each element will be a result of the callback. This function is synchronous and will block the whole event loop. Keep this in mind and consider dividing into chunks large data sets.

Output:

[ [ undefined, 'michelle@gmail.com', '' ],
  [ undefined, 'johann@gmail.com', 'johann-md@gmail.com' ] ]

Flat Map

So as a result we got an array of arrays of emailAdresses. Our target structure is a plain list, so lets convert this 2 dimensions array into a flat list. We are going to engage two new JavaScript functions: reduce and concat. Reduce iterates through an array and produce a single value and concat simply merges two arrays. Combination of them is a single plain array.

et generateResponse = (data) => {
return Promise.coroutine(function* () {
let emails = data
.map(getEmailsFromIndividual)
.reduce(flatten)
return emails
})()
}
let getEmailsFromIndividual = (individual) => {
if (!Array.isArray(individual.contacts)) return []
return individual
.contacts
.map(value => value['emailAddress'])
}
let flatten = (flatArray, arrayElement) => {
return flatArray.concat(arrayElement)
}
view raw functional_flatmap.js hosted with ❤ by GitHub

Output:

[ undefined,
  'michelle@gmail.com',
  '',
  undefined,
  'johann@gmail.com',
  'johann-md@gmail.com' ]

Filter

Great! Flat list is much easier to understand and manipulate. It is tempting to start producing target structure, but before that let’s perform a clean up – remove duplicates and empty values. In order for that we will use the filter function – it will remove all elements which do not satisfy with a condition.

let generateResponse = (data) => {
return Promise.coroutine(function* () {
let emails = data
.map(getEmailsFromIndividual)
.reduce(flatten)
.filter(value => value != undefined && value != "")
.filter(distinct)
return emails
})()
}
let getEmailsFromIndividual = (individual) => {
if (!Array.isArray(individual.contacts)) return []
return individual
.contacts
.map(value => value['emailAddress'])
}
let flatten = (flatArray, arrayElement) => {
return flatArray.concat(arrayElement)
}
let distinct = (val, index, contacts) => {
return contacts.indexOf(val) === index
}
view raw functional_filter.js hosted with ❤ by GitHub

Output:

[ 'michelle@gmail.com',
  'johann@gmail.com',
  'johann-md@gmail.com' ]

Pretty neat! Now we are ready to build the final structure. So we aim to an array of dictionaries with two properties - emailAddress and token. Token is a one time JWT token which we are going to store in MongoDb. So this will require asynchronous requests as MongoDb API is asynchronous. Let’s engage Bluebird’s function map which can iterate through an array and execute a Promise for each element assigning back the return value.

/**
* @author Oleg Kubrakov yellowred.github.com
* @since 2017
**/
const Promise = require('Bluebird')
const jwt = require('jsonwebtoken');
let generateOneTimeToken = (value) => {
// we use promise to stub a request to MongoDB
let token = jwt.sign({ value }, 'privkey');
return Promise.resolve({value, token});
}
let data = [
{
name: "Michelle Rung",
contacts: [
{ phone: "7281681726" },
{ emailAddress: "michelle@gmail.com" },
{ emailAddress: "michelle@gmail.com" },
{ emailAddress: "" }
]
},
{
name: "Johann Sigh",
contacts: [
{ phone: "7812787-2667" },
{ emailAddress: "johann@gmail.com" },
{ emailAddress: "johann-md@gmail.com" }
]
}
]
let generateResponse = (data) => {
return Promise.coroutine(function* () {
let emails = data
.map(getEmailsFromIndividual)
.reduce(flatten)
.filter(value => value != undefined && value != "")
.filter(distinct)
return yield Promise.map(
emails,
emailAddress => valueAndToken(emailAddress)
)
})()
}
let getEmailsFromIndividual = (individual) => {
if (!Array.isArray(individual.contacts)) return []
return individual
.contacts
.map(value => value['emailAddress'])
}
let flatten = (flatArray, arrayElement) => {
return flatArray.concat(arrayElement)
}
let distinct = (val, index, contacts) => {
return contacts.indexOf(val) === index
}
let valueAndToken = (emailAddress) => {
return generateOneTimeToken(emailAddress)
.then(tokenObject => {
return {
electronicAddress: emailAddress,
token: tokenObject.token
}
})
}
generateResponse(data).then(console.log)
view raw functional.js hosted with ❤ by GitHub

Output:

[ { electronicAddress: 'michelle@gmail.com',
    token: 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ2YWx1ZSI6Im9iYW1hQGdtYWlsLmNvbSIsImlhdCI6MTQ4OTczMzg1Mn0.lfoJ6zxUmVNE2AZ2nSY6fgoGEu6wN_xqemu3k1Z4XyQ' },
  { electronicAddress: 'johann@gmail.com',
    token: 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ2YWx1ZSI6InRydW1wQGdtYWlsLmNvbSIsImlhdCI6MTQ4OTczMzg1Mn0.Rw1dZsx4NIosZVpWFREQSg7Wz8C5i2_b15CrNrZjHNc' },
  { electronicAddress: 'johann-md@gmail.com',
    token: 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ2YWx1ZSI6InRydW1wLXByZXNpZGVudEBnbWFpbC5jb20iLCJpYXQiOjE0ODk3MzM4NTJ9.xZJAX35U2_5kL-LLSn2Jju6XGj9nqQ475TU4c7RgJd0' } ]

Well we have met our goal and refactored the imperative code to functional fashion. All the cycles, temporary variables and buffers are boiled down to a chain of functions which separates concerns to different logical blocks. Next time you look into this code you might omit the rest of the code except the main function generateResponse. This code will have less bugs as it avoids to keep state in variables and instead always produce a new value.