RudderStack's Transformations feature lets you write custom functions to implement specific use-cases on your event data, like:

• Filtering or sampling events
• Cleaning or aggregating data
• Data masking or removing sensitive PII to ensure data privacy
• Enriching events by implementing static logic or leveraging an external API
• Using an API to implement specific actions on the events

See the Sample RudderStack Transformations GitHub repository for some useful templates that you can use to create your own transformations.

Key features

• Easier to build, manage, debug, and reuse.
• Add custom logic to your source events in real-time, before sending them to your destination.
• Supports JavaScript and Python.
• Use the transformations across your Event Streams, Cloud Extract, and Reverse ETL pipelines in cloud mode.
• You can version control your transformations.
• Create an organization-wide sandbox where your team can store all the transformations before publishing them in production.
• Programmatically manage your transformations using the Transformations API.

Use case

Suppose you want to set the context.os field to Android for all the events, irrespective of the actual platform RudderStack tracks the event from. To do this, you can write a simple transformation as shown:

Copy
export function transformEvent(event, metadata) {
    event.context.os = { name: "Android"};
    return event;
}

Copy
def transformEvent(event, metadata):
    event['context']['os'] = { "name": "Android"}
    return event

The transformEvent function overrides the event's context.os.name and sets it as Android, as seen below:

Python transformations

RudderStack now supports writing your transformations in Python, giving you full flexibility to use custom Python code to transform your source events on the fly. This feature is especially useful for the data teams that generally deal with Python.

Limitation

RudderStack supports only some of the built-in Python packages to write your transformations. These are datetime, json, math, random, requests, time, and urllib, along with the external package python-dateutil.

Adding a transformation

To add a new transformation in the RudderStack dashboard, follow these steps:

1. Log in to the RudderStack dashboard.
2. Go to Enhance > Transformations and click New Transformation.
3. Name your transformation and add an optional description.
4. In the Transformation window, select the language to write your transformation, as shown:

5. Add your transformation function, as shown:

5. To test your transformation, click the Run Test button. By default, RudderStack provides a sample event payload to test if your transformation logic works as expected.

6. Finally, click Save to save the transformation.

Connecting transformation to a destination

You can connect a transformation to a destination in two cases:

Case 1: Setting up a new destination

You can connect an existing transformation or create a new transformation from scratch while setting up a destination, as shown:

Case 2: Connecting to an existing destination

1. In the dashboard, go to the Transformation tab and click Add a transformation, as shown:

2. Select the transformation to connect to the destination and click Choose.

Deleting a transformation

To delete a transformation, go to Enhance > Transformations and click the Delete button next to the transformation that you want to delete.

transformEvent function

While using a transformation, RudderStack applies the transformEvent function on each event that takes the following two arguments:

After the transformation is complete, the transformEvent function returns the final event to be sent to the destination.

Accessing event metadata

RudderStack injects a function metadata(event) into your transformations as an argument. This lets you access the event metadata variables to customize your transformations.

The following properties, if available, are present in the metadata response:

An example of using metadata is shown below:

Copy
export function transformEvent(event, metadata) {
  const meta = metadata(event);
  event.sourceId = meta.sourceId;

  return event;
}

Copy
def transformEvent(event, metadata):
    meta = metadata(event)
    event['sourceId'] = meta['sourceId']
    return event

Making external API requests

You can make any number of external API requests in your transformation functions and use the fetched responses to enrich your events.

JavaScript

RudderStack injects an asynchronous fetch function in your transformations. It makes an API call to the given URL and returns the response in the JSON format.

You can use the fetch function in your transformations, as shown:

Copy
export async function transformEvent(event, metadata) {
  const res = await fetch("post_url", {
    method: "POST",  // POST, PUT, DELETE, GET, etc.
    headers: {
      "Content-Type": "application/json;charset=UTF-8",
      Authorization: "Bearer <authorization_token>"
    },
    body: JSON.stringify(event)
  });
  event.response = JSON.stringify(res);
  return event;
}

To see the fetch function in action, refer to the Clearbit enrichment example.

fetchV2

fetchv2 is a wrapper for the fetch call. It enables you to fetch the response properties more efficiently while making the external API calls.

The fetchv2 response contains the following properties:

The following example highlights the use of the fetchV2 function in a transformation to capture failure due to a timeout:

Copy
export async function transformEvent(event) {
  try {
    const res = await fetchV2("url", { timeout: 1000});
    if (res.status == 200) {
      event.response = JSON.stringify(res.body);
    }
  } catch (err) {
    log(err.message);
  }
  return event;
}

Python

You can use Python's requests package to fetch response properties while making the external API calls, as shown:

Copy
import requests

def transformEvent(event, metadata):
    res = requests.get("url")
    if res.status_code == 200:
        event["response"] = res.json();
    return event

FAQ

Refer to the FAQ guide for answers to all the frequently asked questions on RudderStack Transformations.