Adding transformations through our platform gives you the authonomy you need to see your data the way you want, as soon as you need it.

​
Overview

The feature allows you to:

  • Select input tables: Use the interface to select the current tables that contain your target data.
  • Use your own transformation script: Utilize the coding space to add your own transformation script that manipulates the chosen tables and generate the new one.
  • Define details: Define your new table name, the description of what this transformation is doing, the frequency you want to trigger this pipeline and update the new table and more.

To start using our templates go to Add Transformation and follow this guide.

​
How to add your transformation

​
Select your input tables

At this point, we consider you already have some tables in the Catalog and now want to modify some aspect of them to adapt to your needs.

Select the tables from each layer: You might want to modify only one or more tables, either from the same or different layers. Feel free to select as many tables as needed for your transformation script to work.

​
Add your transformation script

Regardless of the way you wrote your transformation script (through our templates or other way) what you need to input here is the body of a ‘user_transformation’ function. It will be the code that transform your initial dataframes into your final dataframe.

While pasting your code here, please be carefull to not forget any piece of code, to avoid duplications, disordered code, etc. Make sure your code here is exactly the one you’ve tested and returned your desired final dataframe.

The script you’ll have in the end consists basically of:

def user_transformation(delta_dict): #the method signature, that should NOT be modified, by any means.
    # Custom imports section, that are the imports you had to use for your operations
      
    # Load dataframes section
      
    # Transformation section, that contains your core transformation script 

    # Return section, that should simply return your new dataframe name
    return new_df

We’re working to make this experience more streamlined and will keep you posted as we have more updates!

​
Define your output table name

Define a meaningful name for your new table. We recommend that you choose something that will remind you for what service/activity/purpose that table was created for.

​
Add custom dependencies

If your code requires specific dependencies - not added by default - you can list them here. Don’t forget to inform the versions as in the provided example.

​
Configure your transformation

By clicking Next, you’ll be prompted to inform some details about your new transformation. Give it a description: Describe in more details why this transformation was created, who’s going to use it and any other information that would help you identify this modification and the reason why the new table exists.

Configure your transformation pipeline trigger: Creating a transformation will also create a new pipeline that will run in a certain frequency to make sure your new table is up to date every time you need it. Therefore, at this point, you have to decide how your pipeline is going to be triggered: automatically in a specific date/time, as soon as other pipeline finishes or even manually. Learn more about triggers here.

​
Check your new transformation!

Click Nekt and hopefully everything will work out. You will be informed your new pipeline is set up.

You can now monitor it on the Transformations page to see its execution and completion. You might need to click on the icon to run the pipeline if you want to see it before the moment you selected for it to be triggered. As soon as the pipeline is executed, you’ll find your new table on the Catalog section.

​
Support

If you encounter any issues or have questions about using the feature, please message us!

We’re here to assist you in leveraging our data platform to its fullest potential.

OverviewEdit Transformation