The filter step is used to filter out your data within a transformation flow. Certain input steps (such as, single table and third-party connectors) also allow for filtering when being configured, but with the help of this step, it is possible to filter data any time in the flow.
This is done by using the Filter step in the Data Transformation module. Note: This is a built-in step, and therefore will be available in the Transformations List by default.
When configuring this step, you will see a Filter popup that allows you to create a filter. Following is an understanding of this popup .
|
Follow the instructions below to configure a filter step:
Use the new popup to add filters.
You can also remove all the filters using the Clear option or edit them by clicking on the Edit Filters option.
The following example shows a full transformation flow that involves a filter step. This transformation will involve setting up an input step, creating a custom calculated field, filtering the data, and then storing it into a database. You could always include more steps to your transformation flow.
Click on the Create button in the top-right corner.
Then select Transformation Flow.
If you do not see this option, you may not have security access to transformation flows. Learn how to get access here. |
You will be taken to the transformation flow builder.
Hover your cursor over the input steps button on the left side. A panel with a list of all data extraction steps will appear.
Drag one of these steps onto the canvas. For this procedure, we will use the single table step as an example. (Click here to learn about all the different input steps.)
On doing so, a popup will appear to load data from a data source.
Click on the data source that you require.
Then choose the database table, and click on Submit.
The selected table's fields will appear in the transformation flow panel to be configured.
Select only the fields that you want data to be extracted from.
You can make further changes, such as renaming the step, including a description, etc.
Once you’re done with the input step configuration, click on the Apply button.
On doing so, the data preview panel will display the data extracted from the configured database table.
In our example, we will now create a calculated field based on our data.
Drag the calculated field step and create a connection with the input step.
Then click on the calculated field step icon to configure it. The transformation flow panel wil be updated.
Click on Add Item.
A popup window will appear to create a new calculated field.
If the calculation is valid, click Save.
The data preview panel will display a new column field for this formula's result.
Now include the filter step in your flow by following the steps below.
Expand the transformation steps panel by hovering on its icon in the step builder, and then drag the Filter step onto the canvas.
Add filters using the settings on the new popup.
|
The configure panel on the right will also display the filter that was created.
You can remove a filter using the Clear option, or edit it by clicking on the Edit Filters option.
Once you are ready to save your data into a writeable database, follow the steps below.
Extend the output steps panel by hovering on its icon, and drag the SQL database output step onto the canvas.
Connect the filter step (or the previous step in your flow) to the output step by creating a connection.
Note: By default, the output step will be highlighted as red to signify that it contains errors. This is because it has not been configured yet.
And then configure the output step through the panel on the right-side. Click here to learn more about configuring this step.
You can now execute the draft flow by clicking on the run button in the top header menu. (This does a quick execution of the data rows in the data preview panel.)
Or save the flow for a full execution. To do that, click on the Publish button.
Then provide details in the popup that appears, such as providing a proper name, and selecting rights to access the flow.
Finally, click on the Save button.