---
title: "Processing Implementation"
canonical: "https://wiki.yellowfinbi.com/space/user80/1540314/Processing%20Implementation"
format: markdown
---
A number of methods need to be implemented to get the processing logic of the step to work. Some methods are different depending on whether it is a Row Step or a Cached Step.

 

## Common methods

Following are some of the common methods for defining how a step processes data.

| **Method** | **Description & Example** |
| --- | --- |
| <span style="color: #000000">public ETLStepAPIVersion getAPIVersion()</span> | This method is used to define the version of the Yellowfin Step API to maintain step updates. This is usually the latest version in the enum ETLStepAPIVersion. The API version is used to determine compatibility. |
| <span style="color: #000000">public Collection<ETLException> validate()</span> | This method performs pre-run validations. The implementation should check if any mandatory options were not set up, or if credentials are incorrect or a host is reachable. Errors should be captured in an instance of ETLException and added to the collection returned from the method. Alternatively, one could use the convenience method getInvalidConfigETLException() instead of constructing an ETLException. |
| <span style="color: #000000">public Map<String, String> getValidatedStepOptions()</span> | This method allows step options to be validated. Mapping of optionKey to optionValue will be saved in the Yellowfin config database. This is where one can remove invalid option values. Manipulating the mapping returned by <span style="color: #000000">this.getStepOptions()</span> will have no effect. |
| <span style="color: #000000">public void setupGeneratedFields()</span> | Implement this method when the step is required to output a new field. The data in the new field may be generated using other fields. The new field may also replace an existing field or be a duplicate. Yellowfin provides convenience methods for each operation. This method is expected to create a new instance of ETLStepMetadataFieldBean or duplicate an existing field. It is important that this method runs only if fields were not set up before, or if they should be set up again. If they are to be recreated because of a change in an option, the old field must be removed. Otherwise new fields will be generated every time the step is reconfigured. |
| <span style="color: #000000">public Integer getMinInputSteps()</span><br><span style="color: #000000">public Integer getMaxInputSteps()</span><br><span style="color: #000000">public Integer getMinOutputSteps()</span><br><span style="color: #000000">public Integer getMaxOutputSteps()</span> | These methods should be overridden if the step has multiple inputs or outputs. Yellowfin provides default implementations which return min/max values based on the Step Category. These values are defined in the ETLStepCategory enum element for that category. |
| YFLogger | Although this isn’t a method, it is common for all steps. A step can write to the Data Transformation log using the YFLogger. It should be declared as an instance variable. YFLogger is a wrapper for log4j’s Logger class. |

 

 

 

---

 

## Row Step Implementation 

A Row step extends the **AbstractETLRowStep** class. It requires the implementation of only one method, **processWireData()**.


### processWireData();

When the framework invokes processWireData(), data from each column of the current row is already on the correct Wire. Each wire is mapped to a metadata field and may be accessed using <span style="color: #000000">**this.getWireForField(fieldUUID)**</span>. Data should be retrieved from the wire, processed, and put back on the same wire or a different one.


- **Method Parameters: **The method has one parameter, a **List<ETLStepMetadataFieldBean>**. These are the fields from the input step. They are provided for convenience. Although they may be used to fetch wires, the preferred way is by using Default Metadata Fields, as shown in the snippet below.
- **Return Value: **The return value is a boolean which indicates whether the row of data should be output to the next step. A filter step, for example, would return false when data in a row does not satisfy filter conditions.
- **Exceptions:** This method throws **ETLException** and **InterruptedException**. Processing errors, if any, must be wrapped in an instance of ETLException and thrown so that Yellowfin can display them for the user. Convenience method <span style="color: #000000">**this.**</span>**throwUnhandledETLException(e)** may be used. Exceptions must not be caught and swallowed. It is also not advisable to catch **java.lang.Exception **as an InterruptedException will be caught as well. If it is unavoidable, then InterruptedException should be caught and thrown in a separate catch block.  
  
Code snippet:

 

Here’s a sample implementation for appending a number to a specific field.

```java
@Override
protected boolean processWireData(List<ETLStepMetadataFieldBean> fields)
                                  throws ETLException, InterruptedException {
    	
    // The options should've been validated by the validate() method,
    // so no need for further checks here
    String appendFieldUUID = this.getStepOption("APPEND_FIELD");
    String newFieldUUID = this.getStepOption("NEW_FIELD");
    String appendValue = this.getStepOption("APPEND_VALUE");
 
    Wire<Object, String> appendFieldWire = this.getWireForField(appendFieldUUID);
    Wire<Object, String> newFieldWire = this.getWireForField(newFieldUUID);
 
    Object data = appendFieldWire.getValue();
    String newFieldData = null;
    if (data == null) {
        newFieldData = appendValue;
    } else {
        newFieldData = data.toString() + appendValue;
    }
	
    newFieldWire.send(newFieldData);
	
    return true;
}


```

In this example,** processWireData()** runs for every row of data. In a real-world implementation, objects which will not change in subsequent method calls, should be cached in member variables. For example, appendFieldUUID, newFieldUUID, appendValue, appendFieldWire and newFieldWire should be member variables, populated only when processWireData() runs for the first time.



---



## Cached Step Implementation 

A cached step extends **AbstractETLCachedStep**. Only one method, **processEndRows()**, needs to be implemented. Cached steps usually have more than one input step. Data extraction steps are also often implemented as a cached step because the step will have no input. It generates data by executing an SQL query, for instance.


### processEndRows();


- **Input Steps**: For input steps, the framework invokes processEndRows() as soon as the process starts running. An input step needn’t bother about data caching. However, the implementation must put data on wires and emit data to its output(s). Here’s a sample implementation of the method for a data generator step:
- **Transformation steps: **For cached transformation steps, the framework invokes processEndRows() when every input has finished sending its data to the cached step. Each input step’s data is stored in a separate memory cache. The step implementation must send data on wires and emit data from the step to its output(s). The example below implements proc for a Union-All step and shows how data caches are used.