Page History

DataSource Data Source Function List

A connector DataSource data source is an implementation of the abstract java class AbstractDataSource. When defining a DataSource data source  the following functions require implementation: 

...

• protected final byte[] loadBlob(String key); 
• protected final boolean saveBlob(String key, byte[] data); 
• protected final boolean areBlobsAvailable(); 
• public final Object getAttribute(String key); 
• public final Integer getSourceId(); 

DataSource Data Source Function Definitions 

...

public abstract Collection<AbstractDataSet> getDataSets(); 

Return the collection of DataSets data sets that are available in this DataSourcedata source. See the DataSet Data Set section for defining a DataSetdata set. 

...

public abstract JDBCMetaData getDataSourceMetaData(); 

Return the connection meta-data metadata required for this DataSourcedata source. See the MetaData Metadata section for defining Meta Data metadata for a Data Sourcedata source. 

...

public ScheduleDefinition(FrequencyTypeCode frequencyTypeCode, String frequencyCode, Integer frequencyUnit);
 

Each frequencyTypeCode is defined below. For any type that requires n, that value is defined in the frequencyUnit. 

...

saveBlob() allows for saving a blob (byte[]) for later use. This is a way of saving data from background tasks for later use. The parameter key is the unique identifier for the data to be saved. data Data is the byte[] to be associated with that key. Writing null to data will delete the saved data for the specified key. Blobs can only be loaded on data sources that have been saved. areBlobsAvailable() can be used to see if blob access is available. 

...

getAttribute() allows for fetching attributes from the connection meta-datametadata. For example, a Username username may be specified for the connection through the Yellowfin UI. Using the key of the parameter, the contents of the Username meta-data username metadata field can be fetched for use when retrieving data from external APIs. 

...

getSourceId() can be used to fetch the unique internal id ID of source that this connector is associated with. This may be helpful for segregating data by connection in some kind of external cache or database. 

...

Recommendations for using saveBlob() and loadBlob() 

...

Minimize Stored/Cached Data 

It is recommended to only store data that cannot be retrieved from an external source reliably. This could be in the case of “sliding window” access to data, where only a limited amount of historical data is available, and this needs to be downloaded prior to it becoming unavailable. Extremely slow data sets can also use locally stored data sets to improve query speed. 

If significant amounts of data are stored in the blob system, it is recommended to truncate the data after a certain period. This might mean deleting all data when it reaches a certain age, or storing less granular information for older data. For instance, store raw data for three months, daily aggregated data for one year, and weekly aggregated data for older data. To achieve this, a background job would need to re-aggregate and re-store reaggregate and restore the data. 

...

...

Minimize Blob Size 

There is significant load induced on the Yellowfin database and server when storing and loading large sized blobs. If possible, distribute stored data across multiple blobs. 

For instance, there may be an instance where 100,000 tweets are stored in the blob storage system. This might be stored with the key “ALL_TWEETS”. However, to minimize loading times of blobs, and to not overload the caching system, this could be split and stored in smaller chunks. 

...

When a query is requested from the connector, filters can be used to determine which blobs need to be used, and thus loaded from the blob storage system. A For example, a query with the specified date range of 2016-02-05 to 2016-03-05 would just need to load the data for February and March, “201602_TWEETS” and “201603_TWEETS”. 

There may be significant overhead required to join datasets data sets from blobs. It is recommended that this be taken into consideration and to compare the performance of multiple smaller blobs versus larger ones. 

...

Application-level aggregation is toggled based on the capabilities of the connector. If any DataSet data set columns (as returned by getColumns()) support native aggregations (where the connector returns aggregated data), then aggregations will be available. 

Application-level filtering is also toggled based on the capabilities of columns in the connector. If any DataSet data set columns (as returned by getColumns()) support native filtering (where the connector applies its own filters), then the application-level filtering will be disabled, and only connector column filters will be available. Connector filters (as returned by getFilters()) can co-exist coexist with application-level filters. 

...

Custom messages can be returned to the Yellowfin UI if an error occurs whilst running a connector report. This can be done by throwing a ThirdPartyException() with a custom message from the connector pluginplug-in. 

throw new ThirdPartyException("Unable to connect to the Twitter API at this time."); 

The custom error message will be shown as a standard “Oh-No” error where the report would usually be rendered. This would usually be thrown from the execute() function on a DataSetdata set. 

...