generic_connector module

The generic connector inherits from the abstract BaseConnector class. This is the primary class to inherit connectors from, as it provides the methods for performing DataFrame operations.

All Connectors maintain an attribute “df”, or DataFrame. Functions that mutate this dataframe are passed in the “transform” method. Once a transformation is complete, the “write” method then handles the results by executing the output handler’s execute action.

class generic_connector.GenericConnecter(input_handler: libs.handler.base_handler.BaseHandler, output_handler: libs.handler.base_handler.BaseHandler, log_handler: libs.handler.base_handler.BaseHandler, write_logs: bool = True, jsonify: bool = True, STEPS: dict = {})

Bases: libs.connector.base_connector.BaseConnector

property df: pandas.core.frame.DataFrame

enrich(handler: libs.handler.base_handler.BaseHandler, left_column: str, right_column: str)

Accepts a handler, a left column (and existing column from the df attribute), and a right column (existing column from the df argument). Extends the df attribute with the df argument using a left join.

The handler is appended to the __input_handlers instance attribute. Whenever “refresh” is called, the handlers read their data source and recreate the DataFrame.

get_input_handler(index: int) → libs.handler.base_handler.BaseHandler

property input_columns: list

property input_handler: libs.handler.base_handler.BaseHandler

join(handler: libs.handler.base_handler.BaseHandler, left_column: str, right_column: str, join_type: str = 'left')

Accepts a dataframe, a left column (and existing column from the df attribute), a right column (existing column from the df argument), and a join type. Joins the df attribute with the new table.

property log_handler: libs.log_handler.log_handler.LogHandler

mutate(*funcs)

Accepts a number of functions that mutate a dataframe and returns a dictionary. The return dictionary must have a key of results and all keys in the dictionary must be arrays of records. Each function mutates the df attribute and the logger logs each of the steps.

property output_columns: list

property output_handler: libs.handler.base_handler.BaseHandler

static parse_steps(steps: dict)

read()

Calls the Handler’s “execute” method and loads it into a dataframe.

reduce()

Accepts a dataframe, a left column (and existing column from the df attribute), a right column (existing column from the df argument), and a join type. Joins the df attribute with the new table.

transform(handler: libs.handler.base_handler.BaseHandler, *funcs)

Accepts a handler and a number of functions that mutate a dataframe using another dataframe and returns a dictionary. The return dictionary must have a key of results and all keys in the dictionary must be arrays of records. Each function mutates the df attribute and the logger logs each of the steps.

Handlers of transform type do not take action on “refresh”.

write()

property write_logs: bool