base_connector module

Abstract class for connectors. A connector is used to connect to an input data source, perform data operations on the data, and output the results to some output data source. All connectors operations are performed in Pandas and are passed to the “transform” method as functions that accept a DataFrame and return a dictionary with a key of “results”. In this way, multiple operations can mutate the data in a single job. Jobs are then logged and a trace of operations can be retrieved.

By inheriting this class, polymorphism is ensured amongst all connectors so that code bases can expect the same interface.

class base_connector.BaseConnector

Bases: abc.ABC

abstract property df

abstract enrich(df: pandas.core.frame.DataFrame, left_column: str, right_column: str)

abstract get_input_handler(index: int) → libs.handler.base_handler.BaseHandler

abstract property input_columns: list

abstract property input_handler: libs.handler.base_handler.BaseHandler

abstract join(df: pandas.core.frame.DataFrame, left_column: str, right_column: str, join: str = 'left')

abstract property log_handler: libs.log_handler.log_handler.LogHandler

abstract mutate(*funcs)

abstract property output_columns: list

abstract property output_handler: libs.handler.base_handler.BaseHandler

abstract parse_steps()

abstract read()

abstract reduce()

abstract transform(df: pandas.core.frame.DataFrame)

abstract write()

abstract property write_logs: bool

class base_connector.BaseDbConnector

Bases: base_connector.BaseConnector