core

config

class shekels.core.config.Config(raw_data=None, trusted_data=None, deserialize_mapping=None, init=True, partial=True, strict=True, validate=False, app_data=None, lazy=False, **kwargs)[source]

Bases: Model

Configuration of database.

data_path

Path to CSV file.

Type:

str

columns

Columns to be displayed in data.

Type:

list[str]

default_query

Placeholder SQL query string.

Type:

str

font_family

Font family.

Type:

str

color_scheme

Color scheme.

Type:

dict

conform

List of conform actions.

Type:

lit[dict]

plots

List of plots.

Type:

list[dict]

_schema = <schematics.deprecated.patch_schema.<locals>.Schema object>
color_scheme = <DictType(StringType) instance on Config as 'color_scheme'>
columns = <ListType(StringType) instance on Config as 'columns'>
conform = <ListType(ModelType) instance on Config as 'conform'>
data_path = <StringType() instance on Config as 'data_path'>
default_query = <StringType() instance on Config as 'default_query'>
font_family = <StringType() instance on Config as 'font_family'>
plots = <ListType(ModelType) instance on Config as 'plots'>
class shekels.core.config.ConformAction(raw_data=None, trusted_data=None, deserialize_mapping=None, init=True, partial=True, strict=True, validate=False, app_data=None, lazy=False, **kwargs)[source]

Bases: Model

Schematic for conform actions.

action

Must be ‘overwrite’ or ‘substitute’.

Type:

str

source_column

Source column to be matched.

Type:

str

target_column

Target column to be overwritten.

Type:

str

mapping

Mapping of matched key in source column with replacement value in target column.

Type:

dict

_schema = <schematics.deprecated.patch_schema.<locals>.Schema object>
action = <StringType() instance on ConformAction as 'action'>
mapping = <DictType(UnionType) instance on ConformAction as 'mapping'>
source_column = <StringType() instance on ConformAction as 'source_column'>
target_column = <StringType() instance on ConformAction as 'target_column'>
validate()[source]

Validates the state of the model. If the data is invalid, raises a DataError with error messages. Also, performs a stricter validation on mapping if action is substitute.

Parameters:

  • partial (bool, optional) – Allow partial data to validate. Essentially drops the required=True settings from field definitions. Default: False.

  • convert (bool, optional) – Controls whether to perform import conversion before validating. Can be turned off to skip an unnecessary conversion step if all values are known to have the right datatypes (e.g., when validating immediately after the initial import). Default: True.

Raises:

DataError – If data is invalid.

class shekels.core.config.FigureItem(raw_data=None, trusted_data=None, deserialize_mapping=None, init=True, partial=True, strict=True, validate=False, app_data=None, lazy=False, **kwargs)[source]

Bases: Model

Schematic for a plot figure.

kind

Type of plot. Default: bar.

Type:

str

color_scheme

Color scheme for plot. Default: {‘grey1’: ‘#181818’, ‘bg’: ‘#242424’}.

Type:

dict[str, str]

x_axis

Column to use as x axis: Default: None.

Type:

str

y_axis

Column to use as y axis: Default: None.

Type:

str

title

Title of plot. Default: None.

Type:

str

x_title

Title of plot x axis. Default: None.

Type:

str

y_title

Title of plot y axis. Default: None.

Type:

str

bins

Number of bins if histogram. Default: 50.

Type:

int

bar_mode

How bars in bar graph are presented. Default: stack.

Type:

str

_schema = <schematics.deprecated.patch_schema.<locals>.Schema object>
bar_mode = <StringType() instance on FigureItem as 'bar_mode'>
bins = <IntType() instance on FigureItem as 'bins'>
color_scheme = <DictType(StringType) instance on FigureItem as 'color_scheme'>
kind = <StringType() instance on FigureItem as 'kind'>
title = <StringType() instance on FigureItem as 'title'>
x_axis = <StringType() instance on FigureItem as 'x_axis'>
x_title = <StringType() instance on FigureItem as 'x_title'>
y_axis = <StringType() instance on FigureItem as 'y_axis'>
y_title = <StringType() instance on FigureItem as 'y_title'>
class shekels.core.config.FilterAction(raw_data=None, trusted_data=None, deserialize_mapping=None, init=True, partial=True, strict=True, validate=False, app_data=None, lazy=False, **kwargs)[source]

Bases: Model

Schematic for filter actions.

column

Column name.

Type:

str

comparator

String representation of comparator.

Type:

str

value

Value to be compared.

Type:

object

_schema = <schematics.deprecated.patch_schema.<locals>.Schema object>
column = <StringType() instance on FilterAction as 'column'>
comparator = <StringType() instance on FilterAction as 'comparator'>
value = <BaseType() instance on FilterAction as 'value'>
class shekels.core.config.GroupAction(raw_data=None, trusted_data=None, deserialize_mapping=None, init=True, partial=True, strict=True, validate=False, app_data=None, lazy=False, **kwargs)[source]

Bases: Model

Schematic for group actions.

columns

Columns to group data by.

Type:

str or list[str]

metric

Aggregation metric.

Type:

str

datetime_column

Datetime column for time grouping. Default: date.

Type:

str, optinal

_schema = <schematics.deprecated.patch_schema.<locals>.Schema object>
columns = <ListType(StringType) instance on GroupAction as 'columns'>
datetime_column = <StringType() instance on GroupAction as 'datetime_column'>
metric = <StringType() instance on GroupAction as 'metric'>
class shekels.core.config.PivotAction(raw_data=None, trusted_data=None, deserialize_mapping=None, init=True, partial=True, strict=True, validate=False, app_data=None, lazy=False, **kwargs)[source]

Bases: Model

Schematic for group actions.

columns

Columns whose unique values become separate traces within a plot.

Type:

list[str]

values

Columns whose values become the values within each trace of a plot. Default: [].

Type:

list[str], optional

index

Column whose values become the y axis values of a plot. Default: None.

Type:

str, optional

_schema = <schematics.deprecated.patch_schema.<locals>.Schema object>
columns = <ListType(StringType) instance on PivotAction as 'columns'>
index = <StringType() instance on PivotAction as 'index'>
values = <ListType(StringType) instance on PivotAction as 'values'>
class shekels.core.config.PlotItem(raw_data=None, trusted_data=None, deserialize_mapping=None, init=True, partial=True, strict=True, validate=False, app_data=None, lazy=False, **kwargs)[source]

Bases: Model

Schematic for a plot.

filters

How data is filtered. Default: [].

Type:

list[dict]

group

How data is grouped. Default: {}.

Type:

dict

pivot

How data is pivoted. Default: {}.

Type:

dict

figure

Plot figure details. Default: {}.

Type:

dict

min_width

Minimum width of plot. Default: 0.25.

Type:

float

_schema = <schematics.deprecated.patch_schema.<locals>.Schema object>
figure = <ModelType(FigureItem) instance on PlotItem as 'figure'>
filters = <ListType(ModelType) instance on PlotItem as 'filters'>
group = <ModelType(GroupAction) instance on PlotItem as 'group'>
min_width = <FloatType() instance on PlotItem as 'min_width'>
pivot = <ModelType(PivotAction) instance on PlotItem as 'pivot'>
shekels.core.config.is_bar_mode(item)[source]

Ensures mode is a legal bar mode.

Parameters:

item (str) – Mode.

Raises:

ValidationError – If mode is not a legal bar mode.

shekels.core.config.is_color_scheme(item)[source]

Determines if given dict is a valid color scheme.

Parameters:

item (dict) – Color scheme dictionary.

Raises:

ValidationError – If item contains invalid keys.

Return type:

None

shekels.core.config.is_comparator(item)[source]

Ensures that given string is a legal comparator.

Legal comparators:

  • ==

  • !=

  • >

  • >=

  • <

  • <=

  • ~

  • !~

Parameters:

item (str) – String to be tested.

Raises:

ValidationError – If item is not a legal comparator.

Return type:

None

shekels.core.config.is_csv(filepath)[source]

Determines if given filepath is a CSV.

Parameters:

filepath (str or Path) – Filepath.

Raises:

ValidationError – If filepath is not a CSV.

Return type:

None

shekels.core.config.is_metric(item)[source]

Ensures that given string is a legal metric.

Legal metrics:

  • max

  • mean

  • min

  • std

  • sum

  • var

  • count

Parameters:

item (str) – String to be tested.

Raises:

ValidationError – If item is not a legal metric.

Return type:

None

shekels.core.config.is_percentage(number)[source]

Ensures number is between 0 and 100.

Parameters:

number (float) – Number to be tested.

Raises:

ValidationError – If number is not between 0 and 100.

shekels.core.config.is_plot_kind(item)[source]

Ensures item is a kind of plotly plot.

Parameters:

item (str) – Kind of plot.

Raises:

ValidationError – If item is a plot kind.

database

class shekels.core.database.Database(config)[source]

Bases: object

Database is a class for wrapping a mint transaction DataFrame with a simple CRUD-like API. API methods include: update, read and search.

__init__(config)[source]

Constructs a Database instance.

Parameters:

config (dict) – Configuration.

static _to_records(data)[source]

Converts given DataFrame to a list of JSONifiable dicts.

Parameters:

data (DataFrame) – Data.

Returns:

Records.

Return type:

list[dict]

property config

Returns a copy of this instance’s configuration.

Returns:

Copy of config.

Return type:

dict

property data

Returns a copy of this instance’s data.

Returns:

Copy of data.

Return type:

DataFrame

static from_json(filepath)[source]

Constructs a Database instance from a given JSON filepath.

Parameters:

filepath (Path or str) – Path to JSON config file.

Returns:

Database instance.

Return type:

Database

read()[source]

Returns data if update has been called.

Raises:

RuntimeError – If update has not first been called.

Returns:

Data as records.

Return type:

list[dict]

search(query)[source]

Search data according to given SQL query.

Parameters:

query (str) – SQL query. Make sure to use “FROM data” in query.

Returns:

Formatted data.

Return type:

DataFrame

update()[source]

Loads CSV found in config’s data_path into self._data.

Returns:

self.

Return type:

Database

data_tools

shekels.core.data_tools.conform(data, actions=[], columns=[])[source]

Conform given mint transaction data.

Parameters:

  • data (DataFrame) – Mint transactions DataFrame.

  • actions (list[dict], optional) – List of conform actions. Default: [].

  • columns (list[str], optional) – List of columns. Default: [].

Raises:

  • DataError – If invalid conform action given.

  • ValueError – If source column not found in data columns.

Returns:

Conformed DataFrame.

Return type:

DataFrame

shekels.core.data_tools.conform_figure(figure, color_scheme)[source]

Conforms given figure to use given color scheme.

Parameters:

  • figure (dict) – Plotly figure.

  • color_scheme (dict) – Color scheme dictionary.

Returns:

Conformed figure.

Return type:

dict

shekels.core.data_tools.filter_data(data, column, comparator, value)[source]

Filters given data via comparator(column value, value).

Legal comparators:

  • == lambda a, b: a == b

  • != lambda a, b: a != b

  • > lambda a, b: a > b

  • >= lambda a, b: a >= b

  • < lambda a, b: a < b

  • <= lambda a, b: a <= b

  • ~ lambda a, b: bool(re.search(a, b, flags=re.I))

  • !~ lambda a, b: not bool(re.search(a, b, flags=re.I))

Parameters:

  • data (DataFrame) – DataFrame to be filtered.

  • column (str) – Column name.

  • comparator (str) – String representation of comparator.

  • value (object) – Value to be compared.

Raises:

  • EnforceError – If data is not a DataFrame.

  • EnforceError – If column is not a string.

  • EnforceError – If column not in data columns.

  • EnforceError – If illegal comparator given.

  • EnforceError – If comparator is ~ or !~ and value is not a string.

Returns:

Filtered data.

Return type:

DataFrame

shekels.core.data_tools.get_figure(data, filters=[], group=None, pivot=None, kind='bar', color_scheme={}, x_axis=None, y_axis=None, title=None, x_title=None, y_title=None, bins=50, bar_mode='stack')[source]

Generates a plotly figure dictionary from given data and manipulations.

Parameters:

  • data (DataFrame) – Data.

  • filters (list[dict], optional) – List of filters for data. Default: [].

  • group (dict, optional) – Grouping operation. Default: None.

  • pivot (dict, optional) – Pivot operation. Default: None.

  • kind (str, optional) – Kind of plot. Default: bar.

  • color_scheme (dict[str, str], optional) – Color scheme. Default: {}.

  • x_axis (str) – Column to use as x axis: Default: None.

  • y_axis (str) – Column to use as y axis: Default: None.

  • title (str, optional) – Title of plot. Default: None.

  • x_title (str, optional) – Title of x axis. Default: None.

  • y_title (str, optional) – Title of y axis. Default: None.

  • bins (int, optional) – Number of bins if histogram. Default: 50.

  • bar_mode (str, optional) – How bars in bar graph are presented. Default: stack.

Raises:

  • DataError – If any filter in filters is invalid.

  • DataError – If group is invalid.

  • DataError – If pivot is invalid.

Returns:

Plotly Figure as dictionary.

Return type:

dict

shekels.core.data_tools.get_sql_grammar()[source]

Creates a grammar for parsing SQL queries.

Returns:

SQL parser.

Return type:

MatchFirst

shekels.core.data_tools.group_data(data, columns, metric, datetime_column='date')[source]

Groups given data by given columns according to given metric. If a legal time interval is given in the columns, then an additional special column of that same name is added to the data for grouping.

Legal metrics:

  • max lambda x: x.max()

  • mean lambda x: x.mean()

  • min lambda x: x.min()

  • std lambda x: x.std()

  • sum lambda x: x.sum()

  • var lambda x: x.var()

  • count lambda x: x.count()

Legal time intervals:

  • year

  • quarter

  • month

  • two_week

  • week

  • day

  • hour

  • half_hour

  • quarter_hour

  • minute

  • second

  • microsecond

Parameters:

  • data (DataFrame) – DataFrame to be grouped.

  • columns (str or list[str]) – Columns to group data by.

  • metric (str) – String representation of metric.

  • datetime_column (str, optinal) – Datetime column for time grouping. Default: date.

Raises:

  • EnforceError – If data is not a DataFrame.

  • EnforceError – If columns not in data columns.

  • EnforceError – If illegal metric given.

  • EnforceError – If time interval in columns and datetime_column not in columns.

Returns:

Grouped data.

Return type:

DataFrame

shekels.core.data_tools.parse_rgba(string)[source]

Parses rgb and rgba strings into tuples of numbers.

Example

>>>parse_rgba(‘rgb(255, 0, 0)’) (255, 0, 0) >>>parse_rgba(‘rgba(255, 0, 0, 0.5)’) (255, 0, 0, 0.5) >>>parse_rgba(‘foo’) is None True

Parameters:

string (str) – String to be parsed.

Returns:

(red, green, blue) or (red, green, blue, alpha)

Return type:

tuple

shekels.core.data_tools.pivot_data(data, columns, values=[], index=None)[source]

Pivots a given dataframe via a list of columns.

Legal time columns:

  • date

  • year

  • quarter

  • month

  • two_week

  • week

  • day

  • hour

  • half_hour

  • quarter_hour

  • minute

  • second

  • microsecond

Parameters:

  • data (DataFrame) – DataFrame to be pivoted.

  • columns (list[str]) – Columns whose unique values become separate traces within a plot.

  • values (list[str], optional) – Columns whose values become the values within each trace of a plot. Default: [].

  • index (str, optional) – Column whose values become the y axis values of a plot. Default: None.

Raises:

  • EnforceError – If data is not a DataFrame.

  • EnforceError – If data is of zero length.

  • EnforceError – If columns not in data columns.

  • EnforceError – If values not in data columns.

  • EnforceError – If index not in data columns or legal time columns.

Returns:

Pivoted data.

Return type:

DataFrame

shekels.core.data_tools.query_data(data, query, uri='sqlite:///:memory:')[source]

Parses SQL + regex query and applies it to given data.

Regex operators:

  • ~, regex - Match regular expression

  • !~, not regex - Do not match regular expression

Parameters:

  • data (DataFrame) – DataFrame to be queried.

  • query (str) – SQL query that may include regex operators.

Returns:

Data filtered by query.

Return type:

DataFrame

shekels.core.data_tools.query_dict(data, query)[source]

Query a given diction with a given SQL query.

Parameters:

  • data (dict) – Dictionary to be queried.

  • query (str) – SQL query.

Returns:

Queried dictionary.

Return type:

dict