convtools package

Submodules

convtools.aggregations module

This module brings aggregations with various reduce functions

convtools.aggregations.Aggregate(*args, **kwargs) convtools.base.BaseConversion[source]

Shortcut for GroupBy().aggregate(*args, **kwargs)

class convtools.aggregations.ArrayDistinctReducer(*expressions, initial=<convtools.base._None object>, default=<convtools.base._None object>, where=None)[source]

Bases: convtools.aggregations.MultiStatementReducer

default: Any = <convtools.base.NaiveConversion object>
post_conversion: Optional[convtools.base.BaseConversion] = <convtools.base.InlineExpr object>
prepare_first: Tuple[str, ...] = ('%(result)s = {{ {0}: None }}',)
reduce: Tuple[str, ...] = ('%(result)s[{0}] = None',)
unconditional_init: bool = True
class convtools.aggregations.ArrayReducer(*expressions, initial=<convtools.base._None object>, default=<convtools.base._None object>, where=None)[source]

Bases: convtools.aggregations.MultiStatementReducer

default: Any = <convtools.base.NaiveConversion object>
prepare_first: Tuple[str, ...] = ('%(result)s = [{0}]',)
reduce: Tuple[str, ...] = ('%(result)s.append({0})',)
unconditional_init: bool = True
class convtools.aggregations.AverageReducerDispatcher[source]

Bases: convtools.aggregations.ReducerDispatcher

Dispatcher which chooses between weighted and simple averages

sum_reducer_dispatcher = <convtools.aggregations.SumReducerDispatcher object>
class convtools.aggregations.BaseDictReducer(*args, **kwargs)[source]

Bases: convtools.aggregations.MultiStatementReducer

This reducer accepts 2 expressions:

  • the first one is used to calculate keys of the resulting dict

  • the second one is used to calculate values to be reduced and put as the final value, under the certain key.

Effectively dict reducers allow for double grouping: the first one on the top level, the second one on DictReducer level.

prepare_first: Tuple[str, ...]
reduce: Tuple[str, ...]
class convtools.aggregations.BaseReducer(default, initial)[source]

Bases: convtools.base.BaseConversion, Generic[convtools.aggregations.RBT, convtools.base.CT]

Base of a reduce operation to be used during the aggregation

default: Any
expressions: Tuple[Any, ...]
gen_reduce_code_block(var_agg_data_value: str, reducer_code_input: str, checksum_flag: int, ctx: dict) convtools.aggregations.RBT[source]
post_conversion: Optional[convtools.base.BaseConversion] = None
prepare_default_n_initial(default, initial)[source]
self_content_type = 129
unconditional_init: bool = False
where: Optional[convtools.base.BaseConversion] = None
class convtools.aggregations.CountDistinctReducer(*expressions, initial=<convtools.base._None object>, default=<convtools.base._None object>, where=None)[source]

Bases: convtools.aggregations.MultiStatementReducer

default: Any = <convtools.base.NaiveConversion object>
post_conversion: Optional[convtools.base.BaseConversion] = <convtools.base.Call object>
prepare_first: Tuple[str, ...] = ('%(result)s = {{ {0} }}',)
reduce: Tuple[str, ...] = ('%(result)s.add({0})',)
unconditional_init: bool = True
class convtools.aggregations.CountReducer(*expressions, initial=<convtools.base._None object>, default=<convtools.base._None object>, where=None)[source]

Bases: convtools.aggregations.MultiStatementReducer

default: Any = <convtools.base.NaiveConversion object>
prepare_first: Tuple[str, ...] = ('%(result)s = 1',)
reduce: Tuple[str, ...] = ('%(result)s = {prev_result} + 1',)
unconditional_init: bool = True
class convtools.aggregations.DictArrayDistinctReducer(*args, **kwargs)[source]

Bases: convtools.aggregations.BaseDictReducer

Dict reducer where dict values are lists of distinct values

default: Any = <convtools.base.NaiveConversion object>
expressions: Tuple[Any, ...]
post_conversion: Optional[convtools.base.BaseConversion] = <convtools.base.InlineExpr object>
prepare_first: Tuple[str, ...] = ('%(result)s = _d = defaultdict(dict)', '_d[{0}][{1}] = None')
reduce: Tuple[str, ...] = ('%(result)s[{0}][{1}] = None',)
unconditional_init: bool = True
class convtools.aggregations.DictArrayReducer(*args, **kwargs)[source]

Bases: convtools.aggregations.BaseDictReducer

default: Any = <convtools.base.NaiveConversion object>
expressions: Tuple[Any, ...]
post_conversion: Optional[convtools.base.BaseConversion] = <convtools.base.InlineExpr object>
prepare_first: Tuple[str, ...] = ('%(result)s = _d = defaultdict(list)', '_d[{0}].append({1})')
reduce: Tuple[str, ...] = ('%(result)s[{0}].append({1})',)
unconditional_init: bool = True
class convtools.aggregations.DictCountDistinctReducer(*args, **kwargs)[source]

Bases: convtools.aggregations.BaseDictReducer

Dict reducer where dict values are numbers of distinct values per group

default: Any = <convtools.base.NaiveConversion object>
expressions: Tuple[Any, ...]
post_conversion: Optional[convtools.base.BaseConversion] = <convtools.base.InlineExpr object>
prepare_first: Tuple[str, ...] = ('%(result)s = {{ {0}: {{ {1} }} }}',)
reduce: Tuple[str, ...] = ('if {0} not in %(result)s:', '    %(result)s[{0}] = {{ {1} }}', 'else:', '    %(result)s[{0}].add({1})')
unconditional_init: bool = True
class convtools.aggregations.DictCountReducer(*args, **kwargs)[source]

Bases: convtools.aggregations.BaseDictReducer

default: Any = <convtools.base.NaiveConversion object>
expressions: Tuple[Any, ...]
prepare_first: Tuple[str, ...] = ('%(result)s = {{ {0}: 1 }}',)
reduce: Tuple[str, ...] = ('if {0} not in %(result)s:', '    %(result)s[{0}] = 1', 'else:', '    %(result)s[{0}] = {prev_result}[{0}] + 1')
unconditional_init: bool = True
class convtools.aggregations.DictFirstReducer(*args, **kwargs)[source]

Bases: convtools.aggregations.BaseDictReducer

default: Any = <convtools.base.NaiveConversion object>
expressions: Tuple[Any, ...]
prepare_first: Tuple[str, ...] = ('%(result)s = {{ {0}: {1} }}',)
reduce: Tuple[str, ...] = ('if {0} not in %(result)s:', '    %(result)s[{0}] = {1}')
unconditional_init: bool = True
class convtools.aggregations.DictLastReducer(*args, **kwargs)[source]

Bases: convtools.aggregations.BaseDictReducer

default: Any = <convtools.base.NaiveConversion object>
expressions: Tuple[Any, ...]
prepare_first: Tuple[str, ...] = ('%(result)s = {{ {0}: {1} }}',)
reduce: Tuple[str, ...] = ('%(result)s[{0}] = {1}',)
unconditional_init: bool = True
class convtools.aggregations.DictMaxReducer(*args, **kwargs)[source]

Bases: convtools.aggregations.BaseDictReducer

DictMax reducer which takes first positional item as keys and accumulates max value of second positional item

base_condition_code: Optional[str] = '{1} is not None'
default: Any = <convtools.base.NaiveConversion object>
expressions: Tuple[Any, ...]
prepare_first: Tuple[str, ...] = ('%(result)s = {{ {0}: {1} }}',)
reduce: Tuple[str, ...] = ('if {0} not in %(result)s or {1} > %(result)s[{0}]:', '    %(result)s[{0}] = {1}')
unconditional_init: bool = True
class convtools.aggregations.DictMinReducer(*args, **kwargs)[source]

Bases: convtools.aggregations.DictMaxReducer

expressions: Tuple[Any, ...]
reduce: Tuple[str, ...] = ('if {0} not in %(result)s or {1} < %(result)s[{0}]:', '    %(result)s[{0}] = {1}')
class convtools.aggregations.DictReducer(*args, **kwargs)[source]

Bases: convtools.aggregations.BaseDictReducer

default: Any = <convtools.base.NaiveConversion object>
expressions: Tuple[Any, ...]
prepare_first: Tuple[str, ...] = ('%(result)s = {{ {0}: {1} }}',)
reduce: Tuple[str, ...] = ('%(result)s[{0}] = {1}',)
unconditional_init: bool = True
class convtools.aggregations.DictSumOrNoneReducer(*args, **kwargs)[source]

Bases: convtools.aggregations.BaseDictReducer

Dict reducer where dict values are either numbers or None if there’s been at least one None value within the group

default: Any = <convtools.base.NaiveConversion object>
expressions: Tuple[Any, ...]
post_conversion: Optional[convtools.base.BaseConversion] = <convtools.base.InlineExpr object>
prepare_first: Tuple[str, ...] = ('%(result)s = _d = defaultdict(int)', '_d[{0}] = {1}')
reduce: Tuple[str, ...] = ('if {1} is None:', '    %(result)s[{0}] = None', 'elif %(result)s[{0}] is not None:', '    %(result)s[{0}] = {prev_result}[{0}] + {1}')
unconditional_init: bool = True
class convtools.aggregations.DictSumReducer(*args, **kwargs)[source]

Bases: convtools.aggregations.BaseDictReducer

default: Any = <convtools.base.NaiveConversion object>
expressions: Tuple[Any, ...]
post_conversion: Optional[convtools.base.BaseConversion] = <convtools.base.InlineExpr object>
prepare_first: Tuple[str, ...] = ('%(result)s = _d = defaultdict(int)', '_d[{0}] = {1} or 0')
reduce: Tuple[str, ...] = ('%(result)s[{0}] = {prev_result}[{0}] + ({1} or 0)',)
unconditional_init: bool = True
class convtools.aggregations.FastSumReducer(*expressions, initial=<convtools.base._None object>, default=<convtools.base._None object>, where=None)[source]

Bases: convtools.aggregations.MultiStatementReducer

default: Any = <convtools.base.NaiveConversion object>
prepare_first: Tuple[str, ...] = ('%(result)s = ({0} or 0)',)
reduce: Tuple[str, ...] = ('%(result)s = {prev_result} + {0}',)
unconditional_init: bool = True
class convtools.aggregations.FirstReducer(*expressions, initial=<convtools.base._None object>, default=<convtools.base._None object>, where=None)[source]

Bases: convtools.aggregations.MultiStatementReducer

default: Any = <convtools.base.NaiveConversion object>
prepare_first: Tuple[str, ...] = ('%(result)s = {0}',)
reduce: Tuple[str, ...] = ()
unconditional_init: bool = True
class convtools.aggregations.GroupBy(*by: Tuple[convtools.base.BaseConversion, ...])[source]

Bases: convtools.base.BaseConversion, Generic[convtools.aggregations.RBT, convtools.base.CT]

Generates the function which aggregates the data, grouping by conversions, specified in __init__ method and returns list of items in a format defined by the parameter passed to aggregate method.

If no group keys are passed, then it returns just a single value, defined by the parameter passed to aggregate method.

Current optimizations:
  • piping like c.group_by(...).aggregate().pipe(...) won’t run the aggregation twice, this is handled as 2 statements

  • using the same reduce clause twice (e.g. one used as an argument for some function calls) won’t result in calculating this reduce twice

aggregate(reducer: Union[dict, list, set, tuple, convtools.base.BaseConversion]) convtools.aggregations.GroupBy[source]

Takes the conversion which defines the desired output of aggregation

filter(condition_conv, cast=<convtools.base._None object>) convtools.base.BaseConversion[source]

Shortcut for calling convtools.base.FilterConversion on self

self_content_type = 194
class convtools.aggregations.LastReducer(*expressions, initial=<convtools.base._None object>, default=<convtools.base._None object>, where=None)[source]

Bases: convtools.aggregations.MultiStatementReducer

default: Any = <convtools.base.NaiveConversion object>
prepare_first: Tuple[str, ...] = ('%(result)s = {0}',)
reduce: Tuple[str, ...] = ('%(result)s = {0}',)
unconditional_init: bool = True
class convtools.aggregations.ListSortedOnceWrapper(list_: list, key=None, reverse=False)[source]

Bases: object

Wraps a list, exposes append method only. Once the list is filled up, it is sorted (only once) in-place and is returned when get method called.

append
get() list[source]
key
list_
reverse
sorted
class convtools.aggregations.MaxReducer(*expressions, initial=<convtools.base._None object>, default=<convtools.base._None object>, where=None)[source]

Bases: convtools.aggregations.MultiStatementReducer

base_condition_code: Optional[str] = '{0} is not None'
default: Any = <convtools.base.NaiveConversion object>
prepare_first: Tuple[str, ...] = ('%(result)s = {0}',)
reduce: Tuple[str, ...] = ('if {0} > %(result)s:', '    %(result)s = {0}')
unconditional_init: bool = True
class convtools.aggregations.MaxRowReducer(*expressions, initial=<convtools.base._None object>, default=<convtools.base._None object>, where=None)[source]

Bases: convtools.aggregations.MultiStatementReducer

Reducer which finds an item with max value of the expression and returns this item

base_condition_code: Optional[str] = '{0} is not None'
default: Any = <convtools.base.NaiveConversion object>
post_conversion: Optional[convtools.base.BaseConversion] = <convtools.base.GetItem object>
prepare_first: Tuple[str, ...] = ('%(result)s = ({0}, %(input)s)',)
reduce: Tuple[str, ...] = ('if %(result)s[0] < {0}:', '    %(result)s = ({0}, %(input)s)')
unconditional_init: bool = True
convtools.aggregations.MedianReducer(conv, *args, **kwargs) convtools.base.BaseConversion[source]
class convtools.aggregations.MinReducer(*expressions, initial=<convtools.base._None object>, default=<convtools.base._None object>, where=None)[source]

Bases: convtools.aggregations.MultiStatementReducer

base_condition_code: Optional[str] = '{0} is not None'
default: Any = <convtools.base.NaiveConversion object>
prepare_first: Tuple[str, ...] = ('%(result)s = {0}',)
reduce: Tuple[str, ...] = ('if {0} < %(result)s:', '    %(result)s = {0}')
unconditional_init: bool = True
class convtools.aggregations.MinRowReducer(*expressions, initial=<convtools.base._None object>, default=<convtools.base._None object>, where=None)[source]

Bases: convtools.aggregations.MaxRowReducer

expressions: Tuple[Any, ...]
reduce: Tuple[str, ...] = ('if %(result)s[0] > {0}:', '    %(result)s = ({0}, %(input)s)')
class convtools.aggregations.ModeReducer(conv, *args, **kwargs)[source]

Bases: convtools.aggregations.DictCountReducer

expressions: Tuple[Any, ...]
post_conversion: Optional[convtools.base.BaseConversion] = <convtools.base.InlineExpr object>
class convtools.aggregations.MultiStatementReducer(*expressions, initial=<convtools.base._None object>, default=<convtools.base._None object>, where=None)[source]

Bases: convtools.aggregations.BaseReducer

Defines the reduce operation, which is based on multiple python statements, to be used during the aggregation

base_condition_code: Optional[str] = None
gen_reduce_code_block(var_agg_data_value: str, reducer_code_input: str, checksum_flag: int, ctx: dict) convtools.aggregations.RBT[source]
prepare_first: Tuple[str, ...]
reduce: Tuple[str, ...]
class convtools.aggregations.PercentileReducer(percentile: float, conv, *args, interpolation='linear', **kwargs)[source]

Bases: convtools.aggregations.SortedArrayReducer

Calculates percentile (floats from 0 to 100 inclusive)

>>> c.ReduceFuncs.Percentile(95, c.item("amount"))
>>> c.ReduceFuncs.Percentile(95, c.item("amount"), interpolation="lower")
interpolation options:
  • “linear”

  • “lower”

  • “higher”

  • “midpoint”

  • “nearest”

interpolation_to_method: Dict[str, Callable] = {'higher': <function PercentileReducer.percentile_higher>, 'linear': <function PercentileReducer.percentile_linear>, 'lower': <function PercentileReducer.percentile_lower>, 'midpoint': <function PercentileReducer.percentile_midpoint>, 'nearest': <function PercentileReducer.percentile_nearest>}
static percentile_higher(data, quantile)[source]
static percentile_linear(data, quantile)[source]
static percentile_lower(data, quantile)[source]
static percentile_midpoint(data, quantile)[source]
static percentile_nearest(data, quantile)[source]
property post_conversion
class convtools.aggregations.Reduce(to_call_with_2_args: typing.Union[typing.Callable, convtools.base.InlineExpr], *expressions: typing.Tuple[typing.Any, ...], initial: typing.Union[convtools.base._None, typing.Callable, convtools.base.InlineExpr, typing.Any], default: typing.Union[convtools.base._None, typing.Callable, typing.Any] = <convtools.base._None object>, unconditional_init: bool = False, where=None)[source]

Bases: convtools.aggregations.BaseReducer

Defines the reduce operation, which is based on a callable / expression to be used during the aggregation

gen_reduce_code_block(var_agg_data_value: str, reducer_code_input: str, checksum_flag: int, ctx: dict) convtools.aggregations.RBT[source]
initial: Any
class convtools.aggregations.ReduceBlock(reduce_initial: Iterable[str], reduce_two: Iterable[str], reducer_code_input: str, var_agg_data_value: str, checksum_flag: int, unconditional_init: bool)[source]

Bases: object

Represents a section of code of a single reducer

as_key()[source]
consume(reduce_block: convtools.aggregations.RBT) convtools.aggregations.RBT[source]
iter_reduce_initial_lines() Iterable[str][source]
iter_reduce_lines(lines_info_dict) Iterable[str][source]
iter_reduce_two_lines() Iterable[str][source]
var_checksum = 'checksum_'
var_expected_checksum = 'expected_checksum_'
class convtools.aggregations.ReduceBlocks[source]

Bases: Generic[convtools.aggregations.RBT]

Represents a set of reduce blocks

add_block(reduce_block: convtools.aggregations.RBT)[source]
gen_aggregate_code(var_row, var_checksum, var_expected_checksum) convtools.utils.Code[source]
gen_group_by_code(var_row, var_agg_data, var_signature_to_agg_data, code_signature) convtools.utils.Code[source]
iter_blocks() Iterable[convtools.aggregations.RBT][source]
class convtools.aggregations.ReduceConditionalBlock(*args, **kwargs)[source]

Bases: convtools.aggregations.ReduceBlock

Represents a section of code of a single reducer with an incoming condition

class convtools.aggregations.ReduceFuncs[source]

Bases: object

Exposes the list of reduce functions

Array

Aggregates values into array

alias of convtools.aggregations.ArrayReducer

ArrayDistinct

Aggregates distinct values into array, preserves order

alias of convtools.aggregations.ArrayDistinctReducer

ArraySorted

Aggregates values into array, sorting them in the end

alias of convtools.aggregations.SortedArrayReducer

Average = <convtools.aggregations.AverageReducerDispatcher object>

Calculates the arithmetic mean or weighted mean.

Count

Counts objects

alias of convtools.aggregations.CountReducer

CountDistinct

Counts distinct values

alias of convtools.aggregations.CountDistinctReducer

Dict

Aggregates values into dict; dict values are last values per group

alias of convtools.aggregations.DictReducer

DictArray

Aggregates values into dict; dict values are lists of group values

alias of convtools.aggregations.DictArrayReducer

DictArrayDistinct

Aggregates values into dict; dict values are lists of unique group values preserves order

alias of convtools.aggregations.DictArrayDistinctReducer

DictCount

Aggregates values into dict; dict values are numbers of values in groups

alias of convtools.aggregations.DictCountReducer

DictCountDistinct

Aggregates values into dict; dict values are numbers of unique values in groups

alias of convtools.aggregations.DictCountDistinctReducer

DictFirst

Aggregates values into dict; dict values are first values per group

alias of convtools.aggregations.DictFirstReducer

DictLast

Aggregates values into dict; dict values are last values per group

alias of convtools.aggregations.DictLastReducer

DictMax

Aggregates values into dict; dict values are max group values

alias of convtools.aggregations.DictMaxReducer

DictMin

Aggregates values into dict; dict values are min group values

alias of convtools.aggregations.DictMinReducer

DictSum

Aggregates values into dict; dict values are sums of group values, skipping None

alias of convtools.aggregations.DictSumReducer

DictSumOrNone

Aggregates values into dict; dict values are sums of group values, any None makes the total sum None

alias of convtools.aggregations.DictSumOrNoneReducer

First

Stores the first value per group

alias of convtools.aggregations.FirstReducer

Last

Stores the last value per group

alias of convtools.aggregations.LastReducer

Max

Finds max value, skips None

alias of convtools.aggregations.MaxReducer

MaxRow

Finds a row with max value, skips None

alias of convtools.aggregations.MaxRowReducer

Median(*args, **kwargs) convtools.base.BaseConversion

Calculates the median value.

Min

Finds min value, skips None

alias of convtools.aggregations.MinReducer

MinRow

Finds a row with min value, skips None

alias of convtools.aggregations.MinRowReducer

Mode

Calculates the most common value. In case of multiple values, returns the last of them.

alias of convtools.aggregations.ModeReducer

Percentile

Calculates percentile: floats in [0, 100]

alias of convtools.aggregations.PercentileReducer

Sum = <convtools.aggregations.SumReducerDispatcher object>

Calculates the sum, skips false values

SumOrNone

Calculates the sum, any None makes the total sum None

alias of convtools.aggregations.SumOrNoneReducer

TopK

Returns a list of the most frequent values. The resulting list is sorted in descending order of values frequency.

alias of convtools.aggregations.TopReducer

class convtools.aggregations.ReducerDispatcher[source]

Bases: object

class convtools.aggregations.SortedArrayReducer(*args, key=None, reverse=False, **kwargs)[source]

Bases: convtools.aggregations.MultiStatementReducer

Array reducer which sorts the list just once in the end

default: Any = <convtools.base.NaiveConversion object>
expressions: Tuple[Any, ...]
property post_conversion
prepare_first(ctx)[source]
reduce: Tuple[str, ...] = ('%(result)s.append({0})',)
unconditional_init: bool = True
class convtools.aggregations.SumOrNoneReducer(*expressions, initial=<convtools.base._None object>, default=<convtools.base._None object>, where=None)[source]

Bases: convtools.aggregations.MultiStatementReducer

default: Any = <convtools.base.NaiveConversion object>
expressions: Tuple[Any, ...]
prepare_first: Tuple[str, ...] = ('%(result)s = {0}',)
reduce: Tuple[str, ...] = ('if {0} is None:', '    %(result)s = None', 'elif %(result)s is not None:', '    %(result)s = {prev_result} + {0}')
unconditional_init: bool = True
class convtools.aggregations.SumReducer(*expressions, initial=<convtools.base._None object>, default=<convtools.base._None object>, where=None)[source]

Bases: convtools.aggregations.MultiStatementReducer

default: Any = <convtools.base.NaiveConversion object>
expressions: Tuple[Any, ...]
prepare_first: Tuple[str, ...] = ('%(result)s = ({0} or 0)',)
reduce: Tuple[str, ...] = ('%(result)s = {prev_result} + ({0} or 0)',)
unconditional_init: bool = True
class convtools.aggregations.SumReducerDispatcher[source]

Bases: convtools.aggregations.ReducerDispatcher

class convtools.aggregations.TopReducer(k: int, key_conv, *args, **kwargs)[source]

Bases: convtools.aggregations.DictCountReducer

Returns a list of the most frequent values. The resulting list is sorted in descending order of values frequency.

expressions: Tuple[Any, ...]
property post_conversion

convtools.base module

Base and basic conversions are defined here.

class convtools.base.And(*args, default=None)[source]

Bases: convtools.base.OrAndEqBaseConversion

Takes any number of objects, each is to be wrapped with ensure_conversion and generates the code joining every argument with python and expression.

default defines behavior when args is empty
  • if None is empty will raise ValueError

  • false values - results in False

  • true values - results in True

op = ' and '
convtools.base.ApplyFunc(func, args, kwargs) convtools.base.InlineExpr[source]

Shortcut to NaiveConversion(func).apply(args, kwargs)

class convtools.base.BaseCollectionConversion(*items)[source]

Bases: convtools.base.BaseConversion

This is a base conversion of every collection

gen_collection_from_generator(generator_code, code_input, ctx)[source]
gen_collection_from_items_code(joined_items_code, code_input, ctx)[source]
gen_joined_items_code(code_input, ctx)[source]
gen_optional_items_generator_code(code_input, ctx)[source]
self_content_type = 0
class convtools.base.BaseComprehensionConversion(item, *, where=None)[source]

Bases: convtools.base.BaseConversion

This is the base conversion to generate a code, which creates a collection like: list/dict/etc.

gen_generator_code(code_input, ctx)[source]
gen_item_code(code_input, ctx)[source]
sorting_requested = None
class convtools.base.BaseConversion[source]

Bases: Generic[convtools.base.CT]

This is the base class of every conversion (so you are not going to use this directly).

A conversion is a definition of some actions to be done to the input passed as data_ argument.

Conversions are nestable (iteration, calling functions) and chainable (method calling or piping).

Every conversion has many important methods like:
  • gen_converter

  • item, attr, call, call_methods, as_type

  • and_, or_, not_, is_, is not, in_, not_in

  • filter

  • pipe

  • overloaded operators

CONVERTERS_CACHE = '_converters_cache'
class ContentTypes[source]

Bases: object

Defines types of conversion content for bitmask calculations

AGGREGATION = 2
ARG_USAGE = 8
BREAKPOINT = 32
FUNCTION_OF_INPUT = 64
LABEL_USAGE = 16
NEW_LABEL = 4
NONE_USAGE = 128
REDUCER = 1
GENERATED_NAMES = '_generated_names'
NAIVE_TO_WARM_UP = '_naive_to_warm_up'
NAMESPACES = '_name_to_code_input'
class OutputHints[source]

Bases: object

NOT_NONE = 1
PREFIXED_HASH_TO_NAME = '_prefixed_hash_to_name'
add(b) convtools.base.InlineExpr[source]
add_hint(hint: int)[source]
add_label(label_name: Union[str, dict]) convtools.base.BaseConversion[source]

Labels data so it can be reused further:

Basic: >>> c.item(“objects”, 0).add_label(“first”)

Advanced: >>> c.item(“objects”).add_label({ >>> “first”: c.item(0), >>> “count”: c.call_func(len, c.this), >>> }).iter_mut( >>> c.Mut.set_attr(“_first”, c.label(“first”)), >>> c.Mut.set_attr(“_count”, c.label(“count”)), >>> )

Parameters

label_name – a name of the label to be applied or a dict with labels to conversions

Returns

the labeled conversion

Return type

LabelConversion

allowed_symbols = 'abcdefghijklmnopqrstuvwxyz0123456789'
and_(*args, **kwargs) convtools.base.And[source]
and_then(conversion, condition=<class 'bool'>) convtools.base.BaseConversion[source]

Applies conversion if condition is true, otherwise leaves untouched. Condition is bool by default

apply(args, kwargs)[source]

Gets compiled into the code which calls the input with params. Both args and kwargs are wrapped with ensure_conversion.

apply_method(method_name: str, args, kwargs) convtools.base.Call[source]

Gets compiled into the code which calls the method_name method of input with params. It’s a shortcut to (...).attr(method_name).apply(args, kwargs)

as_function_ctx(ctx, as_kwargs=False, args_to_skip=None, for_top_level_converter=False, optimize_naive=False) convtools.base.FunctionCtx[source]
as_type(callable_) convtools.base.Call[source]
attr(*attrs, **kwargs) convtools.base.GetAttr[source]
breakpoint()[source]

Shortcut to Breakpoint(self)

call(*args, **kwargs) convtools.base.Call[source]

Gets compiled into the code which calls the input with params. Each *args and **kwargs are wrapped with ensure_conversion.

call_like(*args, **kwargs)[source]
call_method(method_name: str, *args, **kwargs) convtools.base.Call[source]

Gets compiled into the code which calls the method_name method of input with params. It’s a shortcut to (...).attr(method_name).call(*args, **kwargs)

check_dependency(b)[source]
clone() convtools.base.CT[source]
compile_converter(converter_name: str, code: str, ctx: dict) Callable[source]
cumulative(prepare_first, reduce_two, label_name=None)[source]

Shortcut for Cumulative

cumulative_reset(label_name)[source]

Shortcut for CumulativeReset

depends_on(*args)[source]
div(b) convtools.base.InlineExpr[source]
drop_while(condition) convtools.base.BaseConversion[source]
ensure_conversion(conversion, **kwargs) convtools.base.BaseConversion[source]

Runs ensure_conversion on the input object and adds the resulting conversion to the list of dependencies

eq(b, *args) convtools.base.Eq[source]
exceptions_to_dump_sources = (<class 'Exception'>, <class 'KeyboardInterrupt'>)
execute(*args, debug=None, **kwargs) Any[source]

Shortcut for generating converter and running it

filter(condition_conv, cast=<convtools.base._None object>) convtools.base.BaseConversion[source]

Shortcut for calling convtools.base.FilterConversion on self

flatten() convtools.base.Call[source]

Conversion which calls itertools.chain.from_iterable on self. Returns iterable

floor_div(b) convtools.base.InlineExpr[source]
gen_code_and_update_ctx(code_input, ctx) str[source]

The main method which generates the code and stores necessary info in the context (which will be passed as locals() and globals() on to the exec function). However you should not override this method directly, please implement the _gen_code_and_update_ctx one.

gen_converter(method=False, class_method=False, signature=None, debug=None, converter_name='converter')[source]

Compiles a function which act according to the conversion definition.

Parameters
  • debug (bool) – If True, prints the generated code (formats with black if available). By default: None

  • signature (str) – Defines the signature of the function to be compiled. data_ argument is what going to be used as the input. e.g. signature="self, dt, data_, **kwargs"

  • method (bool) – True is a shortcut for: signature="self, data_"

  • class_method (bool) – True is a shortcut for: signature="cls, data_"

  • converter_name (str) – prefix of the name of the function to be compiled

Returns

The compiled function

gen_name(prefix, ctx, item_to_hash) str[source]

Generates name of variable to be used in the generated code. This also ensures that same items_to_hash will yield same names.

get_dependencies(types=None)[source]
gt(b) convtools.base.InlineExpr[source]
gte(b) convtools.base.InlineExpr[source]
ignores_input() Optional[bool][source]
in_(arg) convtools.base.InlineExpr[source]
is_(arg) convtools.base.InlineExpr[source]
is_dependency_trackable(dependency: convtools.base.BaseConversion)[source]
is_itself_callable() Optional[bool][source]
is_itself_callable_like() Optional[bool][source]
is_not(arg) convtools.base.InlineExpr[source]
item(*args, **kwargs) convtools.base.GetItem[source]
iter(element_conv, *, where=None) convtools.base.BaseConversion[source]

Shortcut for self.pipe(c.generator_comp(element_conv, where=condition))

Parameters
  • element_conv (object) – conversion to be run on each element

  • where (object) – condition inside the comprehension

iter_mut(*mutations: convtools.base.BaseMutation) convtools.base.IterMutConversion[source]

Conversion which results in a generator of mutated elements

Parameters

mutations (BaseMutation) – conversion to be run on each element

iter_windows(width, step=1)[source]

Iterates through an iterable and yields tuples, which are obtained by sliding a windows of a given width, moving it by specified step size

len() convtools.base.BaseConversion[source]

Shortcut for CallFunc(len, self)

lt(b) convtools.base.InlineExpr[source]
lte(b) convtools.base.InlineExpr[source]
mod(b) convtools.base.InlineExpr[source]
mul(b) convtools.base.InlineExpr[source]
neg() convtools.base.InlineExpr[source]
not_() convtools.base.InlineExpr[source]
not_eq(b) convtools.base.InlineExpr[source]
not_in(arg) convtools.base.InlineExpr[source]
or_(*args, **kwargs) convtools.base.Or[source]
output_hints = 0
pipe(next_conversion, *args, label_input=None, label_output=None, **kwargs) convtools.base.BaseConversion[source]

Shortcut for PipeConversion

classmethod replace_word(where: str, word: str, with_what: str) str[source]
self_content_type = 64
sort(key=None, reverse=False) convtools.base.BaseConversion[source]

Shortcut for calling convtools.base.SortConversion on self

sub(b) convtools.base.InlineExpr[source]
take_while(condition) convtools.base.BaseConversion[source]
tap(*mutations: convtools.base.BaseMutation) convtools.base.TapConversion[source]

Allows to tap into the processing of a conversion and mutate it in place. Accepts multiple mutations, order matters.

Parameters

mutations (iterable of BaseMutation) – mutations to process the conversion

to_code(code_input, ctx) Optional[convtools.utils.Code][source]
trackable_dependency = False
used_in_narrow_context = False
valid_pipe_output = True
weight = 185000
class convtools.base.BaseMethodConversion(self_conv)[source]

Bases: convtools.base.BaseConversion

This conversion is required to take into account method calls. We need to preserve the instance we are calling a method on.

e.g. like obj[‘key’] OR obj.func() OR obj.attr1

get_self_and_input_code(code_input: str, ctx: dict) Tuple[str, str][source]
class convtools.base.BaseMutation[source]

Bases: convtools.base.BaseConversion

used_in_narrow_context = True
weight = 1016
class convtools.base.Breakpoint(to_debug)[source]

Bases: convtools.base.BaseConversion

Defines the conversion which wraps another one and puts a breakpoint after it

static debug_func(obj)
self_content_type = 32
class convtools.base.Call(*args, self_conv=<convtools.base._None object>, **kwargs)[source]

Bases: convtools.base.BaseMethodConversion

This conversion writes the code which takes the input code and calls it as a function. It takes both positional and keyword arguments to be passed.

self_content_type = 0
convtools.base.CallFunc(func, *args, **kwargs) convtools.base.Call[source]

Shortcut to NaiveConversion(func).call(*args, **kwargs)

class convtools.base.CodeGenerationOptions[source]

Bases: convtools.utils.BaseOptions

reducers_run_stage = None
class convtools.base.CodeGenerationOptionsCtx(*args, **kwds)[source]

Bases: convtools.utils.BaseCtx

options_cls

alias of convtools.base.CodeGenerationOptions

exception convtools.base.ConversionException[source]

Bases: Exception

class convtools.base.ConverterOptions[source]

Bases: convtools.utils.BaseOptions

Converter options (+ see default values below):

  • debug = False - same as .gen_converter(debug=...)

debug = False
class convtools.base.ConverterOptionsCtx(*args, **kwds)[source]

Bases: convtools.utils.BaseCtx

Thread-safe context to manage options.

Example:

with ConverterOptionsCtx() as options:
    options.debug = True
    # ...
options_cls

alias of convtools.base.ConverterOptions

class convtools.base.Cumulative(parent: Any, prepare_first: Any, reduce_two: Any, label_name: Optional[str] = None)[source]

Bases: convtools.base.BaseConversion

A conversion which calculates cumulative values. The main use case is using it within iterables:

>>> assert (
>>>     c.iter(c.cumulative(c.this, c.this + c.PREV))
>>>     .as_type(list)
>>>     .execute([0, 1, 2, 3, 4])
>>> ) == [0, 1, 3, 6, 10]
PREV = <convtools.base.LazyEscapedString object>
class convtools.base.CumulativeReset(parent: Any, label_name: str)[source]

Bases: convtools.base.BaseConversion

A conversion which resets cumulative values to their initial states. The main use case is within nested iterables:

>>> assert (
>>>     c.iter(
>>>         c.cumulative_reset("abc")
>>>         .iter(c.cumulative(c.this, c.this + c.PREV, label_name="abc"))
>>>         .as_type(list)
>>>     )
>>>     .as_type(list)
>>>     .execute([[0, 1, 2], [3, 4]])
>>> ) == [[0, 1, 3], [3, 7]]
class convtools.base.Dict(*key_value_pairs)[source]

Bases: convtools.base.BaseCollectionConversion

Gets compiled into the code which generates a dict

gen_collection_from_generator(generator_code, code_input, ctx)[source]
gen_collection_from_items_code(joined_items_code, code_input, ctx)[source]
gen_joined_items_code(code_input, ctx)[source]
weight = 1850
class convtools.base.DictComp(key, value, *, where=None)[source]

Bases: convtools.base.BaseComprehensionConversion

Generates python dict comprehension code.

filter(condition_conv, cast=<convtools.base._None object>)[source]

Shortcut for calling convtools.base.FilterConversion on self

gen_item_code(code_input, ctx)[source]
sort(key=None, reverse=False) convtools.base.BaseConversion[source]

Shortcut for calling convtools.base.SortConversion on self

class convtools.base.DropWhile(condition)[source]

Bases: convtools.base.BaseConversion

convtools implementation of itertools.dropwhile

class convtools.base.Eq(*args, default=None)[source]

Bases: convtools.base.OrAndEqBaseConversion

Takes any number of objects, each is to be wrapped with ensure_conversion and generates the code joining every argument with python `` == `` operator

default defines behavior when args is empty
  • if None is empty will raise ValueError

  • false values - results in False

  • true values - results in True

op = ' == '
class convtools.base.EscapedString(s: str)[source]

Bases: convtools.base.BaseConversion

Defines the conversion which returns the result of running the python code, passed to init

self_content_type = 0
weight = 100
class convtools.base.FilterConversion(condition_conv, cast=<convtools.base._None object>)[source]

Bases: convtools.base.BaseConversion

Generates the code to iterate the input, taking items for which the provided conversion resolves to a truth value.

as_type(callable_)[source]
filter(condition_conv, cast=<convtools.base._None object>)[source]

Shortcut for calling convtools.base.FilterConversion on self

sort(key=None, reverse=False)[source]

Shortcut for calling convtools.base.SortConversion on self

class convtools.base.FunctionCtx(conversion, ctx, args_as_def_names, kwargs_as_def_names, args_to_pass, kwargs_to_pass, namespace_ctx, optimize_naive)[source]

Bases: object

Defines a code generation context for a conversion, which is to help with wrapping a conversion code in a function so that all required arguments are both defines as parameters and passed properly. It also helps with adding additional parameters.

Another important thing is that some parameters like LazyEscapedString are renamed and replaced with arguments of the new function. So it ensures (see prevent_rendering_while_active) that those lazy strings are not generating code while we are building inner function code (they just won’t be available there).

add_arg(def_name, arg_to_pass=None, left=False)[source]
add_kwarg(def_name, arg_to_pass=None, left=False)[source]
call_with_all_args(conversion)[source]
gen_conversion(name, code)[source]
gen_function(name, code)[source]
get_def_all_args_code()[source]
class convtools.base.GeneratorComp(item, *, where=None)[source]

Bases: convtools.base.BaseComprehensionConversion

Generates python generator comprehension code.

as_type(callable_)[source]
class convtools.base.GetAttr(*indexes, default=<convtools.base._None object>, self_conv=<convtools.base._None object>)[source]

Bases: convtools.base.GetItem

GetAttr gets compiled into the code which runs getattr. If called without params, just returns the input.

If an index is a conversion itself, then it is being calculated against an input.

caching_is_possible = False
get_hardcoded_version()[source]
prefix = 'attr_or_default'
template = '\ndef {converter_name}({code_args}):\n    try:\n        return {get_or_default_code}\n    except AttributeError:\n        return {default_code}\n'
valid_attr = re.compile("^'[A-Za-z_][a-zA-Z0-9_]*'$")
weight = 368
wrap_path_item(code_input, path_item)[source]
class convtools.base.GetItem(*indexes, default=<convtools.base._None object>, self_conv=<convtools.base._None object>)[source]

Bases: convtools.base.BaseMethodConversion

GetItem gets compiled into the code which does dictionary/index lookups.

If called without params, just returns the input.

If an index is a conversion itself, then it is being calculated against an input.

caching_is_possible = True
get_hardcoded_version()[source]
get_or_default_functions = [<function get_1_or_default>, <function get_2_or_default>, <function get_3_or_default>]
naive_or_unsafe_conversions = (<class 'convtools.base.NaiveConversion'>, <class 'convtools.base.EscapedString'>, <class 'convtools.base.InlineExpr'>)
prefix = 'item_or_default'
template = '\ndef {converter_name}({code_args}):\n    try:\n        return {get_or_default_code}\n    except (TypeError, KeyError, IndexError):\n        return {default_code}\n'
weight = 381
wrap_path_item(code_input, path_item)[source]
class convtools.base.If(condition=True, if_true=<convtools.base._None object>, if_false=<convtools.base._None object>, no_input_caching=False)[source]

Bases: convtools.base.BaseConversion

Generates the if expression code.

Checks the code of the input, if it doesn’t seem to be complex, then just proceeds with it as is. If it’s not simple (some index/attribute lookups or function calls are in there), then it caches the input for further reuse in if_true and if_false clauses.

self_content_type = 0
weight = 100
class convtools.base.IfMultiple(*condition_to_value_pairs, else_)[source]

Bases: convtools.base.BaseConversion

Builds a short-circuit conversion, which evaluates multiple conversions and stops at first true one:

>>> converter = c.if_multiple(
>>>     (c.this < 10, c.this / 2),
>>>     (c.this == 10, None),
>>>     else_=c.this * 2
>>> ).gen_converter()

is equivalent of:

>>> def converter(data_):
>>>     if data_ < 10:
>>>         return data_ / 2
>>>     if data_ == 10:
>>>         return None
>>>     return data_ * 2
self_content_type = 0
class convtools.base.InlineExpr(code_str, weight=185000)[source]

Bases: convtools.base.BaseConversion

This conversion allows to avoid function call overhead. It inlines a raw python code expression into the code of resulting conversion.

call_like(*args, **kwargs)[source]
is_itself_callable_like() Optional[bool][source]
pass_args(*args, **kwargs)[source]

The method passes arguments to the code to be inlined.

Parameters
Returns

Clone of the conversion after arguments are passed.

Return type

InlineExpr

self_content_type = 0
class convtools.base.InputArg(name: str)[source]

Bases: convtools.base.BaseConversion

Allows to use arguments passed into the compiled converter.

Unless the signature argument is passed to gen_converter function, all input arguments used in the conversion definition will be expected as keyword-only arguments (affecting the resulting converter signature).

self_content_type = 8
trackable_dependency = True
weight = 100
class convtools.base.IterMutConversion(obj, *mutations: convtools.base.BaseMutation)[source]

Bases: convtools.base.TapConversion

This conversion generates the code which iterates and mutates the elements in-place. The result is a generator. IterMutConversion takes any number of mutations

class convtools.base.LabelConversion(label_name: str)[source]

Bases: convtools.base.BaseConversion

Allows to reference a conversion result by label, after it was cached by PipeConversion or BaseConversion.add_label.

labels_code_name = '_labels'
self_content_type = 16
weight = 381
class convtools.base.LazyEscapedString(name)[source]

Bases: convtools.base.BaseConversion

A lazy named conversion which allows to build a conversion on the outside to then generate some code around it (properly passing required args, etc.)

self_content_type = 0
trackable_dependency = True
weight = 100
class convtools.base.List(*items)[source]

Bases: convtools.base.BaseCollectionConversion

Gets compiled into the code which generates a list

gen_collection_from_generator(generator_code, code_input, ctx)[source]
gen_collection_from_items_code(joined_items_code, code_input, ctx)[source]
weight = 837
class convtools.base.ListComp(item, *, where=None)[source]

Bases: convtools.base.BaseComprehensionConversion

Generates python list comprehension code.

filter(condition_conv, cast=<convtools.base._None object>)[source]

Shortcut for calling convtools.base.FilterConversion on self

sort(key=None, reverse=False) convtools.base.BaseConversion[source]

Shortcut for calling convtools.base.SortConversion on self

class convtools.base.NaiveConversion(value: Any, name_prefix='v')[source]

Bases: convtools.base.BaseConversion

Naive conversion gets compiled into the code, which just returns the value it’s been initialized with. Allows to make any object available inside other conversions.

is_itself_callable() Optional[bool][source]
is_itself_callable_like() Optional[bool][source]
self_content_type = 0
types_to_repr = {<class 'int'>, <class 'NoneType'>, <class 'bool'>}
weight = 100
class convtools.base.Namespace(conversion: Any, name_to_code: Dict[str, Optional[Union[bool, str]]])[source]

Bases: convtools.base.BaseConversion

Wrapping conversion which isolates LazyEscapedString (parent conversions won’t detect them as dependencies) and defines code inputs for them

is_dependency_trackable(dependency: convtools.base.BaseConversion)[source]
self_content_type = 0
weight = 0
class convtools.base.NamespaceControlledUnit(namespace_ctx: convtools.base.NamespaceCtx, conversion)[source]

Bases: convtools.base.BaseConversion

Wrapping conversion which prevents the inner one from being rendered while it is inside the parent NamespaceCtx

conversion
namespace_ctx
weight = 0
class convtools.base.NamespaceCtx(name_to_code: Dict[str, str], ctx)[source]

Bases: object

Context manager which defines code inputs for LazyEscapedString

NAMESPACES = '_name_to_code_input'
active = False
ctx = None
classmethod name_to_code(ctx) Dict[str, str][source]
prevent_rendering_while_active(conversion)[source]
class convtools.base.Not(arg)[source]

Bases: convtools.base.BaseConversion

Conversion which applies not operator to its input

self_content_type = 0
weight = 255
class convtools.base.OptionalCollectionItem(conversion, skip_value=None, skip_if=<convtools.base._None object>, keep_if=<convtools.base._None object>)[source]

Bases: convtools.base.BaseConversion

Wrapping conversion which makes key/value/item of a collection optional.

valid_pipe_output = False
class convtools.base.Or(*args, default=None)[source]

Bases: convtools.base.OrAndEqBaseConversion

Takes any number of objects, each is to be wrapped with ensure_conversion and generates the code joining every argument with python or expression

default defines behavior when args is empty
  • if None is empty will raise ValueError

  • false values - results in False

  • true values - results in True

op = ' or '
class convtools.base.OrAndEqBaseConversion(*args, default=None)[source]

Bases: convtools.base.BaseConversion

Base class of Or/And/Eq operator conversions

op = ''
self_content_type = 0
weight = 255
class convtools.base.PipeConversion(what, where, *args, label_input=None, label_output=None, **kwargs)[source]

Bases: convtools.base.BaseConversion

Passes the result of one conversion as an input to another. If next_conversion is callable, it gets called with the previous result passed as the first param.

Supports predicate/sorting/type casting push down (each is directly applied to the where conversion.

Supports labeling both pipe input and output data (allows to apply conversions before labeling).

as_type(callable_)[source]
filter(condition_conv, cast=<convtools.base._None object>)[source]

Shortcut for calling convtools.base.FilterConversion on self

function_call_threshold = 1351.28
replace(where)[source]
sort(key=None, reverse=False)[source]

Shortcut for calling convtools.base.SortConversion on self

weight = 0
class convtools.base.Set(*items)[source]

Bases: convtools.base.BaseCollectionConversion

Gets compiled into the code which generates a set

gen_collection_from_generator(generator_code, code_input, ctx)[source]
gen_collection_from_items_code(joined_items_code, code_input, ctx)[source]
weight = 1744
class convtools.base.SetComp(item, *, where=None)[source]

Bases: convtools.base.BaseComprehensionConversion

Generates python set comprehension code (obviously non-sortable)

filter(condition_conv, cast=<convtools.base._None object>)[source]

Shortcut for calling convtools.base.FilterConversion on self

class convtools.base.SortConversion(key=None, reverse=False)[source]

Bases: convtools.base.BaseConversion

Generates the code to sort the input.

class convtools.base.TakeWhile(condition)[source]

Bases: convtools.base.BaseConversion

convtools implementation of itertools.takewhile

filter(condition_conv, cast=<convtools.base._None object>)[source]

Shortcut for calling convtools.base.FilterConversion on self

class convtools.base.TapConversion(obj, *mutations: convtools.base.BaseMutation)[source]

Bases: convtools.base.BaseConversion

This conversion generates the code which mutates the input data in-place. TapConversion takes any number of mutations

weight = 1016
class convtools.base.ThisConversion[source]

Bases: convtools.base.BaseConversion

Defines the conversion which just returns the input.

Also, provided that you use this inside comprehension conversions, it references an item from an iterator.

weight = 0
class convtools.base.Tuple(*items)[source]

Bases: convtools.base.BaseCollectionConversion

Gets compiled into the code which generates a tuple

gen_collection_from_generator(generator_code, code_input, ctx)[source]
gen_collection_from_items_code(joined_items_code, code_input, ctx)[source]
weight = 594
class convtools.base.TupleComp(item, *, where=None)[source]

Bases: convtools.base.BaseComprehensionConversion

Generates python tuple comprehension code.

filter(condition_conv, cast=<convtools.base._None object>)[source]

Shortcut for calling convtools.base.FilterConversion on self

sort(key=None, reverse=False) convtools.base.BaseConversion[source]

Shortcut for calling convtools.base.SortConversion on self

convtools.base.debug_func(obj)[source]
convtools.base.ensure_conversion(conversion: Any, explicitly_allowed_cls=None) convtools.base.BaseConversion[source]
Helps to define conversions based on its type:
  • any conversion is returned untouched

  • list/dict/set/tuple collections are wrapped with c.list, c.dict, c.set, c.tuple (see below). If it’s not desired, use c.naive instead

  • slice gets recreated, each slice.start, slice.stop, slice.step is wrapped with ensure_conversion

  • everything else is wrapped with c.naive (see below)

Parameters

conversion (object) – any object

Returns

a conversion based on conversion type:

Return type

BaseConversion

convtools.base.get_1_or_default(data_, key_, default_)[source]
convtools.base.get_2_or_default(data_, key_1, key_2, default_)[source]
convtools.base.get_3_or_default(data_, key_1, key_2, key_3, default_)[source]
convtools.base.iskeyword()

x.__contains__(y) <==> y in x.

convtools.base.var_name_from_string(s)[source]

convtools.chunks module

Provides conversions for slicing iterables into chunks.

class convtools.chunks.BaseChunkBy[source]

Bases: convtools.base.BaseConversion

aggregate(*args, **kwargs) convtools.base.BaseConversion[source]
class convtools.chunks.ChunkBy(*by, size: Optional[int] = None)[source]

Bases: convtools.chunks.BaseChunkBy

Slices iterable into chunks by element values and/or size of chunks.

>>> # simple #1
>>> c.chunk_by(size=1000)
>>>
>>> # simple #2
>>> c.chunk_by(c.item("x"))
>>>
>>> # with aggregate
>>> c.chunk_by(
>>>     c.item("x"),
>>>     size=1000
>>> ).aggregate({
>>>     "x": c.ReduceFuncs.Last(c.item("x")),
>>>     "y": c.ReduceFuncs.Sum(c.item("y")),
>>> })
>>> # example #1
>>> c.chunk_by_condition(c.CHUNK.len() < 100)
>>>
>>> # example #2
>>> c.chunk_by_condition(
>>>     c.and_(
>>>         c.CHUNK.item(-1) == c.this,
>>>         c.CHUNK.item(-1) - c.this < 100
>>>     )
>>> )
>>> # with aggregate
>>> c.chunk_by_condition(
>>>     c.CHUNK.len() < 100
>>> ).aggregate({
>>>     "x": c.ReduceFuncs.Last(c.item("x")),
>>>     "y": c.ReduceFuncs.Sum(c.item("y")),
>>> })

It also provides a shortcut for running convtools.aggregations.Aggregate on chunks.

class convtools.chunks.ChunkByCondition(condition)[source]

Bases: convtools.chunks.BaseChunkBy

Slices iterable into chunks based on condition, which is a function of a current chunk and a current element.

>>> # example #1
>>> c.chunk_by_condition(c.CHUNK.len() < 100)
>>>
>>> # example #2
>>> c.chunk_by_condition(
>>>     c.and_(
>>>         c.CHUNK.item(-1) == c.this,
>>>         c.CHUNK.item(-1) - c.this < 100
>>>     )
>>> )
>>> # with aggregate
>>> c.chunk_by_condition(
>>>     c.CHUNK.len() < 100
>>> ).aggregate({
>>>     "x": c.ReduceFuncs.Last(c.item("x")),
>>>     "y": c.ReduceFuncs.Sum(c.item("y")),
>>> })

It also provides a shortcut for running convtools.aggregations.Aggregate on chunks.

CHUNK = <convtools.base.LazyEscapedString object>

convtools.columns module

Implements base conversions to reference and define columns in table conversions

class convtools.columns.ColumnChanges[source]

Bases: object

MUTATE = 4
REARRANGE = 2
RENAME = 1
class convtools.columns.ColumnDef(name: str, index: Optional[Any], conversion: Optional[convtools.base.BaseConversion])[source]

Bases: object

Defines a column within a table conversion. It holds the following:

  • name of the column in the output

  • index of the column in the input in simple cases, otherwise None

  • conversion to obtain the value from the input, None in simple cases

as_tuple()[source]
conversion
index
is_same_as(other: convtools.columns.ColumnDef) bool[source]
name
class convtools.columns.ColumnRef(name: str, id_=None)[source]

Bases: convtools.base.BaseConversion

Helper conversion, which allows to reference a column of a table conversion

set_index(index: Union[str, int])[source]
trackable_dependency = True
class convtools.columns.MetaColumns(duplicate_columns='raise')[source]

Bases: object

A helper container for naming & keeping column definitions

add(name, index, conversion)[source]
drop(*column_names: str) convtools.columns.MetaColumns[source]
get_name_to_column() Dict[str, convtools.columns.ColumnDef][source]
is_same_as(other: convtools.columns.MetaColumns)[source]
take(*column_names) convtools.columns.MetaColumns[source]

convtools.conversion module

The main module exposing public API (via conversion object)

class convtools.conversion.Conversion[source]

Bases: object

The object, which exposes public API

from convtools import conversion as c

convert = c.aggregate(
    c.ReduceFuncs.DictSum(
        c.item("name"),
        c.item("value")
    )
).gen_converter(debug=True)

assert convert([
    {"name": "Bob", "value": 10},
    {"name": "Bob", "value": 7},
    {"name": "Ron", "value": 3},
]) == {'Bob': 17, 'Ron': 3}
class BaseConversion

Bases: Generic[convtools.base.CT]

This is the base class of every conversion (so you are not going to use this directly).

A conversion is a definition of some actions to be done to the input passed as data_ argument.

Conversions are nestable (iteration, calling functions) and chainable (method calling or piping).

Every conversion has many important methods like:
  • gen_converter

  • item, attr, call, call_methods, as_type

  • and_, or_, not_, is_, is not, in_, not_in

  • filter

  • pipe

  • overloaded operators

CONVERTERS_CACHE = '_converters_cache'
class ContentTypes

Bases: object

Defines types of conversion content for bitmask calculations

AGGREGATION = 2
ARG_USAGE = 8
BREAKPOINT = 32
FUNCTION_OF_INPUT = 64
LABEL_USAGE = 16
NEW_LABEL = 4
NONE_USAGE = 128
REDUCER = 1
GENERATED_NAMES = '_generated_names'
NAIVE_TO_WARM_UP = '_naive_to_warm_up'
NAMESPACES = '_name_to_code_input'
class OutputHints

Bases: object

NOT_NONE = 1
PREFIXED_HASH_TO_NAME = '_prefixed_hash_to_name'
add(b) convtools.base.InlineExpr
add_hint(hint: int)
add_label(label_name: Union[str, dict]) convtools.base.BaseConversion

Labels data so it can be reused further:

Basic: >>> c.item(“objects”, 0).add_label(“first”)

Advanced: >>> c.item(“objects”).add_label({ >>> “first”: c.item(0), >>> “count”: c.call_func(len, c.this), >>> }).iter_mut( >>> c.Mut.set_attr(“_first”, c.label(“first”)), >>> c.Mut.set_attr(“_count”, c.label(“count”)), >>> )

Parameters

label_name – a name of the label to be applied or a dict with labels to conversions

Returns

the labeled conversion

Return type

LabelConversion

allowed_symbols = 'abcdefghijklmnopqrstuvwxyz0123456789'
and_(*args, **kwargs) convtools.base.And
and_then(conversion, condition=<class 'bool'>) convtools.base.BaseConversion

Applies conversion if condition is true, otherwise leaves untouched. Condition is bool by default

apply(args, kwargs)

Gets compiled into the code which calls the input with params. Both args and kwargs are wrapped with ensure_conversion.

apply_method(method_name: str, args, kwargs) convtools.base.Call

Gets compiled into the code which calls the method_name method of input with params. It’s a shortcut to (...).attr(method_name).apply(args, kwargs)

as_function_ctx(ctx, as_kwargs=False, args_to_skip=None, for_top_level_converter=False, optimize_naive=False) convtools.base.FunctionCtx
as_type(callable_) convtools.base.Call
attr(*attrs, **kwargs) convtools.base.GetAttr
breakpoint()

Shortcut to Breakpoint(self)

call(*args, **kwargs) convtools.base.Call

Gets compiled into the code which calls the input with params. Each *args and **kwargs are wrapped with ensure_conversion.

call_like(*args, **kwargs)
call_method(method_name: str, *args, **kwargs) convtools.base.Call

Gets compiled into the code which calls the method_name method of input with params. It’s a shortcut to (...).attr(method_name).call(*args, **kwargs)

check_dependency(b)
clone() convtools.base.CT
compile_converter(converter_name: str, code: str, ctx: dict) Callable
cumulative(prepare_first, reduce_two, label_name=None)

Shortcut for Cumulative

cumulative_reset(label_name)

Shortcut for CumulativeReset

depends_on(*args)
div(b) convtools.base.InlineExpr
drop_while(condition) convtools.base.BaseConversion
ensure_conversion(conversion, **kwargs) convtools.base.BaseConversion

Runs ensure_conversion on the input object and adds the resulting conversion to the list of dependencies

eq(b, *args) convtools.base.Eq
exceptions_to_dump_sources = (<class 'Exception'>, <class 'KeyboardInterrupt'>)
execute(*args, debug=None, **kwargs) Any

Shortcut for generating converter and running it

filter(condition_conv, cast=<convtools.base._None object>) convtools.base.BaseConversion

Shortcut for calling convtools.base.FilterConversion on self

flatten() convtools.base.Call

Conversion which calls itertools.chain.from_iterable on self. Returns iterable

floor_div(b) convtools.base.InlineExpr
gen_code_and_update_ctx(code_input, ctx) str

The main method which generates the code and stores necessary info in the context (which will be passed as locals() and globals() on to the exec function). However you should not override this method directly, please implement the _gen_code_and_update_ctx one.

gen_converter(method=False, class_method=False, signature=None, debug=None, converter_name='converter')

Compiles a function which act according to the conversion definition.

Parameters
  • debug (bool) – If True, prints the generated code (formats with black if available). By default: None

  • signature (str) – Defines the signature of the function to be compiled. data_ argument is what going to be used as the input. e.g. signature="self, dt, data_, **kwargs"

  • method (bool) – True is a shortcut for: signature="self, data_"

  • class_method (bool) – True is a shortcut for: signature="cls, data_"

  • converter_name (str) – prefix of the name of the function to be compiled

Returns

The compiled function

gen_name(prefix, ctx, item_to_hash) str

Generates name of variable to be used in the generated code. This also ensures that same items_to_hash will yield same names.

get_dependencies(types=None)
gt(b) convtools.base.InlineExpr
gte(b) convtools.base.InlineExpr
ignores_input() Optional[bool]
in_(arg) convtools.base.InlineExpr
is_(arg) convtools.base.InlineExpr
is_dependency_trackable(dependency: convtools.base.BaseConversion)
is_itself_callable() Optional[bool]
is_itself_callable_like() Optional[bool]
is_not(arg) convtools.base.InlineExpr
item(*args, **kwargs) convtools.base.GetItem
iter(element_conv, *, where=None) convtools.base.BaseConversion

Shortcut for self.pipe(c.generator_comp(element_conv, where=condition))

Parameters
  • element_conv (object) – conversion to be run on each element

  • where (object) – condition inside the comprehension

iter_mut(*mutations: convtools.base.BaseMutation) convtools.base.IterMutConversion

Conversion which results in a generator of mutated elements

Parameters

mutations (BaseMutation) – conversion to be run on each element

iter_windows(width, step=1)

Iterates through an iterable and yields tuples, which are obtained by sliding a windows of a given width, moving it by specified step size

len() convtools.base.BaseConversion

Shortcut for CallFunc(len, self)

lt(b) convtools.base.InlineExpr
lte(b) convtools.base.InlineExpr
mod(b) convtools.base.InlineExpr
mul(b) convtools.base.InlineExpr
neg() convtools.base.InlineExpr
not_() convtools.base.InlineExpr
not_eq(b) convtools.base.InlineExpr
not_in(arg) convtools.base.InlineExpr
or_(*args, **kwargs) convtools.base.Or
output_hints = 0
pipe(next_conversion, *args, label_input=None, label_output=None, **kwargs) convtools.base.BaseConversion

Shortcut for PipeConversion

classmethod replace_word(where: str, word: str, with_what: str) str
self_content_type = 64
sort(key=None, reverse=False) convtools.base.BaseConversion

Shortcut for calling convtools.base.SortConversion on self

sub(b) convtools.base.InlineExpr
take_while(condition) convtools.base.BaseConversion
tap(*mutations: convtools.base.BaseMutation) convtools.base.TapConversion

Allows to tap into the processing of a conversion and mutate it in place. Accepts multiple mutations, order matters.

Parameters

mutations (iterable of BaseMutation) – mutations to process the conversion

to_code(code_input, ctx) Optional[convtools.utils.Code]
trackable_dependency = False
used_in_narrow_context = False
valid_pipe_output = True
weight = 185000
CHUNK = <convtools.base.LazyEscapedString object>
class CodeGenerationOptionsCtx(*args, **kwds)

Bases: convtools.utils.BaseCtx

options_cls

alias of convtools.base.CodeGenerationOptions

exception ConversionException

Bases: Exception

LEFT = <convtools.joins.LeftJoinCondition object>
Mut

Shortcut to Mutations

alias of convtools.mutations.Mutations

OptionsCtx

alias of convtools.base.ConverterOptionsCtx

PREV = <convtools.base.LazyEscapedString object>
RIGHT = <convtools.joins.RightJoinCondition object>
class ReduceFuncs

Bases: object

Exposes the list of reduce functions

Array

alias of convtools.aggregations.ArrayReducer

ArrayDistinct

alias of convtools.aggregations.ArrayDistinctReducer

ArraySorted

alias of convtools.aggregations.SortedArrayReducer

Average = <convtools.aggregations.AverageReducerDispatcher object>

Calculates the arithmetic mean or weighted mean.

Count

alias of convtools.aggregations.CountReducer

CountDistinct

alias of convtools.aggregations.CountDistinctReducer

Dict

alias of convtools.aggregations.DictReducer

DictArray

alias of convtools.aggregations.DictArrayReducer

DictArrayDistinct

alias of convtools.aggregations.DictArrayDistinctReducer

DictCount

alias of convtools.aggregations.DictCountReducer

DictCountDistinct

alias of convtools.aggregations.DictCountDistinctReducer

DictFirst

alias of convtools.aggregations.DictFirstReducer

DictLast

alias of convtools.aggregations.DictLastReducer

DictMax

alias of convtools.aggregations.DictMaxReducer

DictMin

alias of convtools.aggregations.DictMinReducer

DictSum

alias of convtools.aggregations.DictSumReducer

DictSumOrNone

alias of convtools.aggregations.DictSumOrNoneReducer

First

alias of convtools.aggregations.FirstReducer

Last

alias of convtools.aggregations.LastReducer

Max

alias of convtools.aggregations.MaxReducer

MaxRow

alias of convtools.aggregations.MaxRowReducer

Median(*args, **kwargs) convtools.base.BaseConversion
Min

alias of convtools.aggregations.MinReducer

MinRow

alias of convtools.aggregations.MinRowReducer

Mode

alias of convtools.aggregations.ModeReducer

Percentile

alias of convtools.aggregations.PercentileReducer

Sum = <convtools.aggregations.SumReducerDispatcher object>

Calculates the sum, skips false values

SumOrNone

alias of convtools.aggregations.SumOrNoneReducer

TopK

alias of convtools.aggregations.TopReducer

static aggregate(*args, **kwargs) convtools.base.BaseConversion

Shortcut for GroupBy().aggregate(*args, **kwargs)

and_

alias of convtools.base.And

and_then(conversion, condition=<class 'bool'>) convtools.base.BaseConversion

Shortcut to convtools.base.BaseConversion.and_then

apply(args, kwargs)[source]
static apply_func(func, args, kwargs) convtools.base.InlineExpr

Shortcut to NaiveConversion(func).apply(args, kwargs)

attr

alias of convtools.base.GetAttr

breakpoint()[source]

c.this.breakpoint() shortcut

call(*args, **kwargs)[source]
static call_func(func, *args, **kwargs) convtools.base.Call

Shortcut to NaiveConversion(func).call(*args, **kwargs)

chunk_by

alias of convtools.chunks.ChunkBy

chunk_by_condition

alias of convtools.chunks.ChunkByCondition

col

alias of convtools.columns.ColumnRef

cumulative(prepare_first, reduce_two, label_name=None)[source]
cumulative_reset(label_name)[source]
dict

alias of convtools.base.Dict

dict_comp

alias of convtools.base.DictComp

drop_while

alias of convtools.base.DropWhile

eq

alias of convtools.base.Eq

escaped_string

alias of convtools.base.EscapedString

filter

alias of convtools.base.FilterConversion

flatten()[source]

c.this.flatten() shortcut

generator_comp

alias of convtools.base.GeneratorComp

group_by

alias of convtools.aggregations.GroupBy

if_

alias of convtools.base.If

if_multiple

alias of convtools.base.IfMultiple

inline_expr

alias of convtools.base.InlineExpr

input_arg

Shortcut to InputArg

alias of convtools.base.InputArg

item

alias of convtools.base.GetItem

iter

alias of convtools.base.GeneratorComp

iter_mut(*args, **kwargs)[source]
iter_windows(*args, **kwargs)[source]

Iterates through an iterable and yields tuples, which are obtained by sliding a windows of a given width, moving it by specified step size

join

alias of convtools.joins.JoinConversion

label

alias of convtools.base.LabelConversion

list

Shortcut to List

alias of convtools.base.List

list_comp

alias of convtools.base.ListComp

max(arg1, arg2, *args)[source]

c.call_func(max, …) shortcut

min(arg1, arg2, *args)[source]

c.call_func(min, …) shortcut

naive

alias of convtools.base.NaiveConversion

not_

alias of convtools.base.Not

optional

alias of convtools.base.OptionalCollectionItem

or_

alias of convtools.base.Or

reduce(to_call_with_2_args, *args, **kwargs)[source]
repeat(obj, times=None)[source]

shortcut to call itertools.repeat

set

alias of convtools.base.Set

set_comp

alias of convtools.base.SetComp

sort

alias of convtools.base.SortConversion

take_while

alias of convtools.base.TakeWhile

tap(*args, **kwargs)[source]
this = <convtools.base.ThisConversion object>
tuple

alias of convtools.base.Tuple

tuple_comp

alias of convtools.base.TupleComp

zip(*args, **kwargs)[source]

Conversion which calls zip on conversions.

Parameters
  • args – conversions to zip - returns tuples

  • kwargs – named conversions to zip - returns dicts

convtools.heuristics module

This module is to contain helpers, which collect info about python code execution

class convtools.heuristics.Weights[source]

Bases: object

ATTR_LOOKUP = 368
DICT_INIT = 1850
DICT_LOOKUP = 381
FUNCTION_CALL = 1016
LIST_INIT = 837
LOGICAL = 255
MATH_SIMPLE = 392
SET_INIT = 1744
STEP = 100
TUPLE_INIT = 594
UNPREDICTABLE = 185000
convtools.heuristics.print_new_weights(sleep_time=0.2)[source]

convtools.joins module

This module brings join functionality to the library

class convtools.joins.JoinCondition[source]

Bases: convtools.base.LazyEscapedString

NAME: str
col(column_name: str) convtools.base.BaseConversion[source]
class convtools.joins.JoinConversion(left_conversion: convtools.base.BaseConversion, right_conversion: convtools.base.BaseConversion, condition: convtools.base.BaseConversion, how='inner')[source]

Bases: convtools.base.BaseConversion

Generates conversion which joins left_conversion and right_conversion using condition. The result is a generator of joined pairs

Parameters
  • left_conversion (BaseConversion) – left collection to join

  • right_conversion (BaseConversion) – right collection to join

  • condition (BaseConversion) – join condition. If is True, results in cross join

  • how (str) – one of the following: "inner", "left", "right", "outer"

self_content_type = 192
classmethod validate_how(how: str)[source]
exception convtools.joins.JoinException[source]

Bases: Exception

class convtools.joins.LeftJoinCondition[source]

Bases: convtools.joins.JoinCondition

NAME: str = 'left_row'
class convtools.joins.RightJoinCondition[source]

Bases: convtools.joins.JoinCondition

NAME: str = 'right_row'

convtools.mutations module

This module brings in-place mutations

class convtools.mutations.BaseIndexMutation(index)[source]

Bases: convtools.base.BaseMutation

class convtools.mutations.BaseNameValueMutation(name, value)[source]

Bases: convtools.base.BaseMutation

A base in-place mutation where name and value are needed to define it

class convtools.mutations.Custom(conversion)[source]

Bases: convtools.base.BaseMutation

Mutation to be used in conjunction with tap method. Runs the code, defined by the conversion argument and returns the input as is.

class convtools.mutations.DelAttr(index)[source]

Bases: convtools.mutations.BaseIndexMutation

class convtools.mutations.DelItem(index)[source]

Bases: convtools.mutations.BaseIndexMutation

class convtools.mutations.Mutations[source]

Bases: object

Mutations to be used in conjunction with the conversion tap method.

custom

Runs the code, defined by the conversion argument and returns the input as is.

alias of convtools.mutations.Custom

del_attr

Deletes an attribute from an object.

alias of convtools.mutations.DelAttr

del_item

Pops a key from a dict.

alias of convtools.mutations.DelItem

set_attr

Sets a value of an attribute.

alias of convtools.mutations.SetAttr

set_item

Sets a value by key.

alias of convtools.mutations.SetItem

class convtools.mutations.SetAttr(name, value)[source]

Bases: convtools.mutations.BaseNameValueMutation

class convtools.mutations.SetItem(name, value)[source]

Bases: convtools.mutations.BaseNameValueMutation

convtools.utils module

Helpers live here: like: - recently used cache - options ctx manager

class convtools.utils.BaseCtx(*args, **kwds)[source]

Bases: Generic[convtools.utils.OT]

Context manager to manage option objects

classmethod get_option_value(option_name)[source]
options_cls: Type[convtools.utils.OT]
class convtools.utils.BaseCtxMeta(name, bases, kwargs)[source]

Bases: type

class convtools.utils.BaseOptions[source]

Bases: object

Container object, which carries current options

clone()[source]
to_defaults(option_name=None)[source]
class convtools.utils.BaseOptionsMeta(name, bases, kwargs)[source]

Bases: type

class convtools.utils.Code[source]

Bases: object

A building block for generating code, which is composed of multiple statements.

add_code(code: convtools.utils.Code)[source]
add_line(line: str, next_line_indent_incr: int, expression_line: Optional[str] = None)[source]
as_expression() Optional[str][source]
has_lines()[source]
incr_indent_level(incr: int)[source]
to_string(base_indent_level: int, single_indent: str = '    ')[source]
class convtools.utils.CodePiece(code_str, abs_path, is_dumped)[source]

Bases: object

abs_path
code_str
is_dumped
class convtools.utils.CodeStorage[source]

Bases: object

Container which stores a map of converter names to generated code pieces. It allows to dump code sources on disk into a debug directory.

add_sources(converter_name, code_str)[source]
dump_sources()[source]
class convtools.utils.LazyDebugDir[source]

Bases: object

Lazy instance to hold and initialize debug directory for generated code sources

ensure_initialized()[source]
get()[source]
convtools.utils.drop_dumped_code(name_to_code_piece)[source]
convtools.utils.iter_windows(collection, width, step)[source]

Module contents

The only object is exported - conversion. This object exposes the public API of conversions.