Benefits of convtools¶
Fewer bugs & readability¶
- it forces you to split the definition of data conversion from the data itself
- functional approach (pipes, "everything is an expression") lifts the obligation to name variables at each step, lowering the chance of confusing them
convtoolstest coverage is 100% and it's development is test-driven, so whenever a bug is found, we'll do our best to make sure it never occurs again.
DRY¶
In some cases, to keep the code DRY (complying with "Don't Repeat Yourself" principle) or just readable you need to pay with code performance:
- intensive code reuse (function call overhead)
- configs known at compile time (loops, which can be unrolled)
convtools encapsulates non-dry pieces in the code it generates.
Of course, there's also a cost, but it is completely under your control where
and when to run gen_converter to spend a fraction of a second to generate
an ad-hoc converter.
Functionality¶
Since convtools is a code-generating layer, it provides you with an extra
functionality like group_by, joins, pipes, etc.
Performance¶
'The Art of Computer Programming' book by Donald Knuth:
“The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming.”
convtools worries about code efficiency, so you can put more effort into
other parts of your code. The below table provides the speed-up of
convtools-based solutions over the naive ones.
| name | speed_up | py_compiler | arch |
|---|---|---|---|
| Aggregate1 | +28.4% | Clang 15.0.0 (clang-1500.0.40.1) | arm64 |
| GroupBy1 - MANY_GROUPS | +29.8% | Clang 15.0.0 (clang-1500.0.40.1) | arm64 |
| DatetimeFormat2 | +49.5% | Clang 15.0.0 (clang-1500.0.40.1) | arm64 |
| GroupBy1 - FEW_GROUPS | +70.4% | Clang 15.0.0 (clang-1500.0.40.1) | arm64 |
| DatetimeFormat1 | +75.1% | Clang 15.0.0 (clang-1500.0.40.1) | arm64 |
| IterOfIter1 | +90.7% | Clang 15.0.0 (clang-1500.0.40.1) | arm64 |
| TableDictReader | +122.8% | Clang 15.0.0 (clang-1500.0.40.1) | arm64 |
| DateFormat2 | +178.3% | Clang 15.0.0 (clang-1500.0.40.1) | arm64 |
| DatetimeParse2 | +205.4% | Clang 15.0.0 (clang-1500.0.40.1) | arm64 |
| DatetimeParse1 | +251.6% | Clang 15.0.0 (clang-1500.0.40.1) | arm64 |
| DateParse1 | +270.5% | Clang 15.0.0 (clang-1500.0.40.1) | arm64 |
| DateParse2 | +275.2% | Clang 15.0.0 (clang-1500.0.40.1) | arm64 |
| DateFormat1 | +563.8% | Clang 15.0.0 (clang-1500.0.40.1) | arm64 |
| name | speed_up | py_compiler | arch |
|---|---|---|---|
| Aggregate1 | +30.4% | Clang 14.0.3 (clang-1403.0.22.14.1) | arm64 |
| DatetimeFormat2 | +33.5% | Clang 14.0.3 (clang-1403.0.22.14.1) | arm64 |
| GroupBy1 - MANY_GROUPS | +35.3% | Clang 14.0.3 (clang-1403.0.22.14.1) | arm64 |
| DatetimeFormat1 | +55.5% | Clang 14.0.3 (clang-1403.0.22.14.1) | arm64 |
| GroupBy1 - FEW_GROUPS | +57.9% | Clang 14.0.3 (clang-1403.0.22.14.1) | arm64 |
| IterOfIter1 | +117.1% | Clang 14.0.3 (clang-1403.0.22.14.1) | arm64 |
| DateFormat2 | +139.8% | Clang 14.0.3 (clang-1403.0.22.14.1) | arm64 |
| TableDictReader | +161.3% | Clang 14.0.3 (clang-1403.0.22.14.1) | arm64 |
| DatetimeParse2 | +210.6% | Clang 14.0.3 (clang-1403.0.22.14.1) | arm64 |
| DatetimeParse1 | +245.9% | Clang 14.0.3 (clang-1403.0.22.14.1) | arm64 |
| DateParse1 | +254.1% | Clang 14.0.3 (clang-1403.0.22.14.1) | arm64 |
| DateParse2 | +262.3% | Clang 14.0.3 (clang-1403.0.22.14.1) | arm64 |
| DateFormat1 | +516.3% | Clang 14.0.3 (clang-1403.0.22.14.1) | arm64 |
| name | speed_up | py_compiler | arch |
|---|---|---|---|
| GroupBy1 - FEW_GROUPS | -6.7% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| GroupBy1 - MANY_GROUPS | -5.5% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| Aggregate1 | +27.6% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DatetimeFormat2 | +38.4% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DatetimeFormat1 | +56.3% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| IterOfIter1 | +97.8% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DateFormat2 | +148.6% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| TableDictReader | +164.9% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DatetimeParse2 | +210.1% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DatetimeParse1 | +252.9% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DateParse1 | +262.0% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DateParse2 | +263.8% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DateFormat1 | +490.1% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| name | speed_up | py_compiler | arch |
|---|---|---|---|
| GroupBy1 - FEW_GROUPS | -0.4% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| GroupBy1 - MANY_GROUPS | +0.2% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| Aggregate1 | +28.7% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DatetimeFormat2 | +37.7% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DatetimeFormat1 | +56.7% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| IterOfIter1 | +80.4% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DateFormat2 | +144.3% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| TableDictReader | +167.4% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DatetimeParse2 | +216.8% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DateParse1 | +257.3% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DatetimeParse1 | +258.6% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DateParse2 | +262.3% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DateFormat1 | +471.9% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| name | speed_up | py_compiler | arch |
|---|---|---|---|
| GroupBy1 - MANY_GROUPS | -3.5% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| GroupBy1 - FEW_GROUPS | -2.8% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| Aggregate1 | +39.3% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DatetimeFormat2 | +47.5% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DatetimeFormat1 | +69.5% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| IterOfIter1 | +85.8% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| TableDictReader | +167.1% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DateFormat2 | +170.2% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DatetimeParse2 | +223.9% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DatetimeParse1 | +263.0% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DateParse2 | +265.2% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DateParse1 | +270.1% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DateFormat1 | +489.1% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| name | speed_up | py_compiler | arch |
|---|---|---|---|
| GroupBy1 - MANY_GROUPS | +1.6% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| GroupBy1 - FEW_GROUPS | +3.6% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| Aggregate1 | +35.2% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DatetimeFormat2 | +47.1% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DatetimeFormat1 | +67.0% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| IterOfIter1 | +84.5% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DateFormat2 | +182.1% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DatetimeParse2 | +224.2% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DatetimeParse1 | +265.2% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DateParse1 | +266.0% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DateParse2 | +275.9% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| TableDictReader | +276.6% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
| DateFormat1 | +478.5% | Clang 14.0.0 (clang-1400.0.29.201) | arm64 |
In cases where there are multiple speed test results, the worst is taken. See benchmarks on Github for source code.