Changeset - 14d799bfdd84
[Not reviewed]
0 8 0
Brett Smith - 6 years ago 2017-11-09 20:30:21
brettcsmith@brettcsmith.org
template: Templates amounts are now written with fuller expressions.

This makes it possible to support transactions that divide funds in
ways other than a simple percentage split. For example, there
might be tax withheld, or a flat fee expense on each transaction.
8 files changed with 229 insertions and 95 deletions:
0 comments (0 inline, 0 general)
README.rst
Show inline comments
...
 
@@ -24,15 +24,15 @@ Writing templates
 
A template looks like a Ledger entry with a couple of differences:
 

	
 
* You don't write the first payee line.
 
* You allocate percentages to each account, rather than specific currency amounts.
 
* Instead of writing specific currency amounts, you write simple expressions to calculate amounts.
 

	
 
Here's a simple template for Patreon patron payments::
 

	
 
  [DEFAULT]
 
  template patreon income =
 
    ;Tag: Value
 
    Income:Patreon  -100%%
 
    Accrued:Accounts Receivable:Patreon  100%%
 
    Income:Patreon  -{amount}
 
    Accrued:Accounts Receivable:Patreon  {amount}
 

	
 
Let's walk through this line by line.
 

	
...
 
@@ -42,52 +42,24 @@ Every setting in your configuration file has to be in a section.  ``[DEFAULT]``
 

	
 
The first line of the template is a Ledger tag.  The program will leave all kinds of tags and Ledger comments alone, except to indent them nicely.
 

	
 
The next two lines split the money across accounts.  They follow almost the same format as they do in Ledger: there's an account named, followed by a tab or two or more spaces, and then an amount.  However, rather than a currency amount, specify a percentage amount between -100% and 100%.  You can specify any number of splits you want, as long as the negative splits total -100% and the positive splits total 100%.  ``%`` is a special character in INI file syntax; writing ``%%`` is the way to get a literal ``%`` in the string.  Alternatively, you can omit the ``%`` altogether; import2ledger will still treat any floating point value at the end of the line as a percentage.
 
The next two lines split the money across accounts.  They follow almost the same format as they do in Ledger: there's an account named, followed by a tab or two or more spaces, and then an expression.  Each time import2ledger generates an entry, it will evaluate this expression using the data it imported to calculate the actual currency amount to write.  Your expression can use numbers, basic arithmetic operators (including parentheses for grouping), and imported data referred to as ``{variable_name}``.
 

	
 
If the amount of currency being imported doesn't split evenly, spare change will be allocated to the last split to keep the entry balanced.
 
import2ledger uses decimal math to calculate each amount, and rounds to the number of digits appropriate for that currency.  If the amount of currency being imported doesn't split evenly, spare change will be allocated to the last split to keep the entry balanced on both sides.
 

	
 
Refer to the `Python documentation for INI file structure <https://docs.python.org/3/library/configparser.html#supported-ini-file-structure>`_ for full details of the syntax.  Note that import2ledger doesn't use ``;`` as a comment prefix, since that's the primary comment prefix in Ledger.
 

	
 
Supported templates
 
~~~~~~~~~~~~~~~~~~~
 

	
 
You can define the following templates.
 

	
 
Patreon
 
^^^^^^^
 

	
 
``template patreon income``
 
  Imports one transaction per patron per month.  Generated from Patreon's monthly patron report CSVs.
 

	
 
``template patreon cardfees``
 
  Imports one expense transaction per month for that month's credit card fees.  Generated from Patreon's earnings report CSV.
 

	
 
``template patreon svcfees``
 
  Imports one expense transaction per month for that month's Patreon service fees.  Generated from Patreon's earnings report CSV.
 

	
 
``template patreon vat``
 
  Imports one transaction per country per month each time Patreon withheld VAT.  Generated from Patreon's VAT report CSV.
 

	
 
  This template can use these variables:
 

	
 
  ``country_name``
 
    The full name of the country VAT was withheld for
 

	
 
  ``country_code``
 
    The two-letter ISO country code of the country VAT was withheld for
 

	
 
Template variables
 
~~~~~~~~~~~~~~~~~~
 

	
 
import2ledger templates have access to a few variables for each transaction that can be included in the output entry using `Python's format string syntax <https://docs.python.org/3/library/string.html#formatstrings>`_.  For most uses, all you need to do is write ``{variable_name}`` in the template, and the variable's value will be filled in there.  For example, this template sets different payees for each side of the transaction::
 
import2ledger templates have access to a few variables for each transaction that can be included anywhere in the entry, not just amount expressions.  In other parts of the entry, they're treated as strings, and you can control their formatting using `Python's format string syntax <https://docs.python.org/3/library/string.html#formatstrings>`_.  For example, this template sets different payees for each side of the transaction::
 

	
 
  template patreon income =
 
    Income:Patreon  -100%%
 
    Income:Patreon  -{amount}
 
    ;Payee: {payee}
 
    Accrued:Accounts Receivable:Patreon  100%%
 
    Accrued:Accounts Receivable:Patreon  {amount}
 
    ;Payee: Patreon
 

	
 
You can use the following variables:
 
Every template can use the following variables:
 

	
 
================== ==========================================================
 
Name               Contents
...
 
@@ -103,7 +75,38 @@ date               The date of the transaction, in your configured output
 
payee              The name of the transaction payee
 
================== ==========================================================
 

	
 
Custom hooks may add more template variables.
 
Specific importers and hooks may provide additional variables.
 

	
 
Supported templates
 
~~~~~~~~~~~~~~~~~~~
 

	
 
You can define the following templates.
 

	
 
Patreon
 
^^^^^^^
 

	
 
``template patreon income``
 
  Imports one transaction per patron per month.  Generated from Patreon's monthly patron report CSVs.
 

	
 
``template patreon cardfees``
 
  Imports one expense transaction per month for that month's credit card fees.  Generated from Patreon's earnings report CSV.
 

	
 
``template patreon svcfees``
 
  Imports one expense transaction per month for that month's Patreon service fees.  Generated from Patreon's earnings report CSV.
 

	
 
``template patreon vat``
 
  Imports one transaction per country per month each time Patreon withheld VAT.  Generated from Patreon's VAT report CSV.
 

	
 
  This template can use these variables:
 

	
 
  ============== ============================================================
 
  Name           Contents
 
  ============== ============================================================
 
  country_name   The full name of the country VAT was withheld for
 
  -------------- ------------------------------------------------------------
 
  country_code   The two-letter ISO country code of the country VAT was
 
                 withheld for
 
  ============== ============================================================
 

	
 
Other output options
 
~~~~~~~~~~~~~~~~~~~~
...
 
@@ -112,6 +115,8 @@ Various options control how import2ledger formats dates and currency amounts.  R
 

	
 
  date_format = %%Y-%%m-%%d
 

	
 
(``%`` is a special character in the configuration file syntax.  ``%%`` represents a literal percent sign.)
 

	
 
Configuration sections
 
~~~~~~~~~~~~~~~~~~~~~~
 

	
import2ledger/config.py
Show inline comments
...
 
@@ -241,6 +241,7 @@ class Configuration:
 
            signed_currencies=[code.strip().upper() for code in section_config['signed_currencies'].split(',')],
 
            signed_currency_fmt=section_config['signed_currency_format'],
 
            unsigned_currency_fmt=section_config['unsigned_currency_format'],
 
            template_name=config_key,
 
        )
 

	
 
    def setup_logger(self, logger, section_name=None):
import2ledger/template.py
Show inline comments
 
import collections
 
import datetime
 
import decimal
 
import functools
 
import io
 
import operator
 
import re
 
import tokenize
 

	
 
import babel.numbers
 

	
 
class AccountSplitter:
 
    Split = collections.namedtuple('Split', ('account', 'percentage'))
 
from . import errors
 

	
 
class TokenTransformer:
 
    def __init__(self, source):
 
        try:
 
            source = source.readline
 
        except AttributeError:
 
            pass
 
        self.in_tokens = tokenize.tokenize(source)
 

	
 
    @classmethod
 
    def from_bytes(cls, b):
 
        return cls(io.BytesIO(b).readline)
 

	
 
    @classmethod
 
    def from_str(cls, s, encoding='utf-8'):
 
        return cls.from_bytes(s.encode(encoding))
 

	
 
    def __iter__(self):
 
        for ttype, tvalue, _, _, _ in self.in_tokens:
 
            try:
 
                transformer = getattr(self, 'transform_' + tokenize.tok_name[ttype])
 
            except AttributeError:
 
                raise ValueError("{} token {!r} not supported".format(ttype, tvalue))
 
            yield from transformer(ttype, tvalue)
 

	
 
    def _noop_transformer(self, ttype, tvalue):
 
        yield (ttype, tvalue)
 

	
 
    transform_ENDMARKER = _noop_transformer
 

	
 
    def transform_ENCODING(self, ttype, tvalue):
 
        self.in_encoding = tvalue
 
        return self._noop_transformer(ttype, tvalue)
 

	
 
    def transform(self):
 
        out_bytes = tokenize.untokenize(self)
 
        return out_bytes.decode(self.in_encoding)
 

	
 

	
 
class AmountTokenTransformer(TokenTransformer):
 
    SUPPORTED_OPS = frozenset('+-*/()')
 

	
 
    transform_NAME = TokenTransformer._noop_transformer
 

	
 
    def transform_NUMBER(self, ttype, tvalue):
 
        yield (tokenize.NAME, 'Decimal')
 
        yield (tokenize.OP, '(')
 
        yield (tokenize.STRING, repr(tvalue))
 
        yield (tokenize.OP, ')')
 

	
 
    def __init__(self, sign, signed_currencies, signed_currency_fmt, unsigned_currency_fmt):
 
        self.splits = []
 
        self.sign = sign
 
    def transform_OP(self, ttype, tvalue):
 
        if tvalue == '{':
 
            try:
 
                name_type, name_value, _, _, _ = next(self.in_tokens)
 
                close_type, close_value, _, _, _ = next(self.in_tokens)
 
                if (name_type != tokenize.NAME
 
                    or close_type != tokenize.OP
 
                    or close_value != '}'):
 
                    raise ValueError()
 
            except (StopIteration, ValueError):
 
                raise ValueError("opening { does not name variable")
 
            yield (tokenize.NAME, name_value)
 
        elif tvalue in self.SUPPORTED_OPS:
 
            yield from self._noop_transformer(ttype, tvalue)
 
        else:
 
            raise ValueError("unsupported operator {!r}".format(tvalue))
 

	
 

	
 
class AccountSplitter:
 
    def __init__(self, signed_currencies, signed_currency_fmt, unsigned_currency_fmt,
 
                 template_name):
 
        self.splits = collections.OrderedDict()
 
        self.signed_currency_fmt = signed_currency_fmt
 
        self.unsigned_currency_fmt = unsigned_currency_fmt
 
        self.signed_currencies = set(signed_currencies)
 
        self.unsplit = sign * 100
 
        self.template_name = template_name
 
        self._last_template_vars = object()
 

	
 
    def add(self, account, percentage_s):
 
        int_pctage = decimal.Decimal(percentage_s.rstrip('%'))
 
        self.unsplit -= int_pctage
 
        percentage = None if self.unsplit == 0 else (int_pctage / 100)
 
        self.splits.append(self.Split(account, percentage))
 

	
 
    def render_next(self, template_vars):
 
    def add(self, account, amount_expr):
 
        try:
 
            account, percentage = next(self._split_iter)
 
        except (AttributeError, StopIteration):
 
            self._split_iter = iter(self.splits)
 
            account, percentage = next(self._split_iter)
 
            self._remainder = self.sign * template_vars['amount']
 
            self._currency = template_vars['currency']
 
            if self._currency in self.signed_currencies:
 
                self._amt_fmt = self.signed_currency_fmt
 
            else:
 
                self._amt_fmt = self.unsigned_currency_fmt
 
        if percentage is None:
 
            amount = self._remainder
 
            clean_expr = AmountTokenTransformer.from_str(amount_expr).transform()
 
        except ValueError as error:
 
            raise errors.UserInputConfigurationError(error.args[0], amount_expr)
 
        self.splits[account] = compile(clean_expr, self.template_name, 'eval')
 

	
 
    def _currency_decimal(self, amount, currency):
 
        return decimal.Decimal(babel.numbers.format_currency(amount, currency, '###0.###'))
 

	
 
    def _balance_amounts(self, amounts, to_amount):
 
        cmp_func = operator.lt if to_amount > 0 else operator.gt
 
        should_balance = functools.partial(cmp_func, 0)
 
        remainder = to_amount
 
        to_account = None
 
        for account, amount in amounts.items():
 
            if should_balance(amount):
 
                remainder -= amount
 
                to_account = account
 
        if to_account is None:
 
            pass
 
        elif (abs(remainder) / abs(to_amount)) >= decimal.Decimal('.1'):
 
            raise errors.UserInputConfigurationError(
 
                "template can't balance amounts to {}".format(to_amount),
 
                self.template_name,
 
            )
 
        else:
 
            amount = decimal.Decimal(babel.numbers.format_currency(
 
                template_vars['amount'] * percentage, self._currency, '###0.###'))
 
        self._remainder -= amount
 
        amt_s = babel.numbers.format_currency(amount, self._currency, self._amt_fmt)
 
        return '    {:45}  {:>19}\n'.format(account, amt_s)
 
            amounts[to_account] += remainder
 

	
 
    def _build_amounts(self, template_vars):
 
        amount_vars = {k: v for k, v in template_vars.items() if isinstance(v, decimal.Decimal)}
 
        amount_vars['Decimal'] = decimal.Decimal
 
        amounts = collections.OrderedDict(
 
            (account, self._currency_decimal(eval(amount_expr, amount_vars),
 
                                             template_vars['currency']))
 
            for account, amount_expr in self.splits.items()
 
        )
 
        self._balance_amounts(amounts, template_vars['amount'])
 
        self._balance_amounts(amounts, -template_vars['amount'])
 
        return amounts
 

	
 
    def _iter_splits(self, template_vars):
 
        amounts = self._build_amounts(template_vars)
 
        if template_vars['currency'] in self.signed_currencies:
 
            amt_fmt = self.signed_currency_fmt
 
        else:
 
            amt_fmt = self.unsigned_currency_fmt
 
        for account, amount in amounts.items():
 
            amt_s = babel.numbers.format_currency(amount, template_vars['currency'], amt_fmt)
 
            yield '    {:45}  {:>19}\n'.format(account, amt_s)
 

	
 
    def render_next(self, template_vars):
 
        if template_vars is not self._last_template_vars:
 
            self._split_iter = self._iter_splits(template_vars)
 
            self._last_template_vars = template_vars
 
        return next(self._split_iter)
 

	
 

	
 
class Template:
 
    ACCOUNT_SPLIT_RE = re.compile(r'(?:\t|  )\s*')
 
    DATE_FMT = '%Y/%m/%d'
 
    SIGNED_CURRENCY_FMT = '¤#,##0.###;¤-#,##0.###'
 
    UNSIGNED_CURRENCY_FMT = '#,##0.### ¤¤'
...
 
@@ -51,12 +155,11 @@ class Template:
 
    def __init__(self, template_s, signed_currencies=frozenset(),
 
                 date_fmt=DATE_FMT,
 
                 signed_currency_fmt=SIGNED_CURRENCY_FMT,
 
                 unsigned_currency_fmt=UNSIGNED_CURRENCY_FMT):
 
                 unsigned_currency_fmt=UNSIGNED_CURRENCY_FMT,
 
                 template_name='<template>'):
 
        self.date_fmt = date_fmt
 
        self.pos_splits = AccountSplitter(
 
            1, signed_currencies, signed_currency_fmt, unsigned_currency_fmt)
 
        self.neg_splits = AccountSplitter(
 
            -1, signed_currencies, signed_currency_fmt, unsigned_currency_fmt)
 
        self.splitter = AccountSplitter(
 
            signed_currencies, signed_currency_fmt, unsigned_currency_fmt, template_name)
 

	
 
        lines = [s.lstrip() for s in template_s.splitlines(True)]
 
        for index, line in enumerate(lines):
...
 
@@ -72,10 +175,14 @@ class Template:
 
            if line[0].isalpha():
 
                if start_index < index:
 
                    self._add_str_func(lines[start_index:index])
 
                account, percentage_s = line.rsplit(None, 1)
 
                splitter = self.neg_splits if percentage_s.startswith('-') else self.pos_splits
 
                splitter.add(account, percentage_s)
 
                self.format_funcs.append(splitter.render_next)
 
                line = line.strip()
 
                match = self.ACCOUNT_SPLIT_RE.search(line)
 
                if match is None:
 
                    raise errors.UserInputError("no amount expression found", line)
 
                account = line[:match.start()]
 
                amount_expr = line[match.end():]
 
                self.splitter.add(account, amount_expr)
 
                self.format_funcs.append(self.splitter.render_next)
 
                start_index = index + 1
 
        if start_index <= index:
 
            self._add_str_func(lines[start_index:])
tests/data/templates.ini
Show inline comments
 
[Simplest]
 
template =
 
 Accrued:Accounts Receivable  100%%
 
 Income:Donations  -100%%
 
 Accrued:Accounts Receivable  {amount}
 
 Income:Donations  -{amount}
 

	
 
[FiftyFifty]
 
template =
 
 Accrued:Accounts Receivable  100%%
 
 Income:Donations  -50%%
 
 Income:Sales  -50%%
 
 Accrued:Accounts Receivable  {amount}
 
 Income:Donations  -.5 * {amount}
 
 Income:Sales  -.5*{amount}
 

	
 
[Complex]
 
template =
 
 ;Tag: Value
 
 ;TransactionID: {txid}
 
 Accrued:Accounts Receivable  100%%
 
 Accrued:Accounts Receivable  {amount}
 
 ;Entity: Supplier
 
 Income:Donations:Specific  -95.5%%
 
 Income:Donations:Specific    -.955*  {amount}
 
 ;Program: Specific
 
 ;Entity: {entity}
 
 Income:Donations:General  -4.5%%
 
 Income:Donations:General     -.045  * {amount}
 
 ;Entity: {entity}
 

	
 
[Multivalue]
 
template =
 
 Expenses:Taxes  {tax}
 
 Accrued:Accounts Receivable  {amount} - {tax}
 
 Income:RBI         -.1*{amount}
 
 Income:Donations   -.9*{amount}
tests/data/test_config.ini
Show inline comments
...
 
@@ -5,15 +5,15 @@ signed_currencies = EUR
 
[Templates]
 
output_path = Template.output
 
one =
 
 Accrued:Accounts Receivable  100%%
 
 Income:Donations  -100%%
 
 Accrued:Accounts Receivable  {amount}
 
 Income:Donations  -{amount}
 
two =
 
 ;Tag1: {value}
 
 Income:Donations  -100%%
 
 Income:Donations  -{amount}
 
 ;IncomeTag: Donations
 
 Expenses:Fundraising  5%%
 
 Expenses:Fundraising  .05 * {amount}
 
 ;ExpenseTag: Fundraising
 
 Accrued:Accounts Receivable  95%%
 
 Accrued:Accounts Receivable  .95 * {amount}
 
 ;AccrualTag: Receivable
 

	
 
[Date]
tests/data/test_main.ini
Show inline comments
...
 
@@ -5,8 +5,8 @@ signed_currencies = USD
 

	
 
[One]
 
template patreon cardfees =
 
 Accrued:Accounts Receivable  -100%%
 
 Expenses:Fees:Credit Card  100%%
 
 Accrued:Accounts Receivable  -{amount}
 
 Expenses:Fees:Credit Card  {amount}
 
template patreon svcfees =
 
 Accrued:Accounts Receivable  -100%%
 
 Expenses:Fundraising  100%%
 
 Accrued:Accounts Receivable  -{amount}
 
 Expenses:Fundraising  {amount}
tests/test_config.py
Show inline comments
...
 
@@ -43,7 +43,7 @@ def test_template_parsing():
 
    except IndexError as error:
 
        assert False, error
 
    assert "\n;Tag1: {value}\n" in tmpl_s
 
    assert "\nIncome:Donations  -100%\n" in tmpl_s
 
    assert "\nIncome:Donations  -{amount}\n" in tmpl_s
 
    assert "\n;IncomeTag: Donations\n" in tmpl_s
 

	
 
@pytest.mark.parametrize('arg_s', [None, '-', 'output.ledger'])
tests/test_templates.py
Show inline comments
...
 
@@ -72,3 +72,17 @@ def test_balancing():
 
        "  Income:Donations  -0.50 USD",
 
        "  Income:Sales  -0.51 USD",
 
    ]
 

	
 
def test_multivalue():
 
    tmpl = template_from('Multivalue')
 
    rendered = tmpl.render('DD', decimal.Decimal('150.00'), 'USD', DATE,
 
                           tax=decimal.Decimal('12.50'))
 
    lines = [normalize_whitespace(s) for s in rendered.splitlines()]
 
    assert lines == [
 
        "",
 
        "2015/03/14 DD",
 
        "  Expenses:Taxes  12.50 USD",
 
        "  Accrued:Accounts Receivable  137.50 USD",
 
        "  Income:RBI  -15.00 USD",
 
        "  Income:Donations  -135.00 USD",
 
    ]
0 comments (0 inline, 0 general)