import2ledger

=============

Introduction

------------

import2ledger provides a Python library and CLI tool to read financial data from various popular sources and generate entries for text-based accounting books from them.

Installation

------------

This is a pretty normal Python module, so if you have a favorite way to install those, it will probably work.  If you just want to install it to run the script locally, try running from the source directory::

  $ python3 -m pip install --user -U --upgrade-strategy only-if-needed .

Configuring import2ledger

-------------------------

Since different organizations follow different accounting rules, you need to define an entry template for each kind of data that you want to be able to import.  You do that in a configuration file.  By default, import2ledger reads a configuration file at ``~/.config/import2ledger.ini``.  You can specify a different path with the ``-C`` option, and you can specify that multiple times to read multiple configuration files in order.

Writing templates

~~~~~~~~~~~~~~~~~

A template looks like a Ledger entry with a couple of differences:

* You don't write the first payee line.

* Instead of writing specific currency amounts, you write simple expressions to calculate amounts.

Here's a simple template for Patreon patron payments::

  [DEFAULT]

  patreon income ledger entry =

    ;Tag: Value

    Income:Patreon  -{amount}

    Accrued:Accounts Receivable:Patreon  {amount}

Let's walk through this line by line.

Every setting in your configuration file has to be in a section.  ``[DEFAULT]`` is the default section, and import2ledger reads configuration settings from here if you don't specify another one.  This documentation explains how to use sections later.

``patreon income ledger entry =`` specifies which entry template this is.  Every template is found from a setting with a name in the pattern ``<SOURCE> <TYPE> ledger entry``.  The remaining lines are indented further than this name; this defines a multiline value.  Don't worry about the exact indentation of your template; import2ledger will indent its output nicely.

The first line of the template is a Ledger tag.  The program will leave all kinds of tags and Ledger comments alone, except to indent them nicely.

import2ledger uses decimal math to calculate each amount, and rounds to the number of digits appropriate for that currency.  If the amount of currency being imported doesn't split evenly, spare change will be allocated to the last split to keep the entry balanced on both sides.

Refer to the `Python documentation for INI file structure <https://docs.python.org/3/library/configparser.html#supported-ini-file-structure>`_ for full details of the syntax.  Note that import2ledger doesn't use ``;`` as a comment prefix, since that's the primary comment prefix in Ledger.

Template variables

~~~~~~~~~~~~~~~~~~

import2ledger templates have access to a few variables for each transaction that can be included anywhere in the entry, not just amount expressions.  In other parts of the entry, they're treated as strings, and you can control their formatting using `Python's format string syntax <https://docs.python.org/3/library/string.html#formatstrings>`_.  For example, this template customizes the payee line and sets different payees for each side of the transaction::

  patreon income ledger entry =

    {date}  Patreon payment from {payee}

    Income:Patreon  -{amount}

    ;Payee: {payee}

    Accrued:Accounts Receivable:Patreon  {amount}

    ;Payee: Patreon

Templates automatically detect whether or not you have a custom payee line by checking if the first line begins with a date variable.  If it does, it's assumed to be your payee line.  Otherwise, the template uses a default payee line of ``{date} {payee}``.

Every template can use the following variables:

================== ==========================================================

Name               Contents

================== ==========================================================

amount             The total amount of the transaction, as a simple decimal

                   number (not currency-formatted)

------------------ ----------------------------------------------------------

currency           The three-letter code for the transaction currency

------------------ ----------------------------------------------------------

date               The date of the transaction, in your configured output

                   format

------------------ ----------------------------------------------------------

payee              The name of the transaction payee

------------------ ----------------------------------------------------------

source_abspath     The absolute path of the file being imported

------------------ ----------------------------------------------------------

source_absdir      The absolute path of the directory that contains the file

                   being imported

------------------ ----------------------------------------------------------

source_dir         The path of the directory that contains the file being

                   imported, as specified on the command line

------------------ ----------------------------------------------------------

source_name        The filename of the file being imported

------------------ ----------------------------------------------------------

source_path        The path of the file being imported, as specified on the

                   command line

------------------ ----------------------------------------------------------

source_stem        The filename of the file being imported, with its

import tokenize

import babel.numbers

from . import HOOK_KINDS

from .. import errors, strparse

logger = logging.getLogger('import2ledger.hooks.ledger_entry')

class TokenTransformer:

    def __init__(self, source):

        try:

            source = source.readline

        except AttributeError:

            pass

        self.in_tokens = tokenize.tokenize(source)

    @classmethod

    def from_bytes(cls, b):

        return cls(io.BytesIO(b).readline)

    @classmethod

    def from_str(cls, s, encoding='utf-8'):

        return cls.from_bytes(s.encode(encoding))

    def __iter__(self):

        for ttype, tvalue, _, _, _ in self.in_tokens:

            try:

                transformer = getattr(self, 'transform_' + tokenize.tok_name[ttype])

            except AttributeError:

                raise ValueError("{} token {!r} not supported".format(ttype, tvalue))

            yield from transformer(ttype, tvalue)

    def _noop_transformer(self, ttype, tvalue):

        yield (ttype, tvalue)

    transform_ENDMARKER = _noop_transformer

    def transform_ENCODING(self, ttype, tvalue):

        self.in_encoding = tvalue

        return self._noop_transformer(ttype, tvalue)

    def transform(self):

        out_bytes = tokenize.untokenize(self)

        return out_bytes.decode(self.in_encoding)

class AmountTokenTransformer(TokenTransformer):

    def __iter__(self):

        tokens = super().__iter__()

        for token in tokens:

            yield token

            if token[0] == tokenize.NAME:

                break

        else:

            raise ValueError("no amount in expression")

        yield from tokens

    def transform_NUMBER(self, ttype, tvalue):

        yield (tokenize.NAME, 'Decimal')

        yield (tokenize.OP, '(')

        yield (tokenize.STRING, repr(tvalue))

        yield (tokenize.OP, ')')

    def transform_OP(self, ttype, tvalue):

        if tvalue == '{':

            try:

                name_type, name_value, _, _, _ = next(self.in_tokens)

                close_type, close_value, _, _, _ = next(self.in_tokens)

                if (name_type != tokenize.NAME

                    or name_value != name_value.lower()

                    or close_type != tokenize.OP

                    or close_value != '}'):

                    raise ValueError()

            except (StopIteration, ValueError):

                raise ValueError("opening { does not name variable")

            yield (tokenize.NAME, name_value)

        elif tvalue in self.SUPPORTED_OPS:

            yield from self._noop_transformer(ttype, tvalue)

        else:

            raise ValueError("unsupported operator {!r}".format(tvalue))

class AccountSplitter:

    TARGET_LINE_LEN = 78

    # -4 because that's how many spaces prefix an account line.

    TARGET_ACCTLINE_LEN = TARGET_LINE_LEN - 4

    def __init__(self, signed_currencies, signed_currency_fmt, unsigned_currency_fmt,

                 template_name):

        self.splits = []

        self.metadata = []

        self.signed_currency_fmt = signed_currency_fmt

        self.unsigned_currency_fmt = unsigned_currency_fmt

        self.signed_currencies = set(signed_currencies)

        self.template_name = template_name

        self._last_template_vars = object()

    def is_empty(self):

        return not self.splits

    def add(self, account, amount_expr):

        try:

            clean_expr = AmountTokenTransformer.from_str(amount_expr).transform()

            compiled_expr = compile(clean_expr, self.template_name, 'eval')

        except (SyntaxError, tokenize.TokenError, ValueError) as error:

date_format = %%Y-%%m-%%d

signed_currencies = USD, CAD

signed_currency_format = ¤#,##0.###

unsigned_currency_format = #,##0.### ¤¤

[Simplest]

template = Accrued:Accounts Receivable  {amount}

 Income:Donations  -{amount}

[FiftyFifty]

template =

 Accrued:Accounts Receivable  {amount}

 Income:Donations  -.5 * {amount}

 Income:Sales  -.5*{amount}

[Complex]

template =

 ;Tag: Value

 ;TransactionID: {txid}

 Accrued:Accounts Receivable  {amount}

 ;Entity: Supplier

 Income:Donations:{program}    -.955*  {amount}

 ;Program: {program}

 ;Entity: {entity}

 Income:Donations:General     -.045  * {amount}

 ;Entity: {entity}

[Multivalue]

template =

 Expenses:Taxes  {tax}

 ;TaxAuthority: IRS

 Accrued:Accounts Receivable  {amount} - {tax}

 Income:RBI         -.1*{amount}

 Income:Donations   -.9*{amount}

[Custom Payee]

template =  {custom_date}  {payee} - Custom

 Accrued:Accounts Receivable  {amount}

 Income:Donations  -{amount}

[Multisplit]

template =

 Assets:Cash  {amount}

 Income:Sales  -{amount} + {item_sales}

 ; :NonItem:

 Income:Sales  -{item_sales}

 ; :Item:

[Empty]

template =

[Nonexistent]

    lines = render_lines(render_vars, 'Multisplit')

    assert lines == [

        "",

        "2015/03/14 JJ",

        "  Assets:Cash  90.00 USD",

        "  Income:Sales  -90.00 USD",

        "  ; :NonItem:",

    ]

def test_multiple_postings_same_account():

    render_vars = template_vars('LL', '80.00', other_vars={

        'item_sales': decimal.Decimal(30),

    })

    lines = render_lines(render_vars, 'Multisplit')

    assert lines == [

        "",

        "2015/03/14 LL",

        "  Assets:Cash  80.00 USD",

        "  Income:Sales  -50.00 USD",

        "  ; :NonItem:",

        "  Income:Sales  -30.00 USD",

        "  ; :Item:",

    ]

def test_custom_payee_line():

    render_vars = template_vars('ZZ', '10.00', other_vars={

        'custom_date': datetime.date(2014, 2, 13),

    })

    lines = render_lines(render_vars, 'Custom Payee')

    assert lines == [

        "",

        "2014/02/13  ZZ - Custom",

        "  Accrued:Accounts Receivable  10.00 USD",

        "  Income:Donations  -10.00 USD",

    ]

def test_line1_not_custom_payee():

    render_vars = template_vars('VV', '15.00', other_vars={

        'custom_date': datetime.date(2014, 2, 12),

    })

    lines = render_lines(render_vars, 'Simplest')

    assert lines == [

        "",

        "2015/03/14 VV",

        "  Accrued:Accounts Receivable  15.00 USD",

        "  Income:Donations  -15.00 USD",

    ]

@pytest.mark.parametrize('amount_expr', [

    '',

    'name',

    '-',

    '()',

    '+()',

    '{}',

    '{{}}',

    '{()}',

    '{name',

    'name}',

    '{42}',

    '(5).real',

    '{amount.real}',

    '{amount.is_nan()}',

    '{Decimal}',

    '{FOO}',

])

def test_bad_amount_expression(amount_expr):

    with pytest.raises(errors.UserInputError):

        ledger_entry.Template(" Income  " + amount_expr)

class Config:

    def __init__(self, use_section):

        self.section_name = use_section

        self.stdout = io.StringIO()

    @contextlib.contextmanager

    def open_output_file(self):

        yield self.stdout

    def get_section(self, name=None):

        return config[self.section_name]

def run_hook(entry_data, config_section):

    hook_config = Config(config_section)

    hook = ledger_entry.LedgerEntryHook(hook_config)

    assert hook.run(entry_data) is None

    stdout = hook_config.stdout.getvalue()

    return normalize_whitespace(stdout).splitlines()

def test_hook_renders_template():

    entry_data = template_vars('BB', '0.99')

    lines = run_hook(entry_data, 'Simplest')

    assert lines == [

        "",

        "2015-03-14 BB",