============================================
conservancy_beancount Development Overview
============================================

Running all tests
-----------------

The project comes with a Tox configuration that defines all the tests we expect to pass, across all the different environments we support. To run them all, run::

  tox

The next sections describe how to run them standalone in your own development environment, for faster iteration or simpler isolation of problems.

Installing
~~~~~~~~~~

Create a virtual environment, for example::

  python3 -m venv ~/.virtualenvs/conservancy-beancount

Install the ``conservancy_beancount`` as an editable package including the test dependencies::

  ~/.virtualenvs/conservancy-beancount/bin/python -m pip install --editable ~/conservancy/conservancy_beancount[test]

Running unit tests
~~~~~~~~~~~~~~~~~~

Run::

  pytest

Alternatively, you can run ``tox`` which will run the tests in a clean virtual
environment with all the relevant dependencies.

Type checking
~~~~~~~~~~~~~

Most of the code is typed, except for very dynamic loader methods. To run the type checker::

  # Beancount includes type declarations but not the `py.typed` flag file mypy
  # is looking for to know that. Create it ourselves.
  python -c 'import beancount, pathlib; pathlib.Path(beancount.__file__).with_name("py.typed").touch()'
  mypy conservancy_beancount

This is expected to pass just like the unit tests.

Plugin
------

Books that use this plugin are expected to say::

      plugin "conservancy_beancount.plugin"

When Beancount sees this directive after loading all the data, it passes everything it loaded through the ``run`` function in ``conservancy_beancount/plugin/__init__.py`` to be manipulated. Our ``run`` function, in turn, passes the data through hooks defined in other files under ``conservancy_beancount/plugin`` to normalize data and report errors. ``HookRegistry`` in ``__init__.py`` is the bridge between these two. It supports dynamically choosing which hooks are loaded at runtime, although we currently aren't using that.

``conservancy_beancount/plugin/core.py`` has the base classes for these hooks. The rest of the files in the directory contain the specific hooks that are loaded and run. All the ``meta_foo.py`` files have hook(s) to validate ``foo`` metadata. ``txn_date.py`` verifies that transactions are in the right file by fiscal year.

setup.cfg entry points
----------------------

The ``options.entry_points`` section of ``setup.cfg`` defines the various tools that are installed by this package. An example looks like::

    accrual-report = conservancy_beancount.reports.accrual:entry_point

This means that the package installs a tool called ``accrual-report`` that runs by calling the ``entry_point`` function defined in the ``conservancy_beancount.reports.accrual`` module (``conservancy_beancount/reports/accrual.py``).

To save you a little tracking, the ``entry_point`` function is one that sets up an exception handler and global logger, then calls the ``main`` function in the same module. If you want to see how a tool is implemented, you should be able to just look up the module in ``setup.cfg``, then go right to its ``main`` function. If you want to see how ``entry_point`` is defined, it's the return value of ``make_entry_point`` in ``conservancy_beancount/cliutil.py``. This level of indirection makes it easier to test tools without worrying about global state.

Module overview
---------------

This isn't a comprehensive list of every module—that would be redundant—but a guided tour of the most important modules, and the most important things in them.

* ``books``: Given the definition of a fiscal year, and a path to a directory of Beancount files organized by fiscal year, this module can load ranges of fiscal years of the books. Every user-facing tool uses this module to load the books from disk.
* ``cliutil``: Common functionality for command line argument parsing (to use with the argparse module), exception handling, error status reporting, and logging (with the logging library).
* ``config``: Read all user configuration, including the main ``~/.config/conservancy_beancount/config.ini`` configuration file, as well as environment variables, ``~/.rtrc``, etc.
* ``data``: Higher-level classes that build on top of Beancount's own data structures. Most notably, this defines a ``Posting`` class that provides the "fall-through" view of metadata we want, where if you look up a key and it's not available on the posting it'll try to return it from the transaction instead. There's also an ``Account`` class, built on top of ``str``, that adds a lot of predicates to make it easy and consistent to check where an account is in the account hierarchy.
* ``rtutil``: Knows how to parse RT links in book documentation and check whether they exist, turn them into web links, etc. Caches results to both RAM and disk so not every process needs to query every link every time.
* ``pdfforms``: Tools to read PDF form data to and from a YAML file. The main use case for this is to make it easier to fill out tax forms. The plan was that users could put bean-query-like queries in the YAML file to automatically fill out fields with numbers from the books. It's not clear that continued development is worth it now that the IRS requires e-Filing, though.
* ``plugin``: The plugin to validate and normalize transaction metadata.
    * ``__init__`` contains the plugin entry point and hook selection.
    * ``core`` includes base classes for the validation and normalization hooks.
    * The rest of the modules contain individual hooks.
* ``reconcile``: Tools to reconcile our books against provider statements.
* ``reports``: Business reports from the books.
    * ``core`` includes base classes for all reports, including the ``Balance``, ``RelatedPostings``, and ``BaseODS`` classes.
    * ``rewrite`` implements the "rewrite rules" available to all reports to modify how postings are reported. The module docstring has plenty of details.
    * Other modules implement individual reports.
* ``tools``: Other user-facing tools that are not financial reports. Most of these support the audit by orchestrating the process or manipulating reports to prepare them for submission to the auditors.