# TODO: Make the directory if it doesn't exist.

"""Compare a bank CSV statement with the books.

This tool takes an AMEX or First Republic CSV statement file and

compares it line-by-line with the Beancount books to make sure that

everything matches. This is designed for situations where transactions

are entered into the books directly, rather than being imported from a

statement after the fact.

The reconciler will attempt to match transactions based on date,

amount, check number and payee, but is forgiving to differences in

dates, the absensce of check number and inexact matches on

payee. Matches are ranked, so where there is only one decent match for

an amount/date this is accepted, but if there are multiple similar

candidates it will refuse to guess.

The reconciler will also attempt to identify where a single statement

entry has been split out into multiple Beancount postings, such as a

single bank transfer representing health insurance for multiple

employees.

Run it like this:

$ statement_reconciler \

  --beancount-file=2021.beancount \

  --account=Liabilities:CreditCard:AMEX \

  --csv-statement=2021-09-10_AMEX_activity.csv \

  --bank-statement=2021-09-10_AMEX_activity.csv \

  --statement-balance=1000

Background:

Beancount users often write importers to create bookkeeping entries

direct from a bank statement or similar. That approach automates data

entry and reconciliation in one step. In some cases though, it's

useful to manually enter transactions and reconcile them later

on. This workflow helpful in cases like writing a paper check when

there's a time lag between committing to making a payment and the

funds being debited. That's the workflow we're using here.

Conservancy currently enter data by hand rather than using Beancount

importers.  This tool is still somewhat like an importer in that it

needs to extract transaction details from a third-party

statement. Instead of creating directives, it just checks to see that

similar directives are already present. This is a bit like diff-ing a

statement with the books (though we're only interested in the presence

of lines, not so much their order).

Problems in scope:

 - errors in the books take hours to find during reconciliation,

   requiring manually comparing statemnts and the books and are

   succeptible to mistakes, such as not noticing when there are two

   payments for the same amount on the statement, but not in the books

   ("you're entering a world of pain")

 - adding statement/reconciliation metadata to books is/was manual and

   prone to mistakes

 - Beancount doesn't provide any infrastructure for programmatically

   updating the books, only appending in the case of importers

 - paper checks are entered in the books when written, but may not be

   cashed until months later (reconcile errors)

 - jumping to an individual transaction in a large ledger isn't

   trivial - Emacs grep mode is the current best option

 - auditors would prefer Bradley didn't perform reconciliation,

   ideally not Rosanne either

 - reconciliation reports are created by hand when there are mismatches

Other related problems we're not dealing with here:

 - after updates to the books files, beancount must be restarted to

   reflect updates

 - updates also invalidate the cache meaning restart takes several

   minutes

 - balance checks are manually updated in

   svn/Financial/Ledger/sanity-check-balances.yaml

 - transactions are entered manually and reconciled after the fact,

   but importing from statements may be useful in some cases

# TODO:

#  - extract the magic numbers

#  - consider merging in helper.py

logger.setLevel(logging.INFO)

    """Clean up payee field to improve quality of fuzzy matching.

    It turns out that bank statement "description" fields are

    difficult to fuzzy match on because they're long and

    noisey. Truncating them (see standardize_XXX_record fns) and

    removing the common junk helps significantly.

    """

def validate_amex_csv(sample: str, account: str) -> None:

    required_cols = {'Date', 'Amount', 'Description', 'Card Member'}

    reader = csv.DictReader(io.StringIO(sample))

    if reader.fieldnames and not required_cols.issubset(reader.fieldnames):

        sys.exit(f"This CSV doesn't seem to have the columns we're expecting, including: {', '.join(required_cols)}")

def entry_point():

    args = parse_args(sys.argv)

    main(args)