Summary
git Use ID
Read financial data from popular sources to generate entries for text-based accounting books
Statistics are disabled for this repository
Downloads are disabled for this repository
Brett Smith c49371c87ed3
6 years ago
Brett Smith cf4a0e37c1ba
6 years ago
Brett Smith 1aeffa31c569
6 years ago
Brett Smith 176aac35ed9b
6 years ago
Brett Smith 356f36b039a8
6 years ago
Brett Smith ab9c65d20dc5
6 years ago
Brett Smith b37575eabc84
6 years ago
Brett Smith 23190c1cdbfb
6 years ago
Brett Smith a408c854d476
6 years ago
Brett Smith f888b13c56b0
6 years ago

import2ledger

Introduction

import2ledger provides a Python library and CLI tool to read financial data from various popular sources and generate entries for text-based accounting books from them.

Installation

This is a pretty normal Python module, so if you have a favorite way to install those, it will probably work. If you just want to install it to run the script locally, try running from the source directory:

$ python3 -m pip install --user -U --upgrade-strategy only-if-needed .

Configuring import2ledger

Since different organizations follow different accounting rules, you need to define an entry template for each kind of data that you want to be able to import. You do that in a configuration file. By default, import2ledger reads a configuration file at <tt class="docutils literal">~/.config/import2ledger.ini</tt>. You can specify a different path with the <tt class="docutils literal">-C</tt> option, and you can specify that multiple times to read multiple configuration files in order.

Writing templates

A template looks like a Ledger entry with a couple of differences:

  • You don't write the first payee line.
  • Instead of writing specific currency amounts, you write simple expressions to calculate amounts.

Here's a simple template for Patreon patron payments:

[DEFAULT]
patreon income ledger entry =
  ;Tag: Value
  Income:Patreon  -{amount}
  Accrued:Accounts Receivable:Patreon  {amount}

Let's walk through this line by line.

Every setting in your configuration file has to be in a section. <tt class="docutils literal">[DEFAULT]</tt> is the default section, and import2ledger reads configuration settings from here if you don't specify another one. This documentation explains how to use sections later.

<tt class="docutils literal">patreon income ledger entry =</tt> specifies which entry template this is. Every template is found from a setting with a name in the pattern <tt class="docutils literal"><SOURCE> <TYPE> ledger entry</tt>. The remaining lines are indented further than this name; this defines a multiline value. Don't worry about the exact indentation of your template; import2ledger will indent its output nicely.

The first line of the template is a Ledger tag. The program will leave all kinds of tags and Ledger comments alone, except to indent them nicely.

The next two lines split the money across accounts. They follow almost the same format as they do in Ledger: there's an account named, followed by a tab or two or more spaces, and then an expression. Each time import2ledger generates an entry, it will evaluate this expression using the data it imported to calculate the actual currency amount to write. Your expression can use:

  • numbers and quoted strings
  • imported data referred to as <tt class="docutils literal">{variable_name}</tt>
  • basic arithmetic operators (including parentheses for grouping)
  • the operators <tt class="docutils literal">==</tt>, <tt class="docutils literal">!=</tt>, <tt class="docutils literal"><</tt>, <tt class="docutils literal"><=</tt>, <tt class="docutils literal">></tt>, <tt class="docutils literal">>=</tt>, <tt class="docutils literal">and</tt>, <tt class="docutils literal">or</tt>, <tt class="docutils literal">not</tt>, and <tt class="docutils literal">in</tt>
  • conditional expressions in the format <tt class="docutils literal">TRUE_EXPR if CONDITION else FALSE_EXPR</tt>

import2ledger uses decimal math to calculate each amount, and rounds to the number of digits appropriate for that currency. If the amount of currency being imported doesn't split evenly, spare change will be allocated to the last split to keep the entry balanced on both sides.

Refer to the Python documentation for INI file structure for full details of the syntax. Note that import2ledger doesn't use <tt class="docutils literal">;</tt> as a comment prefix, since that's the primary comment prefix in Ledger.

Template variables

import2ledger templates have access to a few variables for each transaction that can be included anywhere in the entry, not just amount expressions. In other parts of the entry, they're treated as strings, and you can control their formatting using Python's format string syntax. For example, this template customizes the payee line and sets different payees for each side of the transaction:

patreon income ledger entry =
  {date}  Patreon payment from {payee}
  Income:Patreon  -{amount}
  ;Payee: {payee}
  Accrued:Accounts Receivable:Patreon  {amount}
  ;Payee: Patreon

Templates automatically detect whether or not you have a custom payee line by checking if the first line begins with a date variable. If it does, it's assumed to be your payee line. Otherwise, the template uses a default payee line of <tt class="docutils literal">{date} {payee}</tt>.

Every template can use the following variables:

<colgroup> <col width="24%" /> <col width="76%" /> </colgroup>
Name Contents
amount The total amount of the transaction, as a simple decimal number (not currency-formatted)
currency The three-letter code for the transaction currency
date The date of the transaction, in your configured output format
payee The name of the transaction payee
source_abspath The absolute path of the file being imported
source_absdir The absolute path of the directory that contains the file being imported
source_dir The path of the directory that contains the file being imported, as specified on the command line
source_name The filename of the file being imported
source_path The path of the file being imported, as specified on the command line
source_stem The filename of the file being imported, with its extension removed

Specific importers and hooks may provide additional variables.

Supported templates

You can define the following templates.

Amazon Affiliate Program

<tt class="docutils literal">amazon earnings ledger entry</tt>
Imports one transaction per month, summarizing all earnings over the month. Generated from Amazon's "Fee Earnings" report CSV.

Benevity

<tt class="docutils literal">benevity donations ledger entry</tt>

Imports one transaction per row in Benevity's donations report CSV.

This template can use these variables:

<colgroup> <col width="21%" /> <col width="79%" /> </colgroup>
Name Contents
comment The comment from the donor
corporation The name of the participating corporation
disbursement_id The ID of the disbursement from Benevity that includes this donation
donation_amount The amount donated by the individual donor named
donation_fee The fee taken from the donation. This information was not available in reports before 2019, so this amount will always be 0.00 in those reports.
fee_comment The note from the Fee Comment column in the report. This information was not available in reports before 2020, so this field will always be empty in those reports.
frequency The frequency of this donation as indicated in the report
match_amount The amount of the donation match by the participating corporation
match_fee The fee taken from the donation match. This information is not available in reports from before 2019 or since 2020, so this amount will always be 0.00 in those reports.
merchant_fee The merchant fee taken from the donation total. This information was not available in reports before 2019, so this amount will always be 0.00 in those reports.
net_amount The net amount Benevity owes the charity for this transaction: the donation plus the match minus the fees
project The Project named in this row
transaction_id The ID of this specific donation

BrightFunds

<tt class="docutils literal">brightfunds donorreport ledger entry</tt>

Imports one transaction per row in donor report XLS files that BrightFunds mails each month to recipients.

This template can use these variables:

<colgroup> <col width="21%" /> <col width="79%" /> </colgroup>
Name Contents
company_name The company name as reported in the spreadsheet
corporation The company name as detected by the importer (this is usually what you want)
donor_name The donor name as reported in the spreadsheet (usually you want to use <tt class="docutils literal">payee</tt> instead)
donor_email The donor's e-mail address as reported in the spreadsheet
on_behalf_of From the corresponding spreadsheet column
fund From the corresponding spreadsheet column
type From the corresponding spreadsheet column

Eventbrite

<tt class="docutils literal">eventbrite sales ledger entry</tt>

Imports one transaction per ticket sale transaction. Generated from CSV export of Eventbrite sales report.

NOTE: This importer currently only imports sales in the Completed status. In particular, it does not import refunds. It may be extended in the future.

This template can use these variables. Note that many of the informational fields may be empty if their corresponding columns do not appear in the export.

<colgroup> <col width="21%" /> <col width="79%" /> </colgroup>
Name Contents
attendee_id From the <tt class="docutils literal">Attendee #</tt> spreadsheet column
corporation From the <tt class="docutils literal">Company</tt> spreadsheet column
event_id From the corresponding spreadsheet column
event_name From the corresponding spreadsheet column
eventbrite_fees From the corresponding spreadsheet column
order_id From the <tt class="docutils literal">Order #</tt> spreadsheet column
payment_fees From the <tt class="docutils literal">Eventbrite Payment Processing</tt> spreadsheet column
quantity From the corresponding spreadsheet column
tax From the <tt class="docutils literal">Tax Paid</tt> spreadsheet column
ticket_type From the corresponding spreadsheet column
total_fees From the <tt class="docutils literal">Fees Paid</tt> spreadsheet column

GitHub

<tt class="docutils literal">github sponsors ledger entry</tt>

Imports one transaction per processed sponsorship. Generated from CSV export of GitHub Sponsors report for fiscal hosts (NOTE: Not the project reports!)

This template can use these variables. Note that name and email may both be empty if the sponsor chooses not to make that information public.

<colgroup> <col width="21%" /> <col width="79%" /> </colgroup>
Name Contents
email From the <tt class="docutils literal">sponsor email</tt> column
handle From the <tt class="docutils literal">sponsor handle</tt> column, their username on GitHub
transaction_id From the <tt class="docutils literal">transaction id</tt> column

O'Reilly Media

<tt class="docutils literal">oreilly payments ledger entry</tt>

Imports one transaction per payment. Generated from CSV export of O'Reilly's payment history.

This template can use these variables:

<colgroup> <col width="21%" /> <col width="79%" /> </colgroup>
Name Contents
paid_date From the corresponding spreadsheet column. This is usually the end of the month, while the actual transaction date might be slightly off.
<tt class="docutils literal">oreilly royalties ledger entry</tt>

Imports one transaction per royalty period (usually a month, historically a quarter). Generated from CSV export of O'Reilly's royalties statement.

This template can use these variables:

<colgroup> <col width="21%" /> <col width="79%" /> </colgroup>
Name Contents
start_date From the corresponding spreadsheet column
paid_date From the corresponding spreadsheet column. This corresponds to a row in the payment history as well.

Patreon

<tt class="docutils literal">patreon income ledger entry</tt>

Imports one transaction per patron per month. Generated from Patreon's monthly patron report CSVs.

This template can use these variables:

<colgroup> <col width="20%" /> <col width="80%" /> </colgroup>
Name Contents
email The patron's email address on Patreon
tier The name of the tier the patron is supporting at
patreon_id The patron's user ID on Patreon
<tt class="docutils literal">patreon payout ledger entry</tt>

Imports one transaction per month for that month's payout. Generated from Patreon's payout report CSV.

This template can use these variables:

<colgroup> <col width="20%" /> <col width="80%" /> </colgroup>
Name Contents
pledges_amount A decimal with the amount paid out by Patreon to pledges you've made
transfer_amount A decimal with the amount paid out by Patreon to your bank account
<tt class="docutils literal">patreon cardfees ledger entry</tt>
Imports one expense transaction per month for that month's credit card fees. Generated from Patreon's earnings report CSV.
<tt class="docutils literal">patreon servicefees ledger entry</tt>
Imports one expense transaction per month for that month's Patreon service fees. Generated from Patreon's earnings report CSV.
<tt class="docutils literal">patreon vat ledger entry</tt>

Imports one transaction per country per month each time Patreon withheld VAT. Generated from Patreon's VAT report CSV.

This template can use these variables:

<colgroup> <col width="19%" /> <col width="81%" /> </colgroup>
Name Contents
country_name The full name of the country VAT was withheld for
country_code The two-letter ISO country code of the country VAT was withheld for

Stripe

<tt class="docutils literal">stripe payment ledger entry</tt>

Imports one transaction per payment. Generated from Stripe's payments CSV export.

This template can use these variables:

<colgroup> <col width="19%" /> <col width="81%" /> </colgroup>
Name Contents
customer_id The id assigned to this customer by Stripe
customer_email The customer's e-mail address
description The description given to the payment when it was created
fee The amount of fees charged by Stripe for this payment, as a plain decimal number
payment_id The id assigned to this payment by Stripe
payout_id The id of the associated Stripe payout
tax The amount of tax withheld by Stripe for this payment, as a plain decimal number
<tt class="docutils literal">stripe payout ledger entry</tt>

Imports one transaction per payment. Generated from Stripe's payouts CSV export.

This template can use these variables:

<colgroup> <col width="36%" /> <col width="64%" /> </colgroup>
Name Contents  
adjustment_count Decimal from the corresponding CSV column
adjustment_gross Decimal from the corresponding CSV column
adjustment_fees Decimal from the corresponding CSV column
adjustment_net Decimal from the corresponding CSV column
balance_txid Stripe ID of the balance transaction
collected_fee_count Decimal from the corresponding CSV column
collected_fee_gross Decimal from the corresponding CSV column
collected_fee_refund_count Decimal from the corresponding CSV column
collected_fee_refund_gross Decimal from the corresponding CSV column
destination_id Stripe ID of the payout destination account
failure_txid Stripe ID of the failure balance transaction
payment_count Decimal from the corresponding CSV column
payment_gross Decimal from the corresponding CSV column
payment_fees Decimal from the corresponding CSV column
payment_net Decimal from the corresponding CSV column
payout_id Stripe ID of this payout
refund_count Decimal from the corresponding CSV column
refund_gross Decimal from the corresponding CSV column
refund_fees Decimal from the corresponding CSV column
refund_net Decimal from the corresponding CSV column
total_count Decimal from the corresponding CSV column
total_gross Decimal from the corresponding CSV column
total_fees Decimal from the corresponding CSV column
total_net Decimal from the corresponding CSV column
validation_count Decimal from the corresponding CSV column
validation_fees Decimal from the corresponding CSV column
retried_payout_count Decimal from the corresponding CSV column
retried_payout_net Decimal from the corresponding CSV column

YourCause

<tt class="docutils literal">yourcause donations ledger entry</tt>

Imports one transaction per row in YourCause's donations report CSV.

This template can use these variables:

<colgroup> <col width="23%" /> <col width="77%" /> </colgroup>
Name Contents
comment The comment from the donor
corporation The name of the participating corporation
dedication Text from the corresponding CSV column
dedication_type Text from the corresponding CSV column
designation Text from the corresponding CSV column
donor Name of the individual who donated
donor_amount The amount donated by the individual donor named
match_amount The amount of the donation match by the participating corporation
original_amount The amount of the original donation in the original currency
original_currency The local currency the original donation was made in
payment_id The ID of the payment from YourCause that includes this donation
received_amount Decimal from the corresponding CSV column
transaction_id The ID of this donation from YourCause

Other output options

Various options control how import2ledger formats dates and currency amounts. Run <tt class="docutils literal">import2ledger --help</tt> for a full list; they're listed under the "default overrides" section. You can specify these in your configuration file, too. Just change any dashes (<tt class="docutils literal">-</tt>) in the option name to underscores (<tt class="docutils literal">_</tt>), and separate the switch name from your value with <tt class="docutils literal">=</tt>. For example, add this to your configuration file to use ISO 8601-formatted dates in your Ledger entries:

date_format = %%Y-%%m-%%d

(<tt class="docutils literal">%</tt> is a special character in the configuration file syntax. <tt class="docutils literal">%%</tt> represents a literal percent sign.)

Configuration sections

If you keep different sets of books, it might be helpful to have different configuration settings for each. For example, you might adjust the templates for each set of books to use different accounts or tags. Or you might just need to change the formatting of dates or currency amounts.

import2ledger makes this easy by letting you define a new section of options, and then using them all when you import transactions. For example, say you keep two sets of books, one for US accounts and another for European accounts. You could write a configuration file like:

[us]
date_format = %%m/%%d/%%Y
signed_currencies = USD

[eu]
date_format = %%d.%%m.%%Y
signed_currencies = EUR

When you run import2ledger, you can tell it to load options from one of these sections with the option <tt class="docutils literal">--use-config NAME</tt>, or <tt class="docutils literal">-c NAME</tt> for short. In this example, <tt class="docutils literal">import2ledger -c eu</tt> would load the settings for your European books. import2ledger looks for configuration settings in the following places, and uses the first setting it finds:

  1. The configuration section you specify with <tt class="docutils literal">-c</tt>
  2. Command-line options
  3. The <tt class="docutils literal">[DEFAULT]</tt> configuration section
  4. import2ledger defaults

Running import2ledger

Once you've written a configuration file with your templates and any other configuration you need, you can run it with data to import:

$ import2ledger [options …] file1.csv [file2.csv …]

import2ledger will read all the data sources and generate bookkeeping entries from them.

import2ledger needs to be able to seek within the source files. Because of that, your input files need to be normal files, or symlinks to them. Special files like devices and FIFOs aren't supported by import2ledger.

Exit status

import2ledger reports the following exit codes:

<colgroup> <col width="12%" /> <col width="88%" /> </colgroup>
Exit code Meaning
0 Imported data from all source files
1 Internal error (probably a bug)
2 Error in the user's settings, such as a formatting error in a template or date, or a reference to a nonexistent file or configuration file section
11-99 Failed to import data from at least one source file. The exit code is 10 + the number of files that couldn't be imported, up to the maximum exit code of 99.