Changeset - 44762555d5eb
[Not reviewed]
0 3 2
Brett Smith - 6 years ago 2019-02-25 17:39:19
brettcsmith@brettcsmith.org
yourcause: New importer.
5 files changed with 211 insertions and 1 deletions:
0 comments (0 inline, 0 general)
README.rst
Show inline comments
...
 
@@ -315,64 +315,105 @@ Stripe
 
  -------------------------- ----------------------------------------------
 
  payment_fees               Decimal from the corresponding CSV column
 
  -------------------------- ----------------------------------------------
 
  payment_net                Decimal from the corresponding CSV column
 
  -------------------------- ----------------------------------------------
 
  payout_id                  Stripe ID of this payout
 
  -------------------------- ----------------------------------------------
 
  refund_count               Decimal from the corresponding CSV column
 
  -------------------------- ----------------------------------------------
 
  refund_gross               Decimal from the corresponding CSV column
 
  -------------------------- ----------------------------------------------
 
  refund_fees                Decimal from the corresponding CSV column
 
  -------------------------- ----------------------------------------------
 
  refund_net                 Decimal from the corresponding CSV column
 
  -------------------------- ----------------------------------------------
 
  total_count                Decimal from the corresponding CSV column
 
  -------------------------- ----------------------------------------------
 
  total_gross                Decimal from the corresponding CSV column
 
  -------------------------- ----------------------------------------------
 
  total_fees                 Decimal from the corresponding CSV column
 
  -------------------------- ----------------------------------------------
 
  total_net                  Decimal from the corresponding CSV column
 
  -------------------------- ----------------------------------------------
 
  validation_count           Decimal from the corresponding CSV column
 
  -------------------------- ----------------------------------------------
 
  validation_fees            Decimal from the corresponding CSV column
 
  -------------------------- ----------------------------------------------
 
  retried_payout_count       Decimal from the corresponding CSV column
 
  -------------------------- ----------------------------------------------
 
  retried_payout_net         Decimal from the corresponding CSV column
 
  ========================== ==============================================
 

	
 
YourCause
 
^^^^^^^^^
 

	
 
``yourcause donations ledger entry``
 
  Imports one transaction per row in YourCause's donations report CSV.
 

	
 
  This template can use these variables:
 

	
 
  ================= ==========================================================
 
  Name              Contents
 
  ================= ==========================================================
 
  comment           The comment from the donor
 
  ----------------- ----------------------------------------------------------
 
  corporation       The name of the participating corporation
 
  ----------------- ----------------------------------------------------------
 
  dedication        Text from the corresponding CSV column
 
  ----------------- ----------------------------------------------------------
 
  dedication_type   Text from the corresponding CSV column
 
  ----------------- ----------------------------------------------------------
 
  designation       Text from the corresponding CSV column
 
  ----------------- ----------------------------------------------------------
 
  donor             Name of the individual who donated
 
  ----------------- ----------------------------------------------------------
 
  donor_amount      The amount donated by the individual donor named
 
  ----------------- ----------------------------------------------------------
 
  match_amount      The amount of the donation match by the participating
 
                    corporation
 
  ----------------- ----------------------------------------------------------
 
  original_amount   The amount of the original donation in the original
 
                    currency
 
  ----------------- ----------------------------------------------------------
 
  original_currency The local currency the original donation was made in
 
  ----------------- ----------------------------------------------------------
 
  payment_id        The ID of the payment from YourCause that includes this
 
                    donation
 
  ----------------- ----------------------------------------------------------
 
  received_amount   Decimal from the corresponding CSV column
 
  ----------------- ----------------------------------------------------------
 
  transaction_id    The ID of this donation from YourCause
 
  ================= ==========================================================
 

	
 
Other output options
 
~~~~~~~~~~~~~~~~~~~~
 

	
 
Various options control how import2ledger formats dates and currency amounts.  Run ``import2ledger --help`` for a full list; they're listed under the "default overrides" section.  You can specify these in your configuration file, too.  Just change any dashes (``-``) in the option name to underscores (``_``), and separate the switch name from your value with ``=``.  For example, add this to your configuration file to use ISO 8601-formatted dates in your Ledger entries::
 

	
 
  date_format = %%Y-%%m-%%d
 

	
 
(``%`` is a special character in the configuration file syntax.  ``%%`` represents a literal percent sign.)
 

	
 
Configuration sections
 
~~~~~~~~~~~~~~~~~~~~~~
 

	
 
If you keep different sets of books, it might be helpful to have different configuration settings for each.  For example, you might adjust the templates for each set of books to use different accounts or tags.  Or you might just need to change the formatting of dates or currency amounts.
 

	
 
import2ledger makes this easy by letting you define a new section of options, and then using them all when you import transactions.  For example, say you keep two sets of books, one for US accounts and another for European accounts.  You could write a configuration file like::
 

	
 
  [us]
 
  date_format = %%m/%%d/%%Y
 
  signed_currencies = USD
 

	
 
  [eu]
 
  date_format = %%d.%%m.%%Y
 
  signed_currencies = EUR
 

	
 
When you run import2ledger, you can tell it to load options from one of these sections with the option ``--use-config NAME``, or ``-c NAME`` for short.  In this example, ``import2ledger -c eu`` would load the settings for your European books.  import2ledger looks for configuration settings in the following places, and uses the first setting it finds:
 

	
 
1. The configuration section you specify with ``-c``
 
2. Command-line options
 
3. The ``[DEFAULT]`` configuration section
 
4. import2ledger defaults
 

	
 
Running import2ledger
import2ledger/importers/yourcause.py
Show inline comments
 
new file 100644
 
import decimal
 

	
 
try:
 
    import enum
 
except ImportError:
 
    import enum34 as enum
 

	
 
from . import _csv
 
from .. import strparse
 

	
 
ZERO_DECIMAL = decimal.Decimal(0)
 

	
 
class _DonorType(enum.Enum):
 
    Individual = 1
 
    Company = 2
 

	
 
    @classmethod
 
    def from_donation_type(cls, donation_type):
 
        if donation_type.startswith('Individual-'):
 
            return cls.Individual
 
        elif donation_type.startswith('Employer Match-'):
 
            return cls.Company
 
        else:
 
            raise ValueError("unknown donation type {!r}".format(donation_type))
 

	
 

	
 
class DonationsImporter(_csv.CSVImporterBase):
 
    ANONYMOUS = "Anonymous"
 
    DATE_FMT = '%m/%d/%Y %I:%M:%S %p'
 
    DONATION_TYPE_FIELD = 'Donation Type'
 
    ENTRY_SEED = {
 
        'currency': 'USD',
 
        'donor_amount': ZERO_DECIMAL,
 
        'match_amount': ZERO_DECIMAL,
 
    }
 
    DECIMAL_FIELDS = {
 
        'Local Currency Receipt Amount': 'original_amount',
 
        'Received Amount': 'received_amount',
 
        'Transaction Amount': 'amount',
 
    }
 
    NEEDED_FIELDS = frozenset([
 
        DONATION_TYPE_FIELD,
 
        'Donor First Name',
 
        'Donor Last Name',
 
        'Match Donor First Name',
 
        'Match Donor Last Name',
 
        *DECIMAL_FIELDS,
 
    ])
 
    COPIED_FIELDS = {
 
        'Company_Name': 'corporation',
 
        'Dedication Type': 'dedication_type',
 
        'Dedication': 'dedication',
 
        'Designation': 'designation',
 
        'Local Currency Type': 'original_currency',
 
        'Payment ID': 'payment_id',
 
        'Transaction_ID': 'transaction_id',
 
    }
 

	
 
    def _donor_name(self, row, *keys):
 
        names = [row[key] for key in keys]
 
        if any(s and s != self.ANONYMOUS for s in names):
 
            return ' '.join(names).strip()
 
        else:
 
            return self.ANONYMOUS
 

	
 
    def _read_row(self, row):
 
        donor_type = _DonorType.from_donation_type(row[self.DONATION_TYPE_FIELD])
 
        retval = {retkey: strparse.currency_decimal(row[rowkey])
 
                  for rowkey, retkey in self.DECIMAL_FIELDS.items()}
 
        retval['date'] = strparse.date(row['Donation_Date'], self.DATE_FMT)
 
        if donor_type is _DonorType.Individual:
 
            retval['donor'] = self._donor_name(row, 'Donor First Name', 'Donor Last Name')
 
            retval['donor_amount'] = retval['amount']
 
            retval['payee'] = retval['donor']
 
        else:
 
            retval['donor'] = self._donor_name(row, 'Match Donor First Name', 'Match Donor Last Name')
 
            retval['match_amount'] = retval['amount']
 
            retval['payee'] = row['Company_Name']
 
        return retval
setup.py
Show inline comments
 
#!/usr/bin/env python3
 

	
 
import sys
 

	
 
from setuptools import setup, find_packages
 

	
 
REQUIREMENTS = {
 
    'install_requires': [
 
        'babel',
 
        'enum34;python_version<"3.4"',
 
    ],
 
    'setup_requires': ['pytest-runner'],
 
    'extras_require': {
 
        'brightfunds': ['xlrd'],
 
        'nbpy2017': ['beautifulsoup4', 'html5lib'],
 
    },
 
}
 

	
 
all_extras_require = [
 
    req for reqlist in REQUIREMENTS['extras_require'].values() for req in reqlist
 
]
 

	
 
REQUIREMENTS['extras_require']['all_importers'] = all_extras_require
 
REQUIREMENTS['tests_require'] = [
 
    'pytest',
 
    'PyYAML',
 
    *all_extras_require,
 
]
 

	
 
setup(
 
    name='import2ledger',
 
    description="Import different sources of financial data to Ledger",
 
    version='0.7.0',
 
    version='0.8.0',
 
    author='Brett Smith',
 
    author_email='brettcsmith@brettcsmith.org',
 
    license='GNU AGPLv3+',
 

	
 
    packages=find_packages(include=['import2ledger', 'import2ledger.*']),
 
    entry_points={
 
        'console_scripts': ['import2ledger = import2ledger.__main__:main'],
 
    },
 

	
 
    **REQUIREMENTS,
 
)
tests/data/YourCause.csv
Show inline comments
 
new file 100644
 
Donation_Date,Company_Name,Transaction_ID,Donation Type,Transaction Amount,Received Amount,Is Disbursed?,Payment ID,Payment Create Date,Payment Status Date,Payment Status,Donor Type,Donor ID,Donor First Name,Donor Last Name,Donor Full Name,Donor Email Address,Donor Address,Donor Address2,Donor City,Donor State/Province/Region,Donor Postal Code,Donor Country,Match Donor ID,Match Donor First Name,Match Donor Last Name,Dedication Type,Dedication,Designation,Registration ID,Designated Charity Name,Donation Status,Alternate Recognition Name,Segment Name,Local Currency Receipt Amount,Local Currency Type
 
12/20/2018 1:13:00 PM,Company A,2-34567891,Individual-Credit Card,12,12,TRUE,10006789,01/22/2019 5:46:00 PM,,Outstanding,Individual,E1,Dakota,Smith,Dakota Smith,ds@example.com,123 Any Street,,Anytown,Maine,01234,US,,,,,,Note,410123456,Test Charity,Disbursed,,US Segment,12,USD
 
12/31/2018 10:17:00 PM,Company B,2-34567893,Individual-Credit Card,14,14,TRUE,10006789,01/22/2019 5:46:00 PM,,Outstanding,Individual,E2,,,,,,,,,,,,,,,,,410123456,Test Charity,Disbursed,,US Segment,14,USD
 
12/20/2018 1:13:00 PM,Company A,2-34567895,Employer Match-Credit Card Match,12,12,TRUE,10006789,01/22/2019 5:46:00 PM,,Outstanding,Company,C1,,,Company A,,,,,,,,E1,Dakota,Smith,,,Note,410123456,Test Charity,Disbursed,,US Segment,12,USD
 
12/31/2018 10:17:00 PM,Company B,2-34567897,Employer Match-Credit Card,14,14,TRUE,10006789,01/22/2019 5:46:00 PM,,Outstanding,Individual,E2,,,,,,,,,,,,,,,,,410123456,Test Charity,Disbursed,,US Segment,14,USD
 
12/15/2018 10:17:00 AM,Company C,2-34567899,Individual-Credit Card,12.50000,12.50000,TRUE,10006789,01/22/2019 5:46:00 PM,,Outstanding,Individual,E3,Alex,Jones,Alex Jones,,,,,,,,,,,,,,410123456,Test Charity,Disbursed,,US Segment,10,GBP
tests/data/imports.yml
Show inline comments
...
 
@@ -565,32 +565,116 @@
 
  importer: oreilly.PaymentsImporter
 
  expect:
 
    - date: !!python/object/apply:datetime.date [2018, 3, 29]
 
      paid_date: !!python/object/apply:datetime.date [2018, 3, 29]
 
      currency: USD
 
      amount: !!python/object/apply:decimal.Decimal ["29.34"]
 
      payee: "O'Reilly Media, Inc."
 
    - date: !!python/object/apply:datetime.date [2017, 4, 27]
 
      paid_date: !!python/object/apply:datetime.date [2017, 4, 28]
 
      currency: USD
 
      amount: !!python/object/apply:decimal.Decimal ["26.91"]
 
      payee: "O'Reilly Media, Inc."
 
    - date: !!python/object/apply:datetime.date [2017, 4, 24]
 
      paid_date: !!python/object/apply:datetime.date [2017, 4, 28]
 
      currency: USD
 
      amount: !!python/object/apply:decimal.Decimal ["21.48"]
 
      payee: "O'Reilly Media, Inc."
 
    - date: !!python/object/apply:datetime.date [2016, 12, 16]
 
      paid_date: !!python/object/apply:datetime.date [2016, 12, 16]
 
      currency: USD
 
      amount: !!python/object/apply:decimal.Decimal ["26.19"]
 
      payee: "O'Reilly Media, Inc."
 
    - date: !!python/object/apply:datetime.date [2016, 11, 29]
 
      paid_date: !!python/object/apply:datetime.date [2016, 11, 30]
 
      currency: USD
 
      amount: !!python/object/apply:decimal.Decimal ["27.58"]
 
      payee: "O'Reilly Media, Inc."
 
    - date: !!python/object/apply:datetime.date [2010, 3, 31]
 
      paid_date: !!python/object/apply:datetime.date [2010, 3, 31]
 
      currency: USD
 
      amount: !!python/object/apply:decimal.Decimal ["73.85"]
 
      payee: "O'Reilly Media, Inc."
 

	
 
- source: YourCause.csv
 
  importer: yourcause.DonationsImporter
 
  expect:
 
    - date: !!python/object/apply:datetime.date [2018, 12, 20]
 
      currency: USD
 
      original_currency: USD
 
      payment_id: "10006789"
 
      transaction_id: 2-34567891
 
      amount: !!python/object/apply:decimal.Decimal [12]
 
      original_amount: !!python/object/apply:decimal.Decimal [12]
 
      received_amount: !!python/object/apply:decimal.Decimal [12]
 
      donor_amount: !!python/object/apply:decimal.Decimal [12]
 
      match_amount: !!python/object/apply:decimal.Decimal [0]
 
      dedication_type: ""
 
      dedication: ""
 
      designation: Note
 
      payee: Dakota Smith
 
      donor: Dakota Smith
 
      corporation: Company A
 
    - date: !!python/object/apply:datetime.date [2018, 12, 31]
 
      currency: USD
 
      original_currency: USD
 
      payment_id: "10006789"
 
      transaction_id: 2-34567893
 
      amount: !!python/object/apply:decimal.Decimal [14]
 
      original_amount: !!python/object/apply:decimal.Decimal [14]
 
      received_amount: !!python/object/apply:decimal.Decimal [14]
 
      donor_amount: !!python/object/apply:decimal.Decimal [14]
 
      match_amount: !!python/object/apply:decimal.Decimal [0]
 
      dedication_type: ""
 
      dedication: ""
 
      designation: ""
 
      payee: Anonymous
 
      donor: Anonymous
 
      corporation: Company B
 
    - date: !!python/object/apply:datetime.date [2018, 12, 20]
 
      currency: USD
 
      original_currency: USD
 
      payment_id: "10006789"
 
      transaction_id: 2-34567895
 
      amount: !!python/object/apply:decimal.Decimal [12]
 
      original_amount: !!python/object/apply:decimal.Decimal [12]
 
      received_amount: !!python/object/apply:decimal.Decimal [12]
 
      donor_amount: !!python/object/apply:decimal.Decimal [0]
 
      match_amount: !!python/object/apply:decimal.Decimal [12]
 
      dedication_type: ""
 
      dedication: ""
 
      designation: Note
 
      payee: Company A
 
      donor: Dakota Smith
 
      corporation: Company A
 
    - date: !!python/object/apply:datetime.date [2018, 12, 31]
 
      currency: USD
 
      original_currency: USD
 
      payment_id: "10006789"
 
      transaction_id: 2-34567897
 
      amount: !!python/object/apply:decimal.Decimal [14]
 
      original_amount: !!python/object/apply:decimal.Decimal [14]
 
      received_amount: !!python/object/apply:decimal.Decimal [14]
 
      donor_amount: !!python/object/apply:decimal.Decimal [0]
 
      match_amount: !!python/object/apply:decimal.Decimal [14]
 
      dedication_type: ""
 
      dedication: ""
 
      designation: ""
 
      payee: Company B
 
      donor: Anonymous
 
      corporation: Company B
 
    - date: !!python/object/apply:datetime.date [2018, 12, 15]
 
      currency: USD
 
      original_currency: GBP
 
      payment_id: "10006789"
 
      transaction_id: 2-34567899
 
      amount: !!python/object/apply:decimal.Decimal ["12.50"]
 
      original_amount: !!python/object/apply:decimal.Decimal [10]
 
      received_amount: !!python/object/apply:decimal.Decimal ["12.50"]
 
      donor_amount: !!python/object/apply:decimal.Decimal ["12.50"]
 
      match_amount: !!python/object/apply:decimal.Decimal [0]
 
      dedication_type: ""
 
      dedication: ""
 
      designation: ""
 
      payee: Alex Jones
 
      donor: Alex Jones
 
      corporation: Company C
0 comments (0 inline, 0 general)