A customizable light-weight SQL query tool that works on tabular data, including Beancount.
License: GNU General Public License v2.0
Original report by Chad Whitacre (Bitbucket: chadwhitacre, GitHub: chadwhitacre).
It's easy but annoying to calculate it manually from the subtotals for income and expense.
I hate doing manual formatting of the code, what do you think about deciding on some auto formatter and enforcing it?
I was thinking about using https://github.com/google/yapf, but I am not sure if I should use pep8 (default) or google. Any preferences?
@dnicolodi - I was thinking about setting up the mypy type checker in github actions (after fixing existing errors). Is that ok, or would you prefer another type checker (e.g. pytype)?
The CSV table renderer goes the same column values formatting code used by the display renderer. Most of the work done by the renderers is not required for exporting to CSV. In particular the CSV fields do not need to be aligned and thus the double pass through the data required to compute alignment can be spared.
Original report by Martin Blais (Bitbucket: blais, GitHub: blais).
Hi
I notice that beanquery is using the readline stdlib. This library is (I believe) not available to Python Window's installations. Is the lack of support for Windows intentional?
For reasons that I'm not sure I understand, the PLY author decided to stop upload new version of PLY to PyPI and suggests to embed a copy in the projects that use it https://github.com/dabeaz/ply/blob/559a7a1feabb0853be79d4f4dd9e9e9e5ea0d08c/README.md?plain=1#L30 and dabeaz/ply#255
I am not aware of any bug fixed in PLY after the last release available on PyPI, but moving forward we can think about copying a more recent version into the repository or to move to SLY.
SELECT YEAR, SUM(position) WHERE account ~ '^Income' GROUP BY YEAR ORDER BY YEAR
Unfortunately, the financial year is not the calendar year.
It would be nice for BQL to somehow have the concept of a financial year.
(FWIW, Fava has support for financial years)
SELECT
date,
payee,
narration,
account,
position
WHERE
account ~ "Pending" AND ("trip-montreal-july-2015" in tags)
Original report by Martin Michlmayr (Bitbucket: tbm13, GitHub: tbm).
I posted this to the beancount list and Martin said this isn't supported at the moment. Opening a ticket.
I have expenses in one currency (EUR) and want to convert all of them to another (USD) on the day of the transaction.
This gives the correct result because I pass "date" to CONVERT():
beancount> SELECT sum(convert(position, "USD", date)) WHERE account ~ '^Assets:A';
sum_conver
----------
107.50 USD
But I'd like to see a "register" type view with each individual transaction and the total.
Test case:
plugin "beancount.plugins.implicit_prices"
2000-01-01 open Assets:A
2000-01-01 open Assets:B
2018-12-05 * "Test 1"
Assets:A 20 EUR @ 1.10 USD
Assets:B
2018-12-06 * "Test 2"
Assets:A 45 EUR {1.14 USD} @ 1.14 USD
Assets:B
2018-12-07 * "Test 3"
Assets:A 30 EUR {1.14 USD}
Assets:B
Original report by Martin Michlmayr (Bitbucket: tbm13, GitHub: tbm).
Martin mentioned on the list recently:
one can just use a SQL command, something like SELECT location WHERE
date >= 2019-02-01 and flag ='!' and iterate using "next-error" in
Emacs. Perhaps that should be documented better.
Zack said it's not mentioned in the BQL documentation.
Original report by Martin Blais (Bitbucket: blais, GitHub: blais).
Must be done, even before revamping the SQL shell.
Original report by Martin Michlmayr (Bitbucket: tbm13, GitHub: tbm).
I'm trying to do an ORDER BY query but I get an error because some results don't have values:
beancount> select account, any_meta('who'), sum(position) WHERE account ~ 'Assets:Receivable' GROUP BY 1, 2
account any_me sum_positi
----------------- ------ ----------
Assets:Receivable Martin 100.00 EUR
Assets:Receivable 100.00 EUR
beancount>
With ORDER BY 2:
beancount> select account, any_meta('who'), sum(position) WHERE account ~ 'Assets:Receivable' GROUP BY 1, 2 ORDER BY 2
Traceback (most recent call last):
File "/usr/lib/python3.5/cmd.py", line 214, in onecmd
func = getattr(self, 'do_' + cmd)
AttributeError: 'BQLShell' object has no attribute 'do_select'
...
File "/home/tbm/.local/lib/python3.5/site-packages/beancount/query/query_execute.py", line 328, in execute_query
reverse=(query.ordering == 'DESC'))
TypeError: unorderable types: NoneType() < str()
Test case:
plugin "beancount.plugins.auto"
2018-04-22 * "Test 1"
Assets:Receivable 100.00 EUR
who: "Martin"
Assets:Bank -100.00 EUR
2018-04-22 * "Test 2"
Assets:Receivable 100.00 EUR
Assets:Bank -100.00 EUR
When the development focus will switch to performance, we may investigate an alternative way of handling NULL values. Instead than bubbling up NULLs through the BQL call stack, we can simply raise a dedicate exception, rewinding the stack up till the first function that handles it. This has the advantage of removing the code that check function arguments for None and will only require to replace all return None into raise NullValue. The implementation of functions that handle NULL values will just have to wrap arguments evaluation with something like:
def maybenull(x, context):
try:
return x(context)
except NullValue:
return NoneOriginal report by Martin Michlmayr (Bitbucket: tbm13, GitHub: tbm).
beancount in general cares about the order of transactions, including date ordering, but bean-query's results are non deterministic.
I know I can add an ORDER by date but shouldn't that be automatic. Moreover, will ORDER by date keep the order of transactions in the file if they are on the same day.
beancount> SELECT date, position, sum(position), convert(position, "USD", date), sum(position) WHERE account ~ '^Assets:A';
date position sum_position convert_p sum_position_1
---------- ----------------- ----------------- --------- -----------------
2018-12-07 30 EUR {1.14 USD} 30 EUR {1.14 USD} 34.20 USD 30 EUR {1.14 USD}
2018-12-07 45 EUR {1.14 USD} 45 EUR {1.14 USD} 51.30 USD 45 EUR {1.14 USD}
2018-12-05 20 EUR 20 EUR 22.80 USD 20 EUR
beancount> exit
41538:tbm@jirafa: ~/scratch/cvs/beancount] bean-query test
Input file: "Beancount"
Ready with 7 directives (6 postings in 3 transactions).
beancount> SELECT date, position, sum(position), convert(position, "USD", date), sum(position) WHERE account ~ '^Assets:A';
date position sum_position convert_p sum_position_1
---------- ----------------- ----------------- --------- -----------------
2018-12-07 45 EUR {1.14 USD} 45 EUR {1.14 USD} 51.30 USD 45 EUR {1.14 USD}
2018-12-07 30 EUR {1.14 USD} 30 EUR {1.14 USD} 34.20 USD 30 EUR {1.14 USD}
2018-12-05 20 EUR 20 EUR 22.80 USD 20 EUR
beancount>
beancount> exit
41539:tbm@jirafa: ~/scratch/cvs/beancount] bean-query test
Input file: "Beancount"
Ready with 7 directives (6 postings in 3 transactions).
beancount> SELECT date, position, sum(position), convert(position, "USD", date), sum(position) WHERE account ~ '^Assets:A';
date position sum_position convert_p sum_position_1
---------- ----------------- ----------------- --------- -----------------
2018-12-07 30 EUR {1.14 USD} 30 EUR {1.14 USD} 34.20 USD 30 EUR {1.14 USD}
2018-12-05 20 EUR 20 EUR 22.80 USD 20 EUR
2018-12-07 45 EUR {1.14 USD} 45 EUR {1.14 USD} 51.30 USD 45 EUR {1.14 USD}
beancount>
Test case:
plugin "beancount.plugins.implicit_prices"
2000-01-01 open Assets:A
2000-01-01 open Assets:B
2018-12-05 * "Test 1"
Assets:A 20 EUR @ 1.14 USD
Assets:B
2018-12-07 * "Test 2"
Assets:A 45 EUR {1.14 USD} @ 1.14 USD
Assets:B
2018-12-07 * "Test 3"
Assets:A 30 EUR {1.14 USD}
Assets:B
Currently BQL uses single ' and double " quotes interchangeably as string delimiters and does not have the concept of quoted identifiers. This make is impossible to have column names that collide with BQL keywords or that contain non-alphanumeric characters. This is sometimes annoying.
Standard SQL uses single quotes ' for string literals and double quotes " for quoting identifiers. https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS
Should BQL do the same? It would be a tiny patch to the parser. However, I expect this to break a fair number of user queries.
@blais, do you have an opinion?
See TODO list for more details.
Is there a way to compare balances by month and/or by year. Ie: say I wanted to know how much I spent on each category for each of the last 24 months?
Not yet, but I've been thinking about that and wanting it for a long while.
I think the most generic and flexible way to implement that would be via a combination of bean-query generating a table like this:
Month Account Balance
... ... ...
and combining that with a separate tool that would pivot Balance on Month vs. Account, to produce something like this:
Month
6/15 7/15 8/15
Account
Expenses:Restaurant ... ... ...
Original report by Adam Wolenc adamdblu.
When displaying a nested target, for example a SUM of positions containing more than one currency, bean-query displays the elements of this field in arbitrary order. Is there a way to tell it to sort them by currency?
This is best illustrated by an example.
2000-01-04 open Assets:HOOL:Unvested:FUTURE2000 HOOL.UNVEST
2000-01-04 open Income:HOOL:Awards HOOL.UNVEST
2001-01-04 open Assets:HOOL:RSURefund USD
2001-01-04 open Income:HOOL:HoolStockUnit USD
2000-01-04 * "Award FUTURE2000"
Income:HOOL:Awards -10 HOOL.UNVEST
Assets:HOOL:Unvested:FUTURE2000 10 HOOL.UNVEST
2001-01-04 * "Vest FUTURE2000"
Income:HOOL:HoolStockUnit -1000.00 USD
Assets:HOOL:RSURefund
Sub accounts under Assets:HOOL now contain two currencies.
Running this query repeatedly results in one of two outputs arbitrarily.
SELECT ROOT(account, 2) AS institution,
SUM(VALUE(position)) AS mkt
GROUP BY institution
ORDER BY mkt, institution DESC;
institution mkt
----------- --------------------
Assets:HOOL 10 HOOL.UNVEST, 1000.00 USD
Income:HOOL -10 HOOL.UNVEST, -1000.00 USD
institution mkt
----------- --------------------
Assets:HOOL 1000.00 USD , 10 HOOL.UNVEST
Income:HOOL -1000.00 USD , -10 HOOL.UNVEST
In management reports prepared according to our country's FRS, the statement of financial position (~balance sheet) is presented with holdings in foreign currency converted to market value, except for non-monetary items. Non-monetary items that are measured in terms of historical cost in a foreign currency are translated using the exchange rates as at the dates of the initial transactions. As an example of such assets, prepayments can be monetary or non-monetary. The statement of financial position contains a mix of both (some presented at value, and some presented at cost), and we have to sum() a mix of them to get the total assets, and total equity and liabilities.
Because there are a mix of both and we want the sum over them, functions such as sum(value(position)) or sum(cost(position)) do not return the correct result as they use the market value or cost for all the positions. I browsed the beanquery code to see if metadata such as value-at-cost: 1 after the open statement can be added and then used in the function that handles value(position) in query_env.py. The function would check if the metadata is set for that account and use cost or value appropriately. However, a Position is just an Amount and Cost and metadata about the account cannot be accessed, so this way didn't work out.
Please could you suggest what I could do to achieve a sum(position) query where it uses the value or the cost depending on the type of asset?
I want to see a list of outstanding invoices. Given the example below, I can use:
beancount> SELECT ENTRY_META('invoice'), SUM(position) WHERE account ~ '^Assets:Receivable' GROUP BY ENTRY_META('invoice')
ent sum_posit
--- ---------
A01
B01 30.00 USD
Unfortunately, there's no way to exclude positions that are 0, i.e. that have been paid.
It would be great for BQL to allow queries on aggregates.
Example:
2020-01-01 open Assets:Bank
2020-01-01 open Assets:Receivable
2020-01-01 open Income:Sponsorship
2020-03-01 * "Sponsorship from A"
invoice: "A01"
Assets:Receivable 100.00 USD
Income:Sponsorship -100.00 USD
2020-03-01 * "Sponsorship from B"
invoice: "B01"
Assets:Receivable 30.00 USD
Income:Sponsorship -30.00 USD
2020-03-10 * "Payment from A"
invoice: "A01"
Assets:Bank 100.00 USD
Assets:Receivable -100.00 USD
When I mentioned this on the mailing list, @blais responded:
This would need a filtering expression of the aggregate, it's not
implemented. SQL defines a syntax thought.
This works, and it should not (probably just evaluates to true, but it's misleading):
select * from 'Income:US:Hoogle:Vacation';
Original report by Red Street (GitHub: redstreet).
INT(str/int/Decimal): converts to an integer and returns it.
Example use case: convert metadata into an integer, and use it in computations.
Original report by Daniel Bos (Bitbucket: corani, GitHub: corani).
To make documenting BQL queries easier, it would be nice if it were possible to use comments. SQL supports two ways of adding comments:
Original report by Martin Michlmayr (Bitbucket: tbm13, GitHub: tbm).
I want to use CONVERT() to convert something to its historical cost (i.e. exchange rate on date of transaction, without later revaluation).
I don't think this is currently possible with beancount. Maybe CONVERT() can be extended to allow an optional DATE.
Example:
2014-01-01 open Expenses:Test
2014-01-01 open Assets:Cash
2014-01-09 * "Test" "Test 1"
Expenses:Test 10.00 EUR @ 0.8283 GBP
Assets:Cash -8.28 GBP
2014-06-10 * "Test" "Test 2"
Expenses:Test 10.00 EUR @ 0.8106 GBP
Assets:Cash -8.11 GBP
2014-07-30 * "Test" "Test 3"
Expenses:Test 10.00 EUR @ 0.7914 GBP
Assets:Cash -7.91 GBP
2014-09-11 price EUR 0.8011 GBP
2014-12-31 price EUR 0.7825 GBP
select date, position, convert(position, "GBP"), balance where account ~ 'Expenses'
date position convert_po balance
---------- --------- ---------- ---------
2014-01-09 10.00 EUR 7.8250 GBP 10.00 EUR
2014-06-10 10.00 EUR 7.8250 GBP 20.00 EUR
2014-07-30 10.00 EUR 7.8250 GBP 30.00 EUR
It uses the latest exchange rate (0.7825) for all transactions.
Original report by Martin Michlmayr (Bitbucket: tbm13, GitHub: tbm).
I specify a column name via "AS foo" but then "WHERE foo" says that "foo" is an invalid column. It would be nice if it would recognise the custom column names I make via "AS".
beancount> SELECT ANY_META('entity') as entity WHERE ANY_META('entity') ~ 'Michlmayr';
entity
---------
Michlmayr
Michlmayr
Michlmayr
Michlmayr
beancount> SELECT ANY_META('entity') as entity WHERE entity ~ 'Michlmayr';
ERROR: Invalid column name 'entity' in WHERE clause context.
beancount>
There are currently two ways to input a date literal: the regular YYYY-MM-DD syntax (example SELECT 2022-04-08) and the extended #"string" syntax (example SELECT #"April 8, 2022"). The second uses the heuristic in the dateutils parser to determine the date string format.
The problem is that the format is ambiguous: which date is #"10.11.12"? The ambiguity is even greater because, while Beancount uses the year-month-day format for dates, the heuristic in dateutils prefer the month-day-year format. Having an European background, the most natural date format for me is day-month-year, but dateutils heuristic does not like this format at all.
Would anyone complain loudly if I remove support for the liberally parsed date literals?
I am also inclined to remove the liberal parsing from the parse_date(string, format) function done when the second argument is not provided so that we can fold this function into a special case of the date() constructor. Is anyone relaying on the magic parsing?
Original report by Martin Michlmayr (Bitbucket: tbm13, GitHub: tbm).
I can do this to make sure metadata does NOT exist:
beancount> SELECT account WHERE NOT ANY_META('program');
If I drop the NOT, it also works fine:
beancount> SELECT account WHERE ANY_META('program');
However, this fails:
beancount> SELECT account WHERE ANY_META('program') AND NOT ANY_META('entity');
...
return self.operator(self.left(context), self.right(context))
TypeError: unsupported operand type(s) for &: 'NoneType' and 'bool'
It doesn't make sense that it works on its own but not when combined with another query of the same type.
As a workaround, ANY_META('program') ~ '' AND... it works; but I think the simple query should work.
Original report by Martin Blais (Bitbucket: blais, GitHub: blais).
I'm tracking my gross pay into my checking account from Income:Salary, and then transfering out from there into expense:income-tax, assets:401k, etc...
At the end of the year, I'd like to pull a report that shows me how much total I contributed to the Assets:401K account. However, I can't simply pull a select sum(position) where account ~ "401K" because I am also tracking purchases/sales in sub-accounts of Assets:401K. Rather, I'd like to do something like select account,sum(position) where account ~ "401K" and other_accounts ~ "Checking", but this produces an error:
#!python
beancount> select account,sum(position) where account ~ "Checking" and other_accounts ~ "401K"
Traceback (most recent call last):
File "/usr/lib64/python3.6/cmd.py", line 214, in onecmd
func = getattr(self, 'do_' + cmd)
AttributeError: 'BQLShell' object has no attribute 'do_select'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/me/.config/pyVirtualEnvs/beancount_local/lib/python3.6/site-packages/beancount/query/shell.py", line 270, in run_parser
self.dispatch(statement)
File "/home/me/.config/pyVirtualEnvs/beancount_local/lib/python3.6/site-packages/beancount/query/shell.py", line 250, in dispatch
return method(statement)
File "/home/me/.config/pyVirtualEnvs/beancount_local/lib/python3.6/site-packages/beancount/query/shell.py", line 418, in on_Select
self.options_map)
File "/home/me/.config/pyVirtualEnvs/beancount_local/lib/python3.6/site-packages/beancount/query/query_execute.py", line 316, in execute_query
if c_where is None or c_where(context):
File "/home/me/.config/pyVirtualEnvs/beancount_local/lib/python3.6/site-packages/beancount/query/query_compile.py", line 121, in __call__
return self.operator(self.left(context), self.right(context))
File "/home/me/.config/pyVirtualEnvs/beancount_local/lib/python3.6/site-packages/beancount/query/query_compile.py", line 121, in __call__
return self.operator(self.left(context), self.right(context))
File "/home/me/.config/pyVirtualEnvs/beancount_local/lib/python3.6/site-packages/beancount/query/query_compile.py", line 164, in match
return bool(re.search(right, left, re.IGNORECASE))
File "/home/me/.config/pyVirtualEnvs/beancount_local/lib64/python3.6/re.py", line 182, in search
return _compile(pattern, flags).search(string)
TypeError: expected string or bytes-like object
beancount> select account,sum(position) where account ~ "401K" and other_accounts ~ "Checking"
Traceback (most recent call last):
File "/usr/lib64/python3.6/cmd.py", line 214, in onecmd
func = getattr(self, 'do_' + cmd)
AttributeError: 'BQLShell' object has no attribute 'do_select'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/me/.config/pyVirtualEnvs/beancount_local/lib/python3.6/site-packages/beancount/query/shell.py", line 270, in run_parser
self.dispatch(statement)
File "/home/me/.config/pyVirtualEnvs/beancount_local/lib/python3.6/site-packages/beancount/query/shell.py", line 250, in dispatch
return method(statement)
File "/home/me/.config/pyVirtualEnvs/beancount_local/lib/python3.6/site-packages/beancount/query/shell.py", line 418, in on_Select
self.options_map)
File "/home/me/.config/pyVirtualEnvs/beancount_local/lib/python3.6/site-packages/beancount/query/query_execute.py", line 316, in execute_query
if c_where is None or c_where(context):
File "/home/me/.config/pyVirtualEnvs/beancount_local/lib/python3.6/site-packages/beancount/query/query_compile.py", line 121, in __call__
return self.operator(self.left(context), self.right(context))
File "/home/me/.config/pyVirtualEnvs/beancount_local/lib/python3.6/site-packages/beancount/query/query_compile.py", line 121, in __call__
return self.operator(self.left(context), self.right(context))
File "/home/me/.config/pyVirtualEnvs/beancount_local/lib/python3.6/site-packages/beancount/query/query_compile.py", line 164, in match
return bool(re.search(right, left, re.IGNORECASE))
File "/home/me/.config/pyVirtualEnvs/beancount_local/lib64/python3.6/re.py", line 182, in search
return _compile(pattern, flags).search(string)
TypeError: expected string or bytes-like object
Is this expected behaviour? Am I doing something wrong? Is there another way to pull the report I want?
When generating the balance of an account (at market value for presentation in a management report) for a previous financial year, e.g., ending on 2021-03-31, we may use the following bean-query SQL statement:
SELECT sum(value(position)) FROM CLOSE ON 2021-04-01 CLEAR WHERE account ~ '^Assets:Current:Cash-And-Short-Term-Deposits';Assume the journal input contains price statements such as:
2021-03-31 price USD 1.3472 SGD
2022-03-31 price USD 1.3534 SGD
Unfortunately, the query above seems to use price statements that came after 2021-03-31 to compute the market value of the position (it seems to use the most recent price statement in the journal input which is from 2022-03-31). It does not appear to regard the CLOSE ON 2021-04-01 to limit looking for prices. I'm not able to generate a table of balances of this year and the previous year in a management report, where the previous year's balances have to match the market rates on that financial year's closing reporting date.
I don't know if this is a bug, or if some other clause has to be used to make bean-query limit looking for price statements up to 2021-03-31. Please can you advise?
Original report by Martin Michlmayr (Bitbucket: tbm13, GitHub: tbm).
The syntax manual says "You can also attach flags to the postings themselves, if you want to flag one of the transaction’s legs in particular"
However, when I query f on an account (posting) I don't see the one marked as !:
beancount> SELECT account, position, flag WHERE account ~ ':A'
account position f
-------- --------- -
Assets:A 11.00 EUR *
Assets:A 22.00 EUR *
Assets:A 33.00 EUR *
beancount>
Is there a different way to query the posting flag?
Test case:
2018-01-01 open Assets:A
2018-01-01 open Assets:B
2018-01-01 txn "Test 1"
aXa: "foo"
Assets:A 11.00 EUR
Assets:B
2018-01-01 txn "Test 2"
axa: "bar"
! Assets:A 22.00 EUR
Assets:B
2018-01-01 txn "Test 3"
axa: "bar"
* Assets:A 33.00 EUR
Assets:B
Original report by Stefano Zacchiroli (Bitbucket: zacchiro, GitHub: zacchiro).
The following query:
SELECT ANY_META('trip') AS trip, SUM(position) AS amount
WHERE account ~ '^Assets:Reimbursable'
AND ANY_META('trip')
GROUP BY trip
when 'trip' might be missing from some transaction fails as follows:
File "/home/zack/.local/lib/python3.6/site-packages/beancount/query/query_compile.py", line 121, in __call__
return self.operator(self.left(context), self.right(context))
TypeError: unsupported operand type(s) for &: 'bool' and 'str'
the string in question is probably "None", i.e., the string representation of Python None values.
It would be nice if missing metadata could be treated as False in boolean context by BQL.
Original report by Martin Blais (Bitbucket: blais, GitHub: blais).
#!text
- BUG: Precision for rendering is broken for some subsets of amounts (these
are 1.20 USD amounts):
beancount> select date, payee, narration, position where account ~
'Expenses:Taxes:TY2015:US:SDI';
date narration posit
---------- - --------------------------------------------------- -----
2015-01-02 GOOGLE INC PAYROLL / GOOGLE INC PAYROLL 1 USD
2015-01-16 GOOGLE INC PAYROLL / GOOGLE INC PAYROLL 1 USD
2015-01-23 GOOGLE INC PAYROLL / GOOGLE INC PAYROLL 0 USD
This one works:
beancount> select date, payee, narration, position where account ~
'Expenses:Taxes:TY2015:US:SDI' and narration ~ 'GOOGLE';
Original report by Harpreet Sangha (Bitbucket: eliptus, GitHub: eliptus).
I noticed that when using ORDER BY with bean-query sort direction cannot be specified per keyword.
For example:
beancount> SELECT year, account, COST(SUM(position)) AS total WHERE account ~ "^Expenses:" GROUP BY 1, 2 ORDER BY 1 ASC, 2 DESC
ERROR: Syntax error near ',' (at 108)
SELECT year, account, COST(SUM(position)) AS total WHERE account ~ "^Expenses:" GROUP BY 1, 2 ORDER BY 1 ASC, 2 DESC
When performing queries, it is often useful to express one of the columns as a percentage of the total. An example I see commonly is reporting on Assets allocation - grouping Assets by some key and reporting the percentage of cost (value) per group. If I did not miss anything, this cannot be currently done with the query language alone. Would you find this useful to add to the query language? In fava, this would allow to replace many plugins with a single query and I would find it useful for command-line usage too.
If interested, what do you think would be the best way to add this? I can think of three possibilities
select position / (select sum(position))) - this would be versatile (would allow for a different normalization, such as division by max) and consistent with SQLtotal, like balance) - this looks quite hacky, but would solve this particular issue and seems the easiest to implementOriginal report by Martin Blais (Bitbucket: blais, GitHub: blais).
This query:
SELECT date, description, tags, convert(position, 'USD'), convert(balance, 'USD'), account
WHERE account ~ 'Accommodation'
ORDER BY joinstr(tags)
does not yield the correct result because the balances aren't sorted correctly.
I believe the ability to compute a running aggregator should be something that's supported in the SQL shell rewrite.
From the "Beancount Query Language" document, section "Future Features". A special PIVOT BY clause will eventually be usable to convert the output from a one-dimensional list of results to a two-dimensional table. For example, the following query:
SELECT
account,
YEAR(date) AS year,
SUM(COST(position)) AS balance
WHERE
account ~ 'Expenses:Food' AND
currency = 'USD' AND
year >= 2012
GROUP BY 1,2
ORDER BY 1,2;
Might generate the following table of results:
account year balance
------------------------ ---- -----------
Expenses:Food:Alcohol 2012 57.91 USD
Expenses:Food:Alcohol 2013 33.45 USD
Expenses:Food:Coffee 2012 42.07 USD
Expenses:Food:Coffee 2013 124.69 USD
Expenses:Food:Coffee 2014 38.74 USD
Expenses:Food:Groceries 2012 2172.97 USD
Expenses:Food:Groceries 2013 2161.90 USD
Expenses:Food:Groceries 2014 2072.36 USD
Expenses:Food:Restaurant 2012 4310.60 USD
Expenses:Food:Restaurant 2013 5053.61 USD
Expenses:Food:Restaurant 2014 4209.06 USD
If you add a PIVOT clause to the query, like this:
…
PIVOT BY account, year;
You would get a table like this:
account/year 2012 2013 2014
------------------------ ----------- ----------- -----------
Expenses:Food:Alcohol 57.91 USD 33.45 USD
Expenses:Food:Coffee 42.07 USD 124.69 USD 38.74 USD
Expenses:Food:Groceries 2172.97 USD 2161.90 USD 2072.36 USD
Expenses:Food:Restaurant 4310.60 USD 5053.61 USD 4209.06 USD
If your table has more than three columns, the non-pivoted column would get expanded horizontally, like this:
SELECT
account,
YEAR(date),
SUM(COST(position)) AS balance,
LAST(date) AS updated
WHERE
account ~ 'Expenses:Food' AND
currency = 'USD' AND
year >= 2012
GROUP BY 1,2
ORDER BY 1, 2;
You would get a table like this:
account year balance updated
------------------------ ---- ----------- ----------
Expenses:Food:Alcohol 2012 57.91 USD 2012-07-17
Expenses:Food:Alcohol 2013 33.45 USD 2013-12-13
Expenses:Food:Coffee 2012 42.07 USD 2012-07-19
Expenses:Food:Coffee 2013 124.69 USD 2013-12-16
Expenses:Food:Coffee 2014 38.74 USD 2014-09-21
Expenses:Food:Groceries 2012 2172.97 USD 2012-12-30
Expenses:Food:Groceries 2013 2161.90 USD 2013-12-31
Expenses:Food:Groceries 2014 2072.36 USD 2014-11-20
Expenses:Food:Restaurant 2012 4310.60 USD 2012-12-30
Expenses:Food:Restaurant 2013 5053.61 USD 2013-12-29
Expenses:Food:Restaurant 2014 4209.06 USD 2014-11-28
Pivoting, this would generate this table:
account/balance,updated 2012/balanc 2012/updat 2013/balanc 2013/updat 2014/balanc 2014/updat
------------------------ ----------- ---------- ----------- ---------- ----------- ----------
Expenses:Food:Alcohol 57.91 USD 2012-07-17 33.45 USD 2013-12-13
Expenses:Food:Coffee 42.07 USD 2012-07-19 124.69 USD 2013-12-16 38.74 USD 2014-09-21
Expenses:Food:Groceries 2172.97 USD 2012-12-30 2161.90 USD 2013-12-31 2072.36 USD 2014-11-20
Expenses:Food:Restaurant 4310.60 USD 2012-12-30 5053.61 USD 2013-12-29 4209.06 USD 2014-11-28
Original report by Martin Michlmayr (Bitbucket: tbm13, GitHub: tbm).
I forgot to convert something from ledger's XX @@ FOO to XX {FOO} syntax and got the following error in bean-query. (I originally found the problem because fava crashed. Below is a simplified query.)
beancount> SELECT account WHERE account_sortkey(account) ~ "^[01]" GROUP BY account, cost_date, currency, account_sortkey(account) ORDER BY account_sortkey(account), cost_date
Traceback (most recent call last):
File "/usr/lib/python3.5/cmd.py", line 214, in onecmd
func = getattr(self, 'do_' + cmd)
AttributeError: 'BQLShell' object has no attribute 'do_SELECT'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/tbm/.local/lib/python3.5/site-packages/beancount/query/shell.py", line 266, in run_parser
self.dispatch(statement)
File "/home/tbm/.local/lib/python3.5/site-packages/beancount/query/shell.py", line 246, in dispatch
return method(statement)
File "/home/tbm/.local/lib/python3.5/site-packages/beancount/query/shell.py", line 414, in on_Select
self.options_map)
File "/home/tbm/.local/lib/python3.5/site-packages/beancount/query/query_execute.py", line 328, in execute_query
reverse=(query.ordering == 'DESC'))
TypeError: unorderable types: NoneType() < datetime.date()
beancount>
Test case:
2010-01-01 open Equity:Opening-Balance
2010-01-01 open Assets:Pension:NEST
2018-01-01 * "Opening balance: NEST"
Assets:Pension:NEST 57.62 GBP
Equity:Opening-Balance
2018-01-01 * "Opening balance: NEST Higher Risk Fund"
fund: "NESTHIGHER"
Assets:Pension:NEST 11.7230 NESTHIGHER {{25.61 GBP, 2017-12-14}}
Equity:Opening-Balance
2018-01-02 * "NEST: Bought NEST Higher Risk Fund with December employee contributions"
fund: "NESTHIGHER"
Assets:Pension:NEST 11.7520 NESTHIGHER @@ 25.61 GBP
Assets:Pension:NEST -25.61 GBP
This should work:
beancount> select account, sum(position) where currency = 'IRAUSD' group by account order by type;
ERROR: All non-aggregates must be covered by GROUP-BY clause in aggregate query.
Original report by Martin Michlmayr (Bitbucket: tbm13, GitHub: tbm).
I have to quote the query name, otherwise I get a syntax error. I think in both of these cases (dash and underscore) there shouldn't be a need to quote the name.
bean-query Financial/Ledger/books.beancount "run fundraiser_2017"
ERROR: Syntax error near '2017' (at 15)
run fundraiser_2017
^
bean-query Financial/Ledger/books.beancount "run fundraiser-2017"
ERROR: Syntax error near '-2017' (at 14)
run fundraiser-2017
^
The documentation says that these are the posting definition location in the source file, however the implementation returns the transaction location. The location column reports the location of the posting. Most likely the implementation must be changed to match the documentation.
An easy change but it would be a backward compatibility break and currently there would not be a way to get to the values for the transaction. This should be fixed when the big implementation generalization work lands.
Original report by Red Street (GitHub: redstreet).
With the "order by" clause, it would be nice to choose ascending order on one key, and descending on another. For eg, the equivalent of: ORDER BY payee ASC, date DESC. Currently, this doesn't seem to work.
Original report by Trevor Davis (Bitbucket: trevorld, GitHub: trevorld).
According to this e-mail in the beancount mailing list you suggested that the "flatten" feature in bean-query now works:
https://groups.google.com/d/msg/beancount/p5_pEz3QiiA/SBk3sotfEQAJ
However when I tried to use it I get the following error:
AttributeError: 'BQLShell' object has no attribute 'do_select'
What I was trying to do was get a version of the "holdings" report with the decimal number of the units column extended from hundredths place to the thousandths place (i.e. report 5.235 shares at a given cost instead of 5.24 shares at a given cost). I need to manually compute capital gains in my HSA account since my provider doesn't track that information at all (but California in their infinite wisdom requires that I pay taxes on them) and right now it is a little bit of trial and error figuring out what my exact cost holdings in the HSA account are since the "holdings" report truncates a digit off my holdings and my provider doesn't track that at all since it isn't a federal requirement for them to do so.
$ cat minimal.beancount
plugin "beancount.plugins.implicit_prices"
plugin "beancount.plugins.sellgains"
option "operating_currency" "USD"
2018-01-01 open Assets:Brokerage
2018-01-01 open Equity:Transfer
2018-01-01 open Income:CG:ST
2018-01-01 * "" "Buy 5 Shares"
Assets:Brokerage 5 SHARE {100 USD}
Equity:Transfer
2018-02-05 * "" "Buy 5 Shares"
Assets:Brokerage 5 SHARE {200 USD}
Equity:Transfer
2018-03-05 * "" "Sell 3 Shares"
Assets:Brokerage -2 SHARE {100 USD} @ 150 USD
Assets:Brokerage -1 SHARE {200 USD} @ 150 USD
Income:CG:ST -50 USD
Equity:Transfer
$ bean-check minimal.beancount
$ bean-report minimal.beancount holdings
Account Units Currency Cost Currency Average Cost Price Book Value Market Value
---------------- ----- -------- ------------- ------------ ------ ---------- ------------
Assets:Brokerage 3.00 SHARE USD 100.00 150.00 300.00 450.00
Assets:Brokerage 4.00 SHARE USD 200.00 150.00 800.00 600.00
---------------- ----- -------- ------------- ------------ ------ ---------- ------------
$ bean-query minimal.beancount "select account, cost(sum(position)), value(sum(position)) where account ~ 'Assets:' group by account flatten"
Traceback (most recent call last):
File "/usr/lib/python3.6/cmd.py", line 214, in onecmd
func = getattr(self, 'do_' + cmd)
AttributeError: 'BQLShell' object has no attribute 'do_select'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/beancount-2.2.0_devel-py3.6-linux-x86_64.egg/beancount/query/shell.py", line 271, in run_parser
self.dispatch(statement)
File "/usr/local/lib/python3.6/dist-packages/beancount-2.2.0_devel-py3.6-linux-x86_64.egg/beancount/query/shell.py", line 251, in dispatch
return method(statement)
File "/usr/local/lib/python3.6/dist-packages/beancount-2.2.0_devel-py3.6-linux-x86_64.egg/beancount/query/shell.py", line 419, in on_Select
self.options_map)
File "/usr/local/lib/python3.6/dist-packages/beancount-2.2.0_devel-py3.6-linux-x86_64.egg/beancount/query/query_execute.py", line 380, in execute_query
result_types, result_rows = flatten_results(result_types, result_rows)
File "/usr/local/lib/python3.6/dist-packages/beancount-2.2.0_devel-py3.6-linux-x86_64.egg/beancount/query/query_execute.py", line 421, in flatten_results
value = value[irow] if irow < len(value) else None
KeyError: 0
Working toward the goal on making beanquery a general purpose SQL-like query language for arbitrary data and make the BQL language more regular, we need to decide what to do with the current BQL function that depend on the row context, in particular the open_meta(), commodity_meta() and similar functions.
In an unpublished branch I already implemented structured types (or composite types in PostgreSQL parlance https://www.postgresql.org/docs/14/rowtypes.html) thus one way to solve the problem would be to make accounts and commodities their own structured types and allow a syntax similar to
SELECT
account.meta["foo"],
position.units.commodity.meta["bar"]
FROM
postings
Another way is to rely on sub-queries to do the lookup:
SELECT
(SELECT meta["foo"] FROM accounts WHERE account = p.account),
(SELECT meta["bar"] FROM commodities WHERE commodity = p.position.units.commodity),
FROM
postings AS p
A solution based on a join
SELECT
account,
position,
commodities.meta["foo"]
FROM
postings, commodities
WHERE
commodities.account = postings.account
would not work because commodities is a table of all the commodities definitions, and not all the commodities used in a Beancount ledger need to defined, thus the above query would return only a subset of the postings.
The BQL syntax for the first solution is definitely more attractive but it is less flexible and would require making the BQL types system more complex to allow accounts and commodities to be at the same time strings and structured types, unless explicit type conversions are required. Something like
SELECT
account(account).meta["foo"],
commodity(position.units.commodity).meta["bar"]
FROM
postings
is not horrible but feels a bit redundant. Going the other way around:
SELECT *
FROM
postings
WHERE
account = account("Assets:Test")
is not much better.
The syntax for the second solution is verbose, but it is the most explicit and it allows for the most flexibility.
Comments?
In a mailing list thread https://groups.google.com/g/beancount/c/4EkTUWoLXwI and in beancount/beancount#688 it was proposed to add the possibility of performing arithmetic operations on Amounts. The only operations that make sense are multiplication and division of amount by an integer or a decimal, and addition and subtraction of amounts.
Multiplication and division by an integer or a decimal are not an issue as these operations always succeed and return another Amount object.
For sums and differences the problem is a bit more complex. What to do when amounts with incompatible commodities are summed or subtracted? Following the SQL philosophy, we do not want to raise an exception based on the data involved in the query. The only sensible thing is to return NULL. However this would be dangerous, because one very common thing to do is to sum these values into an Inventory, and in this case the problem would not be evident at all.
@IguanaBen, what is your use case for addition and subtractions of Amounts in BQL?
Original report by Martin Michlmayr (Bitbucket: tbm13, GitHub: tbm).
Sometimes I want to convert one currency to another at a fixed rate that does not necessarily reflect any actual exchange rate. For example, when doing a net worth converted to one currency, I may have some "long time, real" exchange rate in mind even though it's not the current exchange rate.
So for example I want to say convert all GBP to EUR and assume "1 GBP equals 1.45 EUR". It would be nice if CONVERT() would accept something like this, e.g.
CONVERT(position, "GBP 1.45 EUR") or CONVERT(position, "GBP @ 1.45 EUR")
Original report by Martin Blais (Bitbucket: blais, GitHub: blais).
beancount> select account, sum(cost(position)) where account ~'Assets:.*RRSP' and position != 0 group by 1
Traceback (most recent call last):
File "/usr/local/lib/python3.4/cmd.py", line 214, in onecmd
func = getattr(self, 'do_' + cmd)
AttributeError: 'BQLShell' object has no attribute 'do_select'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/blais/p/beancount/src/python/beancount/query/shell.py", line 223, in default
self.dispatch(statement)
File "/home/blais/p/beancount/src/python/beancount/query/shell.py", line 213, in dispatch
return method(statement)
File "/home/blais/p/beancount/src/python/beancount/query/shell.py", line 356, in on_Select
self.options_map)
File "/home/blais/p/beancount/src/python/beancount/query/query_execute.py", line 251, in execute_query
if c_where is None or c_where(context):
File "/home/blais/p/beancount/src/python/beancount/query/query_compile.py", line 111, in call
return self.operator(self.left(context), self.right(context))
File "/home/blais/p/beancount/src/python/beancount/query/query_compile.py", line 93, in call
return self.operator(self.operand(context))
File "/home/blais/p/beancount/src/python/beancount/query/query_compile.py", line 111, in call
return self.operator(self.left(context), self.right(context))
File "/home/blais/p/beancount/src/python/beancount/core/position.py", line 166, in eq
return (self.number == other.number and
AttributeError: 'int' object has no attribute 'number'
beancount> select account, sum(cost(position)) where account ~'Assets:.*RRSP' and position group by 1
Traceback (most recent call last):
File "/usr/local/lib/python3.4/cmd.py", line 214, in onecmd
func = getattr(self, 'do_' + cmd)
AttributeError: 'BQLShell' object has no attribute 'do_select'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/blais/p/beancount/src/python/beancount/query/shell.py", line 223, in default
self.dispatch(statement)
File "/home/blais/p/beancount/src/python/beancount/query/shell.py", line 213, in dispatch
return method(statement)
File "/home/blais/p/beancount/src/python/beancount/query/shell.py", line 356, in on_Select
self.options_map)
File "/home/blais/p/beancount/src/python/beancount/query/query_execute.py", line 251, in execute_query
if c_where is None or c_where(context):
File "/home/blais/p/beancount/src/python/beancount/query/query_compile.py", line 111, in call
return self.operator(self.left(context), self.right(context))
TypeError: unsupported operand type(s) for &: 'bool' and 'Position'
beancount>
Original report by Martin Michlmayr (Bitbucket: tbm13, GitHub: tbm).
This is just an idea so feel free to close if you disagree.
Metadata tags have start with a lowercase letter. I wonder if ANY_META should ignore the case of the first letter (i.e. basically do a lcfirst on the argument).
I guess, more broadly, should metadata tags be completely case insensitive? Currently they aren't.
Actually, the more I think about it, I think doing the lcfirst as suggested doesn't make sense if the decision is to keep the case sensitivity. People used to uppercase tags (e.g. from ledger) should just get used to lowercase.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.