Train lf accuracy higher than execution accuracy about sqlova

[whwang299]

Hi @vincentlux

This might happen due to instability of DBEngine class which activates except in following code.

sqlova/sqlova/utils/utils_wikisql.py

Lines 1646 to 1670 in 7a8636d

	def get_cnt_x_list(engine, tb, g_sc, g_sa, g_sql_i, pr_sc, pr_sa, pr_sql_i):
	cnt_x1_list = []
	g_ans = []
	pr_ans = []
	for b in range(len(g_sc)):
	g_ans1 = engine.execute(tb[b]['id'], g_sc[b], g_sa[b], g_sql_i[b]['conds'])
	# print(f'cnt: {cnt}')
	# print(f"pr_sql_i: {pr_sql_i[b]['conds']}")
	try:
	pr_ans1 = engine.execute(tb[b]['id'], pr_sc[b], pr_sa[b], pr_sql_i[b]['conds'])
	if bool(pr_ans1): # not empty due to lack of the data from incorretly generated sql
	if g_ans1 == pr_ans1:
	cnt_x1 = 1
	else:
	cnt_x1 = 0
	else:
	cnt_x1 = 0
	except:
	# type error etc... Execution-guided decoding may be used here.
	pr_ans1 = None
	cnt_x1 = 0
	cnt_x1_list.append(cnt_x1)
	g_ans.append(g_ans1)
	pr_ans.append(pr_ans1)

DBEngine class is modified from SQLNet code which is based on original WikiSQL evaluation code (I guess).

I'll check whether this still happens when DBEngine from WikiSQL evaluation code is employed.

Thanks!

Wonseok

[vincentlux]

Hi Wonseok,

Thanks for the quick response! This is interesting and I will also check with using the option you suggested. Thanks!

[whwang299]

@vincentlux I found the problem. It is caused by the wrong ordering of predicted where-column.

For example, when the ground truth is
{'sel': 1, 'conds': [[6, 0, 'at&t center 18,797'], [0, 2, 57]], 'agg': 0}
the previous prediction in train function could be
{'agg': 0, 'sel': 1, 'conds': [[0, 0, 'at&t center 18,797'], [6, 2, '57']]}

Hence, the execution accuracy of the model over train-set returned from the train function has been under-estimated.
During the test, there is no problem as this issue is caused by the use of ground-truth during the training which makes the debugging difficult.

I have updated the code. Please let me know if the problem still happens.

Wonseok