tfrancoi / odoo_csv_import Goto Github PK

For a broader and cleaner external use, import_threaded should take a file handler for the failing files instead of a string.
What do you think?

Usually customers give us data with these fields:

CODE;NAME;PHONE;INVOICING_CONTACT;INVOICING_PHONE;...

These should be imported as a main partner and an invoicing contact partner with main partner as parent_id.

I find I can do this writing res.partner.csv with one mapping and res.partner.cs2 with a second mapping. res.partner.2.csv or res.partner.csv2 wont do.

Is mine (res.partner.cs2) the proper way to write more res.partner file?

Based on your work and ideas I went down a different path using an established ETL solution, however, I wanted to feed back my EAFP "toposort" solution based on locking...

On any convenient WriterClass:
https://gist.github.com/blaggacao/b032459295d4d669af10aadcdfef98c2

You had an issue with arguments (*args, **kwargs), which sufraced during the debugging, here is my version:

import threading


class RpcThread(object):

    def __init__(self, max_connection):
        self.semaphore = threading.BoundedSemaphore(max_connection)
        self.max_thread_semaphore = threading.BoundedSemaphore(max_connection * 4)
        self.thread_list = []

    def spawn_thread(self, fun, *args, **kwargs):
        def run(*args, **kwargs):
            self.semaphore.acquire()
            try:
                fun(*args, **kwargs)
            except Exception:
                raise
            finally:
                self.semaphore.release()
                self.max_thread_semaphore.release()
        
        # Tight Loop Interprocess communication starts here
        self.max_thread_semaphore.acquire()
        thread = threading.Thread(None, run, None, args=args, kwargs=kwargs)
        thread.start()
        self.thread_list.append(thread)

    def wait(self):
        for t in self.thread_list:
            t.join()

    def thread_number(self):
        return len(self.thread_list)

Hello, I've been using odoo_csv_import successfully with a v14 database.

One issue I'm haivng is how to set the xml_id on a materialized relationship - product_template_attribute_value.

This is a new object since v13 and is created by the product_template_attribute_line (also a new object) and so I am not seeing how to set this xml_id so that I can reference it later.

Maybe I am missing something obvious?

Thanks.

Working with a config file might serve a general git based workflow, but this is not very scripting friendly and therefore not very library-ish.
It would be great, if this could be complemented by ENV interpolation or plain arguments...

I try to import invoices to odoo
by using this command
odoo_import_thread.py --o2m -c ./connection.conf --file=in.csv --model='account.invoice'

and I get this error

{'rows': {'from': 0, 'to': 1}, 'type': 'warning', 'record': 0, 'field': 'invoice_line_ids', 'message': "Found multiple matches for field 'Invoice Lines' (2 matches)"}
['inv_o2m.test10', '1', 'Customer', 'Pro - A', '10', '20', '0']
batch [0], 0
{'rows': {'from': 0, 'to': 1}, 'record': 0, 'type': 'error', 'message': 'null value in column "name" violates not-null constraint\nDETAIL:  Failing row contains (46, null, null, 10, 51, null, 1, null, 20.00, null, null, null, 10.000, 0.00, null, null, null, null, f, 1, 2021-04-07 15:53:12.852083, 1, 2021-04-07 15:53:12.852083, 0, null, null).\n'}
['inv_o2m.test10', '1', 'Customer', 'Pro - A', '10', '20', '0']

my csv file

id;number;partner_id;invoice_line_ids/product_id;invoice_line_ids/quantity;invoice_line_ids/price_unit;invoice_line_ids/value_difference
inv_o2m.test10;1;Customer;Pro - A;10;20;0
;;;Pro – B;5;10;0

I'm having trouble importing the module from a python script

from odoo_csv_tools import import_threaded

results in
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/mladenyakimov/.virtualenvs/odoo/lib/python3.6/site-packages/odoo_csv_tools/__init__.py", line 1, in <module> import lib ModuleNotFoundError: No module named 'lib'

Hello.

I was trying to import journal items, but the journal entry needs the credit and debit totals to be balanced. So it gives me the following error:
{'rows': {'from': 2, 'to': 2}, 'record': 2, 'type': 'error', 'message': "Unknown error during import: <class 'odoo.exceptions.UserError'>: ('Cannot create unbalanced journal entry.', '')", 'moreinfo': 'Resolve other errors first'}

Any known solution for this problem?

Thanks.
João Jerónimo

How can we get the country id,state_id,parent_id which is already exist in the DB?

Hi

I am importing 100.000+ lines of records.

The time it takes to import the 10 first lines is 2 seconds.
The time it takes to import the las 10 lines is 5x longer.

Can anyone advice ? Is that normal ? If after finishing, I run on the same database the import, the first 10 records will take 2 seconds again, this does not seems to be a db-insert issue.

I used this tool to import data, but when I run odoo -u addon_name, it suddenly starts erasing everything I imported...?

Odoo can import a csv file with an image column containing an URL. This file cannot be imported with odoo_import_thread.py because it expects a base64.

Hello Thibault,

first of all: this is really a nice script!

Now my question:
Is it possible to add a requirements.txt file to easily create a python virtual environment?

Thank you.

Kind regards
Jörg Federspiel

I have tried several models that contain product inventory and have been unsuccessful at changing quantity on hand.
Any suggestions?

Hi,
when trying differents export instead of getting actual id I get a string representation

./env/bin/odoo_export_thread.py -c=config.ini --model=res.partner --worker=4 --field="id,name,phone,website,street,city,country_id" --file=out.cs
gives

...
"__export__.res_partner_172_7a1a6fdb";"FirstName LastName";"";"";"";"Montigny les cormeilles";"France"
...

• id gives __export__.res_partner_172_7a1a6fdb
• country_id gives France

in both case I expected just integers .

How can I get integers ? not an arbitrary string representation ?

Hi

Thanks for this great import tool.
I successfully imported over 7000 products in under 5 minutes. That was amazing.

Now I am trying to import a large inventory adjustment (about 4000 rows) using multiple workers and the python script keeps on timing out.

After reading your documentation very carefully I spotted the groupby argument and how it is supposed to address the concurrent update issue (if that is the problem indeed).

The command I am using:
python odoo_import_thread.py -c connection.conf --file=my_data.csv --model=stock.inventory --worker=8 --size=250

I have tried to add the groupby argument with line_ids/product_id and the script still crashed.

Questions:

1. In that specific instance (importing inventory adjustments) is the groupby argument really necessary?
2. If not, why do you think my script is timing out?

Thank you in advance for your kind help.
Regards,
Raf from New Zealand

stock.inventory.zip

Hi Francois,
such would go along the lines:

• Python Macros for what's the interaction with the rpc api: https://doc.dataiku.com/dss/latest/plugins/reference/index.html
• Python Recipts for your odoo specific mapping transformations? https://doc.dataiku.com/dss/latest/code_recipes/python.html
• Managed Folders to preserve your file based data handling and split up, error/failed: https://doc.dataiku.com/dss/latest/advanced/managed_folders.html

Interested in joining efforts?

First question:

• What are the really Odoo specific mapping parts?

How can I export all supported languages for a given field?

This is too custom to formalize (yet), but I want to share.

https://gist.github.com/blaggacao/ba83a7e500c202cf842010cfddb25e15

In an ideal world, we could build a component based library for Odoo imports, leveraging excellent acsone's click-odoo.

My main use case: I want to run Odoo-Loaders as microservices consuming datastreams and giving feedback on another channel to remote clients*, so a component based approach would just be ideal for greater convergence and collaboration.
See also discussion here: acsone/click-odoo-contrib#11 (comment)

* Why? Integrate with ETL tools.

Does it have an issue with https? I always got the following error when trying simple export from the remote site using https. Yet, it works fine when I use my local odoo installation:

Traceback (most recent call last):
  File "/home/navds/.local/bin/odoo_export_thread.py", line 52, in <module>
    encoding=encoding)
  File "/home/navds/.local/lib/python3.6/site-packages/odoo_csv_tools/export_threaded.py", line 85, in export_data
    ids = object_registry.search(domain, context=context)
  File "/home/navds/.local/lib/python3.6/site-packages/odoolib/main.py", line 317, in proxy
    args, kw)
  File "/home/navds/.local/lib/python3.6/site-packages/odoolib/main.py", line 184, in proxy
    result = self.connector.send(self.service_name, method, *args)
  File "/home/navds/.local/lib/python3.6/site-packages/odoolib/main.py", line 140, in send
    return json_rpc(self.url, "call", {"service": service_name, "method": method, "args": args})
  File "/home/navds/.local/lib/python3.6/site-packages/odoolib/main.py", line 118, in json_rpc
    result = result_req.json()
  File "/home/navds/.local/lib/python3.6/site-packages/requests/models.py", line 897, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3/dist-packages/simplejson/__init__.py", line 518, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The command I use to export:

$ odoo_export_thread.py -c odoo.conf --model "product.template" --field "id,default_code,name" -s ";" --file export/product.template.csv

Hi

When I try to export res_currency with the tool, this only export the records with the field Active = True

How Can I to define --domain = ['|','active','=',False,'active','=',True]

Thank's a lot

Hi,

When I import the field state_id in res.partner, I do it this way (like any other field):

res_partner_mapping = {
    (........ more fields here)
    'country_id/id' : mapper.const('base.es'),
    'state_id': mapper.map_val('myStateField', my_state_dictionary),
}

But the field state_id is related to country_id, so if my dictionary returns "AV", there is an"AV" state in Spain, and another "AV" state in Italy, so it throws this error: "Found multiple matches for field 'State' (2 matches)"

I would need something like this:
state = fields.Many2one('country',related=city.country)
related=city.country is the important point.

What is the right way to map a related field like state_id?

Thank you

So far I'm doing it with xml, but this tool would be much better!! Thanks for the great work!!

I am trying to importing product data to odoo 12 with odoo_csv_import but constantly getting the following error. I do have the id column in the csv file.
odoo$ sudo odoo_import_thread.py -c connection.conf --file=product.template.csv --model=product.template open product.template.csv open product.template.csv No External Id (id) column defined, please add one Traceback (most recent call last): File "/usr/local/bin/odoo_import_thread.py", line 62, in <module> max_connection=max_connection, batch_size=batch_size, skip=int(args.skip), o2m=args.o2m) File "/usr/local/lib/python3.6/dist-packages/odoo_csv_tools/import_threaded.py", line 193, in import_data header, data = read_file(file_csv, delimiter=separator, encoding=encoding, skip=skip) File "/usr/local/lib/python3.6/dist-packages/odoo_csv_tools/import_threaded.py", line 143, in read_file check_id_column(header) File "/usr/local/lib/python3.6/dist-packages/odoo_csv_tools/import_threaded.py", line 131, in check_id_column raise ve File "/usr/local/lib/python3.6/dist-packages/odoo_csv_tools/import_threaded.py", line 128, in check_id_column header.index('id') ValueError: 'id' is not in list