Error on exit from mzML about psims

[mobiusklein]

I'll put this in the XMLDocumentWriter.end method to mitigate the error. It's odd that I never encountered it during the three years prior to psims being formally published, where I wrote some very big files.

I'll likely make another release within the next day, but the change is in 8e7ad24 if you can work with the development version.

[mobiusklein]

Hmm. This doesn't seem to happen on the CI server, ever, but I have seen it on one of the Windows machines I use, but with a different long as the error code. Unfortunately, I've only seen it intermittently too. Does it cause the XML file to be incomplete/malformed when it does happen?

[bretttully]

The error code changes each time, which makes me think it might be dereferencing a null pointer somewhere? I will check the mzML when I'm at my work machine.

[bretttully]

Yes, the mzML file is complete and readable.

Looks like others have seen this too: https://pastebin.com/xJcEME60
and https://bitbucket.org/openpyxl/openpyxl/issues/1046/serialisationerror-unknown-error

[bretttully]

To clarify on the above, the error code is constant for a given file, but it changes for different files. I will keep investigating.

[mobiusklein]

This problem boils out of libxml2, and it is shielded from view at the Python level because the attribute where the error is stored is only visible from C, and the C-extension class is defined @cython.internal, so it can't be accessed without rebuilding lxml from scratch.

[bretttully]

It seems to happen pretty frequently once the mzML file gets about 2 or 3 GB when running in our Docker container. Thinking the easiest way forward might be this:

try:
    with MzMLWriter(open(self._mzml_fname, 'wb')) as writer:
        # do work...
        pass
except etree.SerialisationError as e:
    self.logger.warning(f'Error closing file\n\n{e}\n')

[bretttully]

Looks like it might be a known problem for lxml for large files: https://bugs.launchpad.net/lxml/+bug/1570388

Slightly safer try/except:

try:
    with IndexedMzMLWriter(open(self.mzml_fname, 'wb') , close=True) as writer:
        pass
except etree.SerialisationError as e:
    if str(e).startswith('unknown error'):
        # There seems to be a bug when closing the mzML file, usually when they get large
        # see here: https://bugs.launchpad.net/lxml/+bug/1570388
        self.logger.debug(f'Error closing file: {e}')
    else:
        raise

[mobiusklein]

Fix is live in https://pypi.org/project/psims/0.1.27/

[bretttully]

Have tested, and throws as expected on a file that used to crash:

/Users/dev/anaconda3/envs/dev/lib/python3.7/site-packages/psims/xml.py:882: UserWarning: Error closing file: unknown error -1476144936
  warnings.warn("Error closing file: {}".format(err))

Thanks for the fix