cfv-project / cfv Goto Github PK
View Code? Open in Web Editor NEWCommand-line File Verify
License: GNU General Public License v2.0
Command-line File Verify
License: GNU General Public License v2.0
Some deprecated modules and functions are used. We have to investigate those and move them to a proper supported module/function.
Python provides the -W error
argument to ease finding such issues. For example:
$ python3 -W error test/test.py
Traceback (most recent call last):
File "test/test.py", line 48, in <module>
import cfvtest
File "/[...]/cfv/test/cfvtest.py", line 26, in <module>
import imp
File "/usr/lib/python3.6/imp.py", line 33, in <module>
DeprecationWarning, stacklevel=2)
DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
Once fixed, we should also add these Python flags in our CI pipeline.
Update the Debian packaging to fix outstanding issues that caused the package to be removed from Debian Buster (https://tracker.debian.org/pkg/cfv).
The most severe issues are already resolved.
However, it would be nice to be able to run unit tests, short CLI tests and long CLI tests separately.
Currently the CI pipeline only runs during push.
Therefore, during periods of low activity, we do not notice, if something is breaking.
The CI pipeline should run periodically (e.g. once a week) for the python3 branch.
Tests fail when treating all Python warnings as errors.
>>> testing...
.....................
>>> failed test: /home/runner/work/cfv/cfv/test/cfv -ZNVRMUI --unquote=no --fixpaths="" --strippaths=0 --showpaths=auto-relative --progress=no --announceurl=url --noprivate_torrent -r -p /tmp/tmpmjh48zao --showpaths=0 -q -T --list0=bad
Exception ignored in: <_io.FileIO name='d2.sha512' mode='rb' closefd=True>
Traceback (most recent call last):
File "/home/runner/work/cfv/cfv/lib/cfv/common.py", line 1349, in <module>
class SFV(SFV_Base, CRC_MixIn):
ResourceWarning: unclosed file <_io.FileIO name='d2.sha512' mode='rb' closefd=True>
Exception ignored in: <_io.FileIO name='tmpmjh48zao.sha512' mode='rb' closefd=True>
Traceback (most recent call last):
File "/home/runner/work/cfv/cfv/lib/cfv/common.py", line 1349, in <module>
class SFV(SFV_Base, CRC_MixIn):
ResourceWarning: unclosed file <_io.FileIO name='tmpmjh48zao.sha512' mode='rb' closefd=True>
Exception ignored in: <_io.FileIO name='test4.foo' mode='rb' closefd=True>
Traceback (most recent call last):
File "/home/runner/work/cfv/cfv/lib/cfv/hash.py", line 77, in <lambda>
return lambda filename, callback: _getfilechecksum(filename, hasher, callback), hasher().digest_size
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ResourceWarning: unclosed file <_io.BufferedReader name='test4.foo'>
test4.fooException ignored in: <_io.FileIO name='test.ext.end' mode='rb' closefd=True>
Traceback (most recent call last):
File "/home/runner/work/cfv/cfv/lib/cfv/hash.py", line 77, in <lambda>
return lambda filename, callback: _getfilechecksum(filename, hasher, callback), hasher().digest_size
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ResourceWarning: unclosed file <_io.BufferedReader name='test.ext.end'>
test.ext.endException ignored in: <_io.FileIO name='test2.foo' mode='rb' closefd=True>
Traceback (most recent call last):
File "/home/runner/work/cfv/cfv/lib/cfv/hash.py", line 77, in <lambda>
return lambda filename, callback: _getfilechecksum(filename, hasher, callback), hasher().digest_size
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ResourceWarning: unclosed file <_io.BufferedReader name='test2.foo'>
test2.fooException ignored in: <_io.FileIO name='test3' mode='rb' closefd=True>
Traceback (most recent call last):
File "/home/runner/work/cfv/cfv/lib/cfv/hash.py", line 77, in <lambda>
return lambda filename, callback: _getfilechecksum(filename, hasher, callback), hasher().digest_size
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ResourceWarning: unclosed file <_io.BufferedReader name='test3'>
test3
FAILED (2)
File "/home/runner/work/cfv/cfv/test/test.py", line 1884, in <module>
failed += all_tests()
File "/home/runner/work/cfv/cfv/test/test.py", line 1622, in all_tests
ren_test(fmt)
File "/home/runner/work/cfv/cfv/test/test.py", line 905, in ren_test
test_generic(basecmd + ' --showpaths=0 -q -T --list0=bad', cfv_listdata_bad_test)
File "/home/runner/work/cfv/cfv/test/test.py", line 269, in test_generic
test_log_results(cfvtest.cfvenv + cfvtest.cfvfn + ' ' + cmd, s, o, r, kw)
File "/home/runner/work/cfv/cfv/test/test.py", line 254, in test_log_results
test_log_finish(cmd, s, r, o, kw)
File "/home/runner/work/cfv/cfv/test/test.py", line 237, in test_log_finish
traceback_str = '\n'.join(traceback.format_stack())
Exception ignored in: <_io.FileIO name='d2.sha512' mode='rb' closefd=True>
ResourceWarning: unclosed file <_io.FileIO name='d2.sha512' mode='rb' closefd=True>
Exception ignored in: <_io.FileIO name='tmpmjh48zao.sha512' mode='rb' closefd=True>
ResourceWarning: unclosed file <_io.FileIO name='tmpmjh48zao.sha512' mode='rb' closefd=True>
Exception ignored in: <_io.FileIO name='/tmp/cfv_3.0.1.dev0_test-20240518T103311.log' mode='wb' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/cfv_3.0.1.dev0_test-20240518T103311.log' mode='wt' encoding='UTF-8'>
Future releases should be available via the Python Package Index (https://pypi.org/).
Re-format the source code for better readability.
I would suggest to use the PEP-8 style guide (https://www.python.org/dev/peps/pep-0008/).
We could use flake8 as checking tool (https://pypi.org/project/flake8/).
We also want to support Windows and macOS. Therefore, it would be useful to have Windows and macOS runners in this CI pipeline: https://github.com/cfv-project/cfv/blob/python3/.github/workflows/ci-python3.yml
Instead, it should be renamed to 'termios' (like two lines above):
Line 13 in 1debfeb
Traceback (most recent call last):
*** testing all_unittests_suite
File "/home/runner/work/cfv/cfv/test/test.py", line 1606, in <module>
all_unittest_tests()
File "/home/runner/work/cfv/cfv/test/test.py", line 1542, in all_unittest_tests
suite = cfvtest.all_unittests_suite()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/cfv/cfv/test/cfvtest.py", line 245, in all_unittests_suite
alltests.addTest(unittest.findTestCases(module))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/unittest/loader.py", line 489, in findTestCases
warnings.warn(
DeprecationWarning: unittest.findTestCases() is deprecated and will be removed in Python 3.13. Please use unittest.TestLoader.loadTestsFromModule() instead.
Currently, there are a few issues, why cfv including the test suite does not run on macOS. We have to investigate the issues and fix them.
I consider shifting back macOS support to a later release. If anyone wants to use cfv on macOS, please speak up.
Python 2 will be end-of-life in 2020 (https://www.python.org/dev/peps/pep-0373/#update), so we have to add support for Python 3.
Thanks for Python3 support! Quickly made an AUR package here: https://aur.archlinux.org/packages/cfv-git
cfv seems to ship manpages to /usr/man/man1
, following a really old standard from 1995, which has already been superseded in 1997. The current standard suggests to keep man pages in /usr/share/man
. cfv should follow this standard.
In the CI pipeline the maximum number of open file descriptors is clipped to 4096 for the manyfiles_test
:
max open files is big (65536) clipping to 4096. Use --long to try the real value
Therefore, the manyfiles_test
cannot really detect issues with leaking file descriptors. We should reduce the maximum number of open file descriptors limit to <=4096 before running the tests in the CI pipeline. We can do this with ulimit -n 4096
.
HI,
running tests, I noticed this warning:
cfv/test/cfvtest.py:26: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
import imp
I didn't file a PR as I'm not exactly sure what
Line 205 in 08ee267
Hello,
Big thanks for reviving the cfv project! I just installed it and it works great Have you considered adding Blake 3 support? It is a very fast hash algo.
My other feature request is an option to update the hash only if the file modify time is later than then checksum file.
Cheers,
J
Similar to #16 the flake errors should also be fixed in the python3 branch. Furthermore, the flake test should also be marked fatal.
Flake8 is currently running in non-fatal mode in our CI because of some issues.
We should fix the issues and make flake8 fail our pipeline again.
$ flake8 --exclude=build,venv --ignore= --max-line-length=200 --max-complexity=75 --show-source --statistics .
shell: /usr/bin/bash -e {0}
env:
pythonLocation: /opt/hostedtoolcache/Python/2.7.18/x64
./lib/cfv/osutil.py:112:78: E741 ambiguous variable name 'I'
def strippath(filename, num='a', _splitdrivere=re.compile(r'[a-z]:[/\\]', re.I)):
^
./lib/cfv/common.py:738:34: E741 ambiguous variable name 'l'
def do_test_chksumline(self, l):
^
./lib/cfv/common.py:900:34: E741 ambiguous variable name 'l'
def do_test_chksumline(self, l):
^
./lib/cfv/common.py:1400:34: E741 ambiguous variable name 'l'
def do_test_chksumline(self, l):
^
./lib/cfv/common.py:1505:34: E741 ambiguous variable name 'l'
def do_test_chksumline(self, l):
^
./lib/cfv/common.py:1537:34: E741 ambiguous variable name 'l'
def do_test_chksumline(self, l):
^
./lib/cfv/common.py:1572:34: E741 ambiguous variable name 'l'
def do_test_chksumline(self, l):
^
./lib/cfv/common.py:1649:34: E741 ambiguous variable name 'l'
def do_test_chksumline(self, l):
^
./lib/cfv/strutil.py:6:14: E741 ambiguous variable name 'l'
def safesort(l):
^
./test/cfvtest.py:49:26: E741 ambiguous variable name 'l'
def writelines(self, l):
^
10 E741 ambiguous variable name 'l'
Error: Process completed with exit code 1.
if hashing recurive directories in windows there is no contect argument to not follow symbolic and hard linkss.
Hello,
I haven't been able to run the Python 2.7 based cfv for years on a stock Ubuntu due to some dependencies. As a long-time user of (Python-based) cfv
, I decided to write a Bash script to do roughly the same thing and with less dependencies to "often" (once a decade) changing libs, modules. Just in case someone is interested.
Bash bcfv
"Tested and works on Linux"
As reported by @Terry-Kennedy, a test case fails, when /data1
exists: #8 (comment)
>>> failed test: /sysprog/terry/cfv-python3/test/cfv -ZNVRMUI --unquote=no --fixpaths="" --strippaths=0 --showpaths=autorelative --progress=no --announceurl=url --noprivate_torrent --strippaths=none -T -f teststrip-none.csv4
/data1 : file size does not match (13!=33)
teststrip-none.csv4: 1 files, 0 OK, 1 badsize. 0.000 seconds, 0.0K/s
FAILED (4)
File "test/test.py", line 1876, in <module>
File "test/test.py", line 1719, in all_tests
File "test/test.py", line 269, in test_generic
File "test/test.py", line 254, in test_log_results
File "test/test.py", line 237, in test_log_finish
It is bad practice that the test suite makes assumptions about things outside it's own scope (i.e. temp directory). However, the test case cannot easily be adjusted without modifying it's intention. Therefore, we just change the expected test outcome, depending on if the /data1
exists or not. Leaving only an unexpected error, if someone has a file /data1
with exactly the same content as https://github.com/cfv-project/cfv/blob/python3/test/testdata/data1.
Currently, there are a few issues, why cfv including the test suite does not run on Windows. We have to investigate the issues and fix them.
Hello,
I noticed cfv does not use type hints. Type hints and type checkers such as mypy are great aids for making robust code. Strongly recommended.
Cheers,
J
The test suite can verify created files using external tools like cksfv and *sum. However, if they are not installed the verification is just silently skipped. We should at least issue a warning, if the tools are not installed.
Implemented in GNU coreutils with this commit: coreutils/coreutils@ea94589
There still seem to be a lot of uses of str()
to convert from bytes
to str
. We have to investigate those and move them to a proper decoding using something like .decode(correct_encoding)
.
Python provides the -bb
argument to ease finding such issues. For example:
$ python3 -bb test/test.py --exit-early
>>> testing...
...........................Traceback (most recent call last):
File "test/test.py", line 1874, in <module>
failed += all_tests()
File "test/test.py", line 1620, in all_tests
ren_test(fmt)
File "test/test.py", line 913, in ren_test
flscmp(b'1', 0, flsf)
File "test/test.py", line 897, in flscmp
test_log_results('cmp %s for %s' % (fn, t), r, o, r, None)
BytesWarning: str() on a bytes instance
Once fixed, we should also add these Python flags in our CI pipeline.
@Terry-Kennedy tested cfv on FreeBSD and created a FreeBSD port (#8 (comment)). However, currently we do not have a CI runner for FreeBSD. To ensure support in the long run, we should have our CI pipeline also regularly run on FreeBSD.
Sadly, Github Actions does not directly support FreeBSD. One possible workaround would be using this action running a FreeBSD VM on the macOS runner: https://github.com/marketplace/actions/freebsd-vm
Running the code style checker flake
as expected by the CI script currently causes some errors:
$ flake8 --exclude=build,venv --ignore= --max-line-length=200 --max-complexity=75 --show-source --statistics .
./lib/cfv/common.py:738:34: E741 ambiguous variable name 'l'
def do_test_chksumline(self, l):
^
./lib/cfv/common.py:900:34: E741 ambiguous variable name 'l'
def do_test_chksumline(self, l):
^
./lib/cfv/common.py:1400:34: E741 ambiguous variable name 'l'
def do_test_chksumline(self, l):
^
./lib/cfv/common.py:1505:34: E741 ambiguous variable name 'l'
def do_test_chksumline(self, l):
^
./lib/cfv/common.py:1537:34: E741 ambiguous variable name 'l'
def do_test_chksumline(self, l):
^
./lib/cfv/common.py:1572:34: E741 ambiguous variable name 'l'
def do_test_chksumline(self, l):
^
./lib/cfv/common.py:1649:34: E741 ambiguous variable name 'l'
def do_test_chksumline(self, l):
^
./lib/cfv/osutil.py:112:78: E741 ambiguous variable name 'I'
def strippath(filename, num='a', _splitdrivere=re.compile(r'[a-z]:[/\\]', re.I)):
^
./lib/cfv/strutil.py:6:14: E741 ambiguous variable name 'l'
def safesort(l):
^
./test/cfvtest.py:49:26: E741 ambiguous variable name 'l'
def writelines(self, l):
^
10 E741 ambiguous variable name 'l'
The README should contain updated install instructions and requirements.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.