yes, but you missed my point. If you want something to long term store and compare the file too, yes probably use something better than crc64.
However if you want to make a fast checksum to fingerprint a file for comparison etc etc crc64 is very fast, easy to store and use, and good enough. It adds no time to the read (fastest source I have is about 12Gbgytes/s, which it handles without issue in a single thread) so is light on cpu usage compared to many others…

BTRFS does as well

It’s crazy that ZFS is still the only filesystem that checks that files are intact. EXT4 doesn’t. NTFS doesn’t. Everything just assumes that everything the hard drive returns is correct. A basic run of badblocks on drives more than a few years old shows that’s not the case.

Two comments:
1. Koopman’s paper was focused basically on brute-forcing the CRCs, which at the time was a huge effort and it’s understandable why they hadn’t approached CRC64 for practical limitations. I guess nowadays we are free to contribute to this area, with so much available computing resources, right?

For filesystem data integrity checks it would be much more reliable to use a cryptographic-grade hash functions instead of CRC64. This becomes especially important for large files (where file’s bit-size exceeds Hamming weights even for HD=1, in the case of using CRC). ZFS seems to use fletcher4 by default for this purpose, but it also offers sha256 and sha512 for crazier deployments.

For filesystem data integrity checks it would be much more reliable to use a cryptographic-grade hash functions instead of CRC64. This becomes especially important for large files (where file’s bit-size exceeds Hamming weights even for HD=1, in the case of using CRC). ZFS seems to use fletcher4 by default for this purpose, but it also offers sha256 and sha512 for crazier deployments.

Well, I’m afraid you’re right. Selecting an effective CRC that matches the project requirements is far from trivial. Koopman’s paper has really great info on the topic. There are several other nice papers for reading:
– The Effectiveness of Checksums for Embedded Networks – Theresa C.Maxino, 2006, CMU
– Efficient High Hamming Distance CRCs for Embedded Networks – Justin Ray, Philip Koopman, 2006, CMU
– Selection of Cyclic Redundancy Code and Checksum Algorithms to Ensure Critical Data Integrity – US FAA, 2015

ie if you want to crypto verify a file hasn’t been changed, don’t use a crc.

However, if you want a very fast check to see if a file has randomly changed, a CRC will often be the way to go..

And you can do a CRC64 on modern hardware faster than you can do IO…

ie I recently was cleaning up quite a few files – about 100 million – and was looking for duplicate directories. I made crc64 for every file (requiring every file to be read once) then cascaded them up for a CRC64 for each directory recursively, then looked for the highest directories that were duplicate…

This was much faster than using a md5 hash or something, as CRC64 can be done way faster than the 10Gbytes/sec (or so) reading any of my ssds..

I’ve done this before, and I’ve never found two files that are the same size with the same crc64 that weren’t identical.. Yes, you might be able to make them, but it has never happened to me in the wild..

ps if you want to do crc64 very fast try this library – https://github.com/awesomized/crc-fast-rust – it has no trouble doing crc64 at over 60GBytes/sec on my machine, and the combine etc functions work..