data integrity checks for uploads/downloads #547

Closed

Assignees

Labels

api: storagetype: feature request

Milestone

opened

on Jan 14, 2015

For uploads/downloads, check in order of priority:

md5 computed for local file against md5 contained in Object’s metadata (more expensive, more reliable)
crc32c computed for local file against crc32c contained in Object’s metadata (less expensive, less reliable)

eaxmple gslib code for the checks chain

For copies of objects already in the storage, compare the checksums in the Object’s metadata with the same priority (md5 could not be available in some cases)
example gslib_code

If the checksum check fails, the created file should be deleted by the client, and an exception raised.
example gslib_code

We should allow to override the configuration for the checks (e.g. use only crc32c, do not perform checks), but it should be strongly discouraged.

silvolu
added
type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.

api: storageIssues related to the Cloud Storage API.
on Jan 14, 2015
jgeewax
added this to the Storage Future milestone on Jan 30, 2015
dhermes
added 2 commits that reference this issue on Feb 10, 2015
Moving MD5 hash method from regression to storage helpers.
f5f153b
Moving MD5 hash method from regression to storage helpers.
e062c89
dhermes
self-assigned this
on Mar 9, 2015
lukesneeringer commented on Mar 17, 2017
lukesneeringer
on Mar 17, 2017
Contributor
This seems to have been done.
lukesneeringer
closed this as completed on Mar 17, 2017
dhermes commented on Mar 17, 2017
dhermes
on Mar 17, 2017
Contributor
@lukesneeringer Nope, that was the CRC-32 PR that you closed. We also need the option of an MD5 integrity check.
dhermes
reopened this on Mar 17, 2017
lukesneeringer
added
priority: p2Moderately-important priority. Fix may not be included in next release.
on Apr 19, 2017
lukesneeringer
removed
priority: p2Moderately-important priority. Fix may not be included in next release.
on Aug 11, 2017
chemelnucfin commented on Jan 22, 2018
chemelnucfin
on Jan 22, 2018
Contributor
Feature Requests will now be tracked in our Project Feature Request. Please feel free to continue the discussions here.
chemelnucfin
closed this as completed on Jan 22, 2018
theacodes
unassigned
dhermes
on Sep 28, 2018
9 remaining items
parthea
added 2 commits that reference this issue on Oct 21, 2023
build(deps): [autoapprove] bump certifi from 2022.12.7 to 2023.7.22 (#…
2c3c5e2
chore: Update gapic-generator-python to v1.11.9 (#547)
a5b775d
parthea
added a commit that references this issue on Aug 21, 2025
feat: Add support for python 3.11 (#547)
0db07d2
parthea
added a commit that references this issue on Sep 16, 2025
feat: Add support for python 3.11 (#547)
59030a0
parthea
added 4 commits that reference this issue on Nov 22, 2025
docs: Update `cpu_utilization_percent` limit (#547)
beec8ec
chore: release 0.29.0 (#547)
8ad8f62
chore: Remove py.typed file (#547)
6119127
Add new "quickstart" samples [(#547)](GoogleCloudPlatform/python-docs…
7ad0352
parthea
added 6 commits that reference this issue on Mar 2, 2026
samples: add samples from pubsub/cloud-client (#134)
8c90f22
Add new "quickstart" samples [(#547)](GoogleCloudPlatform/python-docs…
f4d3af7
chore: update python-docs-samples link to main branch (#547)
70aa409
chore: release 2.12.0 (#547)
10a16cb
samples: add cloud client samples from python-docs-samples (#626)
2b99c9d
chore: migrate to main branch (#549)
249080c
parthea
added a commit that references this issue on Sep 8, 2026
docs: migrate code from googleapis/python-bigquery (#18303)
23f9a74

to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

lukesneeringer

Labels

api: storagetype: feature request

Type

No type

Projects

No projects

Milestone

Storage Future

Relationships

None yet

Development

No branches or pull requests