|
| 1 | +Python Client for Google Cloud Storage |
| 2 | +====================================== |
| 3 | + |
| 4 | +|stable| |pypi| |versions| |
| 5 | + |
| 6 | +`Google Cloud Storage`_ is a managed service for storing unstructured data. Cloud Storage |
| 7 | +allows world-wide storage and retrieval of any amount of data at any time. You can use |
| 8 | +Cloud Storage for a range of scenarios including serving website content, storing data |
| 9 | +for archival and disaster recovery, or distributing large data objects to users via direct download. |
| 10 | + |
| 11 | +**NOTE**: `3.0 Major Version Notes`_ are available. Feedback welcome. |
| 12 | + |
| 13 | +A comprehensive list of changes in each version may be found in the `CHANGELOG`_. |
| 14 | + |
| 15 | +- `Product Documentation`_ |
| 16 | +- `Client Library Documentation`_ |
| 17 | +- `github.com/googleapis/google-cloud-python`_ |
| 18 | + |
| 19 | +Certain control plane and long-running operations for Cloud Storage (including Folder |
| 20 | +and Managed Folder operations) are supported via the `Storage Control Client`_. |
| 21 | +The `Storage Control API`_ creates one space to perform metadata-specific, control plane, |
| 22 | +and long-running operations apart from the Storage API. |
| 23 | + |
| 24 | +Read more about the client libraries for Cloud APIs, including the older |
| 25 | +Google APIs Client Libraries, in `Client Libraries Explained`_. |
| 26 | + |
| 27 | +.. |stable| image:: https://img.shields.io/badge/support-stable-gold.svg |
| 28 | + :target: https://github.com/googleapis/google-cloud-python/blob/main/README.rst#stability-levels |
| 29 | +.. |pypi| image:: https://img.shields.io/pypi/v/google-cloud-storage.svg |
| 30 | + :target: https://pypi.org/project/google-cloud-storage/ |
| 31 | +.. |versions| image:: https://img.shields.io/pypi/pyversions/google-cloud-storage.svg |
| 32 | + :target: https://pypi.org/project/google-cloud-storage/ |
| 33 | +.. _Google Cloud Storage: https://cloud.google.com/storage |
| 34 | +.. _Client Library Documentation: https://cloud.google.com/python/docs/reference/storage/latest/summary_overview |
| 35 | +.. _Product Documentation: https://cloud.google.com/storage |
| 36 | +.. _CHANGELOG: https://github.com/googleapis/google-cloud-python/blob/main/CHANGELOG.md |
| 37 | +.. _github.com/googleapis/google-cloud-python: https://github.com/googleapis/google-cloud-python |
| 38 | +.. _Storage Control Client: https://cloud.google.com/python/docs/reference/google-cloud-storage-control/latest |
| 39 | +.. _Storage Control API: https://cloud.google.com/storage/docs/reference/rpc/google.storage.control.v2 |
| 40 | +.. _Client Libraries Explained: https://cloud.google.com/apis/docs/client-libraries-explained |
| 41 | + |
| 42 | + |
| 43 | +Quick Start |
| 44 | +----------- |
| 45 | + |
| 46 | +In order to use this library, you first need to go through the following steps. |
| 47 | +A step-by-step guide may also be found in `Get Started with Client Libraries`_. |
| 48 | + |
| 49 | +1. `Select or create a Cloud Platform project.`_ |
| 50 | +2. `Enable billing for your project.`_ |
| 51 | +3. `Enable the Google Cloud Storage API.`_ |
| 52 | +4. `Setup Authentication.`_ |
| 53 | + |
| 54 | +.. _Get Started with Client Libraries: https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python |
| 55 | +.. _Select or create a Cloud Platform project.: https://console.cloud.google.com/project |
| 56 | +.. _Enable billing for your project.: https://cloud.google.com/billing/docs/how-to/modify-project#enable_billing_for_a_project |
| 57 | +.. _Enable the Google Cloud Storage API.: https://console.cloud.google.com/flows/enableapi?apiid=storage-api.googleapis.com |
| 58 | +.. _Setup Authentication.: https://cloud.google.com/docs/authentication/client-libraries |
| 59 | + |
| 60 | +Installation |
| 61 | +~~~~~~~~~~~~ |
| 62 | + |
| 63 | +Install this library in a virtual environment using `venv`_. `venv`_ is a tool that |
| 64 | +creates isolated Python environments. These isolated environments can have separate |
| 65 | +versions of Python packages, which allows you to isolate one project's dependencies |
| 66 | +from the dependencies of other projects. |
| 67 | + |
| 68 | +With `venv`_, it's possible to install this library without needing system |
| 69 | +install permissions, and without clashing with the installed system |
| 70 | +dependencies. |
| 71 | + |
| 72 | +.. _`venv`: https://docs.python.org/3/library/venv.html |
| 73 | + |
| 74 | + |
| 75 | +Code samples and snippets |
| 76 | +~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 77 | + |
| 78 | +Code samples and snippets live in the `samples/`_ folder. |
| 79 | + |
| 80 | +.. _`samples/`: https://github.com/googleapis/google-cloud-python/tree/main/samples |
| 81 | + |
| 82 | + |
| 83 | +Supported Python Versions |
| 84 | +^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 85 | +Our client libraries are compatible with all current `active`_ and `maintenance`_ versions of |
| 86 | +Python. |
| 87 | + |
| 88 | +Python >= 3.7 |
| 89 | + |
| 90 | +.. _active: https://devguide.python.org/devcycle/#in-development-main-branch |
| 91 | +.. _maintenance: https://devguide.python.org/devcycle/#maintenance-branches |
| 92 | + |
| 93 | +Unsupported Python Versions |
| 94 | +^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 95 | +Python <= 3.6 |
| 96 | + |
| 97 | +If you are using an `end-of-life`_ |
| 98 | +version of Python, we recommend that you update as soon as possible to an actively supported version. |
| 99 | + |
| 100 | +.. _end-of-life: https://devguide.python.org/devcycle/#end-of-life-branches |
| 101 | + |
| 102 | +Mac/Linux |
| 103 | +^^^^^^^^^ |
| 104 | + |
| 105 | +.. code-block:: console |
| 106 | +
|
| 107 | + python3 -m venv <your-env> |
| 108 | + source <your-env>/bin/activate |
| 109 | + pip install google-cloud-storage |
| 110 | +
|
| 111 | +
|
| 112 | +Windows |
| 113 | +^^^^^^^ |
| 114 | + |
| 115 | +.. code-block:: console |
| 116 | +
|
| 117 | + py -m venv <your-env> |
| 118 | + .\<your-env>\Scripts\activate |
| 119 | + pip install google-cloud-storage |
| 120 | +
|
| 121 | +
|
| 122 | +Example Usage |
| 123 | +~~~~~~~~~~~~~ |
| 124 | + |
| 125 | +.. code-block:: python |
| 126 | +
|
| 127 | + # Imports the Google Cloud client library |
| 128 | + from google.cloud import storage |
| 129 | +
|
| 130 | + # Instantiates a client |
| 131 | + storage_client = storage.Client() |
| 132 | +
|
| 133 | + # The name for the new bucket |
| 134 | + bucket_name = "my-new-bucket" |
| 135 | +
|
| 136 | + # Creates the new bucket |
| 137 | + bucket = storage_client.create_bucket(bucket_name) |
| 138 | +
|
| 139 | + print(f"Bucket {bucket.name} created.") |
| 140 | +
|
| 141 | +
|
| 142 | +Tracing With OpenTelemetry |
| 143 | +~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 144 | + |
| 145 | +This is a PREVIEW FEATURE: Coverage and functionality are still in development and subject to change. |
| 146 | + |
| 147 | +This library can be configured to use `OpenTelemetry`_ to generate traces on calls to Google Cloud Storage. |
| 148 | +For information on the benefits and utility of tracing, read the `Cloud Trace Overview <https://cloud.google.com/trace/docs/overview>`_. |
| 149 | + |
| 150 | +To enable OpenTelemetry tracing in the Cloud Storage client, first install OpenTelemetry: |
| 151 | + |
| 152 | +.. code-block:: console |
| 153 | +
|
| 154 | + pip install google-cloud-storage[tracing] |
| 155 | +
|
| 156 | +Set the ``ENABLE_GCS_PYTHON_CLIENT_OTEL_TRACES`` environment variable to selectively opt-in tracing for the Cloud Storage client: |
| 157 | + |
| 158 | +.. code-block:: console |
| 159 | +
|
| 160 | + export ENABLE_GCS_PYTHON_CLIENT_OTEL_TRACES=True |
| 161 | +
|
| 162 | +You will also need to tell OpenTelemetry which exporter to use. An example to export traces to Google Cloud Trace can be found below. |
| 163 | + |
| 164 | +.. code-block:: console |
| 165 | +
|
| 166 | + # Install the Google Cloud Trace exporter and propagator, however you can use any exporter of your choice. |
| 167 | + pip install opentelemetry-exporter-gcp-trace opentelemetry-propagator-gcp |
| 168 | +
|
| 169 | + # [Optional] Install the OpenTelemetry Requests Instrumentation to trace the underlying HTTP requests. |
| 170 | + pip install opentelemetry-instrumentation-requests |
| 171 | +
|
| 172 | +.. code-block:: python |
| 173 | +
|
| 174 | + from opentelemetry import trace |
| 175 | + from opentelemetry.sdk.trace import TracerProvider |
| 176 | + from opentelemetry.sdk.trace.export import BatchSpanProcessor |
| 177 | + from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter |
| 178 | +
|
| 179 | + tracer_provider = TracerProvider() |
| 180 | + tracer_provider.add_span_processor(BatchSpanProcessor(CloudTraceSpanExporter())) |
| 181 | + trace.set_tracer_provider(tracer_provider) |
| 182 | +
|
| 183 | + # Optional yet recommended to instrument the requests HTTP library |
| 184 | + from opentelemetry.instrumentation.requests import RequestsInstrumentor |
| 185 | + RequestsInstrumentor().instrument(tracer_provider=tracer_provider) |
| 186 | +
|
| 187 | +In this example, tracing data will be published to the `Google Cloud Trace`_ console. |
| 188 | +Tracing is most effective when many libraries are instrumented to provide insight over the entire lifespan of a request. |
| 189 | +For a list of libraries that can be instrumented, refer to the `OpenTelemetry Registry`_. |
| 190 | + |
| 191 | +.. _OpenTelemetry: https://opentelemetry.io |
| 192 | +.. _OpenTelemetry Registry: https://opentelemetry.io/ecosystem/registry |
| 193 | +.. _Google Cloud Trace: https://cloud.google.com/trace |
| 194 | + |
| 195 | + |
| 196 | +3.0 Major Version Notes |
| 197 | +----------------------- |
| 198 | + |
| 199 | +Feedback Welcome |
| 200 | +~~~~~~~~~~~~~~~~ |
| 201 | + |
| 202 | +If you experience that backwards compatibility for your application is broken |
| 203 | +with this major version release, please let us know through the Github issues |
| 204 | +system. While some breaks of backwards compatibility may be unavoidable due to |
| 205 | +new features in the major version release, we will do our best to minimize |
| 206 | +them. Thank you. |
| 207 | + |
| 208 | +Exception Handling |
| 209 | +~~~~~~~~~~~~~~~~~~ |
| 210 | + |
| 211 | +In Python Storage 3.0, the dependency ``google-resumable-media`` was integrated. |
| 212 | +The ``google-resumable-media`` dependency included exceptions |
| 213 | +``google.resumable_media.common.InvalidResponse`` and |
| 214 | +``google.resumable_media.common.DataCorruption``, which were often imported |
| 215 | +directly in user application code. The replacements for these exceptions are |
| 216 | +``google.cloud.storage.exceptions.InvalidResponse`` and |
| 217 | +``google.cloud.storage.exceptions.DataCorruption``. Please update application code |
| 218 | +to import and use these exceptions instead. |
| 219 | + |
| 220 | +For backwards compatibility, if ``google-resumable-media`` is installed, the new |
| 221 | +exceptions will be defined as subclasses of the old exceptions, so applications |
| 222 | +should continue to work without modification. This backwards compatibility |
| 223 | +feature may be removed in a future major version update. |
| 224 | + |
| 225 | +Some users may be using the original exception classes from the |
| 226 | +``google-resumable-media`` library without explicitly installing that library. So |
| 227 | +as not to break user applications following this pattern, |
| 228 | +``google-resumable-media`` is still in the list of dependencies in this package's |
| 229 | +setup.py file. Applications which do not import directly from |
| 230 | +``google-resumable-media`` can safely disregard this dependency. |
| 231 | +This backwards compatibility feature **will be removed** in a future major |
| 232 | +version update. Please migrate to using the ``google.cloud.storage.exceptions`` |
| 233 | +classes as above. |
| 234 | + |
| 235 | +Checksum Defaults |
| 236 | +~~~~~~~~~~~~~~~~~ |
| 237 | + |
| 238 | +In Python Storage 3.0, uploads and downloads now have a default of "auto" where |
| 239 | +applicable. "Auto" will use crc32c checksums, except for unusual cases where the |
| 240 | +fast (C extension) crc32c implementation is not available, in which case it will |
| 241 | +use md5 instead. Before Python Storage 3.0, the default was md5 for most |
| 242 | +downloads and None for most uploads. Note that ranged downloads ("start" or |
| 243 | +"end" set) still do not support any checksumming, and some features in |
| 244 | +``transfer_manager.py`` still support crc32c only. |
| 245 | + |
| 246 | +Note: The method ``Blob.upload_from_file()`` requires a file in bytes mode, but |
| 247 | +when checksum is set to None, as was the previous default, would not throw an |
| 248 | +error if passed a file in string mode under some circumstances. With the new |
| 249 | +defaults, it will now raise a TypeError. Please use a file opened in bytes |
| 250 | +reading mode as required. |
| 251 | + |
| 252 | +Miscellaneous |
| 253 | +~~~~~~~~~~~~~ |
| 254 | + |
| 255 | +- The ``BlobWriter`` class now attempts to terminate an ongoing resumable upload if |
| 256 | + the writer exits with an exception. |
| 257 | +- Retry behavior is now identical between media operations (uploads and |
| 258 | + downloads) and other operations, and custom predicates are now supported for |
| 259 | + media operations as well. |
| 260 | +- ``Blob.download_as_filename()`` will now delete the empty file if it results in a |
| 261 | + google.cloud.exceptions.NotFound exception (HTTP 404). |
| 262 | +- Previously, object upload, metadata update, and delete methods had retries |
| 263 | + disabled by default unless the generation or metageneration was specified in |
| 264 | + the request. This has now changed so that retries are enabled by default. |
| 265 | + |
| 266 | + |
| 267 | +Next Steps |
| 268 | +~~~~~~~~~~ |
| 269 | + |
| 270 | +- Read the `Google Cloud Storage Product documentation`_ to learn |
| 271 | + more about the product and see How-to Guides. |
| 272 | +- Read the `Client Library Documentation`_ for Google Cloud Storage API |
| 273 | + to see other available methods on the client. |
| 274 | +- View this `README`_ to see the full list of Cloud |
| 275 | + APIs that we cover. |
| 276 | + |
| 277 | +.. _Google Cloud Storage Product documentation: https://cloud.google.com/storage |
| 278 | +.. _README: https://github.com/googleapis/google-cloud-python/blob/main/README.rst |
0 commit comments