-
Notifications
You must be signed in to change notification settings - Fork 64
blobstore
The Blobstore API is not supported for this runtime.
Note: You should consider using Cloud Storage rather than Blobstore for storing blob data.
The Blobstore API allows your application to serve data objects, called blobs, that are much larger than the size allowed for objects in the Cloud Datastore service. Blobs are useful for serving large files, such as video or image files, and for allowing users to upload large data files. Blobs are created by uploading a file through an HTTP request. Typically, your applications will do this by presenting a form with a file upload field to the user. When the form is submitted, the Blobstore creates a blob from the file's contents and returns an opaque reference to the blob, called a blob key, which you can later use to serve the blob. The application can serve the complete blob value in response to a user request, or it can read the value directly using a streaming file-like interface.
App Engine includes the Blobstore service, which allows applications to serve
data objects limited only by the amount of data that can be uploaded or
downloaded over a single HTTP connection. These objects are called Blobstore
values, or blobs. Blobstore values are served as responses from request
handlers and are created as uploads via web forms. Applications do not create
blob data directly; instead, blobs are created indirectly, by a submitted web
form or other HTTP POST request. Blobstore values can be served to the user,
or accessed by the application in a file-like stream, using the Blobstore API.
Note: Blobs as defined by the Blobstore service are not related to blob property values used by the datastore.
To prompt a user to upload a Blobstore value, your application presents a web form with a file upload field. The application generates the form's action URL by calling the Blobstore API. The user's browser uploads the file directly to the Blobstore via the generated URL. Blobstore then stores the blob, rewrites the request to contain the blob key, and passes it to a path in your application. A request handler at that path in your application can perform additional form processing.
To serve a blob, your application sets a header on the outgoing response, and App Engine replaces the response with the blob value.
Blobs can't be modified after they're created, though they can be deleted. Each blob has a corresponding blob info record, stored in the datastore, that provides details about the blob, such as its creation time and content type. You can use the blob key to fetch blob info records and query their properties.
An application can read a Blobstore value a portion at a time using an API call. The size of the portion can be up to the maximum size of an API return value. This size is a little less than 32 megabytes, represented
in Java by the constant
com.google.appengine.api.blobstore.BlobstoreService.MAX_BLOB_FETCH_SIZE
in Python by the constant
google.appengine.ext.blobstore.MAX_BLOB_FETCH_SIZE
. An application cannot create or modify Blobstore values except through files
uploaded by the user.
Applications can use the Blobstore to accept large files as uploads from users
and to serve those files. Files are called blobs once they're uploaded.
Applications don't access blobs directly by filename. Instead, applications
refer to blobs through the
appengine.BlobKey
type., instead, applications work with blobs through blob info entities
(represented by the
BlobInfo
BlobInfo
class) in the Cloud Datastore.
The user creates a blob by submitting an HTML form that includes one or more file input fields. Your application
sets
blobstoreService.createUploadUrl()
blobstore.create_upload_url()
ascalls
blobstoreService.createUploadUrl()
blobstore.create_upload_url()
to get the destination (action) of this form, passing the function a URL path of
a handler in your application. When the user submits the form, the user's
browser uploads the specified files directly to the Blobstore. The Blobstore
rewrites the user's request and stores the uploaded file data, replacing the
uploaded file data with one or more corresponding blob keys, then passes the
rewritten request to the handler at the URL path you provided to
blobstoreService.createUploadUrl()
blobstore.create_upload_url()
.
This handler can do additional processing based on the blob key. Finally, the handler must return a headers-only, redirect response (301, 302, or 303), typically a browser redirect to another page indicating the status of the blob upload.
The application can read portions of a Blobstore value using a
file-like streaming interface. See the
BlobstoreInputStream
class.
file-like streaming interface. See the
BlobReader
class.
To create and upload a blob, follow this procedure:
Call
blobstoreService.createUploadUrl()
blobstore.create_upload_url()
to create an upload URL for the form that the user will fill out, passing the
application path to load when the POST of the form is completed.
Note that this is how the upload form would look if it were created as a JSP.
There is an asynchronous version,
create_upload_url_async().
It allows your application code to continue running while Blobstore generates
the upload URL.
The form must include a file upload field, and the form's enctype must be set
to multipart/form-data. When the user submits the form, the POST is handled
by the Blobstore API, which creates the blob. The API also creates an info
record for the blob and stores the record in the Cloud Datastore, and passes the
rewritten request to your application on the given path as a blob key.
You must serve the form's page with a Content-Type of text/html; charset=utf-8, or any filenames with non-ASCII characters will be
misinterpreted.
This is the default content type in Go, but you must
remember to do this if you set your own Content-Type. Since Blobstore for
Python 3 doesn't use webapp, you must set your own Content-Type to prevent
your web framework from setting a default content-type and App Engine from
setting this to a guessed type.
You can't use a {{global_ext_app_lb}} with a Serverless
NEG
to handle upload requests sent to the /_ah/upload/ URL returned from the
blobstore.create_upload_url call. Instead, you must route those upload
requests directly to the App Engine service. You can do this by using either the
appspot.com domain or a custom domain that is mapped directly to the App
Engine service.
In this handler, you can store the blob key with the rest of your application's data model. The blob key itself remains accessible from the blob info entity in the Cloud Datastore. Note that after the user submits the form and your handler is called, the blob has already been saved and the blob info added to the Cloud Datastore. If your application doesn't want to keep the blob, you should delete the blob immediately to prevent it from becoming orphaned.
In the following code, getUploads returns a set of blobs that have been
uploaded. The Map object is a list that associates the names of the upload
fields to the blobs they contained.
For all Flask apps, all calls made to methods in the
BlobstoreUploadHandler
class require the request.environ dictionary (request being imported from the
flask module). If your app is a WSGI app without a web framework, you use the
environ parameter in the get_uploads() method.
When rewriting the user's request, the Blobstore empties the MIME parts of the
uploaded files and adds the blob key as a MIME part header. The Blobstore
preserves all other form fields and parts, passing them to the upload handler.
If you don't specify a content type, the Blobstore will try to infer it from the
file extension. If it can't determine a content type, it assigns the content
type application/octet-stream to the newly created blob.
Note: If you are serving images, a more efficient and potentially less-expensive
method is to use
getServingUrl()
get_serving_url
using the App Engine Images API rather than blobstoreService.serve()
send_blob . The getServingUrl method get_serving_url function lets you
serve the image directly, without having to go through your App Engine
instances.
To serve blobs, you must include a blob download handler as a path in your
application. This handler should pass the blob key for the desired blob to
blobstoreService.serve(blobKey, res);
This handler should pass the blob key for the desired blob to blobstore.Send
The application serves a blob by setting a header on the outgoing response. If
using Flask, the
BlobstoreDownloadHandler
class requires the request.environ dictionary (request being imported from the
flask module). If your app is a WSGI app without a web framework, you use the
environ parameter in the send_blob() methods . In this example, the blob
key is passed to the download handler as the URL argument
(req.getParameter('blob-key')).
the URL argument r.FormValue("blobKey").
part of the URL. In practice, the download handler can get the blob key by any means you choose, such as through another method or user action.
Blobs can be served from any application URL. To serve a blob in your application, you put a special header in the response containing the blob key. App Engine replaces the body of the response with the content of the blob.
The Blobstore supports serving part of a large value instead of the full value
in response to a request. To serve a partial value, include the
X-AppEngine-BlobRange header in the outgoing response. Its value is a standard
HTTP byte range. The
byte numbering is zero-based. A blank X-AppEngine-BlobRange instructs the API
to ignore the range header and serve the full blob. Example ranges include:
-
0-499serves the first 500 bytes of the value (bytes 0 through 499, inclusive). -
500-999serves 500 bytes starting with the 501st byte. -
500-serves all bytes starting with the 501st byte to the end of the value. -
-500serves the last 500 bytes of the value.
If the byte range is valid for the Blobstore value, the Blobstore sends a 206
Partial Content status code and the requested byte range to the client. If
the range is not valid for the value, the Blobstore sends 416 Requested
Range Not Satisfiable.
The Blobstore does not support multiple byte ranges in a single request (for
example, 100-199,200-299), whether or not they overlap.
A Go 1.11 demo app is available in
GitHub.
Note that you need to insert /v2/ in the old Blobstore package name.
In the following sample application, the application's main URL loads the form that asks the user for a file to upload, and the upload handler immediately calls the download handler to serve the data. This is to simplify the sample application. In practice, you would probably not use the main URL to request upload data, nor would you immediately serve a blob you had just uploaded.
See the Flask app example in the Blobstore API for Python 3 guide.
The Images service can use a Blobstore value as the source of a transformation. The source image can be as large as the maximum size for a Blobstore value. The Images service still returns the transformed image to the application, so the transformed image must be smaller than 32 megabytes. This is useful for making thumbnail images of large photographs uploaded by users. For information on using the Images service with Blobstore values, see the Images Service documentation.
You can use the Blobstore API to store blobs in Cloud Storage instead of storing
them in Blobstore. You need to set up a bucket as described in the Cloud
Storage documentation and specify the bucket and
filename in the BlobstoreService
createUploadUrl,
specify the bucket name in the UploadOptions parameter.
UploadURLOptions you supply to the UploadURL function.
blobstore.create_upload_url
gs_bucket_name parameter.
In your upload handler, you need to process the returned FileInfo metadata
map of returned blobs
FileInfo metadata and explicitly store the Cloud Storage filename needed to retrieve the blob later.
You can also serve Cloud Storage objects using the Blobstore API.
To serve a Cloud Storage object, use BlobKeyForFile
to generate the required blobkey as described in
Serving a blob.
The following code snippets shows how to do this. This sample is in a request handler that gets the bucket name and object name in the request. It creates the Blobstore service and use it to create a blob key for Cloud Storage, using the supplied bucket and object name:
Note: Once you obtain a blobKey for the Cloud Storage object, you can pass it
around, serialize it, and otherwise use it interchangeably anywhere you can use
a blobKey for objects stored in Blobstore. This allows for usage where an app
stores some data in blobstore and some in Cloud Storage, but treats the data
otherwise identically by the rest of the app. (However, BlobInfo objects are
not available for Cloud Storage objects.)
If you want a more modern object storage solution, consider migrating from App Engine Blobstore to Cloud Storage.
An application can read data from Blobstore values using an interface similar to
a Python file
object. This interface can start reading a value at any byte position, and uses
multiple service calls and buffering, so an application can access the full size
of the value despite the limit on the size of a single service call response.
The
BlobReader
class can take one of three values as an argument to its constructor:
The object implements the familiar file methods for reading the value. The application cannot modify the Blobstore value; file methods for writing are not implemented.
An application can call some Blobstore functions that work in the background.
Blobstore carries out the request while the application does other things. To
make the request, the application calls an asynchronous function. The function
immediately returns an RPC object; this object represents the request. When the
application needs the result of the request, it calls the RPC object's
get_result() method.
If the service has not completed the request when the application calls
get_result(), the method waits until the request is complete (or has reached
the deadline, or an error occurs). The method returns the result object, or
raises an exception if an error occurred while carrying out the request. For
example, this code snippet
becomes
In this example, the application carries out the slow_operation() code at the
same time that Blobstore generates the upload URL.
Space used for Blobstore values contributes to the Stored Data (billable) quota. Blob info entities in the Cloud Datastore count towards Cloud Datastore-related limits. Notice that Cloud Storage is a pay-to-use service; you will be charged according to the Cloud Storage price sheet.
For more information on system-wide safety quotas, see Quotas.
In addition to system-wide safety quotas, the following limits apply specifically to the use of the Blobstore:
- The maximum size of Blobstore data that can be read by the application with one API call is 32 megabytes.
- The maximum number of files that can be uploaded in a single form POST is 500.
For information on working around these size limits, see the documentation for the Send function.