|
1 | | -[](https://travis-ci.org/durch/rust-bigtable) |
2 | | -[](https://crates.io/crates/bigtable) |
3 | 1 | [](https://github.com/durch/rust-bigtable/blob/master/LICENSE.md) |
4 | | -[](https://gitter.im/durch/rust-bigtable?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) |
5 | 2 | [](https://github.com/zrosenbauer/awesome-bigtable) |
6 | 3 |
|
7 | 4 | ## rust-bigtable [[docs](https://durch.github.io/rust-bigtable/)] |
8 | 5 |
|
9 | 6 | Rust library for working with [Google Bigtable](https://cloud.google.com/bigtable/docs/) [Data API](https://cloud.google.com/bigtable/docs/reference/data/rpc/google.bigtable.v2) |
10 | 7 |
|
11 | | -### Intro |
12 | | -Interface towards Cloud Bigtable, supports all [Data API](https://cloud.google.com/bigtable/docs/reference/data/rpc/google.bigtable.v2) methods. |
| 8 | +### Supported API Methods |
13 | 9 |
|
14 | | -+ CheckAndMutateRow |
15 | | -+ MutateRow |
16 | | -+ MutateRows |
17 | | -+ ReadModifyWriteRow |
18 | | -+ ReadRows |
19 | | -+ SampleRowKeys |
| 10 | +Full coverage of the Bigtable Data API v2: |
20 | 11 |
|
21 | | -Includes support for [JWT auth](https://cloud.google.com/docs/authentication): |
| 12 | +| Method | Description | |
| 13 | +|--------|-------------| |
| 14 | +| `ReadRows` | Streams back the contents of all requested rows | |
| 15 | +| `SampleRowKeys` | Returns a sample of row keys in the table | |
| 16 | +| `MutateRow` | Mutates a row atomically | |
| 17 | +| `MutateRows` | Mutates multiple rows in a batch | |
| 18 | +| `CheckAndMutateRow` | Mutates a row atomically based on output of a predicate filter | |
| 19 | +| `ReadModifyWriteRow` | Modifies a row atomically on the server | |
| 20 | +| `PingAndWarm` | Warms up connection channels to the service | |
| 21 | +| `GenerateInitialChangeStreamPartitions` | Generates initial change stream partitions | |
| 22 | +| `ReadChangeStream` | Reads changes from a table's change stream | |
| 23 | +| `PrepareQuery` | Prepares a GoogleSQL query for execution | |
| 24 | +| `ExecuteQuery` | Executes a GoogleSQL query against a Bigtable table | |
22 | 25 |
|
23 | | -### How it works |
| 26 | +### How It Works |
24 | 27 |
|
25 | | -Initial plans was to go full `grpc` over `http/2`, unfortunately Rust support is not there yet, so a middle way was taken :). |
| 28 | +Requests are `protobuf` messages generated from [Google's proto definitions](https://github.com/googleapis/googleapis/blob/master/google/bigtable/v2/bigtable.proto). These messages are converted to JSON and sent to the predefined REST endpoints. Responses are returned as `serde_json::Value`. |
26 | 29 |
|
27 | | -Requests objects are `protobuf` messages, generated using `proto` definitions available from [Google](https://github.com/googleapis/googleapis/blob/master/google/bigtable/v2/bigtable.proto). And all configuration is done through very nice interfaces generated in this way. These messages are then transparently converted to `json`, and sent to predefined `google.api.http` endpoints, also defined [here](https://github.com/googleapis/googleapis/blob/master/google/bigtable/v2/bigtable.proto). Responses are returned as [serde_json::Value](https://docs.serde.rs/serde_json/value/index.html). |
| 30 | +Authentication is handled via [goauth](https://crates.io/crates/goauth) with JWT tokens. |
28 | 31 |
|
29 | | -In theory this should enable easy upgrade to full `grpc` over `http/2` as soon as it becomes viable, the only remaining work would be utilising proper return types, also available as `protobuf` messages. |
| 32 | +### Installation |
30 | 33 |
|
31 | | -### Configuration |
| 34 | +```toml |
| 35 | +[dependencies] |
| 36 | +bigtable = "0.6" |
| 37 | +``` |
32 | 38 |
|
33 | | -You can provide the `json` service accounts credentials obtained from Google Cloud Console or the private key file in `pem` or (see [random_rsa_for_testing](https://github.com/durch/rust-bigtable/blob/master/random_rsa_for_tests) for proper format) format as well as Google Cloud service account with proper scopes (scopes are handled by [goauth](https://crates.io/crates/goauth), as part of authentication), |
| 39 | +### Configuration |
34 | 40 |
|
35 | | -### Usage |
| 41 | +Provide service account credentials from Google Cloud Console as a JSON key file: |
36 | 42 |
|
37 | | -*In your Cargo.toml* |
| 43 | +```rust |
| 44 | +use bigtable::utils::get_auth_token; |
38 | 45 |
|
39 | | -``` |
40 | | -[dependencies] |
41 | | -bigtable = '0.3' |
| 46 | +let token = get_auth_token("service-account-key.json", true)?; |
42 | 47 | ``` |
43 | 48 |
|
44 | | -#### Higher level wrappers (`wraps`) |
| 49 | +### Usage |
| 50 | + |
| 51 | +#### High-Level Wrappers |
45 | 52 |
|
46 | | -There and higher wrappers available for reading and writing rows, so there is not need to craft `protobufs` manually. Write can also be used to update, but not very robustly yet, coming soon :). |
| 53 | +Simple wrappers for common operations: |
47 | 54 |
|
48 | | -##### Read and Write |
| 55 | +```rust |
| 56 | +use bigtable::utils::get_auth_token; |
| 57 | +use bigtable::wraps; |
| 58 | +use bigtable::support::Table; |
| 59 | + |
| 60 | +// Read rows with limit |
| 61 | +let token = get_auth_token("credentials.json", true)?; |
| 62 | +let table = Table::default(); |
| 63 | +let rows = wraps::read_rows(&table, &token, Some(100))?; |
| 64 | + |
| 65 | +// Bulk write rows (uses MutateRows - higher throughput) |
| 66 | +let mut rows = vec![wraps::Row::default()]; |
| 67 | +wraps::bulk_write_rows(&mut rows, &token, table.clone())?; |
| 68 | + |
| 69 | +// Write rows one at a time (uses ReadModifyWriteRow) |
| 70 | +let mut rows = vec![wraps::Row::default()]; |
| 71 | +wraps::write_rows(&mut rows, &token, &table)?; |
| 72 | +``` |
49 | 73 |
|
50 | | -Read wrappers allows for simple limit on the number of rows, it uses the `ReadRows` underlying method. |
| 74 | +#### Direct API Access |
51 | 75 |
|
52 | | -There are two write strategies available, `bulk_write_rows` and `write_rows`. `bulk_write_rows` first collects all writes and fires only one request, underlying method is `MutateRows`, this results in a much higher write throughput. `write_rows` shoots one request per row to be written, underlying method is `ReadModifyWriteRow`. |
| 76 | +For full control, use the request builder directly: |
53 | 77 |
|
54 | 78 | ```rust |
| 79 | +use bigtable::request::BTRequest; |
| 80 | +use bigtable::method::{BigTable, ReadRows, MutateRow}; |
| 81 | +use bigtable::utils::{get_auth_token, encode_str}; |
| 82 | +use bigtable::protos::data::{Mutation, mutation}; |
| 83 | + |
| 84 | +let token = get_auth_token("credentials.json", true)?; |
| 85 | + |
| 86 | +// ReadRows |
| 87 | +let mut req = BTRequest { |
| 88 | + base: None, |
| 89 | + table: Default::default(), |
| 90 | + method: ReadRows::new(), |
| 91 | +}; |
| 92 | +req.method.payload_mut().rows_limit = 10; |
| 93 | +let response = req.execute(&token)?; |
| 94 | + |
| 95 | +// MutateRow with SetCell |
| 96 | +let mut req = BTRequest { |
| 97 | + base: None, |
| 98 | + table: Default::default(), |
| 99 | + method: MutateRow::new(), |
| 100 | +}; |
| 101 | + |
| 102 | +let mut set_cell = mutation::SetCell::new(); |
| 103 | +set_cell.family_name = String::from("cf1"); |
| 104 | +set_cell.column_qualifier = encode_str("col1"); |
| 105 | +set_cell.timestamp_micros = -1; |
| 106 | +set_cell.value = encode_str("value1"); |
| 107 | + |
| 108 | +let mut m = Mutation::new(); |
| 109 | +m.mutation = Some(mutation::Mutation::SetCell(set_cell)); |
| 110 | + |
| 111 | +req.method.payload_mut().row_key = encode_str("row1"); |
| 112 | +req.method.payload_mut().mutations.push(m); |
| 113 | + |
| 114 | +let response = req.execute(&token)?; |
| 115 | +``` |
55 | 116 |
|
56 | | -extern crate bigtable as bt; |
| 117 | +### Testing |
57 | 118 |
|
58 | | -use bt::utils::*; |
59 | | -use bt::wraps; |
| 119 | +Integration tests run against a live Bigtable instance: |
60 | 120 |
|
61 | | -const TOKEN_URL: &'static str = "https://www.googleapis.com/oauth2/v4/token"; |
62 | | -const ISS: &'static str = "some-service-account@developer.gserviceaccount.com"; |
63 | | -const PK: &'static str = "pk_for_the_acc_above.pem"; |
| 121 | +```bash |
| 122 | +# Run integration tests (requires credentials) |
| 123 | +cargo test --test integration_tests -- --ignored --test-threads=1 |
64 | 124 |
|
65 | | -fn read_rows(limit: i64) -> Result<(serde_json::Value), BTErr> { |
| 125 | +# Run doc tests |
| 126 | +cargo test |
| 127 | +``` |
66 | 128 |
|
67 | | - let token = get_auth_token(TOKEN_URL, ISS, PK)?; |
68 | | - let table = Default::default(); |
| 129 | +### Dependencies |
69 | 130 |
|
70 | | - wraps::read_rows(table, &token, Some(limit)) |
| 131 | +- `protobuf` / `protobuf-json-mapping` - Protocol buffer handling and JSON conversion |
| 132 | +- `goauth` / `smpl_jwt` - Google OAuth2 / JWT authentication |
| 133 | +- `curl` - HTTP client |
| 134 | +- `serde_json` - JSON serialization |
71 | 135 |
|
72 | | -} |
| 136 | +### License |
73 | 137 |
|
74 | | -fn write_rows(n: usize, bulk: bool) -> Result<(), BTErr> { |
75 | | - let mut rows: Vec<wraps::Row> = vec!(wraps::Row::default()); // put some real data here |
76 | | - let token = get_auth_token(TOKEN_URL, ISS, PK)?; |
77 | | - let table = Default::default(); // Again use a real table here |
78 | | - if bulk { |
79 | | - let _ = wraps::bulk_write_rows(&mut rows, &token, table); |
80 | | - } else { |
81 | | - let _ = wraps::write_rows(&mut rows, &token, table); |
82 | | - } |
83 | | - Ok(()) |
84 | | -} |
85 | | -``` |
| 138 | +MIT |
0 commit comments