Updated Couchbase Destination for Streamsets Datacollector
Following my original article introducing a Streamsets Datacollector destination for the Couchbase NoSQL Database. I've updated the destination with some new features.
The follow new features in the destination are:
- Support for Couchbase NoSQL Database version 5.0 and higher: Building on the Role Based Access Control (RBAC) security model introduced in version 4.5 for Administrators, Couchbase Server 5.0 introduces RBAC for applications. This allows you to control, at both broad and granular levels, what administrators, end-users, and application services can do. By using RBAC, you can closely align the roles you assign users and administrators to the actual roles they hold within your organisation, so that they have access to just the information they need, and therefore meet compliance requirements. In Streamsets, the destination now allows users to connect to a bucket via a bucket password for version 4 or with a Couchbase username and password for Role Based Access Control in version 5.
- Couchbase Java SDK 2.5.5
- Support for Streamsets 3.1.0.0 or higher: Updated libraries from Streamsets. As of Streamsets 3.0.0.0, OpenJDK is also supported.
- Faster ingestion into a Couchbase bucket by using the Async API of the Couchbase Java SDK: The destination now uses the Async part of the Couchbase Java SDK. This allows from faster ingestion into Couchbase as well as back pressure handling in high volume data ingestion scenarios.
- Unique document key as a record field or custom document key: Document keys can now be set as a record field from the record in Streamsets or defining a custom document key pattern by using the expression language in Streamsets Datacollector. Custom documents keys are useful if you want to structure your document key with key patterns (DocType::ID) and fields from the record which uniquely identify the record. An example of a custom document key would be: "customer::<customer_id_number>". In Streamsets, the expression language would be:
str:concat('customer::', record:value('/customer_id_number'))
The code for the destination can be forked at the Couchbase-Streamsets-Destination repository.