Data Process step
What is Data Process in Boomi?
The Data Process shape in Boomi is used to manipulate document data within a process. It allows you to perform various operations on the data, such as unzipping files, searching and replacing text, splitting or combining documents, or using Groovy scripts. The Data Process shape executes processing steps in a sequential order, with each step operating on the output of the previous step.
Data Process step dialog
Commonly Used Processing Steps
1.Base64 Encode
Encodes document data in Base64.
Usage: It is commonly required to send or receive files in Boomi. The encoding process also converts special characters into processable characters.
Note: It makes the data harder for humans to read and increases its size but often improves compatibility with web requests.
2.Base64 Decode
Decodes Base64-encoded document data to plain text.
Usage: It is often needed in Boomi to access text in Base64-encoded files.
Note: Plain text is often readable but increases data size.
3.Character Decode
It Decodes document data from the selected character set to the system default (typically UTF-8).
In the Character Set, you can use any standard Java platform character set :
4.Character Encode
Encodes document data from the system default (typically UTF-8) to the selected character set. You can use any standard Java platform character set described in Character Decode.
5.Split Documents
Split Documents - Splits a document into multiple documents by line, or based on a profile element value. When working with a batch of data, splitting is typically necessary to be able to perform various validations and routing on a per-record basis. To split data by a database profile, use the batching option in the database operation. The Split Documents process type can be used with the following profile types:
6.Combine Documents
Combines multiple documents into a single document. Each document's data is simply appended to the previous document's data. This combines multiple documents read in from the Start step's Get connector or that were created as a result of splitting documents in a Data Process step. The Combine Documents process type can be used with the following profile types:
7.Custom Scripting
It Provides the ability to use custom scripting to perform special processing requirements. You can access and modify the actual document data and tracked properties using JavaScript or Groovy.
A custom script can be inserted inline as part of the processing step. If you prefer, you can reference a Process Scripting component that is available for reuse. Either way, there is no need to compile because the script is ready to execute when you save the process.
8.Custom Scripting
It Allows you to use regular expressions to search the document for specific strings or characters defined in the search text box. Once the data is located it will be replaced with the string or characters defined in the replace text box.
The Search setting specifies the search method. The default is to search incrementally with the search buffer functioning as a “rolling window” whose size is the specified number of Characters. In other words, Characters specifies the maximum number of characters that can match the search target. For example, to replace “this” with “that” in “this is a test string”, you need to set Characters to 4 or greater. The default is 1,024 characters, which is the size of the buffer cache on most systems and therefore is the recommended setting.
9.Map JSON to Multipart Form Data MIME
It Converts an input JSON document containing simple form data and file attachments into a multipart/form-data MIME document.
If the JSON document contains file attachments, specify the appropriate Document Cache and Cache Index to retrieve the file attachments. To generate the MIME document, the input JSON document must also specify any headers that should be included within the MIME document. The input JSON document must also include type elements to differentiate between simple data and file attachments. The element name of a file attachment is used as the key when retrieving the file attachment from the Document Cache.
Recommended by LinkedIn
If the Type element in the JSON document is defined with a "key" value, then a Content-Type element must be included to indicate the file or data type of the document. If Content-Type is defined by any value that is not text/plain data, the "key" Type element should be treated as a file attachment. If the Type element is defined with a "data" value, the Data Process step infers that the incoming content contains text/plain data and therefore does not require a value for Content-Type.
The following is an example of a JSON input document:
{
"FirstName" : {
"value" : "John",
"Content-Type" : "text/plain; charset=ISO-8859-1",
"Content-Transfer-Encoding" : "8bit",
"type" : "data"
},
"LastName" : {
"value" : "Doe",
"Content-Type" : "text/plain",
"Content-Transfer-Encoding" : "8bit",
"type" : "data"
},
"Email" : {
"value" : "johndoe@boomi.com",
"Content-Type" : "text/plain; charset=ISO-8859-1",
"Content-Transfer-Encoding" : "8bit",
"type" : "data"
},
"Phone" : {
"value" : "6105555555",
"Content-Type" : "text/plain; charset=ISO-8859-1",
"Content-Transfer-Encoding" : "8bit",
"type" : "data"
},
"resumeAttachment" : {
"value" : "resume.docx",
"Content-Type" : "application/document",
"Content-Transfer-Encoding" : "binary",
"type" : "key"
}
}
The following is an example of a JSON to multipart/form-data MIME output document:
Content-Type: multipart/form-data; boundary=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
--xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Content-Disposition: form-data; name="FirstName"
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
John
--xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Content-Disposition: form-data; name="LastName"
Content-Type: text/plain
Content-Transfer-Encoding: 8bit
Doe
--xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Content-Disposition: form-data; name="Email"
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
johndoe@boomi.com
--xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Content-Disposition: form-data; name="Phone"
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
610555555
--xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Content-Disposition: form-data; name="resume Attachment" filename="resume.docx"
Content-Type: application/vnd.openxmlformats-
officedocument.wordprocessingml.document
Content-Transfer-Encoding: binary
t4U«¨ªÚéãkü@×&™ì¾i õO,îã\s8VŸÞ9›(*Eâø3Ï™eJÅ>q~½¾Lcg
--xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx--
10.Map Multipart Form Data MIME to JSON
It Converts an input multipart/form-data MIME document into a single JSON output document by mapping simple data and attachment files into JSON objects. Both simple data form elements and attachment files of the input MIME document are referenced directly in the JSON output document using a type element to differentiate them.
File attachments are stored in the Document Cache. The element form name of the attachment is used as the key, and is indicated in the JSON output document in an element called "type".
The following is an example of a multipart/form-data MIME input document:
Content-Type: multipart/form-data; boundary=Y-BFLHmahNxCutH0ijbUpI9_csBSF9fVt672JxIh
--Y-BFLHmahNxCutH0ijbUpI9_csBSF9fVt672JxIh
Content-Disposition: form-data; name="FirstName"
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
John
--Y-BFLHmahNxCutH0ijbUpI9_csBSF9fVt672JxIh
Content-Disposition: form-data; name="LastName"
Content-Type: text/plain
Content-Transfer-Encoding: 8bit
Doe
--Y-BFLHmahNxCutH0ijbUpI9_csBSF9fVt672JxIh
Content-Disposition: form-data; name="Email"
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
johndoe@boomi.com
--Y-BFLHmahNxCutH0ijbUpI9_csBSF9fVt672JxIh
Content-Disposition: form-data; name="Phone"
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
6105555555
--Y-BFLHmahNxCutH0ijbUpI9_csBSF9fVt672JxIh
Content-Disposition: form-data; name="resumeAttachment" filename="resume.docx"
Content-Type: application/vnd.openxmlformats-officedocument.wordprocessingml.document
Content-Transfer-Encoding: binary
t4U«¨ªÚéãkü@×&™ì¾i õO,îã\s8VŸÞ9›(*Eâø3Ï™eJÅ>q~½¾Lcg
--Y-BFLHmahNxCutH0ijbUpI9_csBSF9fVt672JxIh--
The following is an example of a multipart/form-data MIME to JSON output document:
{
"FirstName" : {
"value" : "John",
"Content-Type" : "text/plain charset=ISO-8859-1",
"Content-Transfer-Encoding" : "8bit",
"type" : "data"
},
"LastName" : {
"value" : "Doe",
"Content-Type" : "text/plain",
"Content-Transfer-Encoding" : "8bit",
"type" : "data"
},
"Email" : {
"value" : "johndoe@boomi.com",
"Content-Type" : "text/plain charset=ISO-8859-1",
"Content-Transfer-Encoding" : "8bit",
"type" : "data"
},
"Phone" : {
"value" : "6105555555",
"Content-Type" : "text/plain charset=ISO-8859-1",
"Content-Transfer-Encoding" : "8bit",
"type" : "data"
},
"resumeAttachment" : {
"value" : "resume.docx",
"Content-Type" : "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
"Content-Transfer-Encoding" : "binary",
"type" : "key"
}
}
11.PGP Encrypt
It Allows you to encrypt a file based on the desired security setting and certificate.
You also have the option to use a document property to set the file name for the PGP-encrypted file.
12.PGP Decrypt
It Allows you to decrypt a file based on the defined certificates.
For a PGP-decrypted file you also have the option to get the file name and put it in a document property.
13.XSLT Transformation
It Allows you to transform and process an input XML document into another output XML document using a user-defined XSLT stylesheet.
In the XSLT Transformation step, you can browse for and select an XSLT Stylesheet component that is available for reuse, or create a new XSLT Stylesheet component to define your XSLT script. When an XML document passes through the XSLT Transformation step, the Data Process step executes the transformation script from the selected XSLT Stylesheet component.
The value of a variable within an XSLT stylesheet can be set and retrieved using a dynamic document property. For example, dynamic document properties can be used with an XSLT transformation to receive runtime data from to be used in the XSLT transformation, or send data from inside the transformation to subsequent steps. For more information, see Related topics for using dynamic document properties with XSLT Transformations.
14.Zip
It Allows you to zip incoming data that is in the WinZip compression format. The incoming data is zipped and the documents are passed through the Data Process step into the process.
You can use a document property to set the zipped file’s file name.
15.Unzip
It Allows you to unzip incoming data that is in the WinZip compression format. The incoming data is unzipped and the documents are passed through the Data Process step into the process.
For the file to be unzipped, you can get the zipped file’s file name and put it in a document property.
THANKS FOR READING!