My Experience Building A Document Integration Process

Why am I writing this article?

• I enjoy designing integration solutions and maybe by sharing my experience I can help others do the same.

What is a document integration process?

• An integration process typically represents a discrete workflow that moves a given type of record from one application to another. The document integration process pictured above queries Salesforce for documents (Content Version records) and indexes those documents to Feith a document management system (DMS) which is hosted on the Berry College campus.

What is the use case?

• High school and college transcripts are uploaded to Salesforce by various different sources. Storing documents in Salesforce long term is expensive. Salesforce is not designed to be a DMS. These documents need to be transferred to Feith for processing and long term storage.

Where and how do I start designing a integration process?

1. Identify the systems/applications you are integrating.

• Salesforce (CRM) [Source]
• Feith (Web Service) [Destination]

2. What do I need to connect and access each system?

• Accounts
• Passwords and Authentication Tokens
• Network Details: Firewall Rules, Ports, IP Addresses

3. Do these accounts have permission to the data your accessing (records, tables, servers)?

• I can't stress how important this step is. Save yourself time and frustration by documenting and verifying these permissions. I am unfortunately speaking from experience here... Yeah... Moving on.

4. How are you going to connect these systems together?

• We chose to use Boomi an integration platform as a service (IPaaS) for this process. There are many different systems, programming languages, methods, etc... that you can use to accomplish this goal. Could I do everything in PowerShell (insert your preferred programming language here), yes. There are pros and cons to every implementation. Pick your poison... I mean antidote.

• Boomi is a great tool that saved us time and money. I would recommend Boomi and you should evaluate the system you are going to use: https://boomi.com/

5. How often is this process going to be executed?

• Will this process run on a schedule (weekly, everyday, hourly, every 5 minutes, etc...)?
• Will the process run when an API call is received by the system hosting the process?
• I chose to run this process on a schedule.

6. Identify the source data and or document(s) that will be used to begin the process.

• In this example the Salesforce Content Version object is my source.
• Identify the columns you need (Id, PathOnClient (File name), VersionData (Base 64 encoded file data), [Custom Field] (Identify type of document), [Custom Field] (Used to track if this document was successfully transferred to Feith). https://developer.salesforce.com/docs/atlas.en-us.object_reference.meta/object_reference/sforce_api_objects_contentversion.htm
• Query/Get/Extract the records from this object. Ensure you are restricting your query to only retrieve the documents you are looking for and where your custom tracking field is empty/Null. Restrict the batch size to something appropriate if you are scheduling this process to run on a schedule.

7. Transform the data and document(s) into a format your target system accepts.

• In this example the Feith (Web Service) is my target.
• Obtain an authentication token from the web service. If you are scheduling this process you should do this step first. This will allow you to reuse the authentication token and eliminate unnecessary calls to the web service.
• Construct a multipart form data request... Do you hear that... Its the sound of thousands of developers shouting "Why won't you work!".

• Where do I begin... I'll show you how I accomplished this in Boomi. I would like to thank who ever created this Boomi community post. I wouldn't have been able to figure this out without it. https://community.boomi.com/s/article/multipart-form-data

1. Construct the meta data portion of the multipart data request in your Map. Save the metadata to a dynamic document property.
2. Construct the multipart form data payload in a message. (See the image below)

3. Ensure all lines end in CRLF i.e. \r\n in multipart requests so after your message shape, add a data process shape to do a search for [\n] and replace with [\r][\n]

4. Use a Set Properties shape for any header or path variables used in your HTTP Client Operation. *Don't set these parameters in the HTTP Connector shape. Doing so will send an empty payload. I learned this the hard way...

5. Set the request content type in the HTTP Client Operation to exactly this "multipart/form-data; boundary=testboundary". You will have to type this into the Content Type field. You can use any boundary you want, but keep it simple.

8. Update the Salesforce Content Version record [Custom Field] used to track if the document was sent over successfully. I used the document id returned in the multipart form data response in step 7. Saving the destination ID to the custom field on the Content Version record will allow you to confirm/reference the document in the destination system.

9. When you are able to successfully test your integration. Add error handling to your process. Expect everything to fail. I mean everything... There are going to be unexpected errors that come up in the future. So prepare and document error handling steps to accommodate for the errors you know are possible.

I left out quite a bit in this article, but this was my general experience creating this process. Did I create the most efficient integration solution for the use case? Probably not. There is always going to be some developer out there manically laughing who created a similar process in assembly.

I wrote this article to help encourage others. Don't be afraid to fail. Trust me, I am a professional at failing. Let me know what you think of this article in the comment section below.

Thank you for reading.

-Faraj Daoud