Dual-write framework (Do and Don’t)

To set up the dual-write solution please read my previous blog here.

The purpose of this blog is to describe the best practices described by Microsoft and share some experiences that would help you why we need dual-write and how to use it effectively. Updated blog with some exciting features introduced in 2022/2023.

(Custom mapping into publisher and importing into a solution for export to other environments.

Also during integrations, we need to be patient as a journey between integration systems (source and destination) is not always a straight line. :)

The Solution

Dual-write is an event-driven framework that provides near-real-time interaction between customer engagement (common data service) apps and Finance and Operations apps. Any data change in Finance and Operations apps causes writes to Common Data Service, and any data change in Common Data Service causes writes to Finance and Operations apps. This automated data flow provides an integrated user experience across the apps.

The solution covers master data scenarios like customers, products and vendors; end-to-end process flows like the prospect to cash; and on-demand functions like pricing and reference data like a ledger, tax, payment terms, and schedules. This includes Prospect to Cash (Supply chain management), Project automation, and Field service module scenarios. It will continue to expand in the future to support more scenarios like party, project, on-hand inventory, and so on. The framework is extensible and accommodates customer-centric business data exchange through a few additional clicks.

There is a good explanation from the Microsoft article here, we will also utilize this to understand the concepts along the way.

Dual-write vs. classic data integration patterns

Dual-write provides synchronous, bi-directional, near-real-time experience between model-driven applications in Dynamics 365 and Finance and Operations applications. Data synchronization happens with little or no intervention and is triggered by creating, updating, and deleting actions on an entity. Dual-write is suitable for interactive business scenarios that span across Dynamics 365 applications.

Whereas classic data integration is suitable for business scenarios that involve bulk ingress/egress of data across Dynamics 365 applications.

Furthermore, both the FO and Dataverse instances must in the same tenant, the solution will not work.

Dual-write has two aspects: an infrastructure aspect and an application aspect.

Infrastructure

Application (Orchestration)

WHY dual-write

Here are some reasons (as per Microsoft):

• Dual-write provides tightly coupled, near-real-time, and bidirectional integration between Finance and Operations apps and model-driven apps in Dynamics 365. This integration makes Microsoft Dynamics 365 the one-stop shop for all your business solutions. Other Dataverse modules like Field service or Project Automation are also included in this solution.
• Data from customers, products, operations, projects, and the Internet of Things (IoT) automatically flows to Common Data Service through dual-write. This connection is useful for businesses that are interested in Power Platform expansions.
• The dual-write infrastructure follows the no-code/low-code principle. Minimal engineering effort is required to extend the standard table-to-table maps and to include custom maps.
• Bi-directional and real-time integrations.
• Dual-write supports both online mode and offline mode. Microsoft is the only company that offers support for online and offline modes.

Requirements and Limitations

The setup of a dual-write connection has the following requirements:

• Finance and Operations apps that have build version 10.0.9 (10.0.383.20013) (Quality update) and platform update 33 or later
• Model-driven apps in Microsoft Dynamics 365 that have platform version 9.1.0000.11732 or later

Dual-write has these limitations:

• You can't run dual-write and the Prospect to cash solution for Data integrator side by side. If you're running the Prospect to Cash solution for Data integrator, you must uninstall it.
• Dual-write setup is not supported on trial instances of Finance and Operations apps.
• Dual-write does not support cross-company data sharing.

Dual-write from Dataverse Perspective

Dual-write consists of two AppSource solutions that are installed on Common Data Service. 

These solutions expand the entity schema, plugins, and workflows on Common Data Service so that they can scale to ERP size.

• When the dual-write AppSource package is installed, Common Data Service will have new concepts such as company and party. These concepts help applications built on Common Data Service, including Dynamics 365 Sales, Dynamics 365 Marketing, Dynamics 365 Customer Service, and Dynamics 365 Field Service, to interact seamlessly with Finance and Operations apps.
• Activities and notes are unified and expanded to support both C1s (users of the system) and C2s (customers of the system).
• To prevent data loss during currency transmission between Finance and Operations apps and the Common Data Service, you'll be able to extend the number of decimal places in the currency data type of customer engagement apps. The feature auto-translates existing records to the new extended state at the metadata layer. During this process, the currency value is translated to decimal data rather than money data, and the currency value supports 10 decimal places. This feature is opt-in, and organizations that don't need more than 4 decimal places of precision do not need to opt-in. For more information, see Currency data-type migration for dual-write.
• Date effectivity will be added to the Dataverse. It will support past, present, and future data on the same entity.
• Product unit conversions are supported for products, quotes, orders, and invoices.

Implementation Scenarios (simple example)

let's say a customer’s representative receives a call from a customer in Dynamic365 Customer Service and the customer asks to change its billing and shipping address and saves the record. The same record should be synchronized back to Finance and Operations. The following parameters help in making a decision whether we need dual write synchronization or some other integration (e.g.: classic).

As you can see, this near-time data synchronization can be best implemented by dual write.

Implementation Considerations (Do and Don'ts)

Instance strategy –Supporting 1-1 Link between Finance and Operations apps and Dataverse instance

 Do:

• Finance and Operations apps and Dataverse instances must be in the same tenant.
• Reduce the latency between Finance and Operations apps/Dataverse instances, preferably hosting them in the same Azure region.
• Finance and Operations apps instance can only be linked to a single Dataverse instance, and vice versa.
• Dual-write out-of-the-box scenarios can only be used with “Dynamics 365 apps enabled” Dataverse instance

Don’t:

• Don’t host Finance and Operations apps /Dataverse instances in different regions with higher latencies
• You can’t link Finance and Operations apps/Dataverse across two different tenants
• You can’t link the Finance and Operations apps instance to multiple Dataverse instances. You also can’t link the Dataverse instance to multiple Finance and Operations app instances.
• You can’t use Dual-write out-of-the-box scenarios with a blank Dataverse instance (i.e. without Dynamics 365 apps)

Initial Write (sync) –Best used to create initial data of low volumes

DO:

• Plan for testing Dual-write related data migration and entity sequencing
• Use initial write (sync) to create an initial reference, set up, and master data of low volumes
• For master data of higher volume
• Migrate the data into each app separately before enabling dual-write.
• Plan for Data migration outside the cutover window
• For Dataverse instances with existing business data, Bootstrap data with company information before running initial writes (sync).

Don’t:

• Don’t use initial write (sync) as a data migration replacement. Data migration requires special performance and error handling considerations that are suitable to be done before enabling dual-write.

Don't for Running Mapping - (never pause):

• Never Pause a mapping that is in a running state, always stop the mapping before making any changes to mapping.
• If you leave the mapping running in the PAuse state for more than 24 hours, you will get a Queue records error for the mapping. Yes, I was also surprised to hear that during a meeting with the Dual-write team. The particular error looks following

Always save changes in your mapping into your own custom publisher - DO

DO

• First, install your own publisher into the environment and save your mapping into your own publisher by selecting SaveAs from mapping.
• This way you are able to save different versions of your mappings in case you do not need all the fields from standard dual mappings.

Import mappings into the solution to export into other environments

• Fantastic feature from Microsoft to import mappings into your solution so you can export them to other environments.
• You can do that from Make Powerapps into Solutions -> Add existing -> More ->Other -> Dual Write table map.

• Application scenarios –Reusable building blocks for the final solution

DO:  

• Evaluate available out-of-the-box scenarios and leverage them in the final solution.
• Leverage harmonized concepts such as company concept and its related security alignments

Don’t:

• Don't start from scratch with 100% custom maps, especially for well-known scenarios such as Customers, Vendors, Contacts, and Orders.
• You can’t run the two different versions of the prospect-to-cash scenario (Dual-write and Data integrator) side by side

 Data management –Ensure scenario compatibility

DO:

• While running infrequent high-volume bulk operations on dual-written fields, consider pausing related Dual-write maps during the operation, for the best possible performance, and resuming it on completion.
• Number sequences for the Finance and Operations apps and Dataverse apps are not connected. In a multi-mastered entity scenario, plan for separate number sequence format or range per app, such as F0001, F0002 in Finance and Operations apps and C0001, C0002 in CDS, Or US0001 to US4999 in Finance and Operations apps and US5000 to US9999 in CDS.

DON’T:

• You can’t use Finance and Operations “Cross company data sharing” with Dual-write as it’s not yet supported

Error management –Proactively configure alert setup and notifications

 DO:

• Consider pausing Dual-write during planned outages & and maintenance windows and resume afterward
• Configure alerts notifications and settings to handle unplanned maintenance and outages proactively

DON’T:

• Don’t leave Dual-write maps running during planning outages & and maintenance windows to avoid user impact
• Don’t set up alert notifications to personal emails, instead use a distribution list.

Application lifecycle management –Leverage Dual-write solution awareness

DO:

• ·Plan for Power Apps Solutions to move customized dual-write entities and field maps across environments
• Use Azure DevOps to store customized Dual-write entities and field mappings

DON’T:

•Don't move Dual-write customizations manually across environments

Applications roadmap –Stay aligned with dual-write journey and future scenarios

DO:

• Align with our roadmap for upcoming features
• Join our Insider Program & Dual-write yammer group to get the previews and share your feedback
• Minimize customizations around known roadmap items such as:
• Global address book concepts in Dataverse.
• Default dimensions concept in Dataverse.
• Project operations integrations

DON’T:

• Avoid building complex customization in replacement of known planned upcoming features

This article explains all the recommended definitions for Dual-write from Microsoft and when and what to follow. This will be beneficial to the community for making decisions about why we need Dual write. 

Resources

Install-Core solution (import/export mappings)

https://learn.microsoft.com/en-us/dynamics365/fin-ops-core/dev-itpro/data-entities/dual-write/app-lifecycle-management

Dual-write home page

https://docs.microsoft.com/en-us/dynamics365/fin-ops-core/dev-itpro/data-entities/dual-write/dual-write-home-page

Dual-write GA announcement

https://cloudblogs.microsoft.com/dynamics365/it/2020/03/27/dual-write-automates-data-flow-between-dynamics-365-applications-and-common-data-service/

System requirements and prerequisites

https://docs.microsoft.com/en-us/dynamics365/fin-ops-core/dev-itpro/data-entities/dual-write/requirements-and-prerequisites

Application lifecycle management

https://docs.microsoft.com/en-us/dynamics365/fin-ops-core/dev-itpro/data-entities/dual-write/app-lifecycle-management

I hope this will be beneficial to the community. In the next article, we will see how to set dual-write.

Adnan Samuel