Creating a DMF data package from files stored in an Azure blob storage

Dynamics 365 for Finance and Operations support different scenarios for data integration. 
Each scenario is listed on the following documentation:

One of the most common scenarios is when a large number of records need to be imported in D365FO. These record normally are coming from a third-party application as batches. If we follow the documentation guidelines for this scenario, the recommended pattern to be applied is Batch Data API
For this kind of pattern we have the following API available: 

  • Recurring Integration API 
  • Data Management API 

Many people uses the Recurring Integration API, however, I found it lacks of features (like asking for the task status) and also I haven't seen any further investment to improve it. For these reasons I choose to use the Data Management API. 

 Data Management API is a robust API and their specification can be found here:


The problem with this API is that is it not simple to use as there are many steps to perform before doing the Import (unlike the Recurring Integration API)

A typical flow to Import data is as follow:

  1. Create a DMF .zip package
  2. Ask for a writable URL to upload the package
  3. Upload the package
  4. Start the Import of the package
  5. Ask for the Import status.
On this post I'll focus on solving step number 1 in a way that can be automatized and used on high performant integration scenarios. 

The content of the package

A DMF package is a .zip file that contains the following files: 
  1. Manifest.xml
  2. PackageHeader.xml
  3. As many data files as referenced (typically .csv files)
One of the advantages against the Recurring Integration is the capability to upload many entities on a single import as make full use of the DMF framework capabilities. 

How to get Manifest.xml and PackageHeader.xml 

The easiest way to generate these files is creating a Data Export project, add the entities that you will be including, and then execute the project and download the package. Pick the Manifest.xml and PackageHeader.xml files.  

Want to know a little more about what are these two files are? check this post from the community Purpose of manifest and Packageheader in data download package - Microsoft Dynamics AX Forum Community Forum

Using this on an Integration

When dealing with integrations we need to automatize any manual task, hence each package needs to be created dynamically as we could be dealing with more than one entity and more than one integration. 

To address the problem above an optimal solution will make use of a cloud staging storage - Azure Blob Storage - and a serverless function - Azure functions - to grab and assemble the .zip package. 

Why Azure Blob Storage and Azure Functions? Simple. They are the most robust and cheapest available options. 

The solution 

In the following public repository is my implementation for this solution. Once you have assemble the package you can continue with the steps described above to make the import. I'll be sharing a robust, tested, production ready integration solution built on LogicApps for the remaining steps in following posts. 

Link to the ZipFunction repo:










Comments

Post a Comment

Popular posts from this blog

Telemetry and Monitory with Applications Insights and Dynamics 365 F&O

Image
Telemetry and Monitory with Applications Insights and Dynamics 365 F&O With version 10.0.39 Applications Insights integrations is GA. Hence, here is a very simple example how to start using this great feature and start monitoring the performance of your application.  For more about D365FO and Application Insights integration:  Monitoring and telemetry using Application Insights - Finance & Operations | Dynamics 365 | Microsoft Learn Dynamics 365 Finance and Operations - Monitoring and Telemetry using Application Insights -TechTalk (youtube.com) In this example we will monitor the execution time of a batch job. In particular, we will use a metric to analyze the execution over time.  // Create a class file and paste this code class SlowBatchJobClass extends RunBaseBatch implements BatchRetryable { [SysEntryPointAttribute] public void run() { int maxWaitInSeconds = 20; // Maximum wait time in seconds           Sys...
Read more
Image
Distributed Printing Architecture with Power Platform  Going back to posting after a long period. Now I have started a new blog where I'll be posting content related to my work with Microsoft technology, and how you can combine Dynamics 365 + Power Platform + Azure to solve virtually any problem in a reliable and robust architecture. In this first post I'll share how to build an architecture for unattended distributed printing from Power Apps. Although there are other options (look at PCF controls ) I found this architecture simple and reliable. This solution will serve to customers looking to: Use network shared printers across their organization. Serve multiple users allowing them to distribute their printing task according to specific business rules (ex. some group will only print on 'Printer 01') Rely on the SO to print as you can use the print command Access your local resources (printers) in a secure way To connect from the canvas app to the A...
Read more

Infamous "The natural key for some table was not found" solved once and for all

Image
Data entities in Dynamics 365 for Finance and Operations (D365FO) are awesome, as long as you can use the wizard to create them. Quite often you will find that some tables throw the infamous "The natural key was not found". This is because many tables actually don't have unique indexes - for many reason - but here I'll present a general solution to create any entity - read-only entities - in very simple steps.  Capture when trying to create an entity for InventTrans table In this example I'll create an entity for the InventTrans table as this table doesn't have an unique index.  Step 1: Use the wizard to create an entity but select any know table that happens to have unique indexes. In this case I'll select InventTable.  In the next screen Deselect all the fields and click Finish. You will end up with an entity with a single datasource Step 2:  In this step we need to create a View that is based on the table that we want to expose as an entity - in this ca...
Read more