Challenge
Recently a client had asked for custom development that required processing of thousands of CRM records, each with many relationships which all get affected. Each of those records had large amounts of data that needed to be processed. This was a huge challenge because we had to allow the user to process these records, yet be able to perform other jobs in CRM. This is a great case of how robust, powerful and flexible CRM can be.
Solution Attempt #1
The first thing that came to mind was writing a callout.
Pros:
· Callouts can be triggered by an update event (along with create, delete, and assign).
· Writing code in .NET is relatively inexpensive using object oriented programming principles.
· Callouts can be programmatically imported and exported and with ease.
· It provides greater control of how the program is supposed to execute and greater debugging capabilities.
Con:
· Callouts are synchronous transactions, which means that the user has to wait until the program has finished executing before moving on. This does not meet one of our requirements.
· They can be made to be asynchronous by spinning up multiple threads, but caution has to be taken so no deadlocks occur.
Solution Attempt #2
The next thing that came to mind was using a workflow that calls a custom assembly file.
Pros:
· You can specify certain rules/conditions upfront to fire the workflow.
· You can call an assembly file which can use the same basic principles of object oriented programming, debugging, etc. that callouts can allow you to use.
· Can be kicked off by a user when ever he/she needs to by pressing “Apply Rule.”
Cons:
· We found that with processing thousands of records, sometimes the workflow service would time out causing errors. The timeout limit can be set to something really high like 2 hours, but that defeats the purpose of the timeout for most workflows. Additionally, there may be times when several hours may not be enough time.
· It’s difficult debugging any issues with the workflow itself.
· No workflow API is exposed that we can access.
· Rules cannot be programmatically imported/exported.
Solution Reached!
After much soul searching, and numerous issues with the above two attempts, a happy medium was reached. We used the workflow service to call an assembly file. Within the assembly file, we spun up one thread. This thread then proceeded to execute the rest of the program, but we returned a value back to the workflow service so it “thought” the workflow had finished running. In reality, there’s a thread in the background processing all the transactions. This was ideal for this particular business’ need because of the large amount of data that needed to be processed, but also allowing the user to continue working on other things.
