Engineering
Thought Machine Data Migrations: Harnessing Datadog for Real-Time Reconciliation
Aug 07, 2023
5 min read
TL;DR
Migrating from one Core Banking system to another is no easy feat. Recognizing this challenge, we at Ikigai Digital want to present our suite of Migration Accelerators. The centerpiece of our offering? A migration-pipeline accelerator designed to expedite the technical aspects of ETL (Extract, Transform, Load) processes. As an integration partner of Thought Machine, with a team of highly skilled subject matter experts, our experience of working with Vault Core has gone into this accelerator work and is being put to practice.
The ETL Journey
- Extraction: Our tool extracts source data, encompassing account information and postings, directly from a bank's legacy system.
- Transformation: The data undergoes continuous transformation based on predefined mapping rules.
- Loading: We leverage Thought Machine Vault Core's data migration APIs to efficiently load data.
- Monitoring: Continuous real-time reconciliation and progress monitoring ensure no missteps along the way.
In this piece, we're highlighting our Real-time Streaming Reconciliation Service. With the support of Datadog, this service provides an interactive dashboard, presenting a detailed view of migrated resources. Stakeholders can immediately spot successes, failures, and accordingly, take swift action.
Diving Deeper: Real-Time Reconciliation with Datadog
Steps
- Ikigai Digital's Migration Accelerator, adept at extracting from any RDBMS, validates the data against Vault Core Data Dictionary. This ensures all customer, account, and posting attributes are in order.
- The data, once verified and transformed, adheres to Vault Core's expected format: Resource Batch, Resource, and Dependency Group structure. Postings, meanwhile, are formatted as Posting Instruction Batch requests.
- This formatted request is streamed to the migration pipeline via a Kafka message, in line with Vault Core Data Migration and Posting Migrations API specs.
- Vault Core processes the request asynchronously, producing migration and resource events. Our Reconciliation Service captures these.
- The Reconciliation Service’s role is pivotal: it cross-checks migration events from Vault Core against source data, delivering real-time reconciliation outcomes.
- Crucially, while the Reconciliation Result indicates the success or failure of resource migration, it ensures sensitive payload data remains concealed.
- The outcome? A stream to DataDog's Dashboard, revealing the live status of reconciliation.
Benefits of Embracing Real-Time Migration Reconciliation
- Instant Progress Insights: With real-time dashboards, banks can monitor data migration developments as they unfold. Early detection of potential snags means swift resolution.
- Granular Migration Analysis: Banks can spot migration issues related to customers and accounts at various stages. Direct access to details and message logs offers immediate troubleshooting avenues.
- Comprehensive Reconciliation: Constant oversight of postings migration grants banks a thorough view of balance movements during migration—spanning accounts, customers, and the bank as a whole.
Ikigai Digital's Thought Machine Migration Accelerator isn't just an ETL tool; it's a holistic, cloud-agnostic data migration solution optimized for banks.
Start your data migration journey with us.
Contact Ikigai Digital for a personalized demo and learn more.
Stay up to date
Every few weeks we share the latest insights and news from the world of fintech.
Enjoyed this post? Join our team
Join our team and help build the next generation of digital banks
Solutions
Media
© 2023 Ikigai Digital