One is Enough! Duplicate CIs in the CMDB

Overview
​​
Implementing a Configuration Management Database (CMDB) is not something you just do, it is something you intentionally plan and methodically implement and consistently maintain.  Every individual Configuration Item (CI) is a crucial cog in a complex machine.  No matter the budget, no matter the timeline, no matter how skilled the engineers, a successful CMDB takes ongoing care and maintenance.  Without that intention you end up with bad data, and as most know, ridding yourself of that bad data can be an overwhelming feat. 
 
One regular CMDB maintenance activity we perform is identifying, remediating, and preventing duplicate CIs.  These are more than just a nuisance or a metric on the CMDB health dashboard, they are just as critical as missing CIs.  If your IT and Business processes can’t reference the correct CI, you’re missing out on valuable data and run the risk of making decisions based on outdated information.  

What is a duplicate CI and how do they happen? 

A duplicate CI is any two or more records in your CMDB that represent the same conceptual object.  CIs are populated via several different methods, including manual entry, import from a flat file (XLSX, Text, etc), integration with another system of record, and ServiceNow Discovery.  Every one of these has the potential to create duplicate CIs.  
 
Do tell, how does this happen, you say?  For starters, a server named DEVUNIX001 could be discovered and added to the CMDB.  Months later when that same server is renamed to COUNIX001, discovery considers it to be a new server and creates a new CI.  Another example is the IP address, where subnets can change and machines receive new IP addresses.  If you’re using IP address to match (i.e. your coalesce value) discovered machines to existing CIs, you could end up with duplicate CIs instead of updating existing records.  If the data sources that feed a CMDB are not properly configured and monitored, any one of them can cause issues.  
 
We’ve all heard the term ‘garbage in, garbage out’ and your CMDB is no exception to this rule.  It’s nearly impossible for any single system to be the original source of truth for all areas of the infrastructure.  We all work in a complex landscape of advanced technology where teams have their own management tools to support their devices, and each one of these systems is likely to become a data source for your CMDB. 
 
Perhaps you use SCCM for Windows computers and servers, Casper for Apple devices, SolarWinds for network gear, AirWatch for mobile devices, and ServiceNow Discovery for Linux machines.  It’s crucial to understand if and how the owners of these systems are keeping their data clean.  And manual entry?  I’m guessing we all know the risks of hand keyed information.  The point being your CMDB is undergoing a technical barrage of data from all sides, and without process and due diligence to manage that you will end up with duplicate CIs.

Why do I care if I have duplicates? 

The nature of a CMDB is that it is constantly being updated - weekly, daily, or even in real time.  If you have duplicate CIs, only one of them will be updated by your automated systems (discovery, integrations, etc) so data on the other CI quickly grows stale.  And why is that a problem?
 
Suppose you’re populating the assignment group on an incident based on information from the CI.  If your user selects the stale CI, this incident may get routed incorrectly.  Or what if a change request has the incorrect CI?  Automated risk calculations and approvals are based off of this stale data.  Perhaps you get lucky and this doesn’t cause a significant outage.  But as Dirty Harry asked, are you feeling lucky?
 
Whether supporting operations or analyzing trends, reliable data is paramount.  Quickly resolving an incident, troubleshooting a problem, approving a change, requesting new software, and so many other activities rely on your end users finding one specific CI record.
 
How do I stop the bleeding? 

Start by utilizing the CMDB Dashboard.  This breaks your CMDB health into three categories: completeness, compliance, and correctness.  The correctness score is further broken into three subcategories of duplicate, orphan, and staleness.  Duplicate CIs are then broken down by class.  
 
Then dig into where your fires are burning hottest:  the classes with the most duplicates.  These offer the greatest return for your effort, as it’s here you’ll likely find patterns that address tens, hundreds, or even thousands of duplicates.
 
De-duplication tasks are created automatically.  Once you’ve taken all the low hanging fruit, monitor these tasks.
 
You can also look for CIs whose discovery source is ‘Duplicate’.  
 
Finally, it’s worth noting that some duplicates are not worth the effort of remediation.  Certain classes such as load balancer pools and IP address objects don’t usually cause any major issues, and they can take a lot of time to chase down.  By prioritizing the addressing the key duplicate areas first, you’ll quickly provide qualitative benefits to your processes and CMDB consumers.

What about future duplicates? 
You don’t want to be in the business of manually remediating every single duplicate.  Often they are created much faster than a person or team could ever clean them up.  It’s important to be strategic and target the root cause.  Don’t start mopping up the water until you’ve fixed the leaky pipe. 

• Ensure your identifiers are appropriate for each class
• Utilize Identification Simulation when possible to vet new rules
  
  

Understand duplicate CIs are a natural occurrence in a CMDB.  Having duplicates doesn’t mean you’re doing it wrong; it just means you’re paying attention to the data and appreciating how it impacts your processes, or more often, how it impedes your processes. 
 
How you handle these duplicates is one of the key parts of your CMDB where you succeed or fail. 

Make sure this is someone’s job
 
This isn’t a nice-to-have, this is key to achieving next-level benefit across all of your CMDB-consuming processes.  If you went to the effort and expense of standing up a CMDB, then invest in keeping it clean and healthy.  
 
One of the tools you can use to help you with this is our CI Watch application by AJUVO.  Our CI Watch Usage identifies all records on the platform that are referencing specific CIs.  This is a critical piece of the puzzle.  
 
Deleting a duplicate without understanding where is was used can result in significant data loss.  If the stale CI was associated to operational tasks that data needs to be migrated.  If the CI has an asset associated with it, the asset will automatically be deleted.  By cleaning up one piece of bad data, you may inadvertently create another issue.  CI Watch not only finds these relationships, it also migrates them for you.  Why not take the guesswork out of your CMDB maintenance?

Or even better, let us help you! 

But what if you don’t have the time or resources or expertise to dig into and manage your duplicate CIs?  And let’s face it, the CMDB is not fun and exciting, people would rather be rolling out Virtual Agent.
 
Speak to our experts at AJUVO about CI Watch, a comprehensive solution by AJUVO to get your CMDB back on track and help keep it there.  We use our own custom applications, dedicated CMDB experts, and consultative approach to help (re)design your CMDB to support your desired business outcomes, align your existing data to achieve that value, and then provide a managed service to keep your CMDB humming along smoothly at a fraction of your normal full-time resource costs.