The data cleanup nobody budgets for
Every AI project in an SME hits the same wall, and it is not the model. It is that the same supplier exists four times.
Last month this blog said that deduplicating your supplier master data is the prerequisite for automating invoice processing. That deserves its own post, because it is the step that decides whether any of this works, and the step everyone skips.
What is actually wrong with your data
Four things, in every SME, without exception.
Duplicates that are not identical. « Muster AG », « Muster AG » with a trailing space, « Muster Aktiengesellschaft », and « Muster » from when somebody was in a hurry. A person recognises all four instantly. A matching routine does not.
Identifiers that changed. Article numbers were restructured in 2019. Most records were updated. Most is the problem.
Meaning hidden in free text. The delivery note field that contains « only Tuesdays, call Herr Keller first ». Real operational knowledge, in a place no system can use.
Records nobody dares delete. Customers from 2014 with an open flag, because deleting them might break a report nobody has run since.
How to do it without a project
Do not attempt a full cleanse. You will not finish it and the effort will be wasted on records nobody touches.
Clean only what your first automation needs. If the project is supplier invoices, clean the suppliers who actually sent an invoice in the last twelve months. In most SMEs that is 60 to 120 records out of several thousand, and it is a day of work rather than a quarter.
Fix forward, not backward. Put a rule in place that stops new duplicates being created — a required field, a search-before-create step. Otherwise you will clean the same data again next year.
Keep the decisions. When you merge two records, note which won and why. Somebody will ask.
The argument for doing it anyway
Clean master data is the one investment here that does not depend on which model, vendor or tool wins. It improves your existing reports, your existing ERP and your existing invoicing today, and it makes every future automation cheaper.
It is also the only part of an AI project that no supplier can do for you, which is precisely why no supplier mentions it.