Two truths, half-a-lie and lie

Recently, we have been quite busy working on a complex JD Edwards to PeopleSoft HR data conversion activity. The objective of conversion is to port all data from JD Edwards to PeopleSoft HR. Yes, you heard it – all HR data, some of which dates back to as early as 1952 🙂

Typically, ERP conversions start off with someone maintaining the source system fedex delivering the data extract. Consider yourselves lucky if your source system is not a hosted application, or you would be paying through your nose to get an extract of ‘your’ data! Extraction of data is rarely in the scope of an ERP conversion. I’ve seen project charters that list extraction as the first and last points under the Out-of-Scope section!

What made this undertaking rare and special for us is that we chose to extract, cleanup, transform and convert data into PeopleSoft. Yep, extraction and cleanup was on the house 🙂

Now that we stand close to handing over converted data for UAT, the whole experience was yet another re-affirmation of a few truths/lies about data conversion.

  • Truth – Quality of source data is a primordial determinant of how simple or how complex the conversion is going to be.
  • Truth –  Involve business owners very early and work out the validation strategy. Make them responsible for validation.
  • Half a lie –  Its sufficient for the project team to be masters of target system. Just a little awareness of the source system is sufficient.
  • Lie – Focus on data conversion during the initial stages and forget about it.

Let me talk about quality of source data. The saying “Garbage In, Garbage Out” may apply in everything related to computers, but never ever when converting data. In that sense, data conversion is a green endeavor, involving a lot of recycling.  Over in this world, when its “Garbage In”, you got to recycle and let “Clean data out”.

In PeopleSoft, there are strict rules around allowing hire, terminate and rehire transactions. You would not be able to rehire an employee until they are terminated. Likewise, you cannot terminate an employee who is already terminated. None of these rules (and many more) were enforced in source system, which led to poor quality of data from a PeopleSoft perspective. We had to come up with massive cleanup algorithms to make source data PeopleSoft friendly.

In our case, source data came from three primary sources, one of which was an audit table. That was enough reason to give some sleepless night during the design phase. The audit table presented us with a major “quality” challenge.  A business transaction, like hiring an employee, promoting an employee etc., results in many rows in the audit table, each row capturing data change in one data element. For example, if promoting an employee requires  the user change department, jobcode, promotion date and compensation rate, the audit table will have four extra rows, one for each data element. We decided to perform a row-to-column transformation of the entire audit table. That was the first step in our progression towards building PS-friendly data.

Devising the strategy to cleanup, transform and convert data is only half the battle. The other half was in building the validation strategy, which involved placing the right kind of checks and balances during and after conversion. Alarms should go off when an employee looks active in PeopleSoft but terminated in the source system. Without comprehensive queries and reports that juxtapose raw and converted data, there’s no proof in the pudding.

Over to “half a lie” – “Just a little awareness of the source system is sufficient”. Its only half true, because as I mentioned earlier, most ERP conversions get the source data fedex-delivered. So, its more than sufficient to put together a mapping document and proceed with the build. In our case, we had to be conversant with the source system and know its nitty-gritties, to achieve the goal.

Over to the lie – “Focus on data conversion during the initial stages and forget about it”. Nothing could be farther from truth. In an implementation, midway scope creeps and changes in business processes always warrant a change in the way data is converted. As users get to know their way around the new system, their insights will have an impact on conversion. In short, effort and cost estimation for conversion activities have to consider such scenarios, else budget revisions become indispensable.