Challenges of End User Testing

The Missing Piece: Standardized Test Data

The mortgage industry has invested heavily in standardizing the exchange of information, primarily under the auspices of MISMO.  This has made it easier for lenders to switch service providers providing the same or similar services, for technology providers to expand their connectivity and integration offerings to their customer base with limited effort, and for service providers to have a standard approach to communication with their customers.  All this standardization should be and is helpful, reducing technology costs among other things, but it stopped short of the goal by not addressing a key piece: the standardization of testing.

The missing standardization is not related to the process of testing, but rather the data used to verify and validate the process.  Today, each service provider defines their own test cases to be used against their systems.  This has left the testers of the various connections with the unenviable task of constantly adjusting the test criteria to meet those defined for the respective service provider being tested.  Due to the many points of integration and service providers involved with the origination process, end to end testing of an LOS cannot be completed “cleanly” due to the data changes needed to interface with the respective service providers.

The LOS can include upwards of 20 separate data exchange transactions with external service providers.  In lieu of actually testing the system in an end to end fashion, the system testers must constantly adjust the data (usually through system administration functions) to meet the terms of each service providers’ test samples, or create multiple loans that are only testing specific streams of the process to validate selected transactions.  Even something as simple as defining “Joe Homeowner” at “123 Main Street, Fairfax, VA  22030” as the information with which services can be ordered through the test environments, becomes a time saver.

Although standardizing this information, is very different from standardizing the transaction, it is no less important when discussing introducing efficiencies and streamlining the exchange of information.  Standardization of the test data includes defining multiple test names, property addresses, social security numbers, and other common fields to be used in the test transactions.  While each service may require a different set of data, some standardization will go a very long way, particularly when attempting to complete end to end testing of the technology.

Each service provider maintains a different approach to testing.  In some instances, the test data is sent to a production environment, where the specific test data is recognized and treated differently from a production request, e.g., not charging the lender for the transaction. Other providers open their production environment for a limited period, allowing queries to be run without any charges being incurred during the verification process. Still others support a testing environment that includes sufficient information to respond to the transactions submitted with the test data.  The standardization being discussed would not inhibit the selected service provider’s testing methodology, rather it would define, and possibly expand, the data used during the testing activities.

What do you think?  Have you experienced this issue?  Is it time to standardize the testing data across services and providers?