Talent-Tech Is Hot … But Without a Vendor Assessment Checklist, A Bad Purchase Is Likely

Everyone is excited about the new technology options that pop up nearly every day in talent management and recruiting. Unfortunately, along with all of the excitement and the new capabilities, there are also many perils.

One of the most common problems is when talent leaders or ad-hoc tech purchasing committees buy technology impulsively, without an in-depth assessment. They often rush into the purchase simply to be able to say that “Yes, of course, we are using technology.” As a result, they end up spending hundreds of thousands of dollars on technology that never actually improves either their talent or business results. That lack of a business impact from tech purchases was highlighted in a recent survey of executives. According to Deloitte, human resources technology spending has “grown by 50 percent … in the last five years.” But the spending is not resulting in improved HR performance because “72 percent of non-HR business leaders rate HR performance as adequate, getting by or underperforming.

I’ve been involved in the hands-on implementation of technology since 1966. And as a result of that all-to-often painful experience, I estimate that because of the use of inadequate and even shallow assessment criteria, up to 80 percent of all corporate talent management technology purchases don’t produce a measurable ROI for the corporation within two years.

There are two primary reasons for the high failure rate of new talent-tech implementations. The first is a weak foundation within HR to support any new technology purchases (Note: I covered this topic in the 10/31/16 ERE.net article entitled “Talent-Tech Is Hot … But Without the Right Foundation, It Will Likely Disappoint”).

The second reason for failure is that purchases of talent tech are made without a comprehensive objective assessment checklist. 

A Checklist Covering the Best Criteria for Assessing Talent-Tech Solutions and Vendors

Anyone with more than a few years of experience in talent management probably already realizes that most of what startup talent vendors say or promise is not 100 percent reliable (to put it kindly). In order to ensure success and a positive ROI from purchasing technology, develop a detailed checklist for assessing all talent vendors and their products. Incidentally, don’t be surprised that when talking to vendors that they automatically reject the need for an assessment checklist. And, when you show them an assessment checklist, don’t be surprised when they automatically discount each of the individual items on this or any similar checklist. My recommendation is to instantly reject any vendor that badmouths the need for a tech purchase checklist with objective criteria. This is because any tech purchase ties a firm to a particular way of acting years to come, so it is essential that you make a perfect buying decision every time.

Every firm, should, of course, develop their own final technology purchase screening criteria that best fits their situation. However, the 15 criteria that I have found to be effective in assessing talent-tech vendors is listed below.  First, the “Top 10” followed by additional considerations.

The Top 10 Most Critical Vendor/Product Assessment Criteria

The most critical assessment factors are listed first.

1. Demand proof that a vendor solution has actually improved business results — don’t ever operate on the false assumption that technology automatically produces superior results over traditional human-based. Instead, in each case, demand proof of tech’s superior capabilities and its proven results for at least one firm. If the vendor doesn’t have proof that whatever they are selling directly and measurably improves one of four key performance factors, pass on buying it. Those key performance factors are 1) an improvement in the on-the-job performance of new hires, 2) an improvement in the productivity of current employees, 3) an improvement in the rate of employee innovation, or 4) an increase in revenue generation. In fact, in my experience, the more excuses that a vendor provides for not having or needing this kind of business impact data /proof, the quicker that the vendor should be disqualified from consideration. If the vendor has some limited business impact data or if they are currently undergoing a Beta test with a company to provide this proof, that is an acceptable short-term answer. If they’re willing to waive a portion of their fees if their technology doesn’t produce the targeted results at your company, that’s also a positive indicator.
2. Split sample proof is the most convincing way to prove that a product works — be wary of vendors that in lieu of producing actual product performance data, instead only provide cherry picked quotes from users or the names of firms that have used the product. In business, the best way to prove that any new technology works is to conduct a split sample side-by-side test. This is where you apply your new talent technology approach to half of a group and do nothing different in a similar control group. Then you compare the performance data from the two equal groups to see how much the solution improves performance. This approach is the way that product and drug managers prove the effectiveness of what they produce. If the vendor doesn’t have split sample proof, ask them to consider it. Where it is feasible, consider doing it at your firm in order to unambiguously prove how well the new technology actually improves results.
3. Assess the effectiveness of each individual product feature — even if the overall product works, you must still check each of the individual features to ensure that each one performs to the quality and volume specifications.
4. Assess the quality of service and vendor responsiveness — even a product that has the latest technology embedded in it won’t produce great results at your firm if the vendor doesn’t provide great service to support their product. Unless your startup is willing to offer a service level agreement, conduct a due diligence “service check” with a sample of their existing customers (that you pick out). Assess their response time, their satisfaction rates, and their service quality. Also, try to avoid “consultant switch,” which is where the initial highly knowledgeable consultant that sells your product is replaced with a less-capable junior associate immediately after the purchase is completed.
5. Vendor reliability and longevity must be assessed — in the volatile world of business, the stability of even established firms needs to be continually questioned. And the volatility among talent management firms is no different. Do you remember those many vendors that pushed referral programs during the last few years? Where are they now? Failure is so common among startup vendors that you cannot afford to be naïve about longevity, because as many as 9 out of 10 startup firms across all fields fail. Assume that the startup that you’re dealing with may have a limited future. It may be bought out by an industry giant (and their product will disappear) or they may simply fade away because they are not well managed, well funded, or their product will be surpassed by a better technology. If you have a choice among vendors in a particular product area, look for these seven factors. They are larger firms, they have a positive cash flow (or even profitability), their top managers have been in the industry for several years, they have a rising stock price (if they are a public firm), and they have sufficient capital funding to last them for more than 18 months. In addition, because the continued growth of a vendor is essential for their product to continually improve, consider looking at their own talent capabilities to determine if, in this tight job market, they are able to continually recruit new talent and retain their best employees. Finally, firms that use 6 Sigma measures or that have ISO type certifications should get extra consideration. It also might be a good idea to assume that there will be an economic downturn next year, so be extra cynical about assessing vendor longevity during tough economic times.
6. The best vendors have an impressive array of performance metrics — almost without exception, the very best vendors are true believers in kaizen or continuous performance improvement. These firms will have extensive metrics built into their product or process that cover important performance areas including volume, quality, time, satisfaction, costs, and quantified business impacts. These metrics are critical because all new technology products have flaws, and the firm won’t be able to identify and fix them unless they have extensive performance metrics. The best will also have comparison benchmark numbers so that you can tell how your firm is doing when compared to others that are also using the product.
7. Determine if your secrets, data, and privacy will be well protected — in many cases, vendors will have access to your firm’s internal data. Obviously, you need to have a data security expert look closely at the quality of the protections that they provide. If the vendor doesn’t seem sufficiently concerned about your protecting company secrets or data breaches, you need to reject that vendor. You also need to protect your data and make sure that it will still be available to you if the firm goes away.
8. Assess whether the vendor is aware of all potential problems — after 20 years of assessing vendors, I have found that the very best are not naïve about potential problems. So when asked, they can quickly provide you with a long list of the potential problems that a purchaser might likely encounter during implementation and operation. If your vendor doesn’t have such a list of problems, I found that, it is unlikely that they have prepared a solution to counter each one if it arises. If necessary, create your own list of potential problems and especially look for the costly “unintended consequences” that might occur in other departments or functions.
9. The very best vendors offer pay-for-product performance — you should start with a healthy skepticism for vendors. However, you can reduce some of that skepticism if the vendor is willing to put their fee at least partially at risk based on the performance of the product. I have found that successful vendors that have a great product are at least willing to talk about the possibility of including a performance clause in the contract. Those vendors that automatically discount the need for pay for performance clause should be automatically rejected.
10. Be wary of initial cost and time estimates — don’t be surprised when vendors provide you with an initial lowball purchase price. But select a sample of current users and survey them to see if there was a significant variation between their initial cost estimate and their final actual costs. Of course, when you are calculating the total cost of the technology, you must include in your own internal implementation cost estimates the cost of manager and employee time during implementation and operation. If time to implementation or time to operational status is important, compare the vendor’s time estimates to the experience at of some sample customers.

Additional Vendor/product Assessment Criteria to Consider 

In addition to the top 10 criteria provided above, you should also consider including the following items in your assessment checklist.