Executive Summary

Measurement of savings, whether described as Return on Investment or some other term, is the most contentious issue in disease management, the single most common objection raised by detractors in this field. Payors which can convince employers and other customers that their methodology is sound enjoy a considerable advantage in the marketplace. Yet until now no objective and qualified arbiter of ROI offered the equivalent of the three accreditation programs for vendors. As a result the correlation between payors which should justifiably enjoy that advantage and payors which do enjoy that advantage is weak to non-existent and is based mor on marketing expertise than biostatistical soundness.

Since 2005, The Disease Management Purchasing Consortium has, uniquely, drawn a "bright line" between those measurements which are valid and those which are not. This measurement is not based on minimum standards, such as adherence to "principles" of disease management savings measurement. It is quite possible, even likely, to measure incorrectly even adhering to those principles, since none of the published "principles" describe the hidden Regression to the Mean which is revealed in our White Paper on Measurement.

The goal of this Certification Program is to leave no doubt in employers' minds about the validity of claims made by health plans or (in the case of the Benefits Consultants Certification) the vendors themselves.

Background

Perhaps 90% of measurement is invalid. Worse, the measurers don't even know that their measurements are invalid...and sometimes they even publish them, present them, or put them on their website. Public examples:

• Mercer routinely “validates” more savings than the organizations being validated even spend on adverse events in the categories being measured.  For instance, they validated $230,000,000 for North Carolina Medicaid without even checking to see if they spent that much in avoidable admissions in the categories which were the focus of the effort.  And they “validated” up to $6-million in savings for a large retailer in asthma, when it turned out that asthma ER and IP visits had not declined at all.
• One wellness vendor, Trestletree, claimed a “300% reduction”: in absenteeism until the DMPC explained to them that it was not indeed possible to reduce costs by more than 100%
• One epidemiologist on several occasions has also published statements saying that it is possible to reduce costs by more than 100%
• One disease management vendor publicly projecting "millions in savings" for an employer with fewer than 10,000 employees
• The Mississippi Medicaid DM RFP states: "[There is a] guaranteed...ROI of 1.05, or 100% of the program cost...plus at least a 5% net savings through reduced medical costs." This clause is completely self-contradictory, as when Casey Stengel told his Mets players to "line up alphabetically by height." 5% net savings implies an ROI of about 2:1. A 1.05:1 ROI equates to much less than 1% net savings.

Even actuaries are not immune. A major actuarial consulting firm, Lewin,  "estimated" $213/month in savings per asthmatic for The Blue Cross Federal Employees Program in Texas, an amount far more than the claims amount the average commercially insured asthmatic even spends on asthma.

The Nature of This Certification

A number of attributes distinguish this Certification program from typical accreditation:

1. The individual contract or (in the case of internally built disease management) program is certified rather than a vendor or health plan as a whole. A vendor may have valid metrics in some contracts and not in others. A health plan may be measuring ESRD validly but not asthma.
2. This is not a minimum standard. It is a high hurdle, and many payors don’t even try.  For instance, in Massachusetts, one payor has not even attempted to get DMPC certification even though both its competitors have it and use it in their marketing efforts.
3. Related to #2 above, it is a set of very specific correct ways to measure savings. These ways are summarized below and captured in the White Paper on Measurement, which (with this document) comprises the application package.
4. It is quite simple and inexpensive both in out-of-pocket costs and in internal costs. No on-site visit or massive data transfer is required. Results are known in days. As with formal accreditations, a payor which is denied Certification will be told why and what to do to achieve it. This information won't be public.
5. Though no longer a requirement, it is recommended that those seeking Certification take a half-day Critical Outcomes Report Analysis course and test. CORA courses and tests are often offered as pre-conference or post-conference workshops at conferences posted on the DMPC website, (Click Here) The reason for the recommendation of the course and quiz is that it isn't enough to measure correctly. One must be able to explain why a certain measurement is correct.

Corollary Program for Benefits Consultants

Today, few benefits consultants are able to advise their employers effectively in disease management savings measurement. Consequently they can't distinguish among programs or advise a decision to stay with a health plan vs. select a vendor independently, and if the latter, decide on a vendor

A benefits consultant — as an individual, not a firm — may receive Certification in Critical Outcomes Report Analysis by attending a workshop, reading the White Paper and submitting for Certification either an RFP, contract or critique of reported results which shows the understanding of what is valid and what is not.

What makes a measurement system valid?

It’s quite simple and adheres to the DMAA Outcomes Measurement Guidelines that one should “measure only what one can manage and impact.”  Since there is no epidemiological evidence that DM can routinely impact medical events other than those in the conditions being managed, valid measurement requires measuring events in the conditions being managed.  Event rates over time are then compared to the DMPC database.

Once the ROI using the event rates is estimated, DMPC looks at the actuarial pre-post measurement for consistency with that.

The four additional measurements which much be included in the actuarial measurement are as follows:

1. In addition to being inflation-adjusted the baseline must be regression-adjusted. There are several ways to do this, all described in the White Paper. No, just "measuring on the whole population" turns out not to qualify. It is proven in the White Paper that this is precisely the technique (in many diseases) which creates hidden regression to the mean; or
2. A true controlled study, wherein the study group and the control group-presumably matched for demographics already but we would review non-matched studies on a case-by-case basis — where both groups consist either of volunteers or total eligibles but not where the study group is volunteering (or on a case-by-case basis, not opting out) but the control group is passively matched.
3. Additional ways of measuring will be considered.

Note that "validity" does not equate to "accuracy." The goal of validity is to eliminate systematic bias. There will still be non-systematic bias. A valid result will be accurate on average, not every time. Even a program which earns this Certification can still be subject to non-systematic bias, but non-systematic bias (1) is logistically much harder to reduce and can't be eliminated and (2) understates results as often as it overstates results. Skeptics are much more concerned with the consistent overstatement in results due to systematic bias.

Rules and process for Certification

1. First Steps

In order to encourage as much participation as possible, the rules are simple, the process is expeditious and the price is low.

Request the White Paper ($500 of the $2000 is due at that point) and then decide if you'd like to proceed. You will probably decide that much needs to be done in order to qualify, and not proceed to request Certification. Or, you may decide to proceed to Certification. To seek Certification, you need to send the remaining $1500 of the fee and submit the relevant documentation (usually contracts and/or reporting-described in an accompanying document.)

2. If the measurement is deemed valid

If the measurement is deemed valid, a Certificate specifying the program and the specific measurement tool(s) which qualifies for Certification is sent within four weeks. When claiming Certification, you MUST specify WHICH measurement tools received Certification, as listed on the Certificate. The reason for this is obvious: It is quite possible to have one valid measure and one or more invalid measures concurrently.

Examples:

• Wrong: "We are Certified in Savings Measurement."
• Wrong: "We are Certified in Savings Measurement for common chronic disease."

Consistently overstating the extent of Certification will be grounds for de-Certification. It is critical to the integrity of the process (and to the integrity of other payors' Certifications that only the certified measures be associated with Certification).

Certification is valid through the length of the contract certified. Certification for internal programs is valid for two years.

3. If the measurement is deemed invalid

You will receive a 1-2-page description of the reason for the invalidity within two weeks. You may resubmit as often as you like and we provide "light" consulting as part of the $2000 to help you be successful.

Pricing

$2000 is the total cost per program or contract certified. ESRD, being comparatively simple, is $1000.

One reapplication following denial is free.

As mentioned, $500 is payable up front, to receive the White Paper. This White Paper costs $500 anyway and is a valuable document on its own. However, attendees at the certain ROI workshops may receive the White Paper at no additional cost.

The only other cost is sending relevant staff to the Critical Outcomes Report Analysis workshops or webinars.

Questions and Answers about the Certification Program for Disease Management Savings Measurement

Q: What if we don't want you to see our prices from the vendor when you are calculating an ROI?

One of the side benefits of Certification is that we can tell you, while we're at it, whether your prices are above or below market. Having said that, if you don't want us to see them, simply delete them before sending the contract, or send us the reporting documents alone, without the contract (though we may need to ask for more information).

Q: What if our vendor specifically says that Al Lewis can't see the contract?

That means you should have done a DMPC bid because the price, by definition, is above market. However, DMPC's associate, Ariel Linden Ph.D. (author of the DMAA's White Paper on Measurement) will handle any such Certification requests. No one else will be privy to the information.

Alternatively, you could send in just the reporting, but it is quite possible we will need to request more information.

Q: Are you also certifying the actual ROI?

No. Only the methodology is certified.   You may say, however, that your ROI is calculated using the DMPC-Certified methodology.   We can certify the ROI but that is not included in this certification.  It’s done case-by-case.