Is Your Data Scamming You? Measure Outcomes for True Success
These days, it seems like you can’t swing a digital stick without hitting three documentaries and a Netflix special about a fraudster. What is it about con artists that fascinate us? In the past few years, we’ve heard an awful lot about confident and unapologetic grifters– Billy McFarland and Fyre Fest, Elizabeth Holmes and Theranos, Simon Leviev of “Tinder Swindler” fame, and Anna Delvey or Sorokin (we still aren’t 100 percent sure of her real name). Their stories appeal to all of us, in part, because they both ask and answer an important question: Why can’t I just skip over the boring work and get straight to enjoying success? And the answer: Um, because you haven’t actually done anything.
Let’s be honest, if there was a way to achieve our goals without carefully working toward and monitoring progress, we’d all do it. That’s not a referendum on our character; that’s just part of being an intelligent primate. If you could snap your fingers and magically educate and provide opportunities for thousands, wouldn’t you do it? It can, therefore, be so tempting to only look at data that shows your institution in a successful light—tempting, but dangerous.
Does an institution have high student enrollment numbers? Well, Billy McFarland’s doomed Fyre Festival sold over 5,000 tickets within 24 hours. That’s a pretty impressive number, as long as you ignore everything else that happened next (500 people stranded without sufficient beds, food, water, or security). Billy McFarland is currently serving a six-year prison sentence for fraud.
Is an institution in great financial standing? Cool. So was Theranos. Elizabeth Holmes secured 945 million dollars from high-profile investors, but, as it turned out, that staggeringly high sum didn’t matter very much in the end, as her company’s product didn’t actually work. Theranos is now a literal joke and Holmes is likely facing prison time.
Does an institution have glowing student satisfaction ratings? Anna Delvey probably would have too. By most accounts, her friends found her magnetic and engaging—until she stole money from them. She successfully assumed the persona of a chic aristocrat to embed herself among New York’s elite, convincing many to foot the bill for her while she experienced “banking problems.” Success was fleeting, though. After serving a prison sentence for theft of services, Delvey is now being held by ICE while she awaits deportation.
Does an institution have great social media statistics? Simon Leviev, “The Tinder Swindler,” was pretty good at that, too. He marketed himself on social media/dating apps as a successful and legitimate businessman, made connections with people around the world, and then wielded his invented persona to romance and emotionally manipulate several women out of millions of dollars, with some losing their entire savings.
In each of these examples, the entities in question were able to parlay their one advantageous statistic into “success,” but only for a short time, and at great expense to other people. All of them were missing the key hallmark of legitimacy: a track record of providing value.
If you have ever questioned why measuring outcomes matters, consider the fact that outcomes are, arguably, the most crucial data in consistently demonstrating an institution’s quality and sustainability. Yes, outcomes provide just one piece of the puzzle, but they’re the piece that speaks to whether an institution is truly doing what it said it would do—i.e., achieving its mission and delivering quality academic programs.
If you are overseeing a pre-med program that receives thousands of applicants, great social media statistics, glowing student reports, and a billion-dollar grant, you should be ecstatic! But, if all your students go on to fail the MCATs, you’ve got a problem. And if it isn’t quickly addressed, or your success is going to be very short-lived. That’s not to say that these other statistics aren’t an important part of the picture—they are, but they can also be mercurial. Let’s say that our hypothetical pre-med program is fortunate enough to be profiled in US News & World Report (a periodical we feel some ambivalence toward). This type of press attention could dramatically affect enrollments, tuition dollars, interest from philanthropists, social media campaigns, and even student perception of the institution. Every single one of these statistics could skyrocket, but that numbers bump would be the result of external perspective, and not indicative of any change in the quality of academic programs. Some statistics are volatile. Outcomes should not be.
We have already written an entire blog post (see, I told you we were fascinated by fraudsters) about why good intentions are important but are not enough to make an effective institution. As I like to tell my kids when they promise me they’ll clean their rooms, “it’s the doing of the thing that matters.”
Monitoring outcomes is the key to not getting scammed by your own statistics. Your institution might have a lot going for it, but are you doing what you set out to do? Only outcomes can tell you that. Are your students truly benefiting from their investment? Are they completing programs and moving on to better, purpose-filled lives? If you only measure one thing (but, please, measure more than one thing), it should be that.
