Big Data vs. Small Data
What data actually drives CRE decisions?
I recently caught up with a data leader at a large commercial real estate tech firm. Our conversation opened with a wonderful chat about the opportunities in CRE tech, even in this down capital market. We brainstormed ways in which CRE firms could use data to drive decisions.
Then, all of a sudden, they confessed…
“Jen, I have to tell you something. I’ve been building analytics and algorithms using big data - the most incredible big datasets, let me tell you. And in all my years working for these large CRE firms, I have yet to see one of them use my research or analytics to make a decision. Sure, they use it to support a decision they already made, but I can’t seem to get them to use it preemptively. Has this been your experience too?”
I was floored by their confession.
At the same time, I was not surprised.
Unpopular Opinion: Most CRE Big Data Is Not Actionable
I expect to get a lot of flack for this unpopular opinion. Before you dismiss me, let me offer a few counterarguments.
Big data absolutely has its place in CRE, primarily in augmenting or replacing workflow. As we’ve seen with ChatGPT and other LLMs, big data eliminates much drudgery around documentation. Learnings from large swaths of unstructured and structured data have also proven quite powerful. For example, we can now automate tasks like invoice processing. We can also improve building efficiency within HVAC or other building systems. These automations do not involve second-order or higher thinking.
Many investors, lenders, bosses, or partners require big data to supplement decisions. They want to see evidence of the validity of CRE firms’ assertions.
Big data has encouraged CRE firms to improve their own data capture, normalization, and retention. I am beyond thrilled with this side effect of big data.
The problem with big data in CRE isn’t in its capability; it’s how we think we need to use it versus how it’s actually impactful.
Most CRE firms spend a ton of money on third-party big data and research. Any leftovers *might* get allocated towards small data. Typically, capturing small data is a byproduct of automating another process. An afterthought, really.
Property management systems are a great example. With Yardi, RealPage, etc., property management firms significantly improve their bookkeeping while also capturing data they use to optimize rent. Hospitality revenue management systems are in the same boat. That said, few other owners, developers, or investors actively track, clean, and analyze their own data.
Rather, CRE firms buy big data by the bushel. We think we need big data primarily because of confirmation bias, which I described in a previous post and excerpted for you here:
This belief in accurate predictions stems from confirmation bias, which is defined as:
“the tendency people have to embrace information that supports their beliefs and reject information that contradicts them”
Our innate human nature desires clear right and wrong answers to every problem. That is why we seek information to back up, aka confirm, our stances.
Most decision-makers at CRE firms have predisposed ideas about macroeconomic trends. They only want third-party support for their decision. Most do not start with a hypothesis and test it, like we were taught in our 6th grade science class. Instead, CRE firms follow the order of 1) coming up with a thesis and then 2) massaging the numbers or finding research to support said thesis.
This is the equivalent of poking dead frogs with no agenda. Gross.
Even if we could use big data to automate higher level decisions, would we? We’re comfortable using big data to automate mundane decisions or improve workflows. But would you ever truly use big data to automate your multi-million dollar (if not more $$$) decisions?
Our egos, fear of the unknown with AI, or even self-preservation as humans will prove this scenario unlikely. The only way this could work is via successive low contrast effects over a long period of time. In other words, we wouldn’t see it coming because each incremental enhancement would be perceived as low impact.
Microeconomics vs. Macroeconomics
Most CRE firms make their most profitable decisions to invest, manage, develop, etc. at the microeconomic level.
In fact, most pivotal CRE decisions are driven by local laws. For instance, many developers specialize in certain locales. Each area has its own complexities with local development requirements. Architects often specialize as well and focus on similar topographical areas.
Real estate investors may acquire opportunistic properties because they know a new, large employer will come to the area. Many underwrite in excruciating detail, down to specific operating expense line items. (Whether they test the accuracy of their assumptions later is up for debate… and in my experience, rare. Becoming more accurate in your underwriting is yet another reason to focus on useful microeconomic data.)
Many real estate firms also make choices based on cultural influences. Managing apartments on the Texas-Mexico border has a great many differences from the Midwest.
Yet, CRE firms still spend beaucoup of money gathering big data and on macroeconomic research. Instead, they should minimize their big data, macro spend. They should focus on accumulating and analyzing their microeconomic, small data goldmine. Many already have one just waiting to be untapped.
To quote Charlie Munger,
“[Most people place] too much emphasis on macroeconomics and not enough on microeconomics. People are often wrong on macroeconomics because of extreme complexity in the system they wish to understand.”
If you’re managing a massive, trillion+ dollar portfolio, you likely have the luxury of big data scientists and analysts on staff. Some wise CRE decision-makers will employ these individuals and data to augment their strategy. The wisest will keep in mind that black swans could be just around the corner. They also know that human nature is not rational and therefore, not always predictable with data. As John Maynard Keynes once said, “The market can stay irrational longer than you can stay solvent.”
The complexity of macroeconomics and big data makes using it to drive decisions difficult and wan for accuracy. I’m not suggesting that CRE firms avoid macroeconomics and understanding big data entirely. But they should spend more time on microeconomics, aka small data, and less on macroeconomics, aka big data. This is inverse to how most real estate firms allocate their tech budget.
Fortunately for growing CRE firms, anyone can gain insights from small data. And the smaller the dataset, the easier it is.
The competitive advantages of small data for higher-level decisions outweigh those of big data in a landslide.
Munger also said, “Micro-economics is what we do and macro-economics is what we put up with.”
CRE firms will always be required to pay for some big data to support their decisions. As such, I offer you this adapted mantra to use going forward, “For CRE decision-making, small data is what we do and big data is what we put up with.”
Jen’s Startup of the Week
I first encountered Banner years ago when evaluating software options for a client, and I must say, I was impressed. Their competitors led the development software market at the time, but Banner quickly gained ground. The Banner team focused on core solutions that CRE developers needed. Through their specificity, they were able to offer similar features at a more affordable price. Now, they are more fully featured and a worthy competitor in the space.
As a quick disclaimer, Banner is not endorsing this newsletter or All About CRE.
Jen’s Reading Corner
One of my readers recently asked what books I read every year. One of the first that came to mind was Edward Tufte’s The Visual Display of Quantitative Information. I first read it about an decade ago, and I reread it often, as much as every year. It never disappoints.
If you like All About CRE and want to support my work:
📧 Forward this to a friend and invite them to subscribe at jentindle.substack.com
☕Buy me a coffee
🪄 Book a one-on-one CRE tech coaching session
😍 Plan a media partnership