Dimensional’s Co-CEO and Chief Investment Officer answers questions about investment returns, benchmarks, and evaluating managers.

Investors often start the year by evaluating how their portfolios have performed. Gerard O’Reilly recently sat down with Scott Mardy, a Vice President and Investment Strategist with the firm, to talk about what investors should consider when evaluating investment performance.

Key Takeaways

• Your performance evaluation framework should answer a simple question: Has your money manager delivered what they committed to deliver?
• The point of analyzing performance data is to help investors make informed investment decisions. The noisier the data, the weaker the inferences you can make.
• If you’re going to invest time understanding how a manager operates and potentially commit assets to them, you want a manager who is as committed to the long term as you are.
• Over short time periods, outperforming or underperforming a benchmark is not necessarily evidence that a manager failed to deliver what they said they would deliver.
• There’s no magic time frame for considering returns—different time frames provide different information.

Scott Mardy: What objective should investors have in mind when they evaluate the performance of an investment strategy?

Gerard O’Reilly: The main objective of analyzing performance data should be to make better-informed investment decisions. Whether the investor is deciding how to allocate their portfolio, monitoring their progress toward an investment goal, or evaluating managers to implement their desired asset allocation, it is important to have the right evaluation framework given the objective.

When selecting and monitoring managers, investors must find a manager that can deliver a particular strategy. A good performance evaluation framework should provide evidence to answer an important question: Has the money manager delivered what they committed to deliver? If the manager has, and the investor’s asset allocation remains appropriate, it is unlikely that the investor needs to replace an existing strategy with another, regardless of whether performance has been strong or disappointing.

Q: Is there a framework you think works well for evaluating whether a manager delivered what they said they would?

A: There are at least four steps I think are important: Investors should (1) clearly understand a strategy’s investment objectives, (2) be familiar with the investment approach the manager will use to meet those objectives and why that approach makes sense, (3) have realistic expectations about relative performance in different market environments, and (4) know what appropriate tools, analysis, and resources are available to monitor and explain returns.

In my opinion, these four steps are most useful when applied to systematic strategies, like the strategies Dimensional manages. When a manager uses a systematic approach, they should be able to describe accurately and concisely what the strategy objectives are as well as the investment processes used to achieve those objectives. This, in turn, implies that expectations about relative performance in different market environments can be effectively communicated. And verifying that those expectations have been met can be more accurately monitored.

For example, suppose small cap stocks underperform the broad market over a certain period. The expectation should be that a well-managed systematic small cap strategy will also underperform the market over that period. While underperformance is a disappointing outcome for those invested in the strategy, an investor can still draw confidence that the manager did what they said they would if the manager can demonstrate that they remained invested according to their mandate, controlled what they could control, and added value where they could.

Realized returns are noisy. Strategies that allow you to learn something useful from those realized returns (whether the performance period is short or long) can help you stay the course and become a long-term investor. This is another one of the potential benefits of a consistent, systematic strategy for investors; performance conversations for such strategies can and should enable investors to understand whether a manager delivered what they committed to deliver.

Q: Can you elaborate on what it means for returns to be “noisy”?

A: Market prices reflect the aggregate expectations of investors. Changing expectations, new news, previously unknown events, and so on can all change prices. When I refer to “noisy” returns, what I mean is that from day to day, prices can and do change in unpredictable ways. The amount of noise in the data is related to the magnitude of the price swings.

The purpose of analyzing performance data should be to learn something that helps investors make informed investment decisions. But the more noise in the data, the weaker the inferences investors can make using the data.

Q: What can an investor do to make stronger inferences from the data?

A: A good rule of thumb when dealing with noisy data is: the more observations you have, the stronger the inferences you can make. Generally, you can increase the number of observations in two ways. The first is to use longer time series. When forming expectations about the future, 240 months of returns are likely to provide more useful information than 12 months of returns.

The second is to look across the full range of strategies a manager offers. This is particularly useful for a manager that applies the same investment philosophy and a similar investment process across a wide range of strategies. For example, suppose an investor is interested in a systematic US large cap strategy. If the manager also has systematic strategies in small cap, non-US developed large and small cap, and emerging markets large and small cap, etc., the investor can use the returns of these strategies as additional observations of the manager’s approach to systematic investing. These additional observations, while not specifically for US large caps, can be useful in assessing the efficacy of the manager’s investment approach.

The full range of these different strategy returns over long time horizons is something we refer to at Dimensional as a manager’s “body of work”. It has been my experience that when investors view our body of work, it really helps them understand how consistent our investment philosophy is and that we have been able to deliver for investors across a wide range of asset categories and market environments.

Q: Are there any other benefits to looking at the full range of strategies offered by a manager?

A: Yes, I believe so. One additional benefit is a better understanding of how consistent that manager’s investment philosophy is and how committed they are to a way of managing money. Unfortunately, fund closures in the investment industry are common. For example, almost half of the US-domiciled equity and fixed income funds that were available 15 years ago are not available today.¹ But investment horizons can be much longer than 15 years.

For example, when investing for retirement an investor may spend thirty or more years accumulating assets, and thirty or more years consuming those assets. If you’re going to invest your time and effort to understand how a manager operates and then commit your clients’ assets to a strategy, you want to be confident the manager is committed to the long term just as you are. Long, consistent track records across a diverse range of strategies can help build that confidence.

Q: In the four-step framework you outlined, what role does a benchmark play?

A: The main roles a benchmark plays are to provide an independent measure of market returns and to act as a baseline to help investors understand the impact of a manager's decisions on strategy returns. To that end, it is generally the case that benchmarks should be widely available, well known to investors, and provide similar exposures to that of the strategy it is being compared with.

The criteria that the benchmark’s returns be widely available and well known provides investors with an easily verifiable and independent data source to help monitor a strategy’s performance. The criteria that the benchmark provides similar exposures to the strategy helps control some of the noise inherent in returns. For example, if a strategy invests only in emerging markets stocks, most managers would use a commercial emerging markets index as a benchmark. By observing the benchmark’s returns, an investor will have an independent baseline understanding of the performance of a broad basket of emerging markets stocks.

Q: How should investors think about the different types of benchmarks available within the same asset class, like small-cap stocks for instance?

A: When thinking about this question, it is important to start by acknowledging that the index rules, which govern a benchmark, are arbitrary. What I mean is that each index provider makes decisions on what stocks to include and when to rebalance. Those decisions can be subjective. In the case of small-cap stocks, the index provider chooses the maximum market capitalization (shares outstanding multiplied by price) of the stocks to include in the index. There is no single definition of the maximum market capitalization for a small cap strategy that is better than all other definitions. The index provider also chooses the days of the year on which the index will rebalance. Of course, there is no one day of the year that is superior to all others to rebalance.

I point this out because comparing the returns of a strategy to a benchmark index is helpful when explaining returns, but there may be very good reasons why a manager’s returns are different than indices. A simple way to illustrate this is by looking at the returns of different indices that include similar stocks. Both the Russell 2000 Index and the S&P 600 Index are US small-cap stock indices, but as of August 2018, the difference in the prior 12-month return between these indices was over 7%. How can something like this happen? Because of differences in the subjective rules, Russell and S&P apply to define their indices. It would be challenging to argue that one index is superior to the other, but they can certainly deliver very different returns over quarterly, annual, or longer performance periods.

What this means for investors is that out- or underperformance relative to one or multiple benchmarks over a short measurement period (or even a longer period) is not by itself evidence the manager did or did not deliver what they said they would.

Q: Can you elaborate on why a strategy’s return might deviate from its benchmark index?

A: Sure. An index provider must make decisions on what securities to include in the index and how often to refresh that list of securities and rebalance the index. For example, if a strategy is “deeper value” than its benchmark value index, that will lead to differences in returns over short and long investment horizons. On expectation, when deeper value stocks do better, the strategy will outperform the index, and vice versa.

Q: What about differences in rebalancing approaches?

A: I believe that a manager should look for opportunities to spread a strategy’s turnover across all the trading days of the year. This can provide many benefits to investors. Market prices contain information about expected returns. By initiating a small portion of strategy turnover every day, a manager can use the up-to-date information in market prices when deciding what to buy or sell, which can help keep a strategy continually focused on higher expected return stocks. Further, the manager can use cash flows from corporate actions (like stock dividends) and cash flows from investors in an efficient manner to rebalance the strategy. This can reduce costs. The manager can also use a flexible approach to trading to lower trading costs. The potential benefits of a daily process are many, and I believe these benefits can lead to higher returns for investors.

Performance periods in which differences in the timing of rebalancing trades lead to underperformance should not be interpreted as the manager “got it wrong.” I believe making the decision to spread turnover across all the trading days of the year is logical and good for investors. A period of underperformance like this does not mean a manager should change their approach. On expectations, it was the right approach to take during the performance period and the right approach going forward.

That is why care should be taken in drawing inferences from returns over a short time period. One goal of performance analysis is to provide information that can help investors make informed investment decisions. Looking at the relative returns of a strategy over a short time period may be useful to further understand the processes used to manage a strategy, but I don’t believe the level of out- or under-performance taken in isolation is likely to yield useful information to make better investment decisions.

Q: You’ve mentioned the long term and short term. What is the right time frame over which to measure performance?

A: I don’t think there is one right time frame over which to look at returns. I do think it is important to recognize that different time frames provide different types of information.

As I mentioned earlier, when analyzing noisy data, having more observations generally leads to better inferences. That implies if investors are looking for reliable evidence of a manager’s ability to deliver higher-than-market returns, the longer the period the better. When I say long, I mean that 15 to 20 years is a good starting point, but longer than that is also good. This allows investors to observe how a manager performed in different market environments, as well as how they adapted to changes in regulation, market micro-structure, and accounting practices. The future is uncertain. Meaningful live track records can be evidence that a manager has the skill and investment processes to deal with that uncertainty in the real world, in real time, and deliver good investment outcomes to their clients.

Q: What about strategies that don’t have the benefit of a long track record?

A: If the manager applies the same investment philosophy and a similar investment process across a wide range of strategies, that can also help investors assess strategies the manager offers that have shorter track records.

Q: What about quarterly, annual, and three- or five-year performance periods?

A: Such periods can still provide useful information—specifically, information about the possible range of outcomes a strategy is likely to deliver relative to its benchmark. Understanding volatility is key to becoming a long-term investor. Short-term performance analysis helps investors set volatility expectations. With the right expectations, there should be fewer surprises. That translates into investors having an easier time staying disciplined.

Quarterly or annual performance reviews are also good ways for investors to check in with people managing their money. It is often the case they may learn something new about the investment approach. I have found that a useful way to facilitate this learning is to begin a quarterly or annual performance meeting with a review of long-term results and an overview of the investment process. Then, address returns over the last quarter or year. Lastly, finish with updates on recent enhancements to the investment process. As mentioned, managers need to be able to adapt to an evolving world and continue to push their knowledge of investing forward. Performance reviews can be a good time to share that knowledge.

Q: Do you have any final thoughts?

A: At the end of the day, investing is about trust. Monitoring and assessing performance can help confirm that what a manager or financial advisor is doing is sensible and that they’re working day in and day out to do what they said they would do. Hopefully, this helps build trust over time and enables truly long-term behavior. What’s that mean? Being able to live with returns when they’re disappointed and when they're strong, and not making rash decisions in either of those scenarios. If you can do that—all the details and nuance set aside—we believe you’ve put yourself in a better position to have good investment experience.

1. Source: “Mutual Fund Landscape 2018”, Dimensional Fund Advisors. US-domiciled open-end mutual fund data is from Morningstar and Center for Research in Security Prices (CRSP) from the University of Chicago. Index funds and fund-of-funds are excluded from the sample. See report for more detail, including Morningstar categories included in the fund samples. Past performance is no guarantee of future results.