S&P 500 Returned 15.7% a Year Over the Last Five Years

S&P 500 Index Performance at a Glance: 2019–2024

The S&P 500 delivered a five-year annualized total return of approximately 15.7% from January 2019 through December 2023, well above its long-run historical average of roughly 10% annually. That headline number looks clean. For investors in the top federal bracket, the after-tax reality is meaningfully different, and understanding that gap is where the actual work begins.

The index represents approximately 500 of the largest U.S. publicly traded companies, covering roughly 80% of domestic market capitalization. According to S&P Dow Jones Indices, it delivered total returns of approximately 31.5% in 2019, 18.4% in 2020, 28.7% in 2021, -18.1% in 2022, and 26.3% in 2023. Those swings matter more at scale. A 18.1% drawdown on a $10M equity position is a $1.8M paper loss, and how you are positioned heading into that kind of year determines whether you recover on the index's timeline or your own.

What Has Been the Average Annual Return of the S&P 500 Over the Last 5 Years?

The average annual returns for the 2019–2023 period were exceptional by historical standards. The 15.7% annualized figure compares favorably to the index's long-run historical S&P 500 returns of approximately 10% per year since 1926.

But averages obscure the sequence. A $5M position in the S&P 500 at the start of 2019 grew to roughly $9.6M by end of 2023, assuming full reinvestment of dividends. The same position entered at the start of 2022 sat at approximately $4.1M by year-end before recovering. Sequence matters, especially for investors drawing distributions.

Sources: S&P Dow Jones Indices. Portfolio values assume full dividend reinvestment, no fees, and no tax drag.

The NBER's landmark study "The Rate of Return on Everything, 1870–2015" found that equities have delivered the highest long-run real returns of any major asset class, averaging approximately 7% annually in real terms across developed markets over 145 years. The 2019–2023 window ran significantly hotter than that baseline.

Rolling return analysis shows that five-year periods starting after significant drawdowns tend to outperform. That context matters when calibrating forward expectations.

How Did the S&P 500 Perform During the COVID-19 Crash and Recovery?

The pandemic drawdown was historically fast. Federal Reserve Bank of St. Louis data documents the S&P 500's intraday low of approximately 2,237 on March 23, 2020, representing a peak-to-trough decline of roughly 34% in just 33 days, the fastest bear market decline on record.

The recovery was equally unusual. Fiscal stimulus on an unprecedented scale, near-zero interest rates, and a rapid rotation into technology and digital infrastructure drove a full recovery by August 2020, less than five months after the low. The index closed 2020 up 18.4% for the full year.

Understanding the bear market dynamics of that period requires separating two distinct phases. The March 2020 collapse was a liquidity crisis and demand shock. The 2022 bear market (-18.1%) was an inflation and rate-repricing event. The causes were different, the affected sectors were different, and the recovery timelines were different. Treating them as equivalent data points in a five-year return calculation misses the structural distinction.

For high-net-worth investors, the 2020 recovery also created a significant tax-loss harvesting window. Those who held S&P 500 index funds through the drawdown without harvesting losses left meaningful after-tax value on the table. Direct indexing platforms would have allowed selective harvesting of individual constituent losses while maintaining market exposure, a structural advantage over standard ETF ownership.

What Sectors Drove S&P 500 Returns from 2019 to 2024?

Technology dominated. The technology sector's impact on index composition grew substantially over this period, with the sector representing approximately 29–31% of S&P 500 market cap by late 2024. Apple, Microsoft, Nvidia, Amazon, Alphabet, Meta, and Tesla collectively account for roughly 35% of the entire index.

That concentration has a direct implication for portfolio construction that generic financial journalism consistently misses: if you hold RSUs, employer stock, or private equity exposure in technology, a standard S&P 500 index fund amplifies your sector concentration rather than diversifying it.

Sector performance trends across the five-year window show a clear hierarchy:

Approximate figures. Sources: S&P Dow Jones Indices sector data.

The sector weight shifts over time tell a structural story. Energy's index weight fell from roughly 6% in 2018 to under 4% by 2024, while technology's share expanded materially. This is not diversification drift; it is the index mechanically reflecting market cap changes. Passive ownership means your allocation to each sector shifts continuously without any active decision on your part.

S&P 500 Index Performance Adjusted for Inflation

Nominal returns are the wrong benchmark for wealth preservation. The inflation-adjusted performance of the S&P 500 over the 2019–2023 window tells a more complicated story.

CPI inflation averaged roughly 4.1% annually over that five-year period, heavily weighted toward 2021 and 2022 when inflation peaked above 8%. The real returns adjusted for inflation reduce the 15.7% nominal annualized return to approximately 11% in real terms, still strong by historical standards but a meaningful reduction.

The 2022 year specifically illustrated the failure mode of conventional portfolio construction. The S&P 500 fell 18.1% while the Bloomberg U.S. Aggregate Bond Index fell approximately 13% in the same year, its worst performance in 40 years. A traditional 60/40 portfolio declined roughly 16% in 2022, its worst annual result since 2008.

That correlation breakdown matters. Bonds were supposed to provide ballast. In an inflationary shock, they did not. For FATFIRE investors relying on a 60/40 framework for capital preservation, 2022 was a stress test that exposed the model's assumptions. Real assets, private credit, commodities, and Treasury Inflation-Protected Securities all warrant consideration as complements to S&P 500 core holdings when the inflation regime shifts.

How Capital Gains Taxes Affect S&P 500 Returns for Top-Bracket Investors

This is where the retail-oriented analysis stops and the relevant analysis begins.

According to the IRS, long-term capital gains on S&P 500 index fund holdings sold after more than one year are taxed at 0%, 15%, or 20% depending on taxable income. High earners above $200,000 (single) or $250,000 (married filing jointly) also owe the 3.8% Net Investment Income Tax on investment income.

That means FATFIRE investors effectively pay 23.8% federal tax on long-term gains before state taxes. California adds another 13.3%. New York adds up to 10.9%. The after-tax drag is substantial.

Sources: IRS Topic No. 409 (2024). State rates approximate.

The practical implication: a 15.7% nominal annualized return compresses to approximately 9.8–10.5% after federal taxes for top-bracket investors, and lower still in high-tax states. Tax location strategy, holding index funds in tax-advantaged accounts versus taxable accounts, can add 50–150 basis points of after-tax return annually according to Vanguard research. That is not a rounding error on a $5M portfolio.

Direct Indexing: The S&P 500 Strategy Built for This Wealth Level

Standard S&P 500 ETFs are designed for accumulation-phase investors with taxable portfolios under $250,000. At the FATFIRE level, direct indexing changes the math.

Direct indexing platforms from Vanguard, Fidelity, Schwab, and Parametric allow investors to own individual S&P 500 constituents directly rather than through a fund wrapper. The structural advantage: you can harvest losses on individual underperforming stocks while maintaining overall market exposure, offsetting capital gains elsewhere in your portfolio.

The potential benefit is 1–2% in additional after-tax alpha annually, according to research from direct indexing providers. On a $5M taxable position, that is $50,000–$100,000 per year in tax savings that a standard VOO or SPY position cannot generate.

Minimum account sizes typically start at $250,000 for basic implementations and $500,000 or more for full customization. For investors with concentrated stock positions, RSUs, or sector-specific private equity, direct indexing also allows you to underweight specific holdings or sectors to reduce overlap, something no standard index fund can do.

This is not a marginal optimization. For a $5M+ taxable portfolio, the decision between a standard S&P 500 ETF and a direct indexing account is worth a dedicated conversation with your tax attorney and wealth manager.

S&P 500 vs. Private Equity: The Honest Comparison for UHNW Investors

The question of whether to hold S&P 500 index exposure or allocate to private equity is one that retail financial journalism cannot answer because it is not written for people who have access to both.

Cambridge Associates data shows that top-quartile U.S. private equity funds have historically outperformed the S&P 500 by 300–500 basis points annually over 10-year horizons. That is a meaningful spread. It is also the top quartile. Median private equity performance is closer to the S&P 500 after fees, and bottom-quartile funds underperform significantly.

The relevant considerations for a $5M+ portfolio:

Liquidity. S&P 500 index funds are liquid daily. Private equity capital is locked for 7–12 years in most fund structures. The liquidity premium is real, but it requires that you do not need the capital.

Fee drag. A standard S&P 500 ETF charges 3–5 basis points annually. Private equity management fees typically run 1.5–2% plus 20% carried interest. The gross return advantage needs to be substantial to survive that fee structure.

Access. Top-quartile private equity returns are not uniformly accessible. Allocation to the best managers requires relationships, track record, and often prior LP commitments. The performance dispersion between top and bottom quartile in private equity is far wider than in public equity.

Concentration. Private equity portfolios are inherently concentrated. A $5M allocation to a single fund may represent 20–30 portfolio companies, with significant sector and vintage-year risk.

The practical answer for most FATFIRE portfolios: S&P 500 index exposure serves as the liquid, low-cost core. Private equity allocations of 10–20% of investable assets make sense for investors with long time horizons, established manager relationships, and no near-term liquidity needs.

Should Ultra-High-Net-Worth Investors Hold S&P 500 Index Funds or Actively Managed Alternatives?

The evidence on active management is not ambiguous. Morningstar's Active/Passive Barometer shows that fewer than half of active U.S. large-blend funds survived and outperformed their passive index counterparts over any given 10-year period. Vanguard research consistently demonstrates that low-cost index funds tracking the S&P 500 outperform the majority of actively managed large-cap funds over rolling five- and ten-year periods after fees.

That said, the standard active vs. passive debate is framed for retail investors. At the FATFIRE level, the relevant alternatives are not actively managed mutual funds. They are factor-based strategies (value, quality, low volatility), direct indexing with customization, and private market allocations.

Factor tilts have mixed evidence. Value and quality factors have shown long-run premia, but the timing and magnitude are inconsistent enough that a plain S&P 500 index remains a defensible core position for most investors. The all-time record highs the index has repeatedly reached reinforce the cost of trying to time factor rotations.

The seasonal market patterns in S&P 500 returns are real but too small and inconsistent to build a trading strategy around. Institutional investors have largely arbitraged away the most predictable seasonal effects.

For the vast majority of FATFIRE portfolios, the S&P 500 index serves as the most cost-efficient, tax-efficient, and liquid core equity holding available. The optimization happens at the wrapper level (direct indexing vs. ETF), the tax location level, and the allocation level (what percentage sits alongside private equity, real assets, and fixed income), not in trying to beat the index with active stock selection.

Sequence-of-Returns Risk at Scale

Research published in the Journal of Financial Planning highlights that sequence-of-returns risk disproportionately affects large portfolios. A 3% withdrawal rate on a $5M portfolio represents $150,000 annually. A 4% withdrawal rate represents $200,000. If the first two years of retirement coincide with a drawdown like 2022, the portfolio's long-term sustainability changes materially, even if the 10-year average return is identical to a scenario with better early returns.

The 2022 bear market is instructive. An investor who retired January 1, 2022 with $10M in a 60/40 portfolio and withdrew $300,000 (3%) ended the year with approximately $8.1M, not $9.7M as a simple average-return calculation might suggest. The sequence, not just the average, determines outcomes.

Mitigation strategies at this wealth level include:

• Maintaining 2–3 years of distributions in short-duration fixed income or money market instruments, insulating the equity portfolio from forced selling during drawdowns.
• Using a liability-matching approach for known near-term expenses (property taxes, planned capital calls, charitable commitments) rather than drawing from the equity portfolio.
• Considering a bucket strategy where the S&P 500 allocation represents the long-duration growth bucket, with separate allocations for medium-term and near-term needs.

These are not novel concepts, but they are frequently underweighted in portfolio construction conversations that focus on expected returns rather than distribution sustainability.

References

• S&P Dow Jones Indices -- "S&P 500 Index Fact Sheet and Annual Returns Data" (2024)
• Federal Reserve Bank of St. Louis (FRED) -- "S&P 500 Index Historical Data" (2024)
• IRS -- "Topic No. 409: Capital Gains and Losses" (2024)
• Morningstar -- "U.S. Active/Passive Barometer Report" (2024)
• Vanguard -- "Vanguard's Principles for Investing Success" (2023)
• NBER (Jordà, Knoll, Kuvshinov, Schularick, and Taylor) -- "The Rate of Return on Everything, 1870–2015" (2019)
• Cambridge Associates -- "US Private Equity Index and Selected Benchmark Statistics" (2024)
• Journal of Financial Planning -- "Sequence-of-Returns Risk and Safe Withdrawal Rates for High-Net-Worth Retirees" (2022)