I've mentioned before how I largely recommend index funds, especially in 401k accounts. To explain why, let's start with Mutual funds.
Mutual funds allow you to invest small amounts of money in diversified portfolios. Each mutual fund has a pre-specified way that the money will be invested and the fund manager is responsible to pick the investments they believe will grow the most. In exchange for managing the funds, a percentage of the invested assets is paid to the fund manager each year.
If you want to evaluate the manager of your mutual fund, the easiest way to do so is to compare the performance of the fund to an index, like the S&P 500. The S&P 500 has been around longer than mutual funds and is used to gauge how well the economy in general is doing. At the most basic level, if the S&P 500 is up 10% and over the same time period your mutual fund is up 12%, the fund manager is outperforming. Naturally, then, you would always want to invest only in mutual funds that are outperforming the index, right? Therein lies the challenge. How do you know beforehand whether a fund manager will outperform the index over the next year? Mutual fund companies and brokers are quick to point out that "past performance is not an indicator of future results", but if we're being honest, everyone looks at past performance and gives it a lot of weight in their decision of whether to invest in a fund.
The true challenge is the fact that there are no mutual funds that have ALWAYS outperformed the index. It isn't hard to find a fund that beat the index last year, or even for the past few years, but there has never been a mutual fund that has beat their benchmark every single year.
Enter the index fund. Unlike actively managed mutual funds, Index funds are passively managed mutual fund that attempt to mimic an index rather than outperform the index. One of the largest and most well-known index funds is the Vangaurd 500 fund, which has been around since 1976. The theory behind an index fund is that if you can't predict what actively managed funds will outperform the market, you should just invest in something that will do what the index does. Additionally, because management of an index fund is simpler, the expenses to manage the fund are much lower than an actively managed fund, so investors are able to keep more of the return. A typical expense ratio for an actively managed fund could be around 1% per year, whereas an index fund is generally 0.10% or lower.
There are a few other reasons that I prefer index funds. First, time savings. As a DIY investor, I don't have the time or resources to do research on every available mutual fund out there to find the needle in the haystack fund that might outperform the index, then constantly be researching to make sure the funds I have selected are still the best ones to have. By owning index funds, I can focus my time on making sure I have the right percentage of my portfolio allocated to stocks vs bonds and not run the risk of having bad funds. In a 401k this is especially true since your company will generally only make a few funds available for you to invest in from the thousands of available funds and the criteria for selecting what funds to make available differs greatly between companies and is unlikely to match your own criteria. Most of the time, the mutual fund options I see in 401k lineups have higher than average fees and lower than average performance (the two go hand in hand).
The next reason I prefer index funds is for the cost savings. The expense ratio for most of my index funds is around 0.07% per year, compared to around 1% for a typical actively managed fund. If an actively managed fund were to invest the same as an index fund, because of their fees, the index fund would end up nearly 1% better each year. While it is certainly possible to have an actively managed fund that charges more fees and still outperforms the market, I would hate to have an actively managed fund that charged higher fees and still did worse than if I had just been in an index fund (which is actually what happens most of the time).
Lastly, I believe that most markets are efficient, especially those for large US company stocks.
What this basically means is that everyone has the same information about a stock and whenever new information becomes available, the stock price adjusts almost immediately to reflect the impact of that new information. This means that it is not possible to make outsized gains on stock based on research, since anything you could discover from publicly available information is available to everyone and has already been priced into the stock. I believe that exceptions to this rule exist in other markets where there are less market participants or less public information available, but for purposes of investing in the US stock market, I don't believe anyone can have a significant information advantage without breaking the law.