Cthulhu and Sanity

With the inclusion of Insmouth in Ravenloft: The Horrors Within, why aren't D&D 5.5e rules for Sanity and Madness?

You might consider looking at the "Fear and Mental Stress" stuff in the 5.5e Dungeon Master's Guide.

There's also nothing stopping you from using the same optional rules for this that appeared in 5e.

In the 2014 Dungeon Master's Guide, there's a section on Madness that might be helpful if you have access. Also in Van Richten's Guide to Ravenloft there is a section about Fear and Stress.

In the Candela Obscura manual, there's a good section about sensitivity when it comes to using things like "sanity" and "madness" in games like this. People have real-world mental health issues, phobias, etc, and while creating or using rules to simulate some kind of "sanity bar" mechanic can sound interesting for a more horror-based or phycological-based campaign, it's important to, if you are going to do it, do it with respect to the actual real-life issues people face, and not turn it into some trope-ridden perversion of it.