Hey, this is great. I’d been trying out various approaches to this with varying success, myself. I had started to get an inkling for using edge directions in some way, but hadn’t quite honed in yet. Since testing whether a point is inside a non-convex polygon requires triangulating then testing the point against each resulting triangle (all rather costly!), it struck me that using Euler’s equation:
v – e + s = 2
Where v is num verts, e is num edges, and s is num polygons, for any planar graph, might help. The total number of polygons should (in most cases) end up being one greater than that equation predicts. The polygon with the largest number of constituent vertices / edges should be the shell that we want to discard. The rest are valid. Thoughts on this? I haven’t gotten around to implementing and testing this, myself.