The expression 'there's a worm in it' means that something is not really going well, that something is stuck somewhere, is not as it should or could be - a muddled situation. With hazelnuts, you may have noticed that there are sometimes very small holes in the hard shell and that the nut kernel is infested with a worm. This is the hazelnut borer. It is a small weevil that lays its eggs in the unripe fruit. The larvae hollow out the nut and leave it through the visible small borehole to spin themselves into a burrow in the ground and survive the winter. The attack does not come from the outside, but from the inside. In other words, a messy situation, whether in a private or business environment, is not necessarily caused by external attacks, but comes from within, because unpleasant things have been able to enter much earlier and settle in unnoticed.
I wish you an extraordinary day!