I'm unclear what your second rule is. If you do "Any file", it matches everything so it doesn't make sense to have any other conditions with it. In general, though, to match empty folders, you'd have something like "Kind is Folder, Size is 0".
Also, is Hazel watching Folder X or watching the parent of Folder X? Hazel doesn't operate on the actual folder it is monitoring so you shouldn't worry about it throwing it away.