Antenna subset modulation (ASM) is a physical layer security technique that is well suited for millimeter wave communication systems. The key idea is to vary the radiation pattern at the symbol rate by selecting one from a subset of patterns with a similar main lobe and different side lobes. This paper shows that ASM is not robust to an eavesdropper that makes multiple simultaneous measurements at multiple angles. The measurements are combined and used to formulate an estimation problem to undo the effects of the side lobe randomization. Simulations show the performance of the estimation algorithms and how the eavesdropper can effectively recover the information if the signal-to-noise ratio exceeds a certain threshold. Using fewer active radio frequency chains makes it harder for the attacker to recover the transmit symbol, at the expense of more grating lobes.