Due to increased government surveillance as well as data breaches, end-to-end encryption has recently received an increasing attention as a way to protect against such threats. End-to-end (E2E) encryption preserve the confidentiality of data on the wire as well as from service providers by performing encryption/decryption at clients keeping the keys strictly within client devices. There are many variants of end-to-end encryption schemes for different communication patterns. In this paper, we systematically analyze the security of these different variants against three types of passive adversaries and one type of active adversaries. We show that the security of some of these systems are broken under these threat models and what can be done to ensure confidentiality in such systems. We also analyze the existing products in the market and show what level of security they provide. Our study shows that most of the E2E encrypted systems are secure against only the weakest passive adversaries. Further, these systems are broken not by cryptanalysis of underlying cryptographic algorithms but by flawed system designs and security assumptions. Specifically we identify that unencrypted metadata and access patterns make these systems susceptible to inference attacks. We conclude with general design guidelines to securely build such systems.