A/B tests (or randomized controlled experiments) play an integral role in the research and development cycles of technology companies. Through real examples of Microsoft, we show that it is difficult to assess the impact of new ideas and why we should A/B testing. But these days the size of data is increasing rapidly, and it can be hard to do A/B testing at scale. We look at scalability issue from real examples and how to solve it. We also show various pitfalls and mistakes that people can experience from experimentations. Finally, we look at statistical theory and methods to improve sensitivity of metrics to have better A/B testing.