In this episode of Kasper On BI we are talking about the history of Business Intelligence at Microsoft with one of the early founders and Technical Fellow at Microsoft Amir Netz.

AI Highlights from YouTube:

Introduction to the episode and guest, Amir Netz, a technical fellow at Microsoft and a founding father of Power BI.Discussion on the history of Business Intelligence (BI) at Microsoft, including the transition from OLAP to self-service BI with Power BI.Description of the early days of multidimensional engines and the challenges faced with OLAP systems.Explanation of how pre-computed aggregates were used in OLAP to handle queries efficiently.Introduction of innovations like the Vertipaq engine for in-memory computations, which was a significant breakthrough for BI tools.Discussion on the development of DAX for calculations and its importance in the Power BI ecosystem.Transition from Power Pivot in Excel to standalone Power BI, emphasizing visual-first experiences over grid-centric experiences.Explanation of why Power BI was taken out of Excel to allow for faster innovation and a better user experience.Insight into the integration of Power BI with other Microsoft products, including Excel and Teams.Reflection on the success of Power BI, attributing it to a combination of community engagement, rapid innovation, and strategic product positioning.Acknowledgment of the role of community in the growth and adoption of Power BI.Discussion on the future of Power BI, aiming for mass adoption similar to Microsoft Office products, promoting a data culture.Conclusion with aspirations for Power BI to become a fundamental tool in business and education, contributing to better decision-making globally.

Based on Amir Netz’s explanation in the transcript, here are the key points about VertiPaq/xVelocity:

Origin and Development:

Around 2008, the team was struggling to build a column store engine that would deliver the performance they neededThe breakthrough came when Amir read an article with the key insight that “data is not distributed uniformly in systems”Amir spent one sleepless night developing a new algorithm based on this insightThe initial code (about 2 million lines) was written by Christian Petrulis, who Amir describes as one of Microsoft’s top developers

Key Characteristics:

It’s a column store engine that works in-memory rather than on diskIt was revolutionary because it abandoned the traditional OLAP approach of pre-computing aggregationsInstead of pre-computing, it leverages in-memory processing for performanceIt has unique compression algorithms that work on data while it’s still compressedIt was designed for “smallish” to medium amounts of data that could fit in memory

Performance:

When they first tested it, the performance was so extraordinary that people thought it was fraudulentAmir mentions it’s still about “an order of magnitude faster than any other column store” even though it’s over a decade oldThe core algorithm has barely been touched in over a decade because it was so well optimizedAmir says it was “pure magic” and felt like it “defied the laws of physics” at the time

Historical Significance:

This technology was crucial for enabling self-service BI because it eliminated the need for long pre-computation timesIt allowed for interactive performance without requiring hours of processing like traditional OLAP systemsThis made it possible for business users to make changes and see results immediately, similar to working in Excel

Amir’s Personal Take:

He considers VertiPaq a “once-in-a-lifetime thing” in terms of technological breakthroughsHe says it’s the only time in his life where he felt they had created something “perfect” that couldn’t be improved uponThe algorithm came to him during one sleepless night, and he was so excited about it that he worked through the night without even getting dressedHe says they had to be “crafty” about introducing it to avoid alarming people about abandoning their existing technology

The development of VertiPaq represented a fundamental shift from disk-based, pre-computed OLAP system to interactive, in-memory analytics, which was crucial for the success of Power BI and modern self-service business intelligence.

Origin: VertiPaq
References:
Created 2025-01-28