FabCon Europe 2024 – A look at the future of Microsoft Fabric

Almost a year after Microsoft announced the general availability of its latest data platform, Microsoft Fabric, the first Microsoft Fabric conference on European soil took place from 24 to 27 September 2024. The European Microsoft Community Conference, or FabCon for short, attracted industry experts, developers and IT professionals from Europe and around the world to Stockholm. As part of the adesso team, I was also on site and right in the middle of the action.

With so many exciting talks, keynotes and workshops – over 130 in total – it was not that easy to decide which ones to attend. I deliberately picked out the cherries from the cake for myself – sessions that seemed particularly exciting and relevant to my work as a cloud and data architect. From deep insights into realised use cases to the latest developments in the feature roadmap, I took away a lot of exciting information. In this blog post, I would like to share some insights into the most interesting moments and key takeaways from FabCon.

Developer Workflows and CI/CD

Microsoft Fabric offers various approaches for this, depending on your needs and preferences. On the one hand, there is the option of creating a flexible, code-intensive experience with tools such as Git, Azure DevOps and the Fabric API. On the other hand, Fabric Deployment Pipelines enable a simpler, code-free approach. Currently, there are still some hurdles to using the Fabric API, in particular due to the non-consistently implemented Service Principal authentication. However, Microsoft has promised to solve this problem by the end of the year. Details of the API endpoints that currently support service principals can be found at the attached link. In addition, the Fabric Connections API, which enables the programmatic manipulation of data connections, will be available soon.

Configuration as Code

There is good news for Terraform fans: an official Terraform provider for Microsoft Fabric has been available in public preview since the end of September, which is an important addition for those working in Fabric DevOps with Terraform. Despite the noticeable interest from the community, deployments with ARM/Bicep are not yet on the roadmap.

Python in Fabric

An interesting new feature in Microsoft Fabric is the ability to use pure Python notebooks without the need for a Spark session to run in the background. This feature is particularly welcome as it makes it much easier to perform housekeeping tasks and to process frequent, data-poor jobs, for example, regular external API calls. It will be interesting to see how developers will divide their workloads between User Data Functions (Link) and Python Notebooks in the future to maximise efficiency.

High-concurrency Spark Sessions in Pipelines

Another important update to Microsoft Fabric, available since the end of September, concerns the use of high-concurrency Spark Sessions within Fabric Pipelines. This feature makes it possible to share Spark Sessions between multiple notebooks. Without this functionality, the number of notebooks that can be started simultaneously in a Fabric Pipeline is severely limited because a new session has to be started for each notebook. This update thus opens up new possibilities for more efficient resource management.

Experience-independent workspace copilot

In addition to the already familiar copilots, which are tailored to specific experiments, Microsoft Fabric is now introducing a workspace copilot. This tool is designed to help users quickly and efficiently find the elements relevant to their specific questions. The copilot uses Power BI search and is specifically designed to help business users identify the data sets relevant to their query.

Power BI Desktop Dark Mode

A small thing or essential – in any case, a long-awaited feature for many.

SQL Server / Dedicated SQL Pool Migration Tool

There is good news for anyone involved in a migration project to a fabric warehouse: Microsoft is currently developing a special migration tool to facilitate the migration of workloads from SQL Server or Synapse Dedicated SQL Pool to a fabric warehouse. The tool will be able to migrate the following elements:

• Schemas (including tables, views, stored procedures, and so on)
• Data
• Data connections

For those who want to learn more about the specifics of Fabric T-SQL, the following link to the ‘Fabric T-SQL surface area’ is very helpful. There you will find an overview of the features that are not yet supported by Fabric T-SQL.

Security and Permissions

In several sessions, Microsoft provided insights into the future Fabric permissions system. The key takeaways are:

• The primary security group for permissions on Fabric Items will be the workspace.
• Data permissions are primarily assigned at the Lake level and distributed downstream from there.

It is expected that the permissions concept will initially be in flux and will have a certain complexity, at least during the transition. This is partly because Fabric has to homogenise the permission systems of several separately developed engines, such as the Spark, SQL and KQL engines, and the semantic model.

The long-term goal, however, is clear: permissions in Fabric (including OneLake) should be at least as detailed and granular as in any other PaaS-based lake.

Fabric, networks and users

One of the security sessions highlighted an important aspect of Fabric's security philosophy, which is not limited to Fabric: the security of Fabric as a SaaS product is primarily achieved by securing Entra identities rather than primarily through network security measures.

Traditional network security through VNETs should be critically reviewed when it comes to integration into Fabric. Instead, it may make more sense to focus on securing user identities more strongly. This includes measures such as conditional access policies and user and entity behaviour analytics (UEBA), which enable a targeted and adaptive security strategy. These approaches often provide more effective protection in the context of modern cloud platforms and SaaS solutions.

Spark Native Execution Engine

The processing of Spark workloads in the fabric has seen an important development. These workloads can now be executed in the public preview on the Microsoft Spark Implementation, known as the Native Execution Engine (Link). This new implementation promises significant performance improvements. Performance increases of up to four times are possible. A key advantage of this engine is its flexibility: it can be activated or deactivated at the item level, allowing users to choose between performance and other factors as needed.

Conclusion

The latest developments and updates around Microsoft Fabric clearly show how the platform continues to adapt dynamically to modern IT and security requirements. From the integration of Python in Fabric Notebooks and the introduction of high-concurrency Spark Sessions to the new Spark Native Execution Engine, each innovation promises greater efficiency and usability for developers and businesses.

The introduction of the workspace copilot and the further development of the security guidelines are further examples of how Fabric aims to create a secure and effective working environment. I am excited to see the new developments in the coming months and look forward to the next FabCon 2025 in Vienna.

Would you like to learn more about exciting topics from the adesso world? Then take a look at our blog posts published so far.