Hi,

Greetings from Vilnius!

It's fantastic to have you onboard and let's learn about modern data solutions together.

I am testing a new format - my latest findings in data ecosystem. Let me know what you think.

Important: Mastermind #2 session about DataOps next Thursday (May 13). Sign up here or reply directly to me.

Summary in 60 seconds

Data mesh and data products

Data as a product is one of the 4 core Data Mesh architecture principles. A successful data product is Usable, Valuable and Feasible.

Data governance social engineering

Until people don’t see benefits of data governance, it will be perceived as a burden that no one wants to do. Focus on showcasing value proposition that you get with proper governance in place. 

PowerBI + AI

PowerBI is much more than standard reporting tool. There is a number of AI features: AI-Augmented Visuals, data enrichment, quick insights, Auto ML or running Python and R custom scripts.

Delta Lake keeps evolving

Databricks works hard on new features; private preview curtain hides brilliant functionalities like Delta Live Tables, managed catalog or workflow management. The latest public release is Databricks Delta Change Feed to track row-level changes of a Delta table.

Latest Azure data updates

• Start your Data Factory data flows execution in less than 5 seconds instead of 1-2 minutes
• CI/CD integration to automate Azure Synapse deployment article shows how to configure Azure DevOps repos and releases to propagate your code to environments
• Databricks Delta Change Feed represents row-level changes between versions of a Delta table. When enabled on a Delta table, the runtime records “change events” for all the data written into the table.
• Azure Data Factory in Norway and other regions
• Azure Machine Learning VS Code Integration
• Azure Purview updates to improve resource set experience

Quiz 

When was the Business Intelligence term first mentioned?

(don't worry if you don't know, an answer is at the bottom of this email)

Data as a product

The 4 core principles of data mesh architecture are:

• Domain Driven Data Ownership Architecture
• Data as a Product
• Self-Service Infrastructure as a Platform
• Federated Computational Governance

Data product is a representation of an analytical data that one domain has produced and is serving to the other domains. Eventually each domain can use each others data to create new aggregations and projections. 

From Data as an asset to Data as a product

From Byproduct to Product

From Data as an output of code to Data and code as one unit

From Guilty until proven innocent to Trust but verify

The whole recording with Zhamak Dehghani is so PACKED WITH VALUABLE INFORMATION - it's better you watch it and take notes yourself :)

PowerBI and AI features

PowerBI is much more than standard reporting tool. PowerBI Premium aims to combine Azure Analysis Services capabilities under one roof. Also, there is a number of AI features:

• AI-Augmented Visuals (e.g. decomposition tree, clustering, analyze, anomaly detection, forecasting)
• AI-Powered Visuals (e.g. smart narrative, Q&A, key influencers)
• Data Exploration & Editing (e.g. generate data insights, Power Query AI features, table detections from JSON/CSV files) 
• Data Enrichment (e.g. text analytics, computer vision)
• Running Python and R language
• Azure ML integration
• Auto ML built-in PowerBI

Read more about PowerBI features here and here. 

Watch interview with Rafal Lukawiecki about AI, Machine Learning and Power BI

My lessons from the mastermind meetup

Visualize benefits

Until people don’t see benefits of governance, it will be perceived as a burden that no one wants to do. Focus on showcasing value proposition that you get with proper governance in place. Remember: documenting measures & data is as important as the data itself

Build momentum

It’s better to build momentum, spark interest in data catalog and value of metadata before buying an expensive COTS offering. Wikipedia page with described data definitions and terms is a great place to start.

Community first

Even the best data catalog product is useless if users don’t contribute. Get your community excited about it. Make the processes of documenting data “sexy”. Enable crowdsourcing, award users for contributions with gamification.

Read more on Data Platform Mastermind

People to follow

• Simon Whiteley and Advancing Analytics YouTube channel - if you work with Azure and/or Databricks, Simon's videos are simply one of the best learning resources.
• Abhishek Choudhary and his GitHub is a deep dive into data engineering topics, focused on open-source solutions.
• Adam Votava and his "5 minutes for 5 hours’ worth of reading" series is an awesome read to help expand knowledge beyond Azure data ecosystem.

Books, articles, videos

• Data Pipelines Pocket Reference by James Densmore - if you are entry level data engineer or you have junior engineers in your team, working with AWS, go for it.
• Azure Cloud Native Architecture Mapbook - great read for all Azure admins, infrastructure responsible, or just simply want to know more details about application development in Azure.
• Reverse ETL summary by James Serra - is there anything special about moving data from your data platform back into operational systems? On the first glimpse, it's just another ETL pipeline. But there is a rise of products specializing in reverse ETL. Take a look.

Quiz answer: In 1958, IBM Researcher Hans Peter Luhn publishes "A Business Intelligence System." Hans is later named the Father of Business Intelligence.

Let me know if you find this structure informative. Should I continue sharing information like this?

Valdas Maksimavičius

IT Architect & Microsoft Data Platform MVP

https://www.dataplatformschool.com 

Vilnius
Lithuania

This email was sent to | Unsubscribe | Forward this email to a friend