Monday, March 21, 2016

Building & Utilizing Data for the Future

As part of the fourth and final post in the series "Enabling Data Driven Organizations", the focus of this post is about how to make an "engine" of both data and people to make your organization data driven.

Now that you have the inventory of your data environment, you understand the value, and have influenced changed in how people work, its now time to have an engine that can run and support future needs of the organization.  What I mean by an engine, is by collecting the inventory and building a product around it, people throughout the organization now will have innovative ideas.  While making enhancements to your product will help the organization, part of being a good product is enabling others to build upon what you already have.  This changes the product to a platform.  

  • A Product is a unique tool with a specific design/purpose.  There are use cases and scope, users, and ultimately an end to its development lifecycle.
  • A Platform is a series of products and API's that not only enable the use cases and scope, but also allow others to utilize that information/content to expand into other use cases and purposes, far beyond the scope what the original product team may come up with.
To enable to this, you need to enable systematic communication both for receiving content and sending out content.  You may ask, why this is needed for both incoming and outgoing content.
  • With incoming content, others within the organization may want to provide additional information to the platform like SLA, data quality, comments, and more.  Allowing this to be systematic enables other products and other teams to ultimately work off of a single platform.
  • With outgoing content, others within the organization may want to consume inventory counts, relationships, and more to provide a custom / unique experience to their user base.


When speaking to the context of the data inventory, building that engine (or platform) which allows others within the organization to expand on data driven tools/products.  The analysts may build notifications on new data sets, the data engineers may build SLA reports on their data pipelines, the data scientists may build dynamic models/relationships, the system owners may build dynamic notifications on relationships between systems, and the data consumers may build metadata management automation

This is where products become platforms (the boys to the men) and what a single product team started morphs into what an entire organization utilizes and drives their day to day work on going forward.  This is where you not only influence but also enable the rest of the organization to be "data driven"

Monday, March 14, 2016

Influencing Change Using Data

As part of the third post in the series "Enabling Data Driven Organizations", the focus of this post is about how to influence change within your "data driven" organization.  In the previous posts in this series we focused on exposing the current inventory of data objects and then understanding the value that provides.  Now that we know the value of the inventory, we need to influence change in how the rest of the organization utilizes this information.

First, I want to focus on who within the organization can utilize the value of knowing the data inventory:


Data Engineer

On a daily basis, Data Engineers are responsible for the flow of data and "building data".  As part of this responsibility, one of the challenges they encounter is understanding where data is sourced from.  A Data Discovery Search and Data Lineage are 2 values that will change how they work.

Data Analyst

On a daily basis, Data Analysts are responsible for utilizing data to present via reporting or standard analysis.  They need to quickly identify if this analysis or report already exists.  A Data Discovery Search and simple inventory reports will change how they work.


Data Scientist

On a daily basis, Data Scientists are responsible for finding new ways to use data and provide complex analysis.  Like Data Engineers, a Data Discovery Search and Data Lineage are 2 values that will change how they work.



System Owners

On a daily basis, System Owners are responsible for maintaining a reliable system like a database or reporting system.  With that responsibility, they need to understand what is used (and not used) as well as impact analysis (downstream) to other systems.  Data Linage (downstream) and Inventory Reporting are 2 values that will change how they work.
Data Consumer

On a daily basis, many individuals in your company are data consumers.  They can be executives which review dashboards or your business analyst tracking their progress.  With companies having multiple systems, a Data Discovery Search across platforms will change how they work.



Now that we have identified who can benefit from exposed data inventory, the next question is how to influence changing how they work.  This can be approached in multiple ways and can be a touchy subject.  How I would suggest to approach this is:

  • Advertise Capabilities - One of the biggest barriers to change is being unaware.  Informed people within the organization often will utilize tools that make them more productive.  If they are more productive than their peer, this rivalry will drive efficiencies.  This can be done through written communication or presentations.
  • Celebrate Success Stories - When a product is built, it is often built with use cases in mind.  Whether these are planned use cases or not, the early adopters that find success should be celebrated and communicated.  Similar to advertising capabilities, this showcases to the rest of the organization the value that people are already utilizing the product.
  • Encourage Change - As you see opportunities of how work has "always been done", offer your wisdom, information, and data to help solve these problems.  This could be decommissioning a table and understanding the impact.  This could be cleaning up a system by understanding what is used vs. not.  Offering the information to help solve problems faster, word will spread and people will follow in the future. 
  • Other Communication Channels - Find the other forms of communication channels that catch attention of the people within the organization.  This could be leadership communication, mass email blasting, flyers, or whatever else that may effective.

This step in making a "data driven organization", will be a challenging one but one that can gain traction quickly if done right.  Identifying who can gain value and educating them through the process will influence the desired change.  

Sunday, March 6, 2016

Understanding the Value of Data Inventory

As part of the second post in the series "Enabling Data Driven Organizations", the focus of this post is about understanding the value of the data inventory.  While hinted in the previous post, the exposure of the data inventory does drive motivation to complete metadata, the value of this information has is so much more.

Once you have the four major object types (reports, data sources, data movement jobs, and business terms), there is so much value that this unlocks.  

  • Clean up Environment - being able to understand the entire inventory of reports, tables and more across multiple systems, this allows you to get a global picture and clean up unused, duplicated, or un-needed content.  This helps ease strain on systems as well as just good overall clean up.
  • Decrease Discovery Process - being able to search and discover tables for content that is new to you, will decrease the time to understand of where data is and increase speed to deliver.
  • Reduce Duplicate Work - being able to understand what exists already, will reduce re-creating an existing (or similar) reports, tables, and more.
  • Mosts Popular & Unused - being able to understand the most used (and least used) reports and tables and help direct new hires as well as drive consistency across teams.
  • Audits - being able to reduce time on audits / reports on the inventory, who has access to the content, who has ran the report / table, etc.
  • Lineage - being able to increase knowledge / trust on where the data came from upstream on a particular reports or table. 
  • Impact Analysis - just like lineage, being able to understand downstream impact from a particular table will help keep focus when issues arise with the table or when you want to decommission that table.
  • Team Activity Tracking - being able to track how much content is being created, modified, and deleted by teams is a create way to track the impact of the team.
All of these are great examples of the value of exposing the data inventory (and usage).  Having this information readily available to make decisions, discover content, increase knowledge, or audits, provides an amazing amount of value.  Whether it decreases time to find information or allows you to focus on the impact a particular table has, this allows you (and your teams) to be much more efficient and be truly "data driven".