Data Democratization - A Perspective

Couple of years ago, we were having following conversation in IT office.

Me: Guys, what does it take to end all our application data to our Enterprise Big Data Lake?

TM1: But why do we need to send it?  Nobody asked for it!

Me: That’s true but does it hurt us? Is it too much work?

TM1: It is not too much work but I am worried about who will see this data

TM2: That’s my worry too, if users access the data without understanding what it means.

TM1: Actually we have a reporting team who develops reports so if users want some data traditionally they ask us. We can give them canned reports anytime they need.

It brings a question to forefront, why are we overly protective about the data and its usage? I don’t blame the team members as that’s how traditionally things were run. Whenever business user wants data, they need to approach IT for a new report development. But we are now staring at a new paradigm. Users want democratized data which can be self-serviced without involving what they term as expensive, time consuming IT development.  But is it as fancy as it seems? Let’s inspect this case for democratization of data versus the traditional data aristocracy.

Why do we need it?

Firstly business needs it as in the traditional approach there are limitations in accessing data that is needed for making a decision. Only access users got to data is through a set of canned reports that are opened to users. This approach suffers serious limitations

• 1.    The canned reports are developed to cater to static business requirements that do not have flexibility to adapt to dynamic nature of business decisions. If we developed a report which answers the question of Top performing business lines and if user wants to make a decision about non-performing business lines, the report does not help anyways.

• 2.       While the new reports can be developed, they take time and money to see the light of the day

• 3.       Canned reports could not break the barriers of application-centric data silos enterprises created. If a decision involves data present in two different systems, we are facing multi-year, millions of dollar investment. Data that exists with the organization is not immediately consumable.

• 4.       Users relied on excel extract and manual crunching of numbers to get data needed on ad-hoc business needs. Thoughtful business users always asked for excel export option to all reports developed in BI systems.

• 5.       Also it is important to highlight that there were limitations from technology side when it comes to handling huge amount of data translating It into business insights.

• 6.       Extracting value out of unstructured data is not easy to accomplish

x

Panacea?

Business users are excited every time they hear “Self-Service” BI as this gives them information they need without having to go through IT. IT is skeptical about business users skill levels in playing with data, despite the fact that they are well-versed with basic data Querying skills. However the advent of new technology enablers has made it possible to march progressively.

Technology Enablers:

Big Data: The advent of technologies like Hadoop which can handle big data without necessarily having to take care of intrinsic complexities of parallelism and distributed systems has come as a boon.

Data Lake:  Data warehouses have gone out of fashion. Data Lake is the new concept of hosting the enterprise data in one place accessible to users who need the data. Data can be ingested into the lake (which is powered by Big Data technologies) in raw format and can be made accessible to those who need. Traditional Extract-Transform-Load (ETL) processes have taken over by new Extract-Load-Transform (ELT) method. It is more than a jargon change. With ETL , business was under pressure to explain to IT what kind of transformations it needs before data can be loaded into warehouse. Now comes ELT , where business can give a one line answer to load everything as-it-is in the Source data format and worry about transformations when it.

Real-time: Real-time data exchange between applications is not new. Message Queues were serving the purpose for years. However the need of the hour is to handle massive amount of data that is being generated real-time. Technologies like Apache Kafka help in solving this problem.

Cloud: For smaller organizations that could not afford the infrastructure of scale, Cloud offerings help in getting the infrastructure and associated scalability without necessarily having to shell out huge bucks upfront.

Analytical Tools: A variety of tools have come into existence which can help users in coming up with insights. Features like mining useful information, discovery of patterns, machine learning algorithms that are enabled with the tools aid the users in finding valuable information without having to do lot of technical stuff.

     

Challenges:

We cannot rule out the challenges in data democratization in this new found excitement! Following aspects still need to be addressed.

Data Security:

While unfettered access to data to all is real fancy thing to achieve, exposing access to sensitive data can pose threats to the Organization. If this data happens to fall in hands of unauthorized users, it can be risky. So even in the new world of data democracy, Identity and Access Management protocols have to be put in place.

Data Quality:

There has to be a data quality thresholds defined and adhered to before the data can be consumed by wider groups within the organization. Any crucial decisions made with data that is not of good quality can potentially impact the top-line/bottom-line.

Overwhelming Data:

Having so much of data available can be twin edged sword. Users might get overwhelmed with the abundance at their disposal. Data Officers must be a full-time job to aid data ingestion protocols as well as consumption methods for the users.

Data Governance:

A board of data governance tea must be formed with involvement from various stakeholders. Data Governor who will be chairing the board must lay out procedures around how the data moves around the organization, quality metrics, how it must be consumed, what should be classification of information, which information should be restricted for general user etc.,

Data Dictionary:

Data element should be tagged with metadata and context it represents, in the absence of which it becomes extremely difficult for end users to make sense out of the data. Properly defined, robust and unambiguous semantic layer and created data dictionary is essential before exposing the data to general user.

Empowering the users:

Empowering the user to make use of the data accessible to them is critical success factor for this approach. Empowering also includes providing necessary tools, training and skills to the users so that they do not draw inaccurate conclusions from the data and, as a result make wrong decisions.

Involving IT:

In the entire paradigm shift, IT has a crucial but rather unconventional role to play. Success of this new future state mostly depends on adopting to new design and architecture patterns. Some of the conventional jobs in IT might get shrunk, if not eliminated. IT engineers have to redefine their roles that would suit the new landscape and find newer opportunities to help business.

As a summary, while data democracy is something that is good and achievable for benefit of challenging business dynamics, there are few things that need to be addressed at the framework level before it can become a reality. 

Feedback.. Please..

After working in corporate world for more than a decade, one thing I realize important is getting proper, timely, constructive and useful feedback. Honestly in past few years I don’t think I received feedback consistently from anybody except my wife. My wife always has feedback about what I say, do, dint say or didn’t doJ . Jokes apart, after amassing decade of experience, managers thinks you know it all. They assume you doing mistakes you are committing deliberately.  

Proper – The feedback should be given in a scheduled meeting so that the person comes prepared to receive it. It has to be done in a closed room where no one can listen to the conversation. It has to be verbal communication. Sending feedback in writing is required for documentation purpose but not adequate to communicate the message. Also giving feedback in front of the team to be strictly avoided.

Timely – Most managers observe the mistakes and wait for performance rating discussion. They would want to use this mistake as a weapon to justify the rating. But if the mistake is done in January, waiting till December conversations is not going to help the individual. He might commit more of the same mistake, assuming he is doing just fine.

Constructive – Lot of managers misunderstand feedback to rebuke. I have seen managers who shout in the name of giving feedback. In an urge to speak-up the mistake to be understood, they shout at the individual. Some managers in an attempt to be too nice, they won’t give any feedback. Neither is appropriate. Intent should be genuine concern on the staff to help him improve. Only constructive feedback explaining what he should have done better is going to help.

Useful – In delivering feedback, the focus should be on the improvement to be brought in, not on the mistake. Sometimes, managers attribute a collective mistake to one individual. As we cannot change the past, the focus should be on improvements to be made in future. 

I know people who have not received quality feedback in years. So if you have a manager who gives you feedback, you consider yourself lucky. Problem with feedback is receiver enters the room expecting praise for all the good stuff he did, giver comes prepared with mistakes committed. A quality feedback would comprise of a mix of acknowledgement, appreciation for the good efforts followed by improvement sought in matters that did not go too well.  Feedback is a luxury. So whenever you are given quality feedback, do not forget to say “Thank you!”. 

The King Returns!!

5 years is a long time. People age advances, especially for sports persons and athletes, 5 years is lot of time. That time frame can possibly end few careers. 2012 is when Roger won his last Wimbledon. He was 31 years old then. With dry years that followed when he left home slam-less, many predicted the end of Roger show. Some suggested he should be content with what he has achieved and hang up boots.

Though the results show he didn’t win any slams in the years 2013-16, the stats mislead us to believe Roger era is done. Roger made it to 5 major semifinals and 3 Finals. Only Novak and Murray have got better record than this. Roger has shown unbelievable temperament during his dry run that players of current generation must learn from. Despite criticism that he hasn’t won, he always maintained “I will play as long as I am enjoying the game”. He persisted and he provided treat to his fans by staying on. No one can deny the fact that most graceful and elegant tennis comes from his racket.

I don’t know how he felt during the period but it hurt his fans like me so much. Pained more so because he was playing flawless and most consistent form anyone would have dreamt about. But it is the same period when Novak peaked and Murray had got much awaited success in Slams who ended stopping Roger most times. In 2016, Roger took a break and did not compete in US open. This lead to new series of gossips and speculations that Roger is done.

2017 a new chapter has begun. Roger won Australian open. Icing on the cake is when he defeated his long time rival Rafa, whom some critics rate above Roger. The ticker that has paused in 2012 started ticking again. But the real ecstasy is when he participated in Wimbledon an won it without dropping a set. Yes I repeat competing with professionals who are on average 5 years/more younger to him and winning it without dropping a set in the tournament!!  Amazing!!

Roger’s turn around is not just amusing. It is embodiment of the wisdom that if we persist and do the right things , success will find us. He has proven to all that and proved his critics wrong, despite the fact that he has got nothing to prove.  King returns in Style!! And I await US open to see if he repeats the 3 slam years he has done earlier during his prime years. 

Link to an old blog on Roger http://sarathkrishna.blogspot.in/2010/07/wimbledon-final-with-out-roger.html

Why Learn Sanskrit?

If I have any complaints on my parents, it is this. They should have appointed a tutor for teaching me Sanskrit while in school. Even though I chose Sanskrit as one of the languages in XI and XII, Sanskrit is too vast to learn in 2 years of part time study. But it introduced me to beauty of language. I still remember the lesson on GoSeva (Service to Cow) by legendary Kalidas. I have been introduced him as greatest of the poets and ‘Upama Kalidasasya’ – translating to he made phenomenal use of Upama Alankara or simile. But unfortunately could not continue the love with the language during engineering days not because of lack of time but because of lack of focus.

It may appear outdated and out of fashion for someone to wanting to learn Sanskrit when the alternatives we have around are likes of French, Spanish, German. I actually know few people who consider Sanskrit as a dead language. It pains me to hear such things.

It comes back to basic question of why learn a new language. Fundamentally language is a means of communication. So on this front in order to converse in Sanskrit we need to have someone who only understands Sanskrit and no other languages we know like hindi, english and regional language. Argument holds good for likes of French as well. In one of his speeches veteran devotional speaker Shri Malladi Chandrasekhara Sastri Garu quipped “if you need to converse with God Sanskrit is the only medium. You can use English to send money orders not for conversation with God”. Some truth in it as most pujas are performed using Sanskrit and learning the language will help you perform it with cognizance rather than following instructions of priest.

Another reason could be to benefit from the rich knowledge repository in that language and its literature. Nothing beats Sanskrit here. Vedas, Upanishads, Kavyas  dramas and all probably the vast in the world and needs more than a life time to read. There used to be Rishi named Bhardwaja who wanted to study Vedas. He prayed to God and asked for 1000years of life so that h can study Vedas. After 1000years he is not done yet and so asked for another 1000. After next 1000 years, he asked God how much he completed. God has picked a handful of sand from a big heap and said you finished this much in 2000 years. That illustrates the vast nature of literature in Sanskrit, pity large portion of it is not available to us today and is still vast.

Employment opportunities is one reason for study for some. It doesn’t apply to me as I hold a job already. But I recently got to know that in the heart of the city I live in, we have 2 French tutors and in 6 km radius we could not find a Sanskrit tutor for the kids who have opted for that language. So there is plenty of opportunity here as well as in universities.

Some may disagree with me but I feel Sanskrit is easier to learn than likes of French. This is because most regional languages originated from Sanskrit and have many words borrowed from Sanskrit. So the learning and comprehending would not be too tough. On the other hand one needs to go through steep learning curve for other languages like French.

Given the merits associated with the language, I would suggest given an opportunity pursue study of Sanskrit. Also encourage your kids to study this alongside the curriculum prescribed by the contemporary education system. 

Appreciate what one has

People fail to appreciate what they have until they lose it and it is the norm of average human. Many a time we see colleagues who whine about the company and leave, only to realize the challenges exist everywhere. We see folks who do not pay attention to health in prime days and suffer illness. They regret later that they should have paid attention to body. “I felt sad that I had no shoe until I met a person who has no legs.”

It is easy to crib about tiny imperfections in our loved one. Little do we think about the positive value they add to our lives and how much vacuum our life would be when they are not around?

We find parents pressurizing the kids for not scoring top grades, comparing them with neighborhood kids.  They fail to appreciate the talents of their own kid and encourage them for the gifts they possess. While there is nothing wrong in expectations, excessive obsession and comparison is not good for the kids. Also the parents will appreciate the good fortune of having a normal kid as there are many parents who are not fortunate enough to have normal kids. There are many aspiring parents who visit temples, doctors alike to get blessed with child. If we think in those peoples perspective, we can appreciate how blessed we are.

We don’t appreciate life when we are alive. This is excellently put by Apple founder Steve Jobs after he is diagnosed with Cancer “Remembering that I'll be dead soon is the most important tool I've ever encountered to help me make the big choices in life. Almost everything--all external expectations, all pride, all fear of embarrassment or failure--these things just fall away in the face of death, leaving only what is truly important.”

Once we realize this and start appreciating what we have, life would be beautiful. All the petty problems disappear and we feel blessed for having the life we have.