Snip/Shot by Toddy Mladenov: Analytics/BI & CI

Here is the question – how often do you ask your Analytics (or BI) team to provide you with data in order to make decision or plan something in your job? When you have this answered, here is one more – how often this team can provide you with the data within (ok, let’s be realistic) 24h? For those of you who answered the second one with: “Every time I ask.”, I can only say: “Waw! You are the luckiest people on the Earth!”. For the rest of us there are two options:

Never ask again (and most probably therefore most of us answered the first question with: “Once in a lifetime”)
Continue working with the BI team and hope someday you will get the answer in reasonable time if at all

There is a high chance I will offend somebody with this post, but please don’t take it personally – it’s all business, and you should know it the best.

Recently I had to collect some data for a web site usage that was intended to help us do capacity planning for our servers. I’ve asked our Analytics team for the following:

Number of page views worldwide for the last X months. I wanted to know about any peaks so I needed the data by day or ideally by hour.

OK! How long do you think should take for the Analytics team to collect this information? A day? A week? A month? Never? I am more leaning towards the last.

Here is how it went:

If you work for a big company your first problem is finding the right person in the right Analytics team in the right organization who can help you. In majority of the cases you will not be able to get all those questions answered by one person or even one team. It may take you days or even weeks until you identify the right people you need to work with.
Once you get over your first obstacle you face the challenge with the analysts’ schedule. They are busy people, you know, and who the heck are you to ask them to spend from their precious time to find answers on your silly questions.
If you are very lucky (I emphasize VERY lucky) you may receive occasional reply with a hint for the data. Don’t even dare to ask clarifying questions – either you get the data or not but… it is like the lottery – this was your “once in a lifetime” chance to get a reply from Business Analyst. You should print the reply, frame it and put it on your wall.
And if you are the luckiest amongst the VERY lucky ones you may start conversation, which… takes weeks until you get the data you really need. And the reason for that is that they never have it in a consumable format (Yes, you are right! Even the page views I requested above).

Just to compare the experience, I requested the following information from two engineers in our team:

Server hits for the last two weeks broken down per service, and (of course) per hour.
Number of page views worldwide for the last X months. I wanted to know about any peaks so I needed the data by day or ideally by hour. (You read it right – this is the same information I asked the Analytics team. However the colleague I asked didn’t have access to all properties so he was able to provide me with information for only one of them.)

Can you guess how fast I got the information in the second case?… Give it another try!… 2 hours! “Why so slow?”, will you ask. Yeah, because I requested it 1/2h before lunch, else I could have gotten it faster. Yes, this is the way our Engineering team works – we have the data and monitor it every day. Here is my question:

IF as engineer I am solely responsible for instrumenting the Web site to log the necessary metrics; and IF as consumer of those metrics I am responsible to look in the analytics tools and find the data I need by myself; THEN why the heck companies need Analytics teams?

I will let you answer this question for yourself but here is my advice to the managers of Business Analytics teams as well as their reports:

Lot of Business Analytics teams have proven to be reactive teams; the only data they have readily available is the one reported on the scorecards for upper management. Also (similar to accounting?!?!), they need almost a month to scramble it and present it in consumable form. If those teams want to be successful they should be pro-active and anticipate not only the questions from upper managers but also those coming from peer teams (including development).
Key to success for the Analytics teams is their close relationship with Development teams responsible for implementation. Quite often analytics instrumentation is at the bottom of the priority list just because Analytics teams don’t make the effort to justify their asks. Good implementation requires good requirements and if Analytics teams are not able to communicate those clearly then the implementation will suck and become useless.
Another key to success for Analytics teams is their close relationship with peer teams. They should exchange data, ork closely on requirements and scorecards, and evangelize the need of analytics.
Business Analytics (or its twin brother Business Intelligence) should serve the whole company and not only upper management. I know that kissing up is important for analysts but this is not “business” analytics – I call this (politely) “reporting” to management.
You guys work with numbers and having in mind that most of use have good understanding how much is 2+2 don’t try to convince us that 2+2=5 or send us data that says 2+2=3. What I want to say with that is get your data straight and if some numbers don’t match provide good explanation.
Business Analytics teams should also provide insights of the data. For example if there is a change month over month they should be able to provide explanation why this happened and what measures can be taken to correct it in the future.
The last one (but not the least important) is that Analytics teams should be available and provide transparency of the data. Monthly emails are not enough (keep in mind – people have email rules for those) - go out and socialize your findings. And tailor your message to different audiences because different people want to know different things.

It is really important how Analytics teams position themselves in the company. I am true believer in the analytics data and I don’t think any company can be successful without analyzing it and making decisions based on it. More and more companies rely on the Analytics teams to drive their business and soon we will learn which of those are strong and which not.

Now, if you ask me whether I believe in the Analytics teams… Yeah, I am about to see one that is useful.

Maybe people who are deeper in SEO than myself have already heard about "SEO gap" and there may be people who think it is useful but...

First, I need to clarify what "SEO gap" is. You start with identifying the search leader and its market share for certain region. Good source for information is Comscore. For example let's take Google as the leading search provider in US and assume its market share is 75% (the number is close but may not be completely acurate). Then you look at your analytics data and you see that from all search engine referrals you have, those from Google are 82%. Here is the math:

SEO Gap = Referrals from Search Engine - Search Engine Market Share

For the example above:

SEO Gap = 82% - 75% = +7%

So far, so good! Now you know how to calculate a number and you even have a name for it. Now what? What does this number tell you?

My personal opinion is that calculating SEO Gap is result of "analysis paralysis" (term is trade marked by Robert Kiyosaki but I really like it:)). On the question "Why is this useful to know?" I never get satisfactory answer from our BI team, but I see this every month in our scorecard. The answer I most often get is:

Knowing the SEO gap you know how much the traffic you get from the search engine deviates from the market share for this search engine and this is your opportunity

Normally I don't get it (stupid me:)) and ask follow-up questions. It is logical to think that being in the negative numbers is not good for your web site and my first question is: "So, we would like to be in the positive numbers - right?" The answer to this one really messes up my mind. No, you don't want to be in the positive numbers because this means that you don't get enough traffic from the non-leading search engines and you eat from their traffic (read "you lose users who use other search engines"). As it comes out the ideal number is 0% (ZERO). Zero means you get exactly enough traffic from this search engine as you are supposed to get. Period. And what? What does this tell me? Why do I need to know this? What is the action I can take when I know that? "If you are not ZERO you know that you need to go and optimize either for this search engine or other engines", is one of the answers. Yes, but I don't know which one is the engine second by market share and how do I compare to it (because you don't have it on the scorecard). And even if I know, how does this help me?... I really got distracted (even in my post). I pay too much attention to a number and I don't look anymore at my business.

Here is why. Let's assume that I receive 10M referrals from search engines in that region and Google has 75% market share. Let's assume that my site is at ZERO SEO gap, which translates to 7.5M referrals from Google. Woohoo! I am good! Sure, I am if I don't look at my competitors. They may get 30M referrals from search engines and their SEO gap may be -35%. If you do the math this translates to 30M * (75% - 35%) = 30M * 40% = 12M. Am I good? Of course not! I suck! Because I spend time calculating number that needs 1/2h to explain and many more hours to translate to the scorecard while my competitors spend time to drive traffic and maybe improve their rankings, keep people on their sites, improve content to encourage linking and who knows what else. If I can summarize with two words: my competitor "drives business", while I "invent numbers".

What I learned is that if I am in the Web business I should not spend my time inventing numbers; I should spend my time (and my BI team's time) more wisely and measure the right things, not some hypothetical "what if" scenarios. I should be practical, not theoretical - my Web site is a business for me and not a science project.

By the way, how do you feel when somebody tells you: "Your goal is to be ZERO"? Really excited - aren't you?