Being intelligent with data

I’m wondering about data access and privacy. There’s a balance. You can mine my data if you give me better service. If you cannot be bothered to be intelligent with my data however, then get lost.

This thought occurs to me when I deal with any organisation. When Netflix loads, does it know the shows I want to see? When my supermarket grocery list repeat order is entered, can they tell me other foods and drinks I might like? And when I logon to my bank account, can they inform me of my past, present and future financial behaviours and exposures, and ensure they are covered?

The gold standard in this space has been Amazon, and their recommendation engine. Using all of your past shopping habits, their data analytics immediately know what to recommend next. Yet, even they have big issues with technical debt. Remember that nugget challenge?

As discussed in Think Like Amazon: 50 1/2 Ideas to Become a Digital Leader: 

Technical debt is when a company chooses a less efficient approach for expediency or where they have a situation where some aspect of their system needs an update. This can occur, for instance, when a software product they depend upon is moving into legacy, and they don’t want to support it anymore.

The other issue is that we too often focus upon new features and functionality. However, when this is done at the expense of paying down our legacy technical debt, we are sacrificing long-term value for short-term results.

In general, a third to two-thirds of the developers in an organisation are dedicated to serving technical debt, which costs money. This is why a company should think of technical debt in the same way as monetary debt. There’s an interest rate for technical debt. It may be in how long it takes people to do a task or, over time, the effort to invest in keeping up with customers and competition. It could be in the additional cost to support a product. It could be that the organisation is less responsive to change, because the technical debt becomes a roadblock for moving forward.

The result is that engineering and coding teams start using workarounds to move forward, which just adds more to the technical debt.

This is illustrated well by a possible Amazon staffer responding to a query on Quora:

What are examples of technical debt at Amazon?

“One problem we had is that, back in the day, we would give very challenging and important tasks to people right out of school. The tasks would get done, but often times the resulting code was horrendous, and/or was not operationally excellent - poor application logs, things that we should record metrics for not getting metrics, etc.

“In order to fix it, we ended up highly prioritizing refactoring, hiring experienced people, and giving the refactoring efforts to proven leaders. The past year, about 80% of our development efforts were on refactoring, better testing processes to allow us to refactor safely, and on occasion total rewrites of components.”

Why have I reverted to a discussion of technical debt in a conversation about data access and privacy?

Mainly because the more technical debt that is built into the organisation, the more difficult it would be to do data analytics to provide personalised customer services, as discussed above. But, more importantly, the less personalised, the more alienated. If you cannot serve me with personalised services using data analytics, I feel you don’t care about me.

On the other side and extreme, if you do analyse my data and do it badly, which technical debt contributes towards, then you just come across as creepy. A lot of banks are either dumb with data or, if they are smart, they can be intelligent or creepy.

A good example of creepy?

The way that Facebook and Meta track and trace my feeds to prioritise the five friends I usually click on, rather than the five hundred other ones who are also interesting. The way that X has started placing people I don’t follow on my feed. People like Elon Musk.

There is a balance between data access and privacy. Technical debt is a huge blockage to doing this effectively but, even if you can address that blockage, can you be intelligent and not creepy?

This data age is challenging.