What's the biggest problem in data engineering? Any guesses?

I'll tell you. It's unclear expectations. This has been the biggest problem I've seen over my years of working.

People don't know what they want. They think they do, but they don't.

• I've been on teams that have moved mountains and carried people to the top of the hill, only to be told, "This isn't what we wanted."
• Hit impossible deadlines, only to see the end result never utilized.
• You set off on a project, only for the scope to change halfway through.

This happens a lot. Not always, but enough of the time for it to be a problem.

I've spoken to enough Data Analysts, fellow Data Engineers, and Data Scientists to know that this is happening all over the place.

This is also the reason why I'm not worried about AI stealing my job. If you can't tell me what you want then how will AI know?

How do you get on the same page as your stakeholders? manager? team?

Remember, You can't play the game if you don't know the rules.

Here's what I do (well, I try to do):

1. Early expectations and continuous communication

The foundation of any project that involves data is communication. Yes, talking and listening — the horror!

We live in a world where we have never been more connected with each other. We have a Starbucks menu long of communication choices, but no one talks, and worst of all no one listens. What do people do when they don't know what they are doing? They guess and make assumptions.

That is the surefire way to project a meltdown.

You're not a mind reader, get the expectations down and talk about it:

• The brain dump step — Connect the dots. Focus on the big picture first. Highlight problem areas and address them early on — as a team.
• Now's the time to get detailed — List out the project's steps — everything needs to go down on paper.
• Catch up with your project manager/manager/team review together.
• Make changes, improve, critique, then rinse and repeat.
• Run through the final plan with stakeholders. Take on any changes.
• Find your point of contact, who is the person or team you are going to for answers because questions will pop up.
• Assign tasks — who is doing what and in what order?

If you write the script together (as a team), everyone is more likely to stick to the plan.

2. Document the crap out of it (As you go)

Hands up if you like doing documentation. — No one?

You live or die by the level of documentation you have in your team.

"How does this pipeline work?"

"I don't know, Joe Blog built it."

"Where is he?"

"No idea. He left a year ago."

This scenario plays out so often in companies, it's laughable.

Getting everyone to pull in the same direction is hard. The best thing you can do for yourself and as a team is to get on top of your documentation — early on.

• Requirements from above need to go down from day one. Everyone needs to read them and speak up if anything is unclear.
• Write the documentation as you go, NOT after the project is finished, because after the project, you want nothing to do with this project ever again. Trust me on this one!
• Any issues that like to show up need to go down too, solutions, things tried, etc.
• Use clear language the way you would talk. Not how a robot would speak because I've read enough Microsoft and AWS documentation to know what's good and what's bad. Write for humans. Not robots.
• Diagrams, pictures, and links are great. The more, the better. Complex problems are made clear with epic diagrams. Cavemen drew pictures for a reason, they make more sense — there is logic in that.

3. Take the Stakeholders along for the ride

We are built to collaborate. Humans are social creatures, and data is an adventure. Everyone loves an adventure, and everyone wants to be part of something. So, make this a team effort. That means stakeholders come along for the ride too (whether they want to or not). They get to see what life is like in the trenches.

Doing so means:

• Stakeholders see the issues you are facing early on and can advise.
• If issues or problems lead to a change in direction, stakeholders know about it and why.
• You help bridge the gap between the data and the business.

Additionally, stakeholders will surprise you. They hold the answers you need to questions that arise (most of the time).

Things you can do to bring them along:

• If you can, get something out the door early on. Something like a rough first version of a report or a mockup draft of the data. This gives stakeholders opportunities to get a picture of how the end project will shape up.
• Get stakeholders involved in testing as early as you can. Pair up and walk them through the early phase of the pipeline or how the data is taking shape.
• When a problem arises for which you don't have an answer yet, bring the team together to work on the problem.
• Never guess or assume anything. Try to flag any issues or problems early on in development.

That's that…

I'm not saying these steps always work, no two projects are the same. If experience has taught me anything, it's to expect the unexpected.

These steps or variations of them have, in the past, worked for me. This is the script I tend to follow. More often than not, if I stick to the script and everyone pulls together, most projects run their course happily.

The best projects I've worked on are the ones when everyone understands their role. Make sure everyone knows what they are doing. Leave no room for assumptions. Assumptions lead to frustrations.

I hope you found some of this useful. Take what works for from this article, apply it to your way of working, and let me know how it goes.

🎯 Thanks for reading! If you liked it, please give me a follow and don't forget to subscribe to get my latest articles.

Want to get in touch? feel free to find me on LinkedIn.