What does a day in the life of a data engineer look like? We asked our team of Data Engineers to answer some questions regarding their role. Discover everything about their job and why they started to work as a Data Engineer at Devoteam G Cloud. And most importantly find out what kind of challenges our Data Engineers resolve after downing 2 cups of coffee and attending their daily stand-up meeting.
We asked Jordy, Sven, Andrea, and Rik to tell us about their job and day-to-day tasks at Devoteam G Cloud. Here is an inside look at Data Engineering at Devoteam G Cloud!
So, what do our data engineers do? Who are they, and what do they do?
Most organisations store their data in a variety of formats across databases and text files. This is where data engineers play an important role. Its role is to build pipelines that transform that data into formats that data scientists can use. But why would you then want to become a Data Engineer? Let’s compare this with being a race car builder and a race car driver. The driver gets the excitement of going full speed with the car and enjoys the experience. But the builder gets the excitement of tuning the engine. He experiments with different setups and gets joy by creating a powerful & robust machine. If you’re the type of person that likes to be challenged, data engineering might be right for you.
“I love handling data and getting insights out of them and this is translated in my daily job. I get the opportunity to manage Cloud IT project from A to Z. Every day is a new opportunity to learn new skills to transform data & that’s awesome.”
– Charles, Head of Data Analytics at Devoteam G Cloud
We asked our Data Engineers to describe a typical workday.
We always start with our team stand up where we talk about what we’re working on. The purpose of this meeting is to listen and try to help colleagues if they are blocked. Luckily our standups are not like this “Yesterday I had a bunch of meetings. Today I have even more meetings. I will hopefully work on feature z.” Our daily standups are meant to be about transparency and collaboration. Our goal is to keep our team in the loop on what’s going on in case anyone needs to jump in.
Step 2 of the day: tackling the hardest challenges of that day or/and studying for one hour for one of our certifications. Afterwards we have some customer business meetings to align with their business or to discuss their requests. We also have technical meetings to follow up on the different projects. Most of the days these meetings are spread over the day.
Apart from that, during the day we are developing data pipelines, writing data transformations, or working on our APIs to expose data to external parties. We’re wrapping up the day by testing some code and fixing smaller problems.
Is data engineering a good career choice in 2021?
Long story short – yes. As the amount of data we produce is growing rapidly, so is the demand for data professionals. “Data is the new gold” as Jordy mentioned. But why is data engineering a good career choice in 2021? Our Data Engineers get the opportunity to constantly learn new top-notch technologies and apply them in their day-to-day projects.
We asked our Data Engineers which skills are most in-demand for data engineers? Read on to find out!
Data engineers are expected to know how to build and maintain database systems, be fluent in programming languages, and understand basic machine learning and algorithms. As a Data Engineer you for sure need to be able to learn from your errors and keep growing!
These are the top 3 skills according to our Data Engineers:
- Python
- SQL
- ETL pipelines
Next to that, you need to be stubborn to not settle for the average but go for the best as our Data Engineer Rik says. Add a good portion of willingness to learn and be a team player and you have the perfect Data Engineer.
Do you have any other skills that you wish were on this list to become a data engineer? Let us know in the comments!
What about learning?
We learn the most when working on internal projects like Flycs but also when working on customer projects like BigQuery Data Warehousing.
We get the opportunity to take the first step in our Google Cloud learning journey by obtaining the Google Cloud Certifications. From the Associate Cloud Engineer to Professional Cloud Architect, and Professional Data Engineer certificate.
Within 6 months I got the associate cloud engineer and professional cloud architect certifications. Learning is a value that is super important at Devoteam G Cloud.
– Rik
As Jordy says, every Data Engineer that starts in our Team begins with the Devoteam G Cloud 101 learning, where you learn a lot about the GCP landscape. The goal of this program is to get familiar with topics used by everyone in the technical team and to give the expertise to implement state-of-the-art data engineer solutions at a customer. It forms the basis of your daily work at Devoteam G Cloud.
Topics in the Devoteam G Cloud 101 learning:
- Big data fundamentals
- Focus on BigQuery, Dataflow, Pub/Sub, Cloud Storage, Datastore, BigTable, Cloud SQL and Cloud Functions
- Data Warehouse Modelling
So far I learned a lot of things: from AI Platform, Pub/Sub, SQL, Dataflow, Datastudio, BigQuery to Datastore, Firestore, Cloud Storage, Tensorflow, and GCP platform
– Sven, Google Cloud Data Engineer at Devoteam G Cloud
Just when you think you feel comfortable with what you’re doing, you find yourself challenged with bigger projects. You will need to pick up new technologies and knowledge. There’s always a challenge of solving problems and new learning processes.
We asked our Data Engineers why they started to work as a Data Engineer at Devoteam G Cloud. Here is their answer:
What attracted me here were mainly 3 things: the opportunity to learn daily, the flat organisation, and the good vibes.
– Andrea, Google Cloud Engineer at Devoteam G CloudDuring my interview, I immediately noticed that I will be surrounded by Google experts that are working with great technologies. I like the fact that I can manage Cloud IT projects from A to Z and constantly.
– SvenThe first time I met Devoteam G Cloud was at the VTK job fair and out of the 120+ companies present, it was by far my first choice to start my career. At the stand, I spoke with both Niels and Tristan and they told me about the exciting cloud products of Google and what amazing things they were doing with these technologies. I got the feeling that they were not faking the ‘we are the most fun to work with’-vibe that all of the other companies are trying to reflect or sell. I love the challenging projects & the Fun@Work culture!
– RikI am overall interested in Data, AI, and Machine Learning. I get the opportunity here to work on some cool projects where I can play around with data and get some insights out of them.
– Jordy, Google Cloud Engineer at Devoteam G Cloud
Which advice would you give to young graduates who aren’t sure if Data Engineering is their thing?
Do not be afraid if you are not an expert, I started as a Data Engineer even though my background was more of a Data Scientist. If you are ready to challenge yourself you will become a great Data Engineer.
– AndreaI thought that I wanted to become a pure Data Scientist, but now as a Data Engineer here at Devoteam G Cloud I’m developing every step of the data lifecycle, from data ingestion to machine learning models. This makes it very interesting. Make sure that you choose a company that aims for life-long learning because technologies are changing so fast.
– SvenData engineers right now are in the perfect spot to taste all the new candy on the market: from machine learning to Cloud computing to Web Mining… the list goes on and on! Find a job where you get the opportunity to learn and experiment with all kinds of things.
– RikWe still have some Data Engineering vacancies!
– Jordy
Do you like working with big data?
Become part of our Data Heroes team that supports our customers & partners. You’ll help product decisions alongside software engineering, design, product management, data science, research, and others.