Disclaimer: This is a living document. That means I will be updating it as I discover new and relevant information as it relates to Data Engineering. None of the resources I share here are paid promotions and so I just hope this helps you like it helped me.
Where do I begin?
From the beginning, once I’ve discovered that the entire Information Technology industry revolves around data I’ve decided that I want to become a Data Engineer. This surely raised many questions. Some of them were…What exactly is a Data Engineer? What do they actually do? Why do they do what they do? How do they do it?…and most importantly!How do I answer these questions?
The hunt was on…
I realized that I like learning from videos so I searched YouTube first.My search led to new or subsequent searches which returned videos from several creators. Some would say I went down the rabbit hole. 🕳
My first search led me to this video How Data Engineering Works from AltexSoft which was VERY HELPFUL. They deserve an award for the work they’ve done! 🏆
It answered ALL of my listed questions above and then some! Yet it also led to more questions like: How do I build an extract, transform, & load (ETL) pipeline? What tools will I need to do that?Although, it provided a thorough overview of the profession however, I was looking for a more hands-on or practical description of Data Engineering (DE). I knew SQL is a major tool and component of ETL Pipeline development and I was ready to start coding but I wanted to get a good refresher before I just jumped right in. So I continued hunting for more information…
Adding Data Engineering skills as a Software Engineer…
On my hunt I discovered Nelson of Amigoscode.com He explains complex concepts in such simple terms where it is very easy to understand them. In this video SQL For Beginners Tutorial he covers many aspects of SQL.
Most of them I have been aware of previously. Some were brand new to me and VERY helpful in my learning process. Yet I needed more. I needed to learn how to build a portfolio in order to demonstrate my skills from end to end.
Thus I continued searching…
That’s when I found Alex The Analyst, as in Data Analyst. His playlist Data Analytics Portfolio Projects were exactly what I needed to quickly learn the basics of what Data Analysts do and build my portfolio.
This is important as a DE you need to know how you’ll be supporting Data Analysts (DA).
Additionally, that led to more questions furthermore my search continued…
Next, I needed to know how Data Scientists (DS) fit into the equation as a result, I discovered Shashank Kalanithi‘s content.His video, How Data Science ACTUALLY Works really helped me understand the relationships between Data Scientists (DS), Data Analysts (DA), and Data Engineers (DE). They’re usually parts of the same team with a common goal of finding new ways to use existing data or extracting actionable insights from data to make the company more money. 💰
That video was also quite entertaining, I like to call it “infotainment”.I love that style of education! ❤️
In addition…
Another video of his Python for Data Analysts and Data Scientists, teaches Python from the Data Analyst’s and Data Scientist’s perspective.
This helped me understand the types of actions, data processing, and programming issues I might run into or could expect. Moreover, in this video Is the Meta Data Engineering Certificate any good? (as a Data Engineer), Shashank explained the Coursera course’s benefits and what topics the course material covers.
If you use this link, to sign up for the 9 course series he will receive a commission(Help a brother out, it will not cost you any MORE money).There is a way to sign up for ALL of the same courses for FREE but I’ll update this blog with that info later.
Accordingly…( If you take the free version of the courses you will gain the skills yet you will not have access to the exams or receive the certificate of completion.)
I am currently enrolled in the course and I will share my experiences once I complete it. So far, I like it a lot because its filling in the blanks or supplementing missing information and guiding me with a hands-on approach which provides experience for my resume! For this purpose, do you notice that the course is from Meta/Facebook?
A deeper dive into Data Engineering…
My search also turned up several books and other online resources which I will share with you in this post. (This post will receive updates as I get new information.) The 1st book resource is Fundamentals of Data Engineering by Joe Reis & Matt Housley
Many of the topics covered in this book I have prior knowledge or experience with. Yet other topics covered I was not aware of and those topics really filled in the gaps and supplemented my prior knowledge.
Subsequently, it cleared up some common misconceptions with Data Engineering..
For example, this book explained the differences between many of the industry’s buzzwords like the Data Warehouse, Data Lake, Data Lakehouse, Delta Lake and Cloud Data Warehouse.
The book also helped me understand the relationships between company and project stakeholders and how Data Engineers fit into that ecosystem. This is important for communication among the stakeholders, as a Data Engineer I would be required to speak to C-suites (ex: CTO, CISO, CEO, etc), Software Engineers, DBAs, and others. (You can Google the acronyms)
This book is highly recommended for anyone looking to get started. Welp, that’s all I have for you for now. I think that is plenty for anyone looking to get started with Data Engineering. I hope this helps!
There will be more so check back regularly.
If you made it this far you’re the real MVP! 🏆
Thank you for reading and happy learning! ❤️