What is Data Pipeline | How to design Data Pipeline ? - ETL vs Data pipeline (2023)

Abone ol 339 B

görünümler 837 B

99% 29 000 1

What is Data Pipeline | How to design Data Pipeline? - ETL vs Data pipeline
#datapipeline
📢📢 Subscribe to my FREE newsletter "Normal I.T. Guy" to know more about what I am thinking, learning and doing in my career & life !!
CLICK THE LINK BELOW 👇👇👇
anshultiwari.substack.com
**Do check out our popular playlists**
1) Latest technology tutorial (2020) -
• What is a Data Va...
2) Google Cloud Platform Beginner Series (2020) -
• Google Cloud Plat...
3) Crunching Data Series (2020) - • Learn - Data Engi...
Hi Friends, I am Anshul Tiwari, and welcome to our youtube channel I.T. k Funde.
More about this video -
In this video, we will understand what is a data pipeline with the help of a real-life example. Data pipelines are designed to move data from one point to another. In this video, we will cover below topics -
1 - What is a Data Pipeline?
2 - Why you need a data pipeline?
3 - Basic design of a data pipeline
4 - Types of Data Pipeline - Batch, Streaming, Lambda architecture
5 - Advanced data pipeline design
We will also learn about various products that can be used in a data pipeline - SAP BODS, Mongo DB, Apache Kafka, Big Query, MDM, Teradata, SAP Business Objects, Tableau.
Credits & Resources -
Wikipedia
www.alooma.com/blog/what-is-a...
*********************FOLLOW US ON*************************
Facebook - ITkFUNDE/
Linkedin - www.linkedin.com/in/ansh9685
Twitter - ansh9685
Blog - blogs.itkfunde.com/
Instagram - itkfunde
****************************************************************
*****************About This Channel************************
Friends ITkFUNDE channel wants to bring I.T related knowledge, information, career advice, and much more to every individual regardless of whether he or she belongs to I.T or not. This channel is for everyone interested in learning something new!

katma

30 Ağu 2020

Paylaş:

Herunterladen:

Yük bağlantısı.....

Ekle:

Çalma listem

Daha sonra izle

YORUMLAR : 1 115

Siva Nagaraju Gamidi 2 yıl önce

Master piece tutorial for data engineering

IT k Funde 2 yıl önce

Thanks Siva

Vijay Jayaram 2 yıl önce

@Indexima I guess the videos on ur channel deals with more on, what do we do after receiving the data, analytics if I understand correctly. Here, its more of pumping the data from one place to a common place, and make it available for interested people down the lane

Gautam De Yıl önce

@IT k Funde It's truly one of the finest and easiest video to follow and relate. Many thanks. Will check other videos.

Lee Whieldon Yıl önce

Thank you for breaking down concepts that are difficult to understand!

altamash jawad Yıl önce

Loved this video, probably the best explanation on advanced data pipeline out there. If in your next videos, maybe create a playlist which can show each of the section of this pipeline in detail with little examples using Python or any language etc. Just an idea, brilliant work!

Eric Dasse Yıl önce

That was great! As a data engineer in the making, this is the first time I have understood the concept of data pipelines so clearly. Thank you very much

lukman aliyu Yıl önce

Hello Eric, I'd love to know how it's going for you at the moment with the DE track

MrBignate12345 2 yıl önce

Would love to learn more about how to choose the right frameworks/technologies for data pipelines and data warehouses/lakes for differing requirements. It would be nice to see a playlist of you designing or comparing solutions for an analytic stack.

IT k Funde 2 yıl önce

Thanks MrBignate I have created various playlists one of which is " Crunching Data Series "...I will surely make more videos on similar topic. It is because of encouragement from audience like you which helps me move forward so thanks and really grateful for your positive feedback.

MrBignate12345 2 yıl önce

Please continue to create videos like these! So easy to understand. Love your visual teaching style and the examples you give.

IT k Funde 2 yıl önce

Thank you MrBignate...The aim is to simplify these techie jargons for everyone to correlate and enjoy learning.

Matthew McCall 4 aylar önce

Excellent high level overview Anshul, I appreciate that you differentiated between batch data and real time data with the Lambda Architecture as it seems most applicable to modern organizations. Your explanation of dashboards as consumers was also very realistic. Your video helped me better understand the general steps in the process. +1 Subscriber.

IT k Funde 4 aylar önce

Thanks Matthew for supporting ❤️

Quoc Le 10 aylar önce

As someone new to Data and AI sales, this was very useful to understand the flow of data and the tools used during each process. Awesome video!

IT k Funde 10 aylar önce

Thanks Quoc ☺️

Prabu R Yıl önce

This would be the Best start for the Data Engineers.. A clear precise and short pictorial representation of Data Pipeline (Basics). Best video so far I had seen.. 😊 Thanks.. Much Appreciated.. 👍

IT k Funde Yıl önce

Thanks Prabu 👍☺️🙏

Vivek Joshi Yıl önce

Do data analysts also use data pipeline creation in their jobs ? Or are they expected to know it ? Asking as some companies write knowledge of ETL in JDs.

Prabu R Yıl önce

@Vivek Joshi knowing any of the ETL tools would help in constructing the pipelines and they can visualize data flow from source to target.. Yes mostly it is used..

Siddharth Shankar 2 yıl önce

Thank you so much for a great and easy to understand data pipeline introduction. I love how you focus on the concepts and not jargons, as it allows for people to understand the essence of data pipeline.

Julian Sihite 7 aylar önce

This guy really explain everything clearly and simple! Good job brother, keep sharing and contributing! You're a great teacher :)

IT k Funde 7 aylar önce

Thanks Julian 😊❤️🙏

Mike Brooks 2 yıl önce

Thanks for a great overview of how the Lambda architecture can expedite the delivery of data to data consumers. For future videos, it would be helpful to map this to the roles, responsibilities, and skill requirements needed to manage this environment.

IT k Funde 2 yıl önce

Thanks Mike for suggestion will try to add this

James McMurtry 2 yıl önce

Great visual layout. Would love to see this applied to an ELT model with Snowflake and it's advantages/disadvantages. Possibly a suggestion on ML complementary tools like Looker and Kraken.

janani sri Yıl önce

One of the best tutorials in youtube so far which gives an overview of data engineering process and that too within 10 minutes. Really appreciate your effort and time you put into making this video. Thank you so much. Please keep doing more such tutorials.

C I 2 yıl önce

This was extremely helpful in my new job working in data. Thank you for your approachable and easy to understand teaching style!

IT k Funde 2 yıl önce

Thanks a lot 🙏

Vinicius Queiroz 3 aylar önce

Thank you very much for this video! If I may make a small contribution: BigQuery is actually a Data Warehouse solution. It stores data in a structured table, in column-first format, ideal for OLAP (much like Snowflake, or Azure Synapse) A Data Lake is capable of storing both structured and unstructured data (which could be GCP's Google Storage, AWS's S3, Azure's Blob Storage, or similar).

ARS Yıl önce

Hello Anshul Sir, I do not have any data engineering background, but I am very strong in visualization. You have aided me in finding the missing blocks about data processing and its treatment for final consumption. Keep doing your good job. Thank you again.

Mardii DKing 2 aylar önce

This topic is so complex as a beginner, but I understand this explanation so well. I didn't even have to go back in the video or rewatch it to understand. This is beautiful.

IT k Funde 2 aylar önce

Thank you so much for your kind words and support 🙏🙏♥♥

Veena Gopal USA 7 aylar önce

Great job ! Very simple and effective way of communicating so that program managers like us can understand the concept behind the architecture choices

IT k Funde 7 aylar önce

Thanks Veena 😊

Yogesh Sharma 2 yıl önce

Very helpful, really appreciate your approach. I have few questions though. How is a deep learning pipeline different from a data pipeline ? Will be helpful if you give few examples so that a developer working on ground relates to.

IT k Funde 2 yıl önce

Thanks Yogesh for suggestion I will try to imclude it when I will start my data related series

NoName 2 yıl önce

Great intro, just what I needed. I learned the distinction between ETL and general pipe lines, and Kafka's place in the architecture.

IT k Funde 2 yıl önce

Thanks Ronnie☺️

Mitchelle Leeuw 2 aylar önce

☺️I’m new in Data Engineering and man you created a clear picture of what I’ve been learning and trying to understand 🙂love this… definitely subscribing 🤩

MOHAMMED ADIL AHMED 2 aylar önce

Am from non IT background And just moving towards the data Field. Appreciate the efforts and the explanation was really wonderful and easy to grasp. I have a query that in the entire process of designing the data pipelines where do we use python . BCS I have seen SQL , mongo dB but not the python..

Kolawole Adekoya 5 aylar önce

Simplified and clear explanation of the concepts. Great diction and presentation. Well done!

IT k Funde 5 aylar önce

thanks Kolawale

Komal Sharma Yıl önce

You beautifully explained the basics & placements of various technologies in an architectural diagram. Very useful. Pls make more videos clarifying the basics

Nitin Bobde 2 yıl önce

Crystal clear explanation. Simple but effective way of explaining concepts. Excellent teacher. You rock.

Kyle 2 yıl önce

Hi Anshul, your video was helpful. I have experience with ETL but didn't know that it was a specific type of data pipeline. Thanks for showing the different type of systems and technologies used for the concept visual that you explained with.

IT k Funde 2 yıl önce

Thank you Kyle coming from an experienced guy means a lot. Hoping for continued support !!

Krishna Patel Yıl önce

Loved the analogy of water pipeline and data pipeline. It made very easy to understand the concept and kept me engaged till the end of the video. I watched your video for the time and its very easy to follow. And yes, I subscribed your channel to get updated. Thanks a lot !! Keep up the good work!

IT k Funde Yıl önce

Thanks Krishna gladbu liked it pls keep supporting ☺️☺️🙏

OG_MJ 2 yıl önce

Thanks for posting this. You are a gifted teacher. I am an experienced ETL Data Ops professional and I am prepping for a an interview for a junior Data Engineering position and I feel much more confident having studied your video !!!

Theōs Osanna 7 aylar önce

There will be another tech company IPOing soonish. Keep an eye out for shares, and a open career position.

Brad Lee Aylar önce

Nice explanation. Can you do a video on the difference between Lambda and Kappa architectures, and what use cases lean more towards one or the other?

IT k Funde Aylar önce

thanks Brad for suggestion

Chimmene Gaius-Assor Yıl önce

Great explanation for introduction to data pipelines. Thanks for clarifying the distinction between ETL and data Pipelines.

Techno Arunav 3 aylar önce

WoW....Your lecture on Data pipeline is so simple and lucid to understand unlike other youtubers. I loved the way you have explained the concepts....Please do more videos on Azure and AWS...Count on me as a new subscriber has been added!!

Aitemir Yeskenov 2 yıl önce

Wonderful explanation. Can't tell you how much I appreciate brilliant tutorials like this one.

George Smith 2 aylar önce

That was incredible. Thank you so much for this crisp, yet encompassing explanation of a data pipeline. I feel I totally understand it after watching for only 10 min.

carlosarte11 Aylar önce

Anshul: Thanks a lot for this great video, you not only explained clearly the concepts, but also gave us the name of useful products for doing each step of the process. Thank you very much.

Biggles Harrumpher 8 gün önce

Great video - it seems while technology has advanced, the concepts of batch loads and real-time data is actually decades old. Back in early 2000's we controlled all ETL and real-time loads with Unix or DOS or SQL scripts that provided return codes for success/failure which triggered alert emails, and we had KPI's for Data quality, backing-out jobs for failed loads, and many other control systems. It just seems there are more 'out-of-the-box' software to handle these now as opposed to custom-built solutions. Great presentation!

Monika Dev 9 aylar önce

iam non IT student and recently joined MS IT where your videos are helping so much..... your explanations are amazing ...... keep educating....can you please explain what is IOT?

IT k Funde 9 aylar önce

Thanks Monika all the best for tour new journey, I have taken your idea

TheSocialDrone 2 yıl önce

An excellent and yet simple explanation of the topic! Thanks a lot for creating and uploading!

Jibril Lamai 2 yıl önce

This is a very good explanation and the best I have seen so far in my quest to understand this concept. Thank you very much. Now I can confidently visualize and explain the same concept with ease and a great understanding of it.

IT k Funde 2 yıl önce

Thanks Jibril glad it helped 🙏☺️

נופר מימון 2 aylar önce

Tnx so much for this explanation! been looking for a couple of days video to understand this subject and this one was without any doubt the best! short,sharp, visualisation that made it really reachable,amazing🙏🏻

IT k Funde 2 aylar önce

thanks dear I am truly grateful for your kind words 🙏😊♥

Gurpreet Singh Yıl önce

Very well explained session on Data Pipeline and comparison with traditional ETL. Thanks so much!

Tom 8 2 yıl önce

Thanks for the video, do u work as a Data Engineer? What are the core skills needed for junior DE? Aside from Python and SQL? Also, at 9:23 u mentioned that customer might want to get the data straight from the streaming, what platform do u usually use to do that? Cheers!

Kei Hinata Yıl önce

Excellent video. Concise and very informative. I learned the basic concepts of ETL, data pipelines, and how data is processed and stored. Thanks so much!

David Dsouza 11 aylar önce

Hi @Kei Hinata I checked out this video on 12 Rules for Data Analysts and Data Engineers and it mentioned that tools are not critical for Data pipeline and ETL processes.@.Best wishes for your career

Sudam Amboji Yıl önce

Layman's terms are so helpful for people like me who are new to data technology.Thank you so much for making it so simple to understand.

IT k Funde Yıl önce

Thanks Sudam thats the motive... Make I.T. interesting for everyone ☺️🙏

Alexander Ulloa Opazo 2 yıl önce

Thank you! I had read a lot of papers about Data Pipeline, but I couldn't get the main idea. However, your video was so easy to understand!! Now I have a better picture of the complete process. Thanks again.

IT k Funde 2 yıl önce

Thank you Alexander !!!

Michael Riego Yıl önce

Your tutorials are very clear and easy to grasp. the way you teach and deliver makes us understand easily. your tutorials are like gems. thank you for sharing your knowledge

Haydar Issa 11 aylar önce

Very elegant way to explain data pipelining and ETL approach. I appreciate the examples given especially the master data management. Well done.

Jerome 2 yıl önce

Good talk. A minor correction - BigQuery is really not a candidate technology for Data Lakes. It's a relational columnar data warehouse. GCS or S3 are used as data lakes

IT k Funde 2 yıl önce

Thanks Jerome I agree GCS is a better option and then you can build your analytical dwh on top using BQ thanks for your inputs ☺️

Adeleida Bingham 11 aylar önce

Your channel is so good!!! Please would you consider sharing your whiteboard diagrams somehow, e.g. LucidChart or just a photo even! It is so hard reducing these complex ideas to the key basics, and they are just spot on.

IT k Funde 11 aylar önce

Thanks Adeleida thats a nice idea let me see if that can be done☺

N U 10 aylar önce

Great job man! Very straight to the point and very informative. Thank you so much!

neo-nair films 9 aylar önce

I think, you'ver covered it all as part of Enterprise Data Architecture. This is from the technology view and bit of functional view as well. Good job!

IT k Funde 9 aylar önce

Thanks

Hasmila omar Yıl önce

It is good that u explain the concept of data pipeline by referring to water pipeline. So much easier to understand and remember. Thank you for your video!!

jonathan sudds 2 yıl önce

Thanks for the video - found the comparison to an actual water pipeline very intuitive. Really enjoyed the video !

Rajiv Jani 6 aylar önce

Superb! In 10 minutes, you have put such a clear picture of data pipeline in my mind that I will never forget! Many thanks for your time and sharing this valuable piece!

Venkata Ramana Murthy Pasumarthi Yıl önce

Excellent presentation on the topic Data Pipeline with live examples.👌👍

IT k Funde Yıl önce

thanks Venkata

Matthew McCall 4 aylar önce

One idea for a video could be prototyping a data pipeline for a use case like registered users, using simple tools like google forms, sheets, and docs.

Ken Ford 2 yıl önce

Great job explaining the difference between Data Pipelines and ETL.

IT k Funde 2 yıl önce

Thanks Ken 🙏☺️

Sameer Rai 2 yıl önce

Great lecture. Can you please tell if we can create a video on various storage such as Data Lake or DWH ?

sharad solanke Yıl önce

Really it's really amazing... First time ever I liked something which is new for me. You explained everything in a very systematic manner... Wish you could have knowledge regarding SAS programming ! I need somebody who can explain SAS like you do. Wish you a very great journey ahead ! 😊 Keep uploading videos. And at last doing what you're really worth is the meaning of life. And you've got it. Your knowledge is helping others to learn concepts in a simple way.

Excellent short and simple explanation of key concepts. Really appreciate your efforts and contributions. Thank you 😊

IT k Funde Yıl önce

Thanks Mate ☺

Harinder Ghuman Yıl önce

Very neat and simplified approach. Cleared my doubts about the need for data pipeline vs ETL. Thanks for sharing!

spikeydude114 2 aylar önce

Great video - really interested in learning more about the MDM part. Really appreciated learning about the tools

Brent COS 2 yıl önce

Very informative, especially for a non-computer science guy like myself. Thanks!

IT k Funde 2 yıl önce

Thanks Brent that is the essence of this channel - Making I.T. interesting for everyone.

Rakesh Khairnar 11 aylar önce

Best explanation of Data Pipeline, though the pipeline itself says the definition but high level understanding of Data Pipeline is presented really nice way. I really like the differentiation of ETL and Data Pipeline. Awesome Video Buddy!!

Mukadi Chisabi Yıl önce

Thanks for this awesome lecture video on Data Pipelines. My question is what is the main objective of using Data pipelines, for example in Astronomy and shouldn't a Data Pipeline mainly be used as a tool used to summarize the data analysis process rather than storage mechanisms

Bhuvanesh Singh Yıl önce

If you can explain it simply enough, you have understood it well enough - superb explanation.. 👍

IT k Funde Yıl önce

Thanks Bhuvanesh means a lot 🙏😊

Nath Jones 8 aylar önce

A very clear explanation of the differences between the two methods. Often I see everything limped under an ETL umbrella, when it may not accurate.

IT k Funde 8 aylar önce

Thanks 🙏

Aman Nakhate Yıl önce

Sir you made data engineering easy 🙏🏻🙌 Great Explanation

IT k Funde Yıl önce

Thanks Aman

Ramakrishna Chimmani 2 yıl önce

Thank you. The best way of explanation. I was looking for this kind of video for long time. As a traditional ETL developer, I questioned my self, why people are using a term called 'Data pipeline' though we have ETL process and what is the exact difference between them. Thanks again.

IT k Funde 2 yıl önce

Thanks Rama for your positive feedback !!

Danish Ansari Yıl önce

Such an awesome explanation, short, crisp and to the point. Great!

nagarajan nr 2 yıl önce

Excellent teaching skills by mapping to simple concepts 👍 looking similar explanation on cleansing, governance, processing and enrichment.

Kopal Soni 9 aylar önce

Thank you for the great explanation. I learned how Kafka and MongoDB would come into the picture for Nosql data. Wish somebody explained this 2 years ago to me.

Sharmila Nadgir Yıl önce

Thank you for this simple and clear explanation of data pipeline. Now I have a clear picture of how data flows from consumer to producer

Mohan R Yıl önce

Simple, easy introduction. Thanks for your patience to explain the concepts.

Tejus Rao 9 aylar önce

Thank you for the video, I learnt what data lake hydration projects are, my previous company had no proper KT, I struggled to grasp what I was doing. This was very nicely explained and cleared the doubts that I had.

IT k Funde 9 aylar önce

Thanks♥️

Explorer007 Yıl önce

Superb video and explanation , I am just a beginner and exploring the spark framework but even I was able to take in the concepts explained , thank you very much for the detailed explanation !!

Abhilash Reddy Yıl önce

Truly amazing 👏 well explained. At least now I know what role apache Kafka and lambda architecture plays in real time data engineering. Thanks for the video.

rama KAMBHAMPATI 2 aylar önce

You are a real "Data Pipeline Spiderman".... fantastic instructor..please share more videos....thanks

IT k Funde 2 aylar önce

Thanks Rama ☺️☺️

Jackie You 2 yıl önce

There is a lot of value in explaining technology in lay man terms like this. Thanks very much!

Karthik Subramanian Yıl önce

Third time I think I am saying this on your channel comments. Your tutorials are useful for professionals like me - I am a marketer in a SaaS product firm. I come here regularly so that I can be better informed when interacting with developers. Well done. And more importantly thanks!

IT k Funde Yıl önce

Thanks Karthik for adding your comments regularly, I read it and I am grateful for your support 🙏❤

K0n5tant 2 yıl önce

Your way of explaining these concepts is excellent, thank you!

IT k Funde 2 yıl önce

Thanks a lot

B Alford 6 gün önce

Very clear and detailed explanation. You put the parts I know in context and helped expand my current knowledge. Thank you

James Lawson Yıl önce

This is so good as it incorporates and evolves the idea of full stack understanding :)

Toyosi Olaofe Folayinka Yıl önce

Wow! This video tutorial was amazing. True knowledge of a concept is when you are able to simplify it so, anyone can understand it.

PapiVidz Yıl önce

Yes I must say this is very concise and how he names the commercial vendors as examples really augments the value further.

Sunder Dase 3 aylar önce

A simple and superb explanation about Data pipeline structure. Thanks a lot. Really appreciate!

Sourabh Suri 2 yıl önce

Thank you so much brother, for clarifying some of the concepts.. Truly appreciate it. Can you suggest - Which way is the Tech Heading now - Data Warehouse Vs. Data Lake? Are DWH a thing of past?

IT k Funde 2 yıl önce

Thanks Sourabh, DWH is here to stay its not going anywhere. Today data world has become enormously huge and there is space for DWH and DL to co exist also Datalake can not solve all business problem. There is a hybrid approach coming up wherein you have your DWH on top of your Datalake

Vivekanand joshi Yıl önce

Data Mesh

Konrad Lampe 2 yıl önce

Great video. Thanks for the brief explanation of data pipeline. I‘m not a technical guy more from the business side and I could easily follow your concept. So, kudos for you and a big thank you!

IT k Funde 2 yıl önce

Thanks Konard 🙏☺️

Niket Bahety Yıl önce

Amazing explanation sir !! Anyone who wants to understand data pipeline this is the perfect lecture👏

IT k Funde Yıl önce

Thanks Niktet

Ranga Gullapalli 2 yıl önce

Wonderful Explanation and you really hit the point straight and clear about data pipe lines in a short and precise manner. ThanQ very much.

IT k Funde 2 yıl önce

Thanks 🙏☺️

h, abhie 2 yıl önce

Nicely explained, thanks! Just curious either for Batch or Realtime processing, among these ETLs (Ab Initio, INFA or Talend, etc) which are best suited for what type of job?

IT k Funde 2 yıl önce

Thanks Abhie its difficult to rate a product it all depends on business needs. So once we have that we can properly asses all products but rating anyone higher is not what i prefer to do without deeper analysis.

h, abhie 2 yıl önce

@IT k Funde sure. But I wasn't asking for rating them. It's about where do these products fit in, in Batch or Realtime?

IT k Funde 2 yıl önce

Oh ok sorry I did not got your query then... So to answer this all these softwares provide both batch and real time data ingestion. Batch is old but now almost every product has realtime data offering they cant rely soley on batch offerings in the age of big data and cloud. Here is an article on Ab initio real time data processing - Ab Initio Lambda - Real-Time Layer www.linkedin.com/pulse/ab-initio-lambda-real-time-layer-chris-day-

Tian Cai 2 yıl önce

Simply one of the best videos on data pipeline on TRvid. Deserves so much more attention.

Manoj 2 yıl önce

Great explanation and examples used. Thanks a ton !!

IT k Funde 2 yıl önce

Thanks Manny

00EagleEye00 2 yıl önce

Hi there. Got a question on raw data (flatfiles). These doesn't have identities or keys so you formulate a candidate key by combining some columns (product, location, target_year). Here's the question, if there are some columns to correct and it belongs to the combination of candidate keys, how can the data be corrected or updated? What approach is to be made ?

Edson Sabino da Silva Yıl önce

Great! The part that I liked the most was the one in wich he explained the difference between ETL and data pipeline

RAGHU G 2 yıl önce

Very good and clear explanation of the whole topic. Good work!

IT k Funde 2 yıl önce

Thanks Raghu🙏☺️

Girish Soragavi 2 yıl önce

Wonderful 👍 this is exactly what I thought the data pipeline would look like and you hit the nail on the head there! Spot on!

IT k Funde 2 yıl önce

Thanks Girish

William Coates Yıl önce

Thank you so much for the straight forward and business friendly explanation of a complex concept. I have subscribed and look forward to viewing more of your content.

Ravindu Abeygunasekara Yıl önce

This is superb!. I am very strange to Data Engineering, and this video gave me a super insight! Keep up the good work

IT k Funde Yıl önce

Thanks Ravindu ☺️

Mayank Satija Yıl önce

Hello Sir, thanks for the amazing video. I am also an ETL developer, just wanted to know about this field's future perspective, how can we grow in this field?

IT k Funde Yıl önce

Thanks Mayank ETL is seeing some disruptive changes where are moving swiftly towards real time and no sql data sources and a need for an agile devops data pipeline. All this means that along with having knowledge on etl products like Informatica, sap bods we should also be good at coding. So I suggest you add python in your skillset ficuuseed around data engineering hope it helps