What is Data Pipeline | How to design Data Pipeline ? - ETL vs Data pipeline (2023) 

IT k Funde
Abone ol 339 B
görünümler 837 B
99% 29 000 1

What is Data Pipeline | How to design Data Pipeline? - ETL vs Data pipeline
📢📢 Subscribe to my FREE newsletter "Normal I.T. Guy" to know more about what I am thinking, learning and doing in my career & life !!
**Do check out our popular playlists**
1) Latest technology tutorial (2020) -
• What is a Data Va...
2) Google Cloud Platform Beginner Series (2020) -
• Google Cloud Plat...
3) Crunching Data Series (2020) - • Learn - Data Engi...
Hi Friends, I am Anshul Tiwari, and welcome to our youtube channel I.T. k Funde.
More about this video -
In this video, we will understand what is a data pipeline with the help of a real-life example. Data pipelines are designed to move data from one point to another. In this video, we will cover below topics -
1 - What is a Data Pipeline?
2 - Why you need a data pipeline?
3 - Basic design of a data pipeline
4 - Types of Data Pipeline - Batch, Streaming, Lambda architecture
5 - Advanced data pipeline design
We will also learn about various products that can be used in a data pipeline - SAP BODS, Mongo DB, Apache Kafka, Big Query, MDM, Teradata, SAP Business Objects, Tableau.
Credits & Resources -
*********************FOLLOW US ON*************************
Facebook - ITkFUNDE/
Linkedin - www.linkedin.com/in/ansh9685
Twitter - ansh9685
Blog - blogs.itkfunde.com/
Instagram - itkfunde
*****************About This Channel************************
Friends ITkFUNDE channel wants to bring I.T related knowledge, information, career advice, and much more to every individual regardless of whether he or she belongs to I.T or not. This channel is for everyone interested in learning something new!



30 Ağu 2020




Yük bağlantısı.....


Çalma listem
Daha sonra izle
YORUMLAR : 1 115   
Siva Nagaraju Gamidi
Master piece tutorial for data engineering
IT k Funde
IT k Funde 2 yıl önce
Thanks Siva
Vijay Jayaram
Vijay Jayaram 2 yıl önce
@Indexima I guess the videos on ur channel deals with more on, what do we do after receiving the data, analytics if I understand correctly. Here, its more of pumping the data from one place to a common place, and make it available for interested people down the lane
Gautam De
Gautam De Yıl önce
@IT k Funde It's truly one of the finest and easiest video to follow and relate. Many thanks. Will check other videos.
Lee Whieldon
Lee Whieldon Yıl önce
Thank you for breaking down concepts that are difficult to understand!
altamash jawad
altamash jawad Yıl önce
Loved this video, probably the best explanation on advanced data pipeline out there. If in your next videos, maybe create a playlist which can show each of the section of this pipeline in detail with little examples using Python or any language etc. Just an idea, brilliant work!
Eric Dasse
Eric Dasse Yıl önce
That was great! As a data engineer in the making, this is the first time I have understood the concept of data pipelines so clearly. Thank you very much
lukman aliyu
lukman aliyu Yıl önce
Hello Eric, I'd love to know how it's going for you at the moment with the DE track
MrBignate12345 2 yıl önce
Would love to learn more about how to choose the right frameworks/technologies for data pipelines and data warehouses/lakes for differing requirements. It would be nice to see a playlist of you designing or comparing solutions for an analytic stack.
IT k Funde
IT k Funde 2 yıl önce
Thanks MrBignate I have created various playlists one of which is " Crunching Data Series "...I will surely make more videos on similar topic. It is because of encouragement from audience like you which helps me move forward so thanks and really grateful for your positive feedback.
MrBignate12345 2 yıl önce
Please continue to create videos like these! So easy to understand. Love your visual teaching style and the examples you give.
IT k Funde
IT k Funde 2 yıl önce
Thank you MrBignate...The aim is to simplify these techie jargons for everyone to correlate and enjoy learning.
Matthew McCall
Matthew McCall 4 aylar önce
Excellent high level overview Anshul, I appreciate that you differentiated between batch data and real time data with the Lambda Architecture as it seems most applicable to modern organizations. Your explanation of dashboards as consumers was also very realistic. Your video helped me better understand the general steps in the process. +1 Subscriber.
IT k Funde
IT k Funde 4 aylar önce
Thanks Matthew for supporting ❤️
Quoc Le
Quoc Le 10 aylar önce
As someone new to Data and AI sales, this was very useful to understand the flow of data and the tools used during each process. Awesome video!
IT k Funde
IT k Funde 10 aylar önce
Thanks Quoc ☺️
Prabu R
Prabu R Yıl önce
This would be the Best start for the Data Engineers.. A clear precise and short pictorial representation of Data Pipeline (Basics). Best video so far I had seen.. 😊 Thanks.. Much Appreciated.. 👍
IT k Funde
IT k Funde Yıl önce
Thanks Prabu 👍☺️🙏
Vivek Joshi
Vivek Joshi Yıl önce
Do data analysts also use data pipeline creation in their jobs ? Or are they expected to know it ? Asking as some companies write knowledge of ETL in JDs.
Prabu R
Prabu R Yıl önce
@Vivek Joshi knowing any of the ETL tools would help in constructing the pipelines and they can visualize data flow from source to target.. Yes mostly it is used..
Siddharth Shankar
Siddharth Shankar 2 yıl önce
Thank you so much for a great and easy to understand data pipeline introduction. I love how you focus on the concepts and not jargons, as it allows for people to understand the essence of data pipeline.
Julian Sihite
Julian Sihite 7 aylar önce
This guy really explain everything clearly and simple! Good job brother, keep sharing and contributing! You're a great teacher :)
IT k Funde
IT k Funde 7 aylar önce
Thanks Julian 😊❤️🙏
Mike Brooks
Mike Brooks 2 yıl önce
Thanks for a great overview of how the Lambda architecture can expedite the delivery of data to data consumers. For future videos, it would be helpful to map this to the roles, responsibilities, and skill requirements needed to manage this environment.
IT k Funde
IT k Funde 2 yıl önce
Thanks Mike for suggestion will try to add this
James McMurtry
James McMurtry 2 yıl önce
Great visual layout. Would love to see this applied to an ELT model with Snowflake and it's advantages/disadvantages. Possibly a suggestion on ML complementary tools like Looker and Kraken.
janani sri
janani sri Yıl önce
One of the best tutorials in youtube so far which gives an overview of data engineering process and that too within 10 minutes. Really appreciate your effort and time you put into making this video. Thank you so much. Please keep doing more such tutorials.
C I 2 yıl önce
This was extremely helpful in my new job working in data. Thank you for your approachable and easy to understand teaching style!
IT k Funde
IT k Funde 2 yıl önce
Thanks a lot 🙏
Vinicius Queiroz
Vinicius Queiroz 3 aylar önce
Thank you very much for this video! If I may make a small contribution: BigQuery is actually a Data Warehouse solution. It stores data in a structured table, in column-first format, ideal for OLAP (much like Snowflake, or Azure Synapse) A Data Lake is capable of storing both structured and unstructured data (which could be GCP's Google Storage, AWS's S3, Azure's Blob Storage, or similar).
ARS Yıl önce
Hello Anshul Sir, I do not have any data engineering background, but I am very strong in visualization. You have aided me in finding the missing blocks about data processing and its treatment for final consumption. Keep doing your good job. Thank you again.
Mardii DKing
Mardii DKing 2 aylar önce
This topic is so complex as a beginner, but I understand this explanation so well. I didn't even have to go back in the video or rewatch it to understand. This is beautiful.
IT k Funde
IT k Funde 2 aylar önce
Thank you so much for your kind words and support 🙏🙏♥♥
Veena Gopal USA
Veena Gopal USA 7 aylar önce
Great job ! Very simple and effective way of communicating so that program managers like us can understand the concept behind the architecture choices
IT k Funde
IT k Funde 7 aylar önce
Thanks Veena 😊
Yogesh Sharma
Yogesh Sharma 2 yıl önce
Very helpful, really appreciate your approach. I have few questions though. How is a deep learning pipeline different from a data pipeline ? Will be helpful if you give few examples so that a developer working on ground relates to.
IT k Funde
IT k Funde 2 yıl önce
Thanks Yogesh for suggestion I will try to imclude it when I will start my data related series
NoName 2 yıl önce
Great intro, just what I needed. I learned the distinction between ETL and general pipe lines, and Kafka's place in the architecture.
IT k Funde
IT k Funde 2 yıl önce
Thanks Ronnie☺️
Mitchelle Leeuw
Mitchelle Leeuw 2 aylar önce
☺️I’m new in Data Engineering and man you created a clear picture of what I’ve been learning and trying to understand 🙂love this… definitely subscribing 🤩
Am from non IT background And just moving towards the data Field. Appreciate the efforts and the explanation was really wonderful and easy to grasp. I have a query that in the entire process of designing the data pipelines where do we use python . BCS I have seen SQL , mongo dB but not the python..
Kolawole Adekoya
Kolawole Adekoya 5 aylar önce
Simplified and clear explanation of the concepts. Great diction and presentation. Well done!
IT k Funde
IT k Funde 5 aylar önce
thanks Kolawale
Komal Sharma
Komal Sharma Yıl önce
You beautifully explained the basics & placements of various technologies in an architectural diagram. Very useful. Pls make more videos clarifying the basics
Nitin Bobde
Nitin Bobde 2 yıl önce
Crystal clear explanation. Simple but effective way of explaining concepts. Excellent teacher. You rock.
Kyle 2 yıl önce
Hi Anshul, your video was helpful. I have experience with ETL but didn't know that it was a specific type of data pipeline. Thanks for showing the different type of systems and technologies used for the concept visual that you explained with.
IT k Funde
IT k Funde 2 yıl önce
Thank you Kyle coming from an experienced guy means a lot. Hoping for continued support !!
Krishna Patel
Krishna Patel Yıl önce
Loved the analogy of water pipeline and data pipeline. It made very easy to understand the concept and kept me engaged till the end of the video. I watched your video for the time and its very easy to follow. And yes, I subscribed your channel to get updated. Thanks a lot !! Keep up the good work!
IT k Funde
IT k Funde Yıl önce
Thanks Krishna gladbu liked it pls keep supporting ☺️☺️🙏
OG_MJ 2 yıl önce
Thanks for posting this. You are a gifted teacher. I am an experienced ETL Data Ops professional and I am prepping for a an interview for a junior Data Engineering position and I feel much more confident having studied your video !!!
Theōs Osanna
Theōs Osanna 7 aylar önce
There will be another tech company IPOing soonish. Keep an eye out for shares, and a open career position.
Brad Lee
Brad Lee Aylar önce
Nice explanation. Can you do a video on the difference between Lambda and Kappa architectures, and what use cases lean more towards one or the other?
IT k Funde
IT k Funde Aylar önce
thanks Brad for suggestion
Chimmene Gaius-Assor
Great explanation for introduction to data pipelines. Thanks for clarifying the distinction between ETL and data Pipelines.
Techno Arunav
Techno Arunav 3 aylar önce
WoW....Your lecture on Data pipeline is so simple and lucid to understand unlike other youtubers. I loved the way you have explained the concepts....Please do more videos on Azure and AWS...Count on me as a new subscriber has been added!!
Aitemir Yeskenov
Aitemir Yeskenov 2 yıl önce
Wonderful explanation. Can't tell you how much I appreciate brilliant tutorials like this one.
George Smith
George Smith 2 aylar önce
That was incredible. Thank you so much for this crisp, yet encompassing explanation of a data pipeline. I feel I totally understand it after watching for only 10 min.
carlosarte11 Aylar önce
Anshul: Thanks a lot for this great video, you not only explained clearly the concepts, but also gave us the name of useful products for doing each step of the process. Thank you very much.
Biggles Harrumpher
Biggles Harrumpher 8 gün önce
Great video - it seems while technology has advanced, the concepts of batch loads and real-time data is actually decades old. Back in early 2000's we controlled all ETL and real-time loads with Unix or DOS or SQL scripts that provided return codes for success/failure which triggered alert emails, and we had KPI's for Data quality, backing-out jobs for failed loads, and many other control systems. It just seems there are more 'out-of-the-box' software to handle these now as opposed to custom-built solutions. Great presentation!
Monika Dev
Monika Dev 9 aylar önce
iam non IT student and recently joined MS IT where your videos are helping so much..... your explanations are amazing ...... keep educating....can you please explain what is IOT?
IT k Funde
IT k Funde 9 aylar önce
Thanks Monika all the best for tour new journey, I have taken your idea
TheSocialDrone 2 yıl önce
An excellent and yet simple explanation of the topic! Thanks a lot for creating and uploading!
Jibril Lamai
Jibril Lamai 2 yıl önce
This is a very good explanation and the best I have seen so far in my quest to understand this concept. Thank you very much. Now I can confidently visualize and explain the same concept with ease and a great understanding of it.
IT k Funde
IT k Funde 2 yıl önce
Thanks Jibril glad it helped 🙏☺️
נופר מימון
נופר מימון 2 aylar önce
Tnx so much for this explanation! been looking for a couple of days video to understand this subject and this one was without any doubt the best! short,sharp, visualisation that made it really reachable,amazing🙏🏻
IT k Funde
IT k Funde 2 aylar önce
thanks dear I am truly grateful for your kind words 🙏😊♥
Gurpreet Singh
Gurpreet Singh Yıl önce
Very well explained session on Data Pipeline and comparison with traditional ETL. Thanks so much!
Tom 8
Tom 8 2 yıl önce
Thanks for the video, do u work as a Data Engineer? What are the core skills needed for junior DE? Aside from Python and SQL? Also, at 9:23 u mentioned that customer might want to get the data straight from the streaming, what platform do u usually use to do that? Cheers!
Kei Hinata
Kei Hinata Yıl önce
Excellent video. Concise and very informative. I learned the basic concepts of ETL, data pipelines, and how data is processed and stored. Thanks so much!
David Dsouza
David Dsouza 11 aylar önce
Hi @Kei Hinata I checked out this video on 12 Rules for Data Analysts and Data Engineers and it mentioned that tools are not critical for Data pipeline and ETL processes.​@.Best wishes for your career
Sudam Amboji
Sudam Amboji Yıl önce
Layman's terms are so helpful for people like me who are new to data technology.Thank you so much for making it so simple to understand.
IT k Funde
IT k Funde Yıl önce
Thanks Sudam thats the motive... Make I.T. interesting for everyone ☺️🙏
Alexander Ulloa Opazo
Thank you! I had read a lot of papers about Data Pipeline, but I couldn't get the main idea. However, your video was so easy to understand!! Now I have a better picture of the complete process. Thanks again.
IT k Funde
IT k Funde 2 yıl önce
Thank you Alexander !!!
Michael Riego
Michael Riego Yıl önce
Your tutorials are very clear and easy to grasp. the way you teach and deliver makes us understand easily. your tutorials are like gems. thank you for sharing your knowledge
Haydar Issa
Haydar Issa 11 aylar önce
Very elegant way to explain data pipelining and ETL approach. I appreciate the examples given especially the master data management. Well done.
Jerome 2 yıl önce
Good talk. A minor correction - BigQuery is really not a candidate technology for Data Lakes. It's a relational columnar data warehouse. GCS or S3 are used as data lakes
IT k Funde
IT k Funde 2 yıl önce
Thanks Jerome I agree GCS is a better option and then you can build your analytical dwh on top using BQ thanks for your inputs ☺️
Adeleida Bingham
Adeleida Bingham 11 aylar önce
Your channel is so good!!! Please would you consider sharing your whiteboard diagrams somehow, e.g. LucidChart or just a photo even! It is so hard reducing these complex ideas to the key basics, and they are just spot on.
IT k Funde
IT k Funde 11 aylar önce
Thanks Adeleida thats a nice idea let me see if that can be done☺
N U 10 aylar önce
Great job man! Very straight to the point and very informative. Thank you so much!
neo-nair films
neo-nair films 9 aylar önce
I think, you'ver covered it all as part of Enterprise Data Architecture. This is from the technology view and bit of functional view as well. Good job!
IT k Funde
IT k Funde 9 aylar önce
Hasmila omar
Hasmila omar Yıl önce
It is good that u explain the concept of data pipeline by referring to water pipeline. So much easier to understand and remember. Thank you for your video!!
jonathan sudds
jonathan sudds 2 yıl önce
Thanks for the video - found the comparison to an actual water pipeline very intuitive. Really enjoyed the video !
Rajiv Jani
Rajiv Jani 6 aylar önce
Superb! In 10 minutes, you have put such a clear picture of data pipeline in my mind that I will never forget! Many thanks for your time and sharing this valuable piece!
Venkata Ramana Murthy Pasumarthi
Excellent presentation on the topic Data Pipeline with live examples.👌👍
IT k Funde
IT k Funde Yıl önce
thanks Venkata
Matthew McCall
Matthew McCall 4 aylar önce
One idea for a video could be prototyping a data pipeline for a use case like registered users, using simple tools like google forms, sheets, and docs.
Ken Ford
Ken Ford 2 yıl önce
Great job explaining the difference between Data Pipelines and ETL.
IT k Funde
IT k Funde 2 yıl önce
Thanks Ken 🙏☺️
Sameer Rai
Sameer Rai 2 yıl önce
Great lecture. Can you please tell if we can create a video on various storage such as Data Lake or DWH ?
sharad solanke
sharad solanke Yıl önce
Really it's really amazing... First time ever I liked something which is new for me. You explained everything in a very systematic manner... Wish you could have knowledge regarding SAS programming ! I need somebody who can explain SAS like you do. Wish you a very great journey ahead ! 😊 Keep uploading videos. And at last doing what you're really worth is the meaning of life. And you've got it. Your knowledge is helping others to learn concepts in a simple way.
Login Log247
Login Log247 Yıl önce
Excellent short and simple explanation of key concepts. Really appreciate your efforts and contributions. Thank you 😊
IT k Funde
IT k Funde Yıl önce
Thanks Mate ☺
Harinder Ghuman
Harinder Ghuman Yıl önce
Very neat and simplified approach. Cleared my doubts about the need for data pipeline vs ETL. Thanks for sharing!
spikeydude114 2 aylar önce
Great video - really interested in learning more about the MDM part. Really appreciated learning about the tools
Brent COS
Brent COS 2 yıl önce
Very informative, especially for a non-computer science guy like myself. Thanks!
IT k Funde
IT k Funde 2 yıl önce
Thanks Brent that is the essence of this channel - Making I.T. interesting for everyone.
Rakesh Khairnar
Rakesh Khairnar 11 aylar önce
Best explanation of Data Pipeline, though the pipeline itself says the definition but high level understanding of Data Pipeline is presented really nice way. I really like the differentiation of ETL and Data Pipeline. Awesome Video Buddy!!
Mukadi Chisabi
Mukadi Chisabi Yıl önce
Thanks for this awesome lecture video on Data Pipelines. My question is what is the main objective of using Data pipelines, for example in Astronomy and shouldn't a Data Pipeline mainly be used as a tool used to summarize the data analysis process rather than storage mechanisms
Bhuvanesh Singh
Bhuvanesh Singh Yıl önce
If you can explain it simply enough, you have understood it well enough - superb explanation.. 👍
IT k Funde
IT k Funde Yıl önce
Thanks Bhuvanesh means a lot 🙏😊
Nath Jones
Nath Jones 8 aylar önce
A very clear explanation of the differences between the two methods. Often I see everything limped under an ETL umbrella, when it may not accurate.
IT k Funde
IT k Funde 8 aylar önce
Thanks 🙏
Aman Nakhate
Aman Nakhate Yıl önce
Sir you made data engineering easy 🙏🏻🙌 Great Explanation
IT k Funde
IT k Funde Yıl önce
Thanks Aman
Ramakrishna Chimmani
Thank you. The best way of explanation. I was looking for this kind of video for long time. As a traditional ETL developer, I questioned my self, why people are using a term called 'Data pipeline' though we have ETL process and what is the exact difference between them. Thanks again.
IT k Funde
IT k Funde 2 yıl önce
Thanks Rama for your positive feedback !!
Danish Ansari
Danish Ansari Yıl önce
Such an awesome explanation, short, crisp and to the point. Great!
nagarajan nr
nagarajan nr 2 yıl önce
Excellent teaching skills by mapping to simple concepts 👍 looking similar explanation on cleansing, governance, processing and enrichment.
Kopal Soni
Kopal Soni 9 aylar önce
Thank you for the great explanation. I learned how Kafka and MongoDB would come into the picture for Nosql data. Wish somebody explained this 2 years ago to me.
Sharmila Nadgir
Sharmila Nadgir Yıl önce
Thank you for this simple and clear explanation of data pipeline. Now I have a clear picture of how data flows from consumer to producer
Mohan R
Mohan R Yıl önce
Simple, easy introduction. Thanks for your patience to explain the concepts.
Tejus Rao
Tejus Rao 9 aylar önce
Thank you for the video, I learnt what data lake hydration projects are, my previous company had no proper KT, I struggled to grasp what I was doing. This was very nicely explained and cleared the doubts that I had.
IT k Funde
IT k Funde 9 aylar önce
Explorer007 Yıl önce
Superb video and explanation , I am just a beginner and exploring the spark framework but even I was able to take in the concepts explained , thank you very much for the detailed explanation !!
Abhilash Reddy
Abhilash Reddy Yıl önce
Truly amazing 👏 well explained. At least now I know what role apache Kafka and lambda architecture plays in real time data engineering. Thanks for the video.
rama KAMBHAMPATI 2 aylar önce
You are a real "Data Pipeline Spiderman".... fantastic instructor..please share more videos....thanks
IT k Funde
IT k Funde 2 aylar önce
Thanks Rama ☺️☺️
Jackie You
Jackie You 2 yıl önce
There is a lot of value in explaining technology in lay man terms like this. Thanks very much!
Karthik Subramanian
Third time I think I am saying this on your channel comments. Your tutorials are useful for professionals like me - I am a marketer in a SaaS product firm. I come here regularly so that I can be better informed when interacting with developers. Well done. And more importantly thanks!
IT k Funde
IT k Funde Yıl önce
Thanks Karthik for adding your comments regularly, I read it and I am grateful for your support 🙏❤
K0n5tant 2 yıl önce
Your way of explaining these concepts is excellent, thank you!
IT k Funde
IT k Funde 2 yıl önce
Thanks a lot
B Alford
B Alford 6 gün önce
Very clear and detailed explanation. You put the parts I know in context and helped expand my current knowledge. Thank you
James Lawson
James Lawson Yıl önce
This is so good as it incorporates and evolves the idea of full stack understanding :)
Toyosi Olaofe Folayinka
Wow! This video tutorial was amazing. True knowledge of a concept is when you are able to simplify it so, anyone can understand it.
PapiVidz Yıl önce
Yes I must say this is very concise and how he names the commercial vendors as examples really augments the value further.
Sunder Dase
Sunder Dase 3 aylar önce
A simple and superb explanation about Data pipeline structure. Thanks a lot. Really appreciate!
Sourabh Suri
Sourabh Suri 2 yıl önce
Thank you so much brother, for clarifying some of the concepts.. Truly appreciate it. Can you suggest - Which way is the Tech Heading now - Data Warehouse Vs. Data Lake? Are DWH a thing of past?
IT k Funde
IT k Funde 2 yıl önce
Thanks Sourabh, DWH is here to stay its not going anywhere. Today data world has become enormously huge and there is space for DWH and DL to co exist also Datalake can not solve all business problem. There is a hybrid approach coming up wherein you have your DWH on top of your Datalake
Vivekanand joshi
Vivekanand joshi Yıl önce
Data Mesh
Konrad Lampe
Konrad Lampe 2 yıl önce
Great video. Thanks for the brief explanation of data pipeline. I‘m not a technical guy more from the business side and I could easily follow your concept. So, kudos for you and a big thank you!
IT k Funde
IT k Funde 2 yıl önce
Thanks Konard 🙏☺️
Niket Bahety
Niket Bahety Yıl önce
Amazing explanation sir !! Anyone who wants to understand data pipeline this is the perfect lecture👏
IT k Funde
IT k Funde Yıl önce
Thanks Niktet
Ranga Gullapalli
Ranga Gullapalli 2 yıl önce
Wonderful Explanation and you really hit the point straight and clear about data pipe lines in a short and precise manner. ThanQ very much.
IT k Funde
IT k Funde 2 yıl önce
Thanks 🙏☺️
h, abhie
h, abhie 2 yıl önce
Nicely explained, thanks! Just curious either for Batch or Realtime processing, among these ETLs (Ab Initio, INFA or Talend, etc) which are best suited for what type of job?
IT k Funde
IT k Funde 2 yıl önce
Thanks Abhie its difficult to rate a product it all depends on business needs. So once we have that we can properly asses all products but rating anyone higher is not what i prefer to do without deeper analysis.
h, abhie
h, abhie 2 yıl önce
@IT k Funde sure. But I wasn't asking for rating them. It's about where do these products fit in, in Batch or Realtime?
IT k Funde
IT k Funde 2 yıl önce
Oh ok sorry I did not got your query then... So to answer this all these softwares provide both batch and real time data ingestion. Batch is old but now almost every product has realtime data offering they cant rely soley on batch offerings in the age of big data and cloud. Here is an article on Ab initio real time data processing - Ab Initio Lambda - Real-Time Layer www.linkedin.com/pulse/ab-initio-lambda-real-time-layer-chris-day-
Tian Cai
Tian Cai 2 yıl önce
Simply one of the best videos on data pipeline on TRvid. Deserves so much more attention.
Manoj 2 yıl önce
Great explanation and examples used. Thanks a ton !!
IT k Funde
IT k Funde 2 yıl önce
Thanks Manny
00EagleEye00 2 yıl önce
Hi there. Got a question on raw data (flatfiles). These doesn't have identities or keys so you formulate a candidate key by combining some columns (product, location, target_year). Here's the question, if there are some columns to correct and it belongs to the combination of candidate keys, how can the data be corrected or updated? What approach is to be made ?
Edson Sabino da Silva
Great! The part that I liked the most was the one in wich he explained the difference between ETL and data pipeline
RAGHU G 2 yıl önce
Very good and clear explanation of the whole topic. Good work!
IT k Funde
IT k Funde 2 yıl önce
Thanks Raghu🙏☺️
Girish Soragavi
Girish Soragavi 2 yıl önce
Wonderful 👍 this is exactly what I thought the data pipeline would look like and you hit the nail on the head there! Spot on!
IT k Funde
IT k Funde 2 yıl önce
Thanks Girish
William Coates
William Coates Yıl önce
Thank you so much for the straight forward and business friendly explanation of a complex concept. I have subscribed and look forward to viewing more of your content.
Ravindu Abeygunasekara
This is superb!. I am very strange to Data Engineering, and this video gave me a super insight! Keep up the good work
IT k Funde
IT k Funde Yıl önce
Thanks Ravindu ☺️
Mayank Satija
Mayank Satija Yıl önce
Hello Sir, thanks for the amazing video. I am also an ETL developer, just wanted to know about this field's future perspective, how can we grow in this field?
IT k Funde
IT k Funde Yıl önce
Thanks Mayank ETL is seeing some disruptive changes where are moving swiftly towards real time and no sql data sources and a need for an agile devops data pipeline. All this means that along with having knowledge on etl products like Informatica, sap bods we should also be good at coding. So I suggest you add python in your skillset ficuuseed around data engineering hope it helps
Rembau Times
Rembau Times 2 yıl önce
Good explanation - very methodical and explains the concept clearly. Thanks for this excellent video.
IT k Funde
IT k Funde 2 yıl önce
Çok Güzel Hareketler 146. Bölüm
How Data Engineering Works
görünümler 323 B