PinnedDinesh Kumar A S·Nov 23, 2022Apache Spark Optimisation — Part 2I got a clean Spark code but the source data is challenging in distribution, quality, volume, throughput. What are the techniques to…A response icon2A response icon2
PinnedDinesh Kumar A S·Nov 15, 2022Apache Spark Optimisation-Part 1Writing an efficient Spark program is mandatory for achieving better performance and not to spend unncessarily on the cost of infra
Dinesh Kumar A S·Jul 8, 2024My Experience with Data Stores — Part 1In my 10+ years of experience in Data Engineering and Analytics, I have worked with various data stores, ranging from basic relational…
Dinesh Kumar A S·Mar 20, 2023Consumer Lag Monitoring for Spark Streaming Application + KafkaWhen you build a consumer to read messages from a message queue such as Apache Kafka and the consumer consumes and processes slower than…
Dinesh Kumar A S·Dec 23, 2022Introduction to Structured Streaming in Apache Spark (PySpark) + KafkaA Streaming pipeline is the need of the hour since the industry is moving towards near real time analytics and streaming applications…A response icon1A response icon1
Dinesh Kumar A S·Dec 7, 20225 Tips for Writing a better SQLSQL is the integral part of the data engineering world. Be it frameworks like PySpark, ETL tools like Informatica and Datastage, SQL is the…
Dinesh Kumar A S·Nov 25, 2022What is Distributed Computing?“What is distributed computing ; Why and where is it needed”. Let us discuss!!
Dinesh Kumar A S·Nov 18, 2022Data Warehousing with SnowflakeSnowflake is gaining popularity and the adoption among the organisations is increasing day by day. Why is it popular? Let us discuss!!