PinnedAmit Joshi·Mar 14, 2023Data Ingestion from S3 to Snowflake using snow pipeGetting the latest data is essential for further business decisions. For that data ingestion process should be continuous. One of the…A response icon1A response icon1
Amit Joshi·Oct 12Databricks Data + AI Tour : Key TakeawaysThe Mumbai tour was dominated by one idea : Agents. AI agents are about to become the main users of your data stack, so the stack has to…
Amit Joshi·Oct 2What Is Jev AI? Understanding the AI Model Everyone Is DiscussingI explored Jev AI from a Data Engineer’s perspective. Here is what caught my attention, how it differs from traditional LLMs, and why I…
Amit Joshi·Apr 16, 2024Understanding display() & show() in PySpark DataFramesWhen working with PySpark, you often need to inspect and display the contents of DataFrames for debugging, data exploration, or to monitor…A response icon2A response icon2
Amit Joshi·Dec 23, 2023Navigating the Medallion ArchitectureIn today’s data-driven world, organizations increasingly adopt the Medallion Architecture as a powerful approach to managing and processing…
Amit Joshi·Dec 13, 2023Query pyspark dataframe without creating a temp viewYes, you heard it right — you can now query a PySpark DataFrame without the need to create a temporary view. In many scenarios, after…A response icon1A response icon1
Amit Joshi·Dec 12, 2023Repartition & Coalesce in Apache SparkIn the world of distributed computing, challenges like data spill and data skew often loom large. Data spill occurs when the volume of…
Amit Joshi·Dec 5, 2023Exploring Basic Terms in Apache AirflowIf you’re taking your first steps into the world of Apache Airflow, you’ve entered a realm of powerful tools for developing, scheduling…
Amit Joshi·Nov 7, 2023Kusto Query Language (KQL) — A Beginner’s GuideData is the lifeblood of modern decision-making and problem-solving. Whether you’re a data analyst, data engineer, or simply someone…
Amit Joshi·Sep 3, 2023A Beginner’s Guide to Prompt Engineering: Maximizing AI Tool’s PotentialArtificial intelligence (AI) tools have become indispensable in our lives, assisting us in various tasks. To harness their full potential…