Ilse Epskamp·May 2, 2024SQL operations you might need one dayIf you work with data you will most likely use SQL every now and then. If data is not populated in your SQL tables according to basic…
Ilse Epskamp·May 2, 2024Run integration test between Data Factory and Databricks in Azure DevOps CI pipelineCheck out this example of how to use Powershell to run an integration test between Azure Data Factory and Databricks in your CI pipeline.
InAzure TutorialsbyIlse Epskamp·Jun 8, 2023Save Excel as CSV with custom delimiterLet’s hope that you as a Data Engineer never need this article, because that means the data you are working with is by default…
Ilse Epskamp·Oct 18, 2022Building a scalable metadata-driven data ingestion frameworkIn this blog I discuss the concepts of a metadata-driven data ingestion framework, enabling scalable data processing and ingestion.A response icon1A response icon1
InAzure TutorialsbyIlse Epskamp·Jul 18, 2022CosmosDB (Gremlin) database design: positioning of edges in partitionsIn this blog we will focus on CosmosDB data patterns, and will look specifically at the position and usage of edges in your graph database.
InAzure TutorialsbyIlse Epskamp·May 30, 2022Add/Remove Azure Resource Locks with PowershellResource locks are a powerful mechanism to protect your resources from unauthorized operations.A response icon1A response icon1
InAzure TutorialsbyIlse Epskamp·May 2, 2022Use dataset parameters to copy data to dynamically defined source and sink directories with ADFIn this blog we show how to configure dynamic source and sink directories for your Data Factory workflows.
InAzure TutorialsbyIlse Epskamp·Apr 18, 2022Do’s and Don’ts when working with CosmosDB Gremlin APISharing some of our experiences with ingesting data into Cosmos DB Gremlin API.A response icon1A response icon1
InAzure TutorialsbyIlse Epskamp·Apr 4, 2022Connect Azure Databricks to CosmosDB Gremlin API and run queriesHow to connect Databricks to CosmosDB (Gremlin API) and run queries on the database including a scenario for usage in an end to end flow.
InAzure TutorialsbyIlse Epskamp·Mar 21, 2022Best practices when using Databricks notebooks in an automated workflowBest practices or lessons learned when designing automated workflows with Databricks including a few examples of useful generic functions.