Engineering

Senior Data Engineer (Azure)

Hyderabad
Work Type: Full Time
Apply Now
Responsibilities:
  • Design, develop, and maintain scalable ETL pipelines using Azure Databricks, Spark, and other Azure services.
  • Optimize and troubleshoot data pipelines for performance, scalability, and reliability.
  • Implement data governance policies to ensure accuracy, consistency, and security.
  • Conduct data analysis, profiling, and validation to maintain high data quality.
  • Integrate Azure Databricks with Azure Data Lake, Azure SQL Database, and Azure Synapse Analytics.
  • Develop and manage ETL pipelines using Azure Data Factory and PySpark.
  • Build reusable frameworks to automate and streamline data processing tasks.
  • Create and maintain technical documentation for data pipelines and processes.
  • Collaborate with cross-functional teams to align solutions with business and technical objectives.
  • Identify opportunities to expand data coverage and enhance analytics capabilities, including search and visualization tools.
  • Monitor and optimize system performance, proactively addressing issues and stakeholder requests.
  • Provide insights on data pipeline performance and recommend optimizations.
  • Stay updated with emerging Azure technologies and best practices to enhance data engineering capabilities.
  • Train and onboard team members and stakeholders on the data platform’s capabilities.


Skills and Experience:
  • 3+ years of hands-on experience with Azure Databricks and data engineering in cloud environments.
  • Extensive experience with Azure Data Factory, Azure SQL Database, Azure Synapse Analytics, Data Lake, Cosmos DB, EventHub, ServiceBus, Topics, Blob Storage, Azure Functions, AKS, and Key Vault.
  • Strong understanding of data warehousing concepts (star schema, snowflake schema, data lakes) and data modeling for scalable architecture.
  • Proficiency in Python and SQL (R is a plus), with hands-on experience in writing complex SQL queries.
  • Hands-on experience with distributed computing frameworks such as PySpark (Spark experience required; Hadoop or Dask is a plus).
  • Working knowledge of Azure Pipelines, GitHub Actions & Workflows, and version control (Git).
  • Experience with data integration, ETL workflows, and pipeline automation using Azure Data Factory and PySpark.
  • Familiarity with ML workflows and integrating ML models into Databricks pipelines.
  • Exposure to Docker and Kubernetes (AKS is a plus) for scalable deployments.
  • Strong analytical skills to troubleshoot complex data issues, with excellent communication and stakeholder collaboration.
  • Experience with Agile methodologies for project tracking (JIRA) and collaboration.
  • Azure certifications such as Azure Data Engineer, Azure Solutions Architect, or Azure Developer ID are good to have
  • Excellent communication and interpersonal skills to collaborate with cross-functional teams and stakeholders


Education:
  • Bachelors or Masters from Premier Institutes preferred.
  • Experience 4-12 years

Submit Your Application

You have successfully applied
  • You have errors in applying
Or
  Autofill with LinkedIn