Senior Python Data Engineer

# Senior Python Data Engineer

We are seeking a Senior Python Data Engineer with deep expertise in large-scale data processing, distributed computing, and backend application development. This role is ideal for an experienced engineer who has built data-intensive applications, implemented data science and machine learning solutions, and developed highly scalable APIs capable of handling large data volumes.

The ideal candidate will possess strong Python development skills, hands-on experience with PySpark and big data ecosystems, expertise in both relational and NoSQL databases, and a solid understanding of data science and machine learning workflows. This individual will collaborate closely with data scientists, architects, product teams, and global engineering teams to deliver scalable, high-performance solutions that drive business outcomes.

This role requires both strong technical execution and the ability to collaborate effectively across distributed teams, including offshore and international stakeholders.

Key Responsibilities

Python Application Development

• Design, develop, and maintain scalable Python applications and services.
• Build reusable, maintainable, and well-documented code components.
• Develop backend services and APIs using Django and/or Flask.
• Participate in architecture reviews and technical design discussions.
• Troubleshoot and optimize production systems.

Big Data Engineering

• Develop and optimize large-scale data processing pipelines using PySpark.
• Process, transform, and analyze large structured and unstructured datasets.
• Build scalable ETL and data ingestion frameworks.
• Optimize distributed data processing workloads for performance and reliability.
• Handle large-volume datasets across enterprise environments.

Data Science & Machine Learning

• Implement data science algorithms and predictive models in production environments.
• Collaborate with data scientists to operationalize machine learning solutions.
• Develop and optimize ML pipelines using scikit-learn and related libraries.
• Support model deployment, validation, and monitoring processes.
• Translate analytical requirements into scalable engineering solutions.

Data Processing & Analytics

• Perform complex data wrangling, cleansing, and transformation activities.
• Build data pipelines utilizing:
• Pandas
• NumPy
• PyArrow
• PySpark
• Improve data quality, consistency, and processing efficiency.
• Develop solutions for handling large-scale datasets and analytical workloads.

API Development

• Design and develop RESTful APIs supporting high-volume data operations.
• Build secure, scalable services capable of handling significant throughput.
• Implement API monitoring, logging, and performance optimization.
• Support integrations with internal and external systems.

Database Development

• Design and optimize data access patterns across relational and NoSQL databases.
• Work with:
• PostgreSQL
• MySQL
• SQL Server
• Oracle
• Develop efficient SQL queries and database integrations.
• Work with NoSQL platforms such as:
• Cassandra
• HBase
• Similar distributed data stores

Software Engineering Excellence

• Follow established coding standards and software development best practices.
• Develop automated unit, integration, and regression tests.
• Participate in peer code reviews and architecture reviews.
• Ensure solutions meet performance, scalability, and maintainability requirements.
• Contribute to CI/CD and DevOps best practices.

Collaboration & Leadership

• Partner with product managers, architects, data scientists, business stakeholders, and engineering teams.
• Collaborate with onsite and offshore teams across multiple time zones.
• Participate in Agile ceremonies and planning activities.
• Mentor junior developers and provide technical guidance.
• Drive technical discussions and influence engineering direction.

Required Qualifications

Experience

• 8+ years of professional Python development experience.
• Proven experience developing enterprise-scale applications and data processing solutions.
• Experience working with large-scale datasets and big data platforms.
• Experience collaborating across global and distributed teams.

Python Development

• Expert-level Python programming skills.
• Strong understanding of object-oriented programming principles.
• Experience developing scalable and maintainable applications.

Frameworks

Strong experience with:

• Django
• Flask
• Experience building production-grade REST APIs.

Big Data Technologies

• Strong hands-on experience with:
• PySpark
• Distributed data processing
• Large-scale ETL pipelines
• Experience handling high-volume datasets.

Data Science & Machine Learning

• Experience implementing and operationalizing data science algorithms.
• Hands-on experience with:
• scikit-learn
• Machine learning workflows
• Model integration and deployment

Data Processing Libraries

Experience using:

• Pandas
• NumPy
• PyArrow
• Other Python data processing libraries

Databases

Strong experience with relational databases:

• PostgreSQL
• SQL Server
• Oracle
• MySQL

Strong SQL development skills.

NoSQL Databases

Experience with:

• Cassandra
• HBase
• Other distributed NoSQL platforms

Engineering Practices

Strong commitment to:

• Automated testing
• Code reviews
• CI/CD
• Source control
• Software engineering best practices