ABOUT THE ROLE
In this role, you will drive a critical migration of AWS data pipelines from HDF5 files to Parquet, made available via open table formats such as Apache Iceberg and Delta Lake, working with C++, Python, and large datasets to modernize a high-performance data architecture. Our Client is a London-based leader in harmonized historical order book and market data, powering advanced trading analytics and AI-driven insights. You’ll collaborate in a focused team to deliver scalable, efficient data foundations that directly impact global capital markets.
RESPONSIBILITIES
- Develop robust, production-quality code using modern C++ (17+) and Python
- Work within the team to migrate existing parsing processes from HDF5-based pipelines to the new Parquet and Iceberg-based data pipeline
- Deliver migration changes with a strong emphasis on simplicity, performance, reliability, and maintainability
- Ensure that existing data delivery channels and downstream systems continue to function correctly throughout the migration
- Analyse and improve existing C++ parsers (feed handlers) and the pipelines in which they operate, including performance and reliability improvements
- Use Python and relevant data-processing libraries and tooling to support data retrieval, transformation, analytics, and validation where appropriate
- Troubleshoot and resolve technical, performance, and data-related issues within distributed AWS-based data pipelines
- Contribute to second-line support of daily parsing processes and associated data as part of the wider development team
- Leverage AI-assisted coding tools where appropriate while maintaining full ownership of code quality, consistency, and functional correctness
REQUIREMENTS
- Strong commercial experience with modern C++ (17+) and developing production-level, well-tested systems
- Commercial experience with Python development
- Strong experience developing and troubleshooting software in Linux environments
- Experience with data processing pipelines and working with large datasets
- Experience with system or data migrations, including maintaining end-to-end functionality throughout the migration
- Upper-Intermediate English level or higher
- Experience with AWS, particularly Amazon S3 and AWS Glue Data Catalog (nice-to -have)
- Experience with Apache Iceberg and PyIceberg (as a plus)
- Experience with Python data-processing libraries such as Pandas, Polars, or Apache Spark (as an advantage)
SoftServe is an equal opportunity employer. Qualified applicants will receive consideration regardless of race, color, ancestry, ethnicity, national origin, religion, sex, sexual orientation, gender identity or expression, age, citizenship, disability, health condition, marital or family status, veteran status, or any other characteristic protected by applicable law.
