Description

Hybrid 3 days onsite / 2 days remote in Tysons Corner, VA

Our client is seeking a highly skilled and experienced Big Data Engineer to design, develop, and optimize large-scale data processing systems. You will work with cross-functional teams to architect data pipelines, implement data integration solutions, and ensure the performance, scalability, and reliability of big data platforms. The ideal candidate will have expertise in distributed systems, cloud platforms, and modern big data technologies such as Hadoop and Spark.

We can facilitate w2 and corp-to-corp consultants. For our w2 consultants, we offer a great benefits package that includes Medical, Dental, and Vision benefits, 401k with company matching, and life insurance.

Rate: $65.00 to $70.00/hr. w2

Responsibilities

Design, develop, and maintain large-scale data processing pipelines using Big Data technologies such as Hadoop, Spark, Python, and Scala.
Implement data ingestion, storage, transformation, and analysis solutions that are scalable, efficient, and reliable.
Stay current with industry trends and emerging Big Data technologies to improve the data architecture.
Collaborate with cross-functional teams to translate business requirements into technical solutions.
Optimize and enhance existing data pipelines for performance, scalability, and reliability.
Develop automated testing frameworks and implement continuous testing for data quality assurance.
Conduct unit, integration, and system testing to ensure the robustness and accuracy of data pipelines.
Partner with data scientists and analysts to support data-driven decision-making.
Write and maintain automated unit, integration, and end-to-end tests.
Monitor and troubleshoot data pipelines in production to identify and resolve issues.
Apply SQL skills including window functions, joins, aggregations, and handling of NULLs, duplicates, and ordering.
Design and tune Apache Spark jobs, including partitioning, caching, broadcast joins, and troubleshooting DAG, stages, tasks, and executor performance.
Leverage AWS services such as S3, EMR, Glue, Lambda, and Athena, including Spark with S3 file formats and consistency considerations.
Write clean, modular, and performant Python or Scala code using functional programming concepts, and manage collections, concurrency, and memory.
Employ AI tools and prompt engineering to enhance development workflows, analysis, and continuous improvement.

Experience Requirements

Bachelor's degree in Computer Science, Information Systems, or related discipline with at least five years of related experience, or equivalent training and work experience. Master's degree and Financial Services experience preferred.
Demonstrated expertise in object-oriented and database technologies leading to enterprise-quality solutions.
Experience delivering enterprise solutions in iterative or Agile environments.
Knowledge of software engineering approaches including test automation, build automation, and configuration management frameworks.
Strong written and verbal technical communication skills.
Ability to build effective working relationships and improve work product quality.
Strong organization, thoroughness, and ability to handle competing priorities.
Ability to maintain focus and develop proficiency in new skills rapidly.
Experience working in a fast paced environment.
Experience with Java, Scala, or Python.
Essential technical skills include AI tool proficiency (e.g., GitHub Copilot, Q Developer, ChatGPT, Claude), strong software development background, and extensive experience with Scrum, Kanban, and continuous improvement.
Big Data technologies: Hadoop, Spark, Hive, and Trino, with understanding of data skew, PB-scale data, and resource-related job failures.
Good to have: managing production data pipelines and ETL systems, CI/CD, writing test cases, and AWS certifications.

Education Requirements

Bachelor's degree in Computer Science, Information Systems, or related discipline required. Master's degree preferred.

Job description