
Data Engineer (US)
AVP VIGILANT TECHNOLOGY PVT LTD
Job description
Data Engineer
Experience:
2–7 Years
Locations:
New York, NY | San Francisco, CA | Seattle, WA | Dallas, TX
Salary:
$110,000–$185,000 per year
Employment Type:
Full-Time
About The Role
We are seeking a skilled
Data Engineer
to design, build, and maintain scalable data pipelines and data platforms. The ideal candidate will have strong experience with
Python, SQL, Apache Spark
, data processing, ETL/ELT, and cloud technologies.
Key Responsibilities
-
Design, develop, and maintain scalable data pipelines and ETL/ELT workflows.
-
Build high-performance data processing solutions using Python and Apache Spark.
-
Write optimized SQL queries for data transformation, analysis, and reporting.
-
Develop batch and real-time data processing pipelines.
-
Integrate data from databases, APIs, cloud platforms, and other sources.
-
Ensure data quality, accuracy, security, reliability, and availability.
-
Optimize data pipelines and processing jobs for performance and scalability.
-
Collaborate with data scientists, analysts, software engineers, and business teams.
-
Monitor, troubleshoot, and resolve data pipeline and production issues.
-
Participate in code reviews, Agile/Scrum ceremonies, and technical documentation.
Required Qualifications
-
2–7 years of professional Data Engineering experience.
-
Strong programming skills in Python.
-
Advanced knowledge of SQL and relational databases.
-
Hands-on experience with Apache Spark / PySpark.
-
Experience developing ETL/ELT pipelines and data workflows.
-
Understanding of data warehousing, data modeling, and distributed data systems.
-
Experience with Git and modern software development practices.
-
Strong analytical, problem-solving, and communication skills.
Preferred Skills
-
Experience with AWS, Azure, or Google Cloud Platform.
-
Knowledge of cloud data services such as S3, Redshift, BigQuery, Snowflake, or Databricks.
-
Experience with Airflow or other workflow orchestration tools.
-
Knowledge of Kafka or other streaming technologies.
-
Experience with Docker and Kubernetes.
-
Familiarity with CI/CD and DevOps practices.
-
Understanding of data governance, security, and privacy best practices.
Benefits
-
Competitive salary: $110K–$185K
-
Medical, dental, and vision insurance
-
401(k) and retirement benefits
-
Paid time off and holidays
-
Professional development opportunities
-
Hybrid/remote flexibility depending on employer and location
-
Collaborative and growth-oriented work environment
Equal Opportunity
We are committed to providing equal employment opportunities to all qualified candidates. Employment decisions are based on qualifications, skills, experience, and business requirements.
Skills: sql,spark,python