Job Description


Job Overview

  • Job ID:

    J55919

  • Job Title:

    Senior Data Engineer

  • Location:

    Boston, MA

  • Duration:

    8 Months + Extension

  • Domain Exposure:

    IT/Software

  • Client:

    To Be Discussed Later

  • Employment Type:

    W-2, 1099, C2C

  • Work Authorization:

    US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other

Job Title: Senior Data Engineer

Location: Boston, MA

Duration: 8 Months + Extension

Bill Rate: $76/hour (C2C)

Job Type: W-2, 1099, C2C

Client: To Be Discussed Later

Work Authorization: US Citizen, Green Card, OPT-EAD, CPT, H-1B, H4-EAD, Other

Job Description:
 
We are seeking an experienced Senior Data Engineer with strong expertise in Snowflake, Informatica, Terraform, and Azure DevOps to support the Statewide Longitudinal Data System (SLDS / COdata) initiative for the State of Colorado.
The Data Engineer will be responsible for designing, developing, implementing, and maintaining scalable ETL/ELT data pipelines that integrate data from multiple state agencies, educational institutions, and third-party partner systems into a centralized Snowflake data platform.
This position will work closely with the Data Architect, Data Scientists, Operations Engineers, and a third-party implementation partner to build and maintain reliable data ingestion processes, implement data transformations, perform data quality validation, and support longitudinal student and workforce data analytics.
The selected candidate will initially participate in a structured knowledge-transfer process, including shadowing, co-running, and independently managing existing data pipelines. Over time, the engineer will assume full responsibility for production operations, pipeline maintenance, performance optimization, and analytics-layer support.
 
Key Responsibilities:
1. Data Engineering & ETL/ELT Development
  • Design, develop, test, and deploy automated ETL/ELT pipelines using Snowflake and Informatica.
  • Integrate data from state agencies, educational institutions, and third-party source systems.
  • Develop complex SQL queries, transformations, stored procedures, and data processing workflows.
  • Build and maintain source-to-target mappings and data integration processes.
  • Ingest and transform structured data into standardized longitudinal data models.
  • Develop Informatica workflows to automate data ingestion, cleansing, and transformation.
2. Snowflake Development & Optimization
  • Develop and maintain Snowflake tables, views, schemas, and transformation objects.
  • Optimize Snowflake queries, warehouse utilization, and pipeline performance.
  • Monitor data processing workloads and resolve performance bottlenecks.
  • Implement scalable and cost-efficient data processing solutions.
  • Maintain RAW, transformation, and analytics data layers.
3. Data Quality, Governance & Identity Matching
  • Implement automated data quality checks covering:Structural and format validation.Data completeness and duplicate detection.
  • Referential integrity and business logic validation.
  • Perform data cleansing, reconciliation, and exception handling.
  • Implement record linkage and identity-matching processes across multiple agency datasets.
  • Maintain data lineage, metadata, audit trails, and data governance standards.
  • Ensure compliance with FERPA, COPPA, ADA, and applicable state data privacy regulations.
  • Implement data suppression and de-identification requirements.
4. Production Support & Pipeline Operations
  • Monitor scheduled data pipelines and ensure successful execution of recurring feed cycles.
  • Troubleshoot failed jobs, pipeline errors, and data ingestion issues.
  • Manage late, corrected, or malformed source files and perform data reloads.
  • Reconcile source and target record counts and investigate discrepancies.
  • Maintain operational runbooks, incident logs, and pipeline documentation.
  • Transition existing production pipeline responsibilities from the third-party implementation partner.
5. DevOps, Automation & Release Management
  • Develop and maintain CI/CD pipelines using Azure DevOps.
  • Utilize Git for source control, branching, pull requests, and code reviews.
  • Implement Infrastructure as Code (IaC) using Terraform.
  • Manage code promotion across Development, Test, UAT, and Production environments.
  • Develop and execute manual and automated test cases using Azure Test Plans.
  • Participate in peer reviews, defect resolution, and release documentation.
6. Collaboration & Knowledge Transfer
  • Collaborate with Data Architects to implement data models and architectural standards.
  • Work with Data Scientists and Visualization Designers to prepare datasets for Tableau dashboards and analytics.
  • Coordinate with Operations Engineers on Snowflake warehouse management, permissions, and deployment processes.
  • Participate in knowledge-transfer sessions with third-party implementation teams.
  • Gradually assume independent ownership of data pipelines and production operations.
Required Technical Qualifications
  • Snowflake: Minimum 5 years of experience developing, managing, and optimizing ETL/ELT pipelines and data transformations within Snowflake.
  • Informatica CDGC: Minimum 1 year of hands-on experience with Informatica Cloud Data Governance and Catalog (CDGC), including metadata management, data profiling, cataloging, and data quality validation.
  • Terraform: Minimum 1 year of experience with Infrastructure as Code (IaC), including provisioning and managing cloud infrastructure.
  • Azure DevOps: Strong experience developing and maintaining automated CI/CD pipelines for build, testing, and deployment.
  • Git: Strong proficiency with version control, branching strategies, pull requests, merge management, and code reviews.
  • Strong SQL development and data transformation skills.
  • Experience designing and implementing complex data ingestion and integration pipelines.
  • Experience with data quality, data validation, cleansing, and reconciliation processes.
  • Ability to troubleshoot production data pipelines and manage operational support activities.
Preferred Qualifications:
  • Experience working with state government, public-sector, or education data systems.
  • Familiarity with Statewide Longitudinal Data Systems (SLDS) or longitudinal student and workforce data.
  • Experience integrating data from multiple government agencies and external partner systems.
  • Experience with record linkage, entity resolution, and deterministic or probabilistic identity matching.
  • Knowledge of FERPA, COPPA, student data privacy, and data suppression standards.
  • Experience with Informatica ETL workflows and metadata-driven data integration.
  • Experience with Azure Test Plans and structured software testing processes.
  • Familiarity with Tableau data integration and analytics-layer development.
  • Experience working with education and workforce datasets from organizations such as CDE, CDHE, CDLE, CDEC, and National Student Clearinghouse.
Project-Specific Responsibilities:
  • Develop and maintain data ingestion pipelines for Colorado Department of Higher Education (CDHE) course-level data.
  • Support conditional ingestion of postsecondary demographic, Pell Grant eligibility, and disability-related data.
  • Integrate PSAT/SAT assessment data and perform student identity matching.
  • Support existing data feeds, including NSC, K-3, CDEC, and Phase 1 integrations.
  • Participate in knowledge transfer and operational transition from January through April 2027.
  • Support analytics-layer stabilization, data reliability improvements, and performance tuning.
  • Develop de-identified and privacy-compliant data extracts for participating higher education institutions.
Expected Deliverables:
  • Production-ready Informatica ETL workflows and Snowflake transformation pipelines.
  • Automated data cleansing and validation modules.
  • Identity matching and record-linkage routines.
  • Snowflake analytics-layer tables, views, and data models.
  • Data mapping, lineage, and dependency documentation.
  • Operational runbooks and incident management procedures.
  • Azure Test Plans, test execution records, and defect logs.
  • Release change logs and production feed reconciliation reports.

Apply Now
Equal Opportunity Employer

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.