Overall Experience level:
7 years of recent GCP experience
Apache Hudi for big data processing and storage
7 years of handson experience Hadoop Hive or Spark Airflow or a workflow orchestration solution
Experience with programming languages: Python Java Scala etc.
Experience with scripting languages: Perl Shell etc.
is looking for a highly energetic and collaborative Data Engineer for a 12month engagement.
Responsibilities:
Responsibilities
Design develop and maintain robust and scalable ETL workflows and data pipelines using tools like Hive Spark and Airflow.
Implement and manage data storage and processing solutions using Apache Hudi and BigQuery.
Develop and optimize data pipelines for structured and unstructured data in GCP environments leveraging GCS for data storage.
Write clean maintainable and efficient code in Scala and Python to process and transform data.
Ensure data quality integrity and consistency by implementing appropriate data validation and monitoring techniques.
Work with crossfunctional teams to understand business requirements and deliver data solutions that drive insights and decisionmaking.
Troubleshoot and resolve performance and scalability issues in data processing and pipelines.
Stay updated with the latest developments in big data technologies and tools and incorporate them into the workflow as appropriate.
Required Skills and Qualifications
Proven experience as a Data Engineer preferably in a big data environment.
Expertise in Hive Spark and Apache Hudi for big data processing and storage.
Handson experience with BigQuery and Google Cloud Platform (GCP) services such as GCS Dataflow and Pub/Sub.
Strong programming skills in Scala and Python with experience in building data pipelines and ETL processes.
Proficiency with workflow orchestration tools like Apache Airflow.
Solid understanding of data warehousing concepts data modelling and schema design.
Knowledge of distributed systems and parallel processing.
Strong problemsolving skills and ability to work with large datasets in a fastpaced environment.
| Job Description: Expectations from this role: Act creatively to develop applications and select appropriate technical options optimizing application development maintenance and performance by employing design patterns and reusing proven solutions account for others developmental activities 1. Interpret the application/feature/component design to develop the same in accordance with specifications. 2. Code debug test document and communicate product/component/feature development stages. 3. Validate results with user representatives; integrates and commissions the overall solution 4. Select appropriate technical options for development such as reusing improving or reconfiguration of existing components or creating own solutions 5. Optimises efficiency cost and quality. 6. Influence and improve customer satisfaction 7. Set FAST goals for self/teamTypical performance measures: 1. Adherence to engineering process and standards (coding standards) 2. Adherence to project schedule / timelines 3. Number of technical issues uncovered during the execution of the project 4. Number of defects in the code 5. Number of defects post delivery 6. Number of non compliance issues 7. On time completion of mandatory compliance trainingsPerformance Areas: Code as per design Follow coding standards templates and checklists Review code for team and peers |
...bonuses, various shifts to fit your lifestyle and opportunities for career growth. Avera is currently seeking Physical Therapy Assistant (PTA) looking to deliver the best possible care in Averas patient-focused settings. Position Highlights: Paid Time Off (PTO...
The VP of Mission Critical Services focuses on fundamental business issues managing risk, increasing revenues, controlling costs. Provides direct management oversight for MCS division, clients and subcontractors in their assigned area. Additionally, this role provides...
...telemarketing, and sales interested in a Work From Home position. This role is performing Lost/... ...Off-Hours Availability: This job is 100% Nights, 100% Weekends, and 100% Holidays. This... ...from home Schedule: Evening shift Holidays Night shift Weekend availability...
...when they are taught and mentored by a diverse staff. The Match Associate Teacher (MAT) Program is designed to quickly and effectively... ...for classrooms, as needed QUALIFICATIONS A Bachelor's degree required; A demonstrated commitment to improving education outcomes...
...please visit Human Resource Compensation Page Family - Public Safety Sub-Family - Police & Security Career Track / Level - O... ...to activities of assigned areas. Assists University Police Officers when necessary and relieves officers in non-criminal matters....