Data Engineer

• The primary responsibility of this role will be to design, develop, and maintain data pipelines for threat intelligence ingestion, validation, and export automation flows.
• Responsibilities: Design, develop, and maintain data pipelines for ingesting threat intelligence data from various sources into our data ecosystem.
• Implement data validation processes to ensure data accuracy, completeness, and consistency.
• Collaborate with threat analysts to understand data requirements and design appropriate solutions.
• Develop automation scripts and workflows for data export processes to external systems or partners.
• Optimize and enhance existing data pipelines for improved performance and scalability.
• Monitor data pipelines and troubleshoot issues as they arise, ensuring continuous data availability and integrity.
• Document technical specifications, data flows, and procedures for data pipeline maintenance and support.
• Stay updated on emerging technologies and best practices in data engineering and incorporate them into our data ecosystem.
• Provide technical guidance and support to other team members on data engineering best practices and methodologies.
• Requirements: Proven experience as a Data Engineer or similar role, with a focus on data ingest, validation, and export automation.
• Strong proficiency in Python.
• Experience with data pipeline orchestration tools such as Apache Airflow, Apache NiFi, or similar.
• Familiarity with cloud platforms such as Snowflake, AWS, Azure, or Google Cloud Platform.
• Experience with data validation techniques and tools for ensuring data quality.
• Experience building and deploying images using containerization technologies such as Docker and Kubernetes.
• Excellent problem-solving skills and attention to detail.
• Strong communication and collaboration skills, with the ability to work effectively in a team environment.

Caderno Nacional