Company: Broadbase Human Resources
Job description: We are seeking a Data Engineer. Data Engineers work with various security system data owners to automate data integration and collection strategies. Work closely with the data science team to ensure data cleanliness and accuracy.
- Support data science team by designing, developing and implementing scalable ETL process for disparate datasets into a Hadoop infrastructure
- Design, develop, implement and maintain data ingestion process from various disparate datasets using StreamSets (experience with StreamSets not mandatory)
- Develop processes to identify data drift and malformed records
- Develop technical documentation and standard operating procedures
- Improve responsiveness and overall performance of the data ingestion pipeline architecture
- Assess, prioritize, and size features
- Contribute to a cross functional agile team
KNOWLEDGE SKILLS AND ABILITIES:
- Working knowledge of entity resolution systems
- Experience with messages systems like Kafka
- Experience with NoSQL and/or graph databases like MongoDB or ArangoDB
- Any of the following databases: SQL, MongoDB, Oracle, Postgres
- Working experience with ETL processing
- Working experience with data workflow products like StreamSets or NiFi
- Working experience with Python RESTful API services, JDBC
- Experience with Hadoop and Hive/Impala
- Experience with Cloudera Data Science Workbench is a plus
- Understanding of pySpark Leadership experience
- Creative thinker
- Ability to multi-task
- Excellent use and understanding of data engineering concepts, principles, and theories
Expected salary: $94000 – 118000 per year
Location: Chantilly, VA
Job date: Sat, 07 Aug 2021 22:02:45 GMT
Apply for the job now!