Data Engineer

Join a multifaceted organisation working in interesting areas like GIS, Telematics and BIG data + have the chance to get involved in some exciting projects already in the works.

Affix is pumped to be working again with this organisation that not only provides data services allowing the exchange of information in their specific industry, but also provides advice, information, tools and a selection of services to help its members deliver to their customers. Key partners include all levels of government, private sector and research institutions providing real societal impact and benefit.

About the Role

This is a full time permanen Data Engineer position. You will assist in the design, development, enhancement, and maintenance of complex data pipelines products that manage business critical operations, and large-scale analytics pipelines.

They have exciting current project work to sink your teeth into too including a full scale Databricks implementation due to go live early next year (based on their increasingly scaling data needs and a successful POC).

Reporting to the Enterprise Data Manager the Data Engineer will work on Big Data systems design focusing on data architecture in their Databricks ecosystem and act as go between between the data team (Data Analysts, Data Administrators, GIS Analysts and Business SMEs) and the contracting vendor for the Databricks implementation itself.

Who we’re looking for

• Experience implementing Big Data solutions in a cloud ecosystem e.g. Data/Delta Lakes
• Experience with Big Data pipelines or DAG Tools such as Azure Data Factory, Spark (especially Databricks, Spark and Delta Lake)
• Hands-on Python/Pyspark/SparkSQL experience; Scala experience is a plus
• Experience with source control (GIT) on the command line
• Experience in writing complex, highly optimised SQL queries across large volumes of data
• Exposure to an Agile Scrum/Kanban environment + delivery-based test/release approach

Nice-to-haves

• Data modelling/Data profiling capabilities with Kimball/star schema methodology
• Experience with Kafka, or other live data streaming technology
• Experience with database deployment pipelines
• Experience with one or more unit testing or data quality frameworks
• Experience in working with Telematics Data and Road Maps
• Experience with conceptual, logical and/or physical database designs

Culture

This is an organisaton that values true innovation and agile ways of working. They care about their team and their personal development and offer benefits like time for education and learning budgets, wellness budgets and bonus additional leave.

Next steps?

Sound likes what you’ve been waiting for? Then, let’s chat! Apply now or contact Cass@affix for any questions: *************@affix.com.au