ASD4,5&6 Data Engineer and Data Analytic Engineer$77,787.84 – $147,915.36 (plus 15.4% super)
Melbourne CBD - VICAbout the TeamASD is entering an exciting phase of growth, innovation and transformation. Our agency has committed to delivering REDSPICE, which is the most significant single investment in the Australian Signals Directorate’s 75 years.Mission Data (MD) Division within the Australian Signals Directorate (ASD) is responsible for providing the foundational data capabilities to support all of ASD's functions. We do this via the provision of data platforms and services, management of ASD's mission data, and providing complex analytic development and support, including Artificial intelligence and Machine Learning capabilities.The RoleData Engineer:Key duties of the position include but are not limited to:
- Design and implement complex data flows to connect operational systems, together with data for analytics and business intelligence systems, whilst ensuring data security and governance.
- Manage the design, development and maintenance of enterprise data warehouse and Data Engineering solutions.
- Work with stakeholders to assemble large, complex data sets that meet their current and potential requirements.
- Manage the build of data streaming systems
- Perform root cause analysis on data and processes to answer specific business questions and identify opportunities for improvement.
- Manage the development of Data Engineering frameworks and best practices for processes aimed at enhancing the availability, security and performance of data assets
- Identify and undertake detailed analysis to identify solutions for a range of complex data problems, including gathering and exploring data and meta-data from a variety of sources.
- Build and sustain strong relationships and negotiate with customers and key stakeholders to deliver tailored recommendations.
- Develop and maintain business domain knowledge, data analysis and visualisation skills.
- Work collaboratively, operates as an effective team member and actively participates in organisational activities.
- Experience managing Hadoop platform
- Software development experience in Python or Java preferably
- Experience with streaming technologies Kafka, Spark, Flink
- Cloud Native technologies and Infrastructure as Code
- System administration in Linux
- Experience designing complex data solutions (ASD 6)
- Experience managing NoSQL databases Elasticsearch, Splunk, MongoDB, Neo4J
- Experience managing NiFi and NiFi flows
- Tertiary qualifications in Data Analytics, Computer Science or a related field, or equivalent work experience.
- Demonstrated ability to apply data analysis principles and practices – exploratory data analysis, data cleaning and Data visualisation.
- Knowledge and experience in development languages. We predominantly use Python but encourage applications from by people with experience in other development languages.
- Exposure to software engineering practices (code reviews, source control management and testing).