Address
SalaryAs a Data Engineer you will be responsible for building ETL flows on Dataiku as well as the creation of tools (python-based) on Dataiku to allow the business to use a self-service ETL platform. You will leverage your strong python skills to build and use workflow orchestration processes on Dataiku, prior experience with other workflow orchestration tools such as Apache Airflow would be beneficial. These processes will require modern development standards such as Test-Driven Development, use of microservices and consideration of data governance/lineage.
The role will involve the creation and management of CI/CD pipelines so prior experience with GIT and version management is required. Also, use of Big Data tools, particularly the Apache stack (Impala, Hive, Spark) for ingesting data to and from Data Warehouses and Data Lakes. Prior experience with cloud services is required, particularly AWS, part of the role will involve interacting with S3 buckets and so knowledge of the CLI/bash scripting and general file management best practices. Is necessary Given that much of the role involves integrations with Dataiku, mastery of this platform will be a core function.
To be successful in this role you will have:
- Experience with python including libraries such as pandas and pyspark as well as testing frameworks, linting and use of object-oriented functionality such as classes
- Experience with SQL, particularly on large datasets
- Experience with the CLI/bash scripting language
- Experience with Big Data and knowledge of data management techniques eg partitioning, file compression
- Experience with Big Data technologies such as Apache Spark, Impala and Hive
- Experience with cloud platforms, preferably AWS but including Azure and GCP
- Management of CI/CD processes (eg Jenkins, Bamboo, CircleCI, TeamCity) and use of Git version management such as Bitbucket, GitHub, Gitlab etc
- Understanding od DevOps concepts and when to apply them to data transformation (or DataOps)
- Familiarity with agile methodology, particularly the use of Jira and Confluence, particularly when working across several large projects
- Good communication and stakeholder management skills
About the Corporate Operations Group
The Corporate Operations Group brings together specialist support services in Digital Transformation & Data, Technology, Operations, Human Resources, Business Services, Business Improvement & Strategy, and the Macquarie Group Foundation. We deliver service excellence to ensure Macquarie is open for business, deliver on transformational change, invest in our people and have deep relationships with our customers.
Our commitment to Diversity and Inclusion
The diversity of our people is one of our greatest strengths, and in combination with our inclusive environment, it enables us to deliver innovative and sustainable outcomes for our people, clients, shareholders and communities. From day one, you'll be encouraged to be yourself and supported to perform at your best. If our purpose of 'empowering people to innovate and invest for a better future' is as inspiring to you as it is to us, please apply. With the right technology, support and resources, our people can work in a range of flexible ways.
We are committed to providing a working environment that embraces and values diversity and inclusion. We encourage candidates to speak with a member of our recruitment team if you require adjustments to our recruitment process to support you, and the type of working arrangements that would help you thrive.
#LI-hybrid
About Macquarie
A career at Macquarie means you'll have the opportunity to develop and utilise new skills, explore interesting fields and do challenging work that will impact the lives of people around the world-whether it's accelerating the green energy transition, helping sustain global food supplies, financing social housing projects or investing in essential infrastructure. At Macquarie, we're empowering people to innovate and invest for a better future.
Our size and international presence means your work can take you anywhere-across business groups, disciplines, sectors and borders. With 18,000+ employees and offices in 33 markets around the world, we're a truly global organisation.
You'll be supported by a diverse team where the unique perspectives, ideas and experiences that all of our people bring are valued. You'll be empowered to address unmet needs in our communities whilst advising and investing alongside our clients and partners. Working with us, you'll have the opportunity to make a difference.
Macquarie is a global financial group providing clients with asset management, retail and business banking, wealth management, leasing and asset financing, market access, commodity trading, renewables development, specialist advisory, capital raising and principal investment.
Find out more at
Job no: COG-959419
Work type: Permanent - Full time
Location: Melbourne, Sydney
Category: Mid-senior, Data & Analytics, Technology
Group: Corporate Operations Group
Division: Technology
Recruiter: Kymberley Weir
Opening Date: 24/11/2022
We will email you new jobs that match this search.
Great, we can send you jobs like this, if this is your first time signing up, please check your inbox to confirm your subscription.
The email address was invalid, please check for errors.
You must agree to the privacy statement Subscribe Recaptcha
Privacy agreement
Search results
Multiple locations
Build ETL flows on Dataiku as well as the creation of tools (python-based) on Dataiku to allow the business to use a self-service ETL platform.
Current opportunities
Multiple locations
Build ETL flows on Dataiku as well as the creation of tools (python-based) on Dataiku to allow the business to use a self-service ETL platform.
Campaigns
Loading...
© Macquarie Group Limited / Script for LinkedIn Tag and Instructions Below:/
