AddressWe are seeking a Web Scraping Services professional with the following requirements.
- Solution Requirements (Use case)
- a) The solution should be able to manage scraping tasks across over one hundred websites efficiently, scaling up and down as needed to accommodate the increased demand.
- b) It should have mechanisms in place to manage various website structures and layouts, adapting to changes without any service interruptions.
- c) The solution should support different data formats (HTML, XML, JSON) and be able to extract structured data from them.
- d) Ability to manage multiple scraping tasks simultaneously to improve efficiency without losing performance or reliability.
- e) Capability to rotate IP addresses and use proxies effectively to avoid being blocked by websites and maintain privacy.
- f) Comprehensive monitoring capabilities to track scraping jobs, detect errors, and manage them, including automatic retry mechanisms.
- g) Configuration options to customize scraping behavior according to specific requirements, such as defining scraping intervals or specifying data extraction rules.
- h) Adherence to legal and ethical guidelines, including recognizing website terms of service and robots.txt rules, to ensure compliance with data usage policies within AER.
- i) Integration and storage with I Manage (internal storage) solution to securely store scraped data.
- Functional Requirements
- a) Data extraction
- i) Ability to scrape data from distinct types of Websites, including static, dynamic, and AJAX-heavy sites.
- ii) Ability to adapt to changes in website structures and layouts with minimal or limited intervention from user.
- a) Data extraction
- iii) Support for extracting data from PDFs and other document formats accessible through web pages.
- iv) Capability to parse and extract information from structured formats like HTML, XML, and JSON.
- v) Capability to rotate IP addresses and use proxies effectively to avoid being blocked by websites and maintain privacy.
- vi) Capability to parse and extract structured data (e.g., Tables) from websites and ingest them into structured datasets.
- vii) Customizable data extraction patterns (using XPath, CSS selectors, regular expressions, etc.) such as scheduled scraping (cron jobs or similar scheduling systems) which may define scraping intervals or other extraction rules.
- viii) Support for incremental crawling to update only new or changed content.
- ix) Automated retry logic for handling failed requests due to network issues or server errors.
- x) Adherence to legal and ethical guidelines, including recognizing website terms of service and robots.txt rules, to ensure compliance with data usage policies within AER.
- b) Data processing
- i) Text cleaning and preprocessing (removing HTML tags, decoding HTML entities, etc.).
- ii) Data transformation capabilities (format conversion, date normalization, numerical formatting).
- iii) Support for multi-language content extraction and processing.
- c) Data storage
- i) Options for storing scraped data in various formats (CSV, JSON, XML, databases).
- ii) Integration capabilities with databases (SQL and NoSQL) and cloud storage services (Azure Blob Storage, AWS S3).
- iii) Mechanism for data deduplication and conflict resolution.
- d) User experience
- i) Intuitive, user-friendly, web-based interface for configuring scraping tasks and visualizing progress. The interface should be able to meet Web Content Accessibility Guidelines standard to 2.1 level.
- ii) The system should enable users to navigate through the systems with minimal clicks and wait times to navigate through the systems and perform activities. After the user has completed initial training, they should be able to reliably complete the basic tasks without further instruction (ideally no-code or low-code) and the interface should facilitate these tasks.
- iii) Access to documentation, community forums and support for user assistance.
- iv) Reporting tools for generating analytics and insights from scraped data.
- v) Alerting mechanisms for errors, completion notifications and system health-status.
- Non-functional Requirements
- a) Performance
- i) High throughput to ensure data is extracted and processed quickly.
- ii) Efficient use of resources to minimize the cost of operation on hardware and cloud services.
- a) Performance
- iii) Services should be able to cater for growth and changing preferences without impacting performance, functionality, or stability.
- b) Reliability
- i) Fault tolerance to manage system failures gracefully.
- ii) The system should be able to cater to SLA of 99.9% availability.
- iii) Data integrity checks to ensure the accuracy and completeness of the scraped data.
- iv) System shall ensure the integrity of the data. Unintentional alterations must be prevented.
- c) Scalability
- i) Ability to scale horizontally to manage increases in workload.
- ii) Flexible architecture to add new features or integrate with other services without significant overhead.
- d) Integration
- i) Ability to integrate easily with other tools using standard API integration preferably REST/SOAP APIs.
- ii) If necessary, there should be a library of pre-built integration adapters for integration with other tools to minimize development costs. Preference would be for these adapters to be available from the seller (as opposed to other third-party providers).
- e) Security
- i) All data in the system must be hosted in Australia and under Australian laws.
- ii) Cloud infrastructure should be Infosec Registered Assessor Program (IRAP) certified.
- iii) The systems and data must be protected in accordance with the Government Information Security Manual (ISM) and Protective Security Policy Framework (PSPF).
- iv) All data in the system must be owned by AER. This includes data in any non-production environments, backups, and archives.
- v) The system must enable Single Sign On (SSO).
- vi) The system must enable Role Based Access Control.
- vii) Data must be always encrypted using the Advanced Encryption Standard or similar algorithm. Alternative approaches to data encryption that provide equivalent security will be considered. Refer to the ISM for guidance on encryption requirements.
- viii) The system must keep track of actions in a format accessible to authorized users. The audit logs must not be able to be changed or deleted after creation.
- Services Requirements
- a) Implementation services that are offered directly by the seller.
- b) Maintenance and support services post-integration, including troubleshooting and best practice guidance.
Please feel free to call Manoj on 0468 492 *** or simply click on Apply now button.
