CLIENT STORY
Large Scale Data Collection & Validation Solution
A case study on the successful implementation of a contemporary data collection & validation solution.
THE ECOSYSTEM AT A GLANCE
- This Federal Agency collects third-party data from many domestic and international sources including banks, insurance companies, small businesses and tax agents.
- A variety of data formats and transmission standards are in use.
- Data payloads range in size from a few kilobytes to hundreds of gigabytes.
- Submission frequency varies between third-party communities.
This Federal Agency collects data from numerous sources to support its compliance activities and to enable streamlined reporting of circumstances for Australian Businesses and Citizens. The timely collection and processing of data to produce a high quality, trusted information asset is critical to ensuring participation in the taxation system and a level playing field for Australians.
Over the past decade the number of data sources, the frequency of data collection and variety of data formats has grown substantially. The Agency identified a need to enhance its technology infrastructure to include a flexible, robust and performant data collection and validation platform, known as the Bulk Data Exchange (BDE). Our team collectively led the $13 millon project to implement an enterprise-class Bulk Data Exchange. This included all facets of the delivery life cycle including, but not limited to project management, scoping, technical design, build, implementation and the hyper-support period.
The team liaised with agency staff and external stakeholders including multi-national software vendors, banks, insurance companies and other government departments to ensure a fit-for-purpose and reliable solution was implemented within the project period. The Bulk Data Exchange is now a cornerstone component of the Agency’s data ecosystem, collecting the bulk of their inbound data (by volume) and enabling secure external data sharing between all levels of Australian government and select international government bodies.
THE CHALLENGES
The Agency has been progressively increasing the volume of data it collects over the past 30 years to improve its understanding of each Business’ and Citizen’s taxation circumstances. This information is often collected from third-party organisations in whatever format it is available and using whatever data transfer protocol minimised the barriers to data collection. This approach whilst maximising the available data sources resulted in a disparate suite of connections, formats and platforms that were time consuming and difficult to effectively manage.
The Agency sought to improve this situation through the creation of a single bulk data exchange platform that was capable of collecting and validating third-party data in various formats, supporting common connection protocols and providing a single administrative and technology control tower to manage key data transfer activities.
HOW WE HELPED
To support the Agency to achieve their targeted outcome the team delivered the enterprise-class BDE Platform. Our expert consultants provided the following capabilities:
- Project Leadership – project initiation and technology leadership of the $13m investment to establish the Bulk Data Exchange.
- Technology Project Management – day-to-day project management of technology components of the project.
- Bridging User Experience and Technology – working closely with client teams to translate the desired user experience into pragmatic, robust technology features, and translating technology constraints into viable business process outcomes.
- Solution Architecture Leadership – to describe the critical technology components and capabilities required for a successful solution.
- Secure Data Exchange Expertise – to design a data transfer solution that balanced IT Security Controls, Authentication Techniques and Usability to ensure confidential data assets remain protected.
- Technical Delivery Excellence – to build, validate and document a reliable, scalable solution that met all client and Australian Government requirements, including the Information Security Manual.
- Information Modelling and Validation Expertise – to analyse, document, develop and quality assure data collection specifications, validation rules, error handling and data transformations for downstream utilisation.
- Application Support and Management – to support the client to introduce the solution into its production environment, training agency support staff, and provide hyper-support during the warranty period.
Additionally, due our expertise and experience several members of our project team were subsequently re-engaged to design and deliver the Agency’s other major data collection platform, the eCommerce Platform, which focuses on direct interaction with small and medium business systems.
THE RESULTS
The Bulk Data Exchange Project has been a major success for the Agency. Within the first week of operation they received several positive unsolicited feedback items from Australian Businesses, including a statement that the solution was “the best new IT System (the Agency) has introduced in years”.
The solution has been identified as one of two long-term strategic data collection channels that will support the Agency’s ongoing requirements between now and 2024. BDE continues to be extended by the business to enable more and more data interconnections across federal, state and territory agencies, local governments and international government partners.
BDE also supports the Agency to connect securely with major private sector organisations including banks, private health funds and share registries through direct machine-to-machine technologies. Additionally, it integrates directly behind the Agency’s Web Channels to provide Australian Businesses and Tax Agents with a low barrier entry point for digital data transfer.