The ScorpionTrack Fleet system is a complex feature rich product that provides thousands of customers with real time vehicle tracking and fleet management services. Millions of rows of data streams into the system everyday with over 8 billion rows of data in total.
Scorpion became a victim to its own success and the system started to creak under the enormous pressure of new vehicles being added on a daily basis. It’s initial MVP had served its purpose of finding a saleable market fit for such a product but there were daily outages that needed to be identified and rectified.
Agile, QA, Devops, Java, Gradle, Pub Sub, Kafka, Hikari, Golang, PHP, Codeigniter, Lumen, ELK, Kibana, Google Cloud, Rackspace, Serverless, Continuous Integrations, Gitlab Pipelines, Datadog, Big Data, Data Warehousing, Redis, Docker, Grafana, Percona PMM, Firebase, Protocol Buffers
Microservice architecture redesign built on a scalable cloud based infrastructure
We were tasked to overhaul the site with performance in mind to scale on demand and be as efficient and cost effective as possible. With the added requirement of keeping the existing website up whilst integrating new software with old. We also continued to build in new features to ensure the product is still competitive in the market.
Some of the standout issues we overcame where:-
Due to the sheer scale of the billions of rows data in the database, the reports were taking an inordinate amount of time to produce. We analysed the bottlenecks within the reports and set about pulling out and rewriting the queries. Smaller reports are now near instantaneous and the larger reports taking a few seconds. Some of these reports even brought down the server.
One of the features that was implemented was a Golang microservice that on the fly takes the lat lng coordinates of the vehicle along with its speed and determines whether that user is breaking the speed limit according to their local speed limit. This is then reported back to the fleet manager.
The system requires millions of address lookups from the Google Maps API for features such as reports and live telemetry which became an issue with the size of the system. The inefficient use of these was causing Google Maps to reject the lookups due to going over the query limit further exacerbating the problem. We set about understanding where the bottlenecks and how the address lookups were used. We found that most lookups didn’t need to be done unless the user specifically requested it by an action so we modified the website to only request an address when needed via an API microservice that manages the requests to Google using queues. This solved the problem with capacity to spare for continued growth.
Because of the size of the database and the ongoing stream of millions of rows of data coming into a fixed dedicated server which had no room to grow but couldn’t be taken down for maintenance due to monitoring stolen vehicles, it was a difficult problem to solve. We did this by building bespoke migration software that handled the transfer of data in a managed way.
To test effectively we developed a simulator that could replicate trackers in the wild. This could be spun up with thousands of phantom trackers to load test the infrastructure and imitate peak times such as rush hour.
Due to the expansion of the fleet product to cater for custom solutions along with advanced mobile apps an API was developed and retrofitted to the existing system. This required rewriting legacy code to handle the modern API stack.
We implemented a sophisticated monitoring solution that could let us understand what was happening to the system over time periods and if there was any specific bottlenecks. Once an incident took place we then investigated and put in place precautions to prevent it happening again.
iOS & Android Apps
100% web based application can be accessed online 24-7 from any location with our IOS and Android apps. This means no software downloads/installation.
With our mobile Apps ScorpionTrack has full support for hand held tablet devices such as the Apple iPad and Android. Access to real-time information allows you to instantly locate and track vehicles from anywhere, at any time.
The product is being methodically broken down into microservices to cater for expansion and modular nature to provide the ability to add new products seamlessly.