Data and Analytics

Data is the core of modern business. Databases themselves were originally created because data always outlives the current application and use cases. This has never been more apparent than in these times of rapid innovation in analytics, data science, and data visualization. Users expect their data to be at their fingertips on all devices, up to date, and explorable in many ways. Gone are the days of trying to solve every problem with a monolithic relational database. We choose the right solution for the problem - relational, document, analytical/big data, graph, textual, NoSql, time series

Get Started

Services

Analytics

Transforming data into valuable insights has been one of the most innovative and rapidly changing areas of technology. Data Lakes can accommodate structured, semi-structured, and unstructured data but require planning and management to avoid becoming “data swamps”. We can help you find the right choices from the overwhelming amount of data governance, reporting, visualization, and analysis tools

Data Engineering

Data movement is complex in modern distributed architectures. To keep up, in addition to traditional ETL, data engineering can include streaming, pipelines, mining and Big Data components. This requires an “everything as software” approach known as DataOps. We can ensure that your data engineering is flexible and maintainable.

Oracle

We are an AWS Oracle Competency Partner and have handled the migration and management of Oracle workloads from on premise to cloud as well as cloud to cloud. We provide configuration, backup, virtualization and clustering - HA, RAC, Data Guard as well as licensing and cost optimization.

Data and Analytics

To get your data aligned

Contact Form




Recent Case Studies

Edmodo


Problem Description:

Edmodo had a growing set of submissions from their bug bounty program and not enough bandwidth to process them. Developers also needed guidance in secure coding practices as security bugs due to insecure code was an issue.

Solutions Highlights:

With a large set of bugs to process, we began by organizing and prioritizing the existing security bugs based on severity and business needs. We then proceeded to review the security bugs to verify them using tools like Burpsuite and nmap. Valid security bugs were documented and assigned to Edmodo team members to be addressed. For some security bugs, we submitted code fixes and security enhancements, then tested and verified them in QA and Production environments. We produced security training material and held training sessions for developers to provide guidance on secure coding practices for Ruby on Rails and ReactJS. The topics covered common OWASP categories and findings from bug reviews. We also performed penetration tests on the Edmodo web and Android mobile application.and produced findings.

Preparing for potential security incidents and risks is a crucial part of building a mature security ecosystem. Security incidents can have serious material and reputational impact. Effectively designed and documented security standards can mitigate these risks and provide a consistent process to respond to incidents quickly. Depending on the industry, many companies and organizations are subject to audit to verify established, documented, and complied with security standards.  

Establishing such documentation can be complicated and requires broad collaboration. We  began by conducting a comprehensive review of existing security documentation for the company. After which, we collaborated with technical leadership to integrate a security documentation process to their Secure Development Lifecycle Process (SDLC) process. Engagement with teams and attending scrum meetings was beneficial for staying aware of current and upcoming features and designs that impact security. We collaborated to produce security standards and documentation for the company. These documents include, but are not limited to Security Risk Assessments and Product Requirement Documents.


View Case Study Details

VTS


Problem Description:

VTS Rise provides modernized tenant experience solution and enables asset and property managers to power their portfolio wide operation and provide a premium experience to tenants, building operators and visitors.

VTS has native mobile applications (iOS, Android) as well as web applications hosted on AWS Cloud that continually serve its customers with seamless experience with their property management.

AVM worked with VTS to help them streamline their observability strategy, train them in understanding Datadog functionality, and guide them with best practices in monitoring and recommend improvements.

Solutions Highlights:

VTS already had a Datadog instance in use but had not instrumented their application for a robust Datadog experience. They needed someone to streamline their Datadog configuration and align it with best practices. Their expectation was to have end to end observability and performance monitoring. 

AVM reviewed their tagging strategy, their existing integration and application instrumentation and provided feedback. AVM collaborated VTS in determining a proper tagging strategy. All dashboards, monitors, and alerts were reviewed holistically. Improvements were made leveraging real user monitoring (RUM), synthetic tests, continuous logging and comprehensive service tagging. The iOS and Android instrumentation was also reviewed, resulting in code change recommendations to reconfigure log collection, linking RUM with logging, and correlating logs with traces.

AVM also provided VTS with guidelines on monitoring and template for incident management to fit into their tech operational strategy.


View Case Study Details

Amuse


Problem Description:

Amuse is an e-commerce cannabis delivery business reinventing how consumers order and consume cannabis hosted in AWS. Amuse has aggressive SLO, which needs constant monitoring of the search latencies, one of the product’s core capabilities. They get an enormous volume of search requests to search the exhaustive list of products. Sometimes the search requests fail, and it becomes difficult to capture the customer’s real customer experience. Magento stack has a very minimal out-of-box instrumentation solution. Monitoring the life cycle of messages that flow through various services is challenging.

Solutions Highlights:

AVM worked with the Amuse team to build instrumentation for the Magento stack powered by the Datadog libraries and Datadog as the observability platform. AVM enabled Amuse to instrument the Magento stack and configured the Datadog agent with the suitable parameter set.

The engineering team at AVM ensured that the spans and traces of the request were complete. Missing Datadog spans and traces were identified during troubleshooting, leading to the instrumentation code changed to fix the Datadog agent issues, and enriching the request context for end-to-end Datadog monitoring.

The solution enabled the Amuse team to use Datadog as a single pane glass for their entire  e-commerce platform


View Case Study Details

Allstate


Problem Description:

Allstate, a Fortune 100 Insurance company, reached out to AVM to help them mature their observability strategy and implementation.This was a key initiative of their digital transformation strategy. Allstate wanted to achieve total observability and AVM collaborated to make it happen by reviewing existing patterns, involving all teams, and working through the launch of a new product

Solutions Highlights:

Allstate was already using Datadog for observability but realized that they were not using the tools at full potential. They wanted actionable dashboards that would allow their teams to quickly source and mitigate issues. This included end to end performance monitoring leveraging real user monitoring (RUM), synthetic tests, logs, application and infrastructure monitoring to trace customer requests through the entire architecture including cloud services, kubernetes and asynchronous, Kafka based, integrations. These all added to the complexity. AVM was able achieve end to end traceability and created comprehensive dashboards.

The initial priority was production to reduce the number of critical incidents. AVM then went beyond the production environment to include test and dev environments and CI/CD pipelines in order to catch issues as early as possible. Chaos testing was introduced to detect gaps in both the architecture and the monitoring.

AVM conducted a full audit of existing dashboards, monitors and alerts then worked with the Allstate team to focus them on what was meaningful and actionable. This required AVM’s extensive knowledge of cloud native AWS solutions, messaging and service based architectures, and observability best practices. AVM wanted to ensure that there were tagging standards to aid in creating and navigating dashboards and metrics. These standards needed to account for many observability lens – application performance, security, cloud spend, devops metrics


View Case Study Details

PriceSpider



Problem Description:

PriceSpider provides a commerce platform for brands to drive conversion, track performance, monitor retailers, monitor digital commerce, ensure guideline compliance, and take action on all these inputs unifying brand marketing and performance marketing. Their clients, including many of the world’s top brands, rely on them for reliable, timely data.

PriceSpider has a suite of products that are logically separated and deployed at different both AWS and GCP clouds. Crawler services continuously produce millions of data every day and these are consumed by multiple applications.

Due to the highly distributed nature of the platform. It’s very challenging to monitor the life cycle of messages that flow through different services.


Solutions Highlights:

AVM worked with PriceSpider team to build a shared data platform that uses Kafka as the core messaging hub and Datadog as the observability platform. All the products within the company were analyzed to understand observability requirements concerning the shared data model.

The engineering team at AVM proposed a correlation model for distributed components along with a proof of concept. Furthermore, custom metric exporters were introduced to monitor managed services that do not provide built-in support for Datadog monitoring.

The proposed distributed observability model helped PriceSpider team to simplify the platform complexity and use Datadog as a single pane glass for their multi-cloud platform.


View Case Study Details

AIRR



Problem Description:

Artificial Intelligence Readiness Roadmap is a self-assessment platform from Incitelogix for businesses to evaluate the readiness of the information infrastructure to invest in Artificial Intelligence projects.

The platform has been built as a 100% serverless application running in AWS. It has a ReactJS based frontend and a NodeJS based Restful backend which uses DynamoDB as the persistent storage.

As a self-serve platform with a minimum operational footprint, it is crucial to proactively monitor the end-to-end application to ensure a better user experience and alert the engineers on errors.


Solutions Highlights:

AVM designed and developed the Artificial Intelligence Readiness Roadmap platform from the ground up following the industry best practices for observability. Datadog was selected as the observability platform, and both frontend and backend instrumented with Datadog RUM and APM for monitoring the end-to-end lifecycle of requests. Synthetic tests and monitors were configured to make sure the service level objectives are met.

Datadog dashboards show the health of the entire platform for both business and support teams. And the engineering team can proactively monitor the performance of the application and improve where necessary.

Overall Datadog helped Incitelogix to lower the MTTR by correlating all synthetic tests to backend metrics, traces, and logs.


View Case Study Details

Pricespider



Problem Description:

PriceSpider provides a commerce platform for brands to drive conversion, track performance, monitor retailers, monitor digital commerce, ensure guideline compliance, and take action on all these inputs unifying brand marketing and performance marketing. Their clients, including many of the world’s top brands, rely on them for reliable, timely data.

PriceSpider processes tens of millions of business events per day. Their Engineering and  Business Ops teams make use of Kibana based dashboards to ensure these events are flowing correctly and customer changes are visible quickly.

In order to onboard new customers and fulfill customer feedback critical events must appear in these dashboards as soon as possible. They were seeing cases where some (but not all) events were severely delayed impacting multiple customers each day. They found nothing in common about these events or any root cause.


Solutions Highlights:

AVM worked with PriceSpider’s Platform Engineering team on understanding the event pipeline  that included AWS Lambda, Amazon Kinesis Data Firehose, and Amazon ElasticSearch Service and its Kibana as well as establishing the KPIs important to all stakeholders.  Based on that, AMV Streaming experts then performed an end to end analysis of the component configurations and historic performance using CloudWatch to pinpoint the problem areas and inefficiencies.

Within weeks AVM distilled many separate issues in the pipeline into new resiliency strategies in Firehose and ElasticSearch that exceed the performance SLAs 99.99% of the time, improved observability and alerting of the complex pipeline, and proposed developing additional Lambda based tools to evolve the pipeline to a self-healing system. In addition, AVM found cost savings of over 50% with a more resilient architecture.


View Case Study Details

Edmodo


Problem Description:

Organized, sophisticated and persistent cyber-threat-actors pose a significant challenge to large, high-value organizations. They are capable of disrupting and destroying cyber infrastructures, denying organizations access to IT services, and stealing sensitive information including intellectual property, trade secrets and customer data. SMB organizations are often challenged by incident response management, in part because incident response procedures may not be established.

Therefore it’s critical for an organization to identify and respond to security incidents and events in a timely manner. Whether a breach is small or large, organizations need to have an incident response policy in place to mitigate the risks of being a victim of the latest cyber-attack.


Solutions Highlights:

Edmodo is an educational technology company offering a communication, collaboration, and coaching platform to K-12 schools and teachers. The Edmodo network enables teachers to share content, distribute quizzes, assignments, and manage communication with students, colleagues, and parents. Responding to any security incident is a critical element for Edmodo’s businesses and data security compliance requirement. It’s also essential for Edmodo to identify and respond to security incidents and events in a timely manner. Whether a breach is small or large, Edmodo wanted to have an incident response plan in place to manage the lifecycle (preperation, detection & analysis, containment, eradication & recovery, and post incident activity) of all security Incidents. The faster they detect and respond to security incidents, the less likely it will have a significant impact on their data, customer trust, reputation, and a potential loss in revenue.

We undertook a comprehensive analysis of their existing policies in place, their current team structure, and security incidents happened in the past and their preparedness to handle any future security incidents. We evaluated NIST SP 800-61 & ISO/IEC 27035 standards and based on their existing org structure and specific need, we decided to go ahead with creating security incident response policy based on NIST SP 800-61 standards.

Performing incident response effectively is a complex undertaking, establishing a successful incident response capability requires substantial planning and resources. Continually monitoring for attacks is essential and establishing clear procedures for prioritizing the handling of incidents is critical, as is implementing effective methods of collecting, analysing, and reporting data.

We created a set of practices, processes, and solutions that enabled Edmodo’s Security Incident Response Team (SIRT) in rapidly detecting incidents, minimizing loss and destruction, mitigating the weaknesses that were exploited, and restoring IT services in the shortest possible time.
With incident response policy in place Edmodo’s SIRT team is now able to quickly detect, investigate, address vulnerabilities and issues, and respond to all IT security incidents in an efficient and timely manner. Faster responses helped them reduce the overall impact of incidents, mitigate damages, and ensure that systems and services continue to operate as planned.

Without incident management, an organization may lose valuable data, experience reduced productivity and revenues due to downtime, or be held liable for breach of service level agreements (SLAs). Even when incidents are minor with no lasting harm, IT teams must devote valuable time to investigating and correcting issues.


View Case Study Details

AIRR


Problem Description:

Artificial intelligence has become a must have strength for almost all organizations to be sustainable in the business. Some enterprises have already embraced it and others are planning to invest and included it in their roadmaps. However still the success rates of these investments are at a low level. This is mainly because the information infrastructure of the organizations is not ready for artificial intelligence projects.

Incite Logix has a well experienced team who can help organizations to be successful in artificial intelligence related implementations. They have gathered intellectual knowledge in this problem domain after successfully completing several projects at different organizations. However it’s a lot of technical and non-technical jargon which is hard for someone to read and understand.


Solutions Highlights:

The AVM team joined with the Incite Logix team to find a solution to bridge the gap between the people and the intel that the company had collected. Our primary goal was to present this intel to the users in an understandable manner and self measure their information infrastructure readiness to embrace AI projects.

As a result the two teams came up with the idea to implement a web and a mobile application where the users can register themselves and measure their information infrastructure readiness through a question and answer model which provides a categorized and aggregated score with an action plan to work on.

We at AVM always work with industry leading cutting edge technologies and decided to design this solution using serverless technologies offered by Amazon Web Services. Following the client-server architecture pattern we developed the frontend as a single page application using ReactJS and deployed in AWS using the CloudFront and S3 buckets. This approach gave us the opportunity to serve the application worldwide with reduced latency. Backend application has been built on top of serverless framework using NodeJS, and has been deployed on AWS using API Gateway and Lambda functions and DynamoDB as the persistent store. We used Amazon Cognito for identity management and it simplified most of implementation efforts and gave a solid layer of security. Next the mobile applications were developed using the Flutter framework and that gave us the opportunity to implement for both IOS and Android platforms parallelly reducing a lot of development efforts. Finally, no solution can be successful without proper monitoring and telemetry. We used Amazon CloudWatch logging with alarms configured where necessary to keep the team informed of any failures. And the application usage monitoring was achieved through Amazon Pinpoint and Google Analytics.

In this way, the complete solution has been developed and deployed and a 100% serverless application with very small running cost footprint.


View Case Study Details

Match


Problem Description:

In early 2017, Match.com had become the largest online dating platform reporting over 35 Million users, with the only competitor eHarmony far from catching up with only 17.5 Million. The advent of this new romantic age that leveraged online technologies in the quest for love, brought with it a whole new category of challenges for the platform operators. The number of requests to their servers were no longer in the thousands but in the Trillions. Yet these new types of challenges facing Match, were perfectly suited to be addressed by leveraging the scale and performance benefits of cloud solutions and integrating these with traditional day-to-day IT operations.

One of the first challenges faced by the company was to modernize any remaining monolithic architecture for increased performance and agility. Previously within their software system, functionally distinguishable aspects of their applications, such as data I/O, processing, error handling and user interfaces, were interwoven rather than being isolated into separate architectural components. Other bottlenecks and issues included the elastic demand capabilities of their web servers, and the high capital expenses of provisioning new resources for the on premise data centers.


Solutions Highlights:

In order to facilitate performance improvements and greater agility we conceptualized and implemented a full service end-to-end cloud migration and adoption strategy based around the cloud services offered by Google (GCP) and Oracle (OCI). First, we helped them re-architect their existing infrastructure and applications into a suite of independently deployable, modular microservices. As such each application runs a unique process and communicates through a well-defined, lightweight mechanism. With the help of Docker Containers we helped them migrate these from their on premise locations to the Google Cloud Platform (GCP). Initially, our team used the ExtraHop platform for a continuous auto-discovery of application dependencies and to identify and map architectural requirements necessary to run these applications on GCP. This allowed us to configure and provision Match’s new cloud-based VM environment in a way that would optimally serve the needs of their applications.

Furthermore, we used HashiCorp’s cloud configuration and orchestration tool Terraform to spin up a highly elastic farm of Apache Web Servers in the Google Cloud, to meet the unpredictable and volatile number of requests coming from the online dating platform. This enabled Match to scale flexibly to meet demands and provided significant cost-savings by scaling down when demands were low and stable. Finally, after this initial cloud solution, Match.com commissioned us to help them migrate their database as well. Subsequently we migrated their Oracle DB from on premise to the Oracle Cloud AZ in Phoenix. This is done with the aim of maintaining and improving performance further through the utilization of Oracle’s Baremetal infrastructure. Simultaneously, we are facilitating significant Oracle licensing cost savings through the provision of dynamically scalable instances (elastic CPU scalability) and automation.


View Case Study Details

H&R Block


Problem Description:

When H&R Block came to us they were facing various challenges. Whilst already largely virtualized, their infrastructure and IT systems contained applications with many legacy features and the performance of many applications was suboptimal. Especially one of the consumer group services tended to underperform and required elastic scalability to service fluctuating numbers of consumers. Finally, as a financial services company, comprehensive and complete data security throughout their cloud solution was of critical importance and one of the main priorities for H&R Block.


Solutions Highlights:

To improve performance of the internal API Gateway and consumer group service, we migrated it to AWS using Terraform for the infrastructure as a code. However, the migration required a lot of planning and analysis, as there were complex multi-dependencies that had to be discovered and mapped out, and many legacy features that needed to be removed.

Furthermore, as this consumer group service dealt with financial data and customers’ private information, overall data security within the cloud solution was of paramount importance. Therefore, it was necessary to ensure our solution design guaranteed in transit and at rest data encryption of the highest standards.

We tackled this challenge by establishing permissions which followed the AWS Security principle of least-privilege. This allowed us to minimize the blast radius and drive the Recovery Point Objective (RPO) and Recovery Time Objective (RTO) to down under an hour.


View Case Study Details

Tickets.com


Problem Description:

As a platform, Tickets.com must handle a large number of requests and manage many varied integration services from multiple third party vendors who are selling tickets through Tickets.com. This multi-party integration of different APIs posed significant problems for setting up and managing the platform’s backend.

Furthermore, as the event industry is subject to a highly volatile and seasonal demand fluctuations, the platform needed to be able to rapidly and inexpensively scale up or down capacities depending on demand.


Solutions Highlights:

In response to these requirements, we helped Tickets.com architect an AWS based cloud solution that contained a simplified third party integration service. Additionally we designed and provisioned elastic RDS services in AWS to help them manage their volatile loads.

These solutions effectively helped Tickets.com gain the scalability and elastic capabilities they needed for future growth of the platform, and helped the company minimize costs, risk, and down time by gaining stability for their services because the predictable performance of the AWS cloud was leveraged.


View Case Study Details

MariaDB


Problem Description:

Made by the original developers of MySQL, MariaDB has become one of the most popular database services on offer today, committed to staying open-source. They wanted to benchmark their column oriented data warehouses with the Greenplum Massively Parallel PostgreSQL (MPP) database, an open-source platform for analytics, machine learning, and artificial intelligence.


Solutions Highlights:

Since data warehouses are designed with scale and volume in mind, the benchmark had to be completed with a large scale database on a cluster setup. AWS was the best fit for this use case, spinning up r4.6xlarge EC2 instances and provisioning large EBS reserved IOPS storage volumes. Within these instances, MariaDB ColumnStore 1.6 and Greenplum 5.11 clusters consisted of four nodes each were configured.

AVM carried out an extensive study of both data warehouse engines. This allowed us to fine tune the appropriate DB parameters for a fair comparison between these engines. We decided to use Star Schema Benchmark tool (TPC-H) defined by the TPC organisation for this project. Benchmarking scripts and datasets for the two platforms were designed. Usage and performance metrics were captured and monitored using AWS CloudWatch. Final TCP-H compliant reports were delivered to the client. As it was large scale compute load, during the whole project we made sure cost is closely watched so client can get the best ROI.


View Case Study Details

eHarmony


Problem Description:

eHarmony is one of the most advanced development firms we have seen: they have followed an agile methodology for years, being one of our first customers to run CI/CD pipelines. Each development team runs their development in parallel development environments in AWS, which they call runaways. They later merge all work from these runways to trunk and deploy to production.

The critical showstopper to making this process fully automated was their Oracle database, a key component of the architecture. The DB is an exceptionally powerful RDBMS, which did not have much tooling around release automation.

eHarmony required a solution which can clone the Oracle DB, mask its data, then apply or rollback an incremental set of release-specific changes in a repeatable, automated way. This would allow them to roll the DB forward and back in a finely grained fashion.


Solutions Highlights:

AVM adopted AWS Elastic Block Storage (EBS) snapshots to clone the Oracle DB data before spinning up any new runways. We applied a set of custom scripts to mask and map the DB data. This is a fairly standard industry solution, which became more interesting later when we started to address incremental changes.

Previously scripts would be applied to the DB manually: no history on these scripts was managed, so it was not possible to reproduce previous activity. We redesigned the DB release process, creating a framework to apply changes in a structured way. This framework consisted of Golang Core and Bash scripts.

Within the framework, DB changes are first pulled from GitHub and analysed against a set of pre-checks. If these checks passed, the framework applied its scripts to the DB and logged any activity in the GitHub repository, tracking the status of execution. A rollback script was also created as part of the process.

With a repository and an audit trail of all DB changes at our disposal, we were able to deliver two new features: the first is being able to roll the DB forward and back, and the second is to roll the DB in a selective fashion. In other words, we could choose which runways to apply which branch scripts to. This allowed us to introduce a revolutionary merging technique to the DB, which worked alongside git branch merges. We designed a wrapper around it, using Ruby on Rails following eHarmony’s request, and plugged it into the CI/CD pipeline as a Jenkins and CodePipeline job.

This is one of the case studies we are most proud of. Have you found yourself in a situation when you have a standard ask, which you would think others have already faced thousands of times before? Ever found yourself in the position where you do an online search, and you see everybody is asking the same question, but there is no answer? In this engagement with eHarmony, we addressed exactly this situation, creating a product that the world was looking for and which can be reused to help many, many more customers going forward.


View Case Study Details

Felix


Problem Description:

Felix delivers intelligent local advertising, connecting businesses with the right customers. They wanted to migrate their on-premises Percona MySQL database to AWS, taking this opportunity to upgrade their DB engine from MySQL 5.0 to 5.7 along the way.


Solutions Highlights:

AVM explored a range of options for the migration to AWS – including Aurora, EC2, and RDS. Data Migration Service (DMS) was also considered for this project. We compared an abundance of options for Felix, ranging from Percona MySQL to community editions of MySQL. Felix benefited hugely from Percora’s threading functionality. Since RDS uses the community flavour, we decided to rule this service out.

The biggest challenge in this project presented itself in the upgrade from MySQL 5.0 to 5.7. This had to be a multi-step migration, since the database was 3TB in size and a downtime of at most one hour was permissible. We did an extensive study for the upgrade, establishing how to speed this up and provide minimal downtime.

We settled for Percona MySQL on EC2, tweaking the instance type and IOPS for optimal performance. To accelerate the DB engine upgrade, we adopted MySQL’s mysqldump functionality with custom parallel execution, choosing this over native data migration tools because it could better handle ref constraints and triggers.


View Case Study Details