The Wayback Machine - https://web.archive.org/web/20180524151334/https://www.dativa.com/data-pipelines/

Data Pipeline Development

Data Pipeline Development

We have built some of the world’s largest real-time data processing pipelines and and have expertise across AWS and open source technologies. We can plumb together the most complicated data systems to provide data directly to the hands of business users.

Our data pipelines are optimized for data throughput and also for queries so users no longer have to wait hours for reports to run and can provide ongoing management and optimization.

We often provide ongoing support and maintenance including proactive query optimization for users of the database to ensure your data scientists can focus on analysis rather than query optimization.

Streamlining Data Operations on AWS

With increasing demand for data engineers, it is becoming harder to recruit staff who can manage and support a data pipeline. Although cloud providers like AWS have simplified the technical operations such as managing servers and hosting facilities, the day to day operations of managing data remain. These include …

Data science under GDPR with pseudonymization in the data pipeline

The EU's new General Data Protection Regulation (GDPR) regulation that takes effect from May 25th might make it tempting to remove all personally identifiable information (PII) from your data pipeline. Deleting everything would undoubtedly provide full compliance with GDPR but it would also significantly frustrate data science efforts across …

Data science needs data operations to make it work

Our passion is putting data to work: the strategy about what you should do; the science behind how you can do it; and then the engineering to turn it into a reality. It's once we've completed the build out a data pipeline, the hard work of operating the platform …

Using Redshift Spectrum to load data pipelines

A lot of our clients use AWS Redshift as the ultimate destination for their data pipeline, and when Amazon launched Redshift Spectrum, our data engineering team wondered whether we could use this technology to provide high-performance throughput without having to load the data into Redshift at all. This article …

See the latest posts