Packaging Python

Dealing With Python Apps The punch story with python is an old one. We implemented an elasticsearch aggregator tool in the Brad release (now deprecated), we also leveraged the elasticsearch curator application that we provide as a deployable administrative service in the Craig release. More importantly, we decided to fully support pyspark and to make Read more about Packaging Python[…]

Leveraging the Elastic Common Schema

Introduction The Elastic Common Schema (ECS) is a new normalized format proposed by the Elastic community. Although still in beta status, it is already usable, concrete and more importantly promising. The ECS idea is simple: benefit from a common specification to structure the data indexed in Elasticsearch. Such a data normalization makes it simple and Read more about Leveraging the Elastic Common Schema[…]

Introducing OmniSci

Before diving into leveraging OmniSci, let us quickly recap the well-known database story. Relational databases have long been the standard for storing pre-processed data. Although they cannot keep up with today’s applications requirements in terms of raw performance and volumetry, they stay by far the most popular and the most widely used databases. Their low Read more about Introducing OmniSci[…]

News From Logstash

In the punch, we do not use logstash for high-performance logs parsing. Why is that? Mainly because logstash is nor easily scalable nor does it provide an end-to-end acknowledgment pattern. This is a serious lack because we cannot afford to lose logs whatever happens. We thus selected an alternative technology (apache storm) to run our equivalent input/filter/output processors, Read more about News From Logstash[…]

Scaling with PML

To process and analyze data for typical analytics or machine learning use cases, it is common to hit storage and processing power issues. If the dataset is too big to be stored on your laptop drive, if it cannot be entirely loaded into memory, you must find practical solutions. It is, of course, possible to Read more about Scaling with PML[…]