Packaging Python

Dealing With Python Apps The punch story with python is an old one. We implemented an elasticsearch aggregator tool in the Brad release (now deprecated), we also leveraged the elasticsearch curator application that we provide as a deployable administrative service in the Craig release. More importantly, we decided to fully support pyspark and to make Read more about Packaging Python[…]

Introducing OmniSci

Before diving into leveraging OmniSci, let us quickly recap the well-known database story. Relational databases have long been the standard for storing pre-processed data. Although they cannot keep up with today’s applications requirements in terms of raw performance and volumetry, they stay by far the most popular and the most widely used databases. Their low Read more about Introducing OmniSci[…]

News From Logstash

In the punch, we do not use logstash for high-performance logs parsing. Why is that? Mainly because logstash is nor easily scalable nor does it provide an end-to-end acknowledgment pattern. This is a serious lack because we cannot afford to lose logs whatever happens. We thus selected an alternative technology (apache storm) to run our equivalent input/filter/output processors, Read more about News From Logstash[…]

Kafka Streams Integration

Context While working on a surveillance solution an interesting problem arose: an analysis component had to be built which had to be light, robust and highly adaptative. This component consists of a set of sequential or parallel tasks, each providing a unitary treatment. The solution is built on top of the Punchplatform the use of Read more about Kafka Streams Integration[…]

Some Thoughts on Elastic Ingest Nodes

The new elastic 5 GA is out. It provides many improvements  and great new features. In this post we have a look at the new Ingest nodes, that provide a way to deploy logstash filters without the burden of deploying and running logstash processes. Each running an input-filter-output pipeline. Since ingest node look to us Read more about Some Thoughts on Elastic Ingest Nodes[…]

Elasticsearch on Steroid

The punchplatform is deployed in a number of industrial projects. By now you hopefully know the punchplatform is built extensively on top of Elasticsearch. We run production for several years now, and some of our elasticsearch clusters are fairly big and hold up to a year of data, parsed and normalised thanks to the standard Read more about Elasticsearch on Steroid[…]

From Big to Tiny Data Platforms

Among many other useful things, big data platforms lets you design data pipeline, fetching or receiving data from one side, applying some transformations, then pushing the resulting data further. If you run a punchplatform, you are likely to do that to collect then parse your data into Elasticsearch, in turn visualizing it using Kibana.  Here Read more about From Big to Tiny Data Platforms[…]