Overview In this advanced blog, we explain how to create a new custom machine learning node in the punch. Our goal is to predict the number of travelers in each train station of Il-de-France, the France area that includes Paris and its suburbs. We will learn the historical traffic trends Read more…
Overview Apache Nifi is a popular technology to handle dataflow between systems. Using a simple interface to create diagrams, you can manage where the data goes and how is it processed. Just like many similar frameworks, Nifi provides a bunch of built-in nodes to deal with various data sources or Read more…
You have interesting data: radar data, security logs, Iot data, pictures, and you wonder how to best combine archiving and indexing capabilities. To perform efficient queries and benefit from optimal (near) real-time performance, you need to select the right (indexed) database. In contrast; to archive lots of data for months Read more…
Dealing With Python Apps The punch story with python is an old one. We implemented an elasticsearch aggregator tool in the Brad release (now deprecated), we also leveraged the elasticsearch curator application that we provide as a deployable administrative service in the Craig release. More importantly, we decided to fully Read more…
In the end of 2016, we integrated Ceph in Punchplatform. This article gives a feedback and explains what are the major advantages of using a Ceph cluster on a Punchplatform instead of standard storage solutions. User features Aside from reducing storage costs through well-known erasure-coding, Ceph in Punchplatform offers some Read more…
Tensorflow is a well-known and one of the most popular frameworks used for Deep Learning. It is an open-source library developed by Google and it is extremely useful to solve complex problems which require deep learning. It relies on making layers of data to learn, each layer identifying features from Read more…
Introduction The Elastic Common Schema (ECS) is a new normalized format proposed by the Elastic community. Although still in beta status, it is already usable, concrete and more importantly promising. The ECS idea is simple: benefit from a common specification to structure the data indexed in Elasticsearch. Such a data Read more…
The punch is deployed in production for small, medium or large-scale applications, on the cloud or on premise, for various use cases such as cybersecurity, applicative or system monitoring. In this short blog we highlight a few concrete sizing example to give you a flavor of the hardware and server Read more…
In the punch, we do not use logstash for high-performance logs parsing. Why is that? Mainly because logstash is nor easily scalable nor does it provide an end-to-end acknowledgment pattern. This is a serious lack because we cannot afford to lose logs whatever happens. We thus selected an alternative technology (apache storm) to Read more…
To process and analyze data for typical analytics or machine learning use cases, it is common to hit storage and processing power issues. If the dataset is too big to be stored on your laptop drive, if it cannot be entirely loaded into memory, you must find practical solutions. It Read more…