Context While working on a surveillance solution an interesting problem arose: an analysis component had to be built which had to be light, robust and highly adaptative. This component consists of a set of sequential or parallel tasks, each providing a unitary treatment. The solution is built on top of Read more…
The PunchPlatform Machine Learning (PML) sdk lets you design and run arbitrary spark applications on your platform data. Why would you do that ? Punchplatforms process large quantities of data, parsed, normalized then indexed in ElasticSeach in real time. As a result you have powerful search capabilities through Kibana dashboards. Read more…
Introduction For few weeks, the punchplatform team uses a sublimeText plugin to edit punch code (aka punchlet). First, for internal usage, we have decided to provide the plugin with each standalone from Brad 4.0.1. Video Tuturial A 10min video is available to discover the installation and first step with the Read more…
In the world of log parsing, strange acronyms are used for products (splunk, punch) and for operators (grok). At least dissect is more explicit : it allows you to efficiently cut a string into interesting sub-parts. The basic task you do to parse and normalise your data. Recently Elastic introduced Read more…
The new elastic 5 GA is out. It provides many improvements and great new features. In this post we have a look at the new Ingest nodes, that provide a way to deploy logstash filters without the burden of deploying and running logstash processes. Each running an input-filter-output pipeline. Since Read more…
The punchplatform is deployed in a number of industrial projects. By now you hopefully know the punchplatform is built extensively on top of Elasticsearch. We run production for several years now, and some of our elasticsearch clusters are fairly big and hold up to a year of data, parsed and Read more…
Among many other useful things, big data platforms lets you design data pipeline, fetching or receiving data from one side, applying some transformations, then pushing the resulting data further. If you run a punchplatform, you are likely to do that to collect then parse your data into Elasticsearch, in turn Read more…
The punchplatform lets you design arbitrary spark pipelines on top of your Elastic data by simple configuration. Refer to the PML documentation. In this post we explain how to execute a straight Spark job. Doing that is useful to quickly prototype, to understand what your are doing, or even simply to learn Read more…