Home | Blog | Impress | Data Protection Policy

Visualize Fluent Bit metrics with EB solys

March 15, 2019, written by Torsten Mosis, torsten.mosis@systemticks.de

Intro

In my last blog post I have described how we can setup a logging infrastructure for automotive software systems by exploiting Fluent Bit, Automotive DLT and EB solys.

If not yet done you should read this post first. It explains how these three standalone tools can be installed and configured to run as a combined log & trace analysis stack.

In this post we want to illustrate how EB solys can be utilized to visualize the runtime metrics collected by Fluent Bit in a natural and reasonable way.

Data Flow

Compared to last time we omit DLT as a intermediate logging router for the sake of convenience and connect Fluent Bit directly with EB solys.

How do Fluent Bit and EB solys interplay with each other?

With so-called input plugins Fluent Bit can be configured to retrieve valuable linux system data, such as cpu load, memory consumption, kernel messages, systemd messages and many more.

These logs can be redirected to certain output plugins, like e.g. a http output plugin.

In the end you can run a script within EB solys, that acts as http endpoint for retrieving the log data.

FS stack

Configuring Fluent Bit

So, let’s setup Fluent Bit for collecting some Linux runtime metrics.

Input Plugins

First we configure our input plugins in the INPUT section. We run the mem plugin for collecting memory consumption data, the disk plugin for measuring disk I/O activities and the netif plugin for determining sent and received bytes over the network interface.

[INPUT]
    Name  mem
    Tag   memory

[INPUT]
    Name  disk
    Tag   diskIO

[INPUT]
    Name  netif
    Tag   netif
    Interface enp0s3

Filtering

The FILTER section allows to alter the incoming data generated by the input plugins. Here we remove some data we are not interested in and rename some elements to our needs.

[FILTER]
    Name record_modifier
    Match memory
    Remove_key Swap.total
    Remove_key Swap.used
    Remove_key Swap.free

[FILTER]
    Name modify
    Match memory
    Rename Mem.free free
    Rename Mem.used used
    Rename Mem.total total

[FILTER]
    Name modify
    Match diskIO
    Rename read_size read
    Rename write_size write

[FILTER]
    Name modify
    Match netif
    Rename enp0s3.rx.bytes rxBytes
    Rename enp0s3.rx.packets rxPackets
    Rename enp0s3.tx.bytes txBytes
    Rename enp0s3.tx.packets txPackets

Output plugins

And finally in the OUTPUT section we tell Fluent Bit where it should flush the information it gathers from the input. In our case we use the http plugin, that flushes all records into a HTTP endpoint. The http pluging simply issues a HTTP POST request with the data records in JSON format.

[OUTPUT]
    Name  http
    Match memory
    Host  10.0.2.2
    Port  7071
    Format json
    URI   /memory

[OUTPUT]
    Name  http
    Match diskIO
    Host  10.0.2.2
    Port  7071
    Format json
    URI   /diskIO

[OUTPUT]
    Name  http
    Match netif
    Host  10.0.2.2
    Port  7071
    Format json
    URI   /netif

The complete configuration file is located here. We will need that later.

Connect Fluent Bit to EB solys

Fluent Bit is now configured to collect certain runtime data. But how do we flush the data into EB solys then?

EB solys already comes with two native connectors for retrieving log and trace data:

  • EB solys target agent (a log processor similar to Fluent Bit, available for Linux and Android)

  • Automotive DLT (shown in the last blog post)

For a deep integration we can think of providing a corresponding Fluent Bit connector. This is not yet in place. Feel free to contribute or suppport. :)

For the time being we can use another facility of EB solys for processing log data captured by Fluent Bit: the EB solys built-in script engine.

EB solys scripting

The origin purpose of the sripting engine is to allow the user to interact with the collected runtime data and the UI resources. This enables the user to aggregate and correlate data from different sources programmatically and visualize them in tables, charts or in arbitrary html views in a very easy and straight-forward manner.

This is a very powerful facility for rapid prototyping, since you can add new features to EB solys without the need to modify or extend its source code.

The script language Xtend, the Script API and concepts are explained in full detail in the EB solys inline doc and here:

In case you want to reproduce the use-case in detail and understand the script in its depth you should read those docs first.

Listen to Http Requests

So, how does such a script look like?

The implementation for our use-case is very straight forward:

  • We open a server socket (the port must be the same as specified in the Fluent Bit configuration)

  • We read the HTTP POST data from new incoming requests

  • We handle the payload accordingly to the given URI (also specified in the Fluent Bit configuration)

@BeforeScript
def setup() {
  server = new ServerSocket(7071)
  threadPool = Executors.newCachedThreadPool
  gson = new Gson
}

@Execute(context=ExecutionContext.GLOBAL, description="Start HTTP Server and listen to Fluent Bit log records...")
def execute() {

  while (true) {
    val socket = server.accept
    threadPool.execute(
      new Runnable() {
        override run() {
          val in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
          val post = readHttpPostData(in)
          switch (post.getUri) {
            case 'memory': handleMemory(post.payload)
            case 'diskIO' : handleDiskIO(post.payload)
            case 'netif' : handleNetIf(post.payload)
            default: { consolePrintln(post.payload) }
          }
        }
      }
    )
  }
}

@AfterScript
def tearDown()
{
  threadPool.shutdownNow
  server.close
}

Handling the log data collected by e.g. the netif plugin looks like that:

  • We define a POJO that holds the log data items

  • We transform the json string into a list of objects of that POJO

  • We create a new runtime event for each sample (e.g. rxBytes) and assign it to a channel

@Data
class NetIfItem {
        double date
        long rxBytes
        long rxPackets
        long txBytes
        long txPackets
}

val netifListType = new TypeToken<ArrayList<NetIfItem>>(){}.getType();

def handleNetIf(String jsonString) {

  val List<NetIfItem> memList = gson.fromJson(jsonString, netifListType)

  memList.forEach[
    val timestamp = date.asMicroSeconds
    createOrGetChannel("Netif.rx.bytes", Unit.KILOBYTE, '').addEvent(timestamp, rxBytes.asKiloBytes)
    createOrGetChannel("Netif.rx.packets", Unit.COUNT, '').addEvent(timestamp, rxPackets)
    createOrGetChannel("Netif.tx.bytes", Unit.KILOBYTE, '').addEvent(timestamp, txBytes.asKiloBytes)
    createOrGetChannel("Netif.tx.packets", Unit.COUNT, '').addEvent(timestamp, txPackets)
  ]
}


def asKiloBytes(long bytes)
{
  bytes / 1024
}

You can download the full script from here and import into EB solys with File/Import Script.

Now it’s time to start Fluent Bit with the configuration file we have defined beforehand:

fluent-bit -c fluent-http_out.conf

Then start the EB solys script as shown in the video below:

After importing you can start the script immediately without the need of any further build, deployment or integration work. The script is compiled in the background on-the-fly and available for execution right away.

As soon as the log data are flushed into EB solys the corresponding channels (as implemented in the script) will be created. When double-clicking a channel a corresponding line chart is opened, which represents the log data as a timeline graph.

You can also show the current values of all channels in a snapshot view. The snapshot view is then useful when you want to display an overview of your system at a certain timestamp by monitoring significant values. It behaves similar to a watch window in a programming IDE or debugger where you can monitor variables at a breakpoint.

Conclusion

In this blog post we have shown how easy it is to feed EB solys with data from a log provider like Fluent Bit and how you can visualize the data in a meaningful and valuable way.

Feel free to contact me via mail torsten.mosis@systemticks.de, in case you have questions or if you need any further support.