[[metricset-details]]
=== Metricset Details

This topic provides additional details about creating metricsets.

[float]
=== Adding Special Configuration Options

Each metricset can have its own configuration variables defined. To make use of
these variables, you must extend the `New` method. For example, let's assume that
you want to add a `password` config option to the metricset. You would extend
`beat.yml` in the following way:

[source,yaml]
----
metricbeat.modules:
- module: {module}
  metricsets: ["{metricset}"]
  password: "test1234"
----

To read in the new `password` config option, you need to modify the `New` method. First you define a config
struct that contains the value types to be read. You can set default values, as needed. Then you pass the config to
the `UnpackConfig` method for loading the configuration.

Your implementation should look something like this:

[source,go]
----
type MetricSet struct {
	mb.BaseMetricSet
	password string
}

func New(base mb.BaseMetricSet) (mb.MetricSet, error) {

	// Unpack additional configuration options.
	config := struct {
		Password string `config:"password"`
	}{
		Password: "",
	}
	err := base.Module().UnpackConfig(&config)
	if err != nil {
		return nil, err
	}

	return &MetricSet{
		BaseMetricSet: base,
		password:      config.Password,
	}, nil
}
----


[float]
==== Timeout Connections to Services

Each time the `Fetch` method is called, it makes a request to the service, so it's
important to handle the connections correctly. We recommended that you set up the
connections in the `New` method and persist them in the `MetricSet` object. This allows
connections to be reused.

One very important point is that connections must respect the timeout variable:
`base.Module().Config().Timeout`. If the timeout elapses before the request completes,
the request must be ended and an error must be returned to make sure the next request
can be started on time. By default the Timeout is set to Period, so one request gets
ended before a new request is made.

If a request must be ended or has an error, make sure that you return a useful error
message. This error message is also sent to Elasticsearch, making it possible to not
only fetch metrics from the service, but also report potential problems or errors with
the metricset.


[float]
==== Data Transformation

If the data transformation that has to happen in the `Fetch` method is
extensive, we recommend that you create a second file called `data.go` in the same package
as the metricset. The `data.go` file should contain a function called `eventMapping(...)`.
A separate file is not required, but is currently a best practice because it isolates the
functionality of the metricset and `Fetch` method from the data mapping.



[float]
==== fields.yml

The `fields.yml` file is used for different purposes:

* Creates the Elasticsearch template
* Creates the Kibana index pattern configuration
* Creates the Exported Fields documentation for the metricset

To make sure the Elasticsearch template is correct, it's important to keep this file up-to-date
with all the changes. There is a `fields.yml` file under `module/{module}/_meta/fields.yml` that contains
the general top level structure for all metricsets. Normally you only need to modify the description in this file.

Here an example for the `fields.yml` file from the MySQL module.

[source,yaml]
----
include::../../metricbeat/module/mysql/_meta/fields.yml[]
----

There is another `fields.yml` file under `module/{module}/{metricset}/_meta/fields.yml` that contains all fields retrieved
by the metricset. As field types, each field must have a core data type
{ref}/mapping-types.html#_core_datatypes[supported by elasticsearch]. Here's a very basic example that shows one group from the MySQL `status` metricset:

[source,yaml]
----
- name: status
  type: group
  description: >
    `status` contains the metrics that were obtained by the status SQL query.
  fields:
    - name: aborted
      type: group
      description: >
        Aborted status fields.
      fields:
        - name: clients
          type: integer
          description: >
            The number of connections that were aborted because the client died without closing the connection properly.

        - name: connects
          type: integer
          description: >
            The number of failed attempts to connect to the MySQL server.
----

As you can see, if there are nested fields, you must use the type `group`.

// TODO: Add link to general fields.yml developer guide

[float]
==== Testing

It's important to also add tests for your metricset. There are three different types of tests that you need for testing a Beat:

* unit tests
* integration tests
* system tests

We recommend that you use all three when you create a metricset. Unit tests are
written in Go and have no dependencies. Integration tests are also written
in Go but require the service from which the module collects metrics to also be running.
System tests for Metricbeat also require the service to be running in most cases and are
written in Python based on our small Python test framework.
We use https://docs.python.org/3/library/venv.html[venv] to deal with Python dependencies.
You can simply run the command `make python-env`  and then `. build/python-env/bin/activate` .

You should use a combination of the three test types to test your metricsets because
each method has advantages and disadvantages. To get started with your own tests, it's best
to look at the existing tests. You'll find the unit and integration tests
in the `_test.go` files under existing modules and metricsets. 
Integration tests usually take the form of `TestFetch` and `TestData`.
The system tests are under `tests/systems`.


[float]
===== Adding a Test Environment

Integration and system tests need an environment that's running the service. You
can create this environment by using Docker and a docker-compose file. If you add a
module that requires a service, you must add the service to the virtual environment.
To do this, you:

* Update the `docker-compose.yml` file with your environment
* Update the `docker-entrypoint.sh` script

The `docker-compose.yml` file is at the root of Metricbeat. Most services have
existing Docker modules and can be added as simply as Redis:

[source,yaml]
----
redis:
  image: redis:3.2.3
----

To allow the Beat to access your service, make sure that you define the environment
variables in the docker-compose file and add the link to the container:

[source,yaml]
----
beat:
  links:
    - redis
  environment:
    - REDIS_HOST=redis
    - REDIS_PORT=6379
----

To make sure the service is running before the tests are started, modify the
`docker-entrypoint.sh` script to add a check that verifies your service is
running. For example, the check for Redis looks like this:

[source,shell]
----
waitFor ${REDIS_HOST} ${REDIS_PORT} Redis
----

The environment expects your service to be available as soon as it receives a response from
the given address and port.

[float]
===== Adding the standard metricset integration tests

There are normally two integration tests that are part of every metricset: `TestFetch` and `TestData`.
Both tests will start up a new instance of your metricset and fetch an event. In order to start a metricset, you need to create a configuration object:

[source,go]
----
func getConfig() map[string]interface{} {
    return map[string]interface{}{
    "module":           "{module}",
    "metricsets":       []string{"{metricset}"},
    "hosts":      []string{GetEnvHost() + ":" + GetEnvPort()}, <1>
  }
}

func GetEnvHost() string { <2>
    host := os.Getenv("{module}_HOST")
    if len(host) == 0 {
    host = "127.0.0.1"
  }
  return host
}

func GetEnvPort() string { <2>
    port := os.Getenv("{module}_PORT")

    if len(port) == 0 {
      port = "1234"
    }
  return port
}

----
<1> Add any additional config options your metricset needs here.
<2> The endpoint used by the metricset needs to be configurable for manual and automated testing. 
Environment variables should be defined in the module under `_meta/env` and included in the `docker-compose.yml` file.

The `TestFetch` integration test will return a single event from your metricset, which you can use to test the validity of the data.
`TestData` will (re)generate the `_meta/data.json` file that documents the data reported by the metricset.

[source,go]
----
import (
	"os"
	"testing"

	"github.com/stretchr/testify/assert"

	"github.com/elastic/beats/libbeat/tests/compose"
	mbtest "github.com/elastic/beats/metricbeat/mb/testing"
)

func TestFetch(t *testing.T) {
	compose.EnsureUp(t, "{module}") <1>

	f := mbtest.NewReportingMetricSetV2Error(t, getConfig())

	events, errs := mbtest.ReportingFetchV2Error(f)
	if len(errs) > 0 {
		t.Fatalf("Expected 0 errord, had %d. %v\n", len(errs), errs)
	}

	assert.NotEmpty(t, events) <2>

}

func TestData(t *testing.T) {

	f := mbtest.NewReportingMetricSetV2Error(t, getConfig())

	err := mbtest.WriteEventsReporterV2Error(f, t, "") <3>
	if !assert.NoError(t, err) {
		t.FailNow()
	}
}
----
<1> Use this to start the docker service associated with your metricset.
<2> Add any further validity checks to verify the metricset is working.
<3> `WriteEventsReporterV2Error` will take the first valid event from the metricset and write it to `_meta/data.json`

[float]
===== Running the Tests

To run all the tests, run `make testsuite`. To only run unit tests, run
`mage unitTest`, or for integration tests `mage integTest`.
Be aware that a running Docker environment is needed for integration and system
tests.

To run `TestData` and generate the `data.json` file, run
`go test -tags=integration -data -run TestData` in the directory where your test is located.

To run the integration tests for a single module, set the `MODULE` environment
variable to the name of the directory of the module. For example you can run the
following command to run integration tests for `apache` module:

[source,shell]
----
MODULE=apache mage integTest
----


[float]
=== Documentation

Each module must be documented. The documentation is based on asciidoc and is in
the file `module/{module}/_meta/docs.asciidoc` for the module and in `module/{module}/{metricset}/_meta/docs.asciidoc`
 for the metricset. Basic documentation with the config file and an example output is automatically
 generated. Use these files to document specific configuration options or usage examples.




////
TODO: The following parts should be added as soon as the content exists or the implementation is completed.

[float]
== Field naming
https://github.com/elastic/beats/blob/master/metricbeat/module/doc.go

[float]
== Dashboards

Dashboards are an important part of each metricset. Data gets much more useful
when visualized. To create dashboards for the metricset, follow the guide here
(link to dashboard guide).
////
