// Use these for links to issue and pulls. Note issues and pulls redirect one to
// each other on Github, so don't worry too much on using the right prefix.
:issue: https://github.com/elastic/beats/issues/
:pull: https://github.com/elastic/beats/pull/

This changelog is intended for community Beat developers. It covers the major
breaking changes to the internal APIs in the official Beats and changes related
to developing a Beat like code generators or `fields.yml`. Only the major
changes will be covered in this changelog that are expected to affect community
developers. Each breaking change added here should have an explanation on how
other Beats should be migrated.

Note: This changelog documents the current changes which are not yet present in
an actual release.

=== Beats version HEAD
https://github.com/elastic/beats/compare/v7.0.0-rc2..master[Check the HEAD diff]

The list below covers the major changes between 7.0.0-rc2 and master only.

==== Breaking changes

- Move Fields from package libbeat/common to libbeat/mapping. {pull}11198[11198]
- For "metricbeat style" generated custom beats, the mage target `GoTestIntegration` has changed to `GoIntegTest` and `GoTestUnit` has changed to `GoUnitTest`. {pull}13341[13341]
- Build docker and kubernetes features only on supported platforms. {pull}13509[13509]
- Need to register new processors to be used in the JS processor in their `init` functions. {pull}13509[13509]
- The custom beat generator now uses mage instead of python, `mage GenerateCustomBeat` can be used to create a new beat, and `mage vendorUpdate` to update the vendored libbeat in a custom beat. {pull}13610[13610]
- Altered all remaining uses of mapval to use the renamed and enhanced version: https://github.com/elastic/go-lookslike[go-lookslike] instead, which is a separate project. The mapval tree is now gone. {pull}14165[14165]
- Move light modules to OSS. {pull}14369[14369]
- Deprecate test flags, `generate` and `update_expected`, in favor of `data`. {pull}15292[15292]
- Python 3 is required now to run python tests and tools. {pull}14798[14798]
- The type `memqueue.Broker` is no longer exported; instead of `memqueue.NewBroker`, call `memqueue.NewQueue` (which provides the same public interface). {pull}16667[16667]
- The disk spool types `spool.Spool` and `spool.Settings` have been renamed to the internal types `spool.diskSpool` and `spool.settings`. {pull}16693[16693]
- `queue.Eventer` has been renamed to `queue.ACKListener` {pull}16691[16691]
- Require logger as first parameter for `outputs.elasticsearch.client#BulkReadItemStatus`. {pull}16761[16761]
- Extract Elasticsearch client logic from `outputs/elasticsearch` package into new `esclientleg` package. {pull}16150[16150]
- Rename `queue.BufferConfig.Events` to `queue.BufferConfig.MaxEvents`. {pull}17622[17622]
- Remove `queue.Feature` and replace `queue.RegisterType` with `queue.RegisterQueueType`. {pull}17666[17666]
- The way configuration files are generated has changed to make it easier to customize parts
  of the config without requiring changes to libbeat config templates. Generation is now
  fully based on Go text/template and no longer uses file concatenation to generate the config.
  Your magefile.go will require a change to adapt the devtool API. See the pull request for
  more details. {pull}18148[18148]
- Introduce APM libbeat instrumentation. `Publish` method on `Client` interface now takes a Context as first argument. {pull}17938[17938]
- The Elasticsearch client settings expect the API key to be raw (not base64-encoded). {issue}18939[18939] {pull}18945[18945]
- `management.ConfigManager` has been renamed to `management.Manager`. {pull}19114[19114]
- `UpdateStatus` has been added to the `management.Manager` interface. {pull}19114[19114]
- Remove `common.MapStrPointer` parameter from `cfgfile.Runnerfactory` interface. {pull}19135[19135]
- Replace `ACKCount`, `ACKEvents`, and `ACKLastEvent` callbacks with `ACKHandler` and interface in `beat.ClientConfig`. {pull}19632[19632]
- Remove global ACK handler support via `SetACKHandler` from publisher pipeline. {pull}19632[19632]
- Make implementing `Close` required for `reader.Reader` interfaces. {pull}20455[20455]
- Remove `NumCPU` as clients should update the CPU count on the fly in case of config changes in a VM. {pull}23154[23154]
- Remove Metricbeat EventFetcher and EventsFetcher interface. Use the reporter interface instead. {pull}25093[25093]
- Update Darwin build image to a debian 10 base that increases the MacOS SDK and minimum supported version used in build to 10.14. {issue}24193[24193]

==== Bugfixes

- Stop using `mage:import` in community beats. This was ignoring the vendorized beats directory for some mage targets, using the code available in GOPATH, this causes inconsistencies and compilation problems if the version of the code in the GOPATH is different to the vendored one. Use of `mage:import` will continue to be unsupported in custom beats till beats is migrated to go modules, or mage supports vendored dependencies. {issue}13998[13998] {pull}14162[14162]
- Metricbeat module builders call host parser only once when instantiating light modules. {pull}20149[20149]
- Fix export dashboard command when running against Elastic Cloud hosted Kibana. {pull}22746[22746]
- Remove `event.dataset` (ECS) annotion from `libbeat.logp`. {issue}27404[27404]
- Errors should be thrown as errors. Metricsets inside Metricbeat will now throw errors as the `error` log level. {pull}27804[27804]
- Avoid panicking in `add_fields` processor when input event.Fields is a nil map. {pull}28219[28219]
- Fix type mismatch in libbeat/metric/system/cgroup/cgv2 when building on mips platforms. {pull}34658[34658]
- Make winlogbeat/sys/wineventlog follow the unsafe.Pointer rules. {pull}36650[36650]
- Skip flakey test in HTTPJSON input {issue}34929[34929] {issue}37146[37146] {pull}37162[37162]

==== Added

- Add configuration for APM instrumentation and expose the tracer trough the Beat object. {pull}17938[17938]
- Metricset generator generates beta modules by default now. {pull}10657[10657]
- The `beat.Event` accessor methods now support `@metadata` keys. {pull}10761[10761]
- Assertion for documented fields in tests fails if any of the fields in the tested event is documented as an alias. {pull}10921[10921]
- Support for Logger in the Metricset base instance. {pull}11106[11106]
- Filebeat modules can now use ingest pipelines in YAML format. {pull}11209[11209]
- Prometheus helper for metricbeat contains now `Namespace` field for `prometheus.MetricsMappings` {pull}11424[11424]
- Update Jinja2 version to 2.10.1. {pull}11817[11817]
- Reduce idxmgmt.Supporter interface and rework export commands to reuse logic. {pull}11777[11777],{pull}12065[12065],{pull}12067[12067],{pull}12160[12160]
- Update urllib3 version to 1.24.2 {pull}11930[11930]
- Add libbeat/common/cleanup package. {pull}12134[12134]
- New helper to check for leaked goroutines on tests. {pull}12106[12106]
- Only Load minimal template if no fields are provided. {pull}12103[12103]
- Add new option `IgnoreAllErrors` to `libbeat.common.schema` for skipping fields that failed while converting. {pull}12089[12089]
- Deprecate setup cmds for `template` and `ilm-policy`. Add new setup cmd for `index-management`. {pull}12132[12132]
- Use the go-lookslike library for testing in heartbeat. Eventually the mapval package will be replaced with it. {pull}12540[12540]
- New ReporterV2 interfaces that can receive a context on `Fetch(ctx, reporter)`, or `Run(ctx, reporter)`. {pull}11981[11981]
- Generate configuration from `mage` for all Beats. {pull}12618[12618]
- Add ClientFactory to TCP input source to add SplitFunc/NetworkFuncs per client. {pull}8543[8543]
- Introduce beat.OutputChooses publisher mode. {pull}12996[12996]
- Ensure that beat.Processor, beat.ProcessorList, and processors.ProcessorList are compatible and can be composed more easily. {pull}12996[12996]
- Add support to close beat.Client via beat.CloseRef (a subset of context.Context). {pull}13031[13031]
- Add checks for types and formats used in fields definitions in `fields.yml` files. {pull}13188[13188]
- Makefile included in generator copies files from beats repository using `git archive` instead of cp. {pull}13193[13193]
- Strip debug symbols from binaries to reduce binary sizes. {issue}12768[12768]
- Compare event by event in `testadata` framework to avoid sorting problems {pull}13747[13747]
- Added a `default_field` option to fields in fields.yml to offer a way to exclude fields from the default_field list. {issue}14262[14262] {pull}14341[14341]
- `supported-versions.yml` can be used in metricbeat python system tests to obtain the build args for docker compose builds. {pull}14520[14520]
- Add support for MODULE environment variable in `mage goIntegTest` in metricbeat to run integration tests for a single module. {pull}17147[17147]
- Add support for a `TEST_TAGS` environment variable to add tags for tests selection following go build tags semantics, this environment variable is used by mage test targets to add build tags. Python tests can also be tagged with a decorator (`@beat.tag('sometag')`). {pull}16937[16937] {pull}17075[17075]
- Add fields validation for histogram subfields. {pull}17759[17759]
- Events intended for the Elasticsearch output can now take an `op_type` metadata field of type events.OpType or string to indicate the `op_type` to use for bulk indexing. {pull}12606[12606]
- Remove vendor folder from repository. {pull}18655[18655]
- Added SQL helper that can be used from any Metricbeat module {pull}18955[18955]
- Update Go version to 1.14.4. {pull}19753[19753]
- Update Go version to 1.14.7. {pull}20508[20508]
- Add packaging for docker image based on UBI minimal 8. {pull}20576[20576]
- Make the mage binary used by the build process in the docker container to be statically compiled. {pull}20827[20827]
- Add Pensando distributed firewall module. {pull}21063[21063]
- Update ecszap to v0.3.0 for using ECS 1.6.0 in logs {pull}22267[22267]
- Add support for customized monitoring API. {pull}22605[22605]

- Update Go version to 1.15.7. {pull}22495[22495]
- Update Go version to 1.15.8. {pull}23955[23955]
- Update Go version to 1.15.9. {pull}24442[24442]
- Update Go version to 1.15.10. {pull}24606[24606]
- Update Go version to 1.15.12. {pull}25629[25629]
- Update Go version to 1.16.4. {issue}25346[25346] {pull}25671[25671]
- Add sorting to array fields for generated data files (*-generated.json) {pull}25320[25320]
- Update to go-concert 0.2.0 {pull}27162[27162]
- Update Go version to 1.16.5. {issue}26182[26182] {pull}26186[26186]
- Introduce `libbeat/beat.Beat.OutputConfigReloader` {pull}28048[28048]
- Update Go version to 1.17.1. {pull}27543[27543]
- Whitelist `GCP_*` environment variables in dev tools {pull}28364[28364]
- Improve tests files with shorter statements. {pull}35667[35667]
- Improve compatibility and reduce flakyness of Python tests {pull}31588[31588]
- Pin PyYAML version to 5.3.1 to avoid CI errors temporarily {pull}36091[36091]
- Make logs for empty and small files less noisy when using fingerprint file identity in filestream. {pull}38421[38421]

==== Deprecated

- Deprecated the `common.Float` type. {issue}28279[28279] {pull}28280[28280]
- Deprecate Beat generators. {pull}28814[28814]
