This post has been republished via RSS; it originally appeared at: Containers articles.
TL;DR: Windows Containers Log Monitor is opensource released on Github.
In this blog post we announced our efforts to improve the tooling experience for containers. The first step in this journey was with a log tool concept we demoed during our container session at //build. This demo was met with clear excitement from attendees and interest from the community. After gathering customer feedback, we set out to bring this tool to production quality and make it open source available for our Windows Containers users.
Today, we are happy to announce the opensource release of the Windows Container Log Monitor, now available on Github! This blog offers a deep dive into the architecture and usage of the tool.
To recap, unlike Linux applications that log to STDOUT, Windows applications log to Windows log locations such as ETW, Event Log, and custom log files. Since many container ecosystem logging solutions are built to pull from the STDOUT pipeline as standard with Linux, Windows containers app logs historically have not been accessible via these solutions. The Log Monitor bridges this gap between Windows log locations and STDOUT, as depicted in the diagram below. The scope of the Log Monitor tool is to bridge Windows application logs to the STDOUT pipeline.
The Log Monitor tool consists of two primary elements: LogMonitor.exe and LogMonitorConfig.json. Like the //build demo POC, the Log Monitor is authored with an observer, publisher-subscriber design pattern; however, instead of subscribing to hardcoded log source providers as in the original concept, the Log Monitor is configured to subscribe to a set of sources defined by the user in the Log Monitor Config file.
The log sources supported by the Log Monitor tool include:
- ETW
- Event Logs
- Log files
Note that these sources differ from the original concept, which collected Perf Counters and did not collect Event Logs. From feedback we concluded that Perf Counters, while useful, fall closer to system metrics than application logs, and were thus out of scope for this release. Additionally, the Log Monitor tool does not monitor the lifecycle of services within the container as in the original concept, but instead provides a nesting pattern so users can define their own service monitor or desired application.
The log tool is supported for Windows, Server Core, and Nano images.
Usage
A key point of feedback we received at //build and in the post-build survey was that the entry point-only solution restricts users from using their own app as the entry point process. We mitigate this with several options on usage:
- The Log Monitor tool can be used as the SHELL or the ENTRYPOINT
- As either the SHELL or ENTRYPOINT, another application can be indicated as a parameter to the Log Monitor tool as a nested process. The Log Monitor tool will spin up that process and monitor its STDOUT so that it also continues to show up in the container STDOUT. This mitigates the foreground/background process issue that existed in the original concept.
An example dockerfile for the SHELL usage pattern (with nesting):
An example dockerfile for the ENTRYPOINT usage pattern (with nesting):
Note that in the SHELL usage pattern the CMD/ENTRYPOINT instruction should be specified in the SHELL form and not exec form. When exec form of the CMD/ENTRYPOINT instruction is used, SHELL is not launched, and the Log Monitor tool will not be launched inside the container.
Additionally, if you are building from an image that pre-defines an entrypoint (i.e. an IIS image that pre-defines IIS.ServiceMonitor as the entry point), you must redefine the entrypoint in your dockerfile, either as the Log Monitor if you are using the ENTRYPOINT pattern, or as the pre-defined entrypoint. If you do not redefine the ENTRYPOINT of your dockerfile and are using the SHELL usage pattern, the Log Monitor tool will not work. This is because the SHELL is tied to the entrypoint definition.
Both example usages wrap the ping.exe application. Other applications such as IIS.ServiceMonitor can be nested with Log Monitor in a similar fashion:
Configuration
A sample Log Monitor Config file would be structured as follows:
In the config file there are the three log types you can configure the Log Monitor tool to pull from:
- Event Logs. The event log provides a standard, centralized way for apps and the OS to record important software and hardware events. Each event is in a structured data format, which makes it easy to parse for event log view panes (i.e. Event Viewer) or examine with a programmatic interface. The configuration file supports the following fields:
- startAtOldestRecord. This Boolean field indicates whether the Log Monitor tool should output event logs from the start of the container boot (true) or from the start of the Log Monitor tool itself (false).
- eventFormatMultiLine. This Boolean field indicates whether the Log Monitor should format the logs to STDOUT as multi-line or single line. If set true, the tool does not format the event messages to a single line (and thus event messages can span multiple lines). If set to false, the tool formats the event log messages to a single line and removes new line characters. Many container tools that build upon docker logs and the STDOUT pipe assume each log entry is on a single line as Linux event messages are typically single line.
- channels. A channel is a sync that collects events. The wevutil tool can be used to enumerate channels via the command “wevutil el.”
- Name. Name of the event channel
- Level. This string field specifies the verboseness of the events collected. Possible values in ascending order of verbosity are Critical, Error, Warning, Information, and Verbose. Verbose encompasses Critical, Error, and Warning, and Information.
- Custom Log File. Many apps log to custom app log files. Some services such as the IIS service also log to files in well-known locations. The Log Monitor tool can be configured to parse through custom log files and bridge these entries to STDOUT. The configuration file supports the following fields:
- directory. This string field indicates the directory that the Log Monitor tool should monitor. Note that there can be more than one directory, though additionally directories should be added as an additional json source object with type=File.
- filter. This string field indicates what files under the directory the Log Monitor should monitor. The string is parsed for regex format.
- includeSubdirectories. This Boolean field indicates whether the Log Monitor should recurse the subdirectories of the indicated directory (true) or remain in the indicated directory (false).
- ETW (Event Tracing for Windows). ETW is an efficient kernel-level tracing facility that lets you log system or application defined events to a log file or access them real-time. Applications often instrument providers that are enabled to generate events as relevant to the application. The Log Monitor tool can be configured to watch certain relevant providers. The configuration file supports the following fields:
- eventFormatMultiLine. The behavior is the same as described above. This does not have to match the Boolean chosen for the Event Log.
- Providers. This field is a list of providers that indicates what providers the Log Monitor tool should monitor. The following fields are supported for each provider:
- providerName.
- ProviderGuid. For Server Core or Windows base images, note that either providerName or ProviderGuid can be provided. For Nanoserver ProviderGuid is required as resolving provider name to GUID is a known issue.
- level: a 1-byte integer that enables filtering based on the severity or verbosity of events
- keywords: This string field is a bitmask that specifies what events to collect. Only events with keywords matching the bitmask are collected This is an optional parameter. Default is 0 and all the events will be collected.
Note that which events, files, and ETW providers an application logs to is defined by the application. There are some standard patterns for some common applications such as IIS. Some sample config files for these popular scenarios are included in the Github repo. If information about which events, files, and ETW providers a particular application logs to is not readily available, wevutil or Event Viewer can help.
What is next?
While we are excited to bring this functionality to our users, this is just the beginning of our journey. By open sourcing this tool, we hope to continuously gather feedback and improve the tool to meet customer needs. Some areas we have already identified and are investigating for future releases:
- Rotating log support
- Environment variable configuration support
- ConfigMap support
- Integrations with log aggregation services at scale
- Configuration updates during container runtime
- Performance
- Sidecar usage patterns
- Log driver support