Get best practices on how to monitor your Kubernetes clusters from field experts in this episode of the Kubernetes Best Practices Series. In this intermediate level deep dive, you will learn about monitoring and logging in Kubernetes from Dennis Zielke, Technology Solutions Professional in the Global Black Belts Cloud Native Applications team at Microsoft.
Multi-cluster view from Azure Monitor
Azure Monitor provides a multi-cluster view showing the health status of all monitored AKS clusters deployed across resource groups in your subscriptions. It shows AKS clusters discovered that are not monitored by the solution. Immediately you can understand cluster health, and from here you can drill down to the node and controller performance page, or navigate to see performance charts for the cluster. For AKS clusters discovered and identified as unmonitored, you can enable monitoring for that cluster at any time.
Container Live Logs provides a real-time view into your Azure Kubernetes Service (AKS) container logs (stdout/stderr) without having to run kubectl commands. When you select this option, new pane appears below the containers performance data table on the Containers view, and it shows live logging generated by the container engine to further assist in troubleshooting issues in real time.
Live logs supports three different methods to control access to the logs:
AKS without Kubernetes RBAC authorization enabled
AKS enabled with Kubernetes RBAC authorization
AKS enabled with Azure Active Directory (AD) SAML based single-sign on
You even can search in the Container Live Logs for Troubleshooting and history.
Azure Monitor for VMs monitors your Azure virtual machines (VM) and virtual machine scale sets at scale. The service analyzes the performance and health of your Windows and Linux VMs, monitoring their processes and their dependencies on other resources and external processes.
As a solution, Azure Monitor for VMs includes support for monitoring performance and application dependencies for VMs that are hosted on-premises or in another cloud provider. Three key features deliver in-depth insight:
Logical components of Azure VMs that run Windows and Linux: Are measured against pre-configured health criteria, and they alert you when the evaluated condition is met.
Pre-defined, trending performance charts: Display core performance metrics from the guest VM operating system.
Dependency map: Displays the interconnected components with the VM from various resource groups and subscriptions.
The features are organized into three perspectives:
Health
Performance
Map
Here we have a look at Azure Monitor Service map of my local machine :
Here in the Event you see two Configuration Changes.
What is awesome to see, when you double click on the link marked with a arrow, then It will start log analytics with the right query to see what those changes are 🙂
You see some Changes in Windows Services and Updates on my local Machine
Communications of the local machine on-premisses
Workbooks combine text, Analytics queries, Azure Metrics, and parameters into rich interactive reports. Workbooks are editable by any other team members who have access to the same Azure resources.
Azure Monitor for containers is a feature designed to monitor the performance of container workloads deployed to either Azure Container Instances or managed Kubernetes clusters hosted on Azure Kubernetes Service (AKS). Monitoring your containers is critical, especially when you’re running a production cluster, at scale, with multiple applications.
Azure Monitor for containers gives you performance visibility by collecting memory and processor metrics from controllers, nodes, and containers that are available in Kubernetes through the Metrics API. Container logs are also collected. After you enable monitoring from Kubernetes clusters, these metrics and logs are automatically collected for you through a containerized version of the Log Analytics agent for Linux and stored in your Log Analytics workspace.
What I really like is that you now can see the Container Live logs from the Azure portal and see what is going on in the background of a Container 🙂
Activate Azure Kubernetes Container Live Logs
Here you see the Container Live logs
This feature provides a real-time view into your Azure Kubernetes Service (AKS) container logs (stdout/stderr) without having to run kubectl commands. When you select this option, new pane appears below the containers performance data table on the Containers view, and it shows live logging generated by the container engine to further assist in troubleshooting issues in real time.
Live logs supports three different methods to control access to the logs:
AKS without Kubernetes RBAC authorization enabled
AKS enabled with Kubernetes RBAC authorization
AKS enabled with Azure Active Directory (AD) SAML based single-sign on
You even can search in the Container Live Logs for Troubleshooting and history :
Search on ssh
Azure Monitor for containers uses a containerized version of the Log Analytics agent for Linux. After initial deployment, there are routine or optional tasks you may need to perform during its lifecycle.
Because of this agent you can work with Log Analytics in Azure Monitor :
When you have your production workload running on Azure Kubernetes Clusters, It’s important to monitor to keep you in Control of the solution in Microsoft Azure and watch for improvements like performance for the business. With Container Live logs you can see what is going on in the Containers when you have issues and that’s great for troubleshooting to get your problem solved fast. Get your workload into Azure Containers and make your Azure DevOps CI/CD Pipelines in the Cloud.
There are a range of tools for monitoring your Azure environment, from the application code running on Azure to the services and infrastructure hosting your application. These tools work together to offer comprehensive cloud monitoring and include:
Azure Monitor – the Azure service that operates as a consolidated pipeline for all monitoring data from Azure services. It gives you access to performance metrics and events that describe the operation of the Azure infrastructure and any Azure services you are using. Azure Monitor is a monitoring data pipeline for your Azure environment, and offers that data directly into Log Analytics as well as 3rd party tools where you can gain insight into that data and combine it with data from on premises or other cloud resources.
Application Insights – the Azure service that offers application performance monitoring and user analytics. It monitors the code you’ve written and applications you’ve deployed on Azure, on-premises, or other clouds. By instrumenting your application with the Application Insights SDK you can get access to a range of data including response times of dependencies, exception traces, debugging snapshots, and execution profiles. It provides powerful tools for analyzing this application telemetry while developing and operating your application. It deeply integrates with Visual Studio to enable you to get right to the problem line(s) of code so you can fix it, and offers usage analytics to analyze customer usage of your applications for product managers as well.
Log Analytics – is an Azure service that ingests log and metric data from Azure services (via Azure Monitor), Azure VMs, and on-premises or other cloud infrastructure and offers flexible log search and out-of-the box analytics on top of this data. It provides rich tools to analyze data across sources, allows complex queries across all logs, and can proactively alert on specified conditions. You can even collect custom data into its central repository so you can query and visualize it. You can also take advantage of Log Analytic’s built-in solutions to immediately gain insights into the security and functionality of your infrastructure.
Azure Monitor enables you to consume telemetry to gain visibility into the performance and health of your workloads on Azure. The most important type of Azure telemetry data is the metrics (also called performance counters) emitted by most Azure resources. Azure Monitor provides several ways to configure and consume these metrics for monitoring and troubleshooting.
Telemetry data is important
Because telemetry data is sending every minute, you get near to real-time monitoring of your data and/or your IT Solution.
There are three types of alerts off of data available from Azure Monitor — metric alerts, near real-time metric alerts (preview) and Activity Log alerts.
Metric alerts– This alert triggers when the value of a specified metric crosses a threshold that you assign. The alert generates a notification when the alert is “Activated” (when the threshold is crossed and the alert condition is met) as well as when it is “Resolved” (when the threshold is crossed again and the condition is no longer met)
Near real-time metric alerts (preview) – These alerts are similar to metric alerts but differ in a few ways. Firstly, as the name suggests these alerts can trigger in near real-time (as fast as 1 min). They also support monitoring multiple(currently two) metrics. The alert generates a notification when the alert is “Activated” (when the thresholds for each metric are crossed at the same time and the alert condition is met) as well as when it is “Resolved” (when at least one metric crosses the threshold again and the condition is no longer met).
Activity log alerts– A streaming log alert that triggers when an Activity Log event is generated that matches filter criteria that you have assigned. These alerts have only one state, “Activated,” since the alert engine simply applies the filter criteria to any new event. These alerts can be used to become notified when a new Service Health incident occurs or when a user or application performs an operation in your subscription, for example, “Delete virtual machine.”
Alerts overview
When you go to the Microsoft Azure Portaland click on the left side on Monitor you can start your Solutions and configure them.
Alerts are created by alert rules that automatically run log searches at regular intervals. If the results of the log search match particular criteria then an alert record is created. The rule can then automatically run one or more actions to proactively notify you of the alert or invoke another process. Different types of alert rules use different logic to perform this analysis.
In addition to creating an alert record in the Log Analytics repository, alerts can take the following actions.
Email. Send an email to proactively notify you of a detected issue.
Runbook. An alert in Log Analytics can start a runbook in Azure Automation. This is typically done to attempt to correct the detected issue. The runbook can be started in the cloud in the case of an issue in Azure or another cloud, or it could be started on a local agent for an issue on a physical or virtual machine.
Webhook. An alert can start a webhook and pass it data from the results of the log search. This allows integration with external services such as an alternate alerting system, or it may attempt to take corrective action for an external web site.
Monitoring your IT Solutions is really important for your Application Life Cycle management to get feedback for improvements and to get Customer satisfaction.
With Microsoft Monitoring from the Cloud with Azure and OMS you get more inside information via telemetry and log analytics to keep you Up-To-Date of
your IT Hybrid Infrastructure. Modern Hybrid Cloud Datacenter(s) need a Modern Secure Monitoring environment to keep yourself and your business in Control all the time in this rapidly fast changing IT World. Monitoring via the Microsoft Cloud gives you :
More Security information, Alerts and Advice to prevent security leaks
Application improvements in your Life Cycle management
Automation of action plans on Events.
The Health of your IT Hybrid Cloud Services
Makes troubleshooting much easier with Diagnostics logs
Integration with on-premises IT Infrastructures
OMS assessments, like Active Directory, SQL, Upgrades, Malware, Security & Audits………… and More
Great Dashboards for DevOps, IT Administrators, IT Managers, or for your Customers.
This diagram shows the classes used or defined in the management pack
The Microsoft Azure SQL Database Management Pack enables you to monitor the availability and performance of applications that are running on Microsoft Azure SQL Database.
Feature Summary
After configuration, the Microsoft Azure SQL Database Monitoring Management Pack offers the following functionalities:
User-friendly wizard to discover Microsoft Azure SQL Database servers.
Provides availability status of Microsoft Azure SQL Database server.
Collects and monitors health of Microsoft Azure SQL Database databases.
Space monitoring:
Used space
Free space
Total allocated quota
Track the total number of databases per server
Successful connections count
Failed connections count
Number of deadlocks
Throttling/long transactions count
Connections blocked by firewall count
Collects and monitors performance information:
Average memory per session
Total memory per session
Total CPU time per session
Total I/O per session
Number of database sessions
Maximum Transaction execution time
Maximum Transaction lock count
Maximum Transaction log space used
Network Egress/Ingress bandwidth
Percentage of CPU used
Percentage of workers used
Percentage of CPU used
Ability to define Custom thresholds for each monitor to configure the warning and critical alerts.
Run-as profile to securely connect to Microsoft Azure SQL Database.
Detailed knowledge to guide the IT operator with troubleshooting the problem.
Custom tasks to redirect the user to the Microsoft Azure SQL Database online portal.
Custom query support to enable application-specific availability and performance monitoring.
The System Center Management Pack for SQL Server 2014 enables the discovery and monitoring of SQL Server 2014 Database Engines, Databases, SQL Server Agents and other related components. This Management Pack is designed to run by Operations Manager 2007 R2 (except dashboards), Operations Manager 2012 or Operations Manager 2012 R2.
The monitoring provided by this management pack includes performance, availability, and configuration monitoring, as well as performance and events data collection. All monitoring workflows have predefined thresholds and complimentary knowledge base articles. You can integrate the monitoring of SQL Server 2014 components into your service-oriented monitoring scenarios.
In addition to health monitoring capabilities, this management pack includes dashboards, diagram views, state views, performance views, alert views and diagnostic tasks that enable near real-time diagnostics and remediation of detected issues.
Note: This management pack does not depend on SQL Server Library management pack. You do not need to import it to enable the monitoring of SQL Server 2014. Note: This management pack is only for SQL Server 2014. Please use this link to download the Operations Manager Management Pack for SQL Server 2005/2008/2012.
Feature Summary
The following list gives an overview of features introduced by System Center Operations Manager Management Pack for SQL Server 2014. Please refer to the SQL Server 2014 Management Pack Guide for more details.
Discovery and monitoring of SQL Server 2014 roles like DB Engine, Integrations Services.
Discovery of SQL Server 2014 components: Databases, SQL Agent and SQL jobs.
The management pack introduces 100+ monitors and 380+ rules to provide a deep monitoring of SQL Server 2014. Please refer to the SQL Server 2014 Management Pack Guide for the full list of monitoring scenarios and workflow inventory.
Monitoring of SQL Server 2014 AlwaysOn:
Automatic discovery and monitoring of availability groups, availability replicas, and availability databases.
Health roll-up from availability database to availability replicas.
Detailed knowledge for each related monitoring workflow.
Seamless integration with SQL Server 2014 policy based management (PBM):
Automatic discovery and monitoring of custom PBM polices targeted to Database or Database components.
Rollup of policy execution health to the health of related SCOM entity.
Monitoring of SQL Server In-Memory OLTP.
Reports for long-term analysis of different problematic areas related to SQL Server 2014, such as SQL Server lock analysis, top deadlocked databases, SQL Server service pack levels, user connection activity. Also, generic reports from the Microsoft Generic Report Library can be used to review availability and performance of objects discovered by System Center Operations Manager Management Pack for SQL Server 2014.
Low-privilege monitoring is supported for both stand-alone installations and clustered environments, except PBM.
Support for Mirroring Monitoring
Discover mirroring databases, witness, and mirroring group.
